7 Ways to Find Content Ideas from the Web and Create Like a Pro

Content creators are always stressed out to come up with new ideas. Regular updates are essential. While if you have been focused on a niche for some time, it’s natural to become numb on the topic. 

You need more perspectives and new triggers to open up your mind. A web scraping tool and a list of valuable sources could be extremely helpful in this case. 

Who Will Benefit from This Article

  • Youtube creators looking for new topics to make videos 
  • Blog writers seeking new ideas to write
  • Content marketers wanting to source ideas efficiently
  • Inbound marketing professionals looking for SEO-friendly topics

Modern people are flooded with opinions and content every day. How do we generate ideas from them? 

Reading is not practical. We need a structured way to spy, analyze and extract the values out of them. A web scraping tool can help build the database where writers/creators could filter through and get inspired.

On top of all the places where you can source ideas, I may also present how web scraping can improve the process. 

7 Ways to Curate Content Ideas

  1. Youtube Channel Crawler
  2. Study Google SERP 
  3. Listen to social media discussions
  4. Spy on Competitors’ Blog
  5. Idea generator
  6. Keyword planning tools (Semrush/Ahrefs)
  7. Look back at what you have created

Youtube Channel Crawler

A few months ago, I shared a post about how to use a Youtube channel crawler to review your channel and spy on competitors’.

In this way, you can grab topic data (titles, tags, description) and performance data (views, tags, likes) of published videos. Filtering through the data, you will know what you shall focus on: the well-performing topics that your channel has never touched.

Always keep up with the enemy. 

A topic performs well on another channel and you haven’t made any out of it yet? That’s potential. Spot the gap and fill it with a video better than any of the existing. 

Besides, comments under the video left by the viewers are important signals too, through which you may evaluate how intriguing the topic is to your target audience and learn what you can improve to rank over your enemy.

Study Google SERP

A Google SERP is helpful if you want to generate SEO-performing content. 

  • Top-ranking articles on the result page tell us what people are looking for. Always match your content with the need of your audience.
  • Title and description (and the query you are searching for) shed light on what topics are welcome and what words or phrases can be utilized as SEO-friendly keywords.
Google SERP Crawler Works for SEO Benefits 

Use your general idea as the query to search on Google and the search results help you niche down to a more targeted scope. 

Listen to Social Media Discussions

Posts discussing how to source content ideas won’t miss out on social media platforms, Youtube, Instagram, Reddit, just to name a few. That’s true, they are valuable and resourceful. Conversations reveal what people think and care about.

Social media and Q&A platforms to source ideas:

  • Twitter
  • Instagram
  • Facebook
  • Youtube
  • Medium
  • Reddit
  • Quora
  • Niche communities (eg. Stackoverflow for techies)

However, the key is to find a place where your target audience gathers. Use hashtags, groups, subreddits, channels, and direct search to filter out the insignificant noises and focus on the voices of your potential audience.

Spy on Competitors’ Blog

If you are a Youtuber, keep a close eye on your competitor’s channel. Same for a blogger. And do not limit yourself to “rival competitors”, any websites or channels that share a group of the audience are worth attention. They help us jump out of the box and see what extra values we can offer to our audience.

Idea Generator

There are web-based tools designed for creators to find related topics. Most of them use algorithms to put recurring words into certain patterns and generate questions or titles for writers to write on. The titles/ideas offered may not be the perfect fit to use directly, while it can broaden your mind sometimes.

>>Question generator: AnswerthePublic

>>Title generator: HubspotTopic

Keywords Planning Tool

The keyword is so SEO-oriented that it is always mistaken as magic for high traffic. Instead, keywords and their popularity do not guarantee traffic but work as a signal to tell you what people are seeking. Hence, a keywords planning tool is an instrument for content marketers to find out customer needs.

A good content idea is not necessarily new, original, or fancy, but it must meet the real needs of your audience.

SEMrush and Ahrefs are good tools to explore the keywords, find out what people are looking for on the Internet. There are keyword tools tailored for Youtubers like Socialblade, KeywordTool.

Look Back at What You Have Created

A super influencer with 1 million followers may not care much about SEO stuff. The new release will be dispensed to his/her followers anyway. Social sharing plays a big role and a good ranking is just a result.

A growing channel is different. SEO matters a lot. Analyzing what you have published and how they performed is important to understand who your audience is and what kind of content attracts them.

If you are running a blog, make sure to set up Google Analytics and Google Search Console to monitor the performance of your posts and make adjustments accordingly.


You may source content ideas as an observer, browsing the existing posts, reviews, blogs, and videos. Make sure the quality and quantity of your input do matter. On the other hand, creating opportunities to have direct interaction with your audience like holding an interview, starting a live conversation, may inspire you in a new way.

Cecilia W

I am a content writer and a digital marketer in data science who believes in the power with which data can endow people’s businesses. I am working on writings that convey real values. If you have any feedback or ideas about web scraping and data analytics, talk to me at cecilia@octoparse.com.