How to Scrape Q&A Sites like Quora

Question-and-Answer sites like Quora, StackOverflow, AskMeHelpDesk, Reddit (partially) are online socialization hubs for digital citizens across the world to ask, answer, and discuss the most prominent issues, doubts, and topics. Extracting data from these online Q&A platforms can be very useful to marketers and data scientists alike. Especially Quora. As it’s not only a multilingual Q&A website but also a social network in itself with many niche influencers. Read this insight to find out how to scrape Quora Q&A for free.

Table of Contents

Why Scrape Quora? Use-cases of Scraping Quora

To emphasize why scraping Quora is of interest to marketers and businesses, let’s take a quick peek at 4 vital Quora stats:

Quora is home to 300 million monthly active users.
On average users spend 4+ minutes on Quora every day.
From traffic volumes, it is the 80th most popular website in the world.
Google search shows as many as 65 million results for Quora[dot]com.

Businesses scrape Quora for the following reasons:

Sentiment analysis

You can scrape questions related to politics, brands, stock market etcetera to perform sentiment analysis.

NLP & machine learning

Most of the users on Quora are real users, who ask questions and answers on the platform in their day-to-day lingo. This could be very useful for training ML models, and natural language processing (NLP).

Intelligent influencer marketing

Quora allows you to run ads but you can also tag upof with influencers in a particular niche to promote your brand. Scraping questions, user profiles etcetera from a specific niche would enable you to partner with the right influencers who have real authority to promote your brands.

Lead generation & content marketing

Questions asked by users can help you identify if they are your target leads. For instance, if you’re an IT services company then people who ask questions like “How much it costs to develop an e-commerce website?” are your potential leads. Insights gained from scraping Quora Q&As can also be your gateway to a stellar content marketing strategy.

How to Scrape Quora for Free?

You can scrape Quora using:

Open source Python Quora-scraper
Using Point and click scraping automation tools

High customizability is the benefit of using open-source scripts or writing your own script to scrape Quora Q&As. But the drawback is open-sourced scripts are not regularly updated, while the site’s structure keeps changing. To make any customizations you need to be acquainted with programming. The easier solution is using free click and scrape tools like Octoparse.

Scrape Quora Using Octoparse

Now, let’s extract data from the Quora Stock markets feed using Octoparse.

Target URL: https://www.quora.com/topic/Stock-Markets-2

Pre-requisites:

Download Octoparse

Step 1: Start a new scraping template by clicking on the +New button and then on Advanced Mode.

Step 2: Enter the target URL in the website form-field, and click on Save.

Step 3: Click on Auto-detect web page data.

The auto-detect feature automatically checks for infinite scroll features, it picks up post patterns and auto-suggests you labels for data too.

When auto-detect is 100% processed, it presents you with detected data and an option to create a workflow. Click-tick Add a page scroll feature and then on Create workflow.

Yay! Your quora topic scraping template is ready.

But we still need to do a few customizations. In the workflow panel, click on Action settings:

And update the data-field labels. Here’s what all we’ve scraped:

Question
Question URL
Image URL
User Profile
User Name
Update Date
User Tagline
Question Description

Click on save and run at the top to save this workflow and start scraping:

You can run the task on your local device, and in the cloud. You can also schedule scrapers while scraping in the cloud. This gives you added flexibility of scraping anytime, anywhere, any number of times.

This fires up scraping and when the scraping gets completed, you can export the data from Octoparse in CSV, XLS, JSON, HTML formats or you can store it in a database of your choice (locally/cloud).

Here’s how the data looks after downloading it in JSON and previewing it in an online JSON prettifier:

Why Choose Octoparse as Your Scraping Tool?

Octoparse is a market-leading AI-powered web scraping tool. It is free, intuitive, and extensively documented. You also have premium plans for scalable scraping with Octoparse. Additionally, for enterprise support, you can avail of Octoparse’s data scraping services. Here are some of the features that make Octoparse a stellar web scraping solution:

Cloud scraping

You don’t need to set up any in-house hardware or manpower infrastructure. Octoparse gives you the flexibility to scrape in the cloud.

Bypass anti-scraping architectures

You can rotate IP proxies, and user-agents to overcome simple anti-scraping techniques. If you face Captcha issues, you can switch to browse mode, solve the captcha and resume scraping.

Supports AJAX and Infinite Scrolls

As you would have observed, while scraping Quora, Octoparse automatically took care of infinite scroll. But suppose you want to implement it manually, you can do so by following this tutorial.

Schedule Scrapers

While scraping in the cloud, you can also schedule your scrapers to scrape at specific times or you can set a frequency to scrape at regular intervals/frequency.

Pre-built templates

Octoparse provides premium users to leverage 100s of pre-built templates spanning over different domains like E-commerce, social media, finance, search, entertainment, etcetera.

Available for both MacBook & Windows OS

Lastly, the Octoparse web scraping tool is available for both Mac and Windows users. It takes only seconds to start scraping your target data.

Conclusion

Social media is a goldmine for user-generated data. Scraping Quora Q&As is like gaining access to the pain points of your customers, the likes/dislikes/interests of your audience. Using an intelligent scraping tool takes away all your pains associated with scraping Quora data. Once you’ve extracted your data you can run neural networks-powered ML algorithms and gain business-critical insights. Here are some more scraping resources for you:

Happy scrapping!