Google Scraper: How To Scrape SERPs

If you aren’t a coder but you need to scrape data from Google for your competitor analysis and growth hacking strategies, Octoparse can help you with that.

Here’s a crystal clear explanation of how you can get the data you need in no time without any coding skills involved.

We’ll give you a step-by-step guide on how you can gather the competitor information you need together with some pointers to keep in mind so Google’s detection system doesn’t get in the way.

But first, we’ll show you why data scraping is a must for your eCommerce business.

If you’re ready to gather some data, let’s get right to it.

Table of Contents

What SERP Scraping Can Do For Your eCommerce Business

The best decisions are made when data is involved.

In a 2019 report by Statista, they stated that search traffic accounts for 29% of the worldwide website traffic. This shows that there is valuable data in search engine result information and more than a quarter of people get to where they want to go on the internet with its help.

If that’s the case, gathering the public information available on search results will help your listings become more visible to searchers and in return bring more traffic to your eCommerce store.

If you can determine and understand what type of information people are looking for on the internet using SERP scraping, you’ll be able to cater your web content to those needs.

For example, if you search the keyword “Halloween costumes, instead of eCommerce stores selling these items, you’ll see blogs giving you ideas about the keyword. This way, you can tailor your content for this search intent – finding ideas for Halloween costumes.

This gives more reasons for Google to rank you high, even higher than your competitors. Think about it:

There’s limited real estate on the first page of the SERP, you want to get above your competitors so you can get part of the traffic share to drive revenue to your business.

Especially for SaaS-like ones providing business solutions for eCommerce stores like a predictive dialer or a Woocommerce cart abandonment email templates, there is a lot of competition, so you have to get the right keywords and use them better than the competitors.

What Data Can You Get From Scraping Search Engine Results?

The most important data you can get from scraping the search engine results are keywords. Not just any keywords, but the keywords that help your competitors rank high on the first page.

You’ll especially need this if you’re new in the industry just like a paddleboard company called Gili Sports. You need to find high-ranking keywords that the competitors use so you can take advantage of them too.

Using data scraping, you will get an idea of their best practices that helped them get the rank. You can then use this information and apply it to your digital marketing strategies.

If you’re working with an SEO agency or an ad designer, you can even give them this scraped data so they can align their efforts with what’s already working for the competitors. So all they have to do is to help you make your content better.

Additionally, by scraping Google’s SERP, your eCommerce business will be able to provide relevant information for people searching for specific keywords, like mentioned at the outset.

In some cases, there’s a mismatch between the keyword and the information you have available for that keyword. In other words, you didn’t go with the search intent for that keyword.

For example, if you’re searching for the keyword “tricep exercise” you’ll find blogs catered for beginners or those who want to do it at home, knowing this, you’ll make your content similar to it.

Or if you have an inventory software business and your main offer is barcode software, with this keyword in mind the content you’ll be making should be a list of the best barcode software available since that’s what you’ll see on the SERPs.

The sooner you find the mismatch, the sooner you’ll be able to adjust and refocus your SEO efforts and content strategy to get in line with the search intent. This way, your content shows up with the right information people are looking for.

SERP Scraping Helps With SEO Monitoring

Primarily, SERP scraping helps with your SEO efforts. By scraping the SERP, you’ll get information such as:

Meta titles
Meta descriptions
Knowledge graphs
Rich snippets

Your content team can then analyze these data and be able to pattern their efforts to what works effectively.

It tells them where the keyword should be placed, what additional information should be included in the description, what visuals will help you get ranked in the featured snippets. This way, you’ll get more chances of getting ranked high on the first page.

For example, if your business is about providing insight for website design, scraping data for “website design” will get you to realize your meta description should include keywords like “professionally-designed” and “user experience”.

SERP Scraping Helps With Digital Advertising

If you’re taking advantage of paid ads for your brand awareness campaigns, scraping the SERP will help you get information like: where and when competitor ads show up.

This way, you can monitor the market and also pattern your efforts with their best practices.

Take this ad data for the keyword “lightroom presets”, it contains important keywords together with pricing you can compare your products with.

SERP Scraping Helps Optimize Shopping Results

In many cases, when a user searches for a specific keyword, the intent is to find a product or an alternative to the product they can buy.

Scraping these data which includes the prices, titles, descriptions, and keywords used will optimize your listing to show up when users search for your competitor or the keywords they utilize.

Say you’re selling Mexican blankets, scraping the SERP will tell you the need to include the product’s fabric type, pattern, color, brand, and review rating.

SERP Scraping Helps You Get Credible External Links For Your Content

When researching for information to back up what you say or explain in your content, SERP scraping will help you get credible sources for it.

Search engines like Google have an academic version of their search engines which researchers use.

You can take advantage of this too minus reading all the information that takes a whole lot of time. You can gather keywords, titles, links, snippets, and citations you can use for your content to make it more reliable.

Now that we know what information you can get from SERP scraping and how it can help your eCommerce business, let’s move right into the best part – the steps in scraping Google using Octoparse.

How To Scrape From Google SERP Using Octoparse

SERP scraping used to require coding skills, but not anymore, Octoparse has made it easier to scrape SERP information and compile it all in an organized spreadsheet ready for you to use.

It might seem overwhelming at first glance. It’s easier than it looks.

Just follow the steps closely and refer to the images and you’ll get the information you need in no time.

Step 1: Open The Target Web Page

In this case, the target web page is Google.com so the text box type “https://www.google.com/” and click on the Start button.

Step 2: Verify Parameters

Just so everything is in place, you must verify a couple of things first before entering the keyword you want to scrape in the search bar.

You can skip this step if the primary language you use is English. If not, say it’s Japanese, German, French, or any other language, here’s what you need to do first to get English results.

To do this, scroll down on the window showing Google and click on the word “Parameters” or whatever it translates to in your language.

Next click on “Search Parameters” then click on the circle before the word “English”, then lastly click on “Save”.

Another thing you’d likely see is this popup saying you need to reinstall Chrome to stay secure.

To get this out of the way, you can change the “User Agent”. You’reclicking scraped scrolling the target web page what device you are accessing the page with. In other words, you are telling Octoparse that you are accessing the page using Google Chrome.

How do you do it? Here’s how:

Click on the “Settings” icon next to the “Save” and “Run” buttons.

When a new page pops up, click on the dropdown menu for the Browser version and change it to “Chrome 77.0”. Then click “Save” and the popup doesn’t show up anymore.

Step 3: Input The Keyword

Now we can start to input the keyword we want to scrape data for.

To do that, click on the “Workflow” tab on the upper left side of the window then click on the “Pin” icon so it doesn’t budge and you can control the workflow settings easily.

Next, click on the search bar until it’s highlighted in green. With it, a “Tip” box appears, on it, click on the “Enter Text” and type the keyword you want to scrape for, then click confirm. You’ll then see it typed up in the Google search bar.

Now at the bottom of the suggested long-tail keywords list click on “Google Search”, then go back to the “Tips” box and click on the “Click Button”.

After that, you’ll see the results page show up.

Step 4: Modify The Settings For The Click Item

On the “Workflow” box, click on the action settings icon on the “Click item” button. Then on the “Click Options” list, check the box on the “Open in a new tab”.

Why do we need to do this?

Because in many cases, the URL after the search is different from the URL before it. This is what we call an Ajax website. To finish, just click on “Ok”.

Step 5: Input Multiple Keywords

Now if you want to scrape data for multiple keywords simultaneously, we need to build a “Loop item”.

For this, you have to click on the “Plus” icon below the “Click item” button located on the “Workflow” box. A list appears and clicks on “Loop”.

A “Loop item” box appears, click on the action settings. In the action, settings change the “Loop mode” from “List of URLs” to “Text list” in the dropdown menu.

Below it, click the pen and paper icon and type down all the keywords you want to scrape. 1 set of keywords per line. Then click “Confirm” and then click “Ok”.

It doesn’t matter if the keywords are unrelated, you can scrape for productivity tools, coupon statistics, HubSpot CRM, or any unrelated keywords, you’ll still be able to determine them from each other later in the spreadsheet anyway.

Now you get access to the “Loop item”. The next thing you need to do is to change the steps in the right order.

To do this move the “Loop Item” box right below the “Go to Web Page 1” button. Then inside the “Loop Item” box put the “Enter text” button and the “Click item” button below it.

It should look something like this:

Step 6: Setup The Most Important Setting

This step involves clicking on the action setting icon on the “Enter text” button, then clicking on the text box for “Use text in the loop to enter the text box” and finishing off by clicking “Ok”.

use text in the loop to enter the text box

Doing this makes the scraping for multiple items work. If now, only the first item or the first keyword on the list will be scraped.

Step 7: Select The Data You Want To Extract

To start extracting data, click on the round blue button with the “I” icon on it. The “Tips” box appears again and on it click the “Auto-detect web page data” then the scraped search results will show up.

Now back on the “Tips” box, uncheck the box for the “Add a page scroll” then click on the “Create workflow” button.

The next thing you need to do is to delete the fields or data items you don’t need by clicking on the action settings on the “Extract Data” button. Click on the “Trash” icon for the fields you don’t need then click “Ok”.

Now you need to include the meta description on your scraped results. To do this, click on the meta description where there is a red highlight then click on the “Extract the text of the selected element”.

It then appears on the scraped results. Chance the field name from “Text” to “Meta Description” then click enter.

A pointer to keep in mind when extracting data is to always select data within the red highlighted area. Clicking anywhere else will not work.

So the next step will involve what you have to do so Google’s anti-scraping techniques don’t get triggered.

Step 8: Slow Down The Scraping Speed

As said at the outset, Google has applied anti-scraping techniques. To prevent triggering the ReCaptcha issue, we need to slow down the scraping speed.

You can do this by clicking on the action settings for the “Extract Data” button, scrolling to the bottom, and under the items on “Before action is performed”, click on the checkbox for “Wait before action”.

Then set the waiting time to 3 seconds. Lastly, click “Ok” to save the changes.

Step 9: Add Page URLs

We’re almost done. At this point, we need to add the page URL since we are scraping for multiple items to avoid getting confused.

Click on the action settings for the “Extract Data” button again. Under the “Define date fields” click on the “+” icon on the upper right then click on “Page-level data” and click on “Page URL”. When done, click “Ok”.

Now we’re done with the setup stage, it’s time to run the task.

Step 10: Run The Task

To run the task, click on the “Save” button located above the “Workflow” box then click on the “Run” button just beside it.

You can either run it in the cloud or on your device.

After choosing where to run it, you’ll see the extracted data that you can export simply by clicking on the “Export Data” button on the bottom right corner of the window that showed up.

At this point you can choose to export us to an Excel file, a CSV, an HTML, or a JSON, then click on “Ok” after choosing.

Now you have your results ready to use. The next time your digital marketing team needs data presented for the next scheduled meeting, you’ll have yours comprehensively scraped up.

Conclusion

Extracting data from search results doesn’t have to be rocket science. By following the steps closely, you’ll be able to get all the information you need quickly, so you can focus on the more important things – improving your content strategy to bring in more traffic for your website.

The next time you need to gather data for your digital marketing efforts won’t be a problem anymore as long as you know how to scrape data the right way.