Requirements:

Target:

We will target to scrape the shopping results of Nike shoes.

Process:

Now, all the things are installed that we will need to prepare our scraper. We will use an npm library Unirest JS to make a get request to our target URL so we can get our raw HTML data. Then we will use Cheerio JS for parsing the extracted HTML data.
We will target this URL:

                         
                            https://www.google.com/complete/search?&hl=en&q=coffee&gl=us&client=chrome
                         
                        

Copy this URL in your browser and press enter. You will see a text file downloading in your browser by entering this URL in your browser. Open this file in your respective code editor. We will now first convert this JSON string into object and get the respective data we need.

                              
    const unirest = require("unirest");

    const searchSuggestions = async () => {
        try {
        const response = await unirest
            .get("https://www.google.com/complete/search?&hl=en&q=coffee&gl=us&client=chrome")
            .headers({
            "User-Agent":
                "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/101.0.4951.54 Safari/537.36",
            })
    
        let data = JSON.parse(response.body);
        let suggestions = [];
        for (let i = 0; i < data[1].length; i++) {
            suggestions.push({
            value: data[1][i],
            relevance: data[4]["google:suggestrelevance"][i],
            type: data[4]["google:suggesttype"][i],
            });
        }
        const verbatimRelevance = data[4]["google:verbatimrelevance"]
    
        console.log(suggestions)
        console.log(verbatimRelevance)
        
        } catch (e) {
        console.log("Error : " + e);
        }
    };
    
    searchSuggestions();
                              
                          

You can also check some of my other Google scrapers of in my Git Repository: https://github.com/Darshan972/GoogleScrapingBlogs

Result:

                                
       [
        { value: 'coffee near me', relevance: 1250, type: 'QUERY' },
        { value: 'coffee shops near me', relevance: 650, type: 'QUERY' },
        { value: 'coffee shop', relevance: 601, type: 'QUERY' },
        { value: 'coffee table', relevance: 600, type: 'QUERY' },
        { value: 'coffee maker', relevance: 553, type: 'QUERY' },
        { value: 'coffee bean', relevance: 552, type: 'QUERY' },
        { value: 'coffee grinder', relevance: 551, type: 'QUERY' },
        { value: 'coffee meets bagel', relevance: 550, type: 'QUERY' }
       ]
        1300                               
                                
                             

Our result should look like this 👆🏻.

With Google Autocomplete API

If you want to scrape Google Autocomplete Results easily, without making a scraper, as scraping can take a lot of time sometimes, you can try this API.
Serpdog also provides 100 free searches per month, and if you want to scale your requests quota, you can buy paid plans also.

                        
    const axios = require('axios');

    axios.get('https://api.serpdog.io/autocomplete?api_key=APIKEY&q=football&gl=us')
        .then(response => {
        console.log(response.data);
        })
        .catch(error => {
        console.log(error);
        });
                        
                    

Results:

                    
   {
    "meta": {
        "api_key": "APIKEY",
        "q": "football",
        "gl": "us"
    },
    "suggestions": [
        {
        "value": "football cleats",
        "relevance": 601,
        "type": "QUERY"
        },
        {
        "value": "football games",
        "relevance": 600,
        "type": "QUERY"
        },
        {
        "value": "football wordle",
        "relevance": 555,
        "type": "QUERY"
        },
        {
        "value": "football today",
        "relevance": 554,
        "type": "QUERY"
        },
        {
        "value": "football gloves",
        "relevance": 553,
        "type": "QUERY"
        },
        {
        "value": "football field",
        "relevance": 552,
        "type": "QUERY"
        },
        {
        "value": "football movies",
        "relevance": 551,
        "type": "QUERY"
        },
        {
        "value": "football positions",
        "relevance": 550,
        "type": "QUERY"
        }
    ],
    "verbatim_relevance": 1300
    }   
                    
                

Conclusion:

In this tutorial, we learned to scrape Google Autocomplete Suggestions using Node JS. Feel free to message me anything you need clarification on. Follow me on Twitter. Thanks for reading!

Author:

My name is Darshan, and I am the founder of serpdog.io. I love to create scrapers. I am currently working for several MNCs to provide them with Google Search Data through a seamless data pipeline.