Browsing SERP pagination

How to access additional pages of results for any search engine.

Using Nimble pagination

Having received the first page of results for a chosen search term, it is often desirable to access the next page of results, available through pagination. Nimble's SERP API makes this easy by providing request-ready paginated URLs at the bottom of each request.

For example:

{
  "status": "success",
  "html_content": "<html>The SERP's full HTML</html>",
  "parsing": {...},
  "url": "https://www.google.com/search?q=Sample+search+phrase&hl=fr",
  "nimble_pagination": {
    "next_page_url": "https://api.webit.live/api/v1/realtime/serp?parse=true&query=test&search_engine=google_search&format=json&render=false&country=FR&locale=fr&ei=vPBQZLakI86ckdUPwOuB6A0&sa=N&ved=2ahUKEwi28uXqwNb-AhVOTqQEHcB1AN0Q8NMDegQIBxAW&start=10",
    "other_pages": [
      "https://api.webit.live/api/v1/realtime/serp?parse=true&query=test&search_engine=google_search&format=json&render=false&country=FR&locale=fr&ei=vPBQZLakI86ckdUPwOuB6A0&sa=N&ved=2ahUKEwi28uXqwNb-AhVOTqQEHcB1AN0Q8tMDegQIBxAE&start=20",
      "https://api.webit.live/api/v1/realtime/serp?parse=true&query=test&search_engine=google_search&format=json&render=false&country=FR&locale=fr&ei=vPBQZLakI86ckdUPwOuB6A0&sa=N&ved=2ahUKEwi28uXqwNb-AhVOTqQEHcB1AN0Q8tMDegQIBxAG&start=30",
      ...
    ]
  }
}

Each of the provided URLs contains identical parameters to the original request, as well as an offset to access paginated pages.

In the below example, a request is written to access the next page's results:

curl --location --request GET 'https://api.webit.live/api/v1/realtime/serp?parse=true&query=test&search_engine=google_search&format=json&render=false&country=FR&locale=fr&ei=vPBQZLakI86ckdUPwOuB6A0&sa=N&ved=2ahUKEwi28uXqwNb-AhVOTqQEHcB1AN0Q8NMDegQIBxAW&start=10' \
--header 'Authorization: Basic <credential string>'

import requests

url = 'https://api.webit.live/api/v1/realtime/serp?parse=true&query=test&search_engine=google_search&format=json&render=false&country=FR&locale=fr&ei=vPBQZLakI86ckdUPwOuB6A0&sa=N&ved=2ahUKEwi28uXqwNb-AhVOTqQEHcB1AN0Q8NMDegQIBxAW&start=10'
headers = {
    'Authorization': 'Basic <credential string>'
}
params = {}

response = requests.get(url, headers=headers, params=params)

print(response.status_code)
print(response.json())

const axios = require('axios');

const url = 'https://api.webit.live/api/v1/realtime/serp?parse=true&query=test&search_engine=google_search&format=json&render=false&country=FR&locale=fr&ei=vPBQZLakI86ckdUPwOuB6A0&sa=N&ved=2ahUKEwi28uXqwNb-AhVOTqQEHcB1AN0Q8NMDegQIBxAW&start=10';
const headers = {
  'Authorization': 'Basic <credential string>'
};
const params = {};

axios.get(url, { headers, params })
  .then(response => {
    console.log(response.status);
    console.log(response.data);
  })
  .catch(error => {
    console.error(error);
  });

package main

import (
	"fmt"
	"net/http"
	"io/ioutil"
)

func main() {
	url := "https://api.webit.live/api/v1/realtime/serp?parse=true&query=test&search_engine=google_search&format=json&render=false&country=FR&locale=fr&ei=vPBQZLakI86ckdUPwOuB6A0&sa=N&ved=2ahUKEwi28uXqwNb-AhVOTqQEHcB1AN0Q8NMDegQIBxAW&start=10"
	headers := map[string]string{
		"Authorization": "Basic <credential string>",
	}

	resp, err := http.Get(url)
	if err != nil {
		fmt.Println(err)
		return
	}
	defer resp.Body.Close()

	body, err := ioutil.ReadAll(resp.Body)
	if err != nil {
		fmt.Println(err)
		return
	}

	fmt.Println(resp.StatusCode)
	fmt.Println(string(body))
}

All of the original request's parameters are transmitted through the URL automatically, with the exception of your credential string, which needs to be manually reapplied.

Infinite scrolling SERPs

Search engines often alternate between pagination and infinite scrolling methods depending on location, device, search term, and other variables. In cases where the SERP you requested uses infinite scrolling instead of pagination, only a single URL next_page_url will be provided, which leads to the next batch of results, as in the example below:

{
  "status": "success",
  "html_content": "<html>The SERP's full HTML</html>",
  "parsing": {...},
  "url": "https://www.google.com/search?q=Sample+search+phrase&hl=fr",
  "nimble_pagination": {
    "next_page_url": "https://api.webit.live/api/v1/realtime/serp?parse=true&query=test&search_engine=google_search&format=json&render=false&country=FR&locale=fr&ei=vPBQZLakI86ckdUPwOuB6A0&sa=N&ved=2ahUKEwi28uXqwNb-AhVOTqQEHcB1AN0Q8NMDegQIBxAW&start=10"
  }
}

PreviousGetting local data NextDelivery methods

Last updated 4 months ago