Handling infinite scrolling pages (lazy loading)

A common use case for Page Interactions is to scroll down pages that differ loading off-screen content until a user scrolls down to reveal it. Also known as "lazy loading", websites often do this to reduce page load times or as an alternative to pagination. Some examples include Facebook's newsfeed, Google News, and YouTube's search pages.

Page Interactions include a dedicated feature named infinite_scroll designed specifically to scroll through lazy loading web pages. It's controlled using three parameters:

Parameter	Required	Description	Values	Max
loading_selector	Optional	An identifier of the HTML element that displays the loading/spinner indicator. This helps identify when loading starts and ends.	String. Eg: "#spinner", ".loadingID"	N/A
delay_after_scroll	Optional	The time delay, in milliseconds, between scrolls.	Integer. Default: 200	N/A
duration	Required	A time interval in milliseconds during which scrolling should be continuously attempted.	Integer. Eg: 1000	28000

Scrolling is initiated as soon as the page load event is fired.

Like all Page Interactions, infinite scrolling is capped by the global 120-second session timeout, and will be terminated if this limit is reached.

Infinite Scroll was designed to be used in one of two main methods:

Identifying the loading element

Where possible, it is recommended to use this method, as it allows for more accurate scrolling operation. Use the loading_selector parameter to identify the loading indicator or spinner that is displayed while the page loads additional content, and refine the scrolling behavior with the delay_after_scroll and duration parameters.

Time-based scrolling

In cases where there is no loading indicator, the delay_after_scroll and duration parameters can be used to define and refine the scrolling action until it is well-timed for the particular webpage in question.

Example

cURL

curl -X POST 'https://api.webit.live/api/v1/realtime/web' \
--header 'Authorization: Basic <credential string>' \
--header 'Content-Type: application/json' \
--data-raw '{
    "url": "https://www.example.com",
    "render": true,
    "format": "json",
    "country": "US",
    "parse": true,
    "render_flow": [
        {
            "infinite_scroll": {
                "loading_selector": ".spinner",
                "delay_after_scroll": 2000,
                "duration": 15000
            }
        }    
    ]
}'

Python

import requests

url = 'https://api.webit.live/api/v1/realtime/web'
headers = {
    'Authorization': 'Basic <credential string>',
    'Content-Type': 'application/json'
}
data = {
    "url": "https://www.example.com",
    "render": True,
    "format": "json",
    "country": "US",
    "parse": True,
    "render_flow": [
        {
            "infinite_scroll": {
                "loading_selector": ".spinner",
                "delay_after_scroll": 2000,
                "duration": 15000
            }
        }
    ]
}

response = requests.post(url, headers=headers, json=data)

print(response.status_code)
print(response.json())

Node.js

const axios = require('axios');

const url = 'https://api.webit.live/api/v1/realtime/web';
const headers = {
  'Authorization': 'Basic <credential string>',
  'Content-Type': 'application/json'
};
const data = {
  "url": "https://www.example.com",
  "render": true,
  "format": "json",
  "country": "US",
  "parse": true,
  "render_flow": [
    {
      "infinite_scroll": {
        "loading_selector": ".spinner",
        "delay_after_scroll": 2000,
        "duration": 15000
      }
    }
  ]
};

axios.post(url, data, { headers })
  .then(response => {
    console.log(response.status);
    console.log(response.data);
  })
  .catch(error => {
    console.error(error);
  });

Go

package main

import (
	"bytes"
	"encoding/base64"
	"fmt"
	"net/http"
	"encoding/json"
)

func main() {
	url := "https://api.webit.live/api/v1/realtime/web"
	payload := []byte(`{
		"url": "https://www.example.com",
		"render": true,
		"format": "json",
		"country": "US",
		"parse": true,
		"render_flow": [
			{
				"infinite_scroll": {
					"loading_selector": ".spinner",
					"delay_after_scroll": 2000,
					"duration": 15000
				}
			}
		]
	}`)
	headers := map[string]string{
		"Authorization":  "Basic <credential string>",
		"Content-Type":   "application/json",
	}

	req, err := http.NewRequest("POST", url, bytes.NewBuffer(payload))
	if err != nil {
		fmt.Println(err)
		return
	}

	for key, value := range headers {
		req.Header.Set(key, value)
	}

	client := &http.Client{}
	resp, err := client.Do(req)
	if err != nil {
		fmt.Println(err)
		return
	}
	defer resp.Body.Close()

	fmt.Println(resp.StatusCode)
	// Read the response body if needed
	// body, err := ioutil.ReadAll(resp.Body)
	// fmt.Println(string(body))
}