Page Interactions

Nimble Labs Beta Feature
Page Interactions allow users to perform operations on a webpage before the web data is collected and returned including clicking, typing, scrolling, and more.
This is useful (and sometimes necessary) in a variety of situations, such as one-page applications that use lazy loading and require scrolling in order to load the desired data. Another example includes E-commerce websites that require button clicks or zip code input to display a product price.
A visual representation of some of the capabilities of Page Interactions
Currently, Page Interactions are synchronous, so operations are run sequentially one by one, and the overall sequence is limited to a maximum timeout of 60 seconds. Page Interactions are supported by real-time, asynchronous, and batch requests.
Using Page Interactions
To perform page interactions, set render to true and add a render_flow argument with a JSON list of the operations you’d like to perform, as in the example below:
cURL
Python
Node.js
Go
curl -X POST 'https://api.webit.live/api/v1/realtime/web' \
--header 'Authorization: Basic <credential string>' \
--header 'Content-Type: application/json' \
--data-raw '{
    "url": "https://www.example.com",
    "render": true,
    "format": "json",
    "country": "US",
    "parse": true,
    "render_flow": [
        {
            "wait": {
                "delay": 500
            }
        }
    ]
}'
import requests
​
url = 'https://api.webit.live/api/v1/realtime/web'
headers = {
    'Authorization': 'Basic <credential string>',
    'Content-Type': 'application/json'
}
data = {
    "url": "https://www.example.com",
    "render": True,
    "format": "json",
    "country": "US",
    "parse": True,
    "render_flow": [
        {
            "wait": {
                "delay": 500
            }
        }
    ]
}
​
response = requests.post(url, headers=headers, json=data)
​
print(response.status_code)
print(response.json())
const axios = require('axios');
​
const url = 'https://api.webit.live/api/v1/realtime/web';
const headers = {
  'Authorization': 'Basic <credential string>',
  'Content-Type': 'application/json'
};
const data = {
  "url": "https://www.example.com",
  "render": true,
  "format": "json",
  "country": "US",
  "parse": true,
  "render_flow": [
    {
      "wait": {
        "delay": 500
      }
    }
  ]
};
​
axios.post(url, data, { headers })
  .then(response => {
    console.log(response.status);
    console.log(response.data);
  })
  .catch(error => {
    console.error(error);
  });
package main
​
import (
	"bytes"
	"encoding/base64"
	"fmt"
	"net/http"
	"encoding/json"
)
​
func main() {
	url := "https://api.webit.live/api/v1/realtime/web"
	payload := []byte(`{
		"url": "https://www.example.com",
		"render": true,
		"format": "json",
		"country": "US",
		"parse": true,
		"render_flow": [
			{
				"wait": {
					"delay": 500
				}
			}
		]
	}`)
	headers := map[string]string{
		"Authorization":  "Basic <credential string>",
		"Content-Type":   "application/json",
	}
​
	req, err := http.NewRequest("POST", url, bytes.NewBuffer(payload))
	if err != nil {
		fmt.Println(err)
		return
	}
​
	for key, value := range headers {
		req.Header.Set(key, value)
	}
​
	client := &http.Client{}
	resp, err := client.Do(req)
	if err != nil {
		fmt.Println(err)
		return
	}
	defer resp.Body.Close()
​
	fmt.Println(resp.StatusCode)
	// Read the response body if needed
	// body, err := ioutil.ReadAll(resp.Body)
	// fmt.Println(string(body))
}
​
Page Interactions will not function if rendering is not enabled in the request. Please be sure to set render:true in order for interactions to function correctly.
Response
When a request with Page Interactions is completed, an additional property named render:true is included that contains a log detailing the result of each interaction. For each interaction, the following fields are present:
Name – the interaction type (eg. scroll, wait_and_click, wait_and_type, etc).
Result – some operations may return information. If there is information to return, result will contain the returned value, otherwise result will be true.
Error – this field is only present if an error occurs, and details the type of error encountered.
Status – the status of the interaction. Interactions can have one of four statuses:
Status
Description
no-run
the interaction was not performed.
in-progress
The interaction was in progress when the global 60-second timeout was triggered.
done
the interaction finished successfully.
error
the interaction encountered an error.
The above request would produce the following response:
{
    "status": "success",
    "html_content":"...",
    "render_flow": {
        "success": true,
        "results": [
            {
                "name": "wait",
                "status": "done",
                "result": true
            }
        ]
    }
}
Chaining Page Interactions
Multiple interactions can be chained together and performed sequentially. To do this, simply add additional steps to the render:true property. When chaining interactions, it’s important to consider the maximum overall execution time of all the requested interactions.
Each request has a maximum timeout of 60 seconds for the overall execution, which includes all render flow interactions such as delays, clicks, scrolls, and timeouts if and when they occur.
Additionally, if any particular interaction encounters an error, interactions that come after it will not be executed. The chain is halted, and the data from the webpage is returned normally.
Below is an example of a request with several interactions chained together in a sequence:
cURL
Python
Node.js
Go
curl -X POST 'https://api.webit.live/api/v1/realtime/web' \
--header 'Authorization: Basic <credential string>' \
--header 'Content-Type: application/json' \
--data-raw '{
    "url": "https://www.example.com",
    "render": true,
    "format": "json",
    "country": "US",
    "parse": true,
    "render_flow": [
        {
            "wait_and_type": {
                "selector": "input[type='\''search'\'']",
                "timeout": 2000,
                "value": "eggplant",
                "click_on_element": true
            }
        },
        {
            "wait_and_click": {
                "selector": "#__next > div:nth-child(1) > div > span > header > form > div > button.absolute.bn.br-100.bg-gold.h3.w3 > i",
                "timeout": 2000
            }
        },
        {
            "wait_and_click": {
                "selector": "#maincontent > main > div > div:nth-child(2) section:nth-child(9) > div > h3 > button",
                "timeout": 5000,
                "scroll": true
            }
        },
        {
            "wait_and_click": {
                "selector": "input[name='\''Organic'\'']",
                "timeout": 2000
            }
        },
        {
            "wait": {
                "delay": 5000
            }
        }
    ]
}'
import requests
​
url = 'https://api.webit.live/api/v1/realtime/web'
headers = {
    'Authorization': 'Basic <credential string>',
    'Content-Type': 'application/json'
}
data = {
    "url": "https://www.example.com",
    "render": True,
    "format": "json",
    "country": "US",
    "parse": True,
    "render_flow": [
        {
            "wait_and_type": {
                "selector": "input[type='search']",
                "timeout": 2000,
                "value": "eggplant",
                "click_on_element": True
            }
        },
        {
            "wait_and_click": {
                "selector": "#__next > div:nth-child(1) > div > span > header > form > div > button.absolute.bn.br-100.bg-gold.h3.w3 > i",
                "timeout": 2000
            }
        },
        {
            "wait_and_click": {
                "selector": "#maincontent > main > div > div:nth-child(2) section:nth-child(9) > div > h3 > button",
                "timeout": 5000,
                "scroll": True
            }
        },
        {
            "wait_and_click": {
                "selector": "input[name='Organic']",
                "timeout": 2000
            }
        },
        {
            "wait": {
                "delay": 5000
            }
        }
    ]
}
​
response = requests.post(url, headers=headers, json=data)
​
print(response.status_code)
print(response.json())
const axios = require('axios');
​
const url = 'https://api.webit.live/api/v1/realtime/web';
const headers = {
  'Authorization': 'Basic <credential string>',
  'Content-Type': 'application/json'
};
const data = {
  "url": "https://www.example.com",
  "render": true,
  "format": "json",
  "country": "US",
  "parse": true,
  "render_flow": [
    {
      "wait_and_type": {
        "selector": "input[type='search']",
        "timeout": 2000,
        "value": "eggplant",
        "click_on_element": true
      }
    },
    {
      "wait_and_click": {
        "selector": "#__next > div:nth-child(1) > div > span > header > form > div > button.absolute.bn.br-100.bg-gold.h3.w3 > i",
        "timeout": 2000
      }
    },
    {
      "wait_and_click": {
        "selector": "#maincontent > main > div > div:nth-child(2) section:nth-child(9) > div > h3 > button",
        "timeout": 5000,
        "scroll": true
      }
    },
    {
      "wait_and_click": {
        "selector": "input[name='Organic']",
        "timeout": 2000
      }
    },
    {
      "wait": {
        "delay": 5000
      }
    }
  ]
};
​
axios.post(url, data, { headers })
  .then(response => {
    console.log(response.status);
    console.log(response.data);
  })
  .catch(error => {
    console.error(error);
  });
package main
​
import (
	"bytes"
	"encoding/base64"
	"fmt"
	"net/http"
)
​
func main() {
	url := "https://api.webit.live/api/v1/realtime/web"
	payload := []byte(`{
		"url": "https://www.example.com",
		"render": true,
		"format": "json",
		"country": "US",
		"parse": true,
		"render_flow": [
			{
				"wait_and_type": {
					"selector": "input[type='search']",
					"timeout": 2000,
					"value": "eggplant",
					"click_on_element": true
				}
			},
			{
				"wait_and_click": {
					"selector": "#__next > div:nth-child(1) > div > span > header > form > div > button.absolute.bn.br-100.bg-gold.h3.w3 > i",
					"timeout": 2000
				}
			},
			{
				"wait_and_click": {
					"selector": "#maincontent > main > div > div:nth-child(2) section:nth-child(9) > div > h3 > button",
					"timeout": 5000,
					"scroll": true
				}
			},
			{
				"wait_and_click": {
					"selector": "input[name='Organic']",
					"timeout": 2000
				}
			},
			{
				"wait": {
					"delay": 5000
				}
			}
		]
	}`)
	headers := map[string]string{
		"Authorization":  "Basic <credential string>",
		"Content-Type":   "application/json",
	}
​
	req, err := http.NewRequest("POST", url, bytes.NewBuffer(payload))
	if err != nil {
		fmt.Println(err)
		return
	}
​
	for key, value := range headers {
		req.Header.Set(key, value)
	}
​
	client := &http.Client{}
	resp, err := client.Do(req)
	if err != nil {
		fmt.Println(err)
		return
	}
	defer resp.Body.Close()
​
	fmt.Println(resp.StatusCode)
	// Read the response body if needed
	// body, err := ioutil.ReadAll(resp.Body)
	// fmt.Println(string(body))
}
When chaining interactions, the response will list a success/failed status for each interaction:
{
    "status": "success",
    "html_content":"...",
    "render_flow": {
        "success": true,
        "results": [
            {
                "name": "wait_and_type",
                "status": "done",
                "result": true
            },
            {
                "name": "wait_and_click",
                "status": "done",
                "result": true
            },
            {
                "name": "wait_and_click",
                "status": "done",
                "result": true
            },
            {
                "name": "wait_and_click",
                "status": "done",
                "result": true
            },
            {
                "name": "wait",
                "status": "done",
                "result": true
            }
        ]
    }
}
Handling infinite scrolling pages (lazy loading)
A common use case for Page Interactions is to scroll down pages that differ loading off-screen content until a user scrolls down to reveal it. Also known as "lazy loading", websites often do this to reduce page load times or as an alternative to pagination. Some examples include Facebook's newsfeed, Google News, and YouTube's search pages.
Page Interactions include a dedicated feature named infinite_scroll designed specifically to scroll through lazy loading web pages. It's controlled using three parameters:
Parameter
Required
Description
Values
Max
loading_selector
Optional
An identifier of the HTML element that displays the loading/spinner indicator. This helps identify when loading starts and ends.
String. Eg: "#spinner", ".loadingID"
N/A
delay_after_scroll
Optional
The time delay, in milliseconds, between scrolls.
Integer. Default: 200
N/A
duration
Required
A time interval in milliseconds during which scrolling should be continuously attempted.
Integer. Eg: 1000
28000
Scrolling is initiated as soon as the page load event is fired.
Like all Page Interactions, infinite scrolling is capped by the global 60-second session timeout, and will be terminated if this limit is reached.
Infinite Scroll was designed to be used in one of two main methods:
Identifying the loading element
Where possible, it is recommended to use this method, as it allows for more accurate scrolling operation. Use the loading_selector parameter to identify the loading indicator or spinner that is displayed while the page loads additional content, and refine the scrolling behavior with the delay_after_scroll and duration parameters.
Time-based scrolling
In cases where there is no loading indicator, the delay_after_scroll and duration parameters can be used to define and refine the scrolling action until it is well-timed for the particular webpage in question.
Example
cURL
Python
Node.js
Go
curl -X POST 'https://api.webit.live/api/v1/realtime/web' \
--header 'Authorization: Basic <credential string>' \
--header 'Content-Type: application/json' \
--data-raw '{
    "url": "https://www.example.com",
    "render": true,
    "format": "json",
    "country": "US",
    "parse": true,
    "render_flow": [
        {
            "infinite_scroll": {
                "loading_selector": ".spinner",
                "delay_after_scroll": 2000,
                "duration": 15000
            }
        }    
    ]
}'
import requests
​
url = 'https://api.webit.live/api/v1/realtime/web'
headers = {
    'Authorization': 'Basic <credential string>',
    'Content-Type': 'application/json'
}
data = {
    "url": "https://www.example.com",
    "render": True,
    "format": "json",
    "country": "US",
    "parse": True,
    "render_flow": [
        {
            "infinite_scroll": {
                "loading_selector": ".spinner",
                "delay_after_scroll": 2000,
                "duration": 15000
            }
        }
    ]
}
​
response = requests.post(url, headers=headers, json=data)
​
print(response.status_code)
print(response.json())
const axios = require('axios');
​
const url = 'https://api.webit.live/api/v1/realtime/web';
const headers = {
  'Authorization': 'Basic <credential string>',
  'Content-Type': 'application/json'
};
const data = {
  "url": "https://www.example.com",
  "render": true,
  "format": "json",
  "country": "US",
  "parse": true,
  "render_flow": [
    {
      "infinite_scroll": {
        "loading_selector": ".spinner",
        "delay_after_scroll": 2000,
        "duration": 15000
      }
    }
  ]
};
​
axios.post(url, data, { headers })
  .then(response => {
    console.log(response.status);
    console.log(response.data);
  })
  .catch(error => {
    console.error(error);
  });
package main
​
import (
	"bytes"
	"encoding/base64"
	"fmt"
	"net/http"
	"encoding/json"
)
​
func main() {
	url := "https://api.webit.live/api/v1/realtime/web"
	payload := []byte(`{
		"url": "https://www.example.com",
		"render": true,
		"format": "json",
		"country": "US",
		"parse": true,
		"render_flow": [
			{
				"infinite_scroll": {
					"loading_selector": ".spinner",
					"delay_after_scroll": 2000,
					"duration": 15000
				}
			}
		]
	}`)
	headers := map[string]string{
		"Authorization":  "Basic <credential string>",
		"Content-Type":   "application/json",
	}
​
	req, err := http.NewRequest("POST", url, bytes.NewBuffer(payload))
	if err != nil {
		fmt.Println(err)
		return
	}
​
	for key, value := range headers {
		req.Header.Set(key, value)
	}
​
	client := &http.Client{}
	resp, err := client.Do(req)
	if err != nil {
		fmt.Println(err)
		return
	}
	defer resp.Body.Close()
​
	fmt.Println(resp.StatusCode)
	// Read the response body if needed
	// body, err := ioutil.ReadAll(resp.Body)
	// fmt.Println(string(body))
}
Capturing screenshots
Page Interactions can be used to capture screenshots of the desired webpage. To take a screenshot, use the following parameters:
cURL
Python
Node.js
Go
curl -X POST 'https://api.webit.live/api/v1/realtime/web' \
--header 'Authorization: Basic <credential string>' \
--header 'Content-Type: application/json' \
--data-raw '{
    "url": "https://www.example.com",
    "render": true,
    "render_flow": [{
        "screenshot": {
                full_page: true,
                timeout: 30000
        }
    }]
}'
import requests
​
url = 'https://api.webit.live/api/v1/realtime/web'
headers = {
    'Authorization': 'Basic <credential string>',
    'Content-Type': 'application/json'
}
data = {
    "url": "https://www.example.com",
    "render": True,
    "format": "json",
    "country": "US",
    "parse": True,
    "render_flow": [{
        "screenshot": {
                full_page: true,
                timeout: 30000
        }
    }]
​
}
​
response = requests.post(url, headers=headers, json=data)
​
print(response.status_code)
print(response.json())
const axios = require('axios');
​
const url = 'https://api.webit.live/api/v1/realtime/web';
const headers = {
  'Authorization': 'Basic <credential string>',
  'Content-Type': 'application/json'
};
const data = {
  "url": "https://www.example.com",
  "render": true,
  "format": "json",
  "country": "US",
  "parse": true,
  "render_flow": [{
      "screenshot": {
        full_page: true,
        timeout: 30000
       }
  }]
​
};
​
axios.post(url, data, { headers })
  .then(response => {
    console.log(response.status);
    console.log(response.data);
  })
  .catch(error => {
    console.error(error);
  });
package main
​
import (
	"bytes"
	"encoding/base64"
	"fmt"
	"net/http"
	"encoding/json"
)
​
func main() {
	url := "https://api.webit.live/api/v1/realtime/web"
	payload := []byte(`{
		"url": "https://www.example.com",
		"render": true,
		"format": "json",
		"country": "US",
		"parse": true,
		"render_flow": [{
        		"screenshot": {
                		full_page: true,
                		timeout: 30000
        		}
    		}]
	}`)
	headers := map[string]string{
		"Authorization":  "Basic <credential string>",
		"Content-Type":   "application/json",
	}
​
	req, err := http.NewRequest("POST", url, bytes.NewBuffer(payload))
	if err != nil {
		fmt.Println(err)
		return
	}
​
	for key, value := range headers {
		req.Header.Set(key, value)
	}
​
	client := &http.Client{}
	resp, err := client.Do(req)
	if err != nil {
		fmt.Println(err)
		return
	}
	defer resp.Body.Close()
​
	fmt.Println(resp.StatusCode)
	// Read the response body if needed
	// body, err := ioutil.ReadAll(resp.Body)
	// fmt.Println(string(body))
}
To take a screenshot, render must be set to true.
Parameter
Required
Description
Values
full_page
Optional
Default = false
Capture a screenshot of the entire page or just the viewable area.
Boolean - true/false
timeout
Optional
Default = 30000
Controls the time to allow screenshots scrolling before terminating screenshot processing.
Integer - milliseconds
Failure Handling
The most common errors that are likely to occur include execution errors or timeouts. Most interaction functions include an optional timeout property that allows users to cap their execution time.
If an error is encountered in a chain of interactions, interactions that come after the failed interaction will not be executed. The data is collected from the web resource normally, and returned in accordance with the parameters of the associated request.
Additionally, the error encountered is returned to the user for debugging purposes. In the below example, the two initial interactions were executed successfully, followed by a failed interaction, and finally an interaction that was not executed because of the previously failed interaction:
{
    "status": "success",
    "html_content":"...",
    "render_flow": {
        "success": false,
        "results": [
            {
                "name": "wait_and_type",
                "status": "done",
                "result": true
            },
            {
                "name": "wait_and_click",
                "status": "done",
                "result": true
            },
            {
                "name": "scroll",
                "status": "error",
                "error": "timeout"
            },
            {
                "name": "wait",
                "status": "no-run"
            }
        ]
    }
}
Operation Reference
The table below lists all of the currently supported operations, as well as their types, default values, examples, and more.
Name
Function
Type
Required
Min
Max
Default
Example
Description
wait
delay (ms)
number
no
0
♾️
–
1000
Pause for a period of x ms
wait_for
selectors
array<string>
yes
–
–
–
[”body”,”id”]
Wait until the listed selector(s) have loaded.
 
timeout (ms)
number
no
0
♾️
1000
2000
Wait for x ms before throwing a timeout exception
wait_and_click
selector
string
yes
–
–
–
“#element-id”
The selector for the desired element
 
timeout (ms)
number
no
0
♾️
1000
2000
Wait for x ms before throwing a timeout exception
 
delay (ms)
number
no
0
♾️
0
1000
An artificial delay added at the end of the operation.
 
scroll
bool
no
–
–
false
false
When true, scroll the selected element into the visible area if it is not already visible.
 
visible
bool
no
Status	Description
no-run	the interaction was not performed.
in-progress	The interaction was in progress when the global 60-second timeout was triggered.
done	the interaction finished successfully.
error	the interaction encountered an error.
Parameter	Required	Description	Values	Max
loading_selector	Optional	An identifier of the HTML element that displays the loading/spinner indicator. This helps identify when loading starts and ends.	String. Eg: "#spinner", ".loadingID"	N/A
delay_after_scroll	Optional	The time delay, in milliseconds, between scrolls.	Integer. Default: 200	N/A
duration	Required	A time interval in milliseconds during which scrolling should be continuously attempted.	Integer. Eg: 1000	28000
Parameter	Required	Description	Values
full_page	Optional Default = false	Capture a screenshot of the entire page or just the viewable area.	Boolean - true/false
timeout	Optional Default = 30000	Controls the time to allow screenshots scrolling before terminating screenshot processing.	Integer - milliseconds
Name	Function	Type	Required	Min	Max	Default	Example	Description
wait	delay (ms)	number	no	0	♾️	–	1000	Pause for a period of x ms
wait_for	selectors	array<string>	yes	–	–	–	[”body”,”id”]	Wait until the listed selector(s) have loaded.
	timeout (ms)	number	no	0	♾️	1000	2000	Wait for x ms before throwing a timeout exception
wait_and_click	selector	string	yes	–	–	–	“#element-id”	The selector for the desired element
	timeout (ms)	number	no	0	♾️	1000	2000	Wait for x ms before throwing a timeout exception
	delay (ms)	number	no	0	♾️	0	1000	An artificial delay added at the end of the operation.
	scroll	bool	no	–	–	false	false	When true, scroll the selected element into the visible area if it is not already visible.
	visible	bool	no