Links
Comment on page

Real-time URL request

Public web data is collected by sending a request with a URL to the Web API. It's also possible to include custom headers, targets, or additional characters if the URL contains non-ASCII characters by supplying request parameters in a valid JSON format. To make a real-time request, use the following syntax:
Nimble APIs requires that a base64 encoded credential string be sent with every request to authenticate your account. For detailed examples, see Web API Authentication.
cURL
Python
Node.js
Go
curl -X POST 'https://api.webit.live/api/v1/realtime/web' \
--header 'Authorization: Basic <credential string>' \
--header 'Content-Type: application/json' \
--data-raw '{
"url": "https://www.example.com",
"method": "GET",
"parse": false,
"render": true,
"country": "CA",
"locale": "en-GB",
"headers": {
"Some-Extra-Header": "Some-Extra-Header"
}
}'
import requests
url = 'https://api.webit.live/api/v1/realtime/web'
headers = {
'Authorization': 'Basic <credential string>',
'Content-Type': 'application/json'
}
data = {
"url": "https://www.example.com",
"method": "GET",
"parse": False,
"render": True,
"country": "CA",
"locale": "en-GB",
"headers": {
"Some-Extra-Header": "Some-Extra-Header"
}
}
response = requests.post(url, headers=headers, json=data)
const axios = require('axios');
const url = 'https://api.webit.live/api/v1/realtime/web';
const headers = {
'Authorization': 'Basic <credential string>',
'Content-Type': 'application/json'
};
const data = {
"url": "https://www.example.com",
"method": "GET",
"parse": false,
"render": true,
"country": "CA",
"locale": "en-GB",
"headers": {
"Some-Extra-Header": "Some-Extra-Header"
}
};
axios.post(url, data, { headers })
.then(response => {
console.log(response.status);
console.log(response.data);
})
.catch(error => {
console.error(error);
});
package main
import (
"bytes"
"encoding/base64"
"fmt"
"net/http"
"encoding/json"
)
func main() {
url := "https://api.webit.live/api/v1/realtime/web"
payload := []byte(`{
"url": "https://www.example.com",
"method": "GET",
"parse": false,
"render": true,
"country": "CA",
"locale": "en-GB",
"headers": {
"Some-Extra-Header": "Some-Extra-Header"
}
}`)
headers := map[string]string{
"Authorization": "Basic <credential string>",
"Content-Type": "application/json",
}
req, err := http.NewRequest("POST", url, bytes.NewBuffer(payload))
if err != nil {
fmt.Println(err)
return
}
for key, value := range headers {
req.Header.Set(key, value)
}
client := &http.Client{}
resp, err := client.Do(req)
if err != nil {
fmt.Println(err)
return
}
defer resp.Body.Close()
fmt.Println(resp.StatusCode)
// Read the response body if needed
// body, err := ioutil.ReadAll(resp.Body)
// fmt.Println(string(body))
}
Every request sent through Nimble API is automatically routed through Nimble IP - our premium proxy network!

Request options

Parameter
Required
Description
url
Required
URL | The page or resource to be fetched. Note: when using a URL with a query string, encode the URL and place it at the end of the query string.
method
Optional (default = GET)
String | The method for requesting a URL from the target server.
country
Optional (default = all)
String | Country used to access the target URL, use ISO Alpha-2 Country Codes i.e. US, DE, GB
locale
Optional (default = EN)
String | LCID standard locale used for the URL request.
headers*
Optional
String | JSON with key/value structure to pass the required headers.
cookies
Optional
String | Attach a cookie or cookies to your request. See Sending cookies with a request for details.
format
Optional (default = JSON)
Enum: JSON | HTML - The data response format. HTML - in case of error, returns JSON with error message.
parse
Optional (default = false)
Enum: true | false - True - the page's content will be parsed and returned in a JSON format. False - Response will include page headers and raw data (without parsing). When using parse = true, format must be set to “JSON”.
parser
Optional (default = null)
String | Define a custom parsing template. See Parsing Templates.
render
Optional (default = false)
Enum: true | false - enables or disables Javascript rendering on the target page.
render_options
N/A
A collection of rendering-related settings. See Rendering options.
render_flow
Optional (default = null)
String | Define a series of actions to be performed on the page prior to data collection. See Page Interactions.
* Please do not include any cookies when sending custom headers. To send cookies, please see the Sending cookies with a request section.

Rendering options

The parameter render_options includes several configuration options that allow users to fine-tune how Nimble renders webpages. These options include:
Parameter
Required
Description
include_iframes
No (default = false)
Boolean - Instructs Nimble APIs to render or not to render iframes.
timeout
No (default = 30000) Maximum: 60000
Integer (ms) - Defines how long Nimble APIs will wait for a page to finish loading.
render_type
No (default = load)
Enum - Defines the state that Nimble APIs will consider a page as fully loaded.
Because many webpages load additional content after the initial page has finished loading (such as single-page applications), deciding exactly when a webpage has finished loading is often ambiguous. To aid with this, Nimble APIs supports several options:
  • load - Nimble will consider a page ready after the standard page load event is fired.
  • domready - Nimble will consider a page ready when the DOMContentLoaded event is fired.
  • idle2 - Nimble will consider a page ready when there are 2 or fewer network connections made during the last 500ms.
  • idle0 - Nimble will consider a page ready when no new network connections are made during the last 500ms.
In the below example, all three rendering options are demonstrated:
cURL
Python
Node.js
Go
curl -X POST 'https://api.webit.live/api/v1/realtime/web' \
--header 'Authorization: Basic <credential string>' \
--header 'Content-Type: application/json' \
--data-raw '{
"url": "https://www.example.com",
"method": "GET",
"parse": true,
"render": true,
"render_options": {
"include_iframes": true,
"render_type": "idle0",
"timeout": 35000
}
}'
import requests
url = 'https://api.webit.live/api/v1/realtime/web'
headers = {
'Authorization': 'Basic <credential string>',
'Content-Type': 'application/json'
}
data = {
"url": "https://www.example.com",
"method": "GET",
"parse": True,
"render": True,
"render_options": {
"include_iframes": True,
"render_type": "idle0",
"timeout": 35000
}
}
response = requests.post(url, headers=headers, json=data)
print(response.status_code)
print(response.json())
const axios = require('axios');
const url = 'https://api.webit.live/api/v1/realtime/web';
const headers = {
'Authorization': 'Basic <credential string>',
'Content-Type': 'application/json'
};
const data = {
"url": "https://www.example.com",
"method": "GET",
"parse": true,
"render": true,
"render_options": {
"include_iframes": true,
"render_type": "idle0",
"timeout": 35000
}
};
axios.post(url, data, { headers })
.then(response => {
console.log(response.status);
console.log(response.data);
})
.catch(error => {
console.error(error);
});
package main
import (
"bytes"
"fmt"
"net/http"
"encoding/json"
)
func main() {
url := "https://api.webit.live/api/v1/realtime/web"
payload := []byte(`{
"url": "https://www.example.com",
"method": "GET",
"parse": true,
"render": true,
"render_options": {
"include_iframes": true,
"render_type": "idle0",
"timeout": 35000
}
}`)
headers := map[string]string{
"Authorization": "Basic <credential string>",
"Content-Type": "application/json",
}
req, err := http.NewRequest("POST", url, bytes.NewBuffer(payload))
if err != nil {
fmt.Println(err)
return
}
for key, value := range headers {
req.Header.Set(key, value)
}
client := &http.Client{}
resp, err := client.Do(req)
if err != nil {
fmt.Println(err)
return
}
defer resp.Body.Close()
fmt.Println(resp.StatusCode)
// Read the response body if needed
// body, err := ioutil.ReadAll(resp.Body)
// fmt.Println(string(body))
}

Sending cookies with a request

Many websites use cookies to store important user information, such as preferences, locales, locations, and more. Nimble's Web API allows for cookies to be defined in a string or object format.
Cookie object
The cookie object allows for one or multiple cookies to be set, with each one containing three parameters:
  • key - the name of the cookie.
  • value - the value of the cookie.
  • domain - The domain with which this cookie is associated. If a domain is not defined, Nimble will use the requested URL's domain by default.
In the following example, a cookie is used to select a particular store on the Lowe's website:
cURL
Python
Node.js
Go
curl --location --request POST 'https://api.webit.live/api/v1/realtime/web/' \
--header 'Authorization: Basic <credential string>' \
--header 'Content-Type: application/json' \
--data-raw '{
"url": "https://www.lowes.com/pd/CRAFTSMAN-M110-140-cc-21-in-Gas-Push-Lawn-Mower-with-Briggs-Stratton-Engine/1000676311",
"country": "US",
"cookies": [
{
"key": "sd",
"value": "%7B%22id%22%3A%222209%22%2C%22zip%22%3A%2204769%22%2C%22city%22%3A%22Presque%20Isle%22%2C%22state%22%3A%22ME%22%2C%22name%22%3A%22Presque%20Isle%20Lowe's%22%2C%22region%22%3A%2218%22%7D",
"domain": "lowes.com"
}
]
}'
import requests
url = 'https://api.webit.live/api/v1/realtime/web'
headers = {
'Authorization': 'Basic <credential string>',
'Content-Type': 'application/json'
}
data = {
"url": "https://www.lowes.com/pd/CRAFTSMAN-M110-140-cc-21-in-Gas-Push-Lawn-Mower-with-Briggs-Stratton-Engine/1000676311",
"country": "CA",
"cookies": [
{
"key": "sd",
"value": "%7B%22id%22%3A%222209%22%2C%22zip%22%3A%2204769%22%2C%22city%22%3A%22Presque%20Isle%22%2C%22state%22%3A%22ME%22%2C%22name%22%3A%22Presque%20Isle%20Lowe's%22%2C%22region%22%3A%2218%22%7D",
"domain": "lowes.com"
}
]
}
response = requests.post(url, headers=headers, json=data)
const axios = require('axios');
const url = 'https://api.webit.live/api/v1/realtime/web';
const headers = {
'Authorization': 'Basic <credential string>',
'Content-Type': 'application/json'
};
const data = {
"url": "https://www.lowes.com/pd/CRAFTSMAN-M110-140-cc-21-in-Gas-Push-Lawn-Mower-with-Briggs-Stratton-Engine/1000676311",
"country": "CA",
"cookies": [
{
"key": "sd",
"value": "%7B%22id%22%3A%222209%22%2C%22zip%22%3A%2204769%22%2C%22city%22%3A%22Presque%20Isle%22%2C%22state%22%3A%22ME%22%2C%22name%22%3A%22Presque%20Isle%20Lowe's%22%2C%22region%22%3A%2218%22%7D",
"domain": "lowes.com"
}
]
};
axios.post(url, data, { headers })
.then(response => {
console.log(response.status);
console.log(response.data);
})
.catch(error => {
console.error(error);
});
package main
import (
"bytes"
"encoding/base64"
"fmt"
"net/http"
"encoding/json"
)
func main() {
url := "https://api.webit.live/api/v1/realtime/web"
payload := []byte(`{
"url": "https://www.lowes.com/pd/CRAFTSMAN-M110-140-cc-21-in-Gas-Push-Lawn-Mower-with-Briggs-Stratton-Engine/1000676311",
"country": "CA",
"cookies": [
{
"key": "sd",
"value": "%7B%22id%22%3A%222209%22%2C%22zip%22%3A%2204769%22%2C%22city%22%3A%22Presque%20Isle%22%2C%22state%22%3A%22ME%22%2C%22name%22%3A%22Presque%20Isle%20Lowe's%22%2C%22region%22%3A%2218%22%7D",
"domain": "lowes.com"
}
]
}`)
headers := map[string]string{
"Authorization": "Basic <credential string>",
"Content-Type": "application/json",
}
req, err := http.NewRequest("POST", url, bytes.NewBuffer(payload))
if err != nil {
fmt.Println(err)
return
}
for key, value := range headers {
req.Header.Set(key, value)
}
client := &http.Client{}
resp, err := client.Do(req)
if err != nil {
fmt.Println(err)
return
}
defer resp.Body.Close()
fmt.Println(resp.StatusCode)
// Read the response body if needed
// body, err := ioutil.ReadAll(resp.Body)
// fmt.Println(string(body))
}
In the above example, the key is set to sd and the value is set to an encoded version of the following location object:
{"id":"2209","zip":"04769","city":"Presque Isle","state":"ME","name":"Presque Isle Lowe's","region":"18"}
These values are unique to Lowe's website, and are not necessarily applicable to other websites.
Cookie string
The cookie string uses a syntax of key1=value1;key2=value2;... format, and does not accept a domain parameter, always defaulting to the requested URL's domain.
For example:
cURL
Python
Node.js
Go
curl --location --request POST 'https://api.webit.live/api/v1/realtime/web/' \
--header 'Authorization: Basic <credential string>' \
--header 'Content-Type: application/json' \
--data-raw '{
"url": "https://www.lowes.com/pd/CRAFTSMAN-M110-140-cc-21-in-Gas-Push-Lawn-Mower-with-Briggs-Stratton-Engine/1000676311",
"country": "US",
"cookies": "key1=value1;sd=%7B%22id%22%3A%222209..."
}'
import requests
url = 'https://api.webit.live/api/v1/realtime/web'
headers = {
'Authorization': 'Basic <credential string>',
'Content-Type': 'application/json'
}
data = {
"url": "https://www.lowes.com/pd/CRAFTSMAN-M110-140-cc-21-in-Gas-Push-Lawn-Mower-with-Briggs-Stratton-Engine/1000676311",
"country": "CA",
"cookies": "key1=value1;sd=%7B%22id%22%3A%222209..."
}
response = requests.post(url, headers=headers, json=data)
const axios = require('axios');
const url = 'https://api.webit.live/api/v1/realtime/web';
const headers = {
'Authorization': 'Basic <credential string>',
'Content-Type': 'application/json'
};
const data = {
"url": "https://www.lowes.com/pd/CRAFTSMAN-M110-140-cc-21-in-Gas-Push-Lawn-Mower-with-Briggs-Stratton-Engine/1000676311",
"country": "CA",
"cookies": "key1=value1;sd=%7B%22id%22%3A%222209..."
};
axios.post(url, data, { headers })
.then(response => {
console.log(response.status);
console.log(response.data);
})
.catch(error => {
console.error(error);
});
package main
import (
"bytes"
"encoding/base64"
"fmt"
"net/http"
"encoding/json"
)
func main() {
url := "https://api.webit.live/api/v1/realtime/web"
payload := []byte(`{
"url": "https://www.lowes.com/pd/CRAFTSMAN-M110-140-cc-21-in-Gas-Push-Lawn-Mower-with-Briggs-Stratton-Engine/1000676311",
"country": "CA",
"cookies": "key1=value1;sd=%7B%22id%22%3A%222209..."
}`)
headers := map[string]string{
"Authorization": "Basic <credential string>",
"Content-Type": "application/json",
}
req, err := http.NewRequest("POST", url, bytes.NewBuffer(payload))
if err != nil {
fmt.Println(err)
return
}
for key, value := range headers {
req.Header.Set(key, value)
}
client := &http.Client{}
resp, err := client.Do(req)
if err != nil {
fmt.Println(err)
return
}
defer resp.Body.Close()
fmt.Println(resp.StatusCode)
// Read the response body if needed
// body, err := ioutil.ReadAll(resp.Body)
// fmt.Println(string(body))
}

Sending POST requests to the Target Server

If you wish to send POST data with your request, such as when submitting a form, please use the following syntax:
cURL
Python
Node.js
Go
curl -X POST 'https://api.webit.live/api/v1/realtime/web' \
--header 'Authorization: Basic <credential string>' \
--header 'Content-Type: application/json' \
--data-raw '{
"url": "https://www.example.com",
"method": "POST",
"headers": {
"Content-Type": "application/json",
"Some-Extra-Header": "some-extra-header"
},
"body": {
"a": 1,
"b": 2
}
}'
import requests
url = 'https://api.webit.live/api/v1/realtime/web'
headers = {
'Authorization': 'Basic <credential string>',
'Content-Type': 'application/json'
}
data = {
"url": "https://www.example.com",
"method": "POST",
"headers": {
"Content-Type": "application/json",
"Some-Extra-Header": "some-extra-header"
},
"body": {
"a": 1,
"b": 2
}
}
response = requests.post(url, headers=headers, json=data)
const axios = require('axios');
const url = 'https://api.webit.live/api/v1/realtime/web';
const headers = {
'Authorization': 'Basic <credential string>',
'Content-Type': 'application/json'
};
const data = {
"url": "https://www.example.com",
"method": "POST",
"headers": {
"Content-Type": "application/json",
"Some-Extra-Header": "some-extra-header"
},
"body": {
"a": 1,
"b": 2
}
};
axios.post(url, data, { headers })
.then(response => {
console.log(response.status);
console.log(response.data);
})
.catch(error => {
console.error(error);
});
package main
import (
"bytes"
"encoding/base64"
"fmt"
"net/http"
"encoding/json"
)
func main() {
url := "https://api.webit.live/api/v1/realtime/web"
payload := []byte(`{
"url": "https://www.example.com",
"method": "POST",
"headers": {
"Content-Type": "application/json",
"Some-Extra-Header": "some-extra-header"
},
"body": {
"a": 1,
"b": 2
}
}`)
headers := map[string]string{
"Authorization": "Basic <credential string>",
"Content-Type": "application/json",
}
req, err := http.NewRequest("POST", url, bytes.NewBuffer(payload))
if err != nil {
fmt.Println(err)
return
}
for key, value := range headers {
req.Header.Set(key, value)
}
client := &http.Client{}
resp, err := client.Do(req)
if err != nil {
fmt.Println(err)
return
}
defer resp.Body.Close()
fmt.Println(resp.StatusCode)
// Read the response body if needed
// body, err := ioutil.ReadAll(resp.Body)
// fmt.Println(string(body))
}
Please note that specifying a content-type header is required when submitting a POST request with data in the body.
Additional request options
Parameter
Required
Description
method
Required
Use POST to send forms to the target site
headers
Optional
JSON with key/value structure to pass the required headers
body
Optional
JSON with key/value structure or string to pass needed payload

Response

Headers
X-Task-ID: string
Payload examples:
If parsing was disabled or omitted in the request, the result data will be the raw HTML of the page you requested. If parsing was enabled, a JSON object with a parsed version of the page will be delivered, with the raw HTML included under the html_content property.
200 OK
{
"status": "success",
"query_time": "2023-01-01T12:00:00.007Z",
"status_code": 200,
"headers": {},
"html_content": string,
"parsing": {
"status": "success",
"entities": { },
"total_entities_count": 0,
"entities_count": { }
},
"url": "https://www.google.com/search?q=hello world"
}
500 Error
{
"status": "error",
"task_id": "<task_id>",
"msg": "can't download the query response - please try again"
}
400 Input Error
{
"status": "failed",
"msg": error
}

Response Codes

Status
Description
200
OK
400
The requested resource could not be reached
401
Unauthorized/invalid credental string
500
Internal service error
501
An error was encountered by the proxy service