Real-time URL request
Public web data is collected by sending a request with a URL to the Web API. It's also possible to include custom headers, targets, or additional characters if the URL contains non-ASCII characters by supplying request parameters in a valid JSON format. To make a real-time request, use the following syntax:
Nimble APIs requires that a base64 encoded credential string be sent with every request to authenticate your account. For detailed examples, see Web API Authentication.
cURL
Python
Node.js
Go
curl -X POST 'https://api.webit.live/api/v1/realtime/web' \
--header 'Authorization: Basic <credential string>' \
--header 'Content-Type: application/json' \
--data-raw '{
"url": "https://www.example.com",
"method": "GET",
"parse": false,
"render": true,
"country": "CA",
"locale": "en-GB",
"headers": {
"Some-Extra-Header": "Some-Extra-Header"
}
}'
import requests
url = 'https://api.webit.live/api/v1/realtime/web'
headers = {
'Authorization': 'Basic <credential string>',
'Content-Type': 'application/json'
}
data = {
"url": "https://www.example.com",
"method": "GET",
"parse": False,
"render": True,
"country": "CA",
"locale": "en-GB",
"headers": {
"Some-Extra-Header": "Some-Extra-Header"
}
}
response = requests.post(url, headers=headers, json=data)
const axios = require('axios');
const url = 'https://api.webit.live/api/v1/realtime/web';
const headers = {
'Authorization': 'Basic <credential string>',
'Content-Type': 'application/json'
};
const data = {
"url": "https://www.example.com",
"method": "GET",
"parse": false,
"render": true,
"country": "CA",
"locale": "en-GB",
"headers": {
"Some-Extra-Header": "Some-Extra-Header"
}
};
axios.post(url, data, { headers })
.then(response => {
console.log(response.status);
console.log(response.data);
})
.catch(error => {
console.error(error);
});
package main
import (
"bytes"
"encoding/base64"
"fmt"
"net/http"
"encoding/json"
)
func main() {
url := "https://api.webit.live/api/v1/realtime/web"
payload := []byte(`{
"url": "https://www.example.com",
"method": "GET",
"parse": false,
"render": true,
"country": "CA",
"locale": "en-GB",
"headers": {
"Some-Extra-Header": "Some-Extra-Header"
}
}`)
headers := map[string]string{
"Authorization": "Basic <credential string>",
"Content-Type": "application/json",
}
req, err := http.NewRequest("POST", url, bytes.NewBuffer(payload))
if err != nil {
fmt.Println(err)
return
}
for key, value := range headers {
req.Header.Set(key, value)
}
client := &http.Client{}
resp, err := client.Do(req)
if err != nil {
fmt.Println(err)
return
}
defer resp.Body.Close()
fmt.Println(resp.StatusCode)
// Read the response body if needed
// body, err := ioutil.ReadAll(resp.Body)
// fmt.Println(string(body))
}
Every request sent through Nimble API is automatically routed through Nimble IP - our premium proxy network!
Parameter | Required | Description |
|---|---|---|
url | Required | URL | The page or resource to be fetched. Note: when using a URL with a query string, encode the URL and place it at the end of the query string. |
method | Optional (default = GET) | String | The method for requesting a URL from the target server. |
country | Optional (default = all) | String | Country used to access the target URL, use ISO Alpha-2 Country Codes i.e. US, DE, GB |
locale | Optional (default = EN) | String | LCID standard locale used for the URL request. |
headers* | Optional | String | JSON with key/value structure to pass the required headers. |
cookies | Optional | String | Attach a cookie or cookies to your request. See Sending cookies with a request for details. |
format | Optional (default = JSON) | Enum: JSON | HTML - The data response format. HTML - in case of error, returns JSON with error message. |
parse | Optional (default = false) | Enum: true | false - True - the page's content will be parsed and returned in a JSON format. False - Response will include page headers and raw data (without parsing). When using parse = true, format must be set to “JSON”. |
parser | Optional (default = null) | |
render | Optional (default = false) | Enum: true | false - enables or disables Javascript rendering on the target page. |
render_options | N/A | |
render_flow | Optional (default = null) | String | Define a series of actions to be performed on the page prior to data collection. See Page Interactions. |
* Please do not include any cookies when sending custom headers. To send cookies, please see the Sending cookies with a request section.
The parameter
render_options includes several configuration options that allow users to fine-tune how Nimble renders webpages. These options include:Parameter | Required | Description |
|---|---|---|
include_iframes | No (default = false) | Boolean - Instructs Nimble APIs to render or not to render iframes. |
timeout | No (default = 30000)
Maximum: 60000 | Integer (ms) - Defines how long Nimble APIs will wait for a page to finish loading. |
render_type | No (default = load) | Enum - Defines the state that Nimble APIs will consider a page as fully loaded. |
Because many webpages load additional content after the initial page has finished loading (such as single-page applications), deciding exactly when a webpage has finished loading is often ambiguous. To aid with this, Nimble APIs supports several options:
- load - Nimble will consider a page ready after the standard page load event is fired.
- domready - Nimble will consider a page ready when the DOMContentLoaded event is fired.
- idle2 - Nimble will consider a page ready when there are 2 or fewer network connections made during the last 500ms.
- idle0 - Nimble will consider a page ready when no new network connections are made during the last 500ms.
In the below example, all three rendering options are demonstrated:
cURL
Python
Node.js
Go
curl -X POST 'https://api.webit.live/api/v1/realtime/web' \
--header 'Authorization: Basic <credential string>' \
--header 'Content-Type: application/json' \
--data-raw '{
"url": "https://www.example.com",
"method": "GET",
"parse": true,
"render": true,
"render_options": {
"include_iframes": true,
"render_type": "idle0",
"timeout": 35000
}
}'
import requests
url = 'https://api.webit.live/api/v1/realtime/web'
headers = {
'Authorization': 'Basic <credential string>',
'Content-Type': 'application/json'
}
data = {
"url": "https://www.example.com",
"method": "GET",
"parse": True,
"render": True,
"render_options": {
"include_iframes": True,
"render_type": "idle0",
"timeout": 35000
}
}
response = requests.post(url, headers=headers, json=data)
print(response.status_code)
print(response.json())
const axios = require('axios');
const url = 'https://api.webit.live/api/v1/realtime/web';
const headers = {
'Authorization': 'Basic <credential string>',
'Content-Type': 'application/json'
};
const data = {
"url": "https://www.example.com",
"method": "GET",
"parse": true,
"render": true,
"render_options": {
"include_iframes": true,
"render_type": "idle0",
"timeout": 35000
}
};
axios.post(url, data, { headers })
.then(response => {
console.log(response.status);
console.log(response.data);
})
.catch(error => {
console.error(error);
});
package main
import (
"bytes"
"fmt"
"net/http"
"encoding/json"
)
func main() {
url := "https://api.webit.live/api/v1/realtime/web"
payload := []byte(`{
"url": "https://www.example.com",
"method": "GET",
"parse": true,
"render": true,
"render_options": {
"include_iframes": true,
"render_type": "idle0",
"timeout": 35000
}
}`)
headers := map[string]string{
"Authorization": "Basic <credential string>",
"Content-Type": "application/json",
}
req, err := http.NewRequest("POST", url, bytes.NewBuffer(payload))
if err != nil {
fmt.Println(err)
return
}
for key, value := range headers {
req.Header.Set(key, value)
}
client := &http.Client{}
resp, err := client.Do(req)
if err != nil {
fmt.Println(err)
return
}
defer resp.Body.Close()
fmt.Println(resp.StatusCode)
// Read the response body if needed
// body, err := ioutil.ReadAll(resp.Body)
// fmt.Println(string(body))
}
Many websites use cookies to store important user information, such as preferences, locales, locations, and more. Nimble's Web API allows for cookies to be defined in a string or object format.
Cookie object
The cookie object allows for one or multiple cookies to be set, with each one containing three parameters:
- key - the name of the cookie.
- value - the value of the cookie.
- domain - The domain with which this cookie is associated. If a domain is not defined, Nimble will use the requested URL's domain by default.
In the following example, a cookie is used to select a particular store on the Lowe's website:
cURL
Python
Node.js
Go
curl --location --request POST 'https://api.webit.live/api/v1/realtime/web/' \
--header 'Authorization: Basic <credential string>' \
--header 'Content-Type: application/json' \
--data-raw '{
"url": "https://www.lowes.com/pd/CRAFTSMAN-M110-140-cc-21-in-Gas-Push-Lawn-Mower-with-Briggs-Stratton-Engine/1000676311",
"country": "US",
"cookies": [
{
"key": "sd",
"value": "%7B%22id%22%3A%222209%22%2C%22zip%22%3A%2204769%22%2C%22city%22%3A%22Presque%20Isle%22%2C%22state%22%3A%22ME%22%2C%22name%22%3A%22Presque%20Isle%20Lowe's%22%2C%22region%22%3A%2218%22%7D",
"domain": "lowes.com"
}
]
}'
import requests
url = 'https://api.webit.live/api/v1/realtime/web'
headers = {
'Authorization': 'Basic <credential string>',
'Content-Type': 'application/json'
}
data = {
"url": "https://www.lowes.com/pd/CRAFTSMAN-M110-140-cc-21-in-Gas-Push-Lawn-Mower-with-Briggs-Stratton-Engine/1000676311",
"country": "CA",
"cookies": [
{
"key": "sd",
"value": "%7B%22id%22%3A%222209%22%2C%22zip%22%3A%2204769%22%2C%22city%22%3A%22Presque%20Isle%22%2C%22state%22%3A%22ME%22%2C%22name%22%3A%22Presque%20Isle%20Lowe's%22%2C%22region%22%3A%2218%22%7D",
"domain": "lowes.com"
}
]
}
response = requests.post(url, headers=headers, json=data)
const axios = require('axios');
const url = 'https://api.webit.live/api/v1/realtime/web';
const headers = {
'Authorization': 'Basic <credential string>',
'Content-Type': 'application/json'
};
const data = {
"url": "https://www.lowes.com/pd/CRAFTSMAN-M110-140-cc-21-in-Gas-Push-Lawn-Mower-with-Briggs-Stratton-Engine/1000676311",
"country": "CA",
"cookies": [
{
"key": "sd",
"value": "%7B%22id%22%3A%222209%22%2C%22zip%22%3A%2204769%22%2C%22city%22%3A%22Presque%20Isle%22%2C%22state%22%3A%22ME%22%2C%22name%22%3A%22Presque%20Isle%20Lowe's%22%2C%22region%22%3A%2218%22%7D",
"domain": "lowes.com"
}
]
};
axios.post(url, data, { headers })
.then(response => {
console.log(response.status);
console.log(response.data);
})
.catch(error => {
console.error(error);
});
package main
import (
"bytes"
"encoding/base64"
"fmt"
"net/http"
"encoding/json"
)
func main() {
url := "https://api.webit.live/api/v1/realtime/web"
payload := []byte(`{
"url": "https://www.lowes.com/pd/CRAFTSMAN-M110-140-cc-21-in-Gas-Push-Lawn-Mower-with-Briggs-Stratton-Engine/1000676311",
"country": "CA",
"cookies": [
{
"key": "sd",
"value": "%7B%22id%22%3A%222209%22%2C%22zip%22%3A%2204769%22%2C%22city%22%3A%22Presque%20Isle%22%2C%22state%22%3A%22ME%22%2C%22name%22%3A%22Presque%20Isle%20Lowe's%22%2C%22region%22%3A%2218%22%7D",
"domain": "lowes.com"
}
]
}`)
headers := map[string]string{
"Authorization": "Basic <credential string>",
"Content-Type": "application/json",
}
req, err := http.NewRequest("POST", url, bytes.NewBuffer(payload))
if err != nil {
fmt.Println(err)
return
}
for key, value := range headers {
req.Header.Set(key, value)
}
client := &http.Client{}
resp, err := client.Do(req)
if err != nil {
fmt.Println(err)
return
}
defer resp.Body.Close()
fmt.Println(resp.StatusCode)
// Read the response body if needed
// body, err := ioutil.ReadAll(resp.Body)
// fmt.Println(string(body))
}
In the above example, the
key is set to sd and the value is set to an encoded version of the following location object:{"id":"2209","zip":"04769","city":"Presque Isle","state":"ME","name":"Presque Isle Lowe's","region":"18"}
These values are unique to Lowe's website, and are not necessarily applicable to other websites.
Cookie string
The cookie string uses a syntax of
key1=value1;key2=value2;... format, and does not accept a domain parameter, always defaulting to the requested URL's domain.For example:
cURL
Python
Node.js
Go
curl --location --request POST 'https://api.webit.live/api/v1/realtime/web/' \
--header 'Authorization: Basic <credential string>' \
--header 'Content-Type: application/json' \
--data-raw '{
"url": "https://www.lowes.com/pd/CRAFTSMAN-M110-140-cc-21-in-Gas-Push-Lawn-Mower-with-Briggs-Stratton-Engine/1000676311",
"country": "US",
"cookies": "key1=value1;sd=%7B%22id%22%3A%222209..."
}'
import requests
url = 'https://api.webit.live/api/v1/realtime/web'
headers = {
'Authorization': 'Basic <credential string>',
'Content-Type': 'application/json'
}
data = {
"url": "https://www.lowes.com/pd/CRAFTSMAN-M110-140-cc-21-in-Gas-Push-Lawn-Mower-with-Briggs-Stratton-Engine/1000676311",
"country": "CA",
"cookies": "key1=value1;sd=%7B%22id%22%3A%222209..."
}
response = requests.post(url, headers=headers, json=data)
const axios = require('axios');
const url = 'https://api.webit.live/api/v1/realtime/web';
const headers = {
'Authorization': 'Basic <credential string>',
'Content-Type': 'application/json'
};
const data = {
"url": "https://www.lowes.com/pd/CRAFTSMAN-M110-140-cc-21-in-Gas-Push-Lawn-Mower-with-Briggs-Stratton-Engine/1000676311",
"country": "CA",
"cookies": "key1=value1;sd=%7B%22id%22%3A%222209..."
};
axios.post(url, data, { headers })
.then(response => {
console.log(response.status);
console.log(response.data);
})
.catch(error => {
console.error(error);
});
package main
import (
"bytes"
"encoding/base64"
"fmt"
"net/http"
"encoding/json"
)
func main() {
url := "https://api.webit.live/api/v1/realtime/web"
payload := []byte(`{
"url": "https://www.lowes.com/pd/CRAFTSMAN-M110-140-cc-21-in-Gas-Push-Lawn-Mower-with-Briggs-Stratton-Engine/1000676311",
"country": "CA",
"cookies": "key1=value1;sd=%7B%22id%22%3A%222209..."
}`)
headers := map[string]string{
"Authorization": "Basic <credential string>",
"Content-Type": "application/json",
}
req, err := http.NewRequest("POST", url, bytes.NewBuffer(payload))
if err != nil {
fmt.Println(err)
return
}
for key, value := range headers {
req.Header.Set(key, value)
}
client := &http.Client{}
resp, err := client.Do(req)
if err != nil {
fmt.Println(err)
return
}
defer resp.Body.Close()
fmt.Println(resp.StatusCode)
// Read the response body if needed
// body, err := ioutil.ReadAll(resp.Body)
// fmt.Println(string(body))
}
If you wish to send POST data with your request, such as when submitting a form, please use the following syntax:
cURL
Python
Node.js
Go
curl -X POST 'https://api.webit.live/api/v1/realtime/web' \
--header 'Authorization: Basic <credential string>' \
--header 'Content-Type: application/json' \
--data-raw '{
"url": "https://www.example.com",
"method": "POST",
"headers": {
"Content-Type": "application/json",
"Some-Extra-Header": "some-extra-header"
},
"body": {
"a": 1,
"b": 2
}
}'
import requests
url = 'https://api.webit.live/api/v1/realtime/web'
headers = {
'Authorization': 'Basic <credential string>',
'Content-Type': 'application/json'
}
data = {
"url": "https://www.example.com",
"method": "POST",
"headers": {
"Content-Type": "application/json",
"Some-Extra-Header": "some-extra-header"
},
"body": {
"a": 1,
"b": 2
}
}
response = requests.post(url, headers=headers, json=data)
const axios = require('axios');
const url = 'https://api.webit.live/api/v1/realtime/web';
const headers = {
'Authorization': 'Basic <credential string>',
'Content-Type': 'application/json'
};
const data = {
"url": "https://www.example.com",
"method": "POST",
"headers": {
"Content-Type": "application/json",
"Some-Extra-Header": "some-extra-header"
},
"body": {
"a": 1,
"b": 2
}
};
axios.post(url, data, { headers })
.then(response => {
console.log(response.status);
console.log(response.data);
})
.catch(error => {
console.error(error);
});
package main
import (
"bytes"
"encoding/base64"
"fmt"
"net/http"
"encoding/json"
)
func main() {
url := "https://api.webit.live/api/v1/realtime/web"
payload := []byte(`{
"url": "https://www.example.com",
"method": "POST",
"headers": {
"Content-Type": "application/json",
"Some-Extra-Header": "some-extra-header"
},
"body": {
"a": 1,
"b": 2
}
}`)
headers := map[string]string{
"Authorization": "Basic <credential string>",
"Content-Type": "application/json",
}
req, err := http.NewRequest("POST", url, bytes.NewBuffer(payload))
if err != nil {
fmt.Println(err)
return
}
for key, value := range headers {
req.Header.Set(key, value)
}
client := &http.Client{}
resp, err := client.Do(req)
if err != nil {
fmt.Println(err)
return
}
defer resp.Body.Close()
fmt.Println(resp.StatusCode)
// Read the response body if needed
// body, err := ioutil.ReadAll(resp.Body)
// fmt.Println(string(body))
}
Please note that specifying a content-type header is required when submitting a POST request with data in the body.
Additional request options
Parameter | Required | Description |
|---|---|---|
method | Required | Use POST to send forms to the target site |
headers | Optional | JSON with key/value structure to pass the required headers |
body | Optional | JSON with key/value structure or string to pass needed payload |
Headers
X-Task-ID: string
Payload examples:
If parsing was disabled or omitted in the request, the result data will be the raw HTML of the page you requested. If parsing was enabled, a JSON object with a parsed version of the page will be delivered, with the raw HTML included under the
html_content property.200 OK
{
"status": "success",
"query_time": "2023-01-01T12:00:00.007Z",
"status_code": 200,
"headers": {},
"html_content": string,
"parsing": {
"status": "success",
"entities": { },
"total_entities_count": 0,
"entities_count": { }
},
"url": "https://www.google.com/search?q=hello world"
}
500 Error
{
"status": "error",
"task_id": "<task_id>",
"msg": "can't download the query response - please try again"
}
400 Input Error
{
"status": "failed",
"msg": error
}
Status | Description |
|---|---|
200 | OK |
400 | The requested resource could not be reached |
401 | Unauthorized/invalid credental string |
500 | Internal service error |
501 | An error was encountered by the proxy service |
Last modified 29d ago