Real-time URL request

Public web data is collected by sending a request with a URL to the Web API. It's also possible to include custom headers, targets, or additional characters if the URL contains non-ASCII characters by supplying request parameters in a valid JSON format. To make a real-time request, use the following syntax:

Nimble APIs requires that a base64 encoded credential string be sent with every request to authenticate your account. For detailed examples, see Web API Authentication.

cURL

curl -X POST 'https://api.webit.live/api/v1/realtime/web' \
--header 'Authorization: Basic <credential string>' \
--header 'Content-Type: application/json' \
--data-raw '{
    "url": "https://www.example.com",
    "method": "GET",
    "parse": false,
    "render": true,
    "country": "CA",
    "locale": "en-GB",
    "headers": {
        "Some-Extra-Header": "Some-Extra-Header"
    }
}'

Python

import requests

url = 'https://api.webit.live/api/v1/realtime/web'
headers = {
    'Authorization': 'Basic <credential string>',
    'Content-Type': 'application/json'
}
data = {
    "url": "https://www.example.com",
    "method": "GET",
    "parse": False,
    "render": True,
    "country": "CA",
    "locale": "en-GB",
    "headers": {
        "Some-Extra-Header": "Some-Extra-Header"
    }
}

response = requests.post(url, headers=headers, json=data)

Node.js

const axios = require('axios');

const url = 'https://api.webit.live/api/v1/realtime/web';
const headers = {
  'Authorization': 'Basic <credential string>',
  'Content-Type': 'application/json'
};
const data = {
  "url": "https://www.example.com",
  "method": "GET",
  "parse": false,
  "render": true,
  "country": "CA",
  "locale": "en-GB",
  "headers": {
    "Some-Extra-Header": "Some-Extra-Header"
  }
};

axios.post(url, data, { headers })
  .then(response => {
    console.log(response.status);
    console.log(response.data);
  })
  .catch(error => {
    console.error(error);
  });

Go

package main

import (
	"bytes"
	"encoding/base64"
	"fmt"
	"net/http"
	"encoding/json"
)

func main() {
	url := "https://api.webit.live/api/v1/realtime/web"
	payload := []byte(`{
		"url": "https://www.example.com",
		"method": "GET",
		"parse": false,
		"render": true,
		"country": "CA",
		"locale": "en-GB",
		"headers": {
			"Some-Extra-Header": "Some-Extra-Header"
		}
	}`)
	headers := map[string]string{
		"Authorization":  "Basic <credential string>",
		"Content-Type":   "application/json",
	}

	req, err := http.NewRequest("POST", url, bytes.NewBuffer(payload))
	if err != nil {
		fmt.Println(err)
		return
	}

	for key, value := range headers {
		req.Header.Set(key, value)
	}

	client := &http.Client{}
	resp, err := client.Do(req)
	if err != nil {
		fmt.Println(err)
		return
	}
	defer resp.Body.Close()

	fmt.Println(resp.StatusCode)
	// Read the response body if needed
	// body, err := ioutil.ReadAll(resp.Body)
	// fmt.Println(string(body))
}

Every request sent through Nimble API is automatically routed through Nimble IP - our premium proxy network!

Request options

Parameter	Required	Description
url	Required	URL | The page or resource to be fetched. Note: when using a URL with a query string, encode the URL and place it at the end of the query string.
method	Optional (default = GET)	String | The method for requesting a URL from the target server.
country	Optional (default = all)	String | Country used to access the target URL, use ISO Alpha-2 Country Codes i.e. US, DE, GB
locale	Optional (default = EN)	String | LCID standard locale used for the URL request.
headers*	Optional	String | JSON with key/value structure to pass the required headers.
cookies	Optional	String | Attach a cookie or cookies to your request. See Sending cookies with a request for details.
format	Optional (default = JSON)	Enum: JSON | HTML - The data response format. HTML - in case of error, returns JSON with error message.
parse	Optional (default = false)	Enum: true | false - True - the page's content will be parsed and returned in a JSON format. False - Response will include page headers and raw data (without parsing). When using parse = true, format must be set to “JSON”.
parser	Optional (default = null)	String | Define a custom parsing template. See Parsing Templates.
render	Optional (default = false)	Enum: true | false - enables or disables Javascript rendering on the target page.
render_options	N/A	A collection of rendering-related settings. See Rendering options.
render_flow	Optional (default = null)	String | Define a series of actions to be performed on the page prior to data collection. See Page Interactions.

* Please do not include any cookies when sending custom headers. To send cookies, please see the Sending cookies with a request section.

Rendering options

The parameter render_options includes several configuration options that allow users to fine-tune how Nimble renders webpages. These options include:

Parameter	Required	Description
include_iframes	No (default = false)	Boolean - Instructs Nimble APIs to render or not to render iframes.
timeout	No (default = 30000) Maximum: 60000	Integer (ms) - Defines how long Nimble APIs will wait for a page to finish loading.
render_type	No (default = load)	Enum - Defines the state that Nimble APIs will consider a page as fully loaded.

Because many webpages load additional content after the initial page has finished loading (such as single-page applications), deciding exactly when a webpage has finished loading is often ambiguous. To aid with this, Nimble APIs supports several options:

• load - Nimble will consider a page ready after the standard page load event is fired.

• domready - Nimble will consider a page ready when the DOMContentLoaded event is fired.

• idle2 - Nimble will consider a page ready when there are 2 or fewer network connections made during the last 500ms.

• idle0 - Nimble will consider a page ready when no new network connections are made during the last 500ms.

In the below example, all three rendering options are demonstrated:

cURL

curl -X POST 'https://api.webit.live/api/v1/realtime/web' \
--header 'Authorization: Basic <credential string>' \
--header 'Content-Type: application/json' \
--data-raw '{
    "url": "https://www.example.com",
    "method": "GET",
    "parse": true,
    "render": true,
    "render_options": {
        "include_iframes": true,
        "render_type": "idle0",
	"timeout": 35000
    }
}'

Python

import requests

url = 'https://api.webit.live/api/v1/realtime/web'
headers = {
    'Authorization': 'Basic <credential string>',
    'Content-Type': 'application/json'
}
data = {
    "url": "https://www.example.com",
    "method": "GET",
    "parse": True,
    "render": True,
    "render_options": {
        "include_iframes": True,
        "render_type": "idle0",
        "timeout": 35000
    }
}

response = requests.post(url, headers=headers, json=data)

print(response.status_code)
print(response.json())

Node.js

const axios = require('axios');

const url = 'https://api.webit.live/api/v1/realtime/web';
const headers = {
  'Authorization': 'Basic <credential string>',
  'Content-Type': 'application/json'
};
const data = {
  "url": "https://www.example.com",
  "method": "GET",
  "parse": true,
  "render": true,
  "render_options": {
    "include_iframes": true,
    "render_type": "idle0",
    "timeout": 35000
  }
};

axios.post(url, data, { headers })
  .then(response => {
    console.log(response.status);
    console.log(response.data);
  })
  .catch(error => {
    console.error(error);
  });

Go

package main

import (
	"bytes"
	"fmt"
	"net/http"
	"encoding/json"
)

func main() {
	url := "https://api.webit.live/api/v1/realtime/web"
	payload := []byte(`{
		"url": "https://www.example.com",
		"method": "GET",
		"parse": true,
		"render": true,
		"render_options": {
			"include_iframes": true,
			"render_type": "idle0",
			"timeout": 35000
		}
	}`)
	headers := map[string]string{
		"Authorization":  "Basic <credential string>",
		"Content-Type":   "application/json",
	}

	req, err := http.NewRequest("POST", url, bytes.NewBuffer(payload))
	if err != nil {
		fmt.Println(err)
		return
	}

	for key, value := range headers {
		req.Header.Set(key, value)
	}

	client := &http.Client{}
	resp, err := client.Do(req)
	if err != nil {
		fmt.Println(err)
		return
	}
	defer resp.Body.Close()

	fmt.Println(resp.StatusCode)
	// Read the response body if needed
	// body, err := ioutil.ReadAll(resp.Body)
	// fmt.Println(string(body))
}

Sending cookies with a request

Many websites use cookies to store important user information, such as preferences, locales, locations, and more. Nimble's Web API allows for cookies to be defined in a string or object format.

Cookie object

The cookie object allows for one or multiple cookies to be set, with each one containing three parameters:

• key - the name of the cookie.

• value - the value of the cookie.

• domain - The domain with which this cookie is associated. If a domain is not defined, Nimble will use the requested URL's domain by default.

In the following example, a cookie is used to select a particular store on the Lowe's website:

cURL

curl --location --request POST 'https://api.webit.live/api/v1/realtime/web/' \
--header 'Authorization: Basic <credential string>' \
--header 'Content-Type: application/json' \
--data-raw '{
    "url": "https://www.lowes.com/pd/CRAFTSMAN-M110-140-cc-21-in-Gas-Push-Lawn-Mower-with-Briggs-Stratton-Engine/1000676311",
    "country": "US",
    "cookies": [
        {
            "key": "sd",
            "value": "%7B%22id%22%3A%222209%22%2C%22zip%22%3A%2204769%22%2C%22city%22%3A%22Presque%20Isle%22%2C%22state%22%3A%22ME%22%2C%22name%22%3A%22Presque%20Isle%20Lowe's%22%2C%22region%22%3A%2218%22%7D",
            "domain": "lowes.com"
        }
    ]
}'

Python

import requests

url = 'https://api.webit.live/api/v1/realtime/web'
headers = {
    'Authorization': 'Basic <credential string>',
    'Content-Type': 'application/json'
}
data = {
    "url": "https://www.lowes.com/pd/CRAFTSMAN-M110-140-cc-21-in-Gas-Push-Lawn-Mower-with-Briggs-Stratton-Engine/1000676311",
    "country": "CA",
    "cookies": [
        {
            "key": "sd",
            "value": "%7B%22id%22%3A%222209%22%2C%22zip%22%3A%2204769%22%2C%22city%22%3A%22Presque%20Isle%22%2C%22state%22%3A%22ME%22%2C%22name%22%3A%22Presque%20Isle%20Lowe's%22%2C%22region%22%3A%2218%22%7D",
            "domain": "lowes.com"
        }
    ] 
}

response = requests.post(url, headers=headers, json=data)

Node.js

const axios = require('axios');

const url = 'https://api.webit.live/api/v1/realtime/web';
const headers = {
  'Authorization': 'Basic <credential string>',
  'Content-Type': 'application/json'
};
const data = {
  "url": "https://www.lowes.com/pd/CRAFTSMAN-M110-140-cc-21-in-Gas-Push-Lawn-Mower-with-Briggs-Stratton-Engine/1000676311",
  "country": "CA",
  "cookies": [
        {
            "key": "sd",
            "value": "%7B%22id%22%3A%222209%22%2C%22zip%22%3A%2204769%22%2C%22city%22%3A%22Presque%20Isle%22%2C%22state%22%3A%22ME%22%2C%22name%22%3A%22Presque%20Isle%20Lowe's%22%2C%22region%22%3A%2218%22%7D",
            "domain": "lowes.com"
        }
    ]
};

axios.post(url, data, { headers })
  .then(response => {
    console.log(response.status);
    console.log(response.data);
  })
  .catch(error => {
    console.error(error);
  });

Go

package main

import (
	"bytes"
	"encoding/base64"
	"fmt"
	"net/http"
	"encoding/json"
)

func main() {
	url := "https://api.webit.live/api/v1/realtime/web"
	payload := []byte(`{
		"url": "https://www.lowes.com/pd/CRAFTSMAN-M110-140-cc-21-in-Gas-Push-Lawn-Mower-with-Briggs-Stratton-Engine/1000676311",
		"country": "CA",
		"cookies": [
		        {
		            "key": "sd",
		            "value": "%7B%22id%22%3A%222209%22%2C%22zip%22%3A%2204769%22%2C%22city%22%3A%22Presque%20Isle%22%2C%22state%22%3A%22ME%22%2C%22name%22%3A%22Presque%20Isle%20Lowe's%22%2C%22region%22%3A%2218%22%7D",
		            "domain": "lowes.com"
		        }
		    ]
	}`)
	headers := map[string]string{
		"Authorization":  "Basic <credential string>",
		"Content-Type":   "application/json",
	}

	req, err := http.NewRequest("POST", url, bytes.NewBuffer(payload))
	if err != nil {
		fmt.Println(err)
		return
	}

	for key, value := range headers {
		req.Header.Set(key, value)
	}

	client := &http.Client{}
	resp, err := client.Do(req)
	if err != nil {
		fmt.Println(err)
		return
	}
	defer resp.Body.Close()

	fmt.Println(resp.StatusCode)
	// Read the response body if needed
	// body, err := ioutil.ReadAll(resp.Body)
	// fmt.Println(string(body))
}

In the above example, the key is set to sd and the value is set to an encoded version of the following location object:

{"id":"2209","zip":"04769","city":"Presque Isle","state":"ME","name":"Presque Isle Lowe's","region":"18"}

These values are unique to Lowe's website, and are not necessarily applicable to other websites.

Cookie string

The cookie string uses a syntax of key1=value1;key2=value2;... format, and does not accept a domain parameter, always defaulting to the requested URL's domain.

For example:

cURL

curl --location --request POST 'https://api.webit.live/api/v1/realtime/web/' \
--header 'Authorization: Basic <credential string>' \
--header 'Content-Type: application/json' \
--data-raw '{
    "url": "https://www.lowes.com/pd/CRAFTSMAN-M110-140-cc-21-in-Gas-Push-Lawn-Mower-with-Briggs-Stratton-Engine/1000676311",
    "country": "US",
    "cookies": "key1=value1;sd=%7B%22id%22%3A%222209..."
}'

Python

import requests

url = 'https://api.webit.live/api/v1/realtime/web'
headers = {
    'Authorization': 'Basic <credential string>',
    'Content-Type': 'application/json'
}
data = {
    "url": "https://www.lowes.com/pd/CRAFTSMAN-M110-140-cc-21-in-Gas-Push-Lawn-Mower-with-Briggs-Stratton-Engine/1000676311",
    "country": "CA",
    "cookies": "key1=value1;sd=%7B%22id%22%3A%222209..."
}

response = requests.post(url, headers=headers, json=data)

Node.js

const axios = require('axios');

const url = 'https://api.webit.live/api/v1/realtime/web';
const headers = {
  'Authorization': 'Basic <credential string>',
  'Content-Type': 'application/json'
};
const data = {
  "url": "https://www.lowes.com/pd/CRAFTSMAN-M110-140-cc-21-in-Gas-Push-Lawn-Mower-with-Briggs-Stratton-Engine/1000676311",
  "country": "CA",
  "cookies": "key1=value1;sd=%7B%22id%22%3A%222209..."
};

axios.post(url, data, { headers })
  .then(response => {
    console.log(response.status);
    console.log(response.data);
  })
  .catch(error => {
    console.error(error);
  });

Go

package main

import (
	"bytes"
	"encoding/base64"
	"fmt"
	"net/http"
	"encoding/json"
)

func main() {
	url := "https://api.webit.live/api/v1/realtime/web"
	payload := []byte(`{
		"url": "https://www.lowes.com/pd/CRAFTSMAN-M110-140-cc-21-in-Gas-Push-Lawn-Mower-with-Briggs-Stratton-Engine/1000676311",
		"country": "CA",
		"cookies": "key1=value1;sd=%7B%22id%22%3A%222209..."
	}`)
	headers := map[string]string{
		"Authorization":  "Basic <credential string>",
		"Content-Type":   "application/json",
	}

	req, err := http.NewRequest("POST", url, bytes.NewBuffer(payload))
	if err != nil {
		fmt.Println(err)
		return
	}

	for key, value := range headers {
		req.Header.Set(key, value)
	}

	client := &http.Client{}
	resp, err := client.Do(req)
	if err != nil {
		fmt.Println(err)
		return
	}
	defer resp.Body.Close()

	fmt.Println(resp.StatusCode)
	// Read the response body if needed
	// body, err := ioutil.ReadAll(resp.Body)
	// fmt.Println(string(body))
}

Sending POST requests to the Target Server

If you wish to send POST data with your request, such as when submitting a form, please use the following syntax:

cURL

curl -X POST 'https://api.webit.live/api/v1/realtime/web' \
--header 'Authorization: Basic <credential string>' \
--header 'Content-Type: application/json' \
--data-raw '{
    "url": "https://www.example.com",
    "method": "POST",
    "headers": {
    "Content-Type": "application/json",
        "Some-Extra-Header": "some-extra-header"
    },
    "body": {
        "a": 1,
        "b": 2
    }
}'

Python

import requests

url = 'https://api.webit.live/api/v1/realtime/web'
headers = {
    'Authorization': 'Basic <credential string>',
    'Content-Type': 'application/json'
}
data = {
    "url": "https://www.example.com",
    "method": "POST",
    "headers": {
        "Content-Type": "application/json",
        "Some-Extra-Header": "some-extra-header"
    },
    "body": {
        "a": 1,
        "b": 2
    }
}

response = requests.post(url, headers=headers, json=data)

Node.js

const axios = require('axios');

const url = 'https://api.webit.live/api/v1/realtime/web';
const headers = {
  'Authorization': 'Basic <credential string>',
  'Content-Type': 'application/json'
};
const data = {
  "url": "https://www.example.com",
  "method": "POST",
  "headers": {
    "Content-Type": "application/json",
    "Some-Extra-Header": "some-extra-header"
  },
  "body": {
    "a": 1,
    "b": 2
  }
};

axios.post(url, data, { headers })
  .then(response => {
    console.log(response.status);
    console.log(response.data);
  })
  .catch(error => {
    console.error(error);
  });

Go

package main

import (
	"bytes"
	"encoding/base64"
	"fmt"
	"net/http"
	"encoding/json"
)

func main() {
	url := "https://api.webit.live/api/v1/realtime/web"
	payload := []byte(`{
		"url": "https://www.example.com",
		"method": "POST",
		"headers": {
			"Content-Type": "application/json",
			"Some-Extra-Header": "some-extra-header"
		},
		"body": {
			"a": 1,
			"b": 2
		}
	}`)
	headers := map[string]string{
		"Authorization":  "Basic <credential string>",
		"Content-Type":   "application/json",
	}

	req, err := http.NewRequest("POST", url, bytes.NewBuffer(payload))
	if err != nil {
		fmt.Println(err)
		return
	}

	for key, value := range headers {
		req.Header.Set(key, value)
	}

	client := &http.Client{}
	resp, err := client.Do(req)
	if err != nil {
		fmt.Println(err)
		return
	}
	defer resp.Body.Close()

	fmt.Println(resp.StatusCode)
	// Read the response body if needed
	// body, err := ioutil.ReadAll(resp.Body)
	// fmt.Println(string(body))
}

Please note that specifying a content-type header is required when submitting a POST request with data in the body.

Additional request options

Parameter	Required	Description
method	Required	Use POST to send forms to the target site
headers	Optional	JSON with key/value structure to pass the required headers
body	Optional	JSON with key/value structure or string to pass needed payload

Response

Headers

X-Task-ID: string

Payload examples:

If parsing was disabled or omitted in the request, the result data will be the raw HTML of the page you requested. If parsing was enabled, a JSON object with a parsed version of the page will be delivered, with the raw HTML included under the html_content property.

200 OK

{
	"status": "success",
	"query_time": "2023-01-01T12:00:00.007Z",
	"status_code": 200,
	"headers": {},
	"html_content": string,
	"parsing": {
		"status": "success",
		"entities": { },
		"total_entities_count": 0,
		"entities_count": { }
	},
	"url": "https://www.google.com/search?q=hello world"
}

500 Error

{
    "status": "error",
    "task_id": "<task_id>",
    "msg": "can't download the query response - please try again"
}

400 Input Error

{
        "status": "failed",
        "msg": error
}

Response Codes

Status	Description
200	OK
400	The requested resource could not be reached
401	Unauthorized/invalid credental string
500	Internal service error
501	An error was encountered by the proxy service