Real-time URL request

Public web data is collected by sending a request with a URL to the Web API. It's also possible to include custom headers, targets, or additional characters if the URL contains non-ASCII characters by supplying request parameters in a valid JSON format. To make a real-time request, use the following syntax:

Nimble APIs requires that a base64 encoded credential string be sent with every request to authenticate your account. For detailed examples, see Web API Authentication.

cURL

curl -X POST 'https://api.webit.live/api/v1/realtime/web' \
--header 'Authorization: Basic <credential string>' \
--header 'Content-Type: application/json' \
--data-raw '{
    "url": "https://www.example.com",
    "method": "GET",
    "parse": false,
    "render": true,
    "country": "US",
    "headers": {
        "Some-Extra-Header": "Some-Extra-Header"
    }
}'

Python

import requests

url = 'https://api.webit.live/api/v1/realtime/web'
headers = {
    'Authorization': 'Basic <credential string>',
    'Content-Type': 'application/json'
}
data = {
    "url": "https://www.example.com",
    "method": "GET",
    "parse": False,
    "render": True,
    "country": "CA",
    "locale": "en-GB",
    "headers": {
        "Some-Extra-Header": "Some-Extra-Header"
    }
}

response = requests.post(url, headers=headers, json=data)

Node.js

const axios = require('axios');

const url = 'https://api.webit.live/api/v1/realtime/web';
const headers = {
  'Authorization': 'Basic <credential string>',
  'Content-Type': 'application/json'
};
const data = {
  "url": "https://www.example.com",
  "method": "GET",
  "parse": false,
  "render": true,
  "country": "CA",
  "locale": "en-GB",
  "headers": {
    "Some-Extra-Header": "Some-Extra-Header"
  }
};

axios.post(url, data, { headers })
  .then(response => {
    console.log(response.status);
    console.log(response.data);
  })
  .catch(error => {
    console.error(error);
  });

Go

package main

import (
	"bytes"
	"encoding/base64"
	"fmt"
	"net/http"
	"encoding/json"
)

func main() {
	url := "https://api.webit.live/api/v1/realtime/web"
	payload := []byte(`{
		"url": "https://www.example.com",
		"method": "GET",
		"parse": false,
		"render": true,
		"country": "CA",
		"locale": "en-GB",
		"headers": {
			"Some-Extra-Header": "Some-Extra-Header"
		}
	}`)
	headers := map[string]string{
		"Authorization":  "Basic <credential string>",
		"Content-Type":   "application/json",
	}

	req, err := http.NewRequest("POST", url, bytes.NewBuffer(payload))
	if err != nil {
		fmt.Println(err)
		return
	}

	for key, value := range headers {
		req.Header.Set(key, value)
	}

	client := &http.Client{}
	resp, err := client.Do(req)
	if err != nil {
		fmt.Println(err)
		return
	}
	defer resp.Body.Close()

	fmt.Println(resp.StatusCode)
	// Read the response body if needed
	// body, err := ioutil.ReadAll(resp.Body)
	// fmt.Println(string(body))
}

Every request sent through Nimble API is automatically routed through Nimble IP - our premium proxy network!

Request options

Parameter	Required	Description
url	Required	URL | The page or resource to be fetched. Note: when using a URL with a query string, encode the URL and place it at the end of the query string.
method	Optional (default = GET)	String | The method for requesting a URL from the target server.
country	Optional (default = all)	String | Country used to access the target URL, use ISO Alpha-2 Country Codes i.e. US, DE, GB
state	Optional	String | For targeting US states (does not include regions or territories in other countries). Two-letter state code, e.g. NY, IL, etc.
city	Optional	String | For targeting large cities and metro areas around the globe. When targeting major US cities, you must include state as well. Click here for a list of available cities.
locale	Optional (default = EN)	String | LCID standard locale used for the URL request. Alternatively, user can use auto for automatic locale based on country targeting.
headers*	Optional	String | JSON with key/value structure to pass the required headers.
cookies	Optional	String | Attach a cookie or cookies to your request. See Sending cookies with a request for details.
format	Optional (default = JSON)	Enum: JSON | HTML - The data response format. HTML - in case of error, returns JSON with error message.
parse	Optional (default = false)	Enum: true | false - True - the page's content will be parsed and returned in a JSON format. False - Response will include page headers and raw data (without parsing). When using parse = true, format must be set to “JSON”.
parser	Optional (default = null)	String | Define a custom parsing template. See Parsing Templates.
render	Optional (default = false)	Enum: true | false - enables or disables Javascript rendering on the target page.
render_options	N/A	A collection of rendering-related settings. See Rendering options.
render_flow	Optional (default = null)	String | Define a series of actions to be performed on the page prior to data collection. See Page Interactions.
is_xhr	Optional (default = false)	Boolean | Instructs the Web API that the target page is an XHR request instead of a standard webpage (only available when render is set to false).
consent_header	Optional (default = false)	Boolean | Adds scraping consent header letting the target website know it is being scraped by Nimble.

* Please do not include any cookies when sending custom headers. To send cookies, please see the Sending cookies with a request section.

Response

Headers

X-Task-ID: string

Payload examples:

If parsing was disabled or omitted in the request, the result data will be the raw HTML of the page you requested. If parsing was enabled, a JSON object with a parsed version of the page will be delivered, with the raw HTML included under the html_content property.

200 OK

{
	"status": "success",
	"query_time": "2023-01-01T12:00:00.007Z",
	"status_code": 200,
	"headers": {},
	"html_content": string,
	"parsing": {
		"status": "success",
		"entities": { },
		"total_entities_count": 0,
		"entities_count": { }
	},
	"url": "https://www.google.com/search?q=hello world"
}

500 Error

{
    "status": "error",
    "task_id": "<task_id>",
    "msg": "can't download the query response - please try again"
}

400 Input Error

{
        "status": "failed",
        "msg": error
}

Response Codes

Status	Description
200	OK
400	The requested resource could not be reached
401	Unauthorized/invalid credental string
500	Internal service error
501	An error was encountered by the proxy service