LogoLogo
Nimble HomeLoginCreate an Account
  • Home
  • Quick Start Tutorials
    • Tutorial Library
      • Track SEO and SEM Ranking
      • Reddit as a Guerilla Marketing Strategy
  • Nimble Platform
    • Nimble Platform Overview
    • Online Pipelines
      • Supermarkets
        • ASDA
        • Tesco Groceries
        • Sainsbury’s
        • Morrisons
      • eCommerce
      • Restaurants
        • Yelp
        • Tabelog
        • Uber Eats Japan
        • Demaecan
        • Uber Eats US
      • Real Estate
        • Zillow
  • Nimble SDK
    • SDK Overview
    • Web API
      • Web API Overview
      • API Product Specs
      • Nimble Web API Quick Start Guide
        • Introduction
        • Nimble APIs Authentication
        • Real-time URL request
        • Delivery methods
        • Batch processing
        • Response codes
        • FAQs
      • Nimble Web API Functions
        • Realtime, Async & Batch Request
        • Geo Location Targeting
        • Javascript Rendering
        • Page Interaction
          • Wait (delay)
          • Wait for Selector
          • Wait and Click
          • Wait and Type
          • Scroll
          • Scroll to
          • Infinite Scrolling
          • Capturing Screenshots
          • Collecting Cookies
          • Executing HTTP Requests
          • Operation Reference
        • Network Capture
          • Filter by URL Matching
          • Filter By Resource Type
            • Real World Demo: Capturing Ajax Requests
          • Wait for Requests
          • Capturing XHR without Rendering
          • Operation Reference
        • Data Parsing
          • Parsing Templates
          • Merge Dynamic Parser
        • Custom Headers & Cookies
        • General Params
      • Vertical Endpoints
        • SERP API
          • Real-time search request
          • Getting local data
          • Browsing SERP pagination
          • Delivery methods
          • Batch Processing
          • Endpoints and Response Codes
        • Maps API
          • Searching for places
          • Getting information about a place
          • Collecting reviews
          • Delivery methods
          • Batch processing
          • Endpoints and Response Codes
    • Web Retrieval API
      • Web Retrieval API Overview
    • Proxy API
      • Nimble IP Overview
      • Nimble IP Quick Start Guide
        • Send a request
        • Nimble IP Autentication
        • Geotargeting and session control
        • Response codes
        • FAQs
      • Nimble IP Functions
        • Country/state/city geotargeting
        • Controlling IP rotation
        • Geo-sessions: longer, stickier, more accurate sessions
        • Using IPv6 Proxies
        • Response Codes
      • Integration Guides
        • Incogniton
        • Kameleo
        • VMLogin
        • AdsPower
        • FoxyProxy
        • Android
        • Multilogin
        • iOS
        • SwitchyOmega
        • Windows
        • macOS
        • Proxifier
        • MuLogin
        • Puppeteer
        • Selenium
        • Scrapy
    • Client Libraries
      • Installation
      • Quick Start
    • LangChain Integration
  • Technologies
    • Browserless Drivers
      • API Driver-Based Pricing
    • IP Optimization Models
    • AI Parsing Skills
  • Management Tools
    • Nimble Dashboard
      • Exploring the User Dashboard
      • Managing Pipelines
      • Reporting and Analytics
      • Account Settings
      • Experimenting with the Playground
      • Billing and history
    • Nimble Admin API
      • Admin API basics
      • Admin API reference
  • General
    • Onboarding Guide
      • Getting started with Nimble's User Dashboard
      • Nimble IP Basics
      • Nimble API Basics
      • Helpful Resources
    • FAQs
      • Account Settings and Security
      • Billing and Pricing
      • Tools and Integrations
      • Nimble API
      • Nimble IP
    • Deprecated APIs
      • E-commerce API
        • E-commerce API Authentication
        • Real-time product request
        • Real-time product search request
        • Delivery methods
        • Batch Processing
        • Endpoints and Response Codes
      • Unlocker Proxy Overview
        • Unlocker Proxy Quick Start Guide
          • Real-time request
          • FAQs
        • Unlocker Proxy FAQ
Powered by GitBook
On this page
  • What?
  • Why?
  • Additional Information
  • Request Option
  • Example Request - Sending POST Request with Headers
  • Example Request - Cookie Object
  • Example Request - Cookie String
  1. Nimble SDK
  2. Web API
  3. Nimble Web API Functions

Custom Headers & Cookies

PreviousMerge Dynamic ParserNextGeneral Params

Last updated 8 months ago

What?

The Nimble Web API provides the ability to enhance your scraping requests by including custom headers or cookies. These can be defined in either string or object format and sent as part of your request to the target domain. This feature is crucial for accessing additional data that may be gated behind specific user settings or preferences stored in cookies, allowing you to interact with the website in a more personalized and context-aware manner.

Why?

  • Unlock Access to Personalized Data: By including custom headers or cookies, you can retrieve data that is tailored to specific user session contexts, which is often otherwise inaccessible.

  • Enhance Data Retrieval Capabilities: Support diverse use cases by leveraging cookies and headers to interact with websites in ways that mimic real user behavior, ensuring you capture the most relevant data.

  • Improve Data Retrieval Efficiency: Reduce the need for additional scraping attempts by ensuring that your initial requests are as comprehensive and accurate as possible, saving time and resources.

Additional Information

  • Supported by real-time, asynchronous, and batch requests.

  • Supported Endpoints:

  • Not supported Endpoints: , , and .

Misconfiguration of custom headers or cookies may negatively impact the success rates of your scraping tasks, as improper use may trigger detection by the target website. This feature is intended for advanced users, so please exercise caution and ensure configurations are accurate.

Request Option

Parameter
Required
Description

url

Required

URL | The page or resource to be fetched. Note: when using a URL with a query string, encode the URL and place it at the end of the query string

headers*

Optional

String | JSON with key/value structure to pass the required headers

method

Required when headers is used to send POST request

Enum | GET, POST. Use POST to send forms to the target site

body

Required when headers is used to send POST request

Object | JSON with key/value structure or string to pass needed payload

cookies

Optional

String | The cookie string uses a syntax of key1=value1;key2=value2;... format, and does not accept a domain parameter, always defaulting to the requested URL's domain. OR List Objects | List of attach cookie objects. The cookie object allows for one or multiple cookies to be set, with each one containing three parameters - key, value, domain

cookies.key

Required when cookies is used

String | The name of the attached cookie

cookies.value

Required when cookies is used

String | The value of the attached cookie.

cookies.domain

Optional

URL | The domain with which this cookie is associated. If a domain is not defined, Nimble will use the requested URL's domain by default.

* Please do not include any cookies when sending custom headers.

Example Request - Sending POST Request with Headers

If you wish to send POST data with your target domain as part of your request, such as when submitting a form, please use the following syntax:

curl -X POST 'https://api.webit.live/api/v1/realtime/web' \
--header 'Authorization: Basic <credential string>' \
--header 'Content-Type: application/json' \
--data-raw '{
    "url": "https://www.example.com",
    "method": "POST",
    "headers": {
    "Content-Type": "application/json",
        "Some-Extra-Header": "some-extra-header"
    },
    "body": {
        "a": 1,
        "b": 2
    }
}'
import requests

url = 'https://api.webit.live/api/v1/realtime/web'
headers = {
    'Authorization': 'Basic <credential string>',
    'Content-Type': 'application/json'
}
data = {
    "url": "https://www.example.com",
    "method": "POST",
    "headers": {
        "Content-Type": "application/json",
        "Some-Extra-Header": "some-extra-header"
    },
    "body": {
        "a": 1,
        "b": 2
    }
}

response = requests.post(url, headers=headers, json=data)
const axios = require('axios');

const url = 'https://api.webit.live/api/v1/realtime/web';
const headers = {
  'Authorization': 'Basic <credential string>',
  'Content-Type': 'application/json'
};
const data = {
  "url": "https://www.example.com",
  "method": "POST",
  "headers": {
    "Content-Type": "application/json",
    "Some-Extra-Header": "some-extra-header"
  },
  "body": {
    "a": 1,
    "b": 2
  }
};

axios.post(url, data, { headers })
  .then(response => {
    console.log(response.status);
    console.log(response.data);
  })
  .catch(error => {
    console.error(error);
  });
package main

import (
	"bytes"
	"encoding/base64"
	"fmt"
	"net/http"
	"encoding/json"
)

func main() {
	url := "https://api.webit.live/api/v1/realtime/web"
	payload := []byte(`{
		"url": "https://www.example.com",
		"method": "POST",
		"headers": {
			"Content-Type": "application/json",
			"Some-Extra-Header": "some-extra-header"
		},
		"body": {
			"a": 1,
			"b": 2
		}
	}`)
	headers := map[string]string{
		"Authorization":  "Basic <credential string>",
		"Content-Type":   "application/json",
	}

	req, err := http.NewRequest("POST", url, bytes.NewBuffer(payload))
	if err != nil {
		fmt.Println(err)
		return
	}

	for key, value := range headers {
		req.Header.Set(key, value)
	}

	client := &http.Client{}
	resp, err := client.Do(req)
	if err != nil {
		fmt.Println(err)
		return
	}
	defer resp.Body.Close()

	fmt.Println(resp.StatusCode)
	// Read the response body if needed
	// body, err := ioutil.ReadAll(resp.Body)
	// fmt.Println(string(body))
}

Please note that specifying a content-type header is required when submitting a POST request with data in the body.

Example Request - Cookie Object

In the following example, a Cookie Object is used to select a particular store on the Lowe's website.

curl --location --request POST 'https://api.webit.live/api/v1/realtime/web/' \
--header 'Authorization: Basic <credential string>' \
--header 'Content-Type: application/json' \
--data-raw '{
    "url": "https://www.lowes.com/pd/CRAFTSMAN-M110-140-cc-21-in-Gas-Push-Lawn-Mower-with-Briggs-Stratton-Engine/1000676311",
    "country": "US",
    "cookies": [
        {
            "key": "sd",
            "value": "%7B%22id%22%3A%222209%22%2C%22zip%22%3A%2204769%22%2C%22city%22%3A%22Presque%20Isle%22%2C%22state%22%3A%22ME%22%2C%22name%22%3A%22Presque%20Isle%20Lowe's%22%2C%22region%22%3A%2218%22%7D",
            "domain": "lowes.com"
        }
    ]
}'
import requests

url = 'https://api.webit.live/api/v1/realtime/web'
headers = {
    'Authorization': 'Basic <credential string>',
    'Content-Type': 'application/json'
}
data = {
    "url": "https://www.lowes.com/pd/CRAFTSMAN-M110-140-cc-21-in-Gas-Push-Lawn-Mower-with-Briggs-Stratton-Engine/1000676311",
    "country": "CA",
    "cookies": [
        {
            "key": "sd",
            "value": "%7B%22id%22%3A%222209%22%2C%22zip%22%3A%2204769%22%2C%22city%22%3A%22Presque%20Isle%22%2C%22state%22%3A%22ME%22%2C%22name%22%3A%22Presque%20Isle%20Lowe's%22%2C%22region%22%3A%2218%22%7D",
            "domain": "lowes.com"
        }
    ] 
}

response = requests.post(url, headers=headers, json=data)
const axios = require('axios');

const url = 'https://api.webit.live/api/v1/realtime/web';
const headers = {
  'Authorization': 'Basic <credential string>',
  'Content-Type': 'application/json'
};
const data = {
  "url": "https://www.lowes.com/pd/CRAFTSMAN-M110-140-cc-21-in-Gas-Push-Lawn-Mower-with-Briggs-Stratton-Engine/1000676311",
  "country": "CA",
  "cookies": [
        {
            "key": "sd",
            "value": "%7B%22id%22%3A%222209%22%2C%22zip%22%3A%2204769%22%2C%22city%22%3A%22Presque%20Isle%22%2C%22state%22%3A%22ME%22%2C%22name%22%3A%22Presque%20Isle%20Lowe's%22%2C%22region%22%3A%2218%22%7D",
            "domain": "lowes.com"
        }
    ]
};

axios.post(url, data, { headers })
  .then(response => {
    console.log(response.status);
    console.log(response.data);
  })
  .catch(error => {
    console.error(error);
  });
package main

import (
	"bytes"
	"encoding/base64"
	"fmt"
	"net/http"
	"encoding/json"
)

func main() {
	url := "https://api.webit.live/api/v1/realtime/web"
	payload := []byte(`{
		"url": "https://www.lowes.com/pd/CRAFTSMAN-M110-140-cc-21-in-Gas-Push-Lawn-Mower-with-Briggs-Stratton-Engine/1000676311",
		"country": "CA",
		"cookies": [
		        {
		            "key": "sd",
		            "value": "%7B%22id%22%3A%222209%22%2C%22zip%22%3A%2204769%22%2C%22city%22%3A%22Presque%20Isle%22%2C%22state%22%3A%22ME%22%2C%22name%22%3A%22Presque%20Isle%20Lowe's%22%2C%22region%22%3A%2218%22%7D",
		            "domain": "lowes.com"
		        }
		    ]
	}`)
	headers := map[string]string{
		"Authorization":  "Basic <credential string>",
		"Content-Type":   "application/json",
	}

	req, err := http.NewRequest("POST", url, bytes.NewBuffer(payload))
	if err != nil {
		fmt.Println(err)
		return
	}

	for key, value := range headers {
		req.Header.Set(key, value)
	}

	client := &http.Client{}
	resp, err := client.Do(req)
	if err != nil {
		fmt.Println(err)
		return
	}
	defer resp.Body.Close()

	fmt.Println(resp.StatusCode)
	// Read the response body if needed
	// body, err := ioutil.ReadAll(resp.Body)
	// fmt.Println(string(body))
}

In the above example, the key is set to sd and the value is set to an encoded version of the following location object:

{
    "id": "2209",
    "zip": "04769",
    "city": "Presque Isle",
    "state": "ME",
    "name": "Presque Isle Lowe's",
    "region": "18"
}

These values are unique to Lowe's website, and are not necessarily applicable to other websites.

Example Request - Cookie String

In the following example, a Cookie String is used to select a particular store on the Lowe's website.

curl --location --request POST 'https://api.webit.live/api/v1/realtime/web/' \
--header 'Authorization: Basic <credential string>' \
--header 'Content-Type: application/json' \
--data-raw '{
    "url": "https://www.lowes.com/pd/CRAFTSMAN-M110-140-cc-21-in-Gas-Push-Lawn-Mower-with-Briggs-Stratton-Engine/1000676311",
    "country": "US",
    "cookies": "key1=value1;sd=%7B%22id%22%3A%222209..."
}'
import requests

url = 'https://api.webit.live/api/v1/realtime/web'
headers = {
    'Authorization': 'Basic <credential string>',
    'Content-Type': 'application/json'
}
data = {
    "url": "https://www.lowes.com/pd/CRAFTSMAN-M110-140-cc-21-in-Gas-Push-Lawn-Mower-with-Briggs-Stratton-Engine/1000676311",
    "country": "CA",
    "cookies": "key1=value1;sd=%7B%22id%22%3A%222209..."
}

response = requests.post(url, headers=headers, json=data)
const axios = require('axios');

const url = 'https://api.webit.live/api/v1/realtime/web';
const headers = {
  'Authorization': 'Basic <credential string>',
  'Content-Type': 'application/json'
};
const data = {
  "url": "https://www.lowes.com/pd/CRAFTSMAN-M110-140-cc-21-in-Gas-Push-Lawn-Mower-with-Briggs-Stratton-Engine/1000676311",
  "country": "CA",
  "cookies": "key1=value1;sd=%7B%22id%22%3A%222209..."
};

axios.post(url, data, { headers })
  .then(response => {
    console.log(response.status);
    console.log(response.data);
  })
  .catch(error => {
    console.error(error);
  });
package main

import (
	"bytes"
	"encoding/base64"
	"fmt"
	"net/http"
	"encoding/json"
)

func main() {
	url := "https://api.webit.live/api/v1/realtime/web"
	payload := []byte(`{
		"url": "https://www.lowes.com/pd/CRAFTSMAN-M110-140-cc-21-in-Gas-Push-Lawn-Mower-with-Briggs-Stratton-Engine/1000676311",
		"country": "CA",
		"cookies": "key1=value1;sd=%7B%22id%22%3A%222209..."
	}`)
	headers := map[string]string{
		"Authorization":  "Basic <credential string>",
		"Content-Type":   "application/json",
	}

	req, err := http.NewRequest("POST", url, bytes.NewBuffer(payload))
	if err != nil {
		fmt.Println(err)
		return
	}

	for key, value := range headers {
		req.Header.Set(key, value)
	}

	client := &http.Client{}
	resp, err := client.Do(req)
	if err != nil {
		fmt.Println(err)
		return
	}
	defer resp.Body.Close()

	fmt.Println(resp.StatusCode)
	// Read the response body if needed
	// body, err := ioutil.ReadAll(resp.Body)
	// fmt.Println(string(body))
}
Web
Social
SERP
Maps
eCommerce