Skip to main content

Documentation Index

Fetch the complete documentation index at: https://docs.nimbleway.com/llms.txt

Use this file to discover all available pages before exploring further.

Scrapy is a popular open-source web crawling and scraping framework for Python. It’s used to crawl websites and extract structured data from their pages, making it ideal for data mining, monitoring, and automated testing. By integrating Nimble with Scrapy, you can route your crawling requests through residential IPs, enabling access to geo-restricted content and avoiding IP-based blocking at scale.

​
Prerequisites

  • Python 3.8+ installed on your system
  • Nimble account with API credentials

​
Follow these steps to integrate Nimble

1

Install the Scrapy-Nimble package

Install the official Nimble middleware for Scrapy:
pip install scrapy-nimble
2

Get your API credentials

  • Log in to your Nimble Dashboard
  • Navigate to Account Settings to find your API credentials
  • You’ll need your username and password for the configuration
3

Configure your Scrapy settings

Update your project’s settings.py file to enable the Nimble middleware:
# Enable Nimble integration
NIMBLE_ENABLED = True

# Your Nimble credentials
NIMBLE_USERNAME = "your-username"
NIMBLE_PASSWORD = "your-password"

# Register the Nimble middleware
DOWNLOADER_MIDDLEWARES = {
    "scrapy_nimble.middlewares.NimbleWebApiMiddleware": 570,
}
Ensure the Nimble middleware priority (570) is set to run before the default HttpCompressionMiddleware (590).

​
Using Nimble in your spiders

Once configured, you can use Nimble features in your spider requests via the meta parameter: 
import scrapy

class MySpider(scrapy.Spider):
    name = "my_spider"

    def start_requests(self):
        yield scrapy.Request(
            url="https://example.com",
            callback=self.parse,
            meta={
                "nimble_render": True,      # Enable JavaScript rendering
                "nimble_country": "US",     # Geo-target to US
                "nimble_locale": "en",      # Set locale
            }
        )

    def parse(self, response):
        # Your parsing logic here
        title = response.css("title::text").get()
        yield {"title": title}

​
Available request options

Meta ParameterTypeDescription
nimble_renderbooleanEnable JavaScript rendering
nimble_countrystringTwo-letter country code for geo-targeting (e.g., “US”, “DE”, “GB”)
nimble_localestringLocale setting for the request

​
Example spider with geo-targeting

import scrapy

class GeoSpider(scrapy.Spider):
    name = "geo_spider"

    countries = ["US", "GB", "DE", "FR"]

    def start_requests(self):
        for country in self.countries:
            yield scrapy.Request(
                url="https://example.com",
                callback=self.parse,
                meta={
                    "nimble_country": country,
                    "nimble_render": True,
                },
                cb_kwargs={"country": country}
            )

    def parse(self, response, country):
        yield {
            "country": country,
            "title": response.css("title::text").get(),
            "content_length": len(response.body),
        }

​
Development setup

For local development, we recommend using a virtual environment: 
# Create and activate virtual environment
python -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate

# Install dependencies
pip install scrapy scrapy-nimble

# Create a new Scrapy project
scrapy startproject myproject
That’s all! Scrapy will now route all requests through Nimble’s infrastructure.