Skip to main content
Scrapy is a popular open-source web crawling and scraping framework for Python. It’s used to crawl websites and extract structured data from their pages, making it ideal for data mining, monitoring, and automated testing. By integrating Nimble with Scrapy, you can route your crawling requests through residential IPs, enabling access to geo-restricted content and avoiding IP-based blocking at scale.

Prerequisites

  • Python 3.8+ installed on your system
  • Nimble account with API credentials

Follow these steps to integrate Nimble

1

Install the Scrapy-Nimble package

Install the official Nimble middleware for Scrapy:
2

Get your API credentials

  • Log in to your Nimble Dashboard
  • Navigate to Settings → API Keys to find your API credentials
  • You’ll need your username and password for the configuration
3

Configure your Scrapy settings

Update your project’s settings.py file to enable the Nimble middleware:
Ensure the Nimble middleware priority (570) is set to run before the default HttpCompressionMiddleware (590).

Using Nimble in your spiders

Once configured, you can use Nimble features in your spider requests via the meta parameter:

Available request options

Example spider with geo-targeting

Development setup

For local development, we recommend using a virtual environment:
That’s all! Scrapy will now route all requests through Nimble’s infrastructure.