# Agent Async Run Source: https://docs.nimbleway.com/api-reference/agent/agent-async-run openapi.json post /v1/agents/async Execute Search Agent Async Endpoint # Agent Run Source: https://docs.nimbleway.com/api-reference/agent/agent-run openapi.json post /v1/agents/run Execute Search Agent Realtime Endpoint # Get Agent Details Source: https://docs.nimbleway.com/api-reference/agent/get-agent-details openapi.json get /v1/agents/{agent_name} # List Agents Source: https://docs.nimbleway.com/api-reference/agent/list-agents openapi.json get /v1/agents # Cancel Crawl Source: https://docs.nimbleway.com/api-reference/crawl/cancel-crawl openapi.json delete /v1/crawl/{id} # Crawl by ID Source: https://docs.nimbleway.com/api-reference/crawl/crawl-by-id openapi.json get /v1/crawl/{id} # Create Crawl Source: https://docs.nimbleway.com/api-reference/crawl/create-crawl openapi.json post /v1/crawl # List Crawls Source: https://docs.nimbleway.com/api-reference/crawl/list-crawls openapi.json get /v1/crawl # Extract Source: https://docs.nimbleway.com/api-reference/extract/extract openapi.json post /v1/extract # Extract Async Source: https://docs.nimbleway.com/api-reference/extract/extract-async openapi.json post /v1/extract/async # Introduction Source: https://docs.nimbleway.com/api-reference/introduction Get started with Nimble's API Nimble API provides powerful web scraping and data extraction capabilities through a simple REST API. Extract data from any website, use pre-built templates for popular platforms, search the web, map site structures, and crawl entire websites at scale. ## API Features Extract clean HTML and structured data from any URL with custom parsing and browser actions. Use pre-built agent for popular platforms like Amazon, Google, LinkedIn, and more. Search the web and retrieve full content from top results with optional AI summaries. Discover all URLs on a website with metadata for planning crawl strategies. Extract data from entire websites with async crawling across multiple pages. Retrieve results from async operations (extract, search, map, crawl). ## Base URL All API requests use the following base URL: ```bash theme={"system"} https://sdk.nimbleway.com/v1 ``` ## Authentication All requests require authentication using a Bearer token in the Authorization header: ```bash theme={"system"} Authorization: Bearer YOUR_API_KEY ``` Get your API key from the [Nimble Dashboard](https://online.nimbleway.com/account-settings/api-keys). ### Example Request ```bash theme={"system"} curl -X POST 'https://sdk.nimbleway.com/v1/extract' \ --header 'Authorization: Bearer YOUR_API_KEY' \ --header 'Content-Type: application/json' \ --data-raw '{ "url": "https://www.example.com", "render": true }' ``` ## Response Codes Nimble uses standard HTTP status codes to indicate the success or failure of requests. | Status | Description | | ------ | -------------------------------------------------------- | | 200 | Request successful. | | 400 | Bad request - check your parameters. | | 401 | Unauthorized - invalid or missing API key. | | 402 | Payment required - insufficient budget or trial expired. | | 422 | Validation error - invalid request parameters. | | 429 | Rate limit exceeded - slow down your requests. | | 500 | Internal server error - Nimble infrastructure issue. | ## Rate Limits Nimble enforces rate limits to ensure service stability and fair usage across all users. ### Default Rate Limits | Driver Type | Rate Limit | | -------------------- | ------------------ | | `vx6`, `vx8`, `vx10` | 83 QPS (5,000 QPM) | **QPS** = Queries Per Second, **QPM** = Queries Per Minute. Rate limits apply per API key. ### Rate Limit Headers Responses include headers showing your current rate limit status: ```bash theme={"system"} X-RateLimit-Limit: 83 X-RateLimit-Remaining: 45 X-RateLimit-Reset: 1609459200 ``` ### Exceeding Rate Limits When you exceed the rate limit, you'll receive a 429 status code with retry information: ```json theme={"system"} { "error": { "code": "RATE_LIMIT_EXCEEDED", "message": "Rate limit exceeded. Current limit: 83 QPS", "retry_after": 2 } } ``` **Need higher limits?** Contact [sales@nimbleway.com](https://login.start-chat.com/modal/601e4da4-50a0-4dc7-8155-bb84f2952c40/27847549-dc04-4144-98a5-d40e568bf6af?magicLinkId=slFQ2j\&UID=65ab324c-ecde-4748-9f52-8e395054774b.1753267174019) for custom rate limits. ## Async Operations Several endpoints support asynchronous processing for long-running operations: * `/v1/extract/async` - Async data extraction * `/v1/agent/async` - Async template extraction * `/v1/search/async` - Async web search * `/v1/map/async` - Async site mapping * `/v1/crawl` - Crawl operations (async only) ### Working with Async Tasks 1. **Start an async operation** - Returns a task ID 2. **Check task status** - Use `/v1/tasks/{id}` to check progress 3. **Retrieve results** - Get results from `/v1/tasks/{id}/results` when complete ```bash theme={"system"} # Start async extraction curl -X POST 'https://sdk.nimbleway.com/v1/extract/async' \ --header 'Authorization: Bearer YOUR_API_KEY' \ --header 'Content-Type: application/json' \ --data-raw '{"url": "https://example.com"}' # Returns: {"task_id": "8e8cfde8-345b-42b8-b3e2-0c61eb11e00f"} # Check task status curl 'https://sdk.nimbleway.com/v1/tasks/8e8cfde8-345b-42b8-b3e2-0c61eb11e00f' \ --header 'Authorization: Bearer YOUR_API_KEY' # Get results when complete curl 'https://sdk.nimbleway.com/v1/tasks/8e8cfde8-345b-42b8-b3e2-0c61eb11e00f/results' \ --header 'Authorization: Bearer YOUR_API_KEY' ``` ## Next Steps Make your first API request in minutes Understand rate limits and technical specifications # Map Source: https://docs.nimbleway.com/api-reference/map/map openapi.json post /v1/map # Search Source: https://docs.nimbleway.com/api-reference/search/search openapi.json post /v1/search # Task Result Source: https://docs.nimbleway.com/api-reference/tasks/task-result openapi.json get /v1/tasks/{id}/results Retrieve the status and results of an async task by ID. Works for all async task types (extract, search, map, agent, crawl). # Task Status Source: https://docs.nimbleway.com/api-reference/tasks/task-status openapi.json get /v1/tasks/{id} Retrieve the status and results of an async task by ID. Works for all async task types (extract, search, map, agent, crawl). # Changelog Source: https://docs.nimbleway.com/changelog/release-notes Stay up to date with the latest Nimble SDK updates and improvements Track new features, improvements, and updates to the Nimble SDK and Web Tools. We ship updates regularly to enhance performance, add capabilities, and improve your developer experience. Have feedback or questions about a release? [Reach out to our team](https://portal.usepylon.com/nimble) ### Release Notes for February, 2026 ### New Features #### GET Agents API New endpoints for discovering and exploring available web search agents. List all available agent templates with `GET /agents` or retrieve detailed information about a specific agent including its input/output schema with `GET /agents/{agent_name}`. \ Perfect for building dynamic agent selections or understanding agent capabilities programmatically. [View Documentation](/nimble-sdk/agentic/agent-gallery) #### Screenshot Format New `screenshot` format option for the Extract API. Capture full-page screenshots as base64-encoded PNG images for visual verification, monitoring, and archival. Screenshots automatically enable JavaScript rendering (VX8/VX10 driver). [View Documentation](/nimble-sdk/web-tools/extract/features/formats) ### Documentation #### Documentation Updates * **Callbacks & Delivery Guide**: Comprehensive guide covering all async result delivery options - polling, webhooks, and cloud delivery to S3/GCS. Includes setup instructions for bucket permissions. [View Guide](/nimble-sdk/admin/callbacks-and-delivery) * **Integration Documentation**: New guide covering all available integration and AI connectors to Nimble SDK capabilities. [View Integrations](/integrations) ### New Features #### Agent Skills API Advanced web search skills powered by Nimble Search API. Built on the open-source [Agent Skills](https://agentskills.io/) standard for cross-platform agent compatibility, connected to you Claude, Cursor and more. Features 8 specialized focus modes (general, coding, news, academic, shopping, social, geo, location) with AI-powered answer generation and smart result filtering. [Try Now](/integrations/agent-skills/nimble-web-tools-skill) ### Release Notes for January, 2026 ### Documentation #### Comprehensive Guides * **Quickstart Pages**: New quickstart guides for all Web Tools with installation steps, basic usage examples, common patterns, and next steps. Get started in under 5 minutes. * **Usage Documentation**: Detailed usage pages for each Web Tool covering all parameters, response formats, best practices, common use cases, and limitations. * **API Reference**: Complete API reference with request/response schemas, error codes, rate limits, and authentication methods for all endpoints. [Try Now](/api-reference/introduction) * **Account Management**: New guide covering API key generation, team member management, permissions, and security best practices. [View Guide](/nimble-sdk/admin/account-management) * **Integration Documentation**: New guide covering all available integration and AI connectors to Nimble SDK capabilities. [View Integrations](/integrations) ### New Features #### Extract API The `/extract` retrieves clean HTML, text, and structured data from any webpage. Supports JavaScript rendering, stealth mode for anti-bot protection, custom parsing with CSS selectors, browser actions, screenshots, and network request capture. [Try Now](/nimble-sdk/web-tools/extract/quickstart) ### New Features #### Web Search Agents API The `/agent` are ready-to-use web search agent extractors for popular websites like Amazon, Google, LinkedIn, and hundreds more. No coding or CSS selectors required - just provide the template name, and template inputs such as search term or product ID and get structured data instantly. Agents are maintained 24/7 by Nimble and automatically updated when sites change. [Try Now](/nimble-sdk/agentic/agents) ### New Features #### Search API The `/search` perform web searches and retrieve parsed content from top results. Supports fast mode for URL discovery, deep search mode with full content extraction, and AI-powered answer summaries from search results. [Try Now](/nimble-sdk/web-tools/search) ### New Features #### Map API The `/map` perform fast URL discovery and site structure mapping. Ideal for discovering all pages on a website, extracting sitemap URLs, and understanding site architecture without deep crawling overhead. Supports subdomain filtering and exact domain matching. [Try Now](/nimble-sdk/web-tools/map) ### New Features #### Crawl API The `/crawl` perform deep website crawling for comprehensive data extraction across multiple pages. Features smart navigation, data normalization, and multi-page scraping with automatic traversing handling. [Try Now](/nimble-sdk/web-tools/crawl) # Build Your First Agent Source: https://docs.nimbleway.com/guides/build-first-agent-tutorial Give Claude one prompt and let the Nimble Agents skill handle agent discovery, custom agent generation, and script writing — end to end ## What You'll Build A Python script that pulls running shoe data from **Amazon**, **Walmart**, and **Nike.com** and outputs a ranked, cross-retailer price comparison — all from a single prompt to your AI coding assistant. The Nimble Agents skill handles the rest: * **Finds** pre-built agents for sites already in the gallery (Amazon, Walmart) * **Generates** a custom agent for any site that isn't (Nike.com) * **Writes** the full analysis script using the agents it collected ## Prerequisites Sign up free and grab your API key from Account Settings Install the Nimble plugin for Claude Code or Cursor You'll also need Python 3.8+ and the Nimble SDK: ```bash theme={"system"} pip install nimble_python ``` *** ## Step 1: Give Claude the Prompt Open a new session in Claude Code (or Cursor) and paste this prompt: ``` Build a Python script that runs competitive analysis on running shoes across Amazon, Walmart, and Nike.com. Use Nimble for all data collection. For each retailer: - Search the Nimble agent gallery for an existing agent. Use it if one exists. - If no agent exists for that site, use the Nimble Agents skill to generate a custom one, then publish it for reuse. The script should collect product name, price, rating (if available), and URL from each source, then print a unified list sorted by price and save the full results to shoe_analysis.json. My Nimble API key is set as the NIMBLE_API_KEY environment variable. ``` ``` Build a Python script that runs competitive analysis on running shoes across Amazon, Walmart, and Nike.com. Use Nimble for all data collection. For each retailer: - Search the Nimble agent gallery for an existing agent. Use it if one exists. - If no agent exists for that site, use the Nimble Agents skill to generate a custom one, then publish it for reuse. The script should collect product name, price, rating (if available), and URL from each source, then print a unified list sorted by price and save the full results to shoe_analysis.json. My Nimble API key is set as the NIMBLE_API_KEY environment variable. ``` That's it. Claude takes it from here. *** ## What Claude Does Next Claude works through the task autonomously using the Nimble Agents skill. Here's what happens under the hood: Claude calls `nimble_agents_list` with the query `"amazon"` and finds `amazon_serp` — a pre-built agent for Amazon search results. It inspects the schema and confirms `keyword` is the required input. Same flow for Walmart — finds `walmart_search`, confirms it takes a `keyword` param and returns product name, price, and rating. No public agent exists for `nike.com`. Claude calls `nimble_agents_generate` with a description of what's needed, waits for the agent to be created, runs a test extraction, and publishes it as `nike_running_shoes_plp`. With all three agents confirmed, Claude writes `shoe_analysis.py` using the Nimble Python SDK and the agent names it just collected. The script Claude produces will look something like this: ```python theme={"system"} import json import os from nimble_python import Nimble nimble = Nimble(api_key=os.environ["NIMBLE_API_KEY"]) KEYWORD = "running shoes" NIKE_URL = "https://www.nike.com/w/mens-running-shoes" def fetch_amazon(): result = nimble.agent.run( agent="amazon_serp", params={"keyword": KEYWORD} ) products = result.data.parsing.get("parsed", []) return [ { "source": "Amazon", "name": p.get("product_name"), "price": p.get("price"), "rating": p.get("average_rating"), "url": p.get("product_url"), } for p in products if p.get("price") ] def fetch_walmart(): result = nimble.agent.run( agent="walmart_search", params={"keyword": KEYWORD} ) products = result.data.parsing.get("parsed", []) return [ { "source": "Walmart", "name": p.get("product_name"), "price": p.get("price"), "rating": p.get("rating"), "url": p.get("product_url"), } for p in products if p.get("price") ] def fetch_nike(): result = nimble.agent.run( agent="nike_running_shoes_plp", params={"url": NIKE_URL} ) products = result.data.parsing.get("parsed", []) return [ { "source": "Nike", "name": p.get("product_name"), "price": p.get("price"), "colors": p.get("colors", []), "url": p.get("url"), } for p in products if p.get("price") ] def run_analysis(): print(f'Competitive analysis: "{KEYWORD}"\n') all_products = [] print("Fetching Amazon...") all_products.extend(fetch_amazon()) print("Fetching Walmart...") all_products.extend(fetch_walmart()) print("Fetching Nike...") all_products.extend(fetch_nike()) sorted_products = sorted(all_products, key=lambda x: x["price"]) print(f"\n{'=' * 55}") print(f" RESULTS — {len(all_products)} products across 3 retailers") print(f"{'=' * 55}\n") for i, p in enumerate(sorted_products[:15], 1): stars = f" ★{p['rating']}" if p.get("rating") else "" colors = f" ({len(p['colors'])} colors)" if p.get("colors") else "" print(f"{i:2}. [{p['source']:7}] ${p['price']:<8.2f} {p['name']}{stars}{colors}") with open("shoe_analysis.json", "w") as f: json.dump(all_products, f, indent=2) print(f"\nFull results saved to shoe_analysis.json") if __name__ == "__main__": run_analysis() ``` The exact agent names (e.g. `nike_running_shoes_plp`) are chosen by the skill at generation time. Claude will use whatever names were returned and write the script accordingly — you don't need to track them manually. *** ## Step 2: Run the Script ```bash theme={"system"} python shoe_analysis.py ``` Expected output: ``` Competitive analysis: "running shoes" Fetching Amazon... Fetching Walmart... Fetching Nike... ======================================================= RESULTS — 62 products across 3 retailers ======================================================= 1. [Walmart ] $19.98 Athletic Works Men's Knit Running Shoe ★4.1 2. [Walmart ] $29.98 Starter Men's Athletic Running Shoe ★4.0 3. [Walmart ] $44.97 Skechers Go Run Consistent ★4.3 4. [Amazon ] $54.99 New Balance Men's 515 V3 Sneaker ★4.5 5. [Walmart ] $54.98 Nike Downshifter 12 ★4.5 6. [Amazon ] $64.99 Adidas Men's Runfalcon 3.0 ★4.6 7. [Amazon ] $69.99 Nike Men's Revolution 7 ★4.6 8. [Nike ] $110.00 Nike Pegasus Trail 5 (3 colors) 9. [Amazon ] $119.95 Brooks Ghost 15 ★4.7 10. [Nike ] $130.00 Nike Pegasus 41 (5 colors) ... Full results saved to shoe_analysis.json ``` *** ## What's Next Target, Best Buy, and others are already in the gallery — just ask Claude to extend the script View and manage your generated Nike agent in the Nimble Platform UI Run all three agents in parallel to cut total runtime by 3× Browse all available public agents across every vertical # Nimble Documentation Source: https://docs.nimbleway.com/home Explore guides, API references, SDKs, and tutorials for building with Nimble's Web Search Agent Platform. Learn how to extract, structure, and stream real-time web data into your applications, AI systems, and analytics workflows with speed, reliability, and scale.

Build with Nimble

The AI-Native SDK for real-time web data

Quickstart Guide → API Reference → Playground → Search Agents Gallery →

See Nimble in Action

Production-ready, real-time web data at any scale

Scalable data collection with stealth unblocking --> Get clean, real-time HTML and structured data from any URL
Input ```python Python theme={"system"} from nimble_python import Nimble nimble = Nimble(api_key="YOUR-API-KEY") result = nimble.extract( url= "https://www.nimbleway.com", formats= ["html", "markdown"] ) ``` ```javascript Node theme={"system"} import Nimble from "@nimble-way/nimble-js"; const nimble = new Nimble({ apiKey: "YOUR-API-KEY" }); const result = await nimble.extract({ url: "https://www.nimbleway.com", formats: ["html", "markdown"], }); ``` ```bash cURL theme={"system"} curl -X POST 'https://sdk.nimbleway.com/v1/extract' \ --header 'Authorization: Bearer ' \ --header 'Content-Type: application/json' \ --data-raw '{ "url": "https://www.nimbleway.com", "formats": ["html", "markdown"] }' ```
Output ```json theme={"system"} { "url": "https://www.nimbleway.com", "status": "success", "data": { "html": "...", "markdown": "# Nimble SDK\n...", "parsing": {...}, "screenshot": [...], "network_capture": [...], ... and more }, "metadata": { "driver": "vx8", ... and more } } ```
Accurate, real-time web search with --> AI Agents search the live web to retrieve precise information
Input ```python Python theme={"system"} from nimble_python import Nimble nimble = Nimble(api_key="YOUR-API-KEY") result = nimble.search( query= "latest AI developments", focus= "general", max_results= 5, include_answer= True, deep_search= False ) ``` ```javascript Node theme={"system"} import Nimble from "@nimble-way/nimble-js"; const nimble = new Nimble({ apiKey: "YOUR-API-KEY" }); const result = await nimble.search({ query: "latest AI developments", focus: "general", max_results: 5, include_answer: true, deep_search: false, }); ``` ```bash cURL theme={"system"} curl -X POST 'https://sdk.nimbleway.com/v1/search' \ --header 'Authorization: Bearer ' \ --header 'Content-Type: application/json' \ --data-raw '{ "query": "latest AI developments", "focus": "general", "max_results": 5, "include_answer": true, "deep_search": false }' ```
Output ```json theme={"system"} { "status": "success", "query": "latest AI developments", "results": [ { "title": "Latest AI Developments", "url": "https://example.com/ai", "snippet": "Recent breakthroughs..." }, ... and more ], "answer": "The latest AI developments..." } ```
Fast URL discovery and site structure mapping --> Easily plan extraction workflows
Input ```python Python theme={"system"} from nimble_python import Nimble nimble = Nimble(api_key="YOUR-API-KEY") result = nimble.map( url= "https://www.nimbleway.com", sitemap: "include" ) ``` ```javascript Node theme={"system"} import Nimble from "@nimble-way/nimble-js"; const nimble = new Nimble({ apiKey: "YOUR-API-KEY" }); const result = await nimble.map({ url: "https://www.nimbleway.com", sitemap: "include", }); ``` ```bash cURL theme={"system"} curl -X POST 'https://sdk.nimbleway.com/v1/map' \ --header 'Authorization: Bearer ' \ --header 'Content-Type: application/json' \ --data-raw '{ "url": "https://www.nimbleway.com", "sitemap": "include" }' ```
Output ```json theme={"system"} { "success": true, "links": [ { "url": "https://www.nimbleway.com", "title": "Nimble - Web Data Platform", "description": "AI-powered web data collection and extraction platform" }, { "url": "https://www.nimbleway.com/pricing", "title": "Pricing", "description": "Nimble pricing plans and features" } ... and more ] } ```
Extract contents from entire websites in a single request --> Collect large volumes of web data automatically
Input ```python Python theme={"system"} from nimble_python import Nimble nimble = Nimble(api_key="YOUR-API-KEY") result = nimble.crawl.run( url="https://docs.nimbleway.com/nimble-sdk", limit=4 ) ``` ```javascript Node theme={"system"} import Nimble from "@nimble-way/nimble-js"; const nimble = new Nimble({ apiKey: "YOUR-API-KEY" }); const result = await nimble.crawl.run({ url: "https://docs.nimbleway.com/nimble-sdk", limit: 4, }); ``` ```bash cURL theme={"system"} curl -X POST 'https://sdk.nimbleway.com/v1/crawl' \ --header 'Authorization: Bearer ' \ --header 'Content-Type: application/json' \ --data-raw '{ "url": "https://docs.nimbleway.com/nimble-sdk", "limit": 4 }' ```
Output ```json theme={"system"} { "crawl_id": "85b21e98-69c5-4aa5-9d25-261a55bbf0db", "url": "https://docs.nimbleway.com/nimble-sdk", "status": "succeeded", "tasks": [ { "task_id": "8d2bb2fc-e469-4549-b5d8-0d307352aeb5", "status": "completed", "created_at": "2026-02-11T13:55:45.846Z", "updated_at": "2026-02-11T13:55:52.138Z" }, ... and more ] } ```
Pre-built Web Search Agents extract and structure live web data from popular domains --> Zero config data collection
Input ```python Python theme={"system"} from nimble_python import Nimble nimble = Nimble(api_key="YOUR-API-KEY") result = nimble.agent.run( agent="amazon_pdp", params={ "asin": "B0DKB1GWML" } ) ``` ```javascript Node theme={"system"} import Nimble from "@nimble-way/nimble-js"; const nimble = new Nimble({ apiKey: "YOUR-API-KEY" }); const result = await nimble.agent.run({ agent: "amazon_pdp", params: { asin: "B0DKB1GWML", }, }); ``` ```bash cURL theme={"system"} curl -X POST 'https://sdk.nimbleway.com/v1/agents/run' \ --header 'Authorization: Bearer ' \ --header 'Content-Type: application/json' \ --data-raw '{ "agent": "amazon_pdp", "params": { "asin": "B0DKB1GWML" } }' ```
Output ```json theme={"system"} { "url": "https://www.amazon.com/dp/B0DKB1GWML", "data": { "html": "...", "parsing": { "product_title": "Apple iPhone 16 Pro Max...", "product_description": ".....", "brand": "Apple", "web_price": 819.97, "availability": true, ... and more } }, "metadata":{ "agent": "amazon_pdp", "driver": "wsa-6m", ... and more } } ```
Route requests through premium residential IPs --> Scale fast with without getting blocked
Input ```bash theme={"system"} # Proxy connection string PROXY=http://account-ACCOUNT-pipeline-PIPELINE-country-US:PASSWORD@ip.nimbleway.com:7000 # Request through US residential IP curl -x $PROXY https://ipinfo.io/json ```
Output ```json theme={"system"} { "ip": "123.0.0.123", "city": "Los Angeles", "region": "California", "country": "US", "loc": "34.0522,-118.2437", "org": "AS12345 Example ISP", "timezone": "America/Los_Angeles" } ```
Nimble
# Nimble Agents Skill Source: https://docs.nimbleway.com/integrations/agent-skills/nimble-agent-skill A guided workflow for finding, generating, and running Nimble agents to get structured data from any website. The Nimble Agents skill provides a step-by-step workflow for extracting structured data from any website using Nimble's agents. ## Quick Install Full plugin with skills and MCP tools Setup guides for all platforms ## How It Works The skill follows a find-or-generate approach: 1. **Search** — Find an existing agent matching your target website 2. **Inspect** — Review the agent's input/output schema 3. **Run** — Execute the agent and get structured results 4. **Generate** — If no existing agent fits, create a custom one using natural language 5. **Publish** — Save generated agents for future reuse Results are always presented in clean markdown tables with numbered follow-up options. ## Prerequisites * **Nimble API Key** — [Sign up](https://online.nimbleway.com/signup) and generate a key from your [Account Settings > API Keys](/nimble-sdk/admin/account-management#api-keys) * **MCP Server Connection** — The skill uses 5 MCP tools (`nimble_agents_list`, `nimble_agents_get`, `nimble_agents_generate`, `nimble_agents_run`, `nimble_agents_publish`) provided by the Nimble MCP server ## Installation See the [Plugin Installation](/integrations/agent-skills/plugin-installation) page for all options. ### Vercel Agent Skills CLI ```bash theme={"system"} npx skills add Nimbleway/agent-skills ``` The agents skill requires the MCP server. After installing via `npx skills`, connect the MCP server manually: ```bash theme={"system"} claude mcp add --transport http nimble-mcp-server https://mcp.nimbleway.com/mcp \ --header "Authorization: Bearer ${NIMBLE_API_KEY}" ``` ## Usage Invoke the skill using the slash command: ``` /nimble:nimble-agents ``` For example: ``` /nimble:nimble-agents extract product details from this Amazon page: https://www.amazon.com/dp/B0DGHRT7PS ``` The skill activates automatically — it searches for a matching agent, lets you pick one, runs it, and returns structured data. You can also describe your task in plain language without the slash command. If the plugin is installed, Claude Code will recognize when to use the agents skill based on your request. Invoke the skill by referencing it in Cursor Agent chat: ``` /nimble-agents use nimble mcp to find amazon agents and run them ``` For example: ``` /nimble-agents extract product details from this Amazon page: https://www.amazon.com/dp/B0DGHRT7PS ``` The skill activates automatically — it searches for a matching agent, lets you pick one, runs it, and returns structured data. You can also describe your task in plain language without the slash command. If the skills and rules are installed, Cursor will recognize when to use the agents skill based on your request. ## MCP Tools Used | Tool | Purpose | | ------------------------ | ----------------------------------------- | | `nimble_agents_list` | Browse agents by keyword | | `nimble_agents_get` | Get agent details and input/output schema | | `nimble_agents_generate` | Create custom agents via natural language | | `nimble_agents_run` | Execute agents and get structured results | | `nimble_agents_publish` | Save generated agents for reuse | ## Source Code Full skill documentation, examples, and references: View the complete skill source including SKILL.md, examples, and API reference # Nimble Web Tools Skill Source: https://docs.nimbleway.com/integrations/agent-skills/nimble-web-tools-skill Real-time web intelligence tools — search, extract, map, and crawl the live web via the [Nimble CLI](https://www.npmjs.com/package/@nimble-way/nimble-cli). Built on the open-source [Agent Skills](https://agentskills.io) standard for cross-platform agent compatibility. ## Tools Overview | Tool | Description | | ----------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Search** | Accurate, real-time web search — AI Agents search the live web to retrieve precise information. 8 focus modes: general, coding, news, academic, shopping, social, geo, location. | | **Extract** | Scalable data collection with stealth unblocking — get clean, real-time HTML and structured data from any URL. Supports JS rendering and browser emulation. | | **Map** | Fast URL discovery and site structure mapping — easily plan extraction workflows. | | **Crawl** | Extract contents from entire websites in a single request — collect large volumes of web data automatically. | ## Quick Install ```bash theme={"system"} npx skills add Nimbleway/agent-skills ``` ## Prerequisites ### 1. Nimble CLI Install the Nimble CLI globally: ```bash theme={"system"} npm i -g @nimble-way/nimble-cli ``` Verify with: ```bash theme={"system"} nimble --version ``` ### 2. Nimble API Key Get your key at [online.nimbleway.com](https://online.nimbleway.com/). Set the `NIMBLE_API_KEY` environment variable using your platform's method: Add to `~/.claude/settings.json`: ```json theme={"system"} { "env": { "NIMBLE_API_KEY": "your-api-key-here" } } ``` ```bash theme={"system"} export NIMBLE_API_KEY="your-api-key-here" ``` Or add to your shell profile (`~/.bashrc`, `~/.zshrc`): ```bash theme={"system"} echo 'export NIMBLE_API_KEY="your-api-key-here"' >> ~/.zshrc ``` * Add skills to `.github/skills/` in your repository * Configure API key using GitHub Actions secrets in the copilot environment * Or set as environment variable in your shell ## Quick Start The nimble-web-tools skill activates automatically when you ask relevant questions: ``` "Search for recent AI developments" -> nimble search --query "recent AI developments" --deep-search=false "Extract the content from this URL" -> nimble extract --url "https://example.com" --parse --format markdown "Map all the pages on this docs site" -> nimble map --url "https://docs.example.com" --limit 100 "Crawl the API docs section" -> nimble crawl run --url "https://docs.example.com/api" --limit 50 ``` ## Command Examples ### Search Accurate, real-time web search with 8 focus modes (general, coding, news, academic, shopping, social, geo, location). ```bash theme={"system"} # Fast search (recommended — always use --deep-search=false) nimble search --query "React hooks tutorial" --topic coding --deep-search=false # Search with AI-generated answer summary nimble search --query "what is WebAssembly" --include-answer --deep-search=false # News search with time filter nimble search --query "AI developments" --topic news --time-range week --deep-search=false # Domain-filtered search nimble search --query "auth best practices" \ --include-domain github.com \ --include-domain stackoverflow.com \ --deep-search=false ``` Use `--deep-search=false` for fast responses (1-3s). The default deep mode fetches full page content and is 5-10x slower — only needed for archiving or full-text analysis. ### Extract Get clean, structured data from any URL with stealth unblocking and JS rendering. ```bash theme={"system"} # Standard extraction (always use --parse --format markdown) nimble extract --url "https://example.com/article" --parse --format markdown # Render JavaScript for SPAs and dynamic content nimble extract --url "https://example.com/app" --render --parse --format markdown # Extract with geolocation nimble extract --url "https://example.com" --country US --city "New York" --parse --format markdown ``` ### Map Discover all URLs on a site to plan extraction workflows. ```bash theme={"system"} # Map URLs on a site nimble map --url "https://docs.example.com" --limit 100 # Include subdomains nimble map --url "https://example.com" --domain-filter subdomains ``` ### Crawl Extract content from entire websites asynchronously. ```bash theme={"system"} # Start a crawl (always set --limit) nimble crawl run --url "https://docs.example.com" --limit 50 # Crawl with path filtering nimble crawl run --url "https://example.com" \ --include-path "/docs" \ --include-path "/api" \ --limit 100 # Check crawl status nimble crawl status --id "crawl-id" # List all crawls nimble crawl list ``` For LLM-friendly output, prefer `map` + `extract --parse --format markdown` on individual pages. Crawl returns raw HTML which can be very large. ## About Agent Skills This skill follows the [Agent Skills](https://agentskills.io) open-source standard, making it compatible with multiple AI agent platforms. Install using the [Skills CLI](https://www.npmjs.com/package/skills) — the standard package manager for the Agent Skills ecosystem. ## Full Command Reference For the complete list of options, flags, and advanced usage patterns, see the [SKILL.md on GitHub](https://github.com/Nimbleway/agent-skills/blob/main/skills/nimble-web-tools/SKILL.md). # Plugin Installation Source: https://docs.nimbleway.com/integrations/agent-skills/plugin-installation Install the unified Nimble plugin for Claude Code, Cursor, or Vercel Agent Skills CLI. The Nimble plugin bundles two skills and an MCP server connection into a single package that works across Claude Code, Cursor, and the Vercel Agent Skills CLI. ## What's Included | Component | Description | | -------------------------- | --------------------------------------------------------------------------- | | **nimble-web-tools** skill | Real-time web intelligence — search, extract, map, and crawl via Nimble CLI | | **nimble-agents** skill | Find, generate, and run agents | | **MCP server config** | Pre-configured connection to the Nimble MCP server | ## Prerequisites 1. **Nimble CLI** — Install globally: ```bash theme={"system"} npm i -g @nimble-way/nimble-cli ``` 2. **Nimble API Key** — [Sign up](https://online.nimbleway.com/signup) and generate a key from your [Account Settings > API Keys](/nimble-sdk/admin/account-management#api-keys) 3. Set the environment variable: ```bash theme={"system"} export NIMBLE_API_KEY="your-api-key-here" ``` Or add to `~/.claude/settings.json` (Claude Code): ```json theme={"system"} { "env": { "NIMBLE_API_KEY": "your-api-key-here" } } ``` ## Install by Platform ### Nimble Marketplace (recommended) ```bash theme={"system"} claude plugin marketplace add Nimbleway/agent-skills claude plugin install nimble@nimble-plugin-marketplace ``` This installs both skills and configures the MCP server automatically. ### Local Plugin Directory Clone the repo and load it as a local plugin: ```bash theme={"system"} git clone https://github.com/Nimbleway/agent-skills.git claude --plugin-dir /path/to/agent-skills ``` ### Step 1: Add the MCP Server Click the button below to add the Nimble MCP server to Cursor instantly: Add to Cursor After installing, replace `NIMBLE_API_KEY` in **Cursor Settings > MCP Servers > nimble-mcp-server** with your actual [Nimble API key](https://online.nimbleway.com/account-settings/api-keys). Add to `.cursor/mcp.json` (project) or `~/.cursor/mcp.json` (global): ```json theme={"system"} { "mcpServers": { "nimble-mcp-server": { "url": "https://mcp.nimbleway.com/mcp", "headers": { "Authorization": "Bearer NIMBLE_API_KEY" } } } } ``` Restart Cursor for changes to take effect. ### Step 2: Add Skills and Rules Install the skills using the Vercel Agent Skills CLI: ```bash theme={"system"} npx skills add Nimbleway/agent-skills -a cursor ``` This copies both skills (`nimble-web-tools` and `nimble-agents`) and Cursor rules into your project. Clone the repo and open the `agent-skills` folder in Cursor: ```bash theme={"system"} git clone https://github.com/Nimbleway/agent-skills.git ``` Cursor auto-discovers: * `skills/` — both skills * `rules/` — Cursor rules (auto-loaded) * `mcp.json` — MCP server connection ```bash theme={"system"} npx skills add Nimbleway/agent-skills ``` This installs both skills into your project. To verify: ```bash theme={"system"} npx skills add Nimbleway/agent-skills --list ``` The agents skill requires the MCP server. After installing via `npx skills`, connect the server manually: ```bash theme={"system"} claude mcp add --transport http nimble-mcp-server https://mcp.nimbleway.com/mcp \ --header "Authorization: Bearer ${NIMBLE_API_KEY}" ``` ## Source Code View the full plugin source, skill definitions, and development scripts # LangChain Source: https://docs.nimbleway.com/integrations/langchain LangChain tools and retrievers for real-time web search and content extraction. ### Overview The [`langchain-nimble`](https://pypi.org/project/langchain-nimble/) package provides production-grade LangChain integrations for Nimble's Search and Extract APIs, enabling developers to build RAG applications and AI agents that can search, access, and retrieve online information from anywhere on the web. The package includes **two retrievers** and **two tools** for comprehensive web data access: **Retrievers:** * `NimbleSearchRetriever` - Web search with deep mode, LLM answers, and filtering * `NimbleExtractRetriever` - Direct URL content extraction **Tools (for AI Agents):** * `NimbleSearchTool` - Agent-integrated web search * `NimbleExtractTool` - Agent-integrated content extraction #### Key Features * **Fast + Deep mode** - Quick metadata retrieval or comprehensive content extraction * **Smart filtering** - Domain and date filtering, topic-based routing * **Multiple parsing formats** - Plain text, markdown (default), or simplified HTML * **Full async support** - Both sync and async operations * **Production-ready** - Retry logic, connection pooling, comprehensive error handling If you'd like to learn more about the underlying Nimble Search APIs, visit the [documentation here](https://docs.nimbleway.com/nimble-sdk/search-apiw). ### Quick Start #### Installation ```bash theme={"system"} pip install -U langchain langchain-nimble langchain-openai ``` #### Setup Get your API credentials from [Nimble's dashboard](https://online.nimbleway.com/account-settings/api-keys) (free trial available) and set as an environment variable: ```bash theme={"system"} export NIMBLE_API_KEY="your-api-key" ``` #### Build an AI Agent with Web Search ```python theme={"system"} from langchain.agents import create_agent from langchain_nimble import NimbleSearchTool, NimbleExtractTool from langchain_openai import ChatOpenAI # Initialize tools search_tool = NimbleSearchTool() extract_tool = NimbleExtractTool() # Create agent with multiple tools agent = create_agent( model=ChatOpenAI(model="gpt-5"), tools=[search_tool, extract_tool], system_prompt=( "You are a helpful research assistant with access to " "real-time web information. You can search the web and " "extract content from specific URLs." ) ) # Use the agent response = agent.invoke({ "messages": [( "user", "What are the latest developments in AI agents? " "Summarize key findings." )] }) print(response["messages"][-1].content) ``` ### RAG with Knowledge Base Extract specific URLs as your knowledge base and use them for RAG: ```python theme={"system"} from langchain_nimble import NimbleExtractRetriever from langchain_openai import ChatOpenAI from langchain_core.output_parsers import StrOutputParser from langchain_core.prompts import ChatPromptTemplate from langchain_core.runnables import RunnablePassthrough # Extract your knowledge base from specific URLs retriever = NimbleExtractRetriever( urls=[ "https://python.langchain.com/docs/concepts/retrievers/", "https://python.langchain.com/docs/concepts/tools/", "https://python.langchain.com/docs/tutorials/agents/" ], output_format="markdown" ) # Build RAG chain llm = ChatOpenAI(model="gpt-5", temperature=0) prompt = ChatPromptTemplate.from_template( """Answer based only on the provided documentation. Documentation: {context} Question: {question}""" ) chain = ( {"context": retriever | (lambda docs: "\n\n".join(doc.page_content for doc in docs)), "question": RunnablePassthrough()} | prompt | llm | StrOutputParser() ) # Ask questions about your knowledge base answer = chain.invoke("What are the key differences between retrievers and tools in LangChain?") print(answer) ``` ### Additional Resources * [GitHub Repository](https://github.com/Nimbleway/langchain-nimble) * [PyPI Package](https://pypi.org/project/langchain-nimble/) * [Example Cookbook](https://github.com/Nimbleway/cookbook) # Available Tools Source: https://docs.nimbleway.com/integrations/mcp-server/available-tools ## **Web Search & Content Extraction** ### **nimble\_deep\_web\_search** Perform comprehensive web searches across multiple search engines with Nimble's advanced infrastructure. This tool serves as your general-purpose web retrieval solution, providing real-time access to search results and extracting content from any website found through Google, Bing, and Yandex searches. **Use Cases**: * Generic web content retrieval and data collection * Real-time information gathering from any website * General-purpose web scraping and research ### **nimble\_extract** Extract and parse content from a specific URL when you already know the target webpage. This tool automatically identifies the main content of a webpage and returns it in your preferred format. **Use Cases**: * Direct URL content extraction (when you have the link) * Targeted webpage scraping and monitoring * Specific article or product page parsing ## **Google Maps & Local Business Intelligence** ### **nimble\_google\_maps\_search** **Description**: Search Google Maps to discover places and businesses with comprehensive location data. Returns structured information about multiple venues matching your search criteria, including ratings, addresses, and unique place identifiers. **Use Cases**: * Local business discovery and analysis * Competitor location mapping * Market saturation studies ### **nimble\_google\_maps\_place** Retrieve comprehensive, real-time information about a specific place from Google Maps. This tool provides the most detailed venue data available, including amenities, accessibility features, business attributes, and rich metadata. **Use Cases**: * Detailed venue profiling * Business intelligence gathering * Location-based app development ### **nimble\_google\_maps\_reviews** Collect and analyze customer reviews for any Google Maps location. Provides detailed review content, ratings, timestamps, and reviewer information, enabling comprehensive reputation analysis and customer sentiment tracking. **Use Cases**: * Reputation monitoring and management * Customer sentiment analysis * Competitive benchmarking ## **E-commerce & Marketplace Intelligence** ### **nimble\_targeted\_engines** **Description**: Discover available pre-trained extraction templates for major e-commerce platforms. This tool lists all supported websites and their extraction capabilities, helping you identify which platforms and data types can be accessed. **Use Cases**: * Platform capability discovery * Integration planning and scope assessment * Multi-platform strategy development ### **nimble\_targeted\_retrieval** **Description**: Execute intelligent data extraction using Nimble's pre-trained AI templates for e-commerce sites. Extracts structured product information from search results, product pages, and category listings across major retail platforms. **Use Cases**: * Price monitoring and comparison * Product catalog management * Competitive intelligence and market research ## **Agents** Browse, generate, and run pre-built agents for structured data from any website. ### **nimble\_agents\_list** Browse the catalog of pre-built agents with keyword search and pagination. Use short keyword queries (e.g. "amazon", "linkedin") to find agents matching your extraction goal. **Use Cases**: * Discovering existing agents for a target website * Browsing available extraction capabilities * Finding the right agent before building a custom one ### **nimble\_agents\_get** Get full details of a specific agent including its input/output schema. Returns the agent's description, required input parameters, and the structure of the extracted data. **Use Cases**: * Inspecting an agent's capabilities before running it * Understanding required input parameters * Reviewing output schema to plan data processing ### **nimble\_agents\_generate** Create a custom agent from a natural-language description. This is a conversational tool — reuse the same `session_id` across calls to handle follow-up questions, poll for completion, and retrieve the generated agent. **Use Cases**: * Building custom extractors for websites without existing agents * Creating tailored data extraction schemas * Automating agent creation from business requirements ### **nimble\_agents\_run** Execute an agent against a target URL or parameters and get structured results. Pass the agent name and input parameters matching the agent's input schema. **Use Cases**: * Extracting structured data from web pages * Running bulk extraction across multiple URLs * Automating recurring data collection tasks ### **nimble\_agents\_publish** Save a generated agent so it becomes reusable and searchable in future queries. Uses the same `session_id` from the generation flow. **Use Cases**: * Persisting custom agents for team-wide reuse * Building a library of organization-specific extractors * Sharing generated agents across projects ## **Data Pipelines** Create, run, and query automated data pipelines for scheduled extraction workflows. ### **nimble\_pipeline\_list** Get Nimble Effortless Pipeline definitions. Lists all pipelines available in your account with their configuration and status. **Use Cases**: * Viewing existing pipeline configurations * Monitoring pipeline status and schedules * Auditing active data collection workflows ### **nimble\_pipeline\_templates** Get Nimble Effortless Pipeline templates. Browse pre-built pipeline templates that can be customized and deployed for common extraction workflows. **Use Cases**: * Discovering available pipeline patterns * Finding starter templates for common use cases * Evaluating pipeline capabilities before building custom ones ### **nimble\_pipeline\_create** Create a new pipeline from a template with optional scheduling. Configure extraction targets, output format, and run frequency. **Use Cases**: * Setting up automated data collection * Creating scheduled extraction workflows * Deploying pipelines from templates with custom parameters ### **nimble\_pipeline\_run** Execute/run an existing pipeline definition. Triggers an immediate run of a configured pipeline. **Use Cases**: * On-demand pipeline execution * Testing pipeline configurations * Triggering data refreshes outside the regular schedule ### **nimble\_pipeline\_metadata** Get table metadata and schema information for a specific pipeline. Returns the structure of the pipeline's output data. **Use Cases**: * Understanding pipeline output format * Planning downstream data processing * Validating schema before querying ### **nimble\_pipeline\_query** Execute SQL queries against pipeline data. Query extracted data using standard SQL syntax for filtering, aggregation, and analysis. **Use Cases**: * Analyzing extracted data with SQL * Building reports and dashboards * Filtering and aggregating pipeline results ### **nimble\_pipeline\_input\_generate** Generate structured input data for pipelines using PLP (Product Listing Pages) generation. Automatically creates input parameters for pipeline runs. **Use Cases**: * Automating pipeline input preparation * Generating product listing page URLs at scale * Bootstrapping pipeline runs with generated inputs ## Nimble Cookbook Explore our open source Nimble Cookbook on GitHub for ready-to-run examples, best practices, and composable recipes that demonstrate real-world data workflows. The cookbook includes practical implementations for common use cases across retail, finance, marketing, and AI applications. # Nimble MCP Server Source: https://docs.nimbleway.com/integrations/mcp-server/mcp-server An MCP server enabling AI agents to extract structured web data using Nimble Technologies. Most AI agents are blind. They hallucinate, they guess, they fail. All because they can't see the real world. [Model Control Protocol (MCP)](https://modelcontextprotocol.io/) is an open standard developed by [Anthropic](https://www.anthropic.com/). It gives AI agents a simple, standardized way to plug into tools, data, and services. The Nimble MCP Server bridges this gap by providing real-time, structured web data that transforms how agents interact with the digital world. With tools for web search, location data, business intelligence, agents, and data pipelines, it turns chaotic web information into signals your agents can actually use. MCP routes results through the LLM context window and can consume tokens quickly. For typical web retrieval, the [Nimble CLI skill](/integrations/agent-skills/nimble-web-tools-skill) is more token-efficient. ## Get Started Skills, rules, and MCP server View all 19+ tools and categories ## Prerequisites **Nimble API Key** — You'll need an API key to authenticate with the MCP server. [Sign up](https://online.nimbleway.com/signup) and generate a key from your [Account Settings > API Keys](/nimble-sdk/admin/account-management#api-keys). ## Setup by Platform ### One-Click Install Click the button below to add the Nimble MCP server to Cursor instantly: Add to Cursor After installing, replace `NIMBLE_API_KEY` in Cursor Settings > MCP Servers > `nimble-mcp-server` with your actual [Nimble API key](https://online.nimbleway.com/account-settings/api-keys). Add to `.cursor/mcp.json` (project) or `~/.cursor/mcp.json` (global): ```json theme={"system"} { "mcpServers": { "nimble-mcp-server": { "url": "https://mcp.nimbleway.com/mcp", "headers": { "Authorization": "Bearer NIMBLE_API_KEY" } } } } ``` Restart Cursor for changes to take effect. ### Plugin Install (recommended) Installs both guided skills and the MCP server: ```bash theme={"system"} claude plugin marketplace add Nimbleway/agent-skills claude plugin install nimble@nimble-plugin-marketplace ``` See [Plugin Installation](/integrations/agent-skills/plugin-installation) for all options. ### MCP-Only If you only need the MCP tools: ```bash theme={"system"} export NIMBLE_API_KEY="Bearer your-nimble-api-key" claude mcp add --transport http nimble-mcp-server https://mcp.nimbleway.com/mcp \ --header "Authorization: ${NIMBLE_API_KEY}" ``` Add this to your `claude_desktop_config.json`: ```json theme={"system"} { "mcpServers": { "nimble-mcp-server": { "command": "npx", "args": [ "-y", "mcp-remote@latest", "https://mcp.nimbleway.com/mcp", "--header", "Authorization:${NIMBLE_API_KEY}" ], "env": { "NIMBLE_API_KEY": "Bearer XXX" } } } } ``` * Replace `Bearer XXX` with your actual Nimble API key. * **After modifying the config file, you must restart Claude Desktop for changes to take effect.** * **macOS**: Quit Claude Desktop completely (Cmd+Q) and relaunch. * **Windows**: Close the window, then open Task Manager (Ctrl+Shift+Esc), end any "Claude" processes, and relaunch. For detailed setup instructions and troubleshooting, see the [official MCP quickstart guide](https://modelcontextprotocol.io/quickstart/user). Any MCP-compatible client can connect to the Nimble MCP server using Streamable HTTP transport. **Server URL:** `https://mcp.nimbleway.com/mcp` **Prerequisites:** [Node.js and npm](https://nodejs.org/) installed (for the `mcp-remote` bridge) ```json theme={"system"} { "mcpServers": { "nimble-mcp-server": { "command": "npx", "args": [ "-y", "mcp-remote@latest", "https://mcp.nimbleway.com/mcp", "--header", "Authorization:${NIMBLE_API_KEY}" ], "env": { "NIMBLE_API_KEY": "Bearer your-nimble-api-key" } } } } ``` Replace `your-nimble-api-key` with your actual Nimble API key. ### Available Tools The Nimble MCP Server provides 19+ tools across five categories: * **Web Search & Content Extraction** — `nimble_extract`, `nimble_deep_web_search` * **Google Maps & Local Business** — `nimble_google_maps_search`, `nimble_google_maps_place`, `nimble_google_maps_reviews` * **E-commerce & Marketplace** — `nimble_targeted_engines`, `nimble_targeted_retrieval` * **Agents** — `nimble_agents_list`, `nimble_agents_get`, `nimble_agents_generate`, `nimble_agents_run`, `nimble_agents_publish` * **Data Pipelines** — `nimble_pipeline_list`, `nimble_pipeline_templates`, `nimble_pipeline_create`, `nimble_pipeline_run`, `nimble_pipeline_metadata`, `nimble_pipeline_query`, `nimble_pipeline_input_generate` View detailed documentation for all tools ### Plugin Installation For a richer experience with guided skills and workflows, install the Nimble plugin for Claude Code, Cursor, or Vercel Agent Skills CLI. Install the unified Nimble plugin for Claude Code, Cursor, or Agent Skills CLI Example using Nimble Maps API in Claude Desktop We can't wait to see what cool things your AI agents will do with our tools! Have ideas for making these tools better or need help getting started? \ Drop us a line - we're here to help your agents get the data they need! # AdsPower Source: https://docs.nimbleway.com/integrations/proxy/integration-guides/ads-power Connect Nimble Proxy with AdsPower AdsPower provides stealthy browsers that allow you to use multiple social or e-commerce accounts on a single computer simultaneously by providing a unique browser fingerprint for each account. It also includes a browser automation tool that enables you to run multiple accounts simultaneously as if they were being accessed from different physical devices. ## **Follow these steps to integrate Nimble** * Download and install the [AdsPower software here.](https://share.adspower.net/rf7gskzRbmRSqfn) * Launch AdsPower and create a new browser profile. * Click "New profile" to generate a new virtual browsing profile. * Enter your profile name, then scroll down to the proxy configuration. * Configure the proxy server with your browser profile. **HTTP is available with the Nimble IP.** * In the Nimble User Dashboard, navigate to the [Pipelines](https://app.nimbleway.com/pipelines) page and click “add pipeline” to get your pipeline's proxy connection details. * In your new pipeline, you will find the IP address, port, username, and password. * Copy the proxy connection details from the Nimble user dashboard to the AdsPower platform. Configure the proxy using the following information: * Proxy type: HTTP * IP address/Host: [ip.nimbleway.com](http://ip.nimbleway.com) * Port: 7000 * Username: your pipeline username * Password: your pipeline password That's all! AdsPower will now use proxies from Nimble IP # Android Source: https://docs.nimbleway.com/integrations/proxy/integration-guides/android Connect Nimble Proxy with Android Android is by far the most popular mobile OS on the planet, with upwards of 70% market share. Connecting an Android device to Nimble IP is easy and straightforward, with no 3rd-party apps needed! ## **Follow these steps to integrate Nimble** * Fill in the Nimble IP proxy info in the setting proxy server window (don't forget to check "enable proxy server"). * In the Nimble User Dashboard, navigate to the [Pipelines](https://app.nimbleway.com/pipelines) page and click “add pipeline” to get your pipeline's proxy connection details. * In your new pipeline, you will find the IP address, port, username, and password. * Copy the proxy connection details from the Nimble user dashboard to the VMLogin platform. 1. To get started, open your device's settings, and then Internet/Network settings. 2. Next, click on the network you’re currently connected to in order to view its properties. 3. Click on the pencil at the top right to edit the network’s settings, and then on “Advanced options”. 4. Scroll down until you see “Proxy”. Set your proxy settings to manual. A few new fields will be added, including “Proxy hostname” and “Proxy port” : * Hostname: [ip.nimbleway.com](http://ip.nimbleway.com) * Port: 7000 5. At this point, your phone will begin sending your requests through Nimble IP, but you still need to authenticate your device. To do so, open Chrome and browse to any website. You’ll immediately be asked to sign in with a username and password : * Username: your pipeline username * Password: your pipeline password 6. Click Sign in, and you’re done! 1. Open your device's settings, and then click on Mobile Networks. 2. Next, click on “Access Point Names”, or on some devices APNs. 3. Click on the currently active APN. 4. Enter in the following details: * IP address/Host: [ip.nimbleway.com](http://ip.nimbleway.com) * Port: 7000 * Username: your pipeline username * Password: your pipeline password Because Android is used by many manufacturers, the process may be slightly different on your device. Furthermore, different manufacturers will allow different apps to use proxies, so be sure to check that the app you’d like to use supports proxies. Congratulations! You’ve connected your Android device, and requests will now be routed through Nimble IP. # FoxyProxy Source: https://docs.nimbleway.com/integrations/proxy/integration-guides/foxy-proxy Connect Nimble Proxy with FoxyProxy FoxyProxy is a simple and free browser extension that lets you easily use Nimble IP on Firefox or Chrome. While setting up your proxy server connection can be done without an extension, FoxyProxy provides a simple and flexible interface as well as additional options that browsers lack. ## **Follow these steps to integrate Nimble** Installing FoxyProxy is very straightforward. \ Simply follow the steps on your browser of choice: * [Chrome download page](https://chrome.google.com/webstore/detail/foxyproxy-standard/gcknhkkoolaabfmlnjonogaaifnjlfnp) * [Firefox download page](https://addons.mozilla.org/en-US/firefox/addon/foxyproxy-standard/) * In the Nimble User Dashboard, navigate to the [Pipelines](https://app.nimbleway.com/pipelines) page and click “add pipeline” to get your pipeline's proxy connection details. * In your new pipeline, you will find the IP address, port, username, and password. * Copy the proxy connection details from the Nimble user dashboard to the FoxyProxy platform. 1. After downloading, click the extensions button on the upper right of Chrome and pin the extension to the toolbar 2. Next, click the FoxyProxy icon to open the quick activation menu, and select “Options” to modify the proxy settings. 3. On the left navigation menu, go to “Proxies”. 4. On the options menu to the right, select “Add New Proxy”. 5. In the Proxy Settings dialog that opens, enter the following credentials: * Host: [ip.nimbleway.com](http://ip.nimbleway.com) * Port: 7000 * Username: your pipeline username * Password: your pipeline password * Check the option to “Save Login Credentials” (click ‘OK’ on the message that pops up, explaining how FoxyProxy handles usernames and passwords) 6. Go to the “General” tab and set a name for your Proxy (we used Nimble) 7. There are two ways to enable the proxy: 1. From the FoxyProxy options page, change the proxy mode from “Disable FoxyProxy” to “Use proxy *yourProfile* for all URLs 2. Alternatively, you can use FoxyProxy’s quick menu to activate the proxy from any page. Click on the FoxyProxy icon in the toolbar, and then select your proxy profile 1. After downloading the add-on, Firefox will automatically add a shortcut to the toolbar. Simply click the FoxyProxy icon to open the quick activation menu, and then click “Options” to modify the proxy settings 2. On the left navigation menu, click “Add” to create a new proxy profile 3. In the “Add Proxy” page, set a name for this proxy setup (we used Nimble), then enter the following details: * Proxy IP address or DNS name: [ip.nimbleway.com](http://ip.nimbleway.com) * Port: 7000 * Username: your pipeline username * Password: your pipeline password 4. There are two ways to enable the proxy: 1. From the FoxyProxy options page, change “Turn Off (Use Firefox Settings)” to the title you set for your proxy profile (we used “Nimble”) 2. Alternatively, you can use FoxyProxy’s quick menu to activate the proxy from any page. Click on the FoxyProxy icon in the toolbar, and then select your proxy profile That's all! VMLogin will now use proxies from Nimble IP # Incogniton Source: https://docs.nimbleway.com/integrations/proxy/integration-guides/incogniton Connect Nimble Proxy with Incogniton Incogniton is a multi-session browser that helps you manage multiple online accounts from a single interface while maintaining your privacy and security. Using Incogniton, you can create multiple virtual browser profiles that act as separate users, each with its own unique digital fingerprint. This allows you to access multiple accounts from a single device without compromising your data or leaving a trace. ## **Follow these steps to integrate Nimble** * Sign up for a free account at [incogniton.com](http://incogniton.com). * Download and install the Incogniton software [here](https://incogniton.com/?utm_source=proxy\&utm_campaign=nimbleway). * Navigate to the "Profile Management" option on the left side of the screen. * Click the "New profile" button in the top right corner. * From the left navigation, select "Proxy" and choose the "Proxy type" from the options provided (HTTP, SOCK4, or SOCK5). **HTTP is available with the Nimble IP**. * In the Nimble User Dashboard, navigate to the [Pipelines](https://app.nimbleway.com/pipelines) page and click “add pipeline” to get your pipeline's proxy connection details. * In your new pipeline, you will find the IP address, port, username, and password. * Copy the proxy connection details from the Nimble user dashboard to the Incogniton platform. Configure the proxy using the following information: * Proxy type: HTTP * IP address/Host: [ip.nimbleway.com](http://ip.nimbleway.com) * Port: 7000 * Username: your pipeline username * Password: your pipeline password That's all! Your Incogniton browser will now use proxies from Nimble IP. # iOS Source: https://docs.nimbleway.com/integrations/proxy/integration-guides/ios Connect Nimble Proxy with iPhone and iPad Setting up iOS to use Nimble IP is simple, straightforward, and requires no third-party apps. iOS natively supports proxy configuration for Wi-Fi connections. By configuring Nimble on your iOS device, all network traffic on that Wi-Fi connection will route through Nimble's residential proxy network. ## **Follow these steps to integrate Nimble** * Open the **Settings** app on your iPhone or iPad * Tap **Wi-Fi** * Tap the **info icon** (ⓘ) next to your connected Wi-Fi network * Scroll down to the bottom of the network details * Tap **Configure Proxy** * Select **Manual** Enter the following configuration: * **Server:** ip.nimbleway.com * **Port:** 7000 * **Authentication:** Toggle ON * **Username:** your pipeline username * **Password:** your pipeline password In the Nimble User Dashboard, navigate to the [Pipelines](https://app.nimbleway.com/pipelines) page to get your credentials. * Tap **Save** in the top-right corner * Your device will now route traffic through Nimble's proxy network The proxy configuration applies only to this specific Wi-Fi network. You'll need to repeat the setup for each Wi-Fi network where you want to use Nimble. ## Configuration options | Parameter | Value | Description | | -------------- | ---------------------- | -------------------------------- | | Server | `ip.nimbleway.com` | Nimble proxy endpoint | | Port | `7000` | Default proxy port | | Authentication | ON | Enable credential authentication | | Username | Your pipeline username | From your Nimble dashboard | | Password | Your pipeline password | From your Nimble dashboard | ## Verify your connection To verify the proxy is working: 1. Open **Safari** or another browser 2. Visit [https://api.ipify.org](https://api.ipify.org) to check your IP address 3. The displayed IP should be different from your regular IP, confirming traffic is routed through Nimble ## Disabling the proxy To disable the Nimble proxy: 1. Go to **Settings** → **Wi-Fi** 2. Tap the **info icon** (ⓘ) next to your Wi-Fi network 3. Tap **Configure Proxy** 4. Select **Off** 5. Tap **Save** ## Important notes * **Per-network configuration:** Proxy settings are tied to specific Wi-Fi networks. If you connect to a different network, you'll need to configure the proxy again. * **Cellular data:** This configuration only affects Wi-Fi traffic. Cellular data connections are not routed through the proxy. * **All traffic:** Once enabled, all device traffic on that Wi-Fi network routes through Nimble, including apps and system services. That's all! Your iOS device will now route Wi-Fi traffic through Nimble's proxy network. # Kameleo Source: https://docs.nimbleway.com/integrations/proxy/integration-guides/kameleo Connect Nimble Proxy with Kameleo Kameleo is a technology that allows you to browse the internet anonymously through stealth browsing. It has a range of tools, including a proprietary API and Selenium Stealth WebDriver, that can automate tasks and create multiple accounts while keeping your identity hidden. The tool is supported by residential and mobile proxy networks, and this guide will show you how to use Nimble's proxy networks with Kameleo. ## **Follow these steps to integrate Nimble** * Sign up for an account at [Kameleo.io](http://Kameleo.io) * Download and install the Kameleo software [here](https://kameleo.io/?ref=13784). * Navigate to the "New Profile" option on the left side of the screen. * Choose the browser and OS footprint you wish to use. * In the Nimble User Dashboard, navigate to the [Pipelines](https://app.nimbleway.com/pipelines) page and click “add pipeline” to get your pipeline's proxy connection details. * In your new pipeline, you will find the IP address, port, username, and password. * Copy the proxy connection details from the Nimble user dashboard to the Kameleo platform. Configure the proxy using the following information: * Proxy type: HTTP * IP address/Host: [ip.nimbleway.com](http://ip.nimbleway.com) * Port: 7000 * Username: your pipeline username * Password: your pipeline password That's all! Kameleo will now use proxies from Nimble IP # macOS Source: https://docs.nimbleway.com/integrations/proxy/integration-guides/macos Connect Nimble Proxy with macOS System Preferences macOS natively supports proxy servers like Nimble IP, allowing installed applications to use the proxy connection for their communications over the internet. By configuring Nimble at the system level, all applications that respect system proxy settings will automatically route their traffic through Nimble's residential proxy network. ## **Follow these steps to integrate Nimble** * Click the **Apple menu** () in the top-left corner * Select **System Settings** (or **System Preferences** on older macOS versions) * Click on **Network** in the sidebar * Select your active network connection (e.g., **Wi-Fi** or **Ethernet**) from the list * Click **Details...** (or **Advanced...** on older versions) * Select the **Proxies** tab * Enable **Secure Web Proxy (HTTPS)** * Enter the proxy configuration: * **Server:** ip.nimbleway.com * **Port:** 7000 * Check **Proxy server requires password** * Enter your credentials: * **Username:** your pipeline username * **Password:** your pipeline password * In the Nimble User Dashboard, navigate to the [Pipelines](https://app.nimbleway.com/pipelines) page to get your credentials. * Click **OK** to close the proxy settings * Click **Apply** to save your network configuration Your secure web traffic will now route through Nimble's proxy network. ## Configuration options | Parameter | Value | Description | | --------- | ---------------------- | -------------------------- | | Server | `ip.nimbleway.com` | Nimble proxy endpoint | | Port | `7000` | Default proxy port | | Username | Your pipeline username | From your Nimble dashboard | | Password | Your pipeline password | From your Nimble dashboard | ## Verify your connection To verify the proxy is working: 1. Open **Safari** or another browser 2. Visit [https://api.ipify.org](https://api.ipify.org) to check your IP address 3. The displayed IP should be different from your regular IP, confirming traffic is routed through Nimble ## Disabling the proxy To disable the Nimble proxy: 1. Return to **System Settings** → **Network** → **Details** → **Proxies** 2. Uncheck **Secure Web Proxy (HTTPS)** 3. Click **OK** and **Apply** Some applications may have their own proxy settings that override system preferences. Check individual application settings if traffic isn't routing through the proxy as expected. That's all! macOS will now route secure web traffic through Nimble's proxy network. # MuLogin Source: https://docs.nimbleway.com/integrations/proxy/integration-guides/mulogin Connect Nimble Proxy with MuLogin MuLogin is a multi-session browser that helps you manage multiple online accounts from a single interface while maintaining your privacy and security. Using MuLogin, you can create multiple virtual browser profiles that act as separate users, each with its own unique digital fingerprint. This allows you to access multiple accounts from a single device without compromising your data or leaving a trace. ## **Follow these steps to integrate Nimble** * Sign up for an account at [mulogin.com](https://www.mulogin.com/) * Download and install the MuLogin software [here](https://www.mulogin.com/). * Launch MuLogin and select "Add Browser" from the right-side menu. * Locate the "Proxy Settings" section and choose "HTTP" as the protocol type. * In the Nimble User Dashboard, navigate to the [Pipelines](https://app.nimbleway.com/pipelines) page and click "add pipeline" to get your pipeline's proxy connection details. * In your new pipeline, you will find the IP address, port, username, and password. * Copy the proxy connection details from the Nimble user dashboard to the MuLogin platform. Configure the proxy using the following information: * Proxy type: HTTP * IP address/Host: [ip.nimbleway.com](http://ip.nimbleway.com) * Port: 7000 * Username: your pipeline username * Password: your pipeline password Click "Check the network" to validate your proxy connection. That's all! MuLogin will now use proxies from Nimble IP. # Multilogin Source: https://docs.nimbleway.com/integrations/proxy/integration-guides/multilogin Connect Nimble Proxy with Multilogin Multilogin provides you with browser and device fingerprints, and together with Nimble IP, you can access sites online from multiple destinations with full anonymity. Using Multilogin's virtual browser profiles with Nimble's residential proxies, you can manage multiple online accounts while maintaining unique digital identities for each profile. ## **Follow these steps to integrate Nimble** * Sign up for an account at [multilogin.com](https://multilogin.com/) * Download and install the Multilogin software * Launch Multilogin and log in to your account * Click **Create New** to create a new browser profile * Enter a name for your profile * Select your preferred operating system and browser type * Navigate to the **Proxy** settings section * Click **Edit proxy settings** * Select **HTTP** as the connection type * Enter the following proxy details: * **Address:** ip.nimbleway.com * **Port:** 7000 * **Username:** your pipeline username * **Password:** your pipeline password * In the Nimble User Dashboard, navigate to the [Pipelines](https://app.nimbleway.com/pipelines) page to get your credentials. Enable **"Timezone, WebRTC and Geolocation fingerprints based on the external IP"** for consistent fingerprinting that matches your proxy location. * Click **Check proxy** to verify the connection is working * You should see a success message with your proxy IP * Click **Create profile** to save your configuration Your Multilogin profile will now route all traffic through Nimble's proxy network. ## Configuration options | Parameter | Value | Description | | --------------- | ---------------------- | -------------------------- | | Connection type | HTTP | Proxy protocol | | Address | `ip.nimbleway.com` | Nimble proxy endpoint | | Port | `7000` | Default proxy port | | Username | Your pipeline username | From your Nimble dashboard | | Password | Your pipeline password | From your Nimble dashboard | ## Geo-targeting You can control geo-targeting in two ways: ### Option 1: Pipeline settings (recommended) Configure your target country directly in your Nimble pipeline settings. This applies to all connections using that pipeline. ### Option 2: Username string Append the country code to your username for profile-level control: ``` your-username-country-us # Route through US your-username-country-gb # Route through UK your-username-country-de # Route through Germany ``` This is useful when you need different profiles to connect through different countries. That's all! Multilogin will now use proxies from Nimble IP. # Proxifier Source: https://docs.nimbleway.com/integrations/proxy/integration-guides/proxifier Connect Nimble Proxy with Proxifier Proxifier is a network gateway that allows applications that do not support working through proxy servers to operate through a SOCKS or HTTPS proxy. Using Proxifier with Nimble, you can selectively route specific applications through residential IPs without redirecting all your traffic through the proxy network. ## **Follow these steps to integrate Nimble** * Download and install Proxifier from [proxifier.com](https://www.proxifier.com/) * Launch Proxifier after installation This guide uses Windows as an example, but macOS follows the same configuration steps. * Navigate to **Profile** → **Proxy Servers** * Click **Add** to create a new proxy server * Enter the following server details: * **Address:** ip.nimbleway.com * **Port:** 7000 * **Protocol:** HTTPS * Enable the **Authentication** checkbox * Select **Username and Password** * Enter your credentials: * **Username:** your pipeline username * **Password:** your pipeline password * In the Nimble User Dashboard, navigate to the [Pipelines](https://app.nimbleway.com/pipelines) page to get your credentials. * Click **Check** to test the connection * You should see a success message confirming the proxy is working * Click **OK** to save your configuration We recommend you do not set Nimble IP as your default proxy server to avoid routing all traffic through the proxy unnecessarily. ## Adding proxy rules You can configure which applications use the Nimble proxy: 1. Go to **Profile** → **Proxification Rules** 2. Click **Add** to create a new rule 3. Configure the rule: * **Name:** Give your rule a descriptive name * **Applications:** Select specific applications to route through the proxy * **Target hosts:** Optionally specify domains or ports * **Action:** Select your Nimble proxy server This allows you to route only specific applications (like browsers or scraping tools) through Nimble while keeping other traffic direct. ## DNS configuration For optimal performance, configure DNS resolution through Nimble: 1. Go to **Profile** → **Name Resolution** 2. Select **Resolve hostnames through proxy** 3. Click **OK** to save This ensures DNS queries are also routed through Nimble's infrastructure. ## Configuration options | Parameter | Value | Description | | --------- | ---------------------- | -------------------------- | | Address | `ip.nimbleway.com` | Nimble proxy endpoint | | Port | `7000` | Default proxy port | | Protocol | HTTPS | Secure proxy protocol | | Username | Your pipeline username | From your Nimble dashboard | | Password | Your pipeline password | From your Nimble dashboard | That's all! Proxifier will now route selected applications through Nimble's proxy network. # Puppeteer Source: https://docs.nimbleway.com/integrations/proxy/integration-guides/puppeteer Connect Nimble Proxy with Puppeteer Puppeteer is a Node.js library that provides a high-level API to control Chrome or Chromium browsers. It's commonly used for web scraping, automated testing, and generating screenshots or PDFs of web pages. By integrating Nimble's proxy network with Puppeteer, you can route your automated browser requests through residential IPs, enabling access to geo-restricted content and avoiding IP-based blocking. ## Prerequisites * Node.js installed on your system * Nimble account with proxy credentials ## **Follow these steps to integrate Nimble** Install Puppeteer and the proxy-chain package, which handles proxy authentication: ```bash theme={"system"} npm install puppeteer proxy-chain ``` * In the Nimble User Dashboard, navigate to the [Pipelines](https://app.nimbleway.com/pipelines) page and click "add pipeline" to get your pipeline's proxy connection details. * In your new pipeline, you will find the IP address, port, username, and password. * Your proxy URL format will be: `http://username:password@ip.nimbleway.com:7000` Use the following code to connect Puppeteer through Nimble's proxy: ```javascript theme={"system"} const puppeteer = require('puppeteer'); const proxyChain = require('proxy-chain'); (async () => { // Replace with your Nimble pipeline credentials const proxyUrl = 'http://YOUR_USERNAME:YOUR_PASSWORD@ip.nimbleway.com:7000'; // Anonymize the proxy URL (handles authentication) const anonymizedProxy = await proxyChain.anonymizeProxy(proxyUrl); // Launch browser with proxy const browser = await puppeteer.launch({ args: [ `--proxy-server=${anonymizedProxy}`, '--no-sandbox', '--disable-setuid-sandbox' ], headless: true }); try { const page = await browser.newPage(); // Navigate to your target URL await page.goto('https://example.com'); // Get page content const content = await page.content(); console.log('Page loaded successfully'); // Take a screenshot (optional) await page.screenshot({ path: 'screenshot.png' }); } finally { // Clean up await browser.close(); await proxyChain.closeAnonymizedProxy(anonymizedProxy, true); } })(); ``` The `proxy-chain` package is required because Puppeteer doesn't natively support proxy authentication. It creates a local proxy server that handles the authentication for you. ## Configuration options | Parameter | Value | Description | | --------- | ---------------------- | -------------------------- | | Host | `ip.nimbleway.com` | Nimble proxy endpoint | | Port | `7000` | Default proxy port | | Username | Your pipeline username | From your Nimble dashboard | | Password | Your pipeline password | From your Nimble dashboard | ## Geo-targeting example To route requests through a specific country, append the country code to your username: ```javascript theme={"system"} // Route through US proxies const proxyUrl = 'http://YOUR_USERNAME-country-us:YOUR_PASSWORD@ip.nimbleway.com:7000'; // Route through UK proxies const proxyUrl = 'http://YOUR_USERNAME-country-gb:YOUR_PASSWORD@ip.nimbleway.com:7000'; ``` ## Verify your connection You can verify your proxy connection by checking your IP address: ```javascript theme={"system"} const puppeteer = require('puppeteer'); const proxyChain = require('proxy-chain'); (async () => { const proxyUrl = 'http://YOUR_USERNAME:YOUR_PASSWORD@ip.nimbleway.com:7000'; const anonymizedProxy = await proxyChain.anonymizeProxy(proxyUrl); const browser = await puppeteer.launch({ args: [`--proxy-server=${anonymizedProxy}`], headless: true }); const page = await browser.newPage(); await page.goto('https://api.ipify.org?format=json'); const ipInfo = await page.evaluate(() => document.body.textContent); console.log('Your proxy IP:', ipInfo); await browser.close(); await proxyChain.closeAnonymizedProxy(anonymizedProxy, true); })(); ``` That's all! Puppeteer will now route all requests through Nimble's proxy network. # Scrapy Source: https://docs.nimbleway.com/integrations/proxy/integration-guides/scrapy Connect Nimble Proxy with Scrapy Scrapy is a popular open-source web crawling and scraping framework for Python. It's used to crawl websites and extract structured data from their pages, making it ideal for data mining, monitoring, and automated testing. By integrating Nimble with Scrapy, you can route your crawling requests through residential IPs, enabling access to geo-restricted content and avoiding IP-based blocking at scale. ## Prerequisites * Python 3.8+ installed on your system * Nimble account with API credentials ## **Follow these steps to integrate Nimble** Install the official Nimble middleware for Scrapy: ```bash theme={"system"} pip install scrapy-nimble ``` * Log in to your [Nimble Dashboard](https://app.nimbleway.com) * Navigate to Account Settings to find your API credentials * You'll need your username and password for the configuration Update your project's `settings.py` file to enable the Nimble middleware: ```python theme={"system"} # Enable Nimble integration NIMBLE_ENABLED = True # Your Nimble credentials NIMBLE_USERNAME = "your-username" NIMBLE_PASSWORD = "your-password" # Register the Nimble middleware DOWNLOADER_MIDDLEWARES = { "scrapy_nimble.middlewares.NimbleWebApiMiddleware": 570, } ``` Ensure the Nimble middleware priority (570) is set to run before the default `HttpCompressionMiddleware` (590). ## Using Nimble in your spiders Once configured, you can use Nimble features in your spider requests via the `meta` parameter: ```python theme={"system"} import scrapy class MySpider(scrapy.Spider): name = "my_spider" def start_requests(self): yield scrapy.Request( url="https://example.com", callback=self.parse, meta={ "nimble_render": True, # Enable JavaScript rendering "nimble_country": "US", # Geo-target to US "nimble_locale": "en", # Set locale } ) def parse(self, response): # Your parsing logic here title = response.css("title::text").get() yield {"title": title} ``` ## Available request options | Meta Parameter | Type | Description | | ---------------- | ------- | ------------------------------------------------------------------ | | `nimble_render` | boolean | Enable JavaScript rendering | | `nimble_country` | string | Two-letter country code for geo-targeting (e.g., "US", "DE", "GB") | | `nimble_locale` | string | Locale setting for the request | ## Example spider with geo-targeting ```python theme={"system"} import scrapy class GeoSpider(scrapy.Spider): name = "geo_spider" countries = ["US", "GB", "DE", "FR"] def start_requests(self): for country in self.countries: yield scrapy.Request( url="https://example.com", callback=self.parse, meta={ "nimble_country": country, "nimble_render": True, }, cb_kwargs={"country": country} ) def parse(self, response, country): yield { "country": country, "title": response.css("title::text").get(), "content_length": len(response.body), } ``` ## Development setup For local development, we recommend using a virtual environment: ```bash theme={"system"} # Create and activate virtual environment python -m venv venv source venv/bin/activate # On Windows: venv\Scripts\activate # Install dependencies pip install scrapy scrapy-nimble # Create a new Scrapy project scrapy startproject myproject ``` That's all! Scrapy will now route all requests through Nimble's infrastructure. # Selenium Source: https://docs.nimbleway.com/integrations/proxy/integration-guides/selenium Connect Nimble Proxy with Selenium Selenium is a powerful browser automation framework used for web testing, scraping, and automating repetitive browser tasks. It supports multiple programming languages and browsers. By integrating Nimble's proxy network with Selenium, you can route your automated browser requests through residential IPs, enabling access to geo-restricted content and avoiding IP-based blocking. ## Prerequisites * Node.js installed on your system * Nimble account with proxy credentials ## **Follow these steps to integrate Nimble** Install Selenium WebDriver and the proxy-chain package: ```bash theme={"system"} npm install selenium-webdriver proxy-chain ``` * In the Nimble User Dashboard, navigate to the [Pipelines](https://app.nimbleway.com/pipelines) page and click "add pipeline" to get your pipeline's proxy connection details. * In your new pipeline, you will find the IP address, port, username, and password. * Your proxy URL format will be: `http://username:password@ip.nimbleway.com:7000` Use the following code to connect Selenium through Nimble's proxy: ```javascript theme={"system"} const { Builder } = require('selenium-webdriver'); const chrome = require('selenium-webdriver/chrome'); const proxyChain = require('proxy-chain'); (async () => { // Replace with your Nimble pipeline credentials const proxyUrl = 'http://YOUR_USERNAME:YOUR_PASSWORD@ip.nimbleway.com:7000'; // Anonymize the proxy URL (handles authentication) const anonymizedProxy = await proxyChain.anonymizeProxy(proxyUrl); // Configure Chrome options with proxy const options = new chrome.Options(); options.addArguments(`--proxy-server=${anonymizedProxy}`); options.addArguments('--no-sandbox'); options.addArguments('--disable-setuid-sandbox'); // Build the WebDriver const driver = await new Builder() .forBrowser('chrome') .setChromeOptions(options) .build(); try { // Navigate to your target URL await driver.get('https://example.com'); // Get page title const title = await driver.getTitle(); console.log('Page title:', title); // Your automation logic here... } finally { // Clean up await driver.quit(); await proxyChain.closeAnonymizedProxy(anonymizedProxy, true); } })(); ``` The `proxy-chain` package is required because Selenium doesn't natively support proxy authentication with Chrome. It creates a local proxy server that handles the authentication for you. ## Configuration options | Parameter | Value | Description | | --------- | ---------------------- | -------------------------- | | Host | `ip.nimbleway.com` | Nimble proxy endpoint | | Port | `7000` | Default proxy port | | Username | Your pipeline username | From your Nimble dashboard | | Password | Your pipeline password | From your Nimble dashboard | ## Geo-targeting example To route requests through a specific country, append the country code to your username: ```javascript theme={"system"} // Route through US proxies const proxyUrl = 'http://YOUR_USERNAME-country-us:YOUR_PASSWORD@ip.nimbleway.com:7000'; // Route through Germany proxies const proxyUrl = 'http://YOUR_USERNAME-country-de:YOUR_PASSWORD@ip.nimbleway.com:7000'; ``` ## Verify your connection You can verify your proxy connection by checking your IP address: ```javascript theme={"system"} const { Builder } = require('selenium-webdriver'); const chrome = require('selenium-webdriver/chrome'); const proxyChain = require('proxy-chain'); (async () => { const proxyUrl = 'http://YOUR_USERNAME:YOUR_PASSWORD@ip.nimbleway.com:7000'; const anonymizedProxy = await proxyChain.anonymizeProxy(proxyUrl); const options = new chrome.Options(); options.addArguments(`--proxy-server=${anonymizedProxy}`); const driver = await new Builder() .forBrowser('chrome') .setChromeOptions(options) .build(); await driver.get('https://api.ipify.org'); const ip = await driver.findElement({ css: 'body' }).getText(); console.log('Your proxy IP:', ip); await driver.quit(); await proxyChain.closeAnonymizedProxy(anonymizedProxy, true); })(); ``` That's all! Selenium will now route all requests through Nimble's proxy network. # SwitchyOmega Source: https://docs.nimbleway.com/integrations/proxy/integration-guides/switchyomega Connect Nimble Proxy with SwitchyOmega browser extension SwitchyOmega is a free browser extension that lets you manage and switch between multiple proxies quickly and easily. It supports both Chrome and Firefox with an intuitive interface. Using SwitchyOmega with Nimble, you can easily toggle proxy connections on and off, and even set up automatic rules to route specific websites through different proxies. ## **Follow these steps to integrate Nimble** Install the extension for your browser: * **Chrome:** [Chrome Web Store](https://chrome.google.com/webstore/detail/proxy-switchyomega/padekgcemlokbadohgkifijomclgjgif) * **Firefox:** [Firefox Add-ons](https://addons.mozilla.org/en-US/firefox/addon/switchyomega/) After installation, pin the extension to your browser toolbar for easy access. * Click the SwitchyOmega icon in your browser toolbar * Select **Options** to open the settings interface * In the left sidebar, click on **proxy** under Profiles * Enter the following connection details: * **Protocol:** HTTPS * **Server:** ip.nimbleway.com * **Port:** 7000 * Enter your credentials: * **Username:** your pipeline username * **Password:** your pipeline password In the Nimble User Dashboard, navigate to the [Pipelines](https://app.nimbleway.com/pipelines) page to get your credentials. * Click **Apply changes** to save your proxy settings * Click the SwitchyOmega icon in your toolbar * Select your **proxy** profile to activate the connection Your browser traffic will now route through Nimble's proxy network. ## Configuration options | Parameter | Value | Description | | --------- | ---------------------- | -------------------------- | | Protocol | HTTPS | Secure proxy protocol | | Server | `ip.nimbleway.com` | Nimble proxy endpoint | | Port | `7000` | Default proxy port | | Username | Your pipeline username | From your Nimble dashboard | | Password | Your pipeline password | From your Nimble dashboard | ## Auto-switch rules SwitchyOmega allows you to automatically route specific websites through different proxies: Create separate proxy profiles for different use cases (e.g., different geo-locations or pipelines): * Click **New profile** in the left sidebar * Select **Proxy Profile** * Configure each profile with different Nimble pipeline credentials * Click on **auto switch** in the left sidebar * Add rules to route specific domains through specific profiles: * **Condition type:** Host wildcard * **Condition:** `*.example.com` * **Profile:** Select which proxy profile to use * Click **Apply changes** * Click the SwitchyOmega icon in your toolbar * Select **auto switch** to enable automatic profile switching Now specific websites will automatically route through their assigned proxy profiles. ## Quick switching Once configured, you can quickly switch between: * **Direct** - No proxy (direct connection) * **System Proxy** - Use system-level proxy settings * **proxy** - Your Nimble proxy profile * **auto switch** - Automatic rule-based switching Just click the SwitchyOmega icon and select the desired mode. That's all! SwitchyOmega will now manage your Nimble proxy connections. # VMLogin Source: https://docs.nimbleway.com/integrations/proxy/integration-guides/vm-login Connect Nimble Proxy with VMLogin With VMLogin's virtual browsing profiles, you can use physical devices to access and manage multiple online accounts while also benefiting from features like anti-association and fingerprint protection. These virtual profiles help you protect your online identity and enhance the security of your online accounts. ## **Follow these steps to integrate Nimble** * Sign up for an account at [VMLogin](https://www.vmlogin.us/?ref=nimble) * Download and install the VMLogin software [here.](https://www.vmlogin.us/?ref=nimble) * Launch VMLogin and create a new browser profile. * Click "Get random profile" to generate a new virtual browsing profile. * Customize your profile by selecting the settings that best suit you, including the operating system, screen resolution, language, WebGL vendor, time zone, and media device fingerprint. * Fill in the Nimble IP proxy info in the setting proxy server window (don't forget to check "enable proxy server"). * In the Nimble User Dashboard, navigate to the [Pipelines](https://app.nimbleway.com/pipelines) page and click “add pipeline” to get your pipeline's proxy connection details. * In your new pipeline, you will find the IP address, port, username, and password. * Copy the proxy connection details from the Nimble user dashboard to the VMLogin platform. Configure the proxy using the following information: * Proxy type: HTTP * IP address/Host: [ip.nimbleway.com](http://ip.nimbleway.com) * Port: 7000 * Username: your pipeline username * Password: your pipeline password That's all! VMLogin will now use proxies from Nimble IP # Windows Source: https://docs.nimbleway.com/integrations/proxy/integration-guides/windows Connect Nimble Proxy with Windows System Settings Windows 10 and later natively support proxy servers like Nimble IP. The operating system manages server and port settings, while each application handles its own authentication credentials. By configuring Nimble at the system level, applications that respect system proxy settings will route their traffic through Nimble's residential proxy network. ## **Follow these steps to integrate Nimble** * Press the **Windows key** and search for "proxy settings" * Or navigate to **Settings** → **Network & Internet** → **Proxy** * Scroll down to **Manual proxy setup** * Click **Set up** (or toggle **Use a proxy server** to ON on older Windows versions) * Enter the proxy configuration: * **Proxy IP address:** ip.nimbleway.com * **Port:** 7000 * Click **Save** Windows 10+ requires applications to manage proxy authentication individually. When you access a website through your browser, you'll receive a prompt requesting your Nimble credentials: * **Username:** your pipeline username * **Password:** your pipeline password In the Nimble User Dashboard, navigate to the [Pipelines](https://app.nimbleway.com/pipelines) page to get your credentials. Most browsers will remember your credentials after the first entry, so you won't need to re-enter them for each request. ## Configuration options | Parameter | Value | Description | | ---------------- | ---------------------- | -------------------------- | | Proxy IP address | `ip.nimbleway.com` | Nimble proxy endpoint | | Port | `7000` | Default proxy port | | Username | Your pipeline username | From your Nimble dashboard | | Password | Your pipeline password | From your Nimble dashboard | ## Alternative: IP allowlisting Instead of using username/password authentication, you can allowlist your IP address: 1. Log in to your [Nimble Dashboard](https://app.nimbleway.com) 2. Navigate to your Pipeline settings 3. Add your public IP address to the allowlist 4. Connect to the proxy without credentials This is useful for automated systems or when you prefer not to enter credentials. ## Geo-targeting To route requests through a specific country, modify your username: ``` your-username-country-us # Route through US your-username-country-gb # Route through UK your-username-country-de # Route through Germany ``` ## Disabling the proxy To disable the Nimble proxy: 1. Return to **Settings** → **Network & Internet** → **Proxy** 2. Toggle **Use a proxy server** to OFF 3. Or click **Set up** and disable the proxy That's all! Windows will now route web traffic through Nimble's proxy network. # Overview Source: https://docs.nimbleway.com/integrations/proxy/overview Step-by-step guides for integrating Nimble IP with popular third-party software Nimble IP is optimized for compatibility with third-party apps. To help you get started, we've prepared step-by-step guides for integrating Nimble IP with popular third-party software. *** ## Anti-Detect Browsers Manage multiple accounts with unique browser fingerprints while routing traffic through Nimble's residential proxies. Virtual browser profiles with fingerprint protection Multi-session browser for account management Browser fingerprints with full anonymity Stealth browsers for multiple accounts Anonymous stealth browsing technology Multi-session browser with unique fingerprints *** ## Browser Extensions and Desktop Apps Quickly switch between proxies directly from your browser or Desktop. Firefox and Chrome proxy management extension Manage and switch between multiple proxies Network gateway for any application *** ## Automation Tools Integrate Nimble proxies into your web scraping and automation workflows. Node.js browser automation library Browser automation framework Python web crawling framework *** ## Operating Systems Configure system-level proxy settings for your device. Windows 10+ system proxy configuration macOS system preferences proxy setup iPhone and iPad Wi-Fi proxy configuration Android device proxy setup *** ## Quick Reference All integrations use the same core proxy settings: | Parameter | Value | | ------------ | ---------------------- | | **Host** | `ip.nimbleway.com` | | **Port** | `7000` | | **Protocol** | HTTP / HTTPS | | **Username** | Your pipeline username | | **Password** | Your pipeline password | Get your proxy credentials from the [Pipelines](https://app.nimbleway.com/pipelines) page in your Nimble dashboard. # Account Management Source: https://docs.nimbleway.com/nimble-sdk/admin/account-management Manage API keys and collaborate with your team Manage your Nimble account settings, generate API keys, and collaborate with team members all from one centralized dashboard. ## Accessing Account Settings Navigate to your account settings at [online.nimbleway.com](https://online.nimbleway.com): 1. Log in with your username and password 2. Click on your profile or settings icon 3. Select **Account Settings** from the menu *** ## API Keys API keys are used to authenticate your requests to the Nimble API. Each key is unique and should be kept secure. ### Generate Your First API Key In Account Settings, click on the **API Keys** tab Image Click the **Create New API Key** button Image Give your API key a descriptive name Image **Important**: Your API key will only be displayed once. Make sure to copy it immediately and store it securely. Click **Copy** to save your API key to clipboard Image ### Managing API Keys Once created, you can view and manage all your API keys from the API Keys dashboard. Image #### API Key Information Each API key displays the following information: | Column | Description | | -------------- | --------------------------------------------------- | | **Name** | The descriptive name you gave to the key | | **Created** | Date and time when the key was created | | **Created By** | The team member who created the key | | **Last Used** | Most recent date the key was used in an API request | | **Actions** | Delete or manage the key | ### Delete an API Key Deleting an API key is permanent and cannot be undone. Any applications using this key will immediately lose access. To delete an API key: 1. Locate the key you want to delete in the list 2. Click the **Delete** button or trash icon 3. Confirm the deletion in the popup dialog Image ### Best Practices Name keys based on their purpose or environment Create new keys periodically and delete old ones to maintain security Use separate API keys for different services to track usage and limit exposure Keep API keys confidential and never commit them to version control Monitor the "Last Used" column to identify and remove unused API keys *** ## Team Members Collaborate with your team by inviting members to your Nimble account. Control access levels and manage permissions all in one place. ### Invite Team Members In Account Settings, click on the **Team Members** tab Image Click the **Invite Team Member** button Image Type the email address of the person you want to invite Click **Send Invite** - they'll receive an email with instructions to join Image ### Managing Team Members View and manage all team members from the Team Members dashboard. Image #### Team Member Information | Column | Description | | ----------- | -------------------------------------- | | **Name** | Team member's full name | | **Email** | Associated email address | | **Role** | Permission level (Member or Admin) | | **Status** | Active, Pending, or Invited | | **Joined** | Date when they accepted the invitation | | **Actions** | Change role or remove member | ### Change Member Role Promote members to admins or revoke admin privileges: 1. Locate the team member in the list 2. Click the **Role** dropdown 3. Select **Admin** or **Member** 4. Changes take effect immediately Admins have full access to account settings, billing, API keys, and team management ### Remove Team Members To remove a team member from your account: 1. Find the member in the team list 2. Click the **Remove** button or trash icon 3. Confirm the removal Removed members will immediately lose access to your account and all its resources ### Permission Levels **Members can:** * Use existing API keys * View usage statistics * Access documentation * Make API requests **Members cannot:** * Create or delete API keys * Invite or remove team members * Access billing information * Change account settings **Admins can:** * Everything Members can do * Create and delete API keys * Invite and remove team members * Promote/demote member roles * Access billing and payment information * Modify account settings **Full account control with no restrictions** ### Team Collaboration Tips Create unique API keys for each team member to track individual usage and maintain accountability Only promote trusted team members to admin roles to maintain account security Encourage team members to use clear names when creating API keys Periodically review team members and API keys to remove unused accounts *** ## Security Best Practices * Never commit API keys to version control * Use environment variables to store keys * Rotate keys regularly (every 90 days recommended) * Delete keys immediately if compromised * Use separate keys for development and production * Grant minimum necessary permissions * Remove access for departing team members immediately * Use admin roles sparingly * Review team member list regularly * Monitor API key usage patterns * Check "Last Used" dates on API keys * Review team member activity * Set up alerts for unusual usage * Track API consumption by key * Audit changes to account settings *** ## Need Help? Learn how to use your API keys with our quickstart guide Contact our support team for account assistance Test & Explore the API effortleslly, no code is needed # Blogs Source: https://docs.nimbleway.com/nimble-sdk/admin/blogs # Callbacks & Delivery Source: https://docs.nimbleway.com/nimble-sdk/admin/callbacks-and-delivery Configure how you receive results from async operations When using async operations like `/extract/async`, `/agent/async`, or `/crawl`, you have three flexible options for receiving your results. Choose the method that best fits your infrastructure and workflow. Pull results on-demand using task IDs Receive push notifications when tasks complete Automatic delivery to your S3 or GCS bucket *** ## Option 1: Polling (Pull) The simplest approach - submit your async request, receive a task ID, and poll for results when ready. Send a request to the async endpoint. You'll receive a task or crawl ID to track your request. ```python Python theme={"system"} from nimble_python import Nimble nimble = Nimble(api_key="YOUR-API-KEY") response = nimble.extract_async( url="https://www.nimbleway.com", render=True, formats=["html", "markdown"] ) task_id = response.task_id print(f"Task submitted: {task_id}") ``` ```javascript Node theme={"system"} import Nimble from '@nimble-way/nimble-js'; const nimble = new Nimble({ apiKey: "YOUR-API-KEY" }); const response = await nimble.extractAsync({ url: "https://www.nimbleway.com", render: true, formats: ["html", "markdown"] }); const taskId = response.task_id; console.log(`Task submitted: ${taskId}`); ``` ```bash cURL theme={"system"} curl -X POST 'https://sdk.nimbleway.com/v1/extract/async' \ --header 'Authorization: Bearer YOUR-API-KEY' \ --header 'Content-Type: application/json' \ --data-raw '{ "url": "https://www.nimbleway.com", "render": true, "formats": ["html", "markdown"] }' ``` ```python Python theme={"system"} from nimble_python import Nimble nimble = Nimble(api_key="YOUR-API-KEY") response = nimble.agent.run_async( agent="amazon_pdp", params={"asin": "B0DLKFK6LR"} ) task_id = response.task_id print(f"Task submitted: {task_id}") ``` ```javascript Node theme={"system"} import Nimble from '@nimble-way/nimble-js'; const nimble = new Nimble({ apiKey: "YOUR-API-KEY" }); const response = await nimble.agent.runAsync({ agent: "amazon_pdp", params: { asin: "B0DLKFK6LR" } }); const taskId = response.task_id; console.log(`Task submitted: ${taskId}`); ``` ```bash cURL theme={"system"} curl -X POST 'https://sdk.nimbleway.com/v1/agent/async' \ --header 'Authorization: Bearer YOUR-API-KEY' \ --header 'Content-Type: application/json' \ --data-raw '{ "agent": "amazon_pdp", "params": { "asin": "B0DLKFK6LR" } }' ``` ```python Python theme={"system"} from nimble_python import Nimble nimble = Nimble(api_key="YOUR-API-KEY") result = nimble.crawl.run( url="https://www.nimbleway.com", limit=50 ) crawl_id = result.crawl_id print(f"Crawl started: {crawl_id}") ``` ```javascript Node theme={"system"} import Nimble from '@nimble-way/nimble-js'; const nimble = new Nimble({ apiKey: "YOUR-API-KEY" }); const result = await nimble.crawl.run({ url: "https://www.nimbleway.com", limit: 50 }); const crawlId = result.crawl_id; console.log(`Crawl started: ${crawlId}`); ``` ```bash cURL theme={"system"} curl -X POST 'https://sdk.nimbleway.com/v1/crawl' \ --header 'Authorization: Bearer YOUR-API-KEY' \ --header 'Content-Type: application/json' \ --data-raw '{ "url": "https://www.nimbleway.com", "limit": 50 }' ``` Poll the status endpoint to monitor progress. ```python Python theme={"system"} import time while True: my_task = nimble.tasks.get(task_id) print(f"Status: {my_task.task.state}") if my_task.state == "success": break elif my_task.state == "failed": print(f"Task failed: {status.error}") break time.sleep(15) ``` ```javascript Node theme={"system"} while (true) { const myTask = await nimble.tasks.get(taskId); console.log(`Status: ${myTask.task.state}`); if (myTask.state === "success") break; if (myTask.state === "failed") { console.log(`Task failed: ${status.error}`); break; } await new Promise(resolve => setTimeout(resolve, 15000)); } ``` ```bash cURL theme={"system"} curl 'https://sdk.nimbleway.com/v1/tasks/{task_id}' \ --header 'Authorization: Bearer YOUR-API-KEY' # Response: { "task": { "id": "...", "state": "completed", ... } } ``` Crawl has its own status endpoint that shows overall progress and individual page tasks. ```python Python theme={"system"} import time while True: my_crawl = nimble.crawl.status(crawl_id) print(f"Status: {my_crawl.status}") if status.state == "succeeded": break elif status.state == "failed": print(f"Task failed: {status.error}") break time.sleep(2) ``` ```javascript Node theme={"system"} while (true) { const myCrawl = await nimble.crawl.status(crawlId); console.log(`Status: ${myCrawl.status}`); if (status.state === "succeeded") break; if (status.state === "failed") { console.log(`Task failed: ${status.error}`); break; } await new Promise(resolve => setTimeout(resolve, 2000)); } ``` ```bash cURL theme={"system"} curl 'https://sdk.nimbleway.com/v1/crawl/{crawl_id}' \ --header 'Authorization: Bearer YOUR-API-KEY' # Response: { "crawl": { "status": "running", "completed": 10, "total": 50, "tasks": [...] } } ``` Once complete, fetch the full results. ```python Python theme={"system"} results = nimble.tasks.results(task_id) print(f"HTML length: {len(results.data.html)}") print(f"Markdown length: {len(results.data.markdown)}") ``` ```javascript Node theme={"system"} const results = await nimble.tasks.results(taskId); console.log(`HTML length: ${results.data.html.length}`); console.log(`Markdown length: ${results.data.markdown.length}`); ``` ```bash cURL theme={"system"} curl 'https://sdk.nimbleway.com/v1/tasks/{task_id}/results' \ --header 'Authorization: Bearer YOUR-API-KEY' ``` ```python Python theme={"system"} results = nimble.tasks.results(task_id) parsed = results.data.parsing["parsed"] print(f"Product: {parsed['product_title']}") print(f"Price: ${parsed['web_price']}") ``` ```javascript Node theme={"system"} const results = await nimble.tasks.results(taskId); const parsed = results.data.parsing.parsed; console.log(`Product: ${parsed.product_title}`); console.log(`Price: $${parsed.web_price}`); ``` ```bash cURL theme={"system"} curl 'https://sdk.nimbleway.com/v1/tasks/{task_id}/results' \ --header 'Authorization: Bearer YOUR-API-KEY' ``` Fetch results for each completed task in the crawl. ```python Python theme={"system"} # my_crawl["tasks"] from step #2 contains list of task IDs from status response for task in my_crawl["tasks"]: if task.state == "success": task_result = nimble.tasks.get(task_id) print(f"URL: {task_result['url']}") print(f"HTML length: {len(task_result['data'].get('html', ''))}") ``` ```javascript Node theme={"system"} // myCrawl.tasks from step #2 contains list of task IDs from status response for (const task of myCrawl.tasks) { if (task.status === "completed") { const taskResponse = await nimble.tasks.get(taskId); console.log(`URL: ${taskResult.url}`); console.log(`HTML length: ${taskResult.data?.html?.length || 0}`); } } ``` ```bash cURL theme={"system"} # Get results for each task_id from the crawl status response curl 'https://sdk.nimbleway.com/v1/tasks/{task_id}/results' \ --header 'Authorization: Bearer YOUR-API-KEY' ``` ```json theme={"system"} { "url": "https://www.nimbleway.com/blog/post", "task_id": "ec89b1f7-1cf2-40eb-91b4-78716093f9ed", "status": "success", "task": { "id": "ec89b1f7-1cf2-40eb-91b4-78716093f9ed", "state": "success", "created_at": "2026-02-09T23:15:43.549Z", "modified_at": "2026-02-09T23:16:39.094Z", "account_name": "your-account" }, "data": { "html": "...", "markdown": "# Page Title\n\nContent...", "headers": { ... } }, "metadata": { "query_time": "2026-02-09T23:15:43.549Z", "query_duration": 1877, "response_parameters": { "input_url": "https://www.nimbleway.com/blog/post" }, "driver": "vx6" }, "status_code": 200 } ``` ### Polling endpoints reference | API | Submit | Check Status | Get Results | | ------- | ------------------------ | -------------------------- | -------------------------------------------- | | Extract | `POST /v1/extract/async` | `GET /v1/tasks/{task_id}` | `GET /v1/tasks/{task_id}/results` | | Agent | `POST /v1/agent/async` | `GET /v1/tasks/{task_id}` | `GET /v1/tasks/{task_id}/results` | | Crawl | `POST /v1/crawl` | `GET /v1/crawl/{crawl_id}` | `GET /v1/tasks/{task_id}/results` (per page) | *** ## Option 2: Webhooks (Push) Get notified automatically when your tasks complete. Perfect for event-driven architectures. Include `callback_url` (or `callback` object for crawl) in your async request. ```python Python theme={"system"} from nimble_python import Nimble nimble = Nimble(api_key="YOUR-API-KEY") response = nimble.extract_async( url="https://www.nimbleway.com", render=True, formats=["html", "markdown"], callback_url="https://your-server.com/webhooks/nimble" ) task_id = response.task_id print(f"Task submitted: {task_id}") print("Results will be POSTed to your callback URL when ready") ``` ```javascript Node theme={"system"} import Nimble from '@nimble-way/nimble-js'; const nimble = new Nimble({ apiKey: "YOUR-API-KEY" }); const response = await nimble.extractAsync({ url: "https://www.nimbleway.com", render: true, formats: ["html", "markdown"], callback_url: "https://your-server.com/webhooks/nimble" }); const taskId = response.task_id; console.log(`Task submitted: ${taskId}`); ``` ```bash cURL theme={"system"} curl -X POST 'https://sdk.nimbleway.com/v1/extract/async' \ --header 'Authorization: Bearer YOUR-API-KEY' \ --header 'Content-Type: application/json' \ --data-raw '{ "url": "https://www.nimbleway.com", "render": true, "formats": ["html", "markdown"], "callback_url": "https://your-server.com/webhooks/nimble" }' ``` ```python Python theme={"system"} from nimble_python import Nimble nimble = Nimble(api_key="YOUR-API-KEY") response = nimble.extract_async( url="https://www.nimbleway.com", render=True, formats=["html", "markdown"], callback_url="https://your-server.com/webhooks/nimble" ) task_id = response.task_id print(f"Task submitted: {task_id}") print("Results will be POSTed to your callback URL when ready") ``` ```javascript Node theme={"system"} import Nimble from '@nimble-way/nimble-js'; const nimble = new Nimble({ apiKey: "YOUR-API-KEY" }); const response = await nimble.agent.runAsync({ agent: "amazon_pdp", params: { asin: "B0DLKFK6LR", callback_url: "https://your-server.com/webhooks/nimble" }, }); const taskId = response.task_id; console.log(`Task submitted: ${taskId}`); ``` ```bash cURL theme={"system"} curl -X POST 'https://sdk.nimbleway.com/v1/extract/async' \ --header 'Authorization: Bearer YOUR-API-KEY' \ --header 'Content-Type: application/json' \ --data-raw '{ "url": "https://www.nimbleway.com", "render": true, "formats": ["html", "markdown"], "callback_url": "https://your-server.com/webhooks/nimble" }' ``` Crawl uses a `callback` object for advanced webhook options. ```python Python theme={"system"} from nimble_python import Nimble nimble = Nimble(api_key="YOUR-API-KEY") result = nimble.crawl.run( url="https://www.nimbleway.com", limit=100, callback={ "url": "https://your-server.com/webhooks/nimble", "headers": { "X-Custom-Auth": "your-secret-token" }, "events": ["completed", "failed"] } ) print(f"Crawl started: {result.crawl_id}") ``` ```javascript Node theme={"system"} import Nimble from '@nimble-way/nimble-js'; const nimble = new Nimble({ apiKey: "YOUR-API-KEY" }); const result = await nimble.crawl.run({ url: "https://www.nimbleway.com", limit: 100, callback: { url: "https://your-server.com/webhooks/nimble", headers: { "X-Custom-Auth": "your-secret-token" }, events: ["completed", "failed"] } }); console.log(`Crawl started: ${result.crawl_id}`); ``` ```bash cURL theme={"system"} curl -X POST 'https://sdk.nimbleway.com/v1/crawl' \ --header 'Authorization: Bearer YOUR-API-KEY' \ --header 'Content-Type: application/json' \ --data-raw '{ "url": "https://www.nimbleway.com", "limit": 100, "callback": { "url": "https://your-server.com/webhooks/nimble", "headers": { "X-Custom-Auth": "your-secret-token" }, "events": ["completed", "failed"] } }' ``` Nimble sends a POST to your callback URL when complete: ```json theme={"system"} { "task": { "id": "8e8cfde8-345b-42b8-b3e2-0c61eb11e00f", "state": "completed", "status_code": 200, "created_at": "2026-01-24T12:36:24.685Z", "modified_at": "2026-01-24T12:36:24.685Z", "input": {}, "api_type": "extract" } } ``` ### Webhook configuration options | API | Parameter | Type | Description | | ------- | --------------------- | ------ | ------------------------------------------------------- | | Extract | `callback_url` | string | Your callback URL | | Agent | `params.callback_url` | string | Your callback URL | | Crawl | `callback.url` | string | Your callback URL | | | `callback.headers` | object | Custom headers for authentication | | | `callback.metadata` | object | Custom data included in payload | | | `callback.events` | array | Filter events: `started`, `page`, `completed`, `failed` | *** ## Option 3: Cloud Delivery Automatically deliver results directly to your cloud storage bucket. Deliver to any S3 bucket in your AWS account Deliver to any GCS bucket in your GCP project Grant Nimble's service account write access to your bucket. **Nimble Service User ARN:** ``` arn:aws:iam::744254827463:user/webit-uploader ``` Add this bucket policy: ```json theme={"system"} { "Version": "2012-10-17", "Statement": [ { "Sid": "NimbleCloudDelivery", "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam::744254827463:user/webit-uploader" }, "Action": [ "s3:PutObject", "s3:PutObjectACL", "s3:GetBucketLocation" ], "Resource": [ "arn:aws:s3:::YOUR_BUCKET_NAME", "arn:aws:s3:::YOUR_BUCKET_NAME/*" ] } ] } ``` Replace `YOUR_BUCKET_NAME` with your actual bucket name. For KMS-encrypted buckets, add this to your KMS key policy: ```json theme={"system"} { "Sid": "NimbleKMSAccess", "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam::744254827463:user/webit-uploader" }, "Action": [ "kms:Encrypt", "kms:Decrypt", "kms:ReEncrypt*", "kms:GenerateDataKey*", "kms:DescribeKey" ], "Resource": "*" } ``` **Nimble Service Account:** ``` nimbleway-gcp-storage@nimbleway-gcp.iam.gserviceaccount.com ``` 1. Navigate to your bucket in the [Google Cloud Console](https://console.cloud.google.com/storage/browser) 2. Click on the **Permissions** tab 3. Click **Grant Access** 4. Add principal: `nimbleway-gcp-storage@nimbleway-gcp.iam.gserviceaccount.com` 5. Assign role: **Storage Object Creator** 6. Click **Save** Include `storage_type` and `storage_url` in your request. #### Cloud delivery parameters | Parameter | Type | Description | | --------------------- | ------------ | ----------------------------------------------------- | | `storage_type` | `s3` \| `gs` | Cloud provider | | `storage_url` | string | Bucket path with prefix (e.g., `s3://bucket/prefix/`) | | `storage_compress` | boolean | Enable GZIP compression | | `storage_object_name` | string | Custom filename (default: task ID) | ```python Python theme={"system"} from nimble_python import Nimble nimble = Nimble(api_key="YOUR-API-KEY") response = nimble.extract_async( url="https://www.nimbleway.com", render=True, formats=["html", "markdown"], storage_type="s3", storage_url="s3://your-bucket/nimble-results/", storage_compress=True, storage_object_name="my-result" ) task_id = response.task_id print(f"Results will be saved to: s3://your-bucket/nimble-results/my-result.json.gz") ``` ```javascript Node theme={"system"} import Nimble from '@nimble-way/nimble-js'; const nimble = new Nimble({ apiKey: "YOUR-API-KEY" }); const response = await nimble.extractAsync({ url: "https://www.nimbleway.com", render: true, formats: ["html", "markdown"], storage_type: "s3", storage_url: "s3://your-bucket/nimble-results/", storage_compress: true, storage_object_name: "my-result" }); const taskId = response.task_id; console.log(`Results will be saved to: s3://your-bucket/nimble-results/my-result.json.gz`); ``` ```bash cURL theme={"system"} curl -X POST 'https://sdk.nimbleway.com/v1/extract/async' \ --header 'Authorization: Bearer YOUR-API-KEY' \ --header 'Content-Type: application/json' \ --data-raw '{ "url": "https://www.nimbleway.com", "render": true, "formats": ["html", "markdown"], "storage_type": "s3", "storage_url": "s3://your-bucket/nimble-results/", "storage_compress": true, "storage_object_name": "my-result" }' ``` ```python Python theme={"system"} from nimble_python import Nimble nimble = Nimble(api_key="YOUR-API-KEY") response = nimble.extract_async( url="https://www.nimbleway.com", render=True, formats=["html", "markdown"], storage_type="gs", storage_url="gs://your-bucket/nimble-results/", storage_object_name="my-result" ) task_id = response.task_id print(f"Results will be saved to: gs://your-bucket/nimble-results/my-result.json") ``` ```javascript Node theme={"system"} import Nimble from '@nimble-way/nimble-js'; const nimble = new Nimble({ apiKey: "YOUR-API-KEY" }); const response = await nimble.extractAsync({ url: "https://www.nimbleway.com", render: true, formats: ["html", "markdown"], storage_type: "gs", storage_url: "gs://your-bucket/nimble-results/", storage_object_name: "my-result" }); const taskId = response.task_id; console.log(`Results will be saved to: gs://your-bucket/nimble-results/my-result.json`); ``` ```bash cURL theme={"system"} curl -X POST 'https://sdk.nimbleway.com/v1/extract/async' \ --header 'Authorization: Bearer YOUR-API-KEY' \ --header 'Content-Type: application/json' \ --data-raw '{ "url": "https://www.nimbleway.com", "render": true, "formats": ["html", "markdown"], "storage_type": "gs", "storage_url": "gs://your-bucket/nimble-results/", "storage_object_name": "my-result" }' ``` ```python Python theme={"system"} from nimble_python import Nimble nimble = Nimble(api_key="YOUR-API-KEY") response = nimble.agent.run_async( agent="amazon_pdp", params={ "asin": "B0DLKFK6LR", "storage_type": "s3", "storage_url": "s3://your-bucket/nimble-results/", "storage_compress": True, "storage_object_name": "my-result" } ) task_id = response.task_id print(f"Results will be saved to: s3://your-bucket/nimble-results/my-result.json.gz") ``` ```javascript Node theme={"system"} import Nimble from '@nimble-way/nimble-js'; const nimble = new Nimble({ apiKey: "YOUR-API-KEY" }); const response = await nimble.agent.runAsync({ agent: "amazon_pdp", params: { asin: "B0DLKFK6LR", storage_type: "s3", storage_url: "s3://your-bucket/nimble-results/", storage_compress: true, storage_object_name: "my-result" } }); const taskId = response.task_id; console.log(`Results will be saved to: s3://your-bucket/nimble-results/my-result.json.gz`); ``` ```bash cURL theme={"system"} curl -X POST 'https://sdk.nimbleway.com/v1/agent/async' \ --header 'Authorization: Bearer YOUR-API-KEY' \ --header 'Content-Type: application/json' \ --data-raw '{ "agent": "amazon_pdp", "params": { "asin": "B0DLKFK6LR", "storage_type": "s3", "storage_url": "s3://your-bucket/nimble-results/", "storage_compress": true, "storage_object_name": "my-result" } }' ``` ```python Python theme={"system"} from nimble_python import Nimble nimble = Nimble(api_key="YOUR-API-KEY") response = nimble.agent.run_async( agent="amazon_pdp", params={ "asin": "B0DLKFK6LR", "storage_type": "gs", "storage_url": "gs://your-bucket/nimble-results/", "storage_object_name": "my-result" } ) task_id = response.task_id print(f"Results will be saved to: gs://your-bucket/nimble-results/my-result.json") ``` ```javascript Node theme={"system"} import Nimble from '@nimble-way/nimble-js'; const nimble = new Nimble({ apiKey: "YOUR-API-KEY" }); const response = await nimble.agent.runAsync({ agent: "amazon_pdp", params: { asin: "B0DLKFK6LR", storage_type: "gs", storage_url: "gs://your-bucket/nimble-results/", storage_object_name: "my-result" } }); const taskId = response.task_id; console.log(`Results will be saved to: gs://your-bucket/nimble-results/my-result.json`); ``` ```bash cURL theme={"system"} curl -X POST 'https://sdk.nimbleway.com/v1/agent/async' \ --header 'Authorization: Bearer YOUR-API-KEY' \ --header 'Content-Type: application/json' \ --data-raw '{ "agent": "amazon_pdp", "params": { "asin": "B0DLKFK6LR", "storage_type": "gs", "storage_url": "gs://your-bucket/nimble-results/", "storage_object_name": "my-result" } }' ``` When complete, results are written to your bucket as `{task_id}.json` (or `.json.gz` if compressed). *** ## Comparison | Feature | Polling | Webhooks | Cloud Delivery | | --------------------------- | ---------------------------- | ----------------- | --------------------- | | **Setup complexity** | None | Requires endpoint | Requires bucket setup | | **Real-time notifications** | No (you poll) | Yes | No | | **Automatic storage** | No | No | Yes | | **Best for** | Simple integrations, testing | Event-driven apps | Data pipelines ETLs | | **Infrastructure needed** | None | Web server | Cloud storage bucket | ### Combining methods You can combine delivery methods for redundancy: ```python Python theme={"system"} from nimble_python import Nimble nimble = Nimble(api_key="YOUR-API-KEY") # Receive webhook AND store in S3 response = nimble.extract_async( url="https://www.nimbleway.com", formats=["html", "markdown"], callback_url="https://your-server.com/webhooks/nimble", storage_type="s3", storage_url="s3://your-bucket/results/" ) ``` ```javascript Node theme={"system"} import Nimble from "@nimble-way/nimble-js"; const nimble = new Nimble({ apiKey: "YOUR-API-KEY" }); // Receive webhook AND store in S3 const response = await nimble.extractAsync({ url: "https://www.nimbleway.com", formats: ["html", "markdown"], callback_url: "https://your-server.com/webhooks/nimble", storage_type: "s3", storage_url: "s3://your-bucket/results/", }); ``` ```bash cURL theme={"system"} curl -X POST 'https://sdk.nimbleway.com/v1/extract/async' \ --header 'Authorization: Bearer YOUR-API-KEY' \ --header 'Content-Type: application/json' \ --data-raw '{ "url": "https://www.nimbleway.com", "formats": ["html", "markdown"], "callback_url": "https://your-server.com/webhooks/nimble", "storage_type": "s3", "storage_url": "s3://your-bucket/results/" }' ``` *** ## Best Practices * **Check status first** - Use `/tasks/{id}` before fetching full results - **Use reasonable intervals** - Poll every 2-5 seconds, not continuously - **Handle rate limits** - Implement retry logic for 429 responses - **Set timeouts** - Most tasks complete within seconds to minutes * **Use HTTPS** - Always use secure endpoints - **Verify authenticity** - Use custom headers for authentication - **Respond quickly** - Return 200 OK immediately, process async - **Handle retries** - Nimble retries failed deliveries * **Use prefixes** - Organize by date, project, or type - **Enable compression** - Use `storage_compress: true` for large files - **Set lifecycle policies** - Auto-delete old files to manage costs - **Use custom names** - `storage_object_name` for meaningful filenames ## Next Steps Learn about async extraction options Deep website crawling with async delivery Browse available search agents Understand API rate limit # Compliance Source: https://docs.nimbleway.com/nimble-sdk/admin/compliance # Legacy Docs Source: https://docs.nimbleway.com/nimble-sdk/admin/legacy-docs # Pricing Source: https://docs.nimbleway.com/nimble-sdk/admin/pricing Flexible usage-based pricing for Nimble SDK Nimble offers two ways to access the platform: **API pricing** for direct, per-request access, and **Data Services plans** for managed, production-scale workloads. **Free Trial:** Get started with 5,000 free web pages. No credit card required. [Start Building Now](https://online.nimbleway.com/signup) *** ## API Pricing Use any Nimble API without a commitment. Only successful requests are charged. ### Extract, Crawl & Map APIs Priced per 1,000 URLs based on the rendering driver: * **Map** always uses the `VX6` driver. * **Crawl** is billed as the sum of all `Extract` requests within the crawl. | Extract Driver | Description | PAYG Price | | -------------- | --------------------------- | ---------------- | | `VX6` | Standard — no JS rendering | \$0.90 / 1K URLs | | `VX8` | JS rendering | \$1.30 / 1K URLs | | `VX10` | JS rendering + Stealth Mode | \$1.45 / 1K URLs | Nimble selects the optimal driver automatically, selecting driver manually may cause blocks or empty responses ### Agent API Priced per 1,000 URLs based on the rendering driver: #### Community & Custom Agents Web Search Agents (WSA) you create in Studio or from the community gallery: | Agent Driver | Description | PAYG Price | | ------------ | --------------------------- | ---------------- | | `WSA-6` | Standard — no JS rendering | \$0.99 / 1K URLs | | `WSA-8` | JS rendering | \$1.45 / 1K URLs | | `WSA-10` | JS rendering + Stealth Mode | \$1.60 / 1K URLs | #### Nimble-Maintained Agents Pre-built Web Search Agents (WSA), maintained by Nimble: | Agent Driver | Description | PAYG Price | | ------------ | --------------------------- | ---------------- | | `WSA-6M` | Standard — no JS rendering | \$1.08 / 1K URLs | | `WSA-8M` | JS rendering | \$1.55 / 1K URLs | | `WSA-10M` | JS rendering + Stealth Mode | \$1.75 / 1K URLs | Nimble selects the optimal driver automatically, selecting driver manually may cause blocks or empty responses ### Search API | Operation | PAYG Price | | --------------------- | -------------------------------------------------- | | Search | \$1.00 / 1K inputs (up to 100 results per request) | | Answer (AI-generated) | \$4.00 / 1K inputs | Deep Mode search also includes all `Extract` and/or `Agent` API costs. Total cost varies by configuration. ### Proxy API | Product | PAYG Price | | ----------------- | ----------- | | Residential Proxy | \$5.30 / GB | ### Custom Rates Custom API pricing for high-scale workloads. Includes: * **Volume discounts:** lower per-request rates at scale * **Multi-year pricing protection:** rate locks and long-term discounts * **Product bundling discounts:** combine APIs for better rates * **Custom concurrent sessions:** beyond standard limits * **Dedicated support** and SLA options To discuss custom API pricing for your use case, please [Contact Sales](https://nimbleway.com/contact-general/) *** ## Data Services Plans Managed plans for teams running agents and data pipelines at scale. Billed annually. | | **Startup** | **Scale** | **Professional** | **Enterprise** | | -------------- | ----------- | ---------- | ---------------- | -------------- | | **Price** | \$2,000/mo | \$7,500/mo | \$15,000/mo | Custom | | **Agents** | 10 | 20 | Unlimited | Unlimited | | **Credits** | 500K pages | 3M pages | 8M pages | Custom | | **Concurrent** | 5 | 10 | 30 | Custom | | **Storage** | 30 days | 90 days | 365 days | Custom | * 10 agents * 500,000 web page credits * 5 concurrent agents * 30 days data storage * Custom agent ETL * MCP integration * 20 agents * 3,000,000 web page credits * 10 concurrent agents * 90 days data storage * Custom agent ETL * MCP integration * Data Lake integration * City-level agent geo-targeting * SLA & Priority support * Unlimited agents * 8,000,000 web page credits * 30 concurrent agents * 365 days data storage * Data Lake integration * City-level agent geo-targeting * SLA & Priority support Everything in Professional, plus: * Advanced security features * Custom credit volumes and concurrency limits * Dedicated support [Contact Sales](https://nimbleway.com/contact-general/) to discuss your requirements. *** ## Understanding Drivers Drivers are tiers of complexity --> You only pay for the capability your target site actually requires. Nimble selects the optimal driver automatically, so simple pages use the cheapest tier and complex, protected pages escalate only when needed. You can override the driver manually on the `/extract`, `/crawl`, and `/agent` APIs using the `driver` parameter. No JavaScript execution. Fast and lightweight. **Use when:** The page is static HTML: articles, docs, APIs, JSON endpoints. **Not suitable for:** JS-rendered pages, browser actions, network capture. Full JavaScript execution for dynamic content. **Use when:** The page requires JavaScript to load: SPAs, lazy-loaded content, browser actions, network capture. Full JS rendering with advanced anti-bot fingerprinting and evasion. **Use when:** The site actively blocks scrapers: Cloudflare, PerimeterX, or other bot protection layers. Using `VX6`, `VX8`, or `VX10` on an **Agent** request is automatically mapped to the equivalent agent driver. | Alias | Community & Custom | Nimble-Maintained | | ------ | ------------------ | ----------------- | | `VX6` | `WSA-6` | `WSA-6M` | | `VX8` | `WSA-8` | `WSA-8M` | | `VX10` | `WSA-10` | `WSA-10M` | *** ## Billing FAQ Pay-as-you-go usage is billed monthly by credit card. Data Services plans are billed annually. Only successful requests count toward your usage. Additional usage is billed at pay-as-you-go rates on your next invoice. Yes. Plan changes take effect on your next billing cycle. Credits roll over within your billing cycle but expire at the end of each month. [Contact our sales team](https://nimbleway.com/contact-general/) for custom rates based on your volume and requirements. *** ## Get Started Get 5,000 free pages to test the platform Discuss Data Services plans or custom pricing # Rate Limits Source: https://docs.nimbleway.com/nimble-sdk/admin/rate-limits Technical specifications, rate limits, and API constraints Understanding API limitations and specifications helps you optimize performance, plan capacity, and avoid throttling. This guide covers rate limits, driver capabilities, request constraints, and best practices. ## Overview Nimble's API implements tiered specifications based on: * **Driver selection**: Different drivers have different rate limits * **Request complexity**: Complex operations may have additional constraints * **Resource usage**: Fair usage policies ensure platform stability ## Drivers Nimble offers different drivers optimized for various use cases: | **Driver** | **Description** | **Best For** | | ---------- | ------------------------------- | ------------------------------ | | `vx6` | HTTP requests without rendering | Static pages, simple HTML | | `vx8` | JavaScript rendering | Dynamic content, AJAX | | `vx10` | Stealth browser mode | Protected sites, complex sites | Each driver has different pricing and rate limits. Higher-tier drivers cost more per request. ## Rate Limits ### Default rate limits by driver All accounts have the following default rate limits: | **Driver** | **Rate Limit** | | -------------------- | ------------------ | | `vx6`, `vx8`, `vx10` | 83 QPS (5,000 QPM) | **Need higher limits?** Reach out to your CS or open a via a [Support Ticket](https://portal.usepylon.com/nimble) to discuss custom rate limits based on your requirements. ### Response headers Every response includes metadata headers: ``` ratelimit-limit: 20 ratelimit-remaining: 15 x-task-id: 8e8cfde8-345b-42b8-b3e2-0c61eb11e00f ``` Use these headers to: * Monitor rate limit usage * Track request IDs for debugging * Optimize driver selection * Measure performance ### Handling rate limits When you exceed rate limits, the API returns a 429 status code: ```json theme={"system"} { "status": "failed", "msg": "Rate limit exceeded" } ``` **Best practices:** * Implement exponential backoff on 429 responses * Use the `retry_after` value to schedule next request * Monitor rate limit headers in responses * Batch requests when possible * Use lower-tier drivers when sufficient ## HTTP status codes ### Success codes | Code | Status | Description | | :------ | :----- | :--------------------------------------------------- | | **200** | OK | Request succeeded and data was returned successfully | ### Client error codes | Code | Status | Description | | :------ | :---------------- | :--------------------------------------------------------------- | | **400** | Bad Request | Invalid URL, malformed parameters, or missing required fields | | **401** | Unauthorized | Account doesn't exist, invalid API key, or missing credentials | | **402** | Payment Required | No budget, limit reached, trial expired, or trial quota finished | | **403** | Forbidden | Account blocked or not activated | | **429** | Too Many Requests | Rate limit exceeded for your plan | ### Server error codes | Code | Status | Description | | :------ | :-------------------- | :-------------------------------------- | | **500** | Internal Server Error | Unexpected server-side error occurred | | **501** | Not Implemented | Proxy service encountered an error | | **555** | Request Timeout | Request exceeded maximum execution time | Nimble automatically retries failed requests before returning a final failure status (500) # Agents Gallery Source: https://docs.nimbleway.com/nimble-sdk/agentic/agent-gallery Browse, search, and inspect available Web Search Agents Nimble **Agent Discovery APIs** help you find and understand available agents before using them. Browse agents by vertical, search by domain, and inspect detailed schemas to understand input parameters and output fields. **Can't find what you need?** Create a custom agent for any website using the [Agentic Studio](/nimble-sdk/agentic/studio) - no coding required, just describe what you want in natural language. All public agents are **maintained by Nimble** - we monitor them 24/7 and automatically update them when websites change their structure. Browse agents visually with previews, examples, and interactive documentation. ## When to Use Use agent discovery when you need to: * **Browse available agents** - See what platforms are supported * **Filter by vertical** - Find agents for specific industries (e-commerce, search, social) * **Inspect agent details** - Understand input parameters and output schema * **Check capabilities** - Verify localization and pagination support ## Two Ways to Explore Prefer a visual interface? Browse agents with previews, live demos, and exportable code snippets at the [Agent Gallery](https://online.nimbleway.com/pipeline-gallery) in Nimble Platform. *** ## List Agents API Browse and search the agent catalog. `GET https://sdk.nimbleway.com/v1/agents` ### Parameters | Parameter | Type | Description | Required | | --------- | ------- | ---------------------------------------------------- | -------- | | `privacy` | Enum | Filter by privacy level (`all`, `public`, `private`) | No | | `search` | String | Filter agents by keyword. | No | | `offset` | Integer | Pagination offset (default: `0`) | No | | `limit` | Integer | Number of results per page (default: `100`) | No | ### Response Schema Each agent in the response contains: | Field | Type | Description | | -------------- | ------- | -------------------------------------------------------------------------------------- | | `name` | string | Unique agent identifier (use this in API calls) | | `display_name` | string | Human-readable agent name | | `description` | string | What the agent does | | `is_public` | boolean | Whether the agent is publicly available | | `managed_by` | string | Who maintains this agent: `nimble` (public), `community` (public), or `self` (private) | | `vertical` | string | Industry vertical (e.g., "Ecommerce", "Search") | | `entity_type` | string | Page type (e.g., "Product Detail Page", "SERP") | | `domain` | string | Target website domain | ### Usage Examples ```python Python theme={"system"} from nimble_python import Nimble nimble = Nimble(api_key="YOUR-API-KEY") # List all public agents agents = nimble.agent.list(privacy="public") for agent in agents: print(f"{agent.name}: {agent.display_name}") print(f" Domain: {agent.domain}") print(f" Vertical: {agent.vertical}") print(f" Type: {agent.entity_type}") ``` ```javascript Node theme={"system"} import Nimble from "@nimble-way/nimble-js"; const nimble = new Nimble({ apiKey: "YOUR-API-KEY" }); // List all public agents const agents = await nimble.agent.list({ privacy: "public" }); agents.forEach((agent) => { console.log(`${agent.name}: ${agent.display_name}`); console.log(` Domain: ${agent.domain}`); console.log(` Vertical: ${agent.vertical}`); console.log(` Type: ${agent.entity_type}`); }); ``` ```bash cURL theme={"system"} curl -X GET 'https://sdk.nimbleway.com/v1/agents?privacy=public' \ -H 'Authorization: Bearer YOUR-API-KEY' ``` ### Response Example ```json theme={"system"} [ { "name": "amazon_pdp", "is_public": true, "display_name": "Amazon Product Page", "description": "Extract structured data from Amazon product detail pages including pricing, reviews, specifications, and availability.", "vertical": "Ecommerce", "entity_type": "Product Detail Page (PDP)", "domain": "www.amazon.com" }, { "name": "amazon_serp", "is_public": true, "display_name": "Amazon Search Results", "description": "Extract product listings from Amazon search results pages based on a keyword query.", "vertical": "Ecommerce", "entity_type": "Search Engine Results Page (SERP)", "domain": "www.amazon.com" }, { "name": "walmart_pdp", "is_public": true, "display_name": "Walmart Product Page", "description": "Extract structured data from Walmart product detail pages including pricing, reviews, and variants.", "vertical": "Ecommerce", "entity_type": "Product Detail Page (PDP)", "domain": "www.walmart.com" }, { "name": "google_search", "is_public": true, "display_name": "Google Search Results", "description": "Extract structured data from Google Search Results Pages (SERPs) including organic results, ads, and featured snippets.", "vertical": "Search Engine", "entity_type": "Search Engine Results Page (SERP)", "domain": "www.google.com" } ] ``` *** ## Get Agent Details API Get comprehensive information about a specific agent, including input parameters and output schema. ```GET https://sdk.nimbleway.com/v1/agents/{agent_name} theme={"system"} ``` ### Parameters | Parameter | Type | Description | Required | | ------------ | ------ | ----------------------------------- | -------- | | `agent_name` | string | The agent name (e.g., `amazon_pdp`) | Yes | ### Response Schema | Field | Type | Description | | ------------------ | ------- | -------------------------------------------------------------------------------------- | | `name` | string | Unique agent identifier | | `display_name` | string | Human-readable agent name | | `description` | string | What the agent does | | `is_public` | boolean | Whether the agent is publicly available | | `managed_by` | string | Who maintains this agent: `nimble` (public), `community` (public), or `self` (private) | | `vertical` | string | Industry vertical | | `entity_type` | string | Page type | | `domain` | string | Target website domain | | `input_properties` | array | Input parameters for the agent (see below) | | `output_schema` | object | JSON schema describing output fields | | `feature_flags` | object | Supported capabilities (localization, pagination) | #### Input Properties Each item in `input_properties` contains: | Field | Type | Description | | ------------- | ------- | ----------------------------------- | | `name` | string | Parameter name | | `type` | string | Data type (string, number, boolean) | | `description` | string | What the parameter does | | `required` | boolean | Whether the parameter is required | | `default` | string | Default value if not provided | | `examples` | array | Example values | | `rules` | array | Validation rules | #### Feature Flags | Flag | Type | Description | | --------------------------- | ------- | -------------------------------------------- | | `is_localization_supported` | boolean | Supports ZIP code for location-specific data | | `is_pagination_supported` | boolean | Supports paginated results | ### Usage Examples ```python Python theme={"system"} from nimble_python import Nimble nimble = Nimble(api_key="YOUR-API-KEY") # Get agent details agent = nimble.agent.get("amazon_pdp") print(f"Agent: {agent.display_name}") print(f"Description: {agent.description}") print(f"Domain: {agent.domain}") # Check feature flags flags = agent.feature_flags print(f"\nCapabilities:") print(f" Localization: {'Yes' if flags.is_localization_supported else 'No'}") print(f" Pagination: {'Yes' if flags.is_pagination_supported else 'No'}") # List input parameters print(f"\nInput Parameters:") for param in agent.input_properties: required = "required" if param.required else "optional" print(f" {param.name} ({param.type}) - {required}") print(f" {param.description}") if param.examples: print(f" Examples: {', '.join(param.examples)}") ``` ```javascript Node theme={"system"} import Nimble from "@nimble-way/nimble-js"; const nimble = new Nimble({ apiKey: "YOUR-API-KEY" }); // Get agent details const agent = await nimble.agent.get("amazon_pdp"); console.log(`Agent: ${agent.display_name}`); console.log(`Description: ${agent.description}`); console.log(`Domain: ${agent.domain}`); // Check feature flags const flags = agent.feature_flags; console.log("\nCapabilities:"); console.log( ` Localization: ${flags.is_localization_supported ? "Yes" : "No"}`, ); console.log(` Pagination: ${flags.is_pagination_supported ? "Yes" : "No"}`); // List input parameters console.log("\nInput Parameters:"); agent.input_properties.forEach((param) => { const required = param.required ? "required" : "optional"; console.log(` ${param.name} (${param.type}) - ${required}`); console.log(` ${param.description}`); if (param.examples?.length) { console.log(` Examples: ${param.examples.join(", ")}`); } }); ``` ```bash cURL theme={"system"} curl -X GET 'https://sdk.nimbleway.com/v1/agents/amazon_pdp' \ -H 'Authorization: Bearer YOUR-API-KEY' ``` ### Response Example ```json theme={"system"} { "name": "amazon_pdp", "is_public": true, "display_name": "Amazon Product Page", "description": "Extract structured data from Amazon product detail pages including pricing, reviews, specifications, variants, and availability.", "vertical": "Ecommerce", "entity_type": "Product Detail Page (PDP)", "domain": "www.amazon.com", "input_properties": [ { "name": "asin", "required": true, "type": "string", "description": "Amazon Standard Identification Number (ASIN) - the unique 10-character product identifier", "rules": ["Must be exactly 10 alphanumeric characters"], "examples": ["B08N5WRWNW", "B0DLKFK6LR"], "default": null }, { "name": "zip_code", "required": false, "type": "string", "description": "ZIP code for location-specific pricing and availability", "rules": ["5-digit US ZIP code"], "examples": ["90210", "10001"], "default": "90210" } ], "output_schema": { "asin": { "type": "string", "description": "Product ASIN" }, "product_title": { "type": "string", "description": "Full product name" }, "brand": { "type": "string", "description": "Product brand" }, "web_price": { "type": "number", "description": "Current selling price" }, "list_price": { "type": "number", "description": "Original price before discount" }, "average_of_reviews": { "type": "number", "description": "Average rating (0-5)" }, "number_of_reviews": { "type": "number", "description": "Total review count" }, "availability": { "type": "boolean", "description": "Whether in stock" }, "image_url": { "type": "string", "description": "Main product image URL" } }, "feature_flags": { "is_localization_supported": true, "is_pagination_supported": false } } ``` *** ## Available Agents by Vertical ### E-commerce | Agent | Display Name | Domain | Localization | Pagination | | ----------------- | -------------------- | --------------------------------------------- | ------------ | ---------- | | `amazon_pdp` | Amazon Product Page | [www.amazon.com](http://www.amazon.com) | Yes | No | | `amazon_serp` | Amazon Search | [www.amazon.com](http://www.amazon.com) | Yes | Yes | | `amazon_category` | Amazon Category | [www.amazon.com](http://www.amazon.com) | No | Yes | | `walmart_pdp` | Walmart Product Page | [www.walmart.com](http://www.walmart.com) | Yes | No | | `walmart_search` | Walmart Search | [www.walmart.com](http://www.walmart.com) | Yes | Yes | | `target_pdp` | Target Product Page | [www.target.com](http://www.target.com) | Yes | No | | `best_buy_pdp` | Best Buy Product | [www.bestbuy.com](http://www.bestbuy.com) | Yes | No | | `home_depot_pdp` | Home Depot Product | [www.homedepot.com](http://www.homedepot.com) | Yes | No | ### Search Engines | Agent | Display Name | Domain | Localization | Pagination | | -------------------- | --------------------- | --------------------------------------- | ------------ | ---------- | | `google_search` | Google Search Results | [www.google.com](http://www.google.com) | Yes | Yes | | `google_maps_search` | Google Maps Search | maps.google.com | Yes | Yes | | `google_search_aio` | Google AI Overview | [www.google.com](http://www.google.com) | Yes | No | ### Social Media | Agent | Display Name | Domain | Localization | Pagination | | ---------------- | -------------- | ------------------------------------------- | ------------ | ---------- | | `tiktok_account` | TikTok Account | [www.tiktok.com](http://www.tiktok.com) | No | Yes | | `facebook_page` | Facebook Page | [www.facebook.com](http://www.facebook.com) | No | No | | `youtube_shorts` | YouTube Shorts | [www.youtube.com](http://www.youtube.com) | No | Yes | ### LLM Platforms | Agent | Display Name | Domain | Localization | Pagination | | ------------ | ----------------- | --------------------------------------------- | ------------ | ---------- | | `chatgpt` | ChatGPT Responses | chatgpt.com | No | No | | `gemini` | Gemini Responses | gemini.google.com | No | No | | `perplexity` | Perplexity | [www.perplexity.ai](http://www.perplexity.ai) | No | No | | `grok` | Grok Responses | grok.com | No | No | *** ## Common Patterns ### Search Agents by Keyword Use the `search` query parameter to find agents matching a term across `display_name`, `vertical`, and `domain`: ```python Python theme={"system"} from nimble_python import Nimble nimble = Nimble(api_key="YOUR-API-KEY") agents = nimble.agent.list(search="amazon") for agent in agents: print(f"{agent.name}: {agent.display_name} ({agent.domain})") ``` ```javascript Node theme={"system"} import Nimble from "@nimble-way/nimble-js"; const nimble = new Nimble({ apiKey: "YOUR-API-KEY" }); const agents = await nimble.agent.list({ search: "amazon" }); agents.forEach((agent) => { console.log(`${agent.name}: ${agent.display_name} (${agent.domain})`); }); ``` ```bash cURL theme={"system"} curl -X GET 'https://sdk.nimbleway.com/v1/agents?search=amazon' \ -H 'Authorization: Bearer YOUR-API-KEY' ``` ### Filter Agents by Vertical Find all agents for a specific industry: ```python theme={"system"} from nimble_python import Nimble nimble = Nimble(api_key="YOUR-API-KEY") # Get all agents agents = nimble.agent.list(privacy="public") # Filter by vertical ecommerce_agents = [ agent for agent in agents if agent.vertical == 'Ecommerce' ] print(f"Found {len(ecommerce_agents)} e-commerce agents:") for agent in ecommerce_agents: print(f" {agent.name}: {agent.display_name} ({agent.domain})") ``` ### Check Localization Support Verify an agent supports location-specific data: ```python theme={"system"} from nimble_python import Nimble nimble = Nimble(api_key="YOUR-API-KEY") agent = nimble.agent.get("amazon_pdp") if agent.feature_flags.is_localization_supported: print("This agent supports localization!") print("Pass zip_code parameter for location-specific pricing.") else: print("This agent does not support localization.") ``` ### Get Required Parameters Find which parameters are required before calling an agent: ```python theme={"system"} from nimble_python import Nimble nimble = Nimble(api_key="YOUR-API-KEY") agent = nimble.agent.get("amazon_pdp") required_params = [ p for p in agent.input_properties if p.required ] optional_params = [ p for p in agent.input_properties if not p.required ] print("Required parameters:") for p in required_params: examples = ', '.join(p.examples) if p.examples else '' print(f" {p.name}: {p.description}") if examples: print(f" Examples: {examples}") print("\nOptional parameters:") for p in optional_params: default = p.default if p.default else 'none' print(f" {p.name}: {p.description} (default: {default})") ``` ### Validate Parameters Before Calling Check parameters match the expected schema: ```python theme={"system"} from nimble_python import Nimble nimble = Nimble(api_key="YOUR-API-KEY") def validate_params(agent_name, params): """Validate params against agent input_properties""" agent = nimble.agent.get(agent_name) errors = [] # Build lookup of input properties props = {p.name: p for p in agent.input_properties} # Check required fields for prop_name, prop in props.items(): if prop.required and prop_name not in params: errors.append(f"Missing required parameter: {prop_name}") # Check for unknown parameters for param_name in params: if param_name not in props: errors.append(f"Unknown parameter: {param_name}") return errors # Validate before calling errors = validate_params("amazon_pdp", {"asin": "B08N5WRWNW"}) if errors: print("Validation errors:") for error in errors: print(f" - {error}") else: print("Parameters are valid!") ``` *** ## Next Steps Explore endpoints, request parameters, and response schemas for the Agents API Learn how to call agents in your code Build your own agent with natural language Browse agents visually # Search Agents Source: https://docs.nimbleway.com/nimble-sdk/agentic/agents Pre-built extraction agents for popular platforms - zero configuration required Nimble **Web Search Agents (WSA)** are ready-to-use extraction agents for popular websites like Amazon, Google, LinkedIn, and hundreds more. No CSS selectors or scraping expertise required - just provide the agent name and parameters, and get structured data instantly. ## Quick Start ### Example Request ```python Python theme={"system"} from nimble_python import Nimble nimble = Nimble(api_key="YOUR-API-KEY") result = nimble.agent.run( agent="amazon_pdp", params={ "asin": "B0DLKFK6LR" } ) parsed = result.data.parsing["parsed"] print(f"Product: {parsed['product_title']}") print(f"Price: ${parsed['web_price']}") ``` ```javascript Node theme={"system"} import Nimble from "@nimble-way/nimble-js"; const nimble = new Nimble({ apiKey: "YOUR-API-KEY" }); const result = await nimble.agent.run({ agent: "amazon_pdp", params: { asin: "B0DLKFK6LR", }, }); const parsed = result.data.parsing.parsed; console.log(`Product: ${parsed.product_title}`); console.log(`Price: $${parsed.web_price}`); ``` ```bash cURL theme={"system"} curl -X POST 'https://sdk.nimbleway.com/v1/agents/run' \ --header 'Authorization: Bearer ' \ --header 'Content-Type: application/json' \ --data-raw '{ "agent": "amazon_pdp", "params": { "asin": "B0DLKFK6LR" } }' ``` ### Example Response ```json theme={"system"} { "url": "https://www.amazon.com/dp/B08N5WRWNW", "task_id": "b1fa7943-cba5-4ec2-a88c-4d2d6799c794", "status": "success", "data": { "html": "...", "parsing": { "asin": "B08N5WRWNW", "product_title": "Apple AirPods Pro (2nd Generation)", "brand": "Apple", "web_price": 249.0, "list_price": 279.0, "average_of_reviews": 4.7, "number_of_reviews": 125432, "availability": true } }, "status_code": 200 } ``` ## How it works Choose an agent by name (e.g., `amazon_pdp`, `google_search`) and pass the required parameters like product ID, search query, or URL The agent fetches the page, handles anti-bot protection, and extracts data using battle-tested selectors maintained by Nimble Get clean, normalized JSON with consistent field names - ready to use in your application immediately ## Two types of agents \ **Maintained by Nimble** - Gallery of pre-built agents for popular websites. \ \ Battle-tested, auto-healing, and updated 24/7 when sites change. Browse the [Gallery](/nimble-sdk/agentic/agent-gallery). \ **Created by you** - Build agents for any website using the [Agentic Studio](/nimble-sdk/agentic/studio). \ \ No coding required - just describe what you need in natural language. ## Parameters Supported input parameters: The name of the pre-built agent to use. Each agent is designed for a specific platform or data type. **Popular agents:** * `amazon_pdp` - Amazon product pages * `amazon_serp` - Amazon search results * `google_search` - Google search results * `google_maps_search` - Google Maps locations * `walmart_pdp` - Walmart products * `chatgpt` - ChatGPT prompt results * `perplexity` - Perplexity prompt results [Browse all agents →](/nimble-sdk/agentic/agent-gallery) Agent-specific parameters that tell the agent what data to fetch. Each agent has different requirements. **Common param types:** * Product IDs (ASINs, SKUs) * Search queries * URLs or usernames * Page numbers for pagination **Example:** ```json theme={"system"} { "asin": "B08N5WRWNW" } ``` Enable location-based pricing and availability. **Required** when passing `zip_code` or `store_id` in params. Only available for agents that support localization (check agent details). **Example:** ```json theme={"system"} { "agent": "amazon_pdp", "localization": true, "params": { "asin": "B08N5WRWNW", "zip_code": "90210" } } ``` Each agent has unique parameter requirements. Check the [Agent Gallery](/nimble-sdk/agentic/agent-gallery) for exact parameters for each agent. ## Usage ### E-commerce product extraction Extract product data from Amazon, Walmart, and other retailers: ```python Python theme={"system"} from nimble_python import Nimble nimble = Nimble(api_key="YOUR-API-KEY") # Amazon product result = nimble.agent.run( agent="amazon_pdp", params={ "asin": "B08N5WRWNW" } ) parsed = result.data.parsing["parsed"] print(f"Product: {parsed['product_title']}") print(f"Price: ${parsed['web_price']}") print(f"Rating: {parsed['average_of_reviews']}") ``` ```javascript Node theme={"system"} import Nimble from "@nimble-way/nimble-js"; const nimble = new Nimble({ apiKey: "YOUR-API-KEY" }); // Amazon product const result = await nimble.agent.run({ agent: "amazon_pdp", params: { asin: "B08N5WRWNW", }, }); const parsed = result.data.parsing.parsed; console.log(`Product: ${parsed.product_title}`); console.log(`Price: $${parsed.web_price}`); console.log(`Rating: ${parsed.average_of_reviews}`); ``` ```bash cURL theme={"system"} curl -X POST 'https://sdk.nimbleway.com/v1/agents/run' \ --header 'Authorization: Bearer ' \ --header 'Content-Type: application/json' \ --data-raw '{ "agent": "amazon_pdp", "params": { "asin": "B08N5WRWNW" } }' ``` ### Search results extraction Get search results from Google, Amazon, and other platforms: ```python Python theme={"system"} from nimble_python import Nimble nimble = Nimble(api_key="YOUR-API-KEY") # Amazon search result = nimble.agent.run( agent="amazon_serp", params={ "keyword": "wireless headphones" } ) products = result.data.parsing["parsed"] print(f"Found {len(products)} products") for product in products: print(f"- {product['product_name']}: ${product['price']}") ``` ```javascript Node theme={"system"} import Nimble from "@nimble-way/nimble-js"; const nimble = new Nimble({ apiKey: "YOUR-API-KEY" }); // Amazon search const result = await nimble.agent.run({ agent: "amazon_serp", params: { keyword: "wireless headphones", }, }); const products = result.data.parsing.parsed; console.log(`Found ${products.length} products`); products.forEach((product) => { console.log(`- ${product.product_name}: $${product.price}`); }); ``` ```bash cURL theme={"system"} curl -X POST 'https://sdk.nimbleway.com/v1/agents/run' \ --header 'Authorization: Bearer ' \ --header 'Content-Type: application/json' \ --data-raw '{ "agent": "amazon_serp", "params": { "keyword": "wireless headphones" } }' ``` ### Google Maps extraction Find businesses and locations: ```python Python theme={"system"} from nimble_python import Nimble nimble = Nimble(api_key="YOUR-API-KEY") result = nimble.agent.run( agent="google_maps_search", params={ "query": "coffee shops near Times Square" } ) parsed = result.data.parsing["parsed"] for place in parsed["entities"]["SearchResult"]: print(f"- {place['title']} ({place['rating']} stars)") ``` ```javascript Node theme={"system"} import Nimble from "@nimble-way/nimble-js"; const nimble = new Nimble({ apiKey: "YOUR-API-KEY" }); const result = await nimble.agent.run({ agent: "google_maps_search", params: { query: "coffee shops near Times Square", }, }); const parsed = result.data.parsing.parsed; parsed.entities.SearchResult.forEach((place) => { console.log(`- ${place.title} (${place.rating} stars)`); }); ``` ```bash cURL theme={"system"} curl -X POST 'https://sdk.nimbleway.com/v1/agents/run' \ --header 'Authorization: Bearer ' \ --header 'Content-Type: application/json' \ --data-raw '{ "agent": "google_maps_search", "params": { "query": "coffee shops near Times Square" } }' ``` ### LLM platform extraction Get responses from AI platforms like ChatGPT and Perplexity: ```python Python theme={"system"} from nimble_python import Nimble nimble = Nimble(api_key="YOUR-API-KEY") result = nimble.agent.run( agent="chatgpt", params={ "prompt": "What are the best practices for web scraping?" } ) parsed = result.data.parsing["parsed"] print(parsed["response"]) ``` ```javascript Node theme={"system"} import Nimble from "@nimble-way/nimble-js"; const nimble = new Nimble({ apiKey: "YOUR-API-KEY" }); const result = await nimble.agent.run({ agent: "chatgpt", params: { prompt: "What are the best practices for web scraping?", }, }); console.log(result.data.parsing.parsed.response); ``` ```bash cURL theme={"system"} curl -X POST 'https://sdk.nimbleway.com/v1/agents/run' \ --header 'Authorization: Bearer ' \ --header 'Content-Type: application/json' \ --data-raw '{ "agent": "chatgpt", "params": { "prompt": "What are the best practices for web scraping?" } }' ``` ### Localized extraction Get location-specific pricing and availability: ```python Python theme={"system"} from nimble_python import Nimble nimble = Nimble(api_key="YOUR-API-KEY") # Get New York pricing result = nimble.agent.run( agent="amazon_pdp", localization=True, params={ "asin": "B08N5WRWNW", "zip_code": "10001" } ) parsed = result.data.parsing["parsed"] print(f"Price in NYC: ${parsed['web_price']}") ``` ```javascript Node theme={"system"} import Nimble from "@nimble-way/nimble-js"; const nimble = new Nimble({ apiKey: "YOUR-API-KEY" }); // Get New York pricing const result = await nimble.agent.run({ agent: "amazon_pdp", localization: true, params: { asin: "B08N5WRWNW", zip_code: "10001", }, }); console.log(`Price in NYC: $${result.data.parsing.parsed.web_price}`); ``` ```bash cURL theme={"system"} curl -X POST 'https://sdk.nimbleway.com/v1/agents/run' \ --header 'Authorization: Bearer ' \ --header 'Content-Type: application/json' \ --data-raw '{ "agent": "amazon_pdp", "localization": true, "params": { "asin": "B08N5WRWNW", "zip_code": "10001" } }' ``` ### Async extraction Run agent extractions asynchronously: ```python Python theme={"system"} from nimble_python import Nimble nimble = Nimble(api_key="YOUR-API-KEY") response = nimble.agent.run_async( agent="amazon_pdp", params={"asin": "B0DLKFK6LR"} ) task_id = response.task_id print(f"Task submitted: {task_id}") # Poll for completion import time while True: my_task = nimble.tasks.get(task_id) print(f"Status: {my_task.task.state}") if my_task.state == "success": break elif my_task.state == "failed": print(f"Task failed: {status.error}") break time.sleep(15) ``` ```javascript Node theme={"system"} import Nimble from "@nimble-way/nimble-js"; const nimble = new Nimble({ apiKey: "YOUR-API-KEY" }); const response = await nimble.agent.runAsync({ agent: "amazon_pdp", params: { asin: "B0DLKFK6LR" }, }); const taskId = response.task_id; console.log(`Task submitted: ${taskId}`); // Poll for completion while (true) { const myTask = await nimble.tasks.get(taskId); console.log(`Status: ${myTask.task.state}`); if (myTask.state === "success") break; if (myTask.state === "failed") { console.log(`Task failed: ${status.error}`); break; } await new Promise((resolve) => setTimeout(resolve, 15000)); } ``` ```bash cURL theme={"system"} # Submit async extraction curl -X POST 'https://sdk.nimbleway.com/v1/agent/async' \ --header 'Authorization: Bearer ' \ --header 'Content-Type: application/json' \ --data-raw '{ "agent": "amazon_pdp", "params": {"asin": "B08N5WRWNW"} }' # Response includes task_id # {"task": {"id": "abc123..."}} # Check status curl 'https://sdk.nimbleway.com/v1/tasks/{task_id}' \ --header 'Authorization: Bearer ' # Get results when completed curl 'https://sdk.nimbleway.com/v1/tasks/{task_id}/results' \ --header 'Authorization: Bearer ' ``` Async mode supports polling, callbacks and cloud storage ## Agent Gallery Browse pre-built agents **maintained by Nimble** for popular platforms: Browse all agents with interactive documentation and live testing ### E-commerce | Agent | Platform | Description | | ---------------- | -------- | --------------------------------- | | `amazon_pdp` | Amazon | Product details, pricing, reviews | | `amazon_serp` | Amazon | Search results with products | | `walmart_pdp` | Walmart | Product details and pricing | | `walmart_search` | Walmart | Search results | | `target_pdp` | Target | Product details | | `best_buy_pdp` | Best Buy | Product details | ### Search Engines | Agent | Platform | Description | | -------------------- | ----------- | ------------------------------- | | `google_search` | Google | Search results with snippets | | `google_maps_search` | Google Maps | Business listings and locations | | `google_search_aio` | Google | AI Overview results | ### LLM Platforms | Agent | Platform | Description | | ------------ | ------------- | --------------------- | | `chatgpt` | ChatGPT | Prompt responses | | `perplexity` | Perplexity | Search + AI responses | | `gemini` | Google Gemini | Prompt responses | | `grok` | Grok | Prompt responses | ### Social Media | Agent | Platform | Description | | ---------------- | -------- | --------------------------- | | `tiktok_account` | TikTok | Account profiles and videos | | `facebook_page` | Facebook | Page information | | `youtube_shorts` | YouTube | Short-form videos | ## Create Custom Agents Can't find an agent for your target website? Create your own using **Agentic Studio** - no coding required. Go to the [Agentic Studio](https://online.nimbleway.com/workflow-builder/create) in Nimble Platform Enter the website URL and describe what data you need in plain English: *"Extract product name, price, rating, and all review comments"* Our AI analyzes the page and builds an extraction agent automatically - no CSS selectors needed Preview extracted data and refine your description if needed Your custom agent is available via the same Agent API with your chosen name ### Custom agent example ```python Python theme={"system"} from nimble_python import Nimble nimble = Nimble(api_key="YOUR-API-KEY") # Use your custom agent just like a public one result = nimble.agent.run( agent="my_custom_store_pdp", # Your custom agent name params={ "url": "https://www.nike.com/t/air-max-90-mens-shoes" } ) parsed = result.data.parsing["parsed"] print(f"Product: {parsed['product_name']}") print(f"Price: ${parsed['price']}") ``` ```javascript Node theme={"system"} import Nimble from "@nimble-way/nimble-js"; const nimble = new Nimble({ apiKey: "YOUR-API-KEY" }); // Use your custom agent just like a public one const result = await nimble.agent.run({ agent: "my_custom_store_pdp", // Your custom agent name params: { url: "https://www.nike.com/t/air-max-90-mens-shoes", }, }); const parsed = result.data.parsing.parsed; console.log(`Product: ${parsed.product_name}`); console.log(`Price: $${parsed.price}`); ``` ```bash cURL theme={"system"} curl -X POST 'https://sdk.nimbleway.com/v1/agents/run' \ --header 'Authorization: Bearer ' \ --header 'Content-Type: application/json' \ --data-raw '{ "agent": "my_custom_store_pdp", "params": { "url": "https://www.nike.com/t/air-max-90-mens-shoes" } }' ``` **Custom agents are private to your account** and not maintained by Nimble. You're responsible for updating them if the target site changes. For popular sites, always prefer public agents which are **maintained by Nimble** 24/7. ## Response Fields | Field | Type | Description | | -------------- | ------ | --------------------------------- | | `url` | string | The URL that was extracted | | `task_id` | string | Unique identifier for the request | | `status` | string | `success` or `failed` | | `data.html` | string | Raw HTML content | | `data.parsing` | object | Structured extracted data | | `status_code` | number | HTTP status code from target | ## Use cases Track prices across Amazon, Walmart, and other retailers with consistent data formats Gather comprehensive product data from major e-commerce platforms Monitor Google, Bing, and marketplace search results for SEO and visibility Extract competitor data from any website using public or custom agents ## Agents vs other tools | What you need | Use | | ---------------------------------------------- | ----------------------------------------------------------------------------------------- | | Data from popular sites (Amazon, Google, etc.) | **Public Agents** - [browse gallery](/nimble-sdk/agentic/agent-gallery) | | Data from sites not in the gallery | **Custom Agents** - [create in Studio](/nimble-sdk/agentic/studio) | | Data from specific URLs (expert users) | [**Extract**](/nimble-sdk/web-tools/extract/quickstart) - full control with CSS selectors | | Data from entire website | [**Crawl**](/nimble-sdk/web-tools/crawl) | | Search web + extract content from results | [**Search**](/nimble-sdk/web-tools/search) | **Most users should start with Web Search Agents.** Public agents cover popular sites with zero configuration. For anything else, create a custom agent in the Agentic Studio - it's the fastest path to production-ready data extraction. ## Next steps Browse all available public agents with API documentation Explore endpoints, request parameters, and response schemas for the Agents API Create custom agents for any website with natural language # Agent Creation Source: https://docs.nimbleway.com/nimble-sdk/agentic/studio Create custom Web Search Agents with natural language - no coding required ## Overview **Agentic Studio** is a no-code tool for creating custom Web Search Agents. Simply describe what data you need in plain English, and our AI builds a production-ready agent for you. No CSS selectors, no coding, no scraping expertise required. Think of it as having an expert scraper build your agent for you - just tell it what you want. Open the Agentic Studio in Nimble Platform Use the Nimble Agents skill to generate agents and write a full analysis script from a single prompt ## Why use Agentic Studio? Describe what you need in natural language - the AI handles the technical details Agents are optimized for scale with predictable costs, unlike one-off Ask calls Create agents for sites not covered by our public gallery Use your custom agent via the same Agent API immediately ## How it works Go to the [Agentic Studio](https://online.nimbleway.com/workflow-builder/create) in the Nimble Platform Enter the website URL you want to extract from and describe what data you need in plain English. For example: "Extract product name, price, rating, and all review comments" Our AI analyzes the page structure and builds an extraction agent based on your description. It identifies the right selectors and data patterns automatically. Preview the extracted data to make sure it matches your needs. Refine your description if needed - the AI will adjust the agent. Save your agent with a custom name. It's now available via the Agent API - just like public Agent, but private to your account. ## Custom vs. Public Agent | Feature | Public Agent | Custom Agent | | ------------ | --------------------------------- | ------------------------- | | Maintenance | **Maintained by Nimble** 24/7 | Maintained by you | | Setup | Zero - just use agent name | Create in Agentic Studio | | Availability | Popular sites | Any website | | Auto-healing | Yes - we update when sites change | No - you update if needed | | API usage | Same Agent API | Same Agent API | | Visibility | Available to all users | Private to your account | **Always prefer public Agent when available.** They're battle-tested, auto-healing, and **maintained by Nimble**. Only create custom Agent for sites not in the gallery. ## Example: Creating a custom agent Let's say you need to extract data from a niche e-commerce site that's not in our public gallery. **Step 1: Enter the URL** ``` https://example-niche-store.com/products/widget-pro ``` **Step 2: Describe what you need** ``` Extract the product name, current price, original price (if on sale), stock status, all product specifications as key-value pairs, and the first 5 customer reviews with rating and text. ``` **Step 3: Review the extracted data** ```json theme={"system"} { "product_name": "Widget Pro 3000", "current_price": 49.99, "original_price": 79.99, "stock_status": "In Stock", "specifications": [ { "key": "Dimensions", "value": "10 x 5 x 3 inches" }, { "key": "Weight", "value": "1.2 lbs" }, { "key": "Material", "value": "Aluminum" } ], "reviews": [ { "rating": 5, "text": "Excellent product, exactly what I needed!" }, { "rating": 4, "text": "Good quality, fast shipping." } ] } ``` **`Step 4: Save as niche_store_pdp`** **Step 5: Use via API** ```python Python theme={"system"} from nimble_python import Nimble nimble = Nimble(api_key="YOUR-API-KEY") # Use your custom agent just like a public one result = nimble.agent.run( agent="niche_store_pdp", # Your custom agent name params={ "url": "https://example-niche-store.com/products/another-widget" } ) print(f"Product: {result['parsing']['parsed']['product_name']}") print(f"Price: ${result['parsing']['parsed']['current_price']}") ``` ```javascript Node theme={"system"} import Nimble from "@nimble-way/nimble-js"; const nimble = new Nimble({ apiKey: "YOUR-API-KEY" }); // Use your custom agent just like a public one const result = await nimble.agent.run({ agent: "niche_store_pdp", // Your custom agent name params: { url: "https://example-niche-store.com/products/another-widget", }, }); console.log(`Product: ${result.parsing.parsed.product_name}`); console.log(`Price: $${result.parsing.parsed.current_price}`); ``` ```bash cURL theme={"system"} curl -X POST 'https://sdk.nimbleway.com/v1/agents/run' \ --header 'Authorization: Bearer YOUR-API-KEY' \ --header 'Content-Type: application/json' \ --data-raw '{ "agent": "niche_store_pdp", "params": { "url": "https://example-niche-store.com/products/another-widget" } }' ``` ## Tips for better agents **Be specific about data types** Instead of "get the price", say "extract the current price as a number without currency symbols" **Describe the structure you want** Instead of "get reviews", say "extract reviews as an array with rating (1-5) and review text for each" **Mention edge cases** "Extract the sale price if available, otherwise use the regular price" **Test with multiple pages** Try your agent on different product pages to ensure it works consistently ### Agent naming * Use lowercase with underscores: `my_store_pdp` * Be descriptive: `competitor_pricing` not `agent1` * Include the site or type: `niche_store_reviews` ## Limitations * **Maintenance is your responsibility**: Unlike public Agent that are **maintained by Nimble**, you need to update custom Agent if the target site changes its structure * **Single-page extraction**: Agents extract from one page type. For multi-page workflows, create separate agents ## FAQ Yes! Custom Agent are designed for production use. They use the same reliable infrastructure as public Agent with predictable costs and high throughput. Unlike public Agent (**maintained by Nimble** 24/7), custom Agent don't auto-heal. If your extractions start failing or returning incorrect data, you'll need to update your agent in the Agentic Studio. There's no limit on the number of custom Agent you can create. Create as many as you need for your use cases. Yes, custom Agent are available to all members of your Nimble account. They're private to your organization but shared within your team. ## Next steps Start creating your first custom agent Check if a public agent exists for your site first Learn how to use Web Search Agents via API # Installation Source: https://docs.nimbleway.com/nimble-sdk/getting-started/installation Get started with Nimble by installing our SDK or CLI and setting up your environment **Using the API directly?** If you prefer HTTP requests (cURL, fetch, axios, etc.) you can skip installation and go straight to the [API Reference](/api-reference/introduction). ## Prerequisites * A valid [Nimble API key](https://online.nimbleway.com/account-settings/api-keys) * One of the following runtimes for your chosen SDK: * **Python**: 3.9+ * **Node.js**: 20 LTS+ * **Go**: 1.22+ ## Python ```bash httpx theme={"system"} pip install nimble_python ``` ```bash aiohttp theme={"system"} pip install "nimble_python[aiohttp]" ``` [View Python SDK docs →](/nimble-sdk/sdks/python) ## Node.js ```bash theme={"system"} npm install @nimble-way/nimble-js ``` Works with Node.js 20+, Deno v1.28+, Bun 1.0+, and Cloudflare Workers. [View Node SDK docs →](/nimble-sdk/sdks/node) ## Go ```bash theme={"system"} go get github.com/Nimbleway/nimble-go@latest' ``` [View Go SDK docs →](/nimble-sdk/sdks/go) ## CLI The Nimble CLI lets you interact with the API directly from your terminal (requires Go 1.22+): ```bash theme={"system"} go install 'github.com/Nimbleway/nimble-cli/cmd/nimble@latest' ``` Add the Go bin directory to your PATH if needed: ```bash theme={"system"} export PATH="$PATH:$(go env GOPATH)/bin" ``` [View CLI docs →](/nimble-sdk/sdks/cli) ## Authentication All SDKs and the CLI read your API key from the `NIMBLE_API_KEY` environment variable: ```bash macOS / Linux theme={"system"} export NIMBLE_API_KEY="your-api-key" ``` ```bash Windows (PowerShell) theme={"system"} $env:NIMBLE_API_KEY="your-api-key" ``` You can also pass it directly when initializing the client (not recommended for production): ```python Python theme={"system"} from nimble_python import Nimble client = Nimble(api_key="your-api-key") ``` ```typescript Node theme={"system"} import Nimble from "@nimble-way/nimble-js"; const nimble = new Nimble({ apiKey: "your-api-key" }); ``` ```go Go theme={"system"} import ( nimble "github.com/Nimbleway/nimble-go" "github.com/Nimbleway/nimble-go/option" ) client := nimble.NewClient(option.WithAPIKey("your-api-key")) ``` ## Next Steps Make your first request in minutes Sync and async Python client TypeScript/JavaScript client Idiomatic Go client Use Nimble from your terminal Full REST API documentation # Welcome to Nimble Source: https://docs.nimbleway.com/nimble-sdk/getting-started/overview The AI-Native SDK for Real-Time Web Data at scale ## Get Started Nimble provides enterprise-grade web data infrastructure designed for developers who need reliable access to web content at scale. Whether you're building data pipelines, conducting market research, or powering AI applications, our platform delivers the tools and infrastructure to access web data without the complexity of managing proxies, browsers, or stealth technology. Start using Nimble's web data platform in minutes: Sign up and get your API credentials to start building Test & Explore the API effortleslly, no code is needed Quick start guides with code examples, learn how to use nimble in minutes. ## Nimble Products Nimble's products are built as layers of abstraction, each leveraging the infrastructure below it. Start with raw proxy access for maximum control, use the Web API for browser automation, or leverage AI agents for intelligent data retrieval. Choose the layer that best fits your use case. Pre-built extraction agents for popular platforms - zero configuration required Plug-and-play skills for Claude Code, Cursor, and other AI coding assistants High-performance residential proxy network with advanced stealth capabilities \ ([Contact](https://login.start-chat.com/modal/601e4da4-50a0-4dc7-8155-bb84f2952c40/27847549-dc04-4144-98a5-d40e568bf6af?magicLinkId=slFQ2j\&UID=65ab324c-ecde-4748-9f52-8e395054774b.1753267174019) sales to gain access) Browser automation and data extraction API for raw & dynamic content # Quickstart Guide Source: https://docs.nimbleway.com/nimble-sdk/getting-started/quickstart This guide will help you make your first request with Nimble in just a few minutes. First, create your nimble account at [Nimble Sign Up](https://online.nimbleway.com/signup). Already have an account? Just login to [Nimble dashboard](https://online.nimbleway.com/signup). Now, obtain your API key from the [Account Settings](https://online.nimbleway.com/account-settings/api-keys). Install the Nimble SDK in your preferred language: ```bash Python theme={"system"} pip install nimble_python ``` ```bash Node theme={"system"} npm install @nimble-way/sdk ``` **Prefer using the API directly?** Skip the SDK installation and use HTTP requests (see cURL examples below) or visit the [API Reference](/api-reference/introduction) for complete endpoint documentation. **Extract** Get clean HTML and structured data from any URL: ```python Python theme={"system"} from nimble_python import Nimble nimble = Nimble(api_key="YOUR-API-KEY") result = nimble.extract( url= "https://www.example.com", render= True ) print(result) ``` ```javascript Node theme={"system"} import Nimble from '@nimble-way/nimble-js'; const nimble = new Nimble({ apiKey: "YOUR-API-KEY" }); const result = await nimble.extract({ url: "https://www.example.com", render: true }); console.log(result); ``` ```bash cURL theme={"system"} curl -X POST 'https://sdk.nimbleway.com/v1/extract' \ --header 'Authorization: Bearer ' \ --header 'Content-Type: application/json' \ --data-raw '{ "url": "https://www.example.com", "render": true }' ``` **Web Search Agents** Use pre-built templates for popular platforms: ```python Python theme={"system"} from nimble_python import Nimble nimble = Nimble(api_key="YOUR-API-KEY") result = nimble.agent.run( agent= "amazon_pdp", params= { "asin": "B08N5WRWNW" } ) print(result) ``` ```typescript Node theme={"system"} import Nimble from '@nimble-way/nimble-js'; const nimble = new Nimble({ apiKey: "YOUR-API-KEY" }); const result = await nimble.agent.run({ agent: "amazon_pdp", params: { asin: "B08N5WRWNW" } }); console.log(result); ``` ```bash cURL theme={"system"} curl -X POST 'https://sdk.nimbleway.com/v1/agents/run' \ --header 'Authorization: Bearer ' \ --header 'Content-Type: application/json' \ --data-raw '{ "agent": "amazon_pdp", "params": { "asin": "B08N5WRWNW" } }' ``` **Search** Perform web searches and get structured results: ```python Python theme={"system"} from nimble_python import Nimble nimble = Nimble(api_key="YOUR-API-KEY") result = nimble.search( query= "data scraping tools", country= "US" ) print(result) ``` ```javascript Node theme={"system"} import Nimble from '@nimble-way/nimble-js'; const nimble = new Nimble({ apiKey: "YOUR-API-KEY" }); const result = await nimble.search({ query: "data scraping tools", country: "US" }); console.log(result); ``` ```bash cURL theme={"system"} curl -X POST 'https://sdk.nimbleway.com/v1/search' \ --header 'Authorization: Bearer ' \ --header 'Content-Type: application/json' \ --data-raw '{ "query": "data scraping tools", "country": "US" }' ``` **Map** Fast URL discovery and site structure mapping: ```python Python theme={"system"} from nimble_python import Nimble nimble = Nimble(api_key="YOUR-API-KEY") result = nimble.map( URL= "https://www.example.com", sitemap= "only" ) print(result) ``` ```javascript Node theme={"system"} import Nimble from '@nimble-way/nimble-js'; const nimble = new Nimble({ apiKey: "YOUR-API-KEY" }); const result = await nimble.map({ URL: "https://www.example.com", sitemap: "only" }); console.log(result); ``` ```bash cURL theme={"system"} curl -X POST 'https://sdk.nimbleway.com/v1/map' \ --header 'Authorization: Bearer ' \ --header 'Content-Type: application/json' \ --data-raw '{ "URL": "https://www.example.com", "sitemap": "only" }' ``` **Crawl** Discover and map entire websites automatically: ```python Python theme={"system"} from nimble_python import Nimble nimble = Nimble(api_key="YOUR-API-KEY") result = nimble.crawl.run( url= "https://www.example.com", limit= 100 ) print(result) ``` ```javascript Node theme={"system"} import Nimble from '@nimble-way/nimble-js'; const nimble = new Nimble({ apiKey: "YOUR-API-KEY" }); const result = await nimble.crawl.run({ url: "https://www.example.com", limit: 100 }); console.log(result); ``` ```bash cURL theme={"system"} curl -X POST 'https://sdk.nimbleway.com/v1/crawl' \ --header 'Authorization: Bearer ' \ --header 'Content-Type: application/json' \ --data-raw '{ "url": "https://www.example.com", "limit": 100 }' ``` **Ready to build something real?** End-to-end tutorial: use the Nimble Agents skill to build a competitive analysis script for shoes across Amazon, Walmart, and Nike Explore the **Web Tools** documentation: Get clean & structured data from any URL Ready-to-use extraction templates for popular platforms Perform web searches and retrieve parsed content from top results Fast URL discovery and site structure mapping Map and Extract content from entire websites at scale Access premium residential IPs with Nimble's Proxy Infrastructure ([Contact](https://portal.usepylon.com/nimble) sales to gain access) Use the **API** directly, bring Nimble into your **AI coding assistant**, or connect via **MCP**: Full API documentation Plug-and-play skills for Claude Code, Cursor, and other AI coding assistants Connect AI agents and 3rd party tools to Nimble's web data infrastructure Install an **SDK** and start building in your language of choice: `nimble_python` `@nimbleway/sdk` `nimble-go` `nimble-cli` Manage your **account**, review **pricing**, and monitor **service health**: Manage API keys and collaborate with your team Usage-based pricing for all Nimble APIs and Data Services plans Real-time status of all Nimble services # CLI Source: https://docs.nimbleway.com/nimble-sdk/sdks/cli Official Nimble CLI — interact with the Nimble API directly from your terminal The Nimble CLI lets you run extractions, searches, crawls, and agents directly from your terminal — no code required. ## Installation The CLI requires **Go 1.22+**. ```bash theme={"system"} go install 'github.com/Nimbleway/nimble-cli/cmd/nimble@latest' ``` The binary installs to `$HOME/go/bin`. Add it to your `PATH` if needed: ```bash theme={"system"} export PATH="$PATH:$(go env GOPATH)/bin" ``` Verify the installation: ```bash theme={"system"} nimble --version ``` ## Authentication Set your API key as an environment variable: ```bash theme={"system"} export NIMBLE_API_KEY="your-api-key" ``` Get your API key from [Account Settings](https://online.nimbleway.com/account-settings/api-keys). ## Command Structure All commands follow a resource-based pattern: ```bash theme={"system"} nimble [resource] [flags...] ``` Run `nimble --help` to list all resources, or `nimble [resource] --help` for resource-specific usage. ## Core Commands ### Extract Extract clean content from any URL: ```bash theme={"system"} nimble extract --url "https://www.example.com" ``` With JavaScript rendering and geo-targeting: ```bash theme={"system"} nimble extract \ --url "https://www.example.com" \ --render true \ --country US ``` ### Search Perform a real-time web search: ```bash theme={"system"} nimble search --query "best web scraping tools" --country US ``` ### Map Discover all URLs in a domain: ```bash theme={"system"} nimble map --url "https://www.example.com" --sitemap only ``` ### Crawl Crawl an entire website: ```bash theme={"system"} nimble crawl run --url "https://www.example.com" --limit 100 ``` ### Agents Run a pre-built agent: ```bash theme={"system"} nimble agent run --agent amazon_pdp --params '{"asin": "B08N5WRWNW"}' ``` ## Output Formats Control response formatting with the `--format` flag: | Format | Description | | --------- | -------------------------------- | | `auto` | Automatic (default) | | `pretty` | Pretty-printed JSON | | `json` | Compact JSON | | `jsonl` | JSON Lines (one object per line) | | `yaml` | YAML output | | `raw` | Raw API response body | | `explore` | Interactive explorer view | ```bash theme={"system"} nimble extract --url "https://www.example.com" --format pretty ``` ## Transforming Output Use `--transform` with [GJSON syntax](https://github.com/tidwall/gjson) to extract specific fields: ```bash theme={"system"} # Extract just the task_id nimble extract --url "https://www.example.com" --transform "task_id" # Extract a nested field nimble extract --url "https://www.example.com" --transform "data.html" ``` ## File Arguments Pass file contents as argument values using `@filename` syntax: ```bash theme={"system"} # Pass a JSON schema file nimble extract --url "https://www.example.com" --parsing-schema @schema.json # Pass a file inside a JSON structure nimble extract --options '{"parsing_schema": "@schema.json"}' ``` Use `@file://filename` for plain text or `@data://filename` for base64-encoded content. ## Global Flags | Flag | Description | | ------------------- | ------------------------------------------- | | `--help` | Show help for any command | | `--debug` | Enable debug logging with full HTTP details | | `--version`, `-v` | Display the CLI version | | `--base-url` | Override the API base URL | | `--format` | Set output format | | `--format-error` | Set error output format | | `--transform` | Transform output using GJSON syntax | | `--transform-error` | Transform error output using GJSON syntax | ## Debug Mode Enable `--debug` to inspect the full HTTP request and response — useful for troubleshooting: ```bash theme={"system"} nimble extract --url "https://www.example.com" --debug ``` ## Next Steps Full REST API documentation for all endpoints Extract features and parameters Real-time web search with structured results Pre-built extraction agents for popular platforms # Go Source: https://docs.nimbleway.com/nimble-sdk/sdks/go Official Nimble SDK for Go — type-safe client for web extraction, search, crawl, map, and agents Install the Nimble Go library to interact with the Nimble API from your Go applications. Built with idiomatic Go patterns, full context support, and automatic retries. ## Installation ```sh theme={"system"} go get github.com/Nimbleway/nimble-go@latest' ``` Requires **Go 1.22+**. Set your API key via the `NIMBLE_API_KEY` environment variable or pass it directly using `option.WithAPIKey()`. ## Setup Import the package and initialize the client: ```go Go theme={"system"} package main import ( "context" "fmt" "os" nimble "github.com/Nimbleway/nimble-go" "github.com/Nimbleway/nimble-go/option" ) func main() { client := nimble.NewClient( option.WithAPIKey(os.Getenv("NIMBLE_API_KEY")), ) _ = client fmt.Println("Client ready") } ``` ## Quick Start Extract content from a URL: ```go Go theme={"system"} package main import ( "context" "fmt" "os" nimble "github.com/Nimbleway/nimble-go" "github.com/Nimbleway/nimble-go/option" ) func main() { client := nimble.NewClient( option.WithAPIKey(os.Getenv("NIMBLE_API_KEY")), ) result, err := client.Extract(context.TODO(), nimble.ExtractParams{ URL: "https://www.example.com", }) if err != nil { panic(err) } fmt.Printf("Task ID: %s\n", result.TaskID) } ``` ## Core Methods ### Extract Get clean HTML, markdown, or structured data from any URL: ```go Go theme={"system"} result, err := client.Extract(context.TODO(), nimble.ExtractParams{ URL: "https://www.example.com", Render: nimble.Bool(true), }) if err != nil { panic(err) } fmt.Println(result.TaskID) ``` Optional fields are wrapped in `param.Opt[T]`. Use the provided constructors — `nimble.Bool()`, `nimble.String()`, `nimble.Int()` — to set them. ### Search Perform real-time web searches: ```go Go theme={"system"} result, err := client.Search(context.TODO(), nimble.SearchParams{ Query: "best web scraping tools", Country: nimble.String("US"), }) if err != nil { panic(err) } fmt.Println(result) ``` ### Map Discover all URLs within a domain or sitemap: ```go Go theme={"system"} result, err := client.Map(context.TODO(), nimble.MapParams{ URL: "https://www.example.com", Sitemap: nimble.String("only"), }) if err != nil { panic(err) } fmt.Println(result) ``` ### Crawl Recursively crawl and extract an entire website: ```go Go theme={"system"} result, err := client.Crawl.Run(context.TODO(), nimble.CrawlRunParams{ URL: "https://www.example.com", Limit: nimble.Int(100), }) if err != nil { panic(err) } fmt.Println(result) ``` ### Agents Run pre-built agents for structured data from popular platforms: ```go Go theme={"system"} result, err := client.Agent.Run(context.TODO(), nimble.AgentRunParams{ Agent: "amazon_pdp", Params: map[string]interface{}{ "asin": "B08N5WRWNW", }, }) if err != nil { panic(err) } fmt.Println(result) ``` Browse the full agent catalog in the [Agent Gallery](/nimble-sdk/agentic/agent-gallery). ## Error Handling Check `err != nil` for any failed request. Use `errors.As` to inspect typed API errors: ```go Go theme={"system"} import ( "errors" "fmt" nimble "github.com/Nimbleway/nimble-go" ) result, err := client.Extract(context.TODO(), nimble.ExtractParams{ URL: "https://www.example.com", }) if err != nil { var apiErr *nimble.Error if errors.As(err, &apiErr) { fmt.Printf("API error — status: %d, message: %s\n", apiErr.StatusCode, apiErr.Message) } panic(err) } ``` The `*nimble.Error` type exposes: * `StatusCode int` — HTTP status code * `Message string` — Human-readable error description ## Configuration ### Retries The SDK retries automatically on connection errors, timeouts, and server failures (2 retries by default): ```go Go theme={"system"} // Configure globally client := nimble.NewClient( option.WithAPIKey(os.Getenv("NIMBLE_API_KEY")), option.WithMaxRetries(3), ) // Disable retries for a specific request result, err := client.Extract(context.TODO(), nimble.ExtractParams{ URL: "https://www.example.com", }, option.WithMaxRetries(0)) ``` ### Timeouts Use Go's standard `context` package for request timeouts: ```go Go theme={"system"} import ( "context" "time" ) ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second) defer cancel() result, err := client.Extract(ctx, nimble.ExtractParams{ URL: "https://www.example.com", }) ``` Or set a per-request timeout directly: ```go Go theme={"system"} result, err := client.Extract(context.TODO(), nimble.ExtractParams{ URL: "https://www.example.com", }, option.WithRequestTimeout(30*time.Second)) ``` ## Advanced ### Raw Response Access Capture the raw HTTP response alongside the parsed result: ```go Go theme={"system"} import "net/http" var rawResponse *http.Response result, err := client.Extract(context.TODO(), nimble.ExtractParams{ URL: "https://www.example.com", }, option.WithResponseInto(&rawResponse)) if err != nil { panic(err) } fmt.Println(rawResponse.Header.Get("x-request-id")) fmt.Println(result.TaskID) ``` ### Custom Request Options Attach additional headers or query parameters to any request: ```go Go theme={"system"} result, err := client.Extract(context.TODO(), nimble.ExtractParams{ URL: "https://www.example.com", }, option.WithHeader("X-Custom-Header", "value"), option.WithQueryParam("debug", "true"), ) ``` ## Next Steps Full REST API documentation for all endpoints Rendering, formats, stealth mode, and more Real-time web search with structured results Pre-built extraction agents for popular platforms # Node Source: https://docs.nimbleway.com/nimble-sdk/sdks/node Official Nimble SDK for Node.js and TypeScript — async client for web extraction, search, crawl, map, and agents Install the Nimble JavaScript/TypeScript library to interact with the Nimble API from Node.js, Deno, Bun, Cloudflare Workers, and modern browsers. All methods are async and return typed responses. ## Installation ```bash theme={"system"} npm install @nimble-way/sdk ``` Requires **Node.js 20 LTS+** (also supports Deno v1.28+, Bun 1.0+, and Cloudflare Workers). Never hardcode your API key — use environment variables. ## Setup Initialize the client with your API key: ```typescript TypeScript theme={"system"} import Nimble from "@nimble-way/nimble-js"; const nimble = new Nimble({ apiKey: process.env.NIMBLE_API_KEY, }); ``` ## Quick Start Extract content from a URL in one call: ```typescript TypeScript theme={"system"} import Nimble from "@nimble-way/nimble-js"; const nimble = new Nimble({ apiKey: process.env.NIMBLE_API_KEY }); const result = await nimble.extract({ url: "https://www.example.com", render: true, }); console.log(result.task_id); ``` ## TypeScript Types The SDK ships full TypeScript definitions for all request parameters and responses, with IDE autocompletion: ```typescript TypeScript theme={"system"} import Nimble from "@nimble-way/nimble-js"; const nimble = new Nimble({ apiKey: process.env.NIMBLE_API_KEY }); const params: Nimble.ExtractParams = { url: "https://www.example.com", render: true, }; const result: Nimble.ExtractResponse = await nimble.extract(params); console.log(result.task_id); ``` ## Core Methods ### Extract Get clean HTML, markdown, or structured data from any URL. Supports JavaScript rendering, stealth mode, browser actions, and more. ```typescript TypeScript theme={"system"} const result = await nimble.extract({ url: "https://www.example.com", render: true, }); console.log(result.task_id); ``` See the [Extract docs](/nimble-sdk/web-tools/extract/quickstart) for the full parameter list — geo-targeting, output formats, parsing schemas, and more. ### Search Perform real-time web searches and get structured results: ```typescript TypeScript theme={"system"} const result = await nimble.search({ query: "best web scraping tools", country: "US", }); console.log(result); ``` ### Map Discover all URLs within a domain or sitemap: ```typescript TypeScript theme={"system"} const result = await nimble.map({ URL: "https://www.example.com", sitemap: "only", }); console.log(result); ``` ### Crawl Recursively crawl and extract an entire website at scale: ```typescript TypeScript theme={"system"} const result = await nimble.crawl.run({ url: "https://www.example.com", limit: 100, }); console.log(result); ``` ### Agents Run pre-built agents for structured data from popular platforms: ```typescript TypeScript theme={"system"} const result = await nimble.agent.run({ agent: "amazon_pdp", params: { asin: "B08N5WRWNW", }, }); console.log(result); ``` Browse the full agent catalog in the [Agent Gallery](/nimble-sdk/agentic/agent-gallery). ## Error Handling The SDK throws typed `APIError` subclasses for failed requests: ```typescript TypeScript theme={"system"} import Nimble, { APIError } from "nimble-js"; const nimble = new Nimble({ apiKey: process.env.NIMBLE_API_KEY }); try { const result = await nimble.extract({ url: "https://www.example.com" }); } catch (error) { if (error instanceof APIError) { console.log(`Status: ${error.status}`); console.log(`Message: ${error.message}`); } throw error; } ``` | Error Class | HTTP Status | When it occurs | | ----------------------- | ----------- | -------------------------------- | | `BadRequestError` | 400 | Invalid request parameters | | `AuthenticationError` | 401 | Invalid or missing API key | | `PermissionDeniedError` | 403 | Insufficient account permissions | | `NotFoundError` | 404 | Resource not found | | `RateLimitError` | 429 | Request rate exceeded | | `InternalServerError` | 500+ | Server-side error | ## Configuration ### Timeouts & Retries The SDK retries automatically on connection errors and server failures. Defaults: **2 retries**, **3-minute timeout**. ```typescript TypeScript theme={"system"} // Configure globally const nimble = new Nimble({ apiKey: process.env.NIMBLE_API_KEY, maxRetries: 3, timeout: 60 * 1000, // 60 seconds in ms }); // Override per-request const result = await nimble.extract( { url: "https://www.example.com" }, { maxRetries: 0 }, ); ``` ### Logging Control log verbosity via environment variable: ```bash theme={"system"} export NIMBLE_LOG=debug ``` Or configure programmatically using any compatible logger (pino, winston, consola, etc.): ```typescript TypeScript theme={"system"} import Nimble from "@nimble-way/nimble-js"; import pino from "pino"; const nimble = new Nimble({ apiKey: process.env.NIMBLE_API_KEY, logger: pino(), logLevel: "debug", }); ``` ## Advanced ### Raw Response Access Access HTTP headers and status alongside the parsed response: ```typescript TypeScript theme={"system"} // Get just the raw Response object const raw = await nimble .extract({ url: "https://www.example.com" }) .asResponse(); console.log(raw.headers.get("x-request-id")); // Get both parsed result and raw response const { data, response } = await nimble .extract({ url: "https://www.example.com" }) .withResponse(); console.log(response.headers.get("x-request-id")); console.log(data.task_id); ``` ### Custom Fetch / Proxy Provide a custom fetch implementation or configure proxies for Node.js, Bun, and Deno: ```typescript TypeScript theme={"system"} import Nimble from "@nimble-way/nimble-js"; const nimble = new Nimble({ apiKey: process.env.NIMBLE_API_KEY, fetchOptions: { proxy: "http://proxy.example.com:8080", }, }); ``` ### MCP Server The SDK includes a built-in [MCP Server](https://modelcontextprotocol.io) for AI assistant integrations, enabling API exploration and request testing directly from tools like Claude. ## Next Steps Full REST API documentation for all endpoints Rendering, formats, stealth mode, and more Real-time web search with structured results Pre-built extraction agents for popular platforms # Python Source: https://docs.nimbleway.com/nimble-sdk/sdks/python Official Nimble Python SDK — sync and async client for web extraction, search, crawl, map, and agents Install the Nimble Python library to interact with the Nimble API from your Python applications. Supports both synchronous and asynchronous usage with full type hints. ## Installation ```bash pip theme={"system"} pip install nimble_python ``` ```bash pip (async HTTP backend) theme={"system"} pip install "nimble_python[aiohttp]" ``` Requires **Python 3.9+**. Never hardcode your API key — use environment variables instead. ## Setup Initialize the client with your API key. We recommend loading it from the environment: ```python Python theme={"system"} import os from nimble_python import Nimble client = Nimble(api_key=os.environ.get("NIMBLE_API_KEY")) ``` ## Quick Start Extract clean content from a URL in one call: ```python Python theme={"system"} import os from nimble_python import Nimble client = Nimble(api_key=os.environ.get("NIMBLE_API_KEY")) result = client.extract(url="https://www.example.com", render=True) print(result.task_id) ``` ## Core Methods ### Extract Get clean HTML, markdown, or structured data from any URL. Supports JavaScript rendering, stealth mode, browser actions, and more. ```python Python theme={"system"} result = client.extract( url="https://www.example.com", render=True ) print(result.task_id) ``` See the [Extract docs](/nimble-sdk/web-tools/extract/quickstart) for the full parameter list — geo-targeting, output formats, parsing schemas, and more. ### Search Perform real-time web searches and get structured results: ```python Python theme={"system"} result = client.search( query="best web scraping tools", country="US" ) print(result) ``` ### Map Discover all URLs within a domain or sitemap: ```python Python theme={"system"} result = client.map( URL="https://www.example.com", sitemap="only" ) print(result) ``` ### Crawl Recursively crawl and extract an entire website at scale: ```python Python theme={"system"} result = client.crawl.run( url="https://www.example.com", limit=100 ) print(result) ``` ### Agents Run pre-built agents for structured data from popular platforms: ```python Python theme={"system"} result = client.agent.run( agent="amazon_pdp", params={"asin": "B08N5WRWNW"} ) print(result) ``` Browse the full agent catalog in the [Agent Gallery](/nimble-sdk/agentic/agent-gallery). ## Async Client Use `AsyncNimble` for non-blocking, concurrent operations: ```python Python theme={"system"} import asyncio import os from nimble_python import AsyncNimble client = AsyncNimble(api_key=os.environ.get("NIMBLE_API_KEY")) async def main(): result = await client.extract(url="https://www.example.com", render=True) print(result.task_id) asyncio.run(main()) ``` ### aiohttp Backend For high-concurrency workloads, use the `aiohttp` HTTP backend (requires `pip install "nimble_python[aiohttp]"`): ```python Python theme={"system"} import asyncio import os from nimble_python import AsyncNimble, DefaultAioHttpClient async def main(): async with AsyncNimble( api_key=os.environ.get("NIMBLE_API_KEY"), http_client=DefaultAioHttpClient(), ) as client: result = await client.extract(url="https://www.example.com", render=True) print(result.task_id) asyncio.run(main()) ``` Always use `async with` when working with `DefaultAioHttpClient` to ensure proper resource cleanup. ## Error Handling The SDK maps API failures to typed exception classes: ```python Python theme={"system"} import nimble_python import os from nimble_python import Nimble client = Nimble(api_key=os.environ.get("NIMBLE_API_KEY")) try: result = client.extract(url="https://www.example.com") except nimble_python.APIConnectionError as e: print("Connection failed:", e) except nimble_python.AuthenticationError as e: print("Invalid API key — check NIMBLE_API_KEY") except nimble_python.RateLimitError as e: print("Rate limited — back off and retry:", e) except nimble_python.APIStatusError as e: print(f"API error {e.status_code}:", e.message) ``` | Exception | HTTP Status | When it occurs | | ----------------------- | ----------- | -------------------------------- | | `APIConnectionError` | — | Network failure or timeout | | `AuthenticationError` | 401 | Invalid or missing API key | | `PermissionDeniedError` | 403 | Insufficient account permissions | | `RateLimitError` | 429 | Request rate exceeded | | `InternalServerError` | 500+ | Server-side error | ## Configuration ### Timeouts & Retries The SDK retries automatically on connection errors and server failures. Defaults: **2 retries**, **3-minute timeout**. ```python Python theme={"system"} import os from nimble_python import Nimble # Configure globally client = Nimble( api_key=os.environ.get("NIMBLE_API_KEY"), max_retries=3, timeout=60.0, # seconds ) # Override per-request result = client.with_options(max_retries=0).extract( url="https://www.example.com" ) ``` ### Logging Enable SDK-level debug logging via environment variable: ```bash theme={"system"} export NIMBLE_LOG=debug ``` Or configure programmatically: ```python Python theme={"system"} import logging logging.basicConfig() logging.getLogger("nimble_python").setLevel(logging.DEBUG) ``` ## Advanced ### Raw Response Access Access HTTP headers and response metadata alongside the parsed result: ```python Python theme={"system"} response = client.with_raw_response.extract(url="https://www.example.com") print(response.headers.get("x-request-id")) result = response.parse() print(result.task_id) ``` ### Streaming Response Stream large response bodies instead of buffering them in memory: ```python Python theme={"system"} with client.with_streaming_response.extract(url="https://www.example.com") as response: for chunk in response.iter_bytes(): process(chunk) ``` ## Next Steps Full REST API documentation for all endpoints Rendering, formats, stealth mode, and more Real-time web search with structured results Pre-built extraction agents for popular platforms # Agent Skills Source: https://docs.nimbleway.com/nimble-sdk/sdks/skills Nimble agent skills for Claude Code, Cursor, and Vercel — bring real-time web data into your AI workflows ## Prerequisites **1. Nimble CLI** — Required by the `nimble-web-tools` skill: ```bash theme={"system"} npm install -g @nimble-way/nimble-cli ``` **2. Nimble API Key** — Get yours from [Account Settings](https://online.nimbleway.com/account-settings/api-keys) and set it as an environment variable: ```bash theme={"system"} export NIMBLE_API_KEY="your-api-key" ``` Or persist it in `~/.claude/settings.json` for Claude Code: ```json theme={"system"} { "env": { "NIMBLE_API_KEY": "your-api-key" } } ``` ## Installation Install from the Nimble Marketplace — this adds both skills and configures the MCP server automatically: ```bash theme={"system"} claude plugin marketplace add Nimbleway/agent-skills && claude plugin install nimble@nimble-plugin-marketplace ``` Or load directly from a local clone: ```bash theme={"system"} git clone https://github.com/Nimbleway/agent-skills.git && claude --plugin-dir /path/to/agent-skills ``` **Step 1:** Add the Nimble MCP server to Cursor: Add to Cursor Replace `NIMBLE_API_KEY` in **Cursor Settings > MCP Servers** with your actual key. **Step 2:** Install skills and rules: ```bash theme={"system"} npx skills add Nimbleway/agent-skills -a cursor ``` ```bash theme={"system"} npx skills add Nimbleway/agent-skills ``` The `nimble-agents` skill requires the MCP server. Connect it manually after installing: ```bash theme={"system"} claude mcp add --transport http nimble-mcp-server https://mcp.nimbleway.com/mcp \ --header "Authorization: Bearer ${NIMBLE_API_KEY}" ``` *** ## What's Included Agent Skills are plug-and-play extensions that give AI coding assistants direct access to Nimble's web data tools. Install once and your AI can search the live web, extract pages, map sites, and run structured data agents — automatically, from natural language. | Skill | Description | | -------------------- | ------------------------------------------------------------------------------- | | **nimble-web-tools** | Real-time web intelligence — search, extract, map, and crawl via the Nimble CLI | | **nimble-agents** | Find, generate, and run agents to extract structured data from any website | | **MCP Server** | Pre-configured Nimble MCP server connection for agent-to-API access | ## nimble-web-tools Skill The `nimble-web-tools` skill activates automatically when you ask your AI assistant to search the web, extract a page, map a site, or crawl for content. ### What it can do | Tool | What it does | | ----------- | --------------------------------------------------------------------------------------------------------- | | **Search** | Real-time web search with 8 focus modes: general, coding, news, academic, shopping, social, geo, location | | **Extract** | Get clean HTML or markdown from any URL — supports JS rendering and stealth unblocking | | **Map** | Discover all URLs in a domain or sitemap — useful for planning extraction workflows | | **Crawl** | Extract content from an entire website in one request | ### Example prompts ``` "Search for recent AI developments" → nimble search --query "recent AI developments" --deep-search=false "Extract the content from this URL" → nimble extract --url "https://example.com" --parse --format markdown "Map all pages on this docs site" → nimble map --url "https://docs.example.com" --limit 100 "Crawl the API reference section" → nimble crawl run --url "https://docs.example.com/api" --limit 50 ``` ### Search ```bash theme={"system"} # Fast search (always use --deep-search=false for speed) nimble search --query "React hooks tutorial" --topic coding --deep-search=false # Search with AI-generated answer nimble search --query "what is WebAssembly" --include-answer --deep-search=false # News with time filter nimble search --query "AI developments" --topic news --time-range week --deep-search=false # Filter to specific domains nimble search --query "auth best practices" \ --include-domain github.com \ --include-domain stackoverflow.com \ --deep-search=false ``` Use `--deep-search=false` for fast responses (1–3s). The default deep mode fetches full page content and is 5–10× slower — only needed for full-text archiving. ### Extract ```bash theme={"system"} # Standard extraction — always pass --parse --format markdown for LLM-readable output nimble extract --url "https://example.com/article" --parse --format markdown # With JavaScript rendering (for SPAs and dynamic pages) nimble extract --url "https://example.com/app" --render --parse --format markdown # With geo-targeting nimble extract --url "https://example.com" --country US --city "New York" --parse --format markdown ``` ### Map ```bash theme={"system"} # Discover all URLs on a site nimble map --url "https://docs.example.com" --limit 100 # Include subdomains nimble map --url "https://example.com" --domain-filter subdomains ``` ### Crawl ```bash theme={"system"} # Crawl a site (always set --limit) nimble crawl run --url "https://docs.example.com" --limit 50 # Filter to specific paths nimble crawl run --url "https://example.com" \ --include-path "/docs" \ --include-path "/api" \ --limit 100 # Check crawl status nimble crawl status --id "crawl-id" ``` For LLM-friendly output, prefer `map` + `extract --parse --format markdown` on individual pages rather than crawl — crawl returns raw HTML which can be very large. *** ## nimble-agents Skill The `nimble-agents` skill gives your AI assistant access to Nimble's pre-built agent library. It uses a find-or-generate workflow: search for an existing agent, run it, and get structured data. If no agent fits, generate a custom one via natural language. ### How it works Find an existing agent matching your target website using `nimble_agents_list` Review the agent's input/output schema with `nimble_agents_get` Execute the agent and get clean, structured results via `nimble_agents_run` If no existing agent fits, create a custom one with `nimble_agents_generate` Save generated agents for future reuse with `nimble_agents_publish` ### Usage Use the slash command: ``` /nimble:nimble-agents extract product details from this Amazon page: https://www.amazon.com/dp/B0DGHRT7PS ``` Or just describe what you need in plain language — the skill activates automatically when relevant. Reference the skill in Cursor Agent chat: ``` /nimble-agents extract product details from this Amazon page: https://www.amazon.com/dp/B0DGHRT7PS ``` ### MCP Tools | Tool | Purpose | | ------------------------ | ------------------------------------------- | | `nimble_agents_list` | Browse agents by keyword | | `nimble_agents_get` | Get agent details and input/output schema | | `nimble_agents_generate` | Create a custom agent via natural language | | `nimble_agents_run` | Execute an agent and get structured results | | `nimble_agents_publish` | Save a generated agent for reuse | *** ## Source Code Plugin source, skill definitions, and SKILL.md references Detailed platform-specific installation guide Full command reference for nimble-web-tools Full guide for the nimble-agents skill # Crawl Source: https://docs.nimbleway.com/nimble-sdk/web-tools/crawl Extract structured data from entire websites automatically Nimble **Crawl** (async) systematically visits and extracts content from entire websites. Give it a starting URL, and it automatically discovers pages through sitemaps and internal links, then extracts clean structured data from every page it visits. ## Quick Start ### Example Request ```python Python theme={"system"} from nimble_python import Nimble nimble = Nimble(api_key="YOUR-API-KEY") result = nimble.crawl.run( url="https://www.nimbleway.com", limit=10 ) print(f"Crawl started with ID: {result.crawl_id}") ``` ```javascript Node theme={"system"} import Nimble from "@nimble-way/nimble-js"; const nimble = new Nimble({ apiKey: "YOUR-API-KEY" }); const result = await nimble.crawl.run({ url: "https://www.nimbleway.com", limit: 10, }); console.log(`Crawl started with ID: ${result.crawl_id}`); ``` ```bash cURL theme={"system"} curl -X POST 'https://sdk.nimbleway.com/v1/crawl' \ --header 'Authorization: Bearer ' \ --header 'Content-Type: application/json' \ --data-raw '{ "url": "https://www.nimbleway.com", "limit": 10 }' ``` ### Example Response ```json theme={"system"} { "crawl_id": "e3ca2ff1-b82a-472b-b1a9-ef4d29cc549f", "name": null, "url": "https://www.nimbleway.com", "status": "queued", "account_name": "your-account", "created_at": "2026-02-09T23:15:40.785Z", "updated_at": "2026-02-09T23:15:40.785Z", "completed_at": null, "crawl_options": { "sitemap": "include", "crawl_entire_domain": false, "limit": 10, "max_discovery_depth": 5, "ignore_query_parameters": false, "allow_external_links": false, "allow_subdomains": false }, "extract_options": null } ``` ## How it works Provide a starting URL and configure crawl options (limits, filters, extraction settings) * Returns immediately with a `crawl_id` to track progress - The crawl runs in the background on Nimble's infrastructure - Optional: Configure webhooks to receive real-time notifications * Reads sitemaps and follows internal links - Creates individual tasks for each discovered URL - Extracts content from pages as they're visited - Status updates live: track `pending`, `completed`, and `failed` counts * Poll crawl status to monitor progress - Fetch extracted content for completed tasks using `task_id` - Results persist after crawl completes for later retrieval ## Parameters Supported input parameters: The starting point for your crawl. The crawler will begin here and discover other pages from this URL. **Example:** `https://www.nimbleway.com` Give your crawl a memorable name. This helps you identify it later when you have multiple crawls running. **Example:** `my-zillow-crawl` Stop the crawl after finding this many pages. * Min: `1` * Max: `10000` * Default: `5000` Automatically extract content from each page as you crawl it. Accepts all [Extract API](/nimble-sdk/web-tools/extract/quickstart) options. \*\*Example: \*\* ```json theme={"system"} { "extract_options":{ "driver":"vx10", "parse":true, "formats": ["html", "markdown"] } } ``` Decide how to use the website's sitemap for discovering pages. **Options:** * `include` (default) - Use both the sitemap and discovered links * `only` - Just use the sitemap (fastest) * `skip` - Ignore the sitemap and only follow links Let the crawler explore the entire domain, not just pages "under" your starting URL. For example, if you start at `/blog`, enabling this lets it also crawl `/about` and `/contact`. Allow the crawler to follow links to subdomains. For example, from `www.example.com` to `blog.example.com` or `shop.example.com`. Only crawl pages whose URLs match these regex patterns. **Example:** `["/blog/.*", "/articles/.*"]` Skip pages whose URLs match these regex patterns. **Example:** `[".*/tag/.*", ".*/page/[0-9]+"]` Control how many "clicks away" from the starting page the crawler can go. * Min: `1` * Max: `20` * Default: `5` Treat URLs with different query parameters as the same page, preventing duplicate crawls. Get notified when your crawl completes or as pages are discovered. **Configuration:** * `url` (required) - String | Webhook URL to receive notifications * `headers` - Object | Custom headers for authentication * `metadata` - Object | Extra data to include in payloads * `events` - Array | Which events trigger notifications: `started`, `page`, `completed`, `failed` **Example:** ```json theme={"system"} { "callback":{ "url":"", "headers":{}, "metadata":{}, "events":["started", "completed"] } } ``` Crawl the site as if you're browsing from a specific country. Use ISO Alpha-2 country codes like `US`, `GB`, `FR`, `DE`, `CA`, `JP`, etc. Use `ALL` for random country selection. Set the language preference for crawling. Use LCID standard. **Locale Examples:** * `en-US` - English (United States) * `en-GB` - English (United Kingdom) * `fr-FR` - French (France) * `de-DE` - German (Germany) ## Usage ### Basic crawl Crawl a website using default settings: ```python Python theme={"system"} from nimble_python import Nimble nimble = Nimble(api_key="YOUR-API-KEY") result = nimble.crawl.run(url="https://www.nimbleway.com") print(f"Crawl started with ID: {result.crawl_id}") print(f"Status: {result.status}") ``` ```javascript Node theme={"system"} import Nimble from "@nimble-way/nimble-js"; const nimble = new Nimble({ apiKey: "YOUR-API-KEY" }); const result = await nimble.crawl.run({ url: "https://www.nimbleway.com", }); console.log(`Crawl started with ID: ${result.crawl_id}`); console.log(`Status: ${result.status}`); ``` ```bash cURL theme={"system"} curl -X POST 'https://sdk.nimbleway.com/v1/crawl' \ --header 'Authorization: Bearer ' \ --header 'Content-Type: application/json' \ --data-raw '{ "url": "https://www.nimbleway.com" }' ``` ### Filter with URL patterns Use include and exclude patterns to control which URLs are crawled: ```python Python theme={"system"} from nimble_python import Nimble nimble = Nimble(api_key="YOUR-API-KEY") result = nimble.crawl.run( url="https://www.nimbleway.com", include_paths=["/blog/.*", "/use-cases/.*"], exclude_paths=[".*/careers/.*"], limit=500 ) print(f"Crawl ID: {result.crawl_id}") print(f"Status: {result.status}") ``` ```javascript Node theme={"system"} import Nimble from "@nimble-way/nimble-js"; const nimble = new Nimble({ apiKey: "YOUR-API-KEY" }); const result = await nimble.crawl.run({ url: "https://www.nimbleway.com", include_paths: ["/blog/.*", "/use-cases/.*"], exclude_paths: [".*/careers/.*"], limit: 500, }); console.log(`Crawl ID: ${result.crawl_id}`); console.log(`Status: ${result.status}`); ``` ```bash cURL theme={"system"} curl -X POST 'https://sdk.nimbleway.com/v1/crawl' \ --header 'Authorization: Bearer ' \ --header 'Content-Type: application/json' \ --data-raw '{ "url": "https://www.nimbleway.com", "include_paths": ["/blog/.*", "/use-cases/.*"], "exclude_paths": [".*/careers/.*"], "limit": 500 }' ``` ### Crawl entire domain Allow crawler to follow all internal links beyond the starting path: ```python Python theme={"system"} from nimble_python import Nimble nimble = Nimble(api_key="YOUR-API-KEY") result = nimble.crawl.run( url="https://www.nimbleway.com/blog", crawl_entire_domain=True, limit=2000 ) print(f"Crawl ID: {result.crawl_id}") print(f"Status: {result.status}") ``` ```javascript Node theme={"system"} import Nimble from "@nimble-way/nimble-js"; const nimble = new Nimble({ apiKey: "YOUR-API-KEY" }); const result = await nimble.crawl.run({ url: "https://www.nimbleway.com/blog", crawl_entire_domain: true, limit: 2000, }); console.log(`Crawl ID: ${result.crawl_id}`); console.log(`Status: ${result.status}`); ``` ```bash cURL theme={"system"} curl -X POST 'https://sdk.nimbleway.com/v1/crawl' \ --header 'Authorization: Bearer ' \ --header 'Content-Type: application/json' \ --data-raw '{ "url": "https://www.nimbleway.com/blog", "crawl_entire_domain": true, "limit": 2000 }' ``` ### Crawl with extraction Extract structured data from each page during the crawl: ```python Python theme={"system"} from nimble_python import Nimble nimble = Nimble(api_key="YOUR-API-KEY") result = nimble.crawl.run( url="https://www.nimbleway.com", limit=500, extract_options={ "driver": "vx10", "parse": True, "formats": ["html", "markdown"] } ) print(f"Crawl ID: {result.crawl_id}") print(f"Status: {result.status}") ``` ```javascript Node theme={"system"} import Nimble from "@nimble-way/nimble-js"; const nimble = new Nimble({ apiKey: "YOUR-API-KEY" }); const result = await nimble.crawl.run({ url: "https://www.nimbleway.com", limit: 500, extract_options: { driver: "vx10", parse: true, formats: ["html", "markdown"], }, }); console.log(`Crawl ID: ${result.crawl_id}`); console.log(`Status: ${result.status}`); ``` ```bash cURL theme={"system"} curl -X POST 'https://sdk.nimbleway.com/v1/crawl' \ --header 'Authorization: Bearer ' \ --header 'Content-Type: application/json' \ --data-raw '{ "url": "https://www.nimbleway.com", "limit": 500, "extract_options": { "driver": "vx10", "parse": true, "formats": ["html", "markdown"] } }' ``` ### Combined parameters Crawl with multiple parameters for precise control: ```python Python theme={"system"} from nimble_python import Nimble nimble = Nimble(api_key="YOUR-API-KEY") result = nimble.crawl.run( url="https://www.nimbleway.com", name="Nimble Website Crawl", sitemap="include", allow_subdomains=True, include_paths=["/use-cases/.*"], limit=1000, callback={ "url": "https://your-server.com/webhook", "events": ["completed"] } ) print(f"Crawl started: {result.crawl_id}") print(f"Status: {result.status}") ``` ```javascript Node theme={"system"} import Nimble from "@nimble-way/nimble-js"; const nimble = new Nimble({ apiKey: "YOUR-API-KEY" }); const result = await nimble.crawl.run({ url: "https://www.nimbleway.com", name: "Nimble Website Crawl", sitemap: "include", allow_subdomains: true, include_paths: ["/use-cases/.*"], limit: 1000, callback: { url: "https://your-server.com/webhook", events: ["completed"], }, }); console.log(`Crawl started: ${result.crawl_id}`); console.log(`Status: ${result.status}`); ``` ```bash cURL theme={"system"} curl -X POST 'https://sdk.nimbleway.com/v1/crawl' \ --header 'Authorization: Bearer ' \ --header 'Content-Type: application/json' \ --data-raw '{ "url": "https://www.nimbleway.com", "name": "Nimble Website Crawl", "sitemap": "include", "allow_subdomains": true, "include_paths": ["/use-cases/.*"], "limit": 1000, "callback": { "url": "https://your-server.com/webhook", "events": ["completed"] } }' ``` ## Managing Crawls Get all your crawls filtered by status using the REST API: * **Available status filters:** `pending`, `in_progress`, `completed`, `failed`, `canceled` ```python Python theme={"system"} from nimble_python import Nimble nimble = Nimble(api_key="YOUR-API-KEY") # List crawls by status my_crawls = nimble.crawl.list() for crawl in my_crawls.data: print(f"Crawl {crawl.crawl_id}: {crawl.name} - {crawl.status}") ``` ```javascript Node theme={"system"} import Nimble from '@nimble-way/nimble-js'; const nimble = new Nimble({ apiKey: "YOUR-API-KEY" }); // List crawls by status const myCrawls = await nimble.crawl.list(); myCrawls.data.forEach(crawl => { console.log(`Crawl ${crawl.crawl_id}: ${crawl.name} - ${crawl.status}`); }); ``` ```bash cURL theme={"system"} # Filter by status: pending, in_progress, completed, failed, canceled curl -X GET 'https://sdk.nimbleway.com/v1/crawl?status=completed' \ --header 'Authorization: Bearer ' ``` Check progress and get the list of task IDs for a specific crawl using the REST API: ```python Python theme={"system"} my_crawl = nimble.crawl.status(crawl_id) print(f"Status: {my_crawl.status}") ``` ```javascript Node theme={"system"} const myCrawl = await nimble.crawl.status(crawlId); console.log(`Status: ${myCrawl.status}`); ``` ```bash cURL theme={"system"} curl -X GET 'https://sdk.nimbleway.com/v1/crawl/e3ca2ff1-b82a-472b-b1a9-ef4d29cc549f' \ --header 'Authorization: Bearer ' ``` Use the `task_id` from the crawl status response to fetch extracted content for each page using the REST API: ```python Python theme={"system"} # my_crawl["tasks"] from step #2 contains list of task IDs from status response for task in my_crawl.tasks: if task.status == "completed": task_result = nimble.tasks.get(task_id) print(f"URL: {task_result['url']}") print(f"HTML length: {len(task_result['data'].get('html', ''))}") ``` ```javascript Node theme={"system"} // myCrawl.tasks from step #2 contains list of task IDs from status response for (const task of myCrawl.tasks) { if (task.status === "completed") { const taskResponse = await nimble.tasks.get(taskId); console.log(`URL: ${taskResult.url}`); console.log(`HTML length: ${taskResult.data?.html?.length || 0}`); } } ``` ```bash cURL theme={"system"} # Get results for each task_id from the crawl status response curl 'https://sdk.nimbleway.com/v1/tasks/{task_id}/results' \ --header 'Authorization: Bearer YOUR-API-KEY' ``` ```json theme={"system"} { "url": "https://www.nimbleway.com/blog/post", "task_id": "ec89b1f7-1cf2-40eb-91b4-78716093f9ed", "status": "success", "task": { "id": "ec89b1f7-1cf2-40eb-91b4-78716093f9ed", "state": "success", "created_at": "2026-02-09T23:15:43.549Z", "modified_at": "2026-02-09T23:16:39.094Z", "account_name": "your-account" }, "data": { "html": "...", "markdown": "# Page Title\n\nContent...", "headers": { ... } }, "metadata": { "query_time": "2026-02-09T23:15:43.549Z", "query_duration": 1877, "response_parameters": { "input_url": "https://www.nimbleway.com/blog/post" }, "driver": "vx6" }, "status_code": 200 } ``` Stop a running crawl using the REST API. Completed tasks remain available for result retrieval: ```python Python theme={"system"} from nimble_python import Nimble nimble = Nimble(api_key="YOUR-API-KEY") response = nimble.crawl.terminate(crawl_id) print(response) # {"status": "canceled"} ``` ```javascript Node theme={"system"} import Nimble from '@nimble-way/nimble-js'; const nimble = new Nimble({ apiKey: "YOUR-API-KEY" }); const response = await nimble.crawl.terminate(crawlId); console.log(await response); // { status: "canceled" } ``` ```bash cURL theme={"system"} curl -X DELETE 'https://sdk.nimbleway.com/v1/crawl/e3ca2ff1-b82a-472b-b1a9-ef4d29cc549f' \ --header 'Authorization: Bearer ' ``` ## Response Fields When you use **Crawl**, you receive: * **Async operation** - Crawl jobs run in the background, check status or receive webhooks * **Progress tracking** - Monitor `total`, `pending`, `completed`, and `failed` counts * **Task-based results** - Each page becomes a task with extractable content * **Webhook support** - Get notified in real-time as pages are processed ### Create Crawl Response Returns immediate response with crawl job details ```json theme={"system"} { "crawl_id": "e3ca2ff1-b82a-472b-b1a9-ef4d29cc549f", "name": null, "url": "https://www.nimbleway.com", "status": "queued", "account_name": "your-account", "created_at": "2026-02-09T23:15:40.785Z", "updated_at": "2026-02-09T23:15:40.785Z", "completed_at": null, "crawl_options": { "sitemap": "include", "crawl_entire_domain": false, "limit": 10, "max_discovery_depth": 5, "ignore_query_parameters": false, "allow_external_links": false, "allow_subdomains": false }, "extract_options": null } ``` | Field | Type | Description | | ----------------- | ------ | ------------------------------------------------------ | | `crawl_id` | string | Unique identifier for the crawl job | | `name` | string | Optional name you assigned to the crawl | | `url` | string | Starting URL for the crawl | | `status` | string | `queued`, `running`, `succeeded`, `failed`, `canceled` | | `account_name` | string | Your account identifier | | `created_at` | string | Timestamp when crawl was created | | `updated_at` | string | Timestamp of last status update | | `completed_at` | string | Timestamp when crawl completed (null if in progress) | | `crawl_options` | object | Configuration settings applied to this crawl | | `extract_options` | object | Extraction settings (null if not configured) | ### Get Crawl Status by ID Response Returns the crawl object wrapped in a `crawl` key, with progress counters and task list: ```json theme={"system"} { "crawl": { "crawl_id": "e3ca2ff1-b82a-472b-b1a9-ef4d29cc549f", "name": null, "url": "https://www.nimbleway.com", "status": "succeeded", "account_name": "your-account", "total": 10, "pending": 0, "completed": 10, "failed": 0, "created_at": "2026-02-09T23:15:40.785Z", "updated_at": "2026-02-09T23:17:08.083Z", "completed_at": "2026-02-09T23:17:08.079Z", "crawl_options": { "sitemap": "include", "crawl_entire_domain": false, "limit": 10, "max_discovery_depth": 5 }, "extract_options": null, "tasks": [ { "task_id": "ec89b1f7-1cf2-40eb-91b4-78716093f9ed", "status": "completed", "updated_at": "2026-02-09T23:16:39.094Z", "created_at": "2026-02-09T23:15:43.549Z" }, { "task_id": "3f6c136c-4bb5-44af-a21b-c8f1db708c2f", "status": "completed", "updated_at": "2026-02-09T23:16:45.033Z", "created_at": "2026-02-09T23:15:42.966Z" } ] } } ``` | Field | Type | Description | | ----------------------- | ------- | ------------------------------------------------ | | `crawl.crawl_id` | string | Unique identifier for the crawl job | | `crawl.status` | string | Current crawl status | | `crawl.total` | integer | Total URLs discovered | | `crawl.pending` | integer | URLs waiting to be processed | | `crawl.completed` | integer | Successfully processed URLs | | `crawl.failed` | integer | Failed URL extractions | | `crawl.tasks` | array | List of individual page tasks | | `crawl.tasks[].task_id` | string | Task ID to use with `GET /v1/tasks/{id}/results` | | `crawl.tasks[].status` | string | `pending`, `processing`, `completed`, `failed` | ## SDK and API methods | Method | Availability | Description | | :-------------------------------- | :----------- | :------------------------------- | | `nimble.crawl.run(url=..., ...)` | Python SDK | Create a new crawl job | | `GET /v1/crawl` | REST API | List all crawls with pagination | | `GET /v1/crawl/{crawl_id}` | REST API | Get crawl status and task list | | `DELETE /v1/crawl/{crawl_id}` | REST API | Stop a running crawl | | `GET /v1/tasks/{task_id}/results` | REST API | Get extracted content for a page | The Python SDK currently supports creating crawl jobs via `nimble.crawl.run()`. For crawl management operations (listing, status, cancellation) and retrieving task results, use the REST API directly as shown in the examples above. ## Use cases Extract data from hundreds or thousands of pages across an entire website Gather all product information from e-commerce sites automatically Create complete snapshots of websites for analysis or backup Track pricing across entire catalogs over time ### Real-world examples **Scenario:** You need to gather all product information from a competitor's online store. **How Crawl helps:** * Discovers all product pages through sitemaps and navigation * Extracts product details, prices, and descriptions from each page * Handles pagination and category structures automatically * Filters out cart, checkout, and account pages **Result:** Complete product catalog data without manual URL collection. **Scenario:** You're migrating a blog to a new platform and need all content. **How Crawl helps:** * Finds all blog posts through sitemap and internal links * Extracts post content, metadata, and images * Excludes tag pages, author archives, and pagination * Preserves URL structure for redirects **Result:** Complete content export ready for migration. **Scenario:** You want to create an offline backup of documentation. **How Crawl helps:** * Maps entire documentation structure * Extracts content from all pages * Maintains hierarchy and navigation structure * Captures code examples and technical content **Result:** Complete documentation archive for offline access. **Scenario:** You need to track competitor pricing across their entire catalog. **How Crawl helps:** * Discovers all product pages automatically * Extracts pricing information from each page * Runs on schedule to track changes over time * Handles dynamic pricing and regional variations **Result:** Comprehensive price intelligence data. **Scenario:** You're auditing a website's content for SEO optimization. **How Crawl helps:** * Discovers all indexable pages * Extracts titles, meta descriptions, and headings * Identifies orphaned pages and broken links * Maps internal linking structure **Result:** Complete SEO audit data for analysis. ## Crawl vs Map | Need | Use | | --------------------------------- | ------------------------------ | | Extract content from pages | **Crawl** | | Deep link following | **Crawl** | | Complex filtering patterns | **Crawl** | | Webhook notifications | **Crawl** | | Quick URL discovery only | **Map** - completes in seconds | | URL list with titles/descriptions | **Map** | **Try Use Map first** - Discover URLs quickly with Map, then use Crawl to extract content from the pages you need. ## Next steps Explore endpoints, request parameters, and response schemas for the Crawl API # Async Requests Source: https://docs.nimbleway.com/nimble-sdk/web-tools/extract/features/async Run extraction tasks asynchronously for large-scale operations Use async requests when you need to extract data from multiple pages, run long-running operations, or integrate extraction into background job systems. Async requests return immediately with a task ID, letting you check status and retrieve results later. ## When to use async Use async extract when you: * **Background jobs**: Integrate extraction into scheduled or queued workflows * **Long-running operations**: Handle complex browser actions or slow-loading sites * **Webhook integration**: Get notified when extraction completes * **Cloud storage**: Save results directly to S3 or Google Cloud Storage Async requests are ideal for high-volume extraction where you don't need immediate results. For single-page extractions where you need results right away, use the synchronous [Extract API](/nimble-sdk/web-tools/extract/quickstart). ## How it works Send a POST request to `/v1/extract/async` with your extraction parameters. The API returns immediately with a task ID. Optionally include: * `callback_url` to receive a webhook notification when complete * `storage_url` and `storage_type` (s3/gs) to save results directly to cloud storage Use the task ID to check progress at `/v1/tasks/{task_id}`. The task transitions through states: `pending` → `completed` or `failed`. Once complete, fetch results from `/v1/tasks/{task_id}/results` or from your configured cloud storage. ## API endpoint ```bash theme={"system"} POST https://sdk.nimbleway.com/v1/extract/async ``` ## Parameters Async extract accepts all the same parameters as the synchronous extract endpoint, plus optional async-specific parameters: ### Core extraction parameters All parameters from the [Extract API](/nimble-sdk/web-tools/extract/quickstart) are supported: * `url` (required) - The webpage to extract * `formats` - Output formats (html, markdown, text, screenshot) * `render` - Enable JavaScript rendering * `driver` - Choose extraction engine (vx6, vx8, vx10, etc.) * `country`, `state`, `city` - Geo-targeting options * `parse` - Enable parsing with schemas * `parser` - Define extraction schema * `browser_actions` - Automate interactions * `network_capture` - Capture network requests * And all other extract parameters... ### Async-specific parameters Storage provider for results. When specified, results are saved to your cloud storage instead of Nimble's servers. **Options:** `s3` (Amazon S3), `gs` (Google Cloud Storage) Bucket path where results will be saved. Results are stored as `{task_id}.json` at the specified location. **Format:** `s3://your-bucket/path/prefix/` **Example:** `s3://my-bucket/nimble-results/` Compress results with GZIP before saving. Reduces storage costs and transfer time. When `true`, results are saved as `{task_id}.json.gz`. Custom filename for the stored object instead of the default task ID. **Example:** `"my-custom-name"` saves as `my-custom-name.json` Webhook URL to receive a POST request when the task completes. Nimble sends task metadata (without result data) to this URL when extraction finishes. **Example:** `https://your-api.com/webhook/complete` ## Response format The async endpoint returns immediately with task information: ```json theme={"system"} { "task": { "id": "8e8cfde8-345b-42b8-b3e2-0c61eb11e00f", "state": "completed", "status_code": 200, "created_at": "2026-01-24T12:36:24.685Z", "modified_at": "2026-01-24T12:36:24.685Z", "input": {}, "api_type": "extract" } } ``` ### Task states | State | Description | | --------- | --------------------------------------------------- | | `pending` | Task queued, waiting to start | | `running` | Extraction in progress | | `success` | Extraction finished successfully, results available | | `failed` | Extraction failed, check error details | ## Example usage ### Basic async extraction ```python Python theme={"system"} from nimble_python import Nimble nimble = Nimble(api_key="YOUR_API_KEY") # Submit async extraction response = nimble.extract_async( url= "https://www.example.com", render= True, formats= ["html", "markdown"] ) task_id = response.task_id print(f"Task created: {task_id}") # Check status import time while True: my_task = nimble.tasks.get(task_id) print(f"Status: {my_task.task.state}") if my_task.task.state == 'success': # Get results results = nimble.tasks.results(task_id) print(results.data.html[:200]) break elif my_task.task.state == 'failed': print(f"Task failed: {my_task.message}") break time.sleep(2) ``` ```javascript Node theme={"system"} import Nimble from "@nimble-way/nimble-js"; const nimble = new Nimble({ apiKey: "YOUR_API_KEY" }); // Submit async extraction const response = await nimble.extractAsync({ url: "https://www.example.com", render: true, formats: ["html", "markdown"], }); const taskId = response.task_id; console.log(`Task created: ${taskId}`); // Check status while (true) { const myTask = await nimble.tasks.get(taskId); console.log(`Status: ${myTask.task.state}`); if (myTask.task.state === "success") { // Get results const results = await nimble.tasks.results(taskId); console.log(results.data.html.substring(0, 200)); break; } else if (myTask.task.state === "failed") { console.log(`Task failed: ${status.error}`); break; } await new Promise((resolve) => setTimeout(resolve, 2000)); } ``` ```bash cURL theme={"system"} # Submit async extraction curl -X POST 'https://sdk.nimbleway.com/v1/extract/async' \ --header 'Authorization: Bearer YOUR_API_KEY' \ --header 'Content-Type: application/json' \ --data-raw '{ "url": "https://www.example.com", "render": true, "formats": ["html", "markdown"] }' # Response: # { # "task": { # "id": "f32e1196-3827-4b3a-b6ad-89c51040a2ab", # "state": "pending", # "created_at": "2026-02-18T17:17:28.020Z", # "modified_at": "2026-02-18T17:17:28.020Z", # "account_name": "account_name", # "input": { # "url": "https://www.google.com/search?q=trump", # "country": "US", # "locale": "en-US", # "driver": "vx6" # }, # "api_type": "extract" # } # } # Check status curl 'https://sdk.nimbleway.com/v1/tasks/8e8cfde8-345b-42b8-b3e2-0c61eb11e00f' \ --header 'Authorization: Bearer YOUR_API_KEY' # Get results (when completed) curl 'https://sdk.nimbleway.com/v1/tasks/8e8cfde8-345b-42b8-b3e2-0c61eb11e00f/results' \ --header 'Authorization: Bearer YOUR_API_KEY' ``` ### Save to cloud storage Store results directly in Amazon S3: ```python Python theme={"system"} from nimble_python import Nimble nimble = Nimble(api_key="YOUR_API_KEY") response = nimble.extract_async( url= "https://www.example.com", render= True, formats= ["html", "markdown"], storage_type= "s3", storage_url= "s3://my-bucket/nimble-extracts/", storage_compress= True, storage_object_name= "example-com-extraction" ) task_id = response.task_id print(f"Task created: {task_id}") print(f"Results will be saved to: s3://my-bucket/nimble-extracts/example-com-extraction.json.gz") # Results are automatically saved to S3 when complete # You can still check status and retrieve from Nimble's servers if needed ``` ```javascript Node theme={"system"} import Nimble from "@nimble-way/nimble-js"; const nimble = new Nimble({ apiKey: "YOUR_API_KEY" }); // Submit async extraction const response = await nimble.extractAsync({ url: "https://www.example.com", render: true, formats: ["html", "markdown"], storage_type: "s3", storage_url: "s3://my-bucket/nimble-extracts/", storage_compress: True, storage_object_name: "example-com-extraction", }); const taskId = response.task_id; console.log(`Task created: ${taskId}`); console.log( "Results will be saved to: s3://my-bucket/nimble-extracts/example-com-extraction.json.gz", ); // Results are automatically saved to S3 when complete // You can still check status and retrieve from Nimble's servers if needed ``` ```bash cURL theme={"system"} # Submit async extraction curl -X POST 'https://sdk.nimbleway.com/v1/extract/async' \ --header 'Authorization: Bearer YOUR_API_KEY' \ --header 'Content-Type: application/json' \ --data-raw '{ "url": "https://www.example.com", "render": true, "formats": ["html", "markdown"], "storage_type": "s3", "storage_url": "s3://my-bucket/nimble-extracts/", "storage_compress": True, "storage_object_name": "example-com-extraction" }' # Results are automatically saved to S3 when complete # You can still check status and retrieve from Nimble's servers if needed ``` ### Webhook notifications Get notified when extraction completes: ```python Python theme={"system"} from nimble_python import Nimble nimble = Nimble(api_key="YOUR_API_KEY") response = nimble.extract_async( url= "https://www.example.com", render= True, formats= ["html"], callback_url= "https://your-api.com/webhooks/extract-complete" ) task_id = response.task_id print(f"Task created: {task_id}") print("Webhook will be called when extraction completes") ``` ```javascript Node theme={"system"} import Nimble from "@nimble-way/nimble-js"; const nimble = new Nimble({ apiKey: "YOUR_API_KEY" }); // Submit async extraction const response = await nimble.extractAsync({ url: "https://www.example.com", render: true, formats: ["html", "markdown"], callback_url: "https://your-api.com/webhooks/extract-complete", }); const taskId = response.task_id; console.log(`Task created: ${taskId}`); console.log("Webhook will be called when extraction completes"); ``` ```bash cURL theme={"system"} # Submit async extraction curl -X POST 'https://sdk.nimbleway.com/v1/extract/async' \ --header 'Authorization: Bearer YOUR_API_KEY' \ --header 'Content-Type: application/json' \ --data-raw '{ "url": "https://www.example.com", "render": true, "formats": ["html", "markdown"], "callback_url": "https://your-api.com/webhooks/extract-complete" }' # Webhook will be called when extraction completes ``` ## Checking task status Use the Tasks API to check status: ```bash theme={"system"} GET https://sdk.nimbleway.com/v1/tasks/{task_id}/results ``` ### Status Response: ```json theme={"system"} { "task": { "id": "8e8cfde8-345b-42b8-b3e2-0c61eb11e00f", "state": "completed", "status_code": 200, "created_at": "2026-01-24T12:36:24.685Z", "modified_at": "2026-01-24T12:36:24.685Z", "input": {}, "api_type": "extract" } } ``` ## Retrieving results Once the task is complete, retrieve results: ```bash theme={"system"} GET https://sdk.nimbleway.com/v1/tasks/{task_id}/results ``` ### Results Response ```json theme={"system"} { "url": "https://www.nimbleway.com/blog/post", "task_id": "ec89b1f7-1cf2-40eb-91b4-78716093f9ed", "status": "success", "task": { "id": "ec89b1f7-1cf2-40eb-91b4-78716093f9ed", "state": "success", "created_at": "2026-02-09T23:15:43.549Z", "modified_at": "2026-02-09T23:16:39.094Z", "account_name": "your-account" }, "data": { "html": "...", "markdown": "# Page Title\n\nContent...", "headers": { ... } }, "metadata": { "query_time": "2026-02-09T23:15:43.549Z", "query_duration": 1877, "response_parameters": { "input_url": "https://www.nimbleway.com/blog/post" }, "driver": "vx6" }, "status_code": 200 } ``` ## Data Retention & Expiration ### Result retention Results are typically retained for 7 days. If you need longer retention: * Use cloud storage (`storage_url`) to persist results indefinitely * Download results promptly after completion * Implement your own archival system ### Task expiration * **Pending tasks**: Expire after 24 hours if not started * **Completed results**: Available for 24-48 hours (unless using cloud storage) * **Failed tasks**: Retry data available for 24 hours # Browser Actions Source: https://docs.nimbleway.com/nimble-sdk/web-tools/extract/features/browser-actions Full control over browser actions with predefined steps **Browser Actions** gives you full control over the sequence of actions performed in the browser, ensuring predictable and repeatable results. You manually specify each step like click, scroll, wait, or type. Common uses: * **Load more content**: Click "Show more" buttons or scroll to reveal additional items. * **Set filters**: Select dropdowns, checkboxes, or date ranges to refine displayed data. * **Submit searches**: Enter search terms and submit forms to access results pages. * **Trigger lazy-loaded data**: Scroll to load images, products, or infinite feeds. ## When to use Use Browser Actions when you need: * **Predictable execution**: Same steps run every time * **Full control**: Specify exact actions and sequences * **Fast performance**: No AI inference overhead Browser Action flows may break when page structure or selectors change. Monitor and update your flows as needed. ## Parameters **`Must be set to true`** to use browser actions. This enables the browser to execute JavaScript and perform interactions. Browser actions only work when the page is rendered in a real browser environment. **Example:** ```json theme={"system"} "render": true ``` A list of actions to perform in sequence on the page. Each action is an object specifying what to do (click, scroll, wait, type, etc.). **Actions execute in order** - the second action waits for the first to complete, and so on. **Global timeout:** All actions must complete within 240 seconds total. **Available actions:** * `goto` - Navigate to a different URL * `wait` - Pause for a specified duration * `wait_for_element` - Wait for an element to appear * `click` - Click an element * `press` - Press a keyboard key (Enter, Tab, Escape, etc.) * `fill` - Type or paste text into an input field * `scroll` - Scroll the page by pixels or to an element * `auto_scroll` - Automatically scroll to load lazy content * `screenshot` - Capture a screenshot * `get_cookies` - Extract cookies from the page * `fetch` - Make HTTP requests from the browser context Each action has specific parameters detailed in the sections below. **Example:** ```json theme={"system"} "browser_actions": [ { "fill": { "selector": "input[type='search']", "value": "laptops" } }, { "click": { "selector": "button[type='submit']" } }, { "wait": "2s" } ] ``` All browser actions execute sequentially within a global **240-second timeout**. If any action fails, the entire flow stops unless marked as optional. ## Usage Navigate to a different URL during the browser session. **Parameters:** **Direct form** - Pass URL as a string: * `goto` (required) - URL to navigate to **Extended form** - Use object with options: * `url` (required) - URL to navigate to (string) * `timeout` (optional) - Maximum wait time in milliseconds (number, default: 30000) * `referer` (optional) - HTTP referer header to send (string) * `wait_until` (optional) - Navigation wait condition: `'load'` (default), `'domcontentloaded'`, `'networkidle'`, or `'commit'` ```python Python theme={"system"} from nimble_python import Nimble nimble = Nimble(api_key="YOUR-API-KEY") # Direct form - simple navigation result = nimble.extract( url= "https://www.example.com", render= True, browser_actions= [ { "goto": "https://www.example.com/products" # Navigate to products page } ] ) # Extended form - with wait condition result = nimble.extract( url= "https://www.example.com", render= True, browser_actions= [ { "goto": { "url": "https://www.example.com/products", "wait_until": "networkidle", "timeout": 5000, "referer": "https://www.google.com" } } ] ) print(result) ``` ```javascript Node theme={"system"} import Nimble from '@nimble-way/nimble-js'; const nimble = new Nimble({ apiKey: "YOUR-API-KEY" }); // Direct form - simple navigation const result = await nimble.extract({ url: "https://www.example.com", render: true, browser_actions: [ { goto: "https://www.example.com/products" // Navigate to products page } ] }); // Extended form - with wait condition const result2 = await nimble.extract({ url: "https://www.example.com", render: true, browser_actions: [ { goto: { url: "https://www.example.com/products", wait_until: "networkidle", timeout: 5000, referer: "https://www.google.com" } } ] }); console.log(result); ``` ```bash cURL theme={"system"} # Direct form curl -X POST 'https://sdk.nimbleway.com/v1/extract' \ --header 'Authorization: Bearer ' \ --header 'Content-Type: application/json' \ --data-raw '{ "url": "https://www.example.com", "render": true, "browser_actions": [{ "goto": "https://www.example.com/products" }] }' # Extended form curl -X POST 'https://sdk.nimbleway.com/v1/extract' \ --header 'Authorization: Bearer ' \ --header 'Content-Type: application/json' \ --data-raw '{ "url": "https://www.example.com", "render": true, "browser_actions": [{ "goto": { "url": "https://www.example.com/products", "wait_until": "networkidle", "timeout": 5000, "referer": "https://www.google.com" } }] }' ``` Add a delay between actions to simulate human behavior or allow elements to load. **Parameters:** **Direct form** - Pass duration as a string: * `wait` (required) - Duration string (e.g., "2s", "500ms", "1.5s") **Extended form** - Use object with options: * `duration` (required) - Duration string (e.g., "2s", "500ms", "1.5s") * `required` (optional) - Set to `false` to make this step optional (boolean, default: `true`) ```python Python theme={"system"} from nimble_python import Nimble nimble = Nimble(api_key="YOUR-API-KEY") # Direct form - simple delay result = nimble.extract( url= "https://www.example.com", render= True, browser_actions= [ { "wait": "2s" # Wait for 2 seconds } ] ) # Extended form - with options result = nimble.extract( url= "https://www.example.com", render= True, browser_actions= [ { "wait": { "duration": "500ms", "required": False } } ] ) ``` ```javascript Node theme={"system"} import Nimble from '@nimble-way/nimble-js'; const nimble = new Nimble({ apiKey: "YOUR-API-KEY" }); // Direct form - simple delay const result = await nimble.extract({ url: "https://www.example.com", render: true, browser_actions: [ { wait: "2s" // Wait for 2 seconds } ] }); // Extended form - with options const result2 = await nimble.extract({ url: "https://www.example.com", render: true, browser_actions: [ { wait: { duration: "500ms", required: false } } ] }); ``` ```bash cURL theme={"system"} # Direct form curl -X POST 'https://sdk.nimbleway.com/v1/extract' \ --header 'Authorization: Bearer ' \ --header 'Content-Type: application/json' \ --data-raw '{ "url": "https://www.example.com", "render": true, "browser_actions": [{ "wait": "2s" }] }' # Extended form curl -X POST 'https://sdk.nimbleway.com/v1/extract' \ --header 'Authorization: Bearer ' \ --header 'Content-Type: application/json' \ --data-raw '{ "url": "https://www.example.com", "render": true, "browser_actions": [{ "wait": { "duration": "500ms", "required": false } }] }' ``` Wait for DOM elements to appear on the page. **Parameters:** **Direct form** - Pass single selector or array of selectors: * `wait_for_element` (required) - CSS selector string or array of selectors to wait for (any one matching triggers success) **Extended form** - Use object with options: * `selector` (required) - CSS selector string or array of selectors to wait for * `timeout` (optional) - Maximum time in milliseconds to wait (number, default: 30000) * `visible` (optional) - Whether element must be visible (boolean, default: `true`) ```python Python theme={"system"} from nimble_python import Nimble nimble = Nimble(api_key="YOUR-API-KEY") # Direct form - wait for single element result = nimble.extract( url= "https://www.example.com", render= True, browser_actions= [ { "wait_for_element": "#submit-button" } ] ) # Direct form - wait for multiple elements result = nimble.extract( url= "https://www.example.com", render= True, browser_actions= [ { "wait_for_element": ["#element1", ".class2"] } ] ) # Extended form - with options result = nimble.extract( url= "https://www.example.com", render= True, browser_actions= [ { "wait_for_element": { "selector": "#element", "timeout": 30000, "visible": False } } ] ) print(result) ``` ```javascript Node theme={"system"} import Nimble from '@nimble-way/nimble-js'; const nimble = new Nimble({ apiKey: "YOUR-API-KEY" }); // Direct form - wait for single element const result = await nimble.extract({ url: "https://www.example.com", render: true, browser_actions: [ { wait_for_element: "#submit-button" } ] }); // Direct form - wait for multiple elements const result2 = await nimble.extract({ url: "https://www.example.com", render: true, browser_actions: [ { wait_for_element: ["#element1", ".class2"] } ] }); // Extended form - with options const result3 = await nimble.extract({ url: "https://www.example.com", render: true, browser_actions: [ { wait_for_element: { selector: "#element", timeout: 30000, visible: false } } ] }); console.log(result); ``` ```bash cURL theme={"system"} # Direct form - single selector curl -X POST 'https://sdk.nimbleway.com/v1/extract' \ --header 'Authorization: Bearer ' \ --header 'Content-Type: application/json' \ --data-raw '{ "url": "https://www.example.com", "render": true, "browser_actions": [{ "wait_for_element": "#submit-button" }] }' # Direct form - multiple selectors curl -X POST 'https://sdk.nimbleway.com/v1/extract' \ --header 'Authorization: Bearer ' \ --header 'Content-Type: application/json' \ --data-raw '{ "url": "https://www.example.com", "render": true, "browser_actions": [{ "wait_for_element": ["#element1", ".class2"] }] }' # Extended form curl -X POST 'https://sdk.nimbleway.com/v1/extract' \ --header 'Authorization: Bearer ' \ --header 'Content-Type: application/json' \ --data-raw '{ "url": "https://www.example.com", "render": true, "browser_actions": [{ "wait_for_element": { "selector": "#element", "timeout": 30000, "visible": false } }] }' ``` Click on an element by selector or at specific coordinates. **Parameters:** **Direct form** - Pass CSS selector as string: * `click` (required) - CSS selector string **Extended form (selector mode)** - Use object with selector options: * `selector` (required) - CSS selector of element to click (string) * `timeout` (optional) - Maximum time to wait for element (Duration string, default: "30s") * `visible` (optional) - Whether element must be visible (boolean, default: `true`) * `delay` (optional) - Delay before clicking (Duration string, default: "0ms") * `scroll` (optional) - Whether to scroll element into view (boolean, default: `true`) * `count` (optional) - Number of times to click (number, default: 1) * `steps` (optional) - Mouse movement steps for granular control (number, advanced) * `strategy` (optional) - Mouse movement strategy for behavioral simulation (advanced) * `required` (optional) - Set to `false` to make this step optional (boolean, default: `true`) **Extended form (coordinates mode)** - Click at specific coordinates: * `x` (required) - X coordinate (horizontal position, number) * `y` (required) - Y coordinate (vertical position, number) * `relative_to` (optional) - CSS selector to make coordinates relative to this element (string) * `timeout` (optional) - Maximum time for the click operation (Duration string, default: "15s") * `delay` (optional) - Delay before clicking (Duration string, default: "0ms") * `count` (optional) - Number of times to click (number, default: 1) * `steps` (optional) - Mouse movement steps for granular control (number, advanced) * `required` (optional) - Set to `false` to make this step optional (boolean, default: `true`) ```python Python theme={"system"} from nimble_python import Nimble nimble = Nimble(api_key="YOUR-API-KEY") # Direct form - simple click by selector result = nimble.extract( "url": "https://www.example.com", "render": True, "browser_actions": [ { "click": "button.submit" # Click element matching selector } ] ) # Extended form (selector mode) - with options result = nimble.extract( url= "https://www.example.com", render= True, browser_actions= [ { "click": { "selector": "button.load-more", "timeout": "20s", "delay": "500ms", "scroll": True, "visible": True, "count": 2 # Double-click } } ] ) # Extended form (coordinates mode) - click at specific position result = nimble.extract( url= "https://www.example.com", render= True, browser_actions= [ { "click": { "x": 100, "y": 200, "delay": "300ms" } } ] ) # Coordinates relative to element result = nimble.extract( url= "https://www.example.com", render= True, browser_actions= [ { "click": { "x": 10, "y": 10, "relative_to": "#container" # Click 10px from top-left of container } } ] ) print(result) ``` ```javascript Node theme={"system"} import Nimble from '@nimble-way/nimble-js'; const nimble = new Nimble({ apiKey: "YOUR-API-KEY" }); // Direct form - simple click by selector const result = await nimble.extract({ url: "https://www.example.com", render: true, browser_actions: [ { click: "button.submit" // Click element matching selector } ] }); // Extended form (selector mode) - with options const result2 = await nimble.extract({ url: "https://www.example.com", render: true, browser_actions: [ { click: { selector: "button.load-more", timeout: "20s", delay: "500ms", scroll: true, visible: true, count: 2 // Double-click } } ] }); // Extended form (coordinates mode) - click at specific position const result3 = await nimble.extract({ url: "https://www.example.com", render: true, browser_actions: [ { click: { x: 100, y: 200, delay: "300ms" } } ] }); // Coordinates relative to element const result4 = await nimble.extract({ url: "https://www.example.com", render: true, browser_actions: [ { click: { x: 10, y: 10, relative_to: "#container" // Click 10px from top-left of container } } ] }); console.log(result); ``` ```bash cURL theme={"system"} # Direct form - simple click by selector curl -X POST 'https://sdk.nimbleway.com/v1/extract' \ --header 'Authorization: Bearer ' \ --header 'Content-Type: application/json' \ --data-raw '{ "url": "https://www.example.com", "render": true, "browser_actions": [{ "click": "button.submit" }] }' # Extended form (selector mode) - with options curl -X POST 'https://sdk.nimbleway.com/v1/extract' \ --header 'Authorization: Bearer ' \ --header 'Content-Type: application/json' \ --data-raw '{ "url": "https://www.example.com", "render": true, "browser_actions": [{ "click": { "selector": "button.load-more", "timeout": "20s", "delay": "500ms", "scroll": true, "visible": true, "count": 2 } }] }' # Extended form (coordinates mode) - click at specific position curl -X POST 'https://sdk.nimbleway.com/v1/extract' \ --header 'Authorization: Bearer ' \ --header 'Content-Type: application/json' \ --data-raw '{ "url": "https://www.example.com", "render": true, "browser_actions": [{ "click": { "x": 100, "y": 200, "delay": "300ms" } }] }' # Coordinates relative to element curl -X POST 'https://sdk.nimbleway.com/v1/extract' \ --header 'Authorization: Bearer ' \ --header 'Content-Type: application/json' \ --data-raw '{ "url": "https://www.example.com", "render": true, "browser_actions": [{ "click": { "x": 10, "y": 10, "relative_to": "#container" } }] }' ``` Press a keyboard key like Enter, Tab, Escape, or arrow keys. **Parameters:** **Direct form** - Pass key name as a string: * `press` (required) - Key name to press (e.g., "Enter", "Tab", "Escape", "ArrowDown") **Extended form** - Use object with options: * `key` (required) - Key name to press (string) * `delay` (optional) - Delay in milliseconds before pressing (number, default: 0) ```python Python theme={"system"} from nimble_python import Nimble nimble = Nimble(api_key="YOUR-API-KEY") # Direct form - press Enter key result = nimble.extract( url= "https://www.example.com", render= True, browser_actions= [ { "press": "Enter" # Press Enter key } ] ) # Extended form - press with delay result = nimble.extract( url= "https://www.example.com", render= True, browser_actions= [ { "press": { "key": "Tab", "delay": 500 } } ] ) print(result) ``` ```javascript Node theme={"system"} import Nimble from '@nimble-way/nimble-js'; const nimble = new Nimble({ apiKey: "YOUR-API-KEY" }); // Direct form - press Enter key const result = await nimble.extract({ url: "https://www.example.com", render: true, browser_actions: [ { press: "Enter" // Press Enter key } ] }); // Extended form - press with delay const result2 = await nimble.extract({ url: "https://www.example.com", render: true, browser_actions: [ { press: { key: "Tab", delay: 500 } } ] }); console.log(result); ``` ```bash cURL theme={"system"} # Direct form curl -X POST 'https://sdk.nimbleway.com/v1/extract' \ --header 'Authorization: Bearer ' \ --header 'Content-Type: application/json' \ --data-raw '{ "url": "https://www.example.com", "render": true, "browser_actions": [{ "press": "Enter" }] }' # Extended form curl -X POST 'https://sdk.nimbleway.com/v1/extract' \ --header 'Authorization: Bearer ' \ --header 'Content-Type: application/json' \ --data-raw '{ "url": "https://www.example.com", "render": true, "browser_actions": [{ "press": { "key": "Tab", "delay": 500 } }] }' ``` Fill an input field with text. Supports both typing (character-by-character) and pasting (instant) modes. **Parameters:** **Extended form only** - No direct form available: * `selector` (required) - CSS selector of the input element (string) * `value` (required) - Text value to fill (string) * `mode` (optional) - Fill mode: `'type'` for character-by-character (default), `'paste'` for instant (string) * `timeout` (optional) - Maximum time to wait for element in milliseconds (number, default: 30000) * `typing_interval` (optional) - Interval between keystrokes in milliseconds when typing (number, default: 100, min: 10, max: 1000) * `click_on_element` (optional) - Whether to click on element to focus it (boolean, default: `true`) * `scroll` (optional) - Whether to scroll element into view (boolean, default: `true`) * `visible` (optional) - Whether element must be visible (boolean, default: `true`) * `delay` (optional) - Delay in milliseconds before filling (number, default: 0) * `mouse_movement_strategy` (optional) - Mouse movement strategy for behavioral simulation (advanced) * `typing_strategy` (optional) - Typing strategy for behavioral simulation (advanced) * `required` (optional) - Set to `false` to make this step optional (boolean, default: `true`) ```python Python theme={"system"} from nimble_python import Nimble nimble = Nimble(api_key="YOUR-API-KEY") # Type mode (default, character-by-character) result = nimble.extract( url= "https://www.example.com", render= True, browser_actions= [ { "fill": { "selector": "input[name='search']", "value": "eggplant", "mode": "type", # Default mode "typing_interval": 100, # 100ms between keystrokes "click_on_element": True, "scroll": True } } ] ) # Paste mode (instant, faster) result = nimble.extract( url= "https://www.example.com", render= True, browser_actions= [ { "fill": { "selector": "textarea", "value": "Long text content...", "mode": "paste", # Instant paste "timeout": 30000 } } ] ) # Fill with options result = nimble.extract( url= "https://www.example.com", render= True, browser_actions= [ { "fill": { "selector": "input", "value": "text", "scroll": True, "delay": 500, "typing_interval": 150, "visible": True } } ] ) print(result) ``` ```javascript Node theme={"system"} import Nimble from '@nimble-way/nimble-js'; const nimble = new Nimble({ apiKey: "YOUR-API-KEY" }); // Type mode (default, character-by-character) const result = await nimble.extract({ url: "https://www.example.com", render: true, browser_actions: [ { fill: { selector: "input[name='search']", value: "eggplant", mode: "type", // Default mode typing_interval: 100, // 100ms between keystrokes click_on_element: true, scroll: true } } ] }); // Paste mode (instant, faster) const result2 = await nimble.extract({ url: "https://www.example.com", render: true, browser_actions: [ { fill: { selector: "textarea", value: "Long text content...", mode: "paste", // Instant paste timeout: 30000 } } ] }); // Fill with options const result3 = await nimble.extract({ url: "https://www.example.com", render: true, browser_actions: [ { fill: { selector: "input", value: "text", scroll: true, delay: 500, typing_interval: 150, visible: true } } ] }); console.log(result); ``` ```bash cURL theme={"system"} # Type mode (default) curl -X POST 'https://sdk.nimbleway.com/v1/extract' \ --header 'Authorization: Bearer ' \ --header 'Content-Type: application/json' \ --data-raw '{ "url": "https://www.example.com", "render": true, "browser_actions": [{ "fill": { "selector": "input[name='\''search'\'']", "value": "eggplant", "mode": "type", "typing_interval": 100, "click_on_element": true, "scroll": true } }] }' # Paste mode curl -X POST 'https://sdk.nimbleway.com/v1/extract' \ --header 'Authorization: Bearer ' \ --header 'Content-Type: application/json' \ --data-raw '{ "url": "https://www.example.com", "render": true, "browser_actions": [{ "fill": { "selector": "textarea", "value": "Long text content...", "mode": "paste", "timeout": 30000 } }] }' # Fill with options curl -X POST 'https://sdk.nimbleway.com/v1/extract' \ --header 'Authorization: Bearer ' \ --header 'Content-Type: application/json' \ --data-raw '{ "url": "https://www.example.com", "render": true, "browser_actions": [{ "fill": { "selector": "input", "value": "text", "scroll": true, "delay": 500, "typing_interval": 150, "visible": true } }] }' ``` Scroll the page by pixels, to a position, or to an element. **Parameters:** **Direct form** - Pass number or string: * `scroll: 500` - Scroll Y pixels (number, positive = down, negative = up) * `scroll: "bottom"` - Scroll to bottom of page (string) * `scroll: "#comments"` - Scroll to element matching selector (string) **Extended form** - Use object with options: * `y` (optional) - Vertical scroll distance in pixels (number, default: 0) * `x` (optional) - Horizontal scroll distance in pixels (number, default: 0) * `to` (optional) - CSS selector to scroll to (string, alternative to x/y) * `container` (optional) - CSS selector of container element to scroll within (string) * `visible` (optional) - Whether element must be visible when scrolling to it (boolean, default: `true`) * `required` (optional) - Set to `false` to make this step optional (boolean, default: `true`) ```python Python theme={"system"} from nimble_python import Nimble nimble = Nimble(api_key="YOUR-API-KEY") # Direct form - scroll down 500px result = nimble.extract( url= "https://www.example.com", render= True, browser_actions= [ { "scroll": 500 # Scroll down 500 pixels } ] ) # Direct form - scroll up 200px result = nimble.extract( url= "https://www.example.com", render= True, browser_actions= [ { "scroll": -200 # Scroll up 200 pixels } ] ) # Direct form - scroll to bottom result = nimble.extract( url= "https://www.example.com", render= True, browser_actions= [ { "scroll": "bottom" # Scroll to page bottom } ] ) # Direct form - scroll to element result = nimble.extract( url= "https://www.example.com", render= True, browser_actions= [ { "scroll": "#comments" # Scroll to comments section } ] ) # Extended form - scroll by pixels result = nimble.extract( url= "https://www.example.com", render= True, browser_actions= [ { "scroll": { "x": 100, "y": 500 } } ] ) # Extended form - scroll to element in container result = nimble.extract( url= "https://www.example.com", render= True, browser_actions= [ { "scroll": { "to": "#target", "container": "#modal", "visible": True } } ] ) print(result) ``` ```javascript Node theme={"system"} import Nimble from '@nimble-way/nimble-js'; const nimble = new Nimble({ apiKey: "YOUR-API-KEY" }); // Direct form - scroll down 500px const result = await nimble.extract({ url: "https://www.example.com", render: true, browser_actions: [ { scroll: 500 // Scroll down 500 pixels } ] }); // Direct form - scroll up 200px const result2 = await nimble.extract({ url: "https://www.example.com", render: true, browser_actions: [ { scroll: -200 // Scroll up 200 pixels } ] }); // Direct form - scroll to bottom const result3 = await nimble.extract({ url: "https://www.example.com", render: true, browser_actions: [ { scroll: "bottom" // Scroll to page bottom } ] }); // Direct form - scroll to element const result4 = await nimble.extract({ url: "https://www.example.com", render: true, browser_actions: [ { scroll: "#comments" // Scroll to comments section } ] }); // Extended form - scroll by pixels const result5 = await nimble.extract({ url: "https://www.example.com", render: true, browser_actions: [ { scroll: { x: 100, y: 500 } } ] }); // Extended form - scroll to element in container const result6 = await nimble.extract({ url: "https://www.example.com", render: true, browser_actions: [ { scroll: { to: "#target", container: "#modal", visible: true } } ] }); console.log(result); ``` ```bash cURL theme={"system"} # Direct form - scroll down 500px curl -X POST 'https://sdk.nimbleway.com/v1/extract' \ --header 'Authorization: Bearer ' \ --header 'Content-Type: application/json' \ --data-raw '{ "url": "https://www.example.com", "render": true, "browser_actions": [{ "scroll": 500 }] }' # Direct form - scroll up 200px curl -X POST 'https://sdk.nimbleway.com/v1/extract' \ --header 'Authorization: Bearer ' \ --header 'Content-Type: application/json' \ --data-raw '{ "url": "https://www.example.com", "render": true, "browser_actions": [{ "scroll": -200 }] }' # Direct form - scroll to bottom curl -X POST 'https://sdk.nimbleway.com/v1/extract' \ --header 'Authorization: Bearer ' \ --header 'Content-Type: application/json' \ --data-raw '{ "url": "https://www.example.com", "render": true, "browser_actions": [{ "scroll": "bottom" }] }' # Direct form - scroll to element curl -X POST 'https://sdk.nimbleway.com/v1/extract' \ --header 'Authorization: Bearer ' \ --header 'Content-Type: application/json' \ --data-raw '{ "url": "https://www.example.com", "render": true, "browser_actions": [{ "scroll": "#comments" }] }' # Extended form - scroll by pixels curl -X POST 'https://sdk.nimbleway.com/v1/extract' \ --header 'Authorization: Bearer ' \ --header 'Content-Type: application/json' \ --data-raw '{ "url": "https://www.example.com", "render": true, "browser_actions": [{ "scroll": { "x": 100, "y": 500 } }] }' # Extended form - scroll to element in container curl -X POST 'https://sdk.nimbleway.com/v1/extract' \ --header 'Authorization: Bearer ' \ --header 'Content-Type: application/json' \ --data-raw '{ "url": "https://www.example.com", "render": true, "browser_actions": [{ "scroll": { "to": "#target", "container": "#modal", "visible": true } }] }' ``` Automatically scroll the page to load dynamic content (infinite scroll). Continuously scrolls until content stops loading or timeout is reached. **Parameters:** **Direct form** - Pass boolean or number: * `auto_scroll: true` - Use default settings * `auto_scroll: 30000` - Max duration in milliseconds **Extended form** - Use object with options: * `max_duration` (optional) - Maximum duration in milliseconds to scroll (number, default: 25000) * `delay_after_scroll` (optional) - Delay in milliseconds between scroll steps (number, default: 200) * `step_size` (optional) - Pixels to scroll per step (number, defaults to viewport height) * `click_selector` (optional) - CSS selector to click before scrolling (string, e.g., "Load More" button) * `container` (optional) - CSS selector of container element to scroll within (string) * `idle_timeout` (optional) - Stop scrolling after this many milliseconds of no new content (number, default: 7000) * `pause_on_selector` (optional) - Pause scrolling if this selector appears above the cursor position (string) * `required` (optional) - Set to `false` to make this step optional (boolean, default: `true`) ```python Python theme={"system"} from nimble_python import Nimble nimble = Nimble(api_key="YOUR-API-KEY") # Direct form - boolean (use defaults) result = nimble.extract( url= "https://www.example.com", render= True, browser_actions= [ { "auto_scroll": True # Auto scroll with default settings } ] ) # Direct form - number (max duration) result = nimble.extract( url= "https://www.example.com", render= True, browser_actions= [ { "auto_scroll": 30000 # Auto scroll for max 30 seconds } ] ) # Extended form - with options result = nimble.extract( url= "https://www.example.com", render= True, browser_actions= [ { "auto_scroll": { "max_duration": 30000, "container": "#feed", "click_selector": ".load-more", "delay_after_scroll": 500, "idle_timeout": 7000 } } ] ) print(result) ``` ```javascript Node theme={"system"} import Nimble from '@nimble-way/nimble-js'; const nimble = new Nimble({ apiKey: "YOUR-API-KEY" }); // Direct form - boolean (use defaults) const result = await nimble.extract({ url: "https://www.example.com", render: true, browser_actions: [ { auto_scroll: true // Auto scroll with default settings } ] }); // Direct form - number (max duration) const result2 = await nimble.extract({ url: "https://www.example.com", render: true, browser_actions: [ { auto_scroll: 30000 // Auto scroll for max 30 seconds } ] }); // Extended form - with options const result3 = await nimble.extract({ url: "https://www.example.com", render: true, browser_actions: [ { auto_scroll: { max_duration: 30000, container: "#feed", click_selector: ".load-more", delay_after_scroll: 500, idle_timeout: 7000 } } ] }); console.log(result); ``` ```bash cURL theme={"system"} # Direct form - boolean curl -X POST 'https://sdk.nimbleway.com/v1/extract' \ --header 'Authorization: Bearer ' \ --header 'Content-Type: application/json' \ --data-raw '{ "url": "https://www.example.com", "render": true, "browser_actions": [{ "auto_scroll": true }] }' # Direct form - number curl -X POST 'https://sdk.nimbleway.com/v1/extract' \ --header 'Authorization: Bearer ' \ --header 'Content-Type: application/json' \ --data-raw '{ "url": "https://www.example.com", "render": true, "browser_actions": [{ "auto_scroll": 30000 }] }' # Extended form - with options curl -X POST 'https://sdk.nimbleway.com/v1/extract' \ --header 'Authorization: Bearer ' \ --header 'Content-Type: application/json' \ --data-raw '{ "url": "https://www.example.com", "render": true, "browser_actions": [{ "auto_scroll": { "max_duration": 30000, "container": "#feed", "click_selector": ".load-more", "delay_after_scroll": 500, "idle_timeout": 7000 } }] }' ``` Auto scroll continuously scrolls the page until `idle_timeout` milliseconds pass with no new content appearing, or until `max_duration` is reached. Use `click_selector` to handle "Load More" buttons. Capture a screenshot of the page and return as base64-encoded image. **Parameters:** **Direct form** - Pass boolean to capture full page screenshot: * `screenshot` (required) - Set to `true` to capture full page PNG screenshot **Extended form** - Use object with options: * `full_page` (optional) - Capture full scrollable page (boolean, default: `true`) * `format` (optional) - Image format: `'png'` (default), `'jpeg'`, or `'webp'` * `quality` (optional) - Image quality for jpeg/webp (number, 0-100, default: 90) ```python Python theme={"system"} from nimble_python import Nimble nimble = Nimble(api_key="YOUR-API-KEY") # Direct form - full page PNG screenshot result = nimble.extract( url= "https://www.example.com", render= True, browser_actions= [ { "screenshot": True } ] ) # Extended form - with options result = nimble.extract( url= "https://www.example.com", render= True, browser_actions= [ { "screenshot": { "full_page": False, "format": "jpeg", "quality": 90 } } ] ) print(result) ``` ```javascript Node theme={"system"} import Nimble from '@nimble-way/nimble-js'; const nimble = new Nimble({ apiKey: "YOUR-API-KEY" }); // Direct form - full page PNG screenshot const result = await nimble.extract({ url: "https://www.example.com", render: true, browser_actions: [ { screenshot: true } ] }); // Extended form - with options const result2 = await nimble.extract({ url: "https://www.example.com", render: true, browser_actions: [ { screenshot: { full_page: false, format: "jpeg", quality: 90 } } ] }); console.log(result); ``` ```bash cURL theme={"system"} # Direct form curl -X POST 'https://sdk.nimbleway.com/v1/extract' \ --header 'Authorization: Bearer ' \ --header 'Content-Type: application/json' \ --data-raw '{ "url": "https://www.example.com", "render": true, "browser_actions": [{ "screenshot": true }] }' # Extended form curl -X POST 'https://sdk.nimbleway.com/v1/extract' \ --header 'Authorization: Bearer ' \ --header 'Content-Type: application/json' \ --data-raw '{ "url": "https://www.example.com", "render": true, "browser_actions": [{ "screenshot": { "full_page": false, "format": "jpeg", "quality": 90 } }] }' ``` Retrieve all cookies set in the browser for the current page. **Parameters:** **Direct form** - Pass boolean to get all cookies: * `get_cookies` (required) - Set to `true` to retrieve all cookies **Extended form** - Use object with options: * `domain` (optional) - Domain filter for cookies (string, e.g., ".example.com") ```python Python theme={"system"} from nimble_python import Nimble nimble = Nimble(api_key="YOUR-API-KEY") # Direct form - get all cookies result = nimble.extract( url= "https://www.example.com", render= True, browser_actions= [ { "get_cookies": True } ] ) # Extended form - with domain filter result = nimble.extract( url= "https://www.example.com", render= True, browser_actions= [ { "get_cookies": { "domain": ".example.com" } } ] ) print(result) ``` ```javascript Node theme={"system"} import Nimble from '@nimble-way/nimble-js'; const nimble = new Nimble({ apiKey: "YOUR-API-KEY" }); // Direct form - get all cookies const result = await nimble.extract({ url: "https://www.example.com", render: true, browser_actions: [ { get_cookies: true } ] }); // Extended form - with domain filter const result2 = await nimble.extract({ url: "https://www.example.com", render: true, browser_actions: [ { get_cookies: { domain: ".example.com" } } ] }); console.log(result); ``` ```bash cURL theme={"system"} # Direct form curl -X POST 'https://sdk.nimbleway.com/v1/extract' \ --header 'Authorization: Bearer ' \ --header 'Content-Type: application/json' \ --data-raw '{ "url": "https://www.example.com", "render": true, "browser_actions": [{ "get_cookies": true }] }' # Extended form curl -X POST 'https://sdk.nimbleway.com/v1/extract' \ --header 'Authorization: Bearer ' \ --header 'Content-Type: application/json' \ --data-raw '{ "url": "https://www.example.com", "render": true, "browser_actions": [{ "get_cookies": { "domain": ".example.com" } }] }' ``` Make HTTP requests from the browser context and return the response. **Parameters:** **Direct form** - Pass URL as a string: * `fetch` (required) - URL for the HTTP request (GET method) **Extended form** - Use object with options: * `url` (required) - URL to request (string) * `method` (optional) - HTTP method: `'GET'` (default), `'POST'`, `'PUT'`, `'DELETE'`, `'PATCH'`, etc. * `headers` (optional) - HTTP headers to send as key-value object * `body` (optional) - Request body as string (for POST, PUT, PATCH) * `timeout` (optional) - Maximum time in milliseconds to wait for response (number, default: 15000) ```python Python theme={"system"} from nimble_python import Nimble nimble = Nimble(api_key="YOUR-API-KEY") # Direct form - simple GET request result = nimble.extract( url= "https://www.example.com", render= True, browser_actions= [ { "fetch": "https://api.example.com/data" # Simple GET request } ] ) # Extended form - POST with headers and body result = nimble.extract( url= "https://www.example.com", render= True, browser_actions= [ { "fetch": { "url": "https://api.example.com/submit", "method": "POST", "headers": { "Content-Type": "application/json" }, "body": '{"key": "value"}', "timeout": 15000 } } ] ) print(result) ``` ```javascript Node theme={"system"} import Nimble from '@nimble-way/nimble-js'; const nimble = new Nimble({ apiKey: "YOUR-API-KEY" }); // Direct form - simple GET request const result = await nimble.extract({ url: "https://www.example.com", render: true, browser_actions: [ { fetch: "https://api.example.com/data" // Simple GET request } ] }); // Extended form - POST with headers and body const result2 = await nimble.extract({ url: "https://www.example.com", render: true, browser_actions: [ { fetch: { url: "https://api.example.com/submit", method: "POST", headers: { "Content-Type": "application/json" }, body: '{"key": "value"}', timeout: 15000 } } ] }); console.log(result); ``` ```bash cURL theme={"system"} # Direct form curl -X POST 'https://sdk.nimbleway.com/v1/extract' \ --header 'Authorization: Bearer ' \ --header 'Content-Type: application/json' \ --data-raw '{ "url": "https://www.example.com", "render": true, "browser_actions": [{ "fetch": "https://api.example.com/data" }] }' # Extended form curl -X POST 'https://sdk.nimbleway.com/v1/extract' \ --header 'Authorization: Bearer ' \ --header 'Content-Type: application/json' \ --data-raw '{ "url": "https://www.example.com", "render": true, "browser_actions": [{ "fetch": { "url": "https://api.example.com/submit", "method": "POST", "headers": { "Content-Type": "application/json" }, "body": "{\"key\": \"value\"}", "timeout": 15000 } }] }' ``` The first `fetch` in a `browser_actions` is free. Starting from the second request, each additional `fetch` is billed as a VX6 request. You can chain multiple actions together in the `browser_actions` array. They execute sequentially. ```python Python theme={"system"} from nimble_python import Nimble nimble = Nimble(api_key="YOUR-API-KEY") result = nimble.extract( url= "https://www.example.com", render= True, browser_actions= [ { "fill": { "selectors": ["input[type='search']"], "text": "laptop" } }, { "wait": "1s" }, { "click": { "selectors": ["button[type='submit']"] } }, { "wait_for_element": { "selector": ".results", "timeout": 5000 } } ] ) print(result) ``` ```javascript Node theme={"system"} import Nimble from '@nimble-way/nimble-js'; const nimble = new Nimble({ apiKey: "YOUR-API-KEY" }); const result = await nimble.extract({ url: "https://www.example.com", render: true, browser_actions: [ { fill: { selectors: ["input[type='search']"], text: "laptop" } }, { wait: "1s" }, { click: { selectors: ["button[type='submit']"] } }, { wait_for: { selectors: [".results"], timeout: "5s" } } ] }); console.log(result); ``` ```bash cURL theme={"system"} curl -X POST 'https://sdk.nimbleway.com/v1/extract' \ --header 'Authorization: Bearer ' \ --header 'Content-Type: application/json' \ --data-raw '{ "url": "https://www.example.com", "render": true, "browser_actions": [ { "fill": { "selectors": ["input[type='\''search'\'']"], "text": "laptop" } }, { "wait": "1s" }, { "click": { "selectors": ["button[type='\''submit'\'']"] } }, { "wait_for_element": { "selector": ".results", "timeout": 5000 } } ] }' ``` ## Example response When browser actions complete successfully, you'll receive the final page state along with any data captured. The response includes: * **data**: All related extacted data * **data.html**: Final DOM state after all actions * **data.screenshot**: Page screenshot if used * **data.cookies**: Collected cookies if `get_cookies` was used by order * **data.fetch**: HTTP response if `fetch` was used by order * **metadata**: Execution details including task id, driver used, execution time and more * **metadata.browser\_actions**: The browser actions results per step ```json theme={"system"} { "url": "https://www.example.com/", "task_id": "b1fa7943-cba5-4ec2-a88c-4d2d6799c794", "status": "success", "data": { "html": "...", "screenshot": "base64_encoded_image_data", "cookies": [ { "name": "session_id", "value": "abc123xyz", "domain": ".example.com", "path": "/", "expires": 1735689600, "httpOnly": true, "secure": true } ], "fetch": [ { "status": 200, "duration": 235, "method": "POST", "url": "https://api.example.com/submit", "headers": {}, "body": "..." } ] }, "metadata": { "query_time": "2026-02-08T22:00:36.132Z", "query_duration": 1877, "response_parameters": { "input_url": "https://www.example.com/" }, "driver": "vx6" }, "status_code": 200 } ``` ## Best practices ### Use required parameter for optional steps **Make non-critical steps optional:** ```python theme={"system"} # ✅ Continue even if optional element isn't found browser_actions = [ { "click": { "selectors": [".cookie-banner button"], "required": False # Don't fail if banner doesn't exist } }, { "click": { "selectors": [".submit-button"], "required": True # This is critical } } ] # ❌ Fail entire request if cookie banner isn't found browser_actions = [ { "click": { "selectors": [".cookie-banner button"] # required defaults to True } } ] ``` ### Chain actions with appropriate waits **Add delays between actions:** ```python theme={"system"} # ✅ Wait for content to load between actions browser_actions = [ { "click": { "selectors": ["button.load-more"] } }, { "wait": "2s" # Wait for new content to load }, { "wait_for_element": { "selector": ".new-content" } } ] # ❌ No wait - may act on stale content browser_actions = [ { "click": { "selectors": ["button.load-more"] } }, { "click": { "selectors": [".new-content button"] # May not exist yet } } ] ``` ### Use specific selectors **Be precise with CSS selectors:** ```python theme={"system"} # ✅ Specific selector selector = "button[data-testid='submit-form']" # ✅ Unique class or ID selector = "#unique-submit-button" # ❌ Too generic - may match wrong element selector = "button" # ❌ Fragile - breaks with style changes selector = ".btn.btn-primary.btn-lg.mt-4" ``` ### Optimize auto scroll parameters **Set appropriate scroll limits:** ```python theme={"system"} # ✅ Controlled scrolling with limits auto_scroll = { "max_duration": 10000, # 10 seconds max "idle_timeout": 3000, # Stop if no change for 3s "delay_after_scroll": 500, # Wait between scrolls "click_selector": ".load-more" # Handle load more buttons } # ❌ Excessive scrolling - slow and costly auto_scroll = { "max_duration": 60000, # 60 seconds - too long "delay_after_scroll": 100 # Too fast - may miss content }example ``` # Formats Source: https://docs.nimbleway.com/nimble-sdk/web-tools/extract/features/formats Control response format and included data types Format options control what data types are included in your response. Specify one or more formats to receive HTML, markdown, screenshots, or extracted links alongside your extracted data. Common uses: * **Full page content**: Get raw HTML for custom processing * **Readable text**: Convert pages to clean markdown format * **Visual records**: Capture screenshots for monitoring or archival * **Link extraction**: Get all URLs from the page for further crawling You can combine multiple formats in a single request. All specified formats will be included in the response. ## Parameters Choose which types of content you want in your response. You can request multiple formats at once, and each will be included in the result. **Available formats:** * `html` - The full HTML source code of the page * *Best for:* Custom parsing, preserving exact page structure, accessing all DOM elements * `markdown` - Clean, readable markdown version of the page * *Best for:* Content analysis, LLM processing, human-readable output * `screenshot` - Full-page screenshot as base64-encoded PNG * *Best for:* Visual verification, monitoring, archival, debugging * *Note:* Automatically enables rendering (VX8/VX10 driver) **Example:** ```json theme={"system"} "formats": ["html", "markdown", "screenshot"] ``` ## Usage ### HTML format Request one format type - html (default). **Best for:** * Custom HTML parsing * Preserving exact page structure * Accessing all DOM elements and attributes * Archival purposes ```python Python theme={"system"} from nimble_python import Nimble nimble = Nimble(api_key="YOUR-API-KEY") result = nimble.extract( url= "https://www.google.com/search?q=nimble", formats= ["html"] # default ) # Access HTML content html_content = result.data.html print(html_content) ``` ```javascript Node theme={"system"} import Nimble from "@nimble-way/nimble-js"; const nimble = new Nimble({ apiKey: "YOUR-API-KEY" }); const result = await nimble.extract({ url: "https://www.google.com/search?q=nimble", formats: ["html"], // default }); // Access HTML content const htmlContent = result.data.html; console.log(htmlContent); ``` ```bash cURL theme={"system"} curl -X POST 'https://sdk.nimbleway.com/v1/extract' \ --header 'Authorization: Bearer ' \ --header 'Content-Type: application/json' \ --data-raw '{ "url": "https://www.google.com/search?q=nimble", "formats": ["html"] }' ``` ### Markdown format Convert the page to clean, readable markdown. **Best for:** * Clean text extraction * Content analysis * LLM processing * Human-readable output: ```python Python theme={"system"} from nimble_python import Nimble nimble = Nimble(api_key="YOUR-API-KEY") result = nimble.extract( url= "https://www.amazon.com/s?k=ironflask", formats= ["markdown"] ) # Access markdown content markdown_content = result.data.markdown print(markdown_content) ``` ```javascript Node theme={"system"} import Nimble from "@nimble-way/nimble-js"; const nimble = new Nimble({ apiKey: "YOUR-API-KEY" }); const result = await nimble.extract({ url: "https://www.amazon.com/s?k=ironflask", formats: ["markdown"], }); // Access markdown content const markdownContent = result.data.markdown; console.log(markdownContent); ``` ```bash cURL theme={"system"} curl -X POST 'https://sdk.nimbleway.com/v1/extract' \ --header 'Authorization: Bearer ' \ --header 'Content-Type: application/json' \ --data-raw '{ "url": "https://www.amazon.com/s?k=ironflask", "formats": ["markdown"] }' ``` ### Screenshot format Capture a full-page screenshot as a base64-encoded PNG image. **Best for:** * Visual verification and monitoring * Page archival and documentation * Comparing page changes over time Screenshot format automatically enables JavaScript rendering (VX8 or VX10 driver), which may affect pricing. See [Pricing](/nimble-sdk/admin/pricing) for driver costs. ```python Python theme={"system"} from nimble_python import Nimble import base64 nimble = Nimble(api_key="YOUR-API-KEY") result = nimble.extract( url="https://www.nimbleway.com", formats=["screenshot"] ) # Access screenshot (base64-encoded PNG) screenshot_base64 = result.data.screenshot # Save to file with open("screenshot.png", "wb") as f: f.write(base64.b64decode(screenshot_base64)) print("Screenshot saved to screenshot.png") ``` ```javascript Node theme={"system"} import Nimble from "@nimble-way/nimble-js"; import fs from "fs"; const nimble = new Nimble({ apiKey: "YOUR-API-KEY" }); const result = await nimble.extract({ url: "https://www.nimbleway.com", formats: ["screenshot"], }); // Access screenshot (base64-encoded PNG) const screenshotBase64 = result.data.screenshot; // Save to file fs.writeFileSync("screenshot.png", Buffer.from(screenshotBase64, "base64")); console.log("Screenshot saved to screenshot.png"); ``` ```bash cURL theme={"system"} curl -X POST 'https://sdk.nimbleway.com/v1/extract' \ --header 'Authorization: Bearer ' \ --header 'Content-Type: application/json' \ --data-raw '{ "url": "https://www.nimbleway.com", "formats": ["screenshot"] }' ``` ### Multiple formats Combine multiple formats to get different data representations: ```python Python theme={"system"} from nimble_python import Nimble nimble = Nimble(api_key="YOUR-API-KEY") result = nimble.extract( url="https://www.nimbleway.com", formats=["html", "markdown", "screenshot"] ) print(result.data.html) print(result.data.markdown) print(result.data.screenshot) # base64-encoded PNG ``` ```javascript Node theme={"system"} import Nimble from "@nimble-way/nimble-js"; const nimble = new Nimble({ apiKey: "YOUR-API-KEY" }); const result = await nimble.extract({ url: "https://www.nimbleway.com", formats: ["html", "markdown", "screenshot"], }); console.log(result.data.html); console.log(result.data.markdown); console.log(result.data.screenshot); // base64-encoded PNG ``` ```bash cURL theme={"system"} curl -X POST 'https://sdk.nimbleway.com/v1/extract' \ --header 'Authorization: Bearer ' \ --header 'Content-Type: application/json' \ --data-raw '{ "url": "https://www.nimbleway.com", "formats": ["html", "markdown", "screenshot"] }' ``` When combining screenshot with other formats, rendering is automatically enabled for all formats in the request. ## Example response When formats are specified, all requested data is included in the response. The response includes: * **data.html**: Raw HTML if requested * **data.markdown**: Converted markdown if requested * **data.screenshot**: Base64-encoded PNG if requested * **data.links**: Array of extracted URLs if requested * **data.parsing**: Structured data if parsing was used * **metadata**: Execution details and formats included: ```json theme={"system"} { "url": "https://www.example.com/", "task_id": "b1fa7943-cba5-4ec2-a88c-4d2d6799c794", "status": "success", "data": { "html": "......", "markdown": "# Article Title\n\nThis is the article content...", "screenshot": "iVBORw0KGgoAAAANSUhEUgAAA...", "links": [ "https://www.example.com/about", "https://www.example.com/contact", "https://www.example.com/products", "https://external-site.com" ], "parsing": { "title": "Example Article", "author": "John Doe" } }, "metadata": { "query_time": "2026-02-08T22:00:36.132Z", "query_duration": 1877, "response_parameters": { "input_url": "https://www.example.com/" }, "driver": "vx6" }, "status_code": 200 } ``` ## Best practices ### Format selection **Choose formats based on your needs:** * Use `html` when you need full DOM access * Use `markdown` for clean text and content analysis * Use `screenshot` for visual verification * Use `links` for discovering URLs to crawl **Avoid unnecessary formats:** ```python theme={"system"} # ❌ Don't request all formats if you only need one formats=["html", "markdown"] # ✅ Request only what you need formats=["markdown"] ``` ### Performance considerations * Each format adds processing time * Screenshots require rendering and are slower * HTML and markdown are faster to generate * Request only needed formats for optimal performance # Geo Targeting Source: https://docs.nimbleway.com/nimble-sdk/web-tools/extract/features/geo-targeting Built-in residential proxy allowing Geo-targeting options Geo Targeting routes each request through a proxy exit in a specific region. Use it when content changes by location or language. Common uses: * **Localized content**: pricing, availability, shipping, store pages. * **SEO & market research**: region-specific SERPs and ads. * **Fewer blocks**: look like a real local user with residential exits. ## Parameters Make your request appear as if it's coming from a specific country. Perfect for accessing region-specific content like localized prices or availability. **When to use:** * E-commerce sites with different prices by country * Content that varies by region * Accessing geo-restricted pages * SEO analysis for different markets Use ISO Alpha-2 Country Codes i.e. `US`, `DE`, `GB`.\ Use `ALL` for random country selection. **Example:** ```json theme={"system"} "country": "GB" ``` For US or CA locations only - specify which state your request should come from. Use ISO Alpha-2 Country Codes i.e. `NY`, `AZ`, `CA` **Note:** Only works when `country` is set to `US` or `CA`. Useful for state-specific content like local store inventory or regional pricing. **Example:** ```json theme={"system"} "country": "US", "state": "CA" ``` Target a specific city for hyper-local content. Works best when combined with country (and state for US cities). **When to use:** * Local business listings * City-specific promotions * Store availability by location * Hyper-local content **Examples:** `"new_york"`, `"london"`, `"paris"` **Example:** ```json theme={"system"} "country": "US", "state": "NY", "city": "new_york" ``` Replace spaces from city names with underscore (e.g,`New York` becomes `new_york`). Set the browser's preferred language. This changes the `Accept-Language` header and tells websites which language you prefer. **Most common locales:** * `en-US` - English (United States) * `en-GB` - English (United Kingdom) * `fr-FR` - French (France) * `de-DE` - German (Germany) * `es-ES` - Spanish (Spain) * `ja-JP` - Japanese (Japan) * `zh-CN` - Chinese (China) * `pt-BR` - Portuguese (Brazil) * `auto` - Automatically match the country **Best practice:** Match your locale to your country (e.g., `fr-FR` with `country: "FR"`) for the most authentic experience. **Example:** ```json theme={"system"} "country": "FR", "locale": "fr-FR" ``` `country`/`state`/`city` geo targeting changes **where** the request comes from. `locale` changes **which language** the browser prefers. ## Usage This request exits from **New York** and sets the browser language to **English**. ```python Python theme={"system"} from nimble_python import Nimble nimble = Nimble(api_key="YOUR-API-KEY") result = nimble.extract( url="https://ipinfo.io/json", country="US", state="NY", locale="en-US", ) print(result.data.html) ``` ```javascript Node theme={"system"} import Nimble from "@nimble-way/nimble-js"; const nimble = new Nimble({ apiKey: "YOUR-API-KEY" }); const result = await nimble.extract("https://ipinfo.io/json", { country: "US", state: "NY", locale: "en-US", }); console.log(result.data.html); ``` ```bash cURL theme={"system"} curl -X POST 'https://sdk.nimbleway.com/v1/extract' \ --header 'Authorization: Bearer ' \ --header 'Content-Type: application/json' \ --data-raw '{ "url": "https://ipinfo.io/json", "country": "US", "state": "NY", "locale": "en-US" }' ``` ## Best practices ### Use geo targeting when content varies by location Only specify country/state/city when the content differs by location. ```python theme={"system"} # ✅ Use for location-specific content result = nimble.extract({ "url": "https://www.example.com/products", "country": "GB" # Prices and availability vary by country }) # ❌ Unnecessary for global content result = nimble.extract({ "url": "https://www.example.com/about", "country": "DE" # About page is the same everywhere }) ``` ### Match locale to target country Set the locale to match the country for accurate localized content. ```python theme={"system"} # ✅ Locale matches country result = nimble.extract({ "url": "https://www.example.com", "country": "FR", "locale": "fr-FR" }) # ✅ Locale auto matching country result = nimble.extract({ "url": "https://www.example.com", "country": "FR", "locale": "auto" }) # ❌ Locale doesn't match country result = nimble.extract({ "url": "https://www.example.com", "country": "FR", "locale": "en-US" # May not get French content }) ``` ### Start with country-level targeting Use country-level targeting first, then add state/city if needed. ```python theme={"system"} # ✅ Start simple with country result = nimble.extract({ "url": "https://www.example.com", "country": "US" }) # ✅ Add state if content differs result = nimble.extract({ "url": "https://www.example.com", "country": "US", "state": "NY" # Only if NY shows different content }) # ❌ Overly specific when not needed result = nimble.extract({ "url": "https://www.example.com/global-page", "country": "US", "state": "NY", # Unnecessary granularity "city": "new_york" }) ``` # Custom Headers & Cookies Source: https://docs.nimbleway.com/nimble-sdk/web-tools/extract/features/headers-and-cookies Send custom headers and cookies to access personalized or authenticated content Custom headers and cookies enable you to access personalized data, maintain user sessions, and interact with authenticated endpoints. Set headers and cookies in string or object format to mimic real user behavior and retrieve content that requires specific request contexts. ## When to use Use custom headers and cookies when you need to: * **Access authenticated content**: Retrieve data behind login walls * **Maintain sessions**: Preserve user state across requests * **Mimic real users**: Include user-specific headers for personalized data * **API authentication**: Send API keys or tokens in headers * **Regional content**: Set language or region preferences This is an advanced feature. Misconfigured headers or cookies may trigger anti-bot detection or cause requests to fail. ## Parameters Add custom HTTP headers to your request. Useful for authentication, setting custom user agents, or mimicking specific browsers. **Common use cases:** * API authentication (API keys, tokens) * Custom User-Agent strings * Accept-Language preferences * Referer headers * Custom authentication headers **Format:** Provide headers as key-value pairs in an object. **Example:** ```json theme={"system"} "headers": { "User-Agent": "MyApp/1.0", "Accept-Language": "en-US,en;q=0.9", "X-API-Key": "your-api-key" } ``` The HTTP method to use for the request. **Supported methods:** `GET`, `POST`, `PUT`, `DELETE`, `PATCH` **When to use:** * `GET` - Fetching data (default) * `POST` - Submitting forms or data * `PUT` - Updating resources * `DELETE` - Removing resources **Note:** When using `POST` or `PUT`, you'll typically also need to include a `body` parameter. **Example:** ```json theme={"system"} "method": "POST" ``` The data to send in the request body. Used with POST, PUT, or PATCH requests. **Format:** Can be an object (auto-converted to JSON) or a string. **Common uses:** * Form submissions * API payloads * Search queries * Data uploads **Example:** ```json theme={"system"} "method": "POST", "body": { "query": "product search", "filters": ["category:electronics"] } ``` Send cookies as a simple string, just like they appear in the browser. Separate multiple cookies with semicolons. **When to use:** * Quickest way to add cookies * When you have a cookie string from browser dev tools * Session IDs and authentication tokens **Format:** `name1=value1;name2=value2;name3=value3` **Example:** ```json theme={"system"} "cookies": "session_id=abc123xyz;user_pref=en_US;logged_in=true" ``` Send cookies as an array of objects for more control. Each cookie can specify its domain. **When to use:** * Different cookies for different domains * More precise cookie control * Complex cookie setups **Each cookie object needs:** * `key` - Cookie name (required) * `value` - Cookie value (required) * `domain` - Cookie domain (optional, defaults to request URL domain) **Example:** ```json theme={"system"} "cookies": [ { "key": "session_id", "value": "abc123xyz", "domain": "example.com" }, { "key": "user_token", "value": "token456", "domain": "api.example.com" } ] ``` Headers and cookies **cannot** be used together. Please do not include any cookies when sending custom headers.\ \ Just remember that cookies can be provided in either string format OR array format, but not both at the same time. ## Usage ### Send custom headers Include headers as an object: ```python Python theme={"system"} from nimble_python import Nimble nimble = Nimble(api_key="YOUR-API-KEY") result = nimble.extract( url= "https://api.example.com/data", headers= { "User-Agent": "MyApp/1.0", "Accept-Language": "en-US,en;q=0.9", "Some-Extra-Header": "some-extra-header" } ) print(result) ``` ```javascript Node theme={"system"} import Nimble from "@nimble-way/nimble-js"; const nimble = new Nimble({ apiKey: "YOUR-API-KEY" }); const result = await nimble.extract({ url: "https://api.example.com/data", headers: { "User-Agent": "MyApp/1.0", "Accept-Language": "en-US,en;q=0.9", "Some-Extra-Header": "some-extra-header", }, }); console.log(result); ``` ```bash cURL theme={"system"} curl -X POST 'https://sdk.nimbleway.com/v1/extract' \ --header 'Authorization: Bearer ' \ --header 'Content-Type: application/json' \ --data-raw '{ "url": "https://api.example.com/data", "headers": { "User-Agent": "MyApp/1.0", "Accept-Language": "en-US,en;q=0.9", "Some-Extra-Header": "some-extra-header" } }' ``` ### POST requests with headers Send POST requests with custom headers and body: ```python Python theme={"system"} from nimble_python import Nimble nimble = Nimble(api_key="YOUR-API-KEY") result = nimble.extract( url= "https://api.example.com/submit", method= "POST", headers= { "Content-Type": "application/json", "Some-Extra-Header": "some-extra-header" }, body= { "query": "product search", "filters": ["category:electronics"] } ) print(result) ``` ```javascript Node theme={"system"} import Nimble from "@nimble-way/nimble-js"; const nimble = new Nimble({ apiKey: "YOUR-API-KEY" }); const result = await nimble.extract({ url: "https://api.example.com/submit", method: "POST", headers: { "Content-Type": "application/json", "Some-Extra-Header": "some-extra-header", }, body: { query: "product search", filters: ["category:electronics"], }, }); console.log(result); ``` ```bash cURL theme={"system"} curl -X POST 'https://sdk.nimbleway.com/v1/extract' \ --header 'Authorization: Bearer ' \ --header 'Content-Type: application/json' \ --data-raw '{ "url": "https://api.example.com/submit", "method": "POST", "headers": { "Content-Type": "application/json", "Some-Extra-Header": "some-extra-header" }, "body": { "query": "product search", "filters": ["category:electronics"] } }' ``` ### Send cookies (string format) Use simple string format for multiple cookies: ```python Python theme={"system"} from nimble_python import Nimble nimble = Nimble(api_key="YOUR-API-KEY") result = nimble.extract( url= "https://www.example.com/account", cookies= "session_id=abc123xyz;user_pref=en_US;logged_in=true" ) print(result) ``` ```javascript Node theme={"system"} import Nimble from "@nimble-way/nimble-js"; const nimble = new Nimble({ apiKey: "YOUR-API-KEY" }); const result = await nimble.extract({ url: "https://www.example.com/account", cookies: "session_id=abc123xyz;user_pref=en_US;logged_in=true", }); console.log(result); ``` ```bash cURL theme={"system"} curl -X POST 'https://sdk.nimbleway.com/v1/extract' \ --header 'Authorization: Bearer ' \ --header 'Content-Type: application/json' \ --data-raw '{ "url": "https://www.example.com/account", "cookies": "session_id=abc123xyz;user_pref=en_US;logged_in=true" }' ``` ### Send cookies (object format) Use object format for domain-specific cookies: ```python Python theme={"system"} from nimble_python import Nimble nimble = Nimble(api_key="YOUR-API-KEY") result = nimble.extract( url= "https://www.example.com/dashboard", cookies= [ { "key": "session_id", "value": "abc123xyz", "domain": "example.com" }, { "key": "user_token", "value": "token456", "domain": "api.example.com" } ] ) print(result) ``` ```javascript Node theme={"system"} import Nimble from "@nimble-way/nimble-js"; const nimble = new Nimble({ apiKey: "YOUR-API-KEY" }); const result = await nimble.extract({ url: "https://www.example.com/dashboard", cookies: [ { key: "session_id", value: "abc123xyz", domain: "example.com", }, { key: "user_token", value: "token456", domain: "api.example.com", }, ], }); console.log(result); ``` ```bash cURL theme={"system"} curl -X POST 'https://sdk.nimbleway.com/v1/extract' \ --header 'Authorization: Bearer ' \ --header 'Content-Type: application/json' \ --data-raw '{ "url": "https://www.example.com/dashboard", "cookies": [ { "key": "session_id", "value": "abc123xyz", "domain": "example.com" }, { "key": "user_token", "value": "token456", "domain": "api.example.com" } ] }' ``` ### Capture cookies with browser actions and reuse Use browser actions to interact with a page (like selecting a zip code), capture the resulting cookies, and pass them to subsequent requests: ```python Python theme={"system"} from nimble_python import Nimble nimble = Nimble(api_key="YOUR-API-KEY") # Step 1: Perform browser actions and capture cookies initial_result = nimble.extract( url= "https://www.example.com", render= True, browser_actions= [ { "click": { "selector": "#location-selector" } }, { "fill": { "selector": "#zipcode-input", "text": "90210" } }, { "click": { "selector": "#submit-location" } }, { "wait": 2000 }, { "get_cookies": True } ] ) # Extract cookies from the response captured_cookies = initial_result.data.cookies # Step 2: Use captured cookies in subsequent requests # Convert cookies array to string format cookie_string = ";".join([f"{cookie['name']}={cookie['value']}" for cookie in captured_cookies]) result = nimble.extract( url= "https://www.example.com/products", cookies= cookie_string ) print(result) ``` ```javascript Node theme={"system"} import Nimble from "@nimble-way/nimble-js"; const nimble = new Nimble({ apiKey: "YOUR-API-KEY" }); // Step 1: Perform browser actions and capture cookies const initialResult = await nimble.extract({ url: "https://www.example.com", render: true, browser_actions: [ { click: { selector: "#location-selector", }, }, { fill: { selector: "#zipcode-input", text: "90210", }, }, { click: { selector: "#submit-location", }, }, { wait: 2000, }, { get_cookies: true, }, ], }); // Extract cookies from the response const capturedCookies = initialResult.data.cookies; // Step 2: Use captured cookies in subsequent requests // Convert cookies array to string format const cookieString = capturedCookies .map((cookie) => `${cookie.name}=${cookie.value}`) .join(";"); const result = await nimble.extract({ url: "https://www.example.com/products", cookies: cookieString, }); console.log(result); ``` ```bash cURL theme={"system"} # Step 1: Perform browser actions and capture cookies curl -X POST 'https://sdk.nimbleway.com/v1/extract' \ --header 'Authorization: Bearer ' \ --header 'Content-Type: application/json' \ --data-raw '{ "url": "https://www.example.com", "render": true, "browser_actions": [ { "click": { "selector": "#location-selector" } }, { "fill": { "selector": "#zipcode-input", "text": "90210" } }, { "click": { "selector": "#submit-location" } }, { "wait": 2000 }, { "get_cookies": true } ] }' # Response will include cookies in data.cookies array: # { # "status": "success", # "data": { # "html": "...", # "cookies": [ # { # "name": "location_zipcode", # "value": "90210", # "domain": ".example.com" # }, # { # "name": "session_id", # "value": "abc123xyz", # "domain": ".example.com" # } # ] # } # } # Step 2: Use captured cookies in subsequent requests curl -X POST 'https://sdk.nimbleway.com/v1/extract' \ --header 'Authorization: Bearer ' \ --header 'Content-Type: application/json' \ --data-raw '{ "url": "https://www.example.com/products", "cookies": "location_zipcode=90210;session_id=abc123xyz" }' ``` This workflow is useful when you need to set user preferences (like location, language, or filters) via browser interaction, then make multiple faster requests with the same session state. See [Browser Actions](/nimble-sdk/web-tools/extract/features/browser-actions) for more interaction options. ## Important constraints **Don't mix headers and cookies:** ```python theme={"system"} # ❌ Don't send both in same request result = nimble.extract({ "url": "https://www.example.com", "headers": {"Some-Extra-Header": "some-extra-header"}, "cookies": "session_id=abc" # May cause conflicts }) # ✅ Use one method result = nimble.extract({ "url": "https://www.example.com", "headers": {"Some-Extra-Header": "some-extra-header"} }) ``` **Detection risk:** * Improper headers may trigger anti-bot detection * Invalid cookies can cause request failures * Test thoroughly before production use * Monitor success rates when using custom headers/cookies **Header size limits:** * Total header size: 32 KB * Individual header: 8 KB * Cookie header: 4 KB This is an advanced feature requiring careful configuration. Always test in a development environment first. # JavaScript Rendering Source: https://docs.nimbleway.com/nimble-sdk/web-tools/extract/features/js-rendering Control browser rendering for dynamic content and JavaScript execution JavaScript rendering enables full browser execution to capture dynamically loaded content, handle user interactions, and process JavaScript-heavy websites. Control rendering behavior with advanced options for timeout, iframe handling, and load detection. ## When to use Use JavaScript rendering when you need to: * **Load dynamic content**: Capture content loaded via AJAX or fetch requests * **Execute JavaScript**: Process sites that require JS for content display * **Handle SPAs**: Extract data from Single Page Applications (React, Vue, Angular) * **Wait for interactions**: Capture DOM state after user actions * **See final state**: Get the page as users see it, not raw HTML ## Parameters Turn JavaScript rendering on or off. When enabled, the page loads in a real browser that executes JavaScript, just like when you visit it yourself. **When to enable:** * Single Page Applications (React, Vue, Angular) * Content loaded via AJAX or fetch * Sites that need JavaScript to display content * When you need the page as users see it **When to keep disabled (faster, cheaper):** * Static HTML pages * Simple websites without JavaScript * When raw HTML is sufficient **Note:** Rendering requires vx8 or vx10. The vx6 driver doesn't support rendering. **Example:** ```json theme={"system"} "render": true ``` Fine-tune how the browser waits and loads the page. Only applies when `render: true`. **Available options:** * `render_type` - When to consider the page "loaded" * `load` (default) - Wait for the standard page load event - good for most pages * `domready` - Stop as soon as HTML is ready, before images load - fastest option * `idle2` - Wait until only 2 or fewer network requests in the last 500ms - good for dynamic content * `idle0` - Wait until zero network activity for 500ms - most thorough, slowest * `timeout` - Maximum wait time in milliseconds (default: 30,000) * How long to wait before giving up * `include_iframes` - Whether to load content inside iframes (default: false) * Enable for embedded videos, widgets, or important iframe content * `blocked_domains` - List of domains to block during loading * Great for blocking ads, analytics, or tracking scripts * Makes pages load faster and cleaner **Example:** ```json theme={"system"} "render_options": { "render_type": "idle0", "timeout": 60000, "include_iframes": true, "blocked_domains": ["google-analytics.com", "doubleclick.net"] } ``` ## Usage ### Enable basic rendering Set `render: true` to enable JavaScript execution: ```python Python theme={"system"} from nimble_python import Nimble nimble = Nimble(api_key="YOUR-API-KEY") result = nimble.extract( url= "https://www.google.com/search?q=nba+allstars+2026", render= True ) print(result.data.html) ``` ```javascript Node theme={"system"} import Nimble from "@nimble-way/nimble-js"; const nimble = new Nimble({ apiKey: "YOUR-API-KEY" }); const result = await nimble.extract({ url: "https://www.google.com/search?q=nba+allstars+2026", render: true, }); console.log(result.data.html); ``` ```bash cURL theme={"system"} curl -X POST 'https://sdk.nimbleway.com/v1/extract' \ --header 'Authorization: Bearer ' \ --header 'Content-Type: application/json' \ --data-raw '{ "url": "https://www.google.com/search?q=nba+allstars+2026", "render": true }' ``` When `render: false` (default), the API returns the raw HTML without JavaScript execution, suitable for static pages. ### Dynamic content loading Wait for AJAX-loaded content: ```python Python theme={"system"} from nimble_python import Nimble nimble = Nimble(api_key="YOUR-API-KEY") # Wait for all network activity to stop result = nimble.extract( url= "https://www.target.com/deals/all", render= True, render_options= { "render_type": "idle0", "timeout": 45000 } ) print(result.data.html) ``` ```javascript Node theme={"system"} import Nimble from "@nimble-way/nimble-js"; const nimble = new Nimble({ apiKey: "YOUR-API-KEY" }); // Wait for all network activity to stop const result = await nimble.extract({ url: "https://www.target.com/deals/all", render: true, render_options: { render_type: "idle0", timeout: 45000, }, }); console.log(result.data.html); ``` ```bash cURL theme={"system"} curl -X POST 'https://sdk.nimbleway.com/v1/extract' \ --header 'Authorization: Bearer ' \ --header 'Content-Type: application/json' \ --data-raw '{ "url": "https://www.google.com/search?q=nba+allstars+2026", "render": true, "render_options": { "render_type": "idle0", "timeout": 45000 } }' ``` ### Iframe content extraction Include iframe content in extraction: ```python Python theme={"system"} from nimble_python import Nimble nimble = Nimble(api_key="YOUR-API-KEY") # Include iframe content (e.g., embedded videos, widgets) result = nimble.extract( url= "https://www.example.com/page-with-iframes", render= True, render_options= { "include_iframes": True, "render_type": "load", "timeout": 40000 } ) print(result.data.html) ``` ```javascript Node theme={"system"} import Nimble from "@nimble-way/nimble-js"; const nimble = new Nimble({ apiKey: "YOUR-API-KEY" }); // Include iframe content (e.g., embedded videos, widgets) const result = await nimble.extract({ url: "https://www.example.com/page-with-iframes", render: true, render_options: { include_iframes: true, render_type: "load", timeout: 40000, }, }); console.log(result.data.html); ``` ```bash cURL theme={"system"} curl -X POST 'https://sdk.nimbleway.com/v1/extract' \ --header 'Authorization: Bearer ' \ --header 'Content-Type: application/json' \ --data-raw '{ "url": "https://www.example.com/page-with-iframes", "render": true, "render_options": { "include_iframes": true, "render_type": "load", "timeout": 40000 } }' ``` ### Block unnecessary resources Improve performance by blocking ads and tracking: ```python Python theme={"system"} from nimble_python import Nimble nimble = Nimble(api_key="YOUR-API-KEY") # Block ads, analytics, and tracking domains result = nimble.extract( url= "https://www.example.com", render= True, render_options= { "blocked_domains": [ "googletagmanager.com", "google-analytics.com", "facebook.com", "doubleclick.net", "ads.example.com" ], "render_type": "idle2", "timeout": 25000 } ) print(result.data.html) ``` ```javascript Node theme={"system"} import Nimble from "@nimble-way/nimble-js"; const nimble = new Nimble({ apiKey: "YOUR-API-KEY" }); // Block ads, analytics, and tracking domains const result = await nimble.extract({ url: "https://www.example.com", render: true, render_options: { blocked_domains: [ "googletagmanager.com", "google-analytics.com", "facebook.com", "doubleclick.net", "ads.example.com", ], render_type: "idle2", timeout: 25000, }, }); console.log(result.data.html); ``` ```bash cURL theme={"system"} curl -X POST 'https://sdk.nimbleway.com/v1/extract' \ --header 'Authorization: Bearer ' \ --header 'Content-Type: application/json' \ --data-raw '{ "url": "https://www.google.com/search?q=nba+allstars+2026", "render": true, "render_options": { "blocked_domains": [ "googletagmanager.com", "google-analytics.com", "facebook.com", "doubleclick.net", "ads.example.com" ], "render_type": "idle2", "timeout": 25000 } }' ``` ### Fast rendering for simple sites Use DOMContentLoaded for faster responses: ```python Python theme={"system"} from nimble_python import Nimble nimble = Nimble(api_key="YOUR-API-KEY") # Don't wait for images/styles to load result = nimble.extract( url= "https://www.example.com/article", render= True, render_options= { "render_type": "domready", "timeout": 15000 }, formats= ["markdown"] ) print(result.data.markdown) ``` ```javascript Node theme={"system"} import Nimble from "@nimble-way/nimble-js"; const nimble = new Nimble({ apiKey: "YOUR-API-KEY" }); // Don't wait for images/styles to load const result = await nimble.extract({ url: "https://www.example.com/article", render: true, render_options: { render_type: "domready", timeout: 15000, }, formats: ["markdown"], }); console.log(result.data.markdown); ``` ```bash cURL theme={"system"} curl -X POST 'https://sdk.nimbleway.com/v1/extract' \ --header 'Authorization: Bearer ' \ --header 'Content-Type: application/json' \ --data-raw '{ "url": "https://www.google.com/search?q=nba+allstars+2026", "render": true, "render_options": { "render_type": "domready", "timeout": 15000 }, "formats": ["markdown"] }' ``` ### Combining with browser actions Rendering works seamlessly with browser actions: ```python Python theme={"system"} from nimble_python import Nimble nimble = Nimble(api_key="YOUR-API-KEY") # Render page, then perform actions result = nimble.extract( url= "https://www.example.com", render= True, render_options= { "render_type": "idle0", "timeout": 240000 }, browser_actions= [ { "click": { "selector": "button.load-more", "timeout": 5000 } }, { "sleep": "2s" }, { "auto_scroll": 5000 } ] ) print(result) ``` ```javascript Node theme={"system"} import Nimble from "@nimble-way/nimble-js"; const nimble = new Nimble({ apiKey: "YOUR-API-KEY" }); // Render page, then perform actions const result = await nimble.extract({ url: "https://www.example.com", render: true, render_options: { render_type: "idle0", timeout: 240000, }, browser_actions: [ { click: { selector: "button.load-more", timeout: 5000, }, }, { sleep: "2s", }, { auto_scroll: 5000, }, ], }); console.log(result); ``` ```bash cURL theme={"system"} curl -X POST 'https://sdk.nimbleway.com/v1/extract' \ --header 'Authorization: Bearer ' \ --header 'Content-Type: application/json' \ --data-raw '{ "url": "https://www.google.com/search?q=nba+allstars+2026", "render": true, "render_options": { "render_type": "idle0", "timeout": 45000 }, "browser_actions": [ { "click": { "selector": "button.load-more", "timeout": 5000 } }, { "sleep": "2s" }, { "auto_scroll": 5000 } ] }' ``` ## Best practices ### Choose the right render type **`Use load for standard pages:`** ```python theme={"system"} # ✅ Default for most websites render_options = { "render_type": "load" } ``` **`Use domready for speed:`** ```python theme={"system"} # ✅ When images/styles aren't needed render_options = { "render_type": "domready" # Faster response } ``` **`Use idle2 for dynamic content:`** ```python theme={"system"} # ✅ For AJAX-heavy pages render_options = { "render_type": "idle2", # Wait for most requests to finish "timeout": 30000 } ``` **`Use idle0 for complete loading:`** ```python theme={"system"} # ✅ When you need everything loaded render_options = { "render_type": "idle0", # Wait for all network activity "timeout": 60000 # Higher timeout for slow sites } ``` ### Optimize timeout settings **Set appropriate timeouts:** ```python theme={"system"} # ❌ Too short - may miss content render_options = { "render_type": "idle0", "timeout": 5000 # Not enough for slow sites } # ✅ Balanced timeout render_options = { "render_type": "idle0", "timeout": 30000 # 30 seconds is usually sufficient } # ✅ Extended for slow sites render_options = { "render_type": "idle0", "timeout": 60000 # Up to 60 seconds for very slow sites } ``` ### Block unnecessary domains **Common domains to block:** ```python theme={"system"} common_blocked_domains = [ # Analytics "google-analytics.com", "googletagmanager.com", "segment.com", "mixpanel.com", # Advertising "doubleclick.net", "googlesyndication.com", "adservice.google.com", # Social media widgets "facebook.com", "twitter.com", "linkedin.com", # CDN for ads "ads.example.com", "tracking.example.com" ] result = nimble.extract({ "url": "https://www.example.com", "render": True, "render_options": { "blocked_domains": common_blocked_domains } }) ``` **Benefits of blocking:** * Faster page loads * Lower bandwidth usage * Reduced detection risk * Cleaner HTML output ### Handle iframes carefully **Enable only when needed:** ```python theme={"system"} # ❌ Don't enable iframes unnecessarily render_options = { "include_iframes": True # Slower and larger response } # ✅ Enable only for specific content if need_iframe_content: render_options = { "include_iframes": True, "timeout": 45000 # Increase timeout for iframes } else: render_options = { "include_iframes": False } ``` ### Combine with network capture Monitor and capture API calls during rendering: ```python theme={"system"} from nimble_python import Nimble nimble = Nimble(api_key="YOUR-API-KEY") # Render page and capture specific API calls result = nimble.extract( url= "https://www.example.com", render= True, render_options= { "render_type": "idle0", "timeout": 30000 }, network_capture= [ { "method": "GET", "url": { "type": "contains", "value": "/api/products" } } ] ) # Access captured network data api_responses = result.data.network_capture print(api_responses) ``` ### Performance optimization **Choose the minimal driver:** ```python theme={"system"} # ❌ Don't use vx10 when vx8 works result = nimble.extract({ "url": "https://simple-spa.com", "driver": "vx10", # Unnecessary stealth features "render": True }) # ✅ Use appropriate driver result = nimble.extract({ "url": "https://simple-spa.com", "driver": "vx8", # Sufficient for most SPAs "render": True }) ``` **Monitor execution time:** ```python theme={"system"} result = nimble.extract({ "url": "https://www.example.com", "render": True }) # Check rendering performance exec_time = result.metadata.query_duration print(f"Rendering took {exec_time}ms") # Optimize if too slow if exec_time > 10000: print("Consider blocking domains or changing render_type") ``` ### When to skip rendering Use non-rendering (vx6) when: * **Static content**: HTML already contains all data * **API endpoints**: Fetching JSON directly * **High throughput needed**: Rendering is slower * **Simple pages**: No JavaScript required * **Cost optimization**: Non-rendering is cheaper ```python theme={"system"} # ✅ Skip rendering for static pages result = nimble.extract({ "url": "https://static-site.com/page.html", "render": False, # or omit (false is default) }) ``` # Network Capture Source: https://docs.nimbleway.com/nimble-sdk/web-tools/extract/features/network-capture Capture internal API calls and dynamic data without parsing HTML Network Capture intercepts internal API calls made during webpage loading, giving you direct access to structured data in JSON format instead of parsing HTML. Common uses: * **Dynamic content**: Capture lazy-loaded data and real-time updates. * **API access**: Interact directly with backend APIs bypassing UI rendering. * **Performance**: Reduce overhead by accessing machine-readable responses. * **Accuracy**: Get reliable data directly from API endpoints. Network Capture requires page rendering to be enabled (`render: true`). For XHR/AJAX calls that don't need rendering, use the `is_xhr` parameter instead. ## Parameters Intercept and capture network requests made by the page. Perfect for accessing hidden APIs or getting data directly from backend calls instead of parsing HTML. **Requirements:** Only works when `render: true` **Each capture filter is an object with these options:** * `method` - Filter by HTTP method (GET, POST, PUT, DELETE, etc.) * Leave empty to capture any method * `url.type` - How to match URLs * `exact` - Match the complete URL exactly * `contains` - Match URLs containing a specific string * `url.value` - The URL or URL pattern to match * `resource_type` - Filter by request type (array) * Options: `xhr`, `fetch`, `stylesheet`, `script`, `document`, `image` * Example: `["xhr", "fetch"]` to capture only AJAX/fetch requests * `validation` - Validate response content (default: false) * Ensures captured responses are valid * `wait_for_requests_count` - Wait for this many matching requests (default: 0) * Useful when you know how many API calls to expect * `wait_for_requests_count_timeout` - How long to wait in seconds (default: 10) * Timeout for waiting for the expected request count **Pro tip:** Use `contains` with `/api/` to capture all API calls, or be specific with exact URLs. **Example (capture specific API):** ```json theme={"system"} "network_capture": [ { "method": "GET", "url": { "type": "contains", "value": "/api/products" }, "resource_type": ["xhr", "fetch"] } ] ``` **Example (multiple captures):** ```json theme={"system"} "network_capture": [ { "url": { "type": "contains", "value": "/graphql" } }, { "url": { "type": "exact", "value": "https://api.example.com/data" }, "wait_for_requests_count": 1 } ] ``` ## Usage ### Filter by exact URL match Capture a specific API endpoint by matching the complete URL. ```python Python theme={"system"} from nimble_python import Nimble nimble = Nimble(api_key="YOUR-API-KEY") result = nimble.extract( url="https://www.example.com", render=True, network_capture=[ { "method": "GET", "url": { "type": "exact", "value": "https://www.example.com/api/data" } } ] ) print(result) ``` ```javascript Node theme={"system"} import Nimble from "@nimble-way/nimble-js"; const nimble = new Nimble({ apiKey: "YOUR-API-KEY" }); const result = await nimble.extract({ url: "https://www.example.com", render: true, network_capture: [ { method: "GET", url: { type: "exact", value: "https://www.example.com/api/data", }, }, ], }); console.log(result); ``` ```bash cURL theme={"system"} curl -X POST 'https://sdk.nimbleway.com/v1/extract' \ --header 'Authorization: Bearer ' \ --header 'Content-Type: application/json' \ --data-raw '{ "url": "https://www.example.com", "render": true, "network_capture": [ { "method": "GET", "url": { "type": "exact", "value": "https://www.example.com/api/data" } } ] }' ``` ### Filter by URL pattern Use `contains` to capture requests with URLs matching a pattern. This is useful for capturing file types (like `.css` or `.js`), requests with dynamic URL components, or when you don't know the exact URL. ```python Python theme={"system"} from nimble_python import Nimble nimble = Nimble(api_key="YOUR-API-KEY") result = nimble.extract( url="https://www.example.com", render=True, network_capture=[ { "url": { "type": "contains", "value": "/graphql" } } ] ) print(result) ``` ```javascript Node theme={"system"} import Nimble from "@nimble-way/nimble-js"; const nimble = new Nimble({ apiKey: "YOUR-API-KEY" }); const result = await nimble.extract({ url: "https://www.example.com", render: true, network_capture: [ { url: { type: "contains", value: "/api/", }, }, ], }); console.log(result); ``` ```bash cURL theme={"system"} curl -X POST 'https://sdk.nimbleway.com/v1/extract' \ --header 'Authorization: Bearer ' \ --header 'Content-Type: application/json' \ --data-raw '{ "url": "https://www.example.com", "render": true, "network_capture": [ { "url": { "type": "contains", "value": "/api/" } } ] }' ``` ### Filter by resource type Capture specific types of resources like XHR, Fetch, or Script requests. ```python Python theme={"system"} from nimble_python import Nimble nimble = Nimble(api_key="YOUR-API-KEY") result = nimble.extract( url="https://www.example.com", render=True, network_capture=[ { "method": "GET", "resource_type": ["xhr", "fetch"] } ] ) print(result) ``` ```javascript Node theme={"system"} import Nimble from "@nimble-way/nimble-js"; const nimble = new Nimble({ apiKey: "YOUR-API-KEY" }); const result = await nimble.extract({ url: "https://www.example.com", render: true, network_capture: [ { method: "GET", resource_type: ["xhr", "fetch"], }, ], }); console.log(result); ``` ```bash cURL theme={"system"} curl -X POST 'https://sdk.nimbleway.com/v1/extract' \ --header 'Authorization: Bearer ' \ --header 'Content-Type: application/json' \ --data-raw '{ "url": "https://www.example.com", "render": true, "network_capture": [ { "method": "GET", "resource_type": ["xhr", "fetch"] } ] }' ``` ### Multiple filters Combine multiple filters to capture different request types in one call. ```python Python theme={"system"} from nimble_python import Nimble nimble = Nimble(api_key="YOUR-API-KEY") result = nimble.extract( url="https://www.example.com", render=True, network_capture=[ { "method": "GET", "url": { "type": "exact", "value": "https://www.example.com/api/resource" } }, { "url": { "type": "contains", "value": ".css" } } ] ) print(result) ``` ```javascript Node theme={"system"} import Nimble from "@nimble-way/nimble-js"; const nimble = new Nimble({ apiKey: "YOUR-API-KEY" }); const result = await nimble.extract({ url: "https://www.example.com", render: true, network_capture: [ { method: "GET", url: { type: "exact", value: "https://www.example.com/api/resource", }, }, { url: { type: "contains", value: ".css", }, }, ], }); console.log(result); ``` ```bash cURL theme={"system"} curl -X POST 'https://sdk.nimbleway.com/v1/extract' \ --header 'Authorization: Bearer ' \ --header 'Content-Type: application/json' \ --data-raw '{ "url": "https://www.example.com", "render": true, "network_capture": [ { "method": "GET", "url": { "type": "exact", "value": "https://www.example.com/api/resource" } }, { "url": { "type": "contains", "value": ".css" } } ] }' ``` ### Wait for requests Use `wait_for_requests_count` to ensure you capture a minimum number of network requests. The request duration will be extended until the count is reached or the timeout expires. ```python Python theme={"system"} from nimble_python import Nimble nimble = Nimble(api_key="YOUR-API-KEY") result = nimble.extract( url="https://www.example.com", render=True, network_capture=[ { "method": "GET", "resource_type": ["xhr", "script"], "wait_for_requests_count": 3, "wait_for_requests_count_timeout": 5 } ] ) print(result) ``` ```javascript Node theme={"system"} import Nimble from "@nimble-way/nimble-js"; const nimble = new Nimble({ apiKey: "YOUR-API-KEY" }); const result = await nimble.extract({ url: "https://www.example.com", render: true, network_capture: [ { method: "GET", resource_type: ["xhr", "script"], wait_for_requests_count: 3, wait_for_requests_count_timeout: 5, }, ], }); console.log(result); ``` ```bash cURL theme={"system"} curl -X POST 'https://sdk.nimbleway.com/v1/extract' \ --header 'Authorization: Bearer ' \ --header 'Content-Type: application/json' \ --data-raw '{ "url": "https://www.example.com", "render": true, "network_capture": [ { "method": "GET", "resource_type": ["xhr", "script"], "wait_for_requests_count": 3, "wait_for_requests_count_timeout": 5 } ] }' ``` This configuration will wait up to 5 seconds to capture at least 3 network requests matching the filter criteria. ## XHR without rendering For direct API endpoints that don't require page rendering, use `is_xhr` for better performance. ```python Python theme={"system"} from nimble_python import Nimble nimble = Nimble(api_key="YOUR-API-KEY") result = nimble.extract( url="https://api.example.com/endpoint", is_xhr=True, ) print(result) ``` ```javascript Node theme={"system"} import Nimble from "@nimble-way/nimble-js"; const nimble = new Nimble({ apiKey: "YOUR-API-KEY" }); const result = await nimble.extract({ url: "https://api.example.com/endpoint", is_xhr: true, }); console.log(result); ``` ```bash cURL theme={"system"} curl -X POST 'https://sdk.nimbleway.com/v1/extract' \ --header 'Authorization: Bearer ' \ --header 'Content-Type: application/json' \ --data-raw '{ "url": "https://api.example.com/endpoint", "is_xhr": true }' ``` `is_xhr` only works when `render` is `false`. It sends XHR-specific headers and targets the API URL directly. ## Example response When browser actions complete successfully, you'll receive the final page state along with any data captured. The response includes: * **data**: All related extacted data * **data.html**: Final DOM state after all actions * **data.network\_capture**: The network capture response by order * **metadata**: Execution details including task id, driver used, execution time and more ```json theme={"system"} { "url": "https://www.example.com/", "task_id": "b1fa7943-cba5-4ec2-a88c-4d2d6799c794", "status": "success", "data": { "html": "...", "network_capture": [ { "filter": { "method": "GET", "resource_type": ["xhr", "script"] }, "result": [ { "request": { "resource_type": "script", "method": "GET", "url": "https://www.example.com/script/0001.js", "headers": {} }, "response": { "status": 200, "headers": {}, "body": "..." } }, { "request": { "resource_type": "xhr", "method": "GET", "url": "https://www.example.com/script/0002.js", "headers": {} }, "response": { "status": 200, "headers": {}, "body": "..." } } ] } ] }, "metadata": { "query_time": "2026-02-08T22:00:36.132Z", "query_duration": 1877, "response_parameters": { "input_url": "https://www.example.com/" }, "driver": "vx6" }, "status_code": 200 } ``` ## Best practices ### Use specific URL patterns **Be specific with URL matching:** ```python theme={"system"} # ✅ Specific pattern for API endpoints network_capture = [ { "url": { "type": "contains", "value": "/api/v1/products" } } ] # ❌ Too broad - captures everything network_capture = [ { "url": { "type": "contains", "value": "/" } } ] ``` ### Filter by resource type **Narrow down to relevant resources:** ```python theme={"system"} # ✅ Capture only XHR and Fetch requests network_capture = [ { "resource_type": ["xhr", "fetch"] } ] # ✅ Capture scripts and stylesheets network_capture = [ { "resource_type": ["script", "stylesheet"] } ] ``` ### Set appropriate wait counts **Use wait\_for\_requests\_count for dynamic content:** ```python theme={"system"} # ✅ Wait for specific number of requests network_capture = [ { "method": "GET", "resource_type": ["xhr"], "wait_for_requests_count": 3, "wait_for_requests_count_timeout": 10 } ] # ❌ No wait - may miss delayed requests network_capture = [ { "method": "GET", "resource_type": ["xhr"] } ] ``` ### Use XHR mode for direct API calls **Skip rendering when accessing APIs directly:** ```python theme={"system"} # ✅ Direct API access without rendering result = nimble.extract({ "url": "https://api.example.com/data", "is_xhr": True }) # ❌ Unnecessary rendering for API endpoints result = nimble.extract({ "url": "https://api.example.com/data", "render": True, "network_capture": [...] }) ``` # Parsing Schema Source: https://docs.nimbleway.com/nimble-sdk/web-tools/extract/features/parsing-schema Precise data extraction with powerful parser syntax Parsing Schema gives you full control over data extraction using a comprehensive parser syntax. Define exact data structures for predictable, low-cost extraction from HTML, JSON, XML, and network captures. Parsers are the complete recipe for processing web content into structured data. They combine: * selectors - identify elements * extractors - extract data from elements * post-processors (optional) - transform the output # **When to use** Use parsing schema when you need: * **Predictable extraction**: Same selectors extract same data every time * **Full control**: Specify exact selectors paths and data types * **High volume**: Process large datasets efficiently Parsers may break when page structure or selectors change. Monitor source pages and update parsers as needed. # **Parameters** **`Must be set to true`** to enable parsing. This tells Nimble you want to extract structured data using the parser you define. When disabled, you'll just get raw HTML without structured extraction. **Example:** ```json theme={"system"} "parse": true ``` Your custom extraction recipe that defines exactly what data to pull from the page and how to structure it. **Parser structure:** Each field in your parser is a key-value pair where: * **Key** - The name of the field in your output (like `"product_name"` or `"price"`) * **Value** - An object that describes how to extract that field **Every parser needs:** 1. `type` - What kind of parser to use * `terminal` - Extract a single value (like one price) * `terminal_list` - Extract multiple values (like a list of image URLs) * `schema` - Extract a nested object (like product details) * `schema_list` - Extract a list of objects (like multiple products) * `or` - Try multiple strategies, use first one that works * `and` - Combine multiple extraction strategies * `const` - Return a fixed value 2. `selector` - How to find the element on the page * Use CSS selectors (`.product-name`, `#price`, etc.) * Or XPath, JSON paths for other data types 3. `extractor` - What data to grab from the element * `text` - The text content * `attr` - An attribute value (like `href` or `src`) * `json` - Parse JSON data * `raw` - The raw HTML 4. `post_processor` (optional) - Transform the data * Convert to number, format dates, clean text, etc. **Think of it as:** "Find THIS element, grab THIS data from it, and format it like THIS" **Simple example:** ```json theme={"system"} "parser": { "title": { "type": "terminal", "selector": { "type": "css", "css_selector": "h1.product-title" }, "extractor": { "type": "text" } } } ``` **List example:** ```json theme={"system"} "parser": { "images": { "type": "terminal_list", "selector": { "type": "css", "css_selector": ".product-gallery img" }, "extractor": { "type": "attr", "attr": "src" } } } ``` ### Usage **Example parser structure:** ```python Python theme={"system"} from nimble_python import Nimble nimble = Nimble(api_key="YOUR-API-KEY") result = nimble.extract( url="https://www.example.com/product", parse=True, parser={ "product_name": { # key - output field name "type": "terminal", # parser type "selector": { # how to find the element "type": "css", "css_selector": ".product-title" }, "extractor": { # what to extract "type": "text" } }, "price": { "type": "terminal", "selector": { "type": "css", "css_selector": ".price-value" }, "extractor": { "type": "text", "post_processor": { # optional: transform the data "type": "number" } } } } ) print(result) ``` ```javascript Node theme={"system"} import Nimble from "@nimble-way/nimble-js"; const nimble = new Nimble({ apiKey: "YOUR-API-KEY" }); const result = await nimble.extract({ url: "https://www.example.com/product", parse: true, parser: { product_name: { // key - output field name type: "terminal", // parser type selector: { // how to find the element type: "css", css_selector: ".product-title", }, extractor: { // what to extract type: "text", }, }, price: { type: "terminal", selector: { type: "css", css_selector: ".price-value", }, extractor: { type: "text", post_processor: { // optional: transform the data type: "number", }, }, }, }, }); console.log(result); ``` ```bash cURL theme={"system"} curl -X POST 'https://sdk.nimbleway.com/v1/extract' \ --header 'Authorization: Bearer ' \ --header 'Content-Type: application/json' \ --data-raw '{ "url": "https://www.example.com/product", "parse": true, "parser": { "product_name": { "type": "terminal", "selector": { "type": "css", "css_selector": ".product-title" }, "extractor": { "type": "text" } }, "price": { "type": "terminal", "selector": { "type": "css", "css_selector": ".price-value" }, "extractor": { "type": "text", "post_processor": { "type": "number" } } } } }' ``` ### **Example Output** ```json theme={"system"} { "status": "success", "data": { "parsing": { "product_name": "Wireless Headphones", "price": 79.99 } } } ``` ## Parser Types Supported parsing types: * `terminal` - Returns a single terminal/literal as output. * `terminal_list` - Returns a list of literals instead of a single literal. * `schema` - Returns a dictionary/JSON according to its field parsers. * `schema_list` - Returns a list of dictionaries/JSONs instead of a single dictionary/JSON. * `or` - Tries a sequence of parsers and returns the result of the first parser that returns a non-null value * `and` - Runs a sequence of schema parsers and merges their results into a single output. All parsers execute on the same input, and results are combined (first non-null value wins for overlapping keys). * `const` - Always returns its `value` regardless of the input. Useful for adding static data to your output. ### terminal * The most basic parser. ```json theme={"system"} { "type": "terminal", "selector": { ... }, "extractor": { ... } // Defaults to raw extractor if not specified } ``` ```json theme={"system"} { "type": "terminal", "selector": { "type": "css", "css_selector": ".product-name" }, "extractor": { "type": "text" } } ``` ### terminal\_list Returns a list of literals instead of a single literal. ```json theme={"system"} { "type": "terminal_list", "selector": { ... }, "extractor": { ... } // Defaults to raw extractor if not specified } ``` ```json theme={"system"} { "type": "terminal_list", "selector": { "type": "css", "css_selector": ".product-gallery img" }, "extractor": { "type": "attr", "attr": "src" } } ``` ### schema This is the most commonly used parser for structured data extraction. ```json theme={"system"} { "type": "schema", "selector": { ... }, // Optional "fields": { "field_name": { /* Parser */ } } } ``` ```json theme={"system"} { "type": "schema", "selector": { "type": "css", "css_selector": ".product-card" }, "fields": { "name": { "type": "terminal", "selector": { "type": "css", "css_selector": ".product-name" }, "extractor": { "type": "text" } }, "price": { "type": "terminal", "selector": { "type": "css", "css_selector": ".price" }, "extractor": { "type": "text" } } } } ``` ### schema\_list Returns a list of dictionaries/JSONs instead of a single dictionary/JSON. ```json theme={"system"} { "type": "schema_list", "selector": { ... }, // Optional "fields": { "field_name": { /* Parser */ } } } ``` The optional `position` attribute adds an index field to each item in the output list. ```json theme={"system"} { "type": "schema_list", "selector": { ... }, "fields": { "field_name": { /* Parser */ } }, "position": { "field_name": "index", "start_from": 1 // Optional, defaults to 0 } } ``` ```json theme={"system"} { "type": "schema_list", "selector": { "type": "css", "css_selector": ".product-item" }, "fields": { "name": { "type": "terminal", "selector": { "type": "css", "css_selector": ".product-name" }, "extractor": { "type": "text" } }, "price": { "type": "terminal", "selector": { "type": "css", "css_selector": ".price" }, "extractor": { "type": "text" } } } } ``` ### or Useful for handling variations in page structure. ```json theme={"system"} { "type": "or", "parsers": [ { /* Parser 1 */ }, { /* Parser 2 */ }, { /* Parser 3 */ } ] } ``` ```json theme={"system"} { "type": "or", "parsers": [ { "type": "terminal", "selector": { "type": "css", "css_selector": ".sale-price" }, "extractor": { "type": "text" } }, { "type": "terminal", "selector": { "type": "css", "css_selector": ".regular-price" }, "extractor": { "type": "text" } } ] } ``` ### and Runs a sequence of schema parsers and merges their results into a single output. All parsers execute on the same input, and results are combined (first non-null value wins for overlapping keys). ```json theme={"system"} { "type": "and", "parsers": [ { /* Schema Parser 1 */ }, { /* Schema Parser 2 */ } ] } ``` ### const Always returns its `value` regardless of the input. Useful for adding static data to your output. ```json theme={"system"} { "type": "const", "value": "some_value" } ``` ## Parsing Selectors Selectors identify elements (HTML, JSON, XML, Network) in the input web page. Supported selectors: * `css` - Selects elements matching a [CSS selector](https://www.w3schools.com/css/css_selectors.asp). * `xpath` - Enables powerful element selection using [XPath expressions](https://www.w3schools.com/xml/xpath_intro.asp). Particularly useful for XML documents like RSS feeds and sitemaps. * `json` - Extracts JSON elements from the page. All subsequent selectors and extractors receive JSON instead of HTML. * `sequence` - Combines multiple selectors in sequence. Useful for chaining different selector types. * `parent` - Traverses up the DOM tree (for HTML) or context hierarchy (for JSON). Useful when you need to select a parent element after finding a specific child. * `root` - Returns the original page (document). Often used with JSON selector to access fields like `network_capture`, `url`, or `html`. ### css Selects elements matching a [CSS selector](https://www.w3schools.com/css/css_selectors.asp). ```json theme={"system"} { "type": "css", "css_selector": "div.price" } ``` ```json theme={"system"} // Select by class { "type": "css", "css_selector": ".product-name" } // Select by ID { "type": "css", "css_selector": "#main-content" } // Select by attribute { "type": "css", "css_selector": "a[data-product-id]" } // Complex selector { "type": "css", "css_selector": "div.product > h2.title" } ``` ### xpath Enables powerful element selection using [XPath expressions](https://www.w3schools.com/xml/xpath_intro.asp). Particularly useful for XML documents like RSS feeds and sitemaps. ```json theme={"system"} { "type": "xpath", "path": "//book[@category='fiction']" } ``` * `//element` - Select all elements with the given name - `/root/child` - Select child elements of root - `//element[@attr='value']` - Select by attribute value - `//element[position()=1]` - Select first element - `//*[local-name()='element']` - Select ignoring namespaces ```json theme={"system"} { "type": "terminal_list", "selector": { "type": "xpath", "path": "//*[local-name()='loc']" }, "extractor": { "type": "text" } } ``` ```json theme={"system"} { "type": "sequence", "sequence": [ { "type": "xpath", "path": "//item" }, { "type": "xpath", "path": ".//title" } ] } ``` ### json Extracts JSON elements from the page. All subsequent selectors and extractors receive JSON instead of HTML. ```json theme={"system"} { "type": "json", "path": "nested.keys.in.the.json" // jsonpath } ``` The `coercion_filter` field provides advanced control when dealing with multiple JSON objects. It uses JSONPath expressions to filter specific JSON objects. ```json theme={"system"} { "type": "json", "coercion_filter": "$[1]", // Get the second JSON object "path": "nested.keys" } ``` ```json theme={"system"} { "type": "json", "coercion_filter": "$[?(@.type=='product')]", "path": "name" } ``` ```json theme={"system"} { "type": "terminal", "selector": { "type": "sequence", "sequence": [ { "type": "css", "css_selector": "script[type='application/ld+json']" }, { "type": "json", "path": "$.offers.price" } ] }, "extractor": { "type": "raw" } } ``` ### sequence Combines multiple selectors in sequence. Useful for chaining different selector types. ```json theme={"system"} { "type": "sequence", "sequence": [ { /* Selector 1 */ }, { /* Selector 2 */ }, { /* Selector 3 */ } ] } ``` ```json theme={"system"} { "type": "sequence", "sequence": [ { "type": "css", "css_selector": "script#product-data" }, { "type": "json", "path": "$.product" } ] } ``` ### parent Traverses up the DOM tree (for HTML) or context hierarchy (for JSON). Useful when you need to select a parent element after finding a specific child. ```json theme={"system"} { "type": "parent", "times": 1 // Number of levels to traverse (default: 1) } ``` ```json theme={"system"} { "type": "sequence", "sequence": [ { "type": "css", "css_selector": "span.price" }, { "type": "parent", "times": 2 } ] } ``` ### root Returns the original page (document). Often used with JSON selector to access fields like `network_capture`, `url`, or `html`. ```json theme={"system"} { "type": "root" } ``` ```json theme={"system"} { "type": "terminal", "selector": { "type": "sequence", "sequence": [ { "type": "root" }, { "type": "json", "path": "$.url" } ] }, "extractor": { "type": "raw" } } ``` ```json theme={"system"} { "type": "terminal", "selector": { "type": "sequence", "sequence": [ { "type": "root" }, { "type": "json", "path": "$.network_capture" } ] }, "extractor": { "type": "raw" } } ``` ## Parsing Extractors Extractors specify what data to extract from the selected element. Supported extractors: * `text` - Extracts the text content of the element. Works with both HTML (CSS selectors) and XML (XPath selectors). * `strip` (optional, boolean) - If it is set to `false`, leading and trailing whitespaces are preserved in the text. Default is `true`. * `separator` (optional, string) - Specifies a separator string to use when joining text from different child elements. When extracting text from nested HTML elements, this separator will be inserted between text from different elements. If not specified, text from different elements is concatenated without a separator. * `attr` - Extracts an attribute value from the element. Works with both HTML and XML elements. common attr: * `href` - Links * `src` - Images, scripts * `data-*` - Custom data attributes * `class` - CSS classes * `id` - Element IDs * `json` - Extracts JSON content using JSONPath. * `raw` - Extracts an element as-is without coercion. JSON stays as JSON, strings stay as strings. Useful for advanced parsing with complex JSON selectors. If no extractor is specified, the **raw** extractor is used by default. ### text Extracts the text content of the element. This extractor works with both HTML elements (from CSS selectors) and XML elements (from XPath selectors). You can use `strip=false` to keep leading and trailing whitespace characters. The default is to remove them. The text extractor supports both HTML elements (BeautifulSoup Tag) from CSS selectors and XML elements (lxml Element) from XPath selectors. This allows you to use the same extractor regardless of whether you're parsing HTML or XML documents. **Basic usage:** ```json theme={"system"} { "type": "text", "regex": "string" // Optional (deprecated), will return 1st match } ``` ```json theme={"system"} { "type": "text", "regex": "string", // Optional (deprecated), will return 1st match "strip": false } ``` ```json theme={"system"} { "type": "text", "regex": "string", // Optional (deprecated), will return 1st match "separator": " | " } ``` ```json theme={"system"} { "type": "terminal", "selector": { "type": "xpath", "path": "//book/title" }, "extractor": { "type": "text", "strip": true } } ``` ### attr Extracts an attribute value from the element. Works with both HTML and XML elements. Common attributes: `href` ,`src` ,`data-*` ,`class` , `id` ```json theme={"system"} { "type": "attr", "attr": "href" } ``` ```json theme={"system"} { "type": "terminal", "selector": { "type": "css", "css_selector": "img.product-image" }, "extractor": { "type": "attr", "attr": "src" } } ``` ### json Extracts JSON content using JSONPath. ```json theme={"system"} { "type": "json", "path": "nested.keys.in.the.json" } ``` ### raw Extracts an element as-is without coercion. JSON stays as JSON, strings stay as strings. Useful for advanced parsing with complex JSON selectors. ```json theme={"system"} { "type": "raw" } ``` ## Parsing Post Processors Post processors transform extractor output. Define them in the extractor's `post_processor` field. Supported post-procession options: * `url` - Converts relative URLs to absolute URLs based on the page origin. * `regex` - Transforms output using a [regular expression](https://regexr.com/). The optional `group` parameter extracts a specific capturing group (defaults to 0). * `format` - Formats input into a string using Python's str.format, where the input is available as `{data}`. * `date` - Formats dates to ISO format or custom format. * `boolean` - Transforms output to boolean based on conditions: `contains`, `exists`, or `regex`. Use `not: true` to reverse the result. * `number` - Coerces output to a number (int or float). Handles formatted numbers like `"1.5M"` → `1500000` or `"2,100"` → `2100`. * `country` - Converts country names to country codes. * `sequence` - Applies multiple post processors in sequence. ### url Converts relative URLs to absolute URLs based on the page origin. ```json theme={"system"} { "extractor": { "type": "text", "post_processor": { "type": "url" } } } ``` * Input: `"/news/article"` - Output: `"https://www.example.com/news/article"` ### regex Transforms output using a [regular expression](https://regexr.com/). The optional `group` parameter extracts a specific capturing group (defaults to 0). ```json theme={"system"} { "extractor": { "type": "text", "post_processor": { "type": "regex", "regex": "\\d+", "group": 0 // Optional } } } ``` ```json theme={"system"} { "extractor": { "type": "text", "post_processor": { "type": "regex", "regex": "\\d+\\.\\d+" } } } ``` ```json theme={"system"} { "extractor": { "type": "text", "post_processor": { "type": "regex", "regex": "Price: (\\d+)\\.(\\d+)", "group": 1 } } } ``` ### format Formats input into a string using Python's str.format, where the input is available as `{data}`. ```json theme={"system"} { "extractor": { "type": "text", "post_processor": { "type": "format", "format": "${data}" } } } ``` - Input: 5.00 - Output: "\$5.00" ### date Formats dates to ISO format or custom format. ```json theme={"system"} { "extractor": { "type": "text", "post_processor": { "type": "date", "format": "%d/%m/%y" // Optional, defaults to ISO format } } } ``` * Input: `"5 days ago"` - Output (no format): `"2024-07-29T00:00:00"` - Output (with format `%d/%m/%y`): `"29/07/2024"` ### boolean Transforms output to boolean based on conditions: `contains`, `exists`, or `regex`. Use `not: true` to reverse the result. ```json theme={"system"} { "extractor": { "type": "text", "post_processor": { "type": "boolean", "condition": "contains", "contains": "InStock" } } } ``` ```json theme={"system"} { "extractor": { "type": "text", "post_processor": { "type": "boolean", "condition": "exists" } } } ``` ```json theme={"system"} { "extractor": { "type": "text", "post_processor": { "type": "boolean", "condition": "regex", "regex": "\\d+" } } } ``` ```json theme={"system"} { "extractor": { "type": "text", "post_processor": { "type": "boolean", "condition": "contains", "contains": "OutOfStock", "not": true } } } ``` ### number Coerces output to a number (int or float). Handles formatted numbers like `"1.5M"` → `1500000` or `"2,100"` → `2100`. ```json theme={"system"} { "extractor": { "type": "text", "post_processor": { "type": "number", "locale": "en", // Optional: locale for number parsing "force_type": "float" // Optional: "int" or "float" } } } ``` ```json theme={"system"} { "extractor": { "type": "text", "post_processor": { "type": "number", "locale": "de" } } } ``` Input: `"1.000,50"` → Output: `1000.50` ### country Converts country names to country codes. ```json theme={"system"} { "extractor": { "type": "text", "post_processor": { "type": "country" } } } ``` * Input: `"United States"` - Output: `"US"` ### sequence Applies multiple post processors in sequence. ```json theme={"system"} { "extractor": { "type": "text", "post_processor": { "type": "sequence", "sequence": [ { "type": "regex", "regex": "\\d+\\.\\d+" }, { "type": "number" } ] } } } ``` * Input: `"The price is $50.25!"` - After regex: `"50.25"` - After number: `50.25` ## Complete Examples ### Parsing a BBC News Article This example demonstrates parsing a complete BBC news article about a three-legged cat, showing how to extract structured data from HTML using various parser types, selectors, and extractors. **Target URL:** `https://www.bbc.com/news/articles/cervlxymly2o` **Target Schema:** ```json theme={"system"} { "url": "string", "title": "string", "date": "string", "author": { "name": "string", "organization": "string" }, "images": ["string"], "paragraphs": ["string"] } ``` ### Field-by-Field Breakdown The URL is the canonical link for the page, typically found in the HTML `head` tag under a `link` element with `rel="canonical"`. **HTML Structure:** ```html theme={"system"} ``` **Parser:** ```json theme={"system"} { "type": "terminal", "description": "Extracts the canonical URL from the page's head section", "selector": { "type": "css", "css_selector": "link[rel='canonical']" }, "extractor": { "type": "attr", "attr": "href" } } ``` **Explanation:** * **Selector:** `link[rel='canonical']` selects the first link element with `rel="canonical"` * **Extractor:** `attr` with `href` extracts the URL from the href attribute The article title is contained in an `h1` element within a headline block. **HTML Structure:** ```html theme={"system"}

Three-legged cat 'brings town together'

``` **Parser:** ```json theme={"system"} { "type": "terminal", "description": "Main headline of the article", "selector": { "type": "css", "css_selector": "div[data-component='headline-block'] h1" }, "extractor": { "type": "text" } } ``` **Explanation:** * **Selector:** `div[data-component='headline-block'] h1` targets the h1 inside the headline block * **Extractor:** `text` extracts the text content from the element The publication date is in a `time` element and needs to be formatted to ISO format. **HTML Structure:** ```html theme={"system"}
``` **Parser:** ```json theme={"system"} { "type": "terminal", "description": "Publication date in ISO format", "selector": { "type": "css", "css_selector": "div[data-testid='byline-new'] time" }, "extractor": { "type": "text", "post_processor": { "type": "date" } } } ``` **Explanation:** * **Selector:** `div[data-testid='byline-new'] time` targets the time element * **Extractor:** `text` with `date` post-processor converts "29 July 2024" to "2024-07-29T00:00:00" This field requires JavaScript rendering. Add render options to your API request to wait for this element. The author information contains both name and organization, requiring a nested schema parser. **HTML Structure:** ```html theme={"system"}
Martin Heath
BBC News, Northamptonshire
``` **Parser:** ```json theme={"system"} { "type": "schema", "description": "Author information with name and organization", "selector": { "type": "css", "css_selector": "div[data-testid='byline-new-contributors']" }, "fields": { "name": { "type": "terminal", "selector": { "type": "css", "css_selector": "span[class]" }, "extractor": { "type": "text" } }, "organization": { "type": "terminal", "selector": { "type": "css", "css_selector": "span:not([class])" }, "extractor": { "type": "text" } } } } ``` **Explanation:** * **Schema Parser:** Returns a nested object with multiple fields * **Selector Nesting:** Parent selector scopes child selectors to the byline-new-contributors div * **Name Selector:** `span[class]` selects spans with a class attribute * **Organization Selector:** `span:not([class])` selects spans without a class attribute Extract all image URLs from the article, converting relative URLs to absolute. **HTML Structure:** ```html theme={"system"}
``` **Parser:** ```json theme={"system"} { "type": "terminal_list", "description": "All article images with absolute URLs", "selector": { "type": "css", "css_selector": "article img" }, "extractor": { "type": "attr", "attr": "src", "post_processor": { "type": "url" } } } ``` **Explanation:** * **terminal\_list:** Returns an array of values instead of a single value * **Selector:** `article img` selects all img elements within article * **Extractor:** `attr` with `src` gets the image source * **Post-processor:** `url` converts relative URLs to absolute (e.g., `/news/...` → `https://www.bbc.com/news/...`) Extract all article paragraphs as an array of strings. **HTML Structure:** ```html theme={"system"}

A three-legged cat has captured a town's imagination...

The people of Daventry, Northamptonshire, love taking photographs...

``` **Parser:** ```json theme={"system"} { "type": "terminal_list", "description": "Article content paragraphs", "selector": { "type": "css", "css_selector": "article div[data-component='text-block'] p" }, "extractor": { "type": "text" } } ``` **Explanation:** * **terminal\_list:** Returns an array of paragraph texts * **Selector:** `article div[data-component='text-block'] p` selects all p elements in text blocks * **Extractor:** `text` extracts the text content from each paragraph ### Complete Parser ```json theme={"system"} { "type": "schema", "description": "Parses a BBC news article into structured data", "fields": { "url": { "type": "terminal", "description": "Canonical URL of the article", "selector": { "type": "css", "css_selector": "link[rel='canonical']" }, "extractor": { "type": "attr", "attr": "href" } }, "title": { "type": "terminal", "description": "Main headline of the article", "selector": { "type": "css", "css_selector": "div[data-component='headline-block'] h1" }, "extractor": { "type": "text" } }, "date": { "type": "terminal", "description": "Publication date in ISO format", "selector": { "type": "css", "css_selector": "div[data-testid='byline-new'] time" }, "extractor": { "type": "text", "post_processor": { "type": "date" } } }, "author": { "type": "schema", "description": "Author information", "selector": { "type": "css", "css_selector": "div[data-testid='byline-new-contributors']" }, "fields": { "name": { "type": "terminal", "selector": { "type": "css", "css_selector": "span[class]" }, "extractor": { "type": "text" } }, "organization": { "type": "terminal", "selector": { "type": "css", "css_selector": "span:not([class])" }, "extractor": { "type": "text" } } } }, "images": { "type": "terminal_list", "description": "All article images", "selector": { "type": "css", "css_selector": "article img" }, "extractor": { "type": "attr", "attr": "src", "post_processor": { "type": "url" } } }, "paragraphs": { "type": "terminal_list", "description": "Article content paragraphs", "selector": { "type": "css", "css_selector": "article div[data-component='text-block'] p" }, "extractor": { "type": "text" } } } } ``` ### Example Output ```json theme={"system"} { "url": "https://www.bbc.com/news/articles/cervlxymly2o", "title": "Three-legged cat 'brings town together'", "date": "2024-07-29T00:00:00", "author": { "name": "Martin Heath", "organization": "BBC News, Northamptonshire" }, "images": [ "https://ichef.bbci.co.uk/news/480/cpsprodpb/2a87/live/321fae30-4c01-11ef-b2d2-cdb23d5d7c5b.jpg.webp", "https://ichef.bbci.co.uk/news/480/cpsprodpb/a8c2/live/904194b0-4c01-11ef-b2d2-cdb23d5d7c5b.jpg.webp", "https://ichef.bbci.co.uk/news/480/cpsprodpb/7579/live/9ecae4f0-4c01-11ef-aebc-6de4d31bf5cd.jpg.webp" ], "paragraphs": [ "A three-legged cat has captured a town's imagination with his appearances in shops and offices.", "The people of Daventry, Northamptonshire, love taking photographs of the 14-year-old feline and documenting his travels on social media.", "Funds have been raised to buy a street sign with his name on it, and souvenir Salem T-shirts could follow." ] } ``` ### API Request Example ```python Python theme={"system"} from nimble_python import Nimble nimble = Nimble(api_key="YOUR-API-KEY") result = nimble.extract( url="https://www.bbc.com/news/articles/cervlxymly2o", parse=True, render=True, render_flow=[{ "wait_for": { "selectors": ["div[data-testid='byline-new'] time"] } }], parser={ "type": "schema", "description": "Parses a BBC news article into structured data", "fields": { # ... (full parser structure as shown above) } } ) print(result) ``` ```javascript Node theme={"system"} import Nimble from "@nimble-way/nimble-js"; const nimble = new Nimble({ apiKey: "YOUR-API-KEY" }); const result = await nimble.extract({ url: "https://www.bbc.com/news/articles/cervlxymly2o", parse: true, render: true, render_flow: [ { wait_for: { selectors: ["div[data-testid='byline-new'] time"], }, }, ], parser: { type: "schema", description: "Parses a BBC news article into structured data", fields: { // ... (full parser structure as shown above) }, }, }); console.log(result); ``` ```bash cURL theme={"system"} curl -X POST 'https://sdk.nimbleway.com/v1/extract' \ --header 'Authorization: Bearer ' \ --header 'Content-Type: application/json' \ --data-raw '{ "url": "https://www.bbc.com/news/articles/cervlxymly2o", "parse": true, "render": true, "render_flow": [{ "wait_for": { "selectors": ["div[data-testid=\"byline-new\"] time"] } }], "parser": { "type": "schema", "description": "Parses a BBC news article into structured data", "fields": { ... } } }' ``` This parser can be reused for any BBC news article following the same structure - just change the URL! ### Parsing Embedded JSON from Etsy.com Prodcut Page This example demonstrates parsing structured data from embedded JSON-LD (Linked Data JSON) within an HTML page. Many websites embed JSON-LD in their HTML to help search engines understand their content - we can leverage this for easier, more reliable parsing. **What is LD+JSON?** Linked Data JSON is a format for structuring data in a machine-readable way. It's often embedded in webpages using `