Callbacks & Delivery

When using async operations like /extract/async, /agent/async, or /crawl, you have three flexible options for receiving your results. Choose the method that best fits your infrastructure and workflow.

Polling

Pull results on-demand using task IDs

Callbacks

Receive push notifications when tasks complete

Cloud Delivery

Automatic delivery to your S3 or GCS bucket

Option 1: Polling (Pull)

The simplest approach - submit your async request, receive a task ID, and poll for results when ready.

Submit async request

Send a request to the async endpoint. You’ll receive a task or crawl ID to track your request.

Extract
Agent
Crawl

from nimble_python import Nimble

nimble = Nimble(api_key="YOUR-API-KEY")

response = nimble.extract.async(
    url="https://www.nimbleway.com",
    render=True,
    formats=["html", "markdown"]
)

task_id = response.task_id
print(f"Task submitted: {task_id}")

from nimble_python import Nimble

nimble = Nimble(api_key="YOUR-API-KEY")

response = nimble.agent.async(
    agent="amazon_pdp",
    params={"asin": "B0DLKFK6LR"}
)

task_id = response.task_id
print(f"Task submitted: {task_id}")

from nimble_python import Nimble

nimble = Nimble(api_key="YOUR-API-KEY")

result = nimble.crawl(
    url="https://www.nimbleway.com",
    limit=50
)

crawl_id = result.crawl_id
print(f"Crawl started: {crawl_id}")

Check status

Poll the status endpoint to monitor progress.

Extract / Agent
Crawl

import time

while True:
    my_task = nimble.tasks.status(task_id)
    print(f"Status: {my_task.task.state}")

    if my_task.state == "success":
        break
    elif my_task.state == "failed":
        print(f"Task failed: {status.error}")
        break

    time.sleep(15)

Crawl has its own status endpoint that shows overall progress and individual page tasks.

import time

while True:
    my_crawl = nimble.crawl.status(crawl_id)
    print(f"Status: {my_crawl.status}")

    if status.state == "succeeded":
        break
    elif status.state == "failed":
        print(f"Task failed: {status.error}")
        break

    time.sleep(2)

Retrieve results

Once complete, fetch the full results.

Extract
Agent
Crawl

results = nimble.tasks.results(task_id)

print(f"HTML length: {len(results.data.html)}")
print(f"Markdown length: {len(results.data.markdown)}")

results = nimble.tasks.results(task_id)

parsed = results.data.parsing["parsed"]
print(f"Product: {parsed['product_title']}")
print(f"Price: ${parsed['web_price']}")

Fetch results for each completed task in the crawl.

# my_crawl["tasks"] from step #2 contains list of task IDs from status response
for task in my_crawl["tasks"]:
    if task.state == "success":
        task_result = nimble.tasks.status(task_id)

        print(f"URL: {task_result['url']}")
        print(f"HTML length: {len(task_result['data'].get('html', ''))}")

Example Results

{
    "url": "https://www.nimbleway.com/blog/post",
    "task_id": "ec89b1f7-1cf2-40eb-91b4-78716093f9ed",
    "status": "success",
    "task": {
        "id": "ec89b1f7-1cf2-40eb-91b4-78716093f9ed",
        "state": "success",
        "created_at": "2026-02-09T23:15:43.549Z",
        "modified_at": "2026-02-09T23:16:39.094Z",
        "account_name": "your-account"
    },
    "data": {
        "html": "<!DOCTYPE html>...",
        "markdown": "# Page Title\n\nContent...",
        "headers": { ... }
    },
    "metadata": {
        "query_time": "2026-02-09T23:15:43.549Z",
        "query_duration": 1877,
        "response_parameters": {
            "input_url": "https://www.nimbleway.com/blog/post"
        },
		"driver": "vx6"
    },
    "status_code": 200
}

Polling endpoints reference

API	Submit	Check Status	Get Results
Extract	`POST /v1/extract/async`	`GET /v1/tasks/{task_id}`	`GET /v1/tasks/{task_id}/results`
Agent	`POST /v1/agent/async`	`GET /v1/tasks/{task_id}`	`GET /v1/tasks/{task_id}/results`
Crawl	`POST /v1/crawl`	`GET /v1/crawl/{crawl_id}`	`GET /v1/tasks/{task_id}/results` (per page)

Option 2: Webhooks (Push)

Get notified automatically when your tasks complete. Perfect for event-driven architectures.

Submit request with callback URL

Include callback_url (or callback object for crawl) in your async request.

Extract
Agent
Crawl

from nimble_python import Nimble

nimble = Nimble(api_key="YOUR-API-KEY")

response = nimble.extract.async(
    url="https://www.nimbleway.com",
    render=True,
    formats=["html", "markdown"],
    callback_url="https://your-server.com/webhooks/nimble"
)

task_id = response.task_id
print(f"Task submitted: {task_id}")
print("Results will be POSTed to your callback URL when ready")

from nimble_python import Nimble

nimble = Nimble(api_key="YOUR-API-KEY")

response = nimble.extract.async(
    url="https://www.nimbleway.com",
    render=True,
    formats=["html", "markdown"],
    callback_url="https://your-server.com/webhooks/nimble"
)

task_id = response.task_id
print(f"Task submitted: {task_id}")
print("Results will be POSTed to your callback URL when ready")

Crawl uses a callback object for advanced webhook options.

from nimble_python import Nimble

nimble = Nimble(api_key="YOUR-API-KEY")

result = nimble.crawl(
    url="https://www.nimbleway.com",
    limit=100,
    callback={
        "url": "https://your-server.com/webhooks/nimble",
        "headers": {
            "X-Custom-Auth": "your-secret-token"
        },
        "events": ["completed", "failed"]
    }
)

print(f"Crawl started: {result.crawl_id}")

Receive webhook notification

Nimble sends a POST to your callback URL when complete:

{
  "task": {
    "id": "8e8cfde8-345b-42b8-b3e2-0c61eb11e00f",
    "state": "success",
    "api_type": "extract",
    "download_url": "https://sdk.nimbleway.com/v1/tasks/8e8cfde8-345b-42b8-b3e2-0c61eb11e00f/results"
  }
}

Webhook configuration options

API	Parameter	Type	Description
Extract	`callback_url`	string	Your callback URL
Agent	`params.callback_url`	string	Your callback URL
Crawl	`callback.url`	string	Your callback URL
	`callback.headers`	object	Custom headers for authentication
	`callback.metadata`	object	Custom data included in payload
	`callback.events`	array	Filter events: `started`, `page`, `completed`, `failed`

Option 3: Cloud Delivery

Automatically deliver results directly to your cloud storage bucket.

Amazon S3

Deliver to any S3 bucket in your AWS account

Google Cloud Storage

Deliver to any GCS bucket in your GCP project

Configure bucket permissions (one-time)

Grant Nimble’s service account write access to your bucket.

Amazon S3
Google Cloud Storage

Nimble Service User ARN:

arn:aws:iam::744254827463:user/webit-uploader

Add this bucket policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "NimbleCloudDelivery",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::744254827463:user/webit-uploader"
      },
      "Action": [
        "s3:PutObject",
        "s3:PutObjectACL",
        "s3:GetBucketLocation"
      ],
      "Resource": [
        "arn:aws:s3:::YOUR_BUCKET_NAME",
        "arn:aws:s3:::YOUR_BUCKET_NAME/*"
      ]
    }
  ]
}

Replace YOUR_BUCKET_NAME with your actual bucket name.

KMS-Encrypted Buckets

For KMS-encrypted buckets, add this to your KMS key policy:

{
  "Sid": "NimbleKMSAccess",
  "Effect": "Allow",
  "Principal": {
    "AWS": "arn:aws:iam::744254827463:user/webit-uploader"
  },
  "Action": [
    "kms:Encrypt",
    "kms:Decrypt",
    "kms:ReEncrypt*",
    "kms:GenerateDataKey*",
    "kms:DescribeKey"
  ],
  "Resource": "*"
}

Nimble Service Account:

[email protected]

Navigate to your bucket in the Google Cloud Console
Click on the Permissions tab
Click Grant Access
Add principal: [email protected]
Assign role: Storage Object Creator
Click Save

Submit request with storage config

Include storage_type and storage_url in your request.

from nimble_python import Nimble

nimble = Nimble(api_key="YOUR-API-KEY")

response = nimble.extract.async(
    url="https://www.nimbleway.com",
    render=True,
    formats=["html", "markdown"],
    storage_type="s3",
    storage_url="s3://your-bucket/nimble-results/",
    storage_compress=True
)

task_id = response.task_id
print(f"Results will be saved to: s3://your-bucket/nimble-results/{task_id}.json.gz")

from nimble_python import Nimble

nimble = Nimble(api_key="YOUR-API-KEY")

response = nimble.extract.async(
    url="https://www.nimbleway.com",
    render=True,
    formats=["html", "markdown"],
    storage_type="gs",
    storage_url="gs://your-bucket/nimble-results/"
)

task_id = response.task_id
print(f"Results will be saved to: gs://your-bucket/nimble-results/{task_id}.json")

Results delivered automatically

When complete, results are written to your bucket as {task_id}.json (or .json.gz if compressed).

Cloud delivery parameters

Parameter	Type	Description
`storage_type`	`s3` \| `gs`	Cloud provider
`storage_url`	string	Bucket path with prefix (e.g., `s3://bucket/prefix/`)
`storage_compress`	boolean	Enable GZIP compression
`storage_object_name`	string	Custom filename (default: task ID)

Comparison

Feature	Polling	Webhooks	Cloud Delivery
Setup complexity	None	Requires endpoint	Requires bucket setup
Real-time notifications	No (you poll)	Yes	No
Automatic storage	No	No	Yes
Best for	Simple integrations, testing	Event-driven apps	Data pipelines ETLs
Infrastructure needed	None	Web server	Cloud storage bucket

Combining methods

You can combine delivery methods for redundancy:

from nimble_python import Nimble

nimble = Nimble(api_key="YOUR-API-KEY")

# Receive webhook AND store in S3
response = nimble.extract_async(
    url="https://www.nimbleway.com",
    formats=["html", "markdown"],
    callback_url="https://your-server.com/webhooks/nimble",
    storage_type="s3",
    storage_url="s3://your-bucket/results/"
)

Best Practices

Polling

Check status first - Use /tasks/{id} before fetching full results
Use reasonable intervals - Poll every 2-5 seconds, not continuously
Handle rate limits - Implement retry logic for 429 responses
Set timeouts - Most tasks complete within seconds to minutes

Webhooks

Use HTTPS - Always use secure endpoints
Verify authenticity - Use custom headers for authentication
Respond quickly - Return 200 OK immediately, process async
Handle retries - Nimble retries failed deliveries

Cloud Delivery

Use prefixes - Organize by date, project, or type
Enable compression - Use storage_compress: true for large files
Set lifecycle policies - Auto-delete old files to manage costs
Use custom names - storage_object_name for meaningful filenames

Next Steps

Async Extract

Learn about async extraction options

Crawl API

Deep website crawling with async delivery

Agent Gallery

Browse available search agents

Rate Limits

Understand API rate limit

Introduction

Web Tools

Agentic

Admin

Polling

Callbacks

Cloud Delivery

Option 1: Polling (Pull)

Submit async request

Check status

Retrieve results

Polling endpoints reference

Option 2: Webhooks (Push)

Submit request with callback URL

Receive webhook notification

Webhook configuration options

Option 3: Cloud Delivery

Amazon S3

Google Cloud Storage

Configure bucket permissions (one-time)

Submit request with storage config

Results delivered automatically

Cloud delivery parameters

Comparison

Combining methods

Best Practices

Next Steps

Async Extract

Crawl API

Agent Gallery

Rate Limits

Introduction

Web Tools

Agentic

Admin

Polling

Callbacks

Cloud Delivery

​Option 1: Polling (Pull)

Submit async request

Check status

Retrieve results

​Polling endpoints reference

​Option 2: Webhooks (Push)

Submit request with callback URL

Receive webhook notification

​Webhook configuration options

​Option 3: Cloud Delivery

Amazon S3

Google Cloud Storage

Configure bucket permissions (one-time)

Submit request with storage config

Results delivered automatically

​Cloud delivery parameters

​Comparison

​Combining methods

​Best Practices

​Next Steps

Async Extract

Crawl API

Agent Gallery

Rate Limits

Option 1: Polling (Pull)

Polling endpoints reference

Option 2: Webhooks (Push)

Webhook configuration options

Option 3: Cloud Delivery

Cloud delivery parameters

Comparison

Combining methods

Best Practices

Next Steps