Callbacks & Delivery - Nimble Docs

When using async operations like /extract/async, /extract/batch, /agent/async, or /crawl, you have three flexible options for receiving your results. Choose the method that best fits your infrastructure and workflow.

Polling

Pull results on-demand using task IDs

Callbacks

Receive push notifications when tasks complete

Cloud Delivery

Automatic delivery to your S3 or GCS bucket

Option 1: Polling (Pull)

The simplest approach - submit your async request, receive a task ID, and poll for results when ready.

Submit async request

Send a request to the async endpoint. You’ll receive a task or crawl ID to track your request.

Extract
Agent
Crawl
Batch

from nimble_python import Nimble

nimble = Nimble(api_key="YOUR-API-KEY")

response = nimble.extract_async(
    url="https://www.nimbleway.com",
    render=True,
    formats=["html", "markdown"]
)

task_id = response.task_id
print(f"Task submitted: {task_id}")

from nimble_python import Nimble

nimble = Nimble(api_key="YOUR-API-KEY")

response = nimble.agent.run_async(
    agent="amazon_pdp",
    params={"asin": "B0DLKFK6LR"}
)

task_id = response.task_id
print(f"Task submitted: {task_id}")

from nimble_python import Nimble

nimble = Nimble(api_key="YOUR-API-KEY")

result = nimble.crawl.run(
    url="https://www.nimbleway.com",
    limit=50
)

crawl_id = result.crawl_id
print(f"Crawl started: {crawl_id}")

Submit up to 1,000 URLs in one request. Each URL becomes its own task, tracked under a shared batch_id.

from nimble_python import Nimble

nimble = Nimble(api_key="YOUR-API-KEY")

response = nimble.extract_batch(
    inputs=[
        {"url": "https://www.example.com/page1"},
        {"url": "https://www.example.com/page2"},
        {"url": "https://www.example.com/page3"},
    ],
    shared_inputs={
        "render": True,
        "country": "US",
        "formats": ["markdown"],
    }
)

batch_id = response.batch_id
print(f"Batch submitted: {batch_id} ({response.batch_size} tasks)")

Check status

Poll the status endpoint to monitor progress.

Extract / Agent
Batch
Crawl

import time

while True:
    my_task = nimble.tasks.get(task_id)
    print(f"Status: {my_task.task.state}")

    if my_task.task.state == "success":
        break
    elif my_task.task.state == "error":
        print(f"Task failed: {my_task.task.error}")
        break

    time.sleep(15)

Use the lightweight progress endpoint for polling. Once completed is true, fetch task details.

import time

while True:
    progress = nimble.batches.progress(batch_id)
    print(f"Progress: {progress.completed_count} / {progress.total} ({progress.progress:.0%})")

    if progress.completed:
        break

    time.sleep(15)

# Get all task IDs and states
batch = nimble.batches.get(batch_id)
for task in batch.tasks:
    print(f"Task {task.id}: {task.state}")

Crawl has its own status endpoint that shows overall progress and individual page tasks.

import time

while True:
    my_crawl = nimble.crawl.status(crawl_id)
    print(f"Status: {my_crawl.status}")

    if status.state == "succeeded":
        break
    elif status.state == "failed":
        print(f"Task failed: {status.error}")
        break

    time.sleep(2)

Retrieve results

Once complete, fetch the full results.

Extract
Agent
Crawl
Batch

results = nimble.tasks.results(task_id)

print(f"HTML length: {len(results.data.html)}")
print(f"Markdown length: {len(results.data.markdown)}")

results = nimble.tasks.results(task_id)

parsed = results.data.parsing["parsed"]
print(f"Product: {parsed['product_title']}")
print(f"Price: ${parsed['web_price']}")

Fetch results for each completed task in the crawl.

# my_crawl["tasks"] from step #2 contains list of task IDs from status response
for task in my_crawl["tasks"]:
    if task.state == "success":
        task_result = nimble.tasks.results(task.id)

        print(f"URL: {task_result.url}")
        print(f"HTML length: {len(task_result.data.get('html', ''))}")

Once the batch is complete, retrieve results for each successful task using the task IDs from GET /v1/batches/{batch_id}.

# batch.tasks from step #2 — iterate and fetch each result
for task in batch.tasks:
    if task.state == "success":
        result = nimble.tasks.results(task.id)
        print(f"Markdown: {result.data.markdown[:200]}")
    elif task.state == "error":
        print(f"Task {task.id} failed: {task.error}")

Example Results

{
    "url": "https://www.nimbleway.com/blog/post",
    "task_id": "ec89b1f7-1cf2-40eb-91b4-78716093f9ed",
    "status": "success",
    "task": {
        "id": "ec89b1f7-1cf2-40eb-91b4-78716093f9ed",
        "state": "success",
        "created_at": "2026-02-09T23:15:43.549Z",
        "modified_at": "2026-02-09T23:16:39.094Z",
        "account_name": "your-account"
    },
    "data": {
        "html": "<!DOCTYPE html>...",
        "markdown": "# Page Title\n\nContent...",
        "headers": { ... }
    },
    "metadata": {
        "query_time": "2026-02-09T23:15:43.549Z",
        "query_duration": 1877,
        "response_parameters": {
            "input_url": "https://www.nimbleway.com/blog/post"
        },
		"driver": "vx6"
    },
    "status_code": 200
}

Polling endpoints reference

API	Submit	Check Status	Get Results
Extract	`POST /v1/extract/async`	`GET /v1/tasks/{task_id}`	`GET /v1/tasks/{task_id}/results`
Agent	`POST /v1/agent/async`	`GET /v1/tasks/{task_id}`	`GET /v1/tasks/{task_id}/results`
Batch	`POST /v1/extract/batch`	`GET /v1/batches/{batch_id}/progress`	`GET /v1/tasks/{task_id}/results` (per task)
Crawl	`POST /v1/crawl`	`GET /v1/crawl/{crawl_id}`	`GET /v1/tasks/{task_id}/results` (per page)

To list all tasks across your account, use GET /v1/tasks (supports cursor and limit for pagination). To list all batches, use GET /v1/batches.

Option 2: Webhooks (Push)

Get notified automatically when your tasks complete. Perfect for event-driven architectures.

Submit request with callback URL

Include callback_url (or callback object for crawl) in your async request.

Extract
Agent
Crawl

from nimble_python import Nimble

nimble = Nimble(api_key="YOUR-API-KEY")

response = nimble.extract_async(
    url="https://www.nimbleway.com",
    render=True,
    formats=["html", "markdown"],
    callback_url="https://your-server.com/webhooks/nimble"
)

task_id = response.task_id
print(f"Task submitted: {task_id}")
print("Results will be POSTed to your callback URL when ready")

from nimble_python import Nimble

nimble = Nimble(api_key="YOUR-API-KEY")

response = nimble.extract_async(
    url="https://www.nimbleway.com",
    render=True,
    formats=["html", "markdown"],
    callback_url="https://your-server.com/webhooks/nimble"
)

task_id = response.task_id
print(f"Task submitted: {task_id}")
print("Results will be POSTed to your callback URL when ready")

Crawl uses a callback object for advanced webhook options.

from nimble_python import Nimble

nimble = Nimble(api_key="YOUR-API-KEY")

result = nimble.crawl.run(
    url="https://www.nimbleway.com",
    limit=100,
    callback={
        "url": "https://your-server.com/webhooks/nimble",
        "headers": {
            "X-Custom-Auth": "your-secret-token"
        },
        "events": ["completed", "failed"]
    }
)

print(f"Crawl started: {result.crawl_id}")

Receive webhook notification

Nimble sends a POST to your callback URL when complete:

{
  "task": {
    "id": "8e8cfde8-345b-42b8-b3e2-0c61eb11e00f",
    "state": "success",
    "status_code": 200,
    "created_at": "2026-01-24T12:36:24.685Z",
    "modified_at": "2026-01-24T12:36:24.685Z",
    "input": {},
    "api_type": "extract"
  }
}

Webhook configuration options

API	Parameter	Type	Description
Extract	`callback_url`	string	Your callback URL
Agent	`params.callback_url`	string	Your callback URL
Crawl	`callback.url`	string	Your callback URL
	`callback.headers`	object	Custom headers for authentication
	`callback.metadata`	object	Custom data included in payload
	`callback.events`	array	Filter events: `started`, `page`, `completed`, `failed`

Option 3: Cloud Delivery (Async API)

Applies to async API requests - /extract/async, /extract/batch, /agent/async, /crawl. For Nimble Jobs, see Option 4: Jobs cloud delivery below.

Automatically deliver results directly to your cloud storage bucket.

Amazon S3

Deliver to any S3 bucket in your AWS account

Google Cloud Storage

Deliver to any GCS bucket in your GCP project

Configure bucket permissions (one-time)

Grant Nimble’s service account write access to your bucket.

Amazon S3
Google Cloud Storage

Nimble Service User ARN:

arn:aws:iam::744254827463:user/webit-uploader

Add this bucket policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "NimbleCloudDelivery",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::744254827463:user/webit-uploader"
      },
      "Action": [
        "s3:PutObject",
        "s3:PutObjectACL",
        "s3:GetBucketLocation"
      ],
      "Resource": [
        "arn:aws:s3:::YOUR_BUCKET_NAME",
        "arn:aws:s3:::YOUR_BUCKET_NAME/*"
      ]
    }
  ]
}

Replace YOUR_BUCKET_NAME with your actual bucket name.

KMS-Encrypted Buckets

For KMS-encrypted buckets, add this to your KMS key policy:

{
  "Sid": "NimbleKMSAccess",
  "Effect": "Allow",
  "Principal": {
    "AWS": "arn:aws:iam::744254827463:user/webit-uploader"
  },
  "Action": [
    "kms:Encrypt",
    "kms:Decrypt",
    "kms:ReEncrypt*",
    "kms:GenerateDataKey*",
    "kms:DescribeKey"
  ],
  "Resource": "*"
}

Nimble Service Account:

[email protected]

Navigate to your bucket in the Google Cloud Console
Click on the Permissions tab
Click Grant Access
Add principal: [email protected]
Assign role: Storage Object Creator
Click Save

Submit request with storage config

Include storage_type and storage_url in your request.

Cloud delivery parameters

Parameter	Type	Description
`storage_type`	`s3` \| `gs`	Cloud provider
`storage_url`	string	Bucket path with prefix (e.g., `s3://bucket/prefix/`)
`storage_compress`	boolean	Enable GZIP compression
`storage_object_name`	string	Custom filename (default: task ID)

Extract
Agent

from nimble_python import Nimble

nimble = Nimble(api_key="YOUR-API-KEY")

response = nimble.extract_async(
    url="https://www.nimbleway.com",
    render=True,
    formats=["html", "markdown"],
    storage_type="s3",
    storage_url="s3://your-bucket/nimble-results/",
    storage_compress=True,
    storage_object_name="my-result"
)

task_id = response.task_id
print(f"Results will be saved to: s3://your-bucket/nimble-results/my-result.json.gz")

from nimble_python import Nimble

nimble = Nimble(api_key="YOUR-API-KEY")

response = nimble.extract_async(
    url="https://www.nimbleway.com",
    render=True,
    formats=["html", "markdown"],
    storage_type="gs",
    storage_url="gs://your-bucket/nimble-results/",
    storage_object_name="my-result"
)

task_id = response.task_id
print(f"Results will be saved to: gs://your-bucket/nimble-results/my-result.json")

from nimble_python import Nimble

nimble = Nimble(api_key="YOUR-API-KEY")

response = nimble.agent.run_async(
    agent="amazon_pdp",
    params={
        "asin": "B0DLKFK6LR",
        "storage_type": "s3",
        "storage_url": "s3://your-bucket/nimble-results/",
        "storage_compress": True,
        "storage_object_name": "my-result"
    }
)

task_id = response.task_id
print(f"Results will be saved to: s3://your-bucket/nimble-results/my-result.json.gz")

from nimble_python import Nimble

nimble = Nimble(api_key="YOUR-API-KEY")

response = nimble.agent.run_async(
    agent="amazon_pdp",
    params={
        "asin": "B0DLKFK6LR",
        "storage_type": "gs",
        "storage_url": "gs://your-bucket/nimble-results/",
        "storage_object_name": "my-result"
    }
)

task_id = response.task_id
print(f"Results will be saved to: gs://your-bucket/nimble-results/my-result.json")

Results delivered automatically

When complete, results are written to your bucket as {task_id}.json (or .json.gz if compressed).

Option 4: Jobs cloud delivery

Applies to Nimble Jobs only - used as the input source and/or destination for a Job. For async API requests, see Option 3 above.

A Job can read its input set from an S3 bucket and write its assembled results back to one. Both directions are wired through a single bucket policy that grants Nimble’s IAM principal access to specific prefixes. Nimble does not assume a role in the customer account.

Nimble’s IAM principal

Every Job runs under a single IAM user:

arn:aws:iam::744254827463:user/crawlit-scrapy

Use this ARN as the Principal in the bucket policy.

Required permissions

Prefix	Permission
`s3://YOUR_BUCKET` (whole bucket)	`s3:ListBucket`
`s3://YOUR_BUCKET/input/*`	`s3:GetObject`
`s3://YOUR_BUCKET/output/*`	`s3:GetObject`, `s3:PutObject`, `s3:AbortMultipartUpload`, `s3:ListMultipartUploadParts`

s3:AbortMultipartUpload and s3:ListMultipartUploadParts are required because large output files are written in parts. If an upload fails mid-way, Nimble aborts the incomplete multipart upload so partial bytes do not accumulate in the bucket.

s3:DeleteObject is not requested. Nimble never deletes files in the bucket - including the connection-test probe described below.

Bucket policy template

Replace YOUR_BUCKET with the bucket name. Adjust the input/ and output/ prefixes to match the paths used in the Job form.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "NimbleListBucket",
      "Effect": "Allow",
      "Principal": { "AWS": "arn:aws:iam::744254827463:user/crawlit-scrapy" },
      "Action": "s3:ListBucket",
      "Resource": "arn:aws:s3:::YOUR_BUCKET"
    },
    {
      "Sid": "NimbleReadInputPrefix",
      "Effect": "Allow",
      "Principal": { "AWS": "arn:aws:iam::744254827463:user/crawlit-scrapy" },
      "Action": "s3:GetObject",
      "Resource": "arn:aws:s3:::YOUR_BUCKET/input/*"
    },
    {
      "Sid": "NimbleReadWriteOutputPrefix",
      "Effect": "Allow",
      "Principal": { "AWS": "arn:aws:iam::744254827463:user/crawlit-scrapy" },
      "Action": [
        "s3:GetObject",
        "s3:PutObject",
        "s3:AbortMultipartUpload",
        "s3:ListMultipartUploadParts"
      ],
      "Resource": "arn:aws:s3:::YOUR_BUCKET/output/*"
    }
  ]
}

Applying the policy

Open the bucket

In the AWS Console, open S3 and click the bucket Nimble should access.

Edit the bucket policy

Open the Permissions tab. Click Edit under Bucket policy.

Paste the template

Paste the JSON above. Replace YOUR_BUCKET and the prefixes. Save changes.

Verify in Nimble

Open the Job form. Paste the s3://... path and click Test Connection. A green chip confirms the policy is correct.

Keep Block Public Access enabled. The policy grants access only to Nimble’s IAM user, not to the public.

Test Connection - what it does

Mode	Operation	Verifies
Read (input path)	`head_bucket` + list one object under the prefix	`s3:ListBucket` + `s3:GetObject`
Write (output path)	`head_bucket` + `put_object` of an empty `.nimble-connection-test` file at the prefix	`s3:ListBucket` + `s3:PutObject`

The write test leaves an empty .nimble-connection-test file under the output prefix. The file is not deleted - the policy does not grant s3:DeleteObject, so cleanup is not possible. The key is deterministic, so repeated tests overwrite the same object. At most one residue file per output prefix. Remove it manually at any time.

Interpreting the result

Chip	Meaning	Fix
✓ Connection OK	Permissions correct. The prefix has files.	Ready to use.
✓ Connection OK - prefix is empty	Permissions correct. The prefix has no files yet.	Upload the first input file.
✗ Access denied	The bucket policy is missing a required permission.	Re-apply the template above.
✗ Bucket does not exist	The bucket name in the path is wrong.	Verify the bucket name.
✗ Connection failed	Network, throttling, or unexpected S3 error.	Retry. Contact support if it persists.

Comparison

Feature	Polling	Webhooks	Cloud Delivery
Setup complexity	None	Requires endpoint	Requires bucket setup
Real-time notifications	No (you poll)	Yes	No
Automatic storage	No	No	Yes
Best for	Simple integrations, testing	Event-driven apps	Data pipelines ETLs
Infrastructure needed	None	Web server	Cloud storage bucket

Combining methods

You can combine delivery methods for redundancy:

from nimble_python import Nimble

nimble = Nimble(api_key="YOUR-API-KEY")

# Receive webhook AND store in S3
response = nimble.extract_async(
    url="https://www.nimbleway.com",
    formats=["html", "markdown"],
    callback_url="https://your-server.com/webhooks/nimble",
    storage_type="s3",
    storage_url="s3://your-bucket/results/"
)

Best Practices

Polling

Check status first - Use /tasks/{id} before fetching full results - Use reasonable intervals - Poll every 2-5 seconds, not continuously - Handle rate limits - Implement retry logic for 429 responses - Set timeouts - Most tasks complete within seconds to minutes

Webhooks

Use HTTPS - Always use secure endpoints - Verify authenticity - Use custom headers for authentication - Respond quickly - Return 200 OK immediately, process async - Handle retries - Nimble retries failed deliveries

Cloud Delivery

Use prefixes - Organize by date, project, or type - Enable compression - Use storage_compress: true for large files - Set lifecycle policies - Auto-delete old files to manage costs - Use custom names - storage_object_name for meaningful filenames

Next Steps

Async Extract

Learn about async extraction options

Crawl API

Deep website crawling with async delivery

Agent Gallery

Browse available search agents

Rate Limits

Understand API rate limit

Polling

Callbacks

Cloud Delivery

​Option 1: Polling (Pull)

Submit async request

Check status

Retrieve results

​Polling endpoints reference

​Option 2: Webhooks (Push)

Submit request with callback URL

Receive webhook notification

​Webhook configuration options

​Option 3: Cloud Delivery (Async API)

Amazon S3

Google Cloud Storage

Configure bucket permissions (one-time)

Submit request with storage config

​Cloud delivery parameters

Results delivered automatically

​Option 4: Jobs cloud delivery

​Nimble’s IAM principal

​Required permissions

​Bucket policy template

​Applying the policy

​Test Connection - what it does

​Interpreting the result

​Comparison

​Combining methods

​Best Practices

​Next Steps

Async Extract

Crawl API

Agent Gallery

Rate Limits

Option 1: Polling (Pull)

Polling endpoints reference

Option 2: Webhooks (Push)

Webhook configuration options

Option 3: Cloud Delivery (Async API)

Cloud delivery parameters

Option 4: Jobs cloud delivery

Nimble’s IAM principal

Required permissions

Bucket policy template

Applying the policy

Test Connection - what it does

Interpreting the result

Comparison

Combining methods

Best Practices

Next Steps