Skip to main content
POST
Create Crawl

Authorizations

Authorization
string
header
required

Bearer authentication header of the form Bearer <token>, where <token> is your auth token.

Body

application/json

Request body model for the /crawl endpoint

url
string
required

Url to crawl.

Allows the crawler to follow links to external websites.

Example:

false

allow_subdomains
boolean
default:false

Allows the crawler to follow links to subdomains of the main domain.

Example:

false

callback

Webhook configuration for receiving crawl results.

crawl_entire_domain
boolean
default:false

Allows the crawler to follow internal links to sibling or parent URLs, not just child paths.

Example:

false

exclude_paths
string[]

URL pathname regex patterns that exclude matching URLs from the crawl.

Example:
extract_options
object
ignore_query_parameters
boolean
default:false

Do not re-scrape the same path with different (or none) query parameters.

Example:

false

include_paths
string[]

URL pathname regex patterns that include matching URLs in the crawl.

Example:
limit
integer
default:5000

Maximum number of pages to crawl.

Required range: 1 <= x <= 10000
Example:

100

max_discovery_depth
integer
default:5

Maximum depth to crawl based on discovery order.

Required range: 1 <= x <= 20
Example:

3

name
string

Name of the crawl.

Example:

"The best crawl ever"

sitemap
enum<string>
default:include

Sitemap and other methods will be used together to find URLs.

Available options:
skip,
include,
only
Example:

"include"

Response

200 - application/json

Successful Response - Crawl Task Created

Crawl API response

account_name
string
required
crawl_id
string<uuid>
required
crawl_options
object
required
created_at
required
status
enum<string>
required
Available options:
queued,
running,
succeeded,
failed,
canceled
updated_at
required
url
string<uri>
required
completed
number
completed_at
extract_options
object | null
failed
number
name
string | null
pending
number
tasks
object[]
total
number