Delivery methods
The Nimble SERP API supports three delivery methods:
For real-time delivery, see our page on performing a real-time search request. To use Cloud or Push/Pull delivery, use an asynchronous request instead. Asynchronous requests also have the added benefit of running in the background, so you can continue working without waiting for the job to complete.
To send an asynchronous request, use the /async/serp endpoint with the following syntax:
curl -X POST 'https://api.webit.live/api/v1/async/serp' \
--header 'Authorization: Basic <credential string>' \
--header 'Content-Type: application/json' \
--data-raw '{
"search_engine": "google_search",
"country": "US",
"query": "Sample search phrase",
"storage_type": "s3",
"storage_url": "s3://sample-bucket",
"callback_url": "https://my.callback/path"
}'
Request Options
Asynchronous requests share the same parameters as real-time requests, but also include storage and callback parameters.
query
Required
String
The term or phrase to search for.
search_engine
Required
Enum: google_search | bing_search | yandex_search
The search engine from which to collect results.
tab
Optional (default = null)
Enum: news
When using google_search
, setting tab
to news
will provide Google News results instead of standard search results.
num_results
Optional
Integer
Set the mount of retuned search results
domain
Optional
String
Search through a custom top-level domain of Google. eg: "co.uk"
country
Optional (default = all)
String
Country used to access the target URL, use ISO Alpha-2 Country Codes i.e. US, DE, GB
state
Optional
String
For targeting US states (does not include regions or territories in other countries). Two-letter state code, e.g. NY, IL, etc.
city
Optional
String
For targeting large cities and metro areas around the globe. When targeting major US cities, you must include state as well. Click here for a list of available cities.
locale
Optional (default = en)
String
String | LCID standard locale used for the URL request. Alternatively, user can use auto
for automatic locale based on country targeting.
location
Optional
String
Search Google through a custom geolocation, regardless of country or proxy location. eg: "London,Ohio,United States". See Getting local data for more information.
parse
Optional (default = true)
Boolean
Instructs Nimble whether to structure the results into a JSON format or return the raw HTML.
ads_optimization
Optional (default = false)
Boolean
This flag increases the number of paid ads (sponsored ads) in the results. It works by running the requests in 'incognito' mode.
storage_type
Required
ENUM: s3 | gs
Use s3 for Amazon S3 and gs for Google Cloud Platform
storage_url
Required
String
Repository URL: s3://Your.Bucket.Name/your/object/name/prefix/ | Output will be saved to TASK_ID.json
callback_url
Optional
String
A url to callback once the data is delivered. Nimble APIs will send a POST request to the callback_url with the task details once the task is complete (this “notification” will not include the requested data).
storage_compress
Optional (default = false)
Boolean
When set to true
, the response saved to the storage_url
will be compressed using GZIP format. This can help reduce storage size and improve data transfer efficiency. If not set or set to false
, the response will be saved in its original uncompressed format.
Please add Nimble's system/service user to your GCS or S3 bucket to ensure that data can be delivered successfully.
Selecting a delivery method
To enable cloud delivery:
Set the storage_type parameter to either s3 or gs
Set the storage_url parameter to the bucket/folder URL of your cloud storage where you'd like the data to be saved.
To enable Push/Pull delivery:
Leave both the storage_type and storage_url fields blank. Nimble will automatically recognize that Push/Pull delivery has been selected.
Response
Initial Response
When an asynchronous request is initiated, an initial response with a state of “pending” is served to the user, along with a TaskID that can be used to check the task’s progress. The TaskID is also included in the result file name, for easier tracking of the result file in the destination repository.
200 OK
{
"status": "success",
"task": {
"id": "Task_ID",
"state": "pending",
"output_url": "s3://Your.Repository.Path/Task_ID.json",
"callback_url": "https://your.callback.url/path",
"status_url": "https://api.webit.live/api/v1/tasks/Task_ID",
"created_at": "2022-07-12T11:31:06.149Z",
"modified_at": "2022-07-12T11:31:07.765Z",
"input": {
"search_engine": "google_search",
"country": "FR",
"locale": "fr",
"query": "Sample search phrase",
"storage_type": "s3",
"storage_url": "s3://sample-bucket",
"callback_url": "https://my.callback/path"
}
"status_code": 200
}
}
Checking Task Status
Asynchronous tasks can be in one of four potential states:
success
Task was complete and stored in the destination repository.
failed
Nimble was unable to complete the task, no file was created in the repository. A failed state will also be returned in case of unidentified or expired TaskID.
pending
The task is still being processed.
uploading
The results are being uploaded to the destination repository.
To check the status of an asynchronous task, use the following endpoint:
GET https://api.webit.live/api/v1/tasks/<task_id>
For example:
curl -X GET 'https://api.webit.live/api/v1/tasks/Task_ID' \
--header 'Authorization: Basic <credential string>'
The response object has the same structure as the Task Completion object that is sent to the callback_url upon task completion.
Checking Tasks List
To check the status of asynchronous tasks list, use the endpoint
Path https://api.webit.live/api/v1/tasks/list
Parameters
limit
Optional (default = 100)
Number | List item limit
cursor
Optional
String | Cursor for pagination.
Example Request:
curl -X GET 'https://api.webit.live/api/v1/tasks/list?limit=20' \
--header 'Authorization: Basic <credential string>'
Example Response:
{
"data": [
...
],
"pagination": {
"hasNext": true,
"nextCursor": ...,
"total": 102
}
}
The response objects within data
has the same structure as the Task Completion object that is sent to the callback_url upon task completion.
For pagination, run until pagination.hasNext = false or pagination.nextCursor = null
Task Completion
A POST request will be sent to the callback_url once the task is complete. If you opted for Push/Pull delivery, the output_url
field will contain the address from which your data can be downloaded.
{
"status": "success",
"task": {
"id": "Task_ID",
"state": "success",
"output_url": "s3://Your.Repository.Path/Task_ID.json",
"callback_url": "https://your.callback.url/path",
"status_url": "https://api.webit.live/api/v1/tasks/Task_ID",
"created_at": "2022-07-12T11:31:06.149Z",
"modified_at": "2022-07-12T11:31:07.765Z",
"input": {...},
"status_code": 200
}
}
Handling Cloud Delivery Failure
If an error is encountered in saving the result data to your cloud storage for any reason, the data will be available to download from Nimble’s servers for 24 hours. If this is the case, the task state will report “success”, but will include two additional fields – download_url and error. The download_url
field will contain the address from which your data can be downloaded.
{
"status": "success",
"task": {
"id": "Task_ID",
"state": "success",
"output_url": "s3://Your.Repository.Path/Task_ID.json",
"callback_url": "https://your.callback.url/path",
"status_url": "https://api.webit.live/api/v1/tasks/Task_ID",
"created_at": "2022-07-12T11:31:06.149Z",
"modified_at": "2022-07-12T11:31:07.765Z",
"input": {...},
"status_code": 200
"download_url": "https://api.webit.live/api/v1/tasks/d176cfe9-8894-48bb-a058-eeba473a638a/results",
"error": "callback: request to https://your.callback.url/path failed, reason: getaddrinfo ENOTFOUND your.callback.url, upload: The specified bucket does not exist"
}
}
500 Error
{
"status": "error",
"task_id": "<task_id>",
"msg": "can't download the query response - please try again"
}
400 Input Error
{
"status": "failed",
"msg": error
}
Response Codes
200
OK
400
The requested resource could not be reached
401
Unauthorized/invalid credental string
500
Internal Service Error
501
An error was encountered by the proxy service
Last updated