Comment on page
Capturing Network Requests
Modern webpages are often a complex mixture of resources. Data is frequently post-loaded dynamically with javascript from internal APIs, not to mention the long lists of CSS, JS, and image resources that are loaded with every page.
Capturing network requests can be very beneficial for accessing these resources, and Nimble's Web API includes a very flexible tool for intercepting network requests.
Requests can be captured by using the
network_capture field, which receives a JSON array that can include one or more filters. Each filter specifies which requests should be collected for later inspection.The below example demonstrates two simple network capture filters, and how they are performed:
curl -X POST 'https://api.webit.live/api/v1/realtime/web' \
--header 'Authorization: Basic <credential string>' \
--header 'Content-Type: application/json' \
--data-raw '{
"url": "https://www.walmart.com/search?q=test&facet=fulfillment_method%3APickup",
"method": "GET",
"render": true,
"network_capture": [
{
"method": "GET",
"url": {
"type": "exact",
"value": "https://i5.walmartimages.com/a14/capabilities.min.js"
}
},
{
"url": {
"type": "contains",
"value": "/graphql/Search"
}
}
]
}'
Network capture is only available when render is set to true.
In the first example filter, we ask to capture requests that match the URL we've provided exactly and that only use the GET method. Exact matches are very precise and can be used when you've determined exactly which network request you'd like to capture.
In the second example filter, we ask to capture all GraphQL requests by searching for requests to URLs that include "/graphql/Search". The request method has been intentionally left out to capture requests using any method.
Using containing filters helps broaden our network captures, and can be very useful when needing all files of a certain type (css, js, etc.), a particular request that includes dynamic variables, or when the exact request is still not fully known.
In the example below, we see a fictional response to the example above. Notice how each network capture filter is included in the response, along with the collected data:
{
"status": "success",
"query_time": "...",
"html_content": "...",
"status_code": 200,
"headers": {
"content-encoding": "gzip",
"content-type": "text/html; charset=UTF-8",
"date": "...",
"last-modified": "...",
"server": "nginx",
"vary": "Accept-Encoding"
},
"url": "https://www.walmart.com/search?q=test&facet=fulfillment_method%3APickup",
"network_capture": [
{
"filter": {
"method": "GET",
"url": {
"type": "exact",
"value": "https://i5.walmartimages.com/a14/capabilities.min.js"
}
},
"results": [
{
"request": {
"method": "GET",
"url": "https://i5.walmartimages.com/a14/capabilities.min.js",
"headers": {
"sec-ch-ua": "\"Chromium\";v=\"112\", \"Google Chrome\";v=\"112\", \"Not:A-Brand\";v=\"99\"",
"referer": "https://www.example.com/",
"accept-language": "en,en;q=0.9",
"sec-ch-ua-mobile": "?0",
"user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.0.0 Safari/537.36",
"sec-ch-ua-platform": "\"Windows\""
}
},
"response": {
"status": 200,
"status_text": "",
"headers": {
"accept-ranges": "bytes",
"cache-control": "max-age=2702",
"content-length": "538511",
"content-type": "application/javascript",
"date": "...",
"etag": "\"6a6c1c29ddb3737a4398ff1f7b1063bf\"",
"last-modified": "...",
"strict-transport-security": "max-age=15552000; includeSubDomains; preload",
"vary": "Access-Control-Request-Headers,Access-Control-Request-Method,Origin",
"x-amz-request-id": "tx00000000000002e8d4651-0064943c55-6d194055-nyc3c",
"x-hw": "1687437271.dop007.ml1.t,1687437271.cds219.ml1.hn,1687437271.cds216.ml1.c",
"x-rgw-object-type": "Normal"
},
"serialization": "none",
"body": "..."
}
}
]
},
{
"filter": {
"url": {
"type": "contains",
"value": ".css"
}
},
"results": [
{
"request": {
"method": "GET",
"url": "https://www.example.com/wp-content/cache/min/1/10069917ea6c78bacd530335b1684679.css",
"headers": {
"sec-ch-ua": "\"Chromium\";v=\"112\", \"Google Chrome\";v=\"112\", \"Not:A-Brand\";v=\"99\"",
"referer": "https://www.example.com/",
"accept-language": "en,en;q=0.9",
"sec-ch-ua-mobile": "?0",
"user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.0.0 Safari/537.36",
"sec-ch-ua-platform": "\"Windows\""
}
},
"response": {
"status": 200,
"status_text": "",
"headers": {
"cache-control": "max-age=31536000",
"content-encoding": "gzip",
"content-length": "81027",
"content-type": "text/css",
"date": "Thu, 22 Jun 2023 12:34:30 GMT",
"etag": "\"647713da-13c83\"",
"expires": "Fri, 21 Jun 2024 12:34:30 GMT",
"last-modified": "Wed, 31 May 2023 09:31:06 GMT",
"server": "nginx",
"vary": "Accept-Encoding"
},
"serialization": "none",
"body": "..."
}
}
]
}
]
}
In the example response above, we see that network capture returns:
- filter - the filter that was defined initially.
- result - the requests found that match the filter
- request - the headers and metadata of the request.
- response - the headers and data of the response.
- body - under response,
bodywill contain the contents of the captured resource.
Now that we've covered the basics of how to use network capture, the below table provided an in-depth reference to all available parameters and their values.
Parameter | Required | Type | Description |
|---|---|---|---|
method | optional | string | When provided, what HTTP method should be filtered for. Examples: GET, POST, PUT |
url | required | object | Describes the URL(s) to filter and capture. |
type | optional -
default = "exact" | enum | Can be set to "exact" or "contains". |
value | required | string | The URL or URL portion to match to. |