Links
Comment on page

Capturing Network Requests

Modern webpages are often a complex mixture of resources. Data is frequently post-loaded dynamically with javascript from internal APIs, not to mention the long lists of CSS, JS, and image resources that are loaded with every page.
Capturing network requests can be very beneficial for accessing these resources, and Nimble's Web API includes a very flexible tool for intercepting network requests.

Getting Started

Requests can be captured by using the network_capture field, which receives a JSON array that can include one or more filters. Each filter specifies which requests should be collected for later inspection.
The below example demonstrates two simple network capture filters, and how they are performed:
curl -X POST 'https://api.webit.live/api/v1/realtime/web' \
--header 'Authorization: Basic <credential string>' \
--header 'Content-Type: application/json' \
--data-raw '{
"url": "https://www.walmart.com/search?q=test&facet=fulfillment_method%3APickup",
"method": "GET",
"render": true,
"network_capture": [
{
"method": "GET",
"url": {
"type": "exact",
"value": "https://i5.walmartimages.com/a14/capabilities.min.js"
}
},
{
"url": {
"type": "contains",
"value": "/graphql/Search"
}
}
]
}'
Network capture is only available when render is set to true.

Exact match filter

In the first example filter, we ask to capture requests that match the URL we've provided exactly and that only use the GET method. Exact matches are very precise and can be used when you've determined exactly which network request you'd like to capture.

Containing a match

In the second example filter, we ask to capture all GraphQL requests by searching for requests to URLs that include "/graphql/Search". The request method has been intentionally left out to capture requests using any method.
Using containing filters helps broaden our network captures, and can be very useful when needing all files of a certain type (css, js, etc.), a particular request that includes dynamic variables, or when the exact request is still not fully known.

Response

In the example below, we see a fictional response to the example above. Notice how each network capture filter is included in the response, along with the collected data:
{
"status": "success",
"query_time": "...",
"html_content": "...",
"status_code": 200,
"headers": {
"content-encoding": "gzip",
"content-type": "text/html; charset=UTF-8",
"date": "...",
"last-modified": "...",
"server": "nginx",
"vary": "Accept-Encoding"
},
"url": "https://www.walmart.com/search?q=test&facet=fulfillment_method%3APickup",
"network_capture": [
{
"filter": {
"method": "GET",
"url": {
"type": "exact",
"value": "https://i5.walmartimages.com/a14/capabilities.min.js"
}
},
"results": [
{
"request": {
"method": "GET",
"url": "https://i5.walmartimages.com/a14/capabilities.min.js",
"headers": {
"sec-ch-ua": "\"Chromium\";v=\"112\", \"Google Chrome\";v=\"112\", \"Not:A-Brand\";v=\"99\"",
"referer": "https://www.example.com/",
"accept-language": "en,en;q=0.9",
"sec-ch-ua-mobile": "?0",
"user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.0.0 Safari/537.36",
"sec-ch-ua-platform": "\"Windows\""
}
},
"response": {
"status": 200,
"status_text": "",
"headers": {
"accept-ranges": "bytes",
"cache-control": "max-age=2702",
"content-length": "538511",
"content-type": "application/javascript",
"date": "...",
"etag": "\"6a6c1c29ddb3737a4398ff1f7b1063bf\"",
"last-modified": "...",
"strict-transport-security": "max-age=15552000; includeSubDomains; preload",
"vary": "Access-Control-Request-Headers,Access-Control-Request-Method,Origin",
"x-amz-request-id": "tx00000000000002e8d4651-0064943c55-6d194055-nyc3c",
"x-hw": "1687437271.dop007.ml1.t,1687437271.cds219.ml1.hn,1687437271.cds216.ml1.c",
"x-rgw-object-type": "Normal"
},
"serialization": "none",
"body": "..."
}
}
]
},
{
"filter": {
"url": {
"type": "contains",
"value": ".css"
}
},
"results": [
{
"request": {
"method": "GET",
"url": "https://www.example.com/wp-content/cache/min/1/10069917ea6c78bacd530335b1684679.css",
"headers": {
"sec-ch-ua": "\"Chromium\";v=\"112\", \"Google Chrome\";v=\"112\", \"Not:A-Brand\";v=\"99\"",
"referer": "https://www.example.com/",
"accept-language": "en,en;q=0.9",
"sec-ch-ua-mobile": "?0",
"user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.0.0 Safari/537.36",
"sec-ch-ua-platform": "\"Windows\""
}
},
"response": {
"status": 200,
"status_text": "",
"headers": {
"cache-control": "max-age=31536000",
"content-encoding": "gzip",
"content-length": "81027",
"content-type": "text/css",
"date": "Thu, 22 Jun 2023 12:34:30 GMT",
"etag": "\"647713da-13c83\"",
"expires": "Fri, 21 Jun 2024 12:34:30 GMT",
"last-modified": "Wed, 31 May 2023 09:31:06 GMT",
"server": "nginx",
"vary": "Accept-Encoding"
},
"serialization": "none",
"body": "..."
}
}
]
}
]
}
In the example response above, we see that network capture returns:
  • filter - the filter that was defined initially.
  • result - the requests found that match the filter
    • request - the headers and metadata of the request.
    • response - the headers and data of the response.
      • body - under response, body will contain the contents of the captured resource.

Operation Reference

Now that we've covered the basics of how to use network capture, the below table provided an in-depth reference to all available parameters and their values.
Parameter
Required
Type
Description
method
optional
string
When provided, what HTTP method should be filtered for. Examples: GET, POST, PUT
url
required
object
Describes the URL(s) to filter and capture.
type
optional - default = "exact"
enum
Can be set to "exact" or "contains".
value
required
string
The URL or URL portion to match to.