How to Evaluate Search APIs

Web search is one of the most valuable tools an agent can have. It lets the agent work with the live web to find fresh, verifiable, page-level information and ground its answers in real sources instead of relying only on what it already knows. Choosing the right Search API is harder. Providers differ in retrieval quality, freshness, content depth, latency, and how many searches and tokens the agent needs to complete a task. Public benchmarks rarely reflect a real workload. This guide walks through how to test search APIs based on your own real queries. Before starting, review the Search overview to get familiar with Nimble’s capabilities and configuration options.

1. Pick a starting configuration

Start with the configuration closest to the need:

If the need is…	Start with…
General grounding for a Q&A agent	`search_depth: "standard"`
High-volume URL discovery	`search_depth: "lite"`
Real-time freshness, such as prices, availability, or breaking news	`search_depth: "lite"`
Full page content rather than snippets	Add `full_content: true`
News and event monitoring	`search_depth: "lite"` with `focus: "news"`

Treat this as a starting hypothesis. Run a gold set against two or three plausible configurations before choosing. See the Search API reference for all parameters and supported combinations.

2. Build a fair evaluation

Build a gold set. Start with at least 50 real queries that represent what the agent searches for in production. For each query, record the expected answer, source, or relevance criteria. Include the query types that matter to the product, such as current information or breaking news, stable factual questions, finding a known page or source, and open-ended discovery or research. If search is already in production, sample from real query logs. For a new integration, use real user scenarios or expected inputs rather than generic examples. Fifty queries is enough for an initial signal. If providers score closely, expand the most important query categories before deciding. Keep it apples-to-apples. When comparing providers or configurations, keep the query set, result count, grader, timing, and downstream pipeline fixed. Change only the provider or configuration being evaluated. Run freshness-sensitive queries close together in time and record the exact parameters used for every run.

3. Run it

Run the evaluation with a coding agent

Paste this prompt into a coding agent (Claude Code, Cursor, or similar) to run the evaluation end to end.

You are evaluating Nimble Search for an AI-agent workload.

Before you begin

Inspect the workspace for an existing gold set, search integration, evaluation harness, API-key environment variable, and output convention.

Ask the user only for required information you cannot determine:
1. The gold set or the product use case from which to create one.
2. Which providers should be compared, if this is a provider comparison.
3. Any non-default success criteria that matter to the product.

Use these defaults unless the user specifies otherwise:
- Test search_depth: "standard".
- Also test search_depth: "lite" when the workload includes current or time-sensitive information.
- If no baseline is provided, evaluate Nimble independently.
- Save results in a new timestamped search-eval-results directory.
- Use an authenticated Nimble MCP connection or the API with NIMBLE_API_KEY. If neither is available, ask the user to connect MCP or configure the environment variable, never ask them to paste the key into chat.
- Use the evaluation metrics and grading rules defined in this guide.

Gold-set fields
Each example should contain:
- id
- query
- category
- expected answer or relevance criteria
- time_sensitive: true/false

Your task
1. Validate the gold set. Report missing or ambiguous grading criteria before running the evaluation.

1b. Propose a grading rubric before running anything and get it confirmed:
- Top-5 relevance: PASS when at least one result in the top five meets the gold set's relevance criteria, otherwise FAIL.
- Answer correctness: Use normalized exact matching when the expected answer is a short factual value. Otherwise, use the same blinded LLM judge and prompt for every provider, returning PASS, FAIL, or PARTIAL with a one-line rationale.
- Freshness: Score only queries with time_sensitive: true, using timestamped ground truth from the gold set. If that ground truth is missing, flag the query for review rather than scoring it.

Show the rubric, including how PARTIAL affects the final score, to the user for approval. Do not run or grade the evaluation until it is confirmed. Once confirmed, do not change it after seeing results.

2. Run every query against every configuration. Keep the query, result count, grader, and downstream pipeline fixed.
3. For each request, store the query and configuration, request timestamp, latency, request ID, and complete raw response.
4. Grade each result for relevant result in the top five, final-answer correctness, and freshness when applicable.
5. Classify every failure as: Search was not called; Search returned insufficient results; or Search returned sufficient results but synthesis failed.
6. Before calculating cost, pull current per-configuration pricing from each provider's official pricing page or API documentation. Use Nimble pricing (https://www.nimbleway.com/pricing) for Nimble. If pricing is unavailable, ask the user rather than estimating. Then calculate top-five retrieval success, answer accuracy, freshness accuracy, p50 and p95 latency, total cost, and cost per resolved query.
7. Produce a Markdown summary with a recommendation, a CSV with per-query scores, the untouched raw responses, and a list of uncertain cases requiring human review.

Rules
- Do not change the grading rubric after seeing results.
- Treat configuration changes as separate runs.
- Run all configurations and providers for a given time-sensitive query within the same batch, for example within the same five-minute window.
- Do not log or expose API keys.
- Do not invent ground truth. Flag uncertain cases for review.
- Preserve enough information for another person to reproduce the run.

Before finishing, verify that every gold-set query was tested under every requested configuration.

To call the API directly:

import requests

response = requests.post(
    "https://sdk.nimbleway.com/v2/search",
    headers={"Authorization": "Bearer <YOUR_API_KEY>"},
    json={
        "query": "your query here",
        "search_depth": "standard",  # or "lite"
        "full_content": False,    # set True for full page text
        "max_results": 10
    }
)
results = response.json()["results"]

Run this against the full gold set and store the results for grading. See the API reference for all parameters and the quickstart for Python, TypeScript, Go, CLI, and cURL examples.

4. Grade the results

Score every provider on the same small set of metrics. Choose the metrics that matter for the use case and define the grading rules before seeing the results. For at least a portion of the gold set, have a human directly review the results rather than relying solely on automated or LLM-judge scoring. Automated grading is fine for scaling up sample size once it’s been validated against a human-reviewed subset. Evaluate both the search API and agent end-to-end performance, for example with the following metrics:

Metric	What it measures	Why
Response accuracy	Whether the final agent response is correct	This is the actual product outcome.
Search API latency	Time from sending the search request to receiving the results	It isolates the speed of the provider against the full run.
End-to-end latency	Total time from the user request to the agent’s final answer	A fast individual call can still produce a slow answer if the agent needs multiple searches, extractions, or reasoning steps.
Cost per correct answer	Total search, extraction, and model cost for each correct answer	A cheaper call may cost more overall if the agent needs several calls to reach the correct answer.
Link recall and precision	Recall: how many relevant links were retrieved. Precision: how many retrieved links were relevant	Low recall causes missed answers or more searches; low precision adds noise and increases token usage.

Grade time-sensitive queries separately. For anything where the right answer changes over time, such as pricing, availability, breaking news, or inventory, capture what was true at request time and grade against that. A result that was correct when the query ran can look wrong if graded later.

5. Iterate on the configuration

Use the results to choose the next configuration to test. Change one setting at a time and rerun the same gold set.

Results are stale: Test search_depth: "lite".
The agent finds the right URLs but needs more content: Enable full_content: true so page text is returned with the search results.
The answer is in the results but the agent isn’t using it: Change the prompt.

6. Keep evaluating after integrating

A pre-commitment evaluation is a snapshot, not a guarantee. After integrating, log the query, configuration used, latency, and result count for a sample of production traffic, and periodically re-check accuracy against a rotating slice of real queries. Usage patterns and the live web both change in ways a one-time evaluation can’t catch.

Next steps

Quickstart

Make the first Search API request.

Agent skills and MCP setup

Evaluate Nimble from Claude Code, Cursor, Codex, or another coding assistant.

API Reference

Review all parameters and response schemas.

Talk to Sales

Get help building a gold set or running an evaluation.

Getting Started

Search Tools

Web Search Agent

Extraction Tools

Extract Template

Extract Jobs

Guides

SDKs

Admin

Proxy

How to Evaluate Search APIs

1. Pick a starting configuration

2. Build a fair evaluation

3. Run it

4. Grade the results

5. Iterate on the configuration

6. Keep evaluating after integrating

Next steps

Quickstart

Agent skills and MCP setup

API Reference

Talk to Sales

​1. Pick a starting configuration

​2. Build a fair evaluation

​3. Run it

​4. Grade the results

​5. Iterate on the configuration

​6. Keep evaluating after integrating

​Next steps

Quickstart

Agent skills and MCP setup

API Reference

Talk to Sales

1. Pick a starting configuration

2. Build a fair evaluation

3. Run it

4. Grade the results

5. Iterate on the configuration

6. Keep evaluating after integrating

Next steps