> ## Documentation Index
> Fetch the complete documentation index at: https://docs.nimbleway.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Evaluating Results

> Completed is not done: how to check a run, and how to iterate when it falls short

A run that says `completed` has finished. It has not necessarily done what you asked. Partial output, a truncated list and a padded list all complete. This page is how to tell them apart, and what to change when a run falls short.

## Three things to check on every run

**1. Rows returned against rows asked**

Asked for 40, got 12? The run truncated. That happens when the ask was wider than the budget, the criteria were exhausted, or the schema was too wide to fill. Nothing in the status tells you which; the count does.

**2. Cells filled**

Count non-null cells against the schema. The trust report states the fill rate in its reasoning for structured runs. Below about 80% on fields that should be public, the task or the schema needs work before the effort does.

**3. A sample of citations, opened**

Pick five claims, open the URLs, find the excerpt. A citation is where the agent found the claim, not proof the claim is true. Five minutes here catches the failure modes no metric catches: a wrong entity with the same name, a stale figure, an aggregator quoted as fact.

## What the confidence grade is, and is not

The [trust report](https://docs.nimbleway.com/nimble-sdk/web-search-agents/trust.md) grades every claim by fixed rules: primary source, independent corroboration, single secondary source, no usable citation. For structured runs the overall grade is the lower of fill rate and per-claim trust ratio.

That makes it a useful sort key and a poor verifier:

* A truncated run that filled every cell it did return grades `high`.
* A wrong value with a real-looking citation grades `high`.
* A fabricated value the agent attributed to a nearby article grades `high`.

Use the grade to order what to check first. Do not use it to skip checking, and do not use it as the metric when comparing two tiers or two prompts. Compare rows, fill and spot-checked accuracy instead.

## Three results that all say completed

**Full result**

```json theme={"system"}
{
  "type": "json",
  "content": [ /* 38 rows */ ],
  "trust": { "confidence": "high", "reasoning": "High confidence - cells fill rate is 97% and trust per claim ratio is 91%" }
}
```

Asked for 40, got 38, fill 97%. Spot-check five, ship.

**Partial result**

```json theme={"system"}
{
  "type": "json",
  "content": [ /* 9 rows, several fields null */ ],
  "trust": { "confidence": "medium", "reasoning": "Medium confidence - cells fill rate is 61% and trust per claim ratio is 88%" }
}
```

Asked for 40, got 9, fill 61%. The agent ran out of budget writing a wide schema, or the corpus was thin. Narrow the schema first, then scope the ask.

**Verified absence**

```json theme={"system"}
{
  "path": "$[3].latest_round_year",
  "confidence": "high",
  "reasoning": "Backed by a primary source (official)",
  "citations": [{ "url": "https://example.com/about", "excerpts": ["bootstrapped and profitable since 2019"] }]
}
```

The value is `null` and that is correct. Keep it.

## Before a batch: canaries

Pick ten inputs you already know the answers for, including two or three that should return nothing (a dead domain, a company acquired last year, a name that matches two firms). Run them first. All ten must come back right, and the empty ones must come back empty. On one 31-dealer pilot, five seeded dead domains were all flagged with citations; that result, not the grade, is what made the customer trust the other 26.

## At scale: ground truth and a ledger

* **Ground truth**: 50 to 100 rows compared against a source you trust, before the full job. Track filled cells and matches per field, not overall confidence.
* **Your own ledger**: which input rows have come back, keyed on your identifier. The API does not report coverage across runs. Dedupe on the key after every merge; duplicate rows do occur.
* **Top-ups**: re-run the misses with the completed rows passed as an exclusion list in `input_data`.

## Iterate in this order

When a run falls short, change one thing at a time, in this order. Each step is cheaper and more often the cause than the next.

**1. The task text**

Is the entity disambiguated? Is there a row cap, a stop rule, an absence rule? Are the channels named? See [Writing the Task](https://docs.nimbleway.com/nimble-sdk/web-search-agents/best-practices/writing-the-task.md).

**2. The schema**

Too wide? Undescribed fields? Required fields the web cannot fill? See [Output Schema](https://docs.nimbleway.com/nimble-sdk/web-search-agents/best-practices/output-schema.md).

**3. Goals and skill**

Turn caveats into checkable goals. Turn the skill into a procedure with steps.

**4. Sources**

Tier the allow list, block the noise, say that discovery pages are not citations. See [Controlling Sources](https://docs.nimbleway.com/nimble-sdk/web-search-agents/best-practices/sources.md).

**5. Effort**

Last. If rows and fill are still short after the four steps above, go up one tier and compare the same inputs. See [Effort and Cost](https://docs.nimbleway.com/nimble-sdk/web-search-agents/best-practices/effort-and-cost.md).

## Symptom, cause, fix

| You see | Likely cause | Fix |
| - | - | - |
| Far fewer rows than asked, run completed | Ask too wide, or schema too wide to fill | Row cap plus stop rule; narrow the schema; split by region or category |
| Enrichment rows at the end carry only input values | Batch larger than one run's budget | Halve the batch or cut researched fields; the fill rate in the trust reasoning shows how far the run got |
| Exactly the minimum number of rows | "At least N" read as a target | Say "do not stop before N unless the criteria are exhausted" |
| Cells filled with "Not stated publicly" or "Unknown" as text | No absence rule | Nullable fields plus "return null" in the skill |
| A person from company A listed under company B | Batch enrichment without a per-row constraint | "Constrain person fields to the row's own domain" in the skill; smaller batches |
| Same entity twice | No dedupe rule | "One row per company, deduplicated on domain" as a goal; dedupe on your key after merge |
| Citations are search result or directory URLs | Discovery pages allowed as evidence | `prioritize`: "discovery only, cite the underlying page" |
| Empty or near-empty result on a specific site | Login wall, anti-bot, or JavaScript-only page | Test the URL with Extract; use a template; drop the source |
| `422` on run creation | Schema keyword, `$ref`, more than 20 input rows or 200 cells | See [hard limits](https://docs.nimbleway.com/nimble-sdk/web-search-agents/best-practices/output-schema.md#hard-limits) |
| Run still active after 30 minutes | Heavy corpus at `x-high`, or a platform incident | Retry once at `high` with a narrower scope; report the run id if it recurs |

## Next Steps

> **[Trust](https://docs.nimbleway.com/nimble-sdk/web-search-agents/trust.md)**
>
> How grades are computed

> **[Running at Scale](https://docs.nimbleway.com/nimble-sdk/web-search-agents/best-practices/running-at-scale.md)**
>
> Ledgers, batches and top-ups in code

> **[Common Mistakes](https://docs.nimbleway.com/nimble-sdk/web-search-agents/best-practices/common-mistakes.md)**
>
> Wrong and right, side by side


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.