Skip to main content
A run that says completed has finished. It has not necessarily done what you asked. Partial output, a truncated list and a padded list all complete. This page is how to tell them apart, and what to change when a run falls short.

​
Three things to check on every run

1

Rows returned against rows asked

Asked for 40, got 12? The run truncated. That happens when the ask was wider than the budget, the criteria were exhausted, or the schema was too wide to fill. Nothing in the status tells you which; the count does.
2

Cells filled

Count non-null cells against the schema. The trust report states the fill rate in its reasoning for structured runs. Below about 80% on fields that should be public, the task or the schema needs work before the effort does.
3

A sample of citations, opened

Pick five claims, open the URLs, find the excerpt. A citation is where the agent found the claim, not proof the claim is true. Five minutes here catches the failure modes no metric catches: a wrong entity with the same name, a stale figure, an aggregator quoted as fact.

​
What the confidence grade is, and is not

The trust report grades every claim by fixed rules: primary source, independent corroboration, single secondary source, no usable citation. For structured runs the overall grade is the lower of fill rate and per-claim trust ratio. That makes it a useful sort key and a poor verifier:
  • A truncated run that filled every cell it did return grades high.
  • A wrong value with a real-looking citation grades high.
  • A fabricated value the agent attributed to a nearby article grades high.
Use the grade to order what to check first. Do not use it to skip checking, and do not use it as the metric when comparing two tiers or two prompts. Compare rows, fill and spot-checked accuracy instead.

​
Three results that all say completed

Asked for 40, got 38, fill 97%. Spot-check five, ship.

​
Before a batch: canaries

Pick ten inputs you already know the answers for, including two or three that should return nothing (a dead domain, a company acquired last year, a name that matches two firms). Run them first. All ten must come back right, and the empty ones must come back empty. On one 31-dealer pilot, five seeded dead domains were all flagged with citations; that result, not the grade, is what made the customer trust the other 26.

​
At scale: ground truth and a ledger

  • Ground truth: 50 to 100 rows compared against a source you trust, before the full job. Track filled cells and matches per field, not overall confidence.
  • Your own ledger: which input rows have come back, keyed on your identifier. The API does not report coverage across runs. Dedupe on the key after every merge; duplicate rows do occur.
  • Top-ups: re-run the misses with the completed rows passed as an exclusion list in input_data.

​
Iterate in this order

When a run falls short, change one thing at a time, in this order. Each step is cheaper and more often the cause than the next.
1

The task text

Is the entity disambiguated? Is there a row cap, a stop rule, an absence rule? Are the channels named? See Writing the Task.
2

The schema

Too wide? Undescribed fields? Required fields the web cannot fill? See Output Schema.
3

Goals and skill

Turn caveats into checkable goals. Turn the skill into a procedure with steps.
4

Sources

Tier the allow list, block the noise, say that discovery pages are not citations. See Controlling Sources.
5

Effort

Last. If rows and fill are still short after the four steps above, go up one tier and compare the same inputs. See Effort and Cost.

​
Symptom, cause, fix

​
Next Steps

Trust

How grades are computed

Running at Scale

Ledgers, batches and top-ups in code

Common Mistakes

Wrong and right, side by side