Testing agents 101: offline, shadow, canary, CI
ReasonRank Engineering · 2026-07-17
You do not need every evaluation mode on day one. Most teams overcomplicate agent testing and never ship a gated model switch. Use a ladder: prove quality offline, confirm on captured traffic, then move live traffic carefully — and keep CI watching so the win does not rot.
The ladder
- Offline evaluation — shared suite × models (start here)
- Shadow test — replay captured production inputs with no prod impact
- Canary rollout — shift a percentage of live traffic with rollback
- CI quality gate — fail PRs when the suite regresses
Offline evaluation
This is the cheapest honest comparison. Pick an agent with a real suite, select the production model plus cheaper candidates, optionally add an LLM judge and repetitions for stability, and read quality against cost.
Prefer the candidate that clears your quality bar and minimizes cost per success — not the one that is merely cheapest per call. Then project monthly dollars against volume under Savings.
If the model checkboxes are empty, you have not registered model configs yet. That step comes before any test — see Creating a good agent.
Shadow test
Once you have a production baseline and a different candidate, replay recently captured inputs against the candidate offline. You get a dollarized verdict on your traffic shape without changing the gateway.
Requirements people miss:
- Production model set on the agent
- A candidate that is not the same config
- Some captured inputs (ingestion / sample capture)
No samples → nothing to shadow. Fix ingestion first.
Canary rollout
After an offline or shadow win you are willing to defend, shift a slice of live traffic (often ~10%) to the candidate and watch quality. Auto-rollback exists so a regression does not become a weekend incident.
Do not canary from vibes. Canary from a gated comparison.
CI quality gate
Wire the suite into pull requests so a prompt tweak or default-model change cannot quietly tank quality while “saving money.” The gate is the habit that keeps verified savings from evaporating next quarter.
Minimum setup before any test
| Need | Where | | --- | --- | | Agent + cases | Agents | | ≥2 model configs | Model Configs | | Keys or harness | Settings → Providers / BYO harness | | Production model | Agent → Production & savings |
Habit
Providers ship new cheap models constantly. Re-run the suite. Verified savings only stick if you keep measuring.
In the product: Testing agents 101, Runs & results, and CI quality gate.