Method
How the audit decides
The question is narrow on purpose: if you route this task to a cheaper model, how often do the answers change, and what does it save?
1. Agreement, not benchmark scores
You paste examples your current model already answered. The candidate model answers the same task, blind. Agreement is the share of samples where both give the same answer. For extraction, values are compared by type: money within 0.5% and the same currency, dates by day, text ignoring case and punctuation, and an empty field only matches an empty field.
2. Confidence routing
The candidate returns a probability for each yes/no answer. For each threshold from 0.50 to 0.99 the audit keeps the candidate’s answer when its confidence is at or above the threshold and sends the rest to your current model. It recommends the threshold that keeps the most calls while agreement on those calls stays at or above your target (98% by default).
3. Cost projection
- Your current cost per call: your sample length plus the prompt overhead you enter, times the list price of your model, plus its output tokens.
- The candidate’s cost per call: the tokens it actually billed in this run, at its list price ($0.05 per million input, $1 per million output).
- Hybrid: the candidate on every call, plus your current model on the share it doesn’t keep.
- “Worth it” means the hybrid bill is at least 30% lower. Below that, engineering time usually costs more than the saving.
4. Limits
- 25 samples is a screen, not a sign-off: one disagreement moves agreement by 4 points.
- Your current model’s answers are treated as the reference. If they are wrong, agreement measures copying its mistakes.
- Token counts for your current model are estimated (about 4 characters per token).
- List prices change; the audit shows the date they were checked. Batch and cached-input discounts aren’t included.
5. Data
Samples are sent to the candidate model’s API for this run only and aren’t stored by this site. The report stays in your browser unless you download it. Don’t paste personal, health or financial data.