Guide
How to switch from GPT or Claude to a smaller model
1. Pick the right calls
Look for the call that runs most often with the shortest answer. Good candidates:
- Yes/no decisions: is this a sales lead, is this ticket urgent, is this message spam.
- Labels from a fixed list: intent, category, sentiment, language.
- Field extraction: dates, amounts, names, IDs copied out of a document.
Leave open-ended writing, tool use and multi-step reasoning on the large model. A smaller model is most likely to match on narrow, repetitive questions.
2. Work out what the call costs now
Count input tokens (prompt plus message), visible output tokens and reasoning tokens. Reasoning tokens are billed at the output price and are often the largest part of a short call. The savings estimates by model show typical costs per 1,000 calls at list prices checked 2026-11-03; the routing calculator takes your own numbers.
3. Measure agreement on your own inputs
Take 25 real inputs with the answers your current model gave and run the smaller model on the same task, without showing it those answers. Agreement is the share where both give the same answer. Public benchmarks will not tell you this: what matters is your prompt, your data and your labels.
4. Route by confidence, not all-or-nothing
A model that returns a probability lets you keep its answer only when it is sure and send the rest to your current model. That keeps quality on the hard cases and still moves most of the volume. How to choose the confidence threshold.
5. Check the saving clears the bar
The hybrid pays for the small model on every call and for your current model on the share it does not keep. If that is not at least about 30% cheaper than today, the change is rarely worth the engineering and monitoring time.
6. Roll out behind a shadow test
- Run both models on live traffic for a few days and log disagreements, but serve the current model’s answer.
- Switch the confident share over, keep logging a sample of both, and alert if agreement drops.
- Re-check when you change the prompt, the input source or either model version.
Do steps 2 to 5 in one go
The free cutover audit runs your 10 to 25 samples on Levanto Sage 2, reports agreement, picks the threshold that holds your target and projects the monthly cost. Read the method.