LLM Cutover

Guide

How to choose an LLM routing confidence threshold

What the threshold does

The cheap model answers every call and says how sure it is. If its confidence is at or above the threshold, you keep its answer. If not, the call also goes to your current model and you use that answer. The threshold trades two things:

  • Coverage: the share of calls the cheap model keeps. Lower threshold, more coverage, bigger saving.
  • Agreement on kept calls: how often those kept answers match your current model. Higher threshold, usually higher agreement.

Pick it in four steps

  1. Set the agreement you need on kept calls. The audit defaults to 98%; a low-stakes tag might accept less, an action that sends an email might need more.
  2. Collect labelled samples: real inputs plus the answers your current model gave. 25 is a starting point, a few hundred is better before rollout.
  3. For each cut-off (0.50, 0.60, 0.70, 0.80, 0.90, 0.95, 0.99), compute coverage and agreement on the kept share.
  4. Choose the lowest cut-off whose agreement meets the target. If none does, the cheap model is not ready for this task.

Reading the result

  • Coverage stays high as the threshold rises: the model is decisive on this task, a good sign for routing.
  • Agreement is below target even at 0.99: the model is confidently different from your current model. Check whether your labels or the prompt are the problem before giving up.
  • Few samples near the cut-off: the choice is fragile. Add samples before you trust it.

In our own small benchmark on lead qualification (24 samples), every Levanto Sage 2 answer had confidence below 0.2 or above 0.8, so any cut-off between them kept the same calls. Your task may behave differently.

Then check the money

The hybrid pays for the cheap model on every call and for your current model on the share it does not keep. The savings estimates by model show the coverage needed for a 30% saving at list prices; the calculator takes your own volume.

Let the audit do the sweep

The free cutover audit runs Levanto Sage 2 on your 10 to 25 samples, tests each cut-off above and recommends the one that holds your target, with the projected cost. Method · Switching to a smaller model

Run the free audit

Questions

What is a good default confidence threshold?

There isn’t one that holds across tasks. Test every cut-off on your own labelled samples and keep the lowest one that meets your agreement target.

Why not just use the highest threshold?

A higher threshold keeps fewer calls on the cheap model, so you save less. Above the point where agreement already meets your target, raising it only moves cost back to the expensive model.

Does this work with models that do not return probabilities?

Not well. Asking a chat model to rate its own confidence in text is a weak signal. Routing works best with a model that returns a calibrated probability, or with token log-probabilities where your provider exposes them.

How often should I re-check the threshold?

Whenever the prompt, the input mix or either model changes, and on a schedule otherwise. Log a sample of calls through both models and alert if agreement on the kept share drops below target.