Guide
Is my LLM task too simple for GPT or Claude?
Self-check: seven questions
Each “yes” points towards a smaller model, or even plain rules.
- Does the same prompt run on many different inputs, hundreds of times a day or more?
- Is the answer a yes/no, one label from a fixed list, or a few fields copied out of the text?
- Is everything needed to answer already in the input, with no outside knowledge or tools?
- Could a trained person answer each one in under ten seconds?
- Do you throw away the model’s explanation and keep only the label or fields?
- Is a wrong answer cheap to catch or fix later (a human reviews it, or a later step checks it)?
- Are you paying for reasoning tokens on it? On our estimates they can be most of the bill at high effort: see what reasoning effort costs.
Signs it is not too simple
- The output is free text a customer reads.
- The model calls tools, looks things up or plans several steps.
- Inputs vary a lot in length, language or format from call to call.
- A wrong answer is costly and nothing downstream catches it.
Keep those calls on the large model. Moving them is where quality drops.
The 5-sample test
- Pick 5 real inputs: 3 typical and 2 you think are hard.
- Write down the answer your current model gave for each.
- Run the smaller model on the same task without showing it those answers.
- If it disagrees on 2 or more, stop: the task is not simple for that model. If it disagrees on 1, look at why before going further.
- If it matches all 5, test 25 inputs and measure agreement where it is confident.
The free cutover audit does steps 3 to 5 for you with Levanto Sage 2: it takes 5 to 25 samples with your current model’s answers and reports agreement, a confidence threshold and the cost difference.
What we saw on our own bench
We ran Levanto Sage 2 and a simple keyword rule on synthetic customer chats. Small sets, made by us, so read them as an illustration, not a benchmark:
| Question | Levanto Sage 2, clean (24) | Keyword rule, clean | Levanto Sage 2, messier (10) | Keyword rule, messier |
|---|---|---|---|---|
| Qualified sales lead? | 100% | 83.3% | 100% | 100% |
| Escalate to a human? | 100% | 100% | 100% | 70% |
| Scam or phishing? | 100% | 100% | 90% | 90% |
Two lessons. These questions were simple enough that a small model matched the labels on almost every example. And where the wording is predictable, rules can do the same for free, but they break when wording changes. Test both on your own inputs.
Next steps
How to switch to a smaller model · What a call costs, by model · Extraction cost by model · How the audit works