Choosing a Model
Select for the task and its failure cost, not the leaderboard. · AI Product Management · Lesson 43 · 3 min
Choosing a Model · 3 min
Situation
The largest model wins the demo and loses the workflow.
A model writes excellent support drafts but takes too long for an agent handling live chat. A faster model is slightly weaker on rare complex cases but handles routine requests well.
The best product choice depends on the full workload, including latency, cost, and the consequences of mistakes.
Mental model
Compare models on your actual task.
Evaluate quality, latency distribution, cost per successful task, context needs, reliability, supported input types, and tool capabilities. Include operational constraints such as rate limits and availability.
Public benchmarks can help form a shortlist. They do not establish which model meets your own rubric on your users' inputs.
Example
A task-specific comparison can change the ranking.
Test candidate models on routine requests, ambiguous questions, missing information, and difficult exceptions. Evaluate factual correctness and policy adherence separately from style.
Include the whole system: prompts, retrieval, validation, retries, and human review. A cheaper call is not cheaper if it creates frequent retries or expensive corrections.
Trade-off
Routing can help, but it adds another decision.
A smaller model might handle a bounded classification task while a stronger model handles difficult cases. That only works if the routing rule reliably detects the cases needing help.
Evaluate the router and the fallback path too. A model's self-reported confidence is not automatically calibrated enough to decide escalation.
Failure case
Changing models without regression checks.
A new model may improve average quality while mishandling an important category or changing output behavior. Re-run the task suite and inspect failures before a rollout.
Keep a record of the model configuration and the system version so regressions can be investigated. Plan for provider changes without assuming every model is interchangeable.
Remember this
Choose the smallest total system that meets the bar.
The quality threshold, response-time promise, cost, and fallback behavior should determine the model decision. Revisit it when the task distribution or available evidence changes.