AI Cost, Latency and UX
Optimize the time and money spent on a successful user outcome. · AI Product Management · Lesson 53 · 3 min
AI Cost, Latency and UX · 3 min
Situation
A cheap answer becomes an expensive workflow.
A model call costs little, but poor outputs trigger retries and manual correction. Another model costs more per call yet completes the task correctly more often.
Compare cost per successful task, including retrieval, tool calls, retries, infrastructure, and human effort. Token price alone can pick the wrong system.
Mental model
Users experience a timeline, not a model benchmark.
Measure time to first useful feedback, time to a usable result, and the distribution of long waits. Input size, output length, model choice, tool dependencies, and traffic all affect latency.
A fast first token does not necessarily mean the user can act sooner. A partially generated account decision may be useless until validated.
Example
Choose the optimization that fits the task.
A smaller model may handle a bounded classification. Streaming can make a readable draft feel responsive. Batching may suit overnight processing. Caching can avoid repeated work where reuse is valid.
Each choice changes the product contract. Batching delays results; output caching needs freshness and permission rules; routing requires reliable detection of difficult cases.
Failure case
Streaming the wrong thing too early.
If output must pass a safety or correctness check before the user relies on it, showing raw partial text can expose content that is later rejected.
Consider progress states, a validated final result, or a reversible draft. Do not promise a precise completion time unless the system can support that promise.
PM decision
Set a service budget.
Define an acceptable completion-time range, quality bar, and cost envelope for the use case. Test normal and long inputs, busy periods, provider errors, and fallbacks.
A fallback model should be evaluated on the same task. If a lower-quality fallback is unsuitable, a transparent delay or manual path may be the better response.
Remember this
Optimize the whole path to useful completion.
Quality, waiting, and cost interact. Make the trade-off explicit and verify that the optimization improves the outcome the user experiences.