Direct answer
You must decide between two third-party LLMs for a knowledge assistant: a faster, cheaper model with slightly lower factual accuracy versus a slower, costlier model with better factuality, and possibly using both together. The right approach is to define a small set of measurable evaluation criteria tied to the actual product requirement, benchmark both models against real (or realistic) queries on those criteria, and then decide whether a single model suffices or whether a routing/fallback strategy combining both is worth the added complexity.
Structured elaboration
How to evaluate. Build a held-out evaluation set of realistic queries with known-correct answers (or human-graded rubrics for open-ended ones), and measure each candidate model on: factual accuracy (does it get the answer right, and does it hallucinate on out-of-scope questions), latency (p50/p95 response time under realistic load), cost per query at your expected volume, and any user-experience metrics that matter for the product (helpfulness ratings, task completion rate in A/B tests). Benchmarks alone are not sufficient; a model can score well on a public benchmark and still perform differently on your specific domain and query distribution, so the evaluation set must reflect your actual traffic.
Vendor/integration risk. Beyond raw model quality, evaluate SLA guarantees, rate limits, data-handling and privacy terms, and how exposed you are if the vendor changes pricing, deprecates the model, or has an outage, since a third-party dependency carries operational risk that a benchmark score alone won't capture.
Combining both models. A common production pattern is routing: use the fast, cheap model for the majority of queries (the ones it handles well), and escalate to the slower, more accurate model only for queries flagged as higher-risk or lower-confidence, e.g., via the fast model's own confidence signal, query complexity heuristics, or a lightweight classifier trained on where the fast model tends to fail. This captures most of the cost savings of the cheap model while limiting factual-accuracy risk to the harder subset of queries that actually need it.
Worked example
Say the cheap model costs $0.20 per 1,000 queries and the accurate model costs $2.00 per 1,000 queries, roughly a 10x cost difference, and evaluation shows the cheap model is factually correct 92% of the time versus 98% for the expensive model on your held-out set. If a routing classifier can reliably identify the roughly 20% of queries where the cheap model is most likely to be wrong (say, questions requiring precise numeric facts or recent information) and escalate only those to the expensive model, the blended cost is 0.8×$0.20+0.2×$2.00=$0.16+$0.40=$0.56 per 1,000 queries, about a 72% cost reduction from always using the expensive model, while capturing most of its accuracy benefit specifically where it matters most.
Trade-offs & pitfalls
A routing strategy only pays off if the signal for "this query needs the accurate model" is genuinely predictive; a poorly calibrated router either escalates too much (eroding the cost savings) or too little (letting factuality-sensitive queries slip through to the cheap model). It also adds real engineering and operational complexity: two vendor integrations, two SLAs to monitor, and a routing component that itself needs to be evaluated and maintained over time. For an early-stage product without the traffic volume or engineering capacity to build and maintain a router well, a single well-chosen model, even if slightly suboptimal on cost or accuracy, is often the more pragmatic starting point, with routing revisited once volume and evaluation infrastructure justify the added complexity.
