Ten models in two weeks is an architecture warning
By Allan Leone on
The trackers counted roughly ten new model releases from six providers in the first half of this month. If your product has one model's name written into it in more than one place, that pace is a problem you already have.
The release trackers logged about ten new models from six providers in the first two weeks of August. Whatever the precise count, the cadence is now weekly and shows no sign of settling. Every one of those releases is someone's argument for switching.
The interesting question is not which one is best this week. It is what switching costs you, because at this pace you will do it repeatedly, and the answer is usually written into your codebase long before anyone asks.
Where the coupling actually hides
Nobody hardcodes a model name in a hundred places. The lock-in is subtler and lives in four spots.
- Prompts tuned to one model's quirks, where the phrasing is load-bearing and nobody remembers why.
- Output parsing that depends on a particular formatting habit rather than an enforced schema.
- Latency assumptions baked into the interface, so a slower model breaks the animation rather than the feature.
- Cost assumptions baked into the pricing, which is the expensive one to unwind.
Only the first is obvious. The other three are discovered during the swap, which is the worst time.
The cheap insurance
Force structured output at the boundary and validate it. If the contract is a schema rather than a formatting convention, a model that phrases things differently is not a rewrite.
Then design the interface for the slow case. If your loading state only works at eight hundred milliseconds, you have made model choice a design constraint, and you will find that out under deadline.
Do not chase the benchmark
Open models now match proprietary ones on plenty of benchmarks, and benchmark position turns over faster than any team can re-architect. Chasing it is a treadmill with no exit.
The stable position is to be indifferent. Pick on cost, latency and behaviour for your actual task, keep the swap cheap, and re-run your own evaluation when something changes. Your evaluation on your data is the only benchmark that has ever predicted anything about your product.
A model is a dependency. Treat it like one, and the release schedule stops being news.
Tags: ai, product, engineering