My LLM chat API taught me that '50+ models' is a maintenance tax, not a feature
I launched my OpenAI-compatible chat endpoint with 50+ models behind one integration. The marketing math was easy: one client, any provider, swap without rewriting code. The engineering math was harder. The first cost was the model registry. Every provider ships a slightly different /models payload — some include context windows, some don't; some use max_tokens , some use max_completion_tokens ; a few lie about which models are actually available in which regions. I ended up maintaining a normalization layer that's basically a translation table between five different vocabularies for "this model exists." The second cost was routing. I added auto/eco/premium tiers so agents could pick a price-quality tradeoff without knowing provider names. That meant keeping per-model latency and cost data fresh, or the router makes confident wrong decisions. A stale price table sends every "eco" request to a model that got more expensive last Tuesday. The third cost — ...