Model Fleet Governance
Model fleet governance is the operating discipline for managing multiple LLM providers and model versions in production, including routing, evaluation, deprecation, compliance, cost, latency, and regression control.
Key points
- Datadog finds that organizations are increasingly multi-provider: OpenAI still has the largest share, but Google Gemini and Anthropic Claude gained sharply over the prior year [src-037].
- More than 70 percent of organizations in the Datadog telemetry sample use three or more models, and the share using more than six models nearly doubled [src-037].
- Teams are building model portfolios so each workload can use the model that fits its quality, latency, cost, operational-risk, and task requirements [src-037].
- Direct API calls scattered across services create platform engineering, developer-experience, and compliance problems, so Datadog recommends modular routing through a gateway service or managed router such as OpenRouter [src-037].
- Model churn compounds technical debt: teams adopt new releases quickly but retire old production defaults more slowly, leaving overlapping quality, latency, and cost profiles in flight [src-037].
- Governance requires continuous evaluation, benchmarking, routing, regression monitoring, and planned deprecation as providers sunset older models [src-037].
- The EU AI Act adds a legal model-governance layer for general-purpose AI models: technical documentation, downstream-provider information, copyright compliance, training-content summaries, and cooperation with authorities [src-085].
- For systemic-risk GPAI models, model-fleet governance must also track model evaluation, adversarial testing, systemic-risk mitigation, serious-incident reporting, cybersecurity, and potential Commission enforcement [src-085].
Related entities
Related concepts
- Agent Experimentation
- AI Product Experimentation
- Offline Evals to Online Experiments
- Practitioner Model Benchmarking Methodology
- LLM Observability
- General-Purpose AI Model Governance