Model Drift and Production Monitoring

What is it?

A model is trained on a snapshot of the world. As customers, language and prices change, the snapshot gets stale and accuracy slips. Drift monitoring is the smoke alarm that tells you when that is happening.

Also known as: data drift, concept drift, model decay, model degradation, monitoring ML models, PSI population stability index, production monitoring, training serving skew.

Technical definition

Types: covariate/data drift (P(x) changes), label/prior drift, and concept drift (P(y|x) changes) (Gama et al., 2014). Detection uses distribution tests (PSI, KS, Jensen–Shannon), embedding-distance monitors, performance on delayed labels, and proxy signals (confidence, refusal rate, output length). For LLM apps, provider model updates, prompt/template changes and retrieval-corpus changes also cause drift (LLM Observability (Tracing, Logging and Monitoring)). Responses include retraining, re-prompting, re-indexing, threshold changes or rollback.

Entity type: concept · Freshness class: evolving.

When should I use it?

Needed for any model in production beyond a short pilot.

Prerequisites and related concepts

Understand these first: Supervised Learning, MLOps (Machine Learning Operations).

Related concepts: LLM Observability (Tracing, Logging and Monitoring), How to Evaluate AI Systems and Outputs, Overfitting and Regularization, MLflow, What is RAG (Retrieval-Augmented Generation)?.

Implementation patterns

• Baseline distributions at deployment; compare windows.
• Monitor delayed ground truth when available.
• Alert on thresholds with human review.
• Shadow-test candidate models on live traffic.

Technologies and frameworks

Commonly associated: Evidently, WhyLabs-style monitors, MLflow, Prometheus/Grafana, custom statistical tests. Names are examples of what exists, not recommendations, and projects change quickly.

Practical guide

1. Define key metrics.
2. Store baselines.
3. Schedule comparisons.
4. Link alerts to runbooks.
5. Pin model versions where possible.

Common failure modes

• Monitoring only latency and errors.
• Alerting on noise.
• No ground-truth feedback loop.
• Silent provider model changes.

Troubleshooting

• Accuracy dropping with no code changes — Check input distribution shifts, upstream data changes and provider model versions.
• Too many false alerts — Tune thresholds; require persistence over windows.

Contrasted with

• LLM Observability (Tracing, Logging and Monitoring) — Observability traces single requests; drift monitoring watches distributions and trends.

Questions this page answers

• What is model drift?
• What is concept drift?
• How do I monitor ML models in production?
• Why did my model get worse over time?
• What is training-serving skew?

Typical goals: detect when a deployed model gets worse; plan retraining triggers.

Sources and verification

Status: not independently verified. This entry was authored on 2026-10-06 from general technical knowledge, without live checking of the sources below. Treat every source as a lead to confirm (identifier, title, authorship, current version) before publication or citation.

Freshness class: evolving — Established techniques whose best practice shifts as research and tooling move. Re-review every 6 months.

Canonical sources (unverified leads):

• Gama et al., A survey on concept drift adaptation, ACM Computing Surveys (2014) — locator to be identified
• Sculley et al., Hidden Technical Debt in Machine Learning Systems (2015) — locator to be identified

Related concepts