Evaluation & MLOps
Production Monitoring and Drift Detection
How to catch silent regressions in deployed LLMs by monitoring input drift, output quality, and per-user randomised experiments before users tell you something is broken.
intermediate · 9 min read · Premium
The model passed every offline eval. It shipped. Three weeks later, support tickets spike, and you discover a quiet quality regression that started the day a new prompt template rolled out. This is the "silent regression" failure mode - the model never errors, never refuses, just degrades. Production monitoring exists to make the silent loud.
Three things drift
| What drifts | Symptom | Detection |
|---|---|---|
| Input distribution (covariate shift) | New user segment, new product nouns, new languages, longer or shorter prompts | PSI, KL divergence over input embeddings; rolling token-length and topic distributions |
| Output distribution | Response length, refusal rate, format adherence, sentiment shifts | Rolling histograms; rule-based format validators; LLM-judge quality score over a sample |
| Concept (the input-output mapping itself) | The right answer for a given input has changed - product policy, prices, dates | Hardest to detect; usually surfaces as user complaints or eval-set degradation |
Covariate shift means the inputs changed but the correct mapping is the same. Concept drift means the correct mapping itself changed. Both are real; the response is different. Covariate shift may mean you need to retrain or extend coverage; concept drift means stale knowledge and likely a RAG or fine-tuning update.
Input drift metrics that earn their keep
- Population Stability Index (PSI). Bin a feature distribution (token length, topic cluster id) at training time and at serving time, then compute PSI between them. Rule of thumb: under 0.1 is stable, 0.1-0.25 is moderate drift, over 0.25 is significant.
- KL divergence over embedding clusters. Cluster training inputs into 50-200 buckets in embedding space, then track the KL divergence between the training bucket distribution and the rolling production bucket distribution. Catches semantic drift PSI on raw features misses.
- New-token rate. Fraction of input tokens that did not appear in training. Spikes signal a new product, new event, or new language users are bringing in.
Output monitoring
- Rule-based validators. For structured outputs (JSON, SQL, function calls), parse rate and schema-validity rate are the cheapest leading indicators. A 2% drop in JSON parse rate is your earliest warning.
- Quality sampling. Sample 1-5% of production responses, score with an LLM judge using your custom rubric, alert on rolling-window quality drops.
- Response-shape metrics. Mean output length, refusal rate, language distribution, latency percentiles. Each one drifting independently flags a different upstream cause.
Shadow deployment and canary rollout
Before any model swap touches user traffic:
- Shadow. New model receives a copy of production traffic, responses are logged but discarded. You compare distributions and run offline quality scoring against the live baseline.
- Canary. New model serves 1-5% of real traffic. Watch error rates, latency, and quality metrics on the canary slice vs the baseline slice.
- Gradual rollout. Step the new model from 5% to 25% to 50% to 100% over hours or days, with automatic rollback wired to alert thresholds.
Keep reading with Pro.
You're reading the preview. Unlock the full concept plus the library, study plans, the AI mentor, and daily emails.