model-monitoring

Model Monitoring

When to Use

When a model is live in production and needs ongoing health tracking to catch degradation before users notice.

Core Jobs

1. Monitor Data Drift

Input distribution changes → model becomes unreliable.

• Univariate drift: monitor each feature's distribution (KS test, PSI)
• Multivariate drift: monitor joint distribution (MMD, PCA shift)
• Tools: Evidently AI, WhyLogs, Alibi Detect
• Threshold: PSI > 0.2 = alert, > 0.25 = investigate retraining

2. Monitor Prediction Drift

• Distribution of model outputs changing
• Useful when ground truth is delayed (common in production)
• Alert if output distribution shifts significantly from baseline

3. Monitor Ground Truth (when available)

• Compare predictions vs actuals as labels arrive
• Calculate: accuracy, precision, recall (same metrics as eval)
• Set alert thresholds: >10% degradation from baseline = page on-call

4. Set Retraining Triggers

Define trigger strategy:

• Schedule: retrain weekly/monthly regardless of drift
• Performance-based: retrain when accuracy drops below threshold
• Drift-based: retrain when PSI > 0.25 on key features
• Combine: schedule + drift detection for critical models

5. Dashboard Essentials

• Prediction volume over time
• Feature distributions (current vs baseline)
• Model performance metrics (rolling 7-day)
• Latency P50/P95/P99
• Error rates (failed inferences)

Key Outputs

• Monitoring pipeline (drift detection + alerting)
• Model health dashboard
• Retraining trigger policy
• Runbook for drift alerts

Anti-Patterns

• No monitoring after deployment ("set and forget")
• Alerting on every metric — pick 3–5 critical signals
• No ground truth pipeline — can't measure real accuracy
• Retraining without validating the new model first