HarnessML: Diagnosis

HarnessML: Diagnosis

Use after every experiment run. This is the core data science skill — reading results, understanding errors, and forming the next hypothesis. If you skip this, you're not doing science, you're doing random search.

---

The Principle

A metric is a symptom. Diagnosis finds the cause. You don't treat symptoms — you understand the disease.

When you see "Brier improved by 0.004," that is not a conclusion. It's the beginning of a question: where did the improvement come from? Which predictions changed? Did calibration improve, or did discrimination improve? Is the gain robust across folds, or driven by one?

---

Reading the Results

After every run:

pipeline(action="diagnostics")
pipeline(action="compare_latest")

Aggregate Metrics

Start here, but don't stop here.

• Primary metric moved in expected direction? Good — now verify the mechanism matches your hypothesis.
• Primary metric didn't move? The change had no effect, or positive and negative effects canceled out. Check per-fold to see which.
• Secondary metrics moved unexpectedly? A Brier improvement with ECE degradation means you improved discrimination but hurt calibration — that's a specific diagnosis, not just "mixed results."

Per-Fold Breakdown

This is where the real signal is.

• Which folds improved? Do they share characteristics? (Smaller sample, specific time period, certain subgroups)
• Which folds degraded? Same questions.
• Is there a pattern? If the same folds always struggle, that's a systematic model weakness, not random noise.
• What's the fold variance? High variance across folds means the model is unstable — it's learning different things from different data slices.

What fold patterns tell you:

• Consistently bad fold → something about that fold's data is different (temporal shift, subgroup behavior, sample size)
• Improving the bad fold while degrading good ones → your change helps edge cases but hurts the base case (need more targeted approach)
• All folds improve uniformly → genuine structural improvement
• Random fold-to-fold variation → noise, not signal. The change probably doesn't matter.

Train-Test Divergence

• Large train-test gap → overfitting. The model memorizes training data. Solutions: regularization, simpler model, fewer features, more data per fold.
• Small gap but poor test performance → underfitting. The model is too simple. Solutions: more features, more complex model, less regularization.
• Gap changed after your experiment → your change affected model complexity. Did it move in the direction you predicted?

Calibration

pipeline(action="diagnostics")  # includes calibration curves

• ECE (Expected Calibration Error): Are predicted probabilities reliable? Low ECE means when the model says 70%, the event happens ~70% of the time.
• Brier vs ECE: Brier combines discrimination and calibration. You can have good Brier with poor ECE (discriminates well but probabilities are wrong) or good ECE with poor Brier (well-calibrated but doesn't separate classes).
• Calibration curve shape: S-shaped means the model is overconfident in both directions. Flat means the model doesn't use the full probability range.

Feature Importances

• What is the model relying on? If the top features aren't what domain knowledge predicts, investigate why.
• Did importances shift after your change? If you added a feature and it's not in the top importance list, either it's redundant with existing features or the model doesn't find it useful.
• Are importances concentrated? If one feature dominates, the model is fragile. If importances are spread, the model captures diverse signals.

Meta-Learner Coefficients

• Positive coefficient: Model contributes positively to the ensemble.
• Near-zero coefficient: Model adds nothing. The ensemble is ignoring it.
• Negative coefficient: Model actively hurts the ensemble. Its predictions are anti-correlated with the target conditional on other models. This doesn't mean the model is bad individually — it means it's providing signal the ensemble already has, but noisier.

Model Correlation Matrix

• High correlation (>0.95): Two models make nearly identical predictions. One is redundant.
• Moderate correlation (0.7-0.9): Models agree on easy cases, disagree on hard cases. This is good — it's where ensemble diversity helps.
• Low correlation (<0.7): Models see the problem very differently. High diversity, but check that both are individually competent.

---

Forming the Next Hypothesis

This is the purpose of diagnosis. Every diagnostic finding should generate a question:

Finding	Question it generates
Fold 3 always worst	What's different about fold 3's data? Is it a time period, a subgroup, a distribution shift?
Train-test gap on tree models only	Are trees overfitting to interactions that don't generalize? Would regularization or max_depth reduction help?
Feature X has high importance but low univariate correlation	The model found a non-linear or conditional relationship. What interaction or transformation makes this explicit?
Two models have 0.98 correlation	They're learning the same thing. Can I differentiate them with different feature sets? Or should I drop one and add a different model family?
ECE degraded while Brier improved	Discrimination improved but calibration suffered. Is the calibration method appropriate? Would a different calibrator help?
New feature has zero importance	Is it redundant with an existing feature? Or does the model need a different functional form (e.g., binned instead of continuous)?
All folds improved slightly	Genuine structural improvement, but small. Is there more headroom in this direction, or is this the ceiling?

Write down the next hypothesis before closing the diagnosis. If you finish diagnosing and don't know what to try next, you didn't dig deep enough.

---

Residual Analysis

When available, look at where the model's errors concentrate:

• Are errors uniform or clustered? Clustered errors in a specific region of feature space mean the model is missing a pattern in that region.
• Are there systematic over/under-predictions? A model that consistently over-predicts for a subgroup needs a feature that captures that subgroup's behavior.
• Do errors correlate with a feature not in the model? That feature should probably be added.

---

The Diagnosis Template

After every experiment, write:

## Diagnosis: [Experiment ID]

### What happened
[Metric changes — aggregate and per-fold]

### Why it happened
[Mechanism — not just "the metric went down" but WHY]

### What was confirmed
[Parts of the hypothesis that were supported]

### What was surprising
[Things you didn't predict — these are the most valuable]

### Next hypothesis
[What to investigate based on these findings]

Write key diagnostic findings to the notebook so they persist across sessions:

notebook(action="write", type="finding", content="[diagnosis insight]", experiment_id="...")

Not every diagnostic detail — just the insights that should inform future work.