HarnessML: Running Experiments

HarnessML: Running Experiments

Use when executing an experiment. Load experiment-design first to ensure you've thought through the hypothesis. This skill covers the mechanics and the discipline of execution.

---

Execution Flow

1. Create the Experiment

experiments(action="create",
  description="...",
  hypothesis="..."
)

The hypothesis is required. It should follow the structure from experiment-design: what you're changing, why, what you expect, and what it would mean.

For follow-up experiments within the same strategy, link to the parent:

experiments(action="create",
  description="...",
  hypothesis="...",
  parent_id="exp-001",
  branching_reason="Adjusting parameterization based on fold 3 diagnosis"
)

2. Write the Overlay

experiments(action="write_overlay",
  experiment_id="exp-001",
  overlay={"models.xgb_main.params.reg_lambda": 0.5}
)

Overlays modify config in isolation. Production config is never touched until you explicitly promote.

3. Run

experiments(action="run",
  experiment_id="exp-001",
  primary_metric="brier"
)

Or for quick iterations within a strategy:

experiments(action="quick_run",
  description="...",
  hypothesis="...",
  overlay={...}
)

4. Understand the Results

This is the most important step. Do not skip it.

Load the diagnosis skill. Read the results with intention:

• Did the folds you predicted would improve actually improve?
• Did anything unexpected happen?
• Does the pattern of results match your hypothesis, or tell a different story?

pipeline(action="diagnostics")
pipeline(action="compare_latest")

Do not move on until you can explain what happened and why.

5. Log What You Learned

experiments(action="log_result",
  experiment_id="exp-001",
  conclusion="...",
  verdict="...",
  metrics={...},
  baseline_metrics={...}
)

The conclusion is about what you learned, not just what the numbers say. A good conclusion:

• Confirms or refutes the hypothesis with evidence
• Explains the mechanism (why it worked or didn't)
• Identifies what to investigate next

A bad conclusion restates the metrics.

Also record the learning in the project notebook:

notebook(action="write", type="finding", content="...", experiment_id="exp-001")

The experiment journal captures the structured result. The notebook captures the insight — what this means for the bigger picture. If your theory of the target changed, write a theory entry too.

6. Decide What's Next

Every experiment ends with one of:

• The hypothesis was confirmed → What follow-up does this suggest? Is there a related question?
• The hypothesis was partially confirmed → What part was right? What part was wrong? What would disambiguate?
• The hypothesis was refuted → Why? Is it the parameterization (try again differently) or the strategy (move on)?
• The results were surprising → This is the most interesting outcome. Investigate before moving on.

---

Iterating Within a Strategy

When pursuing a strategy (e.g., "add regularization", "try neural network", "engineer temporal features"):

• Try 3-5 meaningfully different configurations before concluding the strategy doesn't work
• Meaningfully different: changing learning_rate from 0.01 to 0.011 is not meaningful. Changing from 0.01 to 0.05 while adjusting n_estimators is.
• Each attempt should be informed by diagnosis of the previous attempt
• Only conclude a strategy is flawed when you can explain the structural reason, not just that individual configs didn't work

The experiment journal should show a trail of attempts with diagnosis between each one, not isolated experiments.

---

Promoting Changes

When an experiment improves the model AND you understand why:

experiments(action="promote", experiment_id="exp-001", primary_metric="brier")

Promote when:

• The improvement is consistent across folds (not driven by one fold)
• You can explain the mechanism
• Secondary metrics didn't degrade in unexpected ways

Do not promote just because the primary metric improved. An unexplained improvement is a time bomb.

---

Verdicts

Verdict	Meaning
keep	Improvement is understood and consistent. Promote.
partial	Directional improvement or improvement on a subset. Worth logging for future combination.
revert	After exhaustive attempts, the strategy is structurally flawed. Document why.
inconclusive	Ambiguous results. May revisit with more information.

Verdicts are consequences of understanding, not goals. You don't run experiments to reach a verdict — you run them to learn. The verdict follows naturally from what you learned.