HarnessML: Feature Engineering

HarnessML: Feature Engineering

Use when creating and testing new features. Every feature is a hypothesis about the data. Treat it that way.

---

The Principle

A feature encodes an assumption: "this signal is predictive because of this mechanism." If you can't state the mechanism, you're adding noise and hoping the model sorts it out. Sometimes it does. Usually it doesn't.

---

The Feature-as-Hypothesis Cycle

1. Hypothesize

Start with a reason. Not "let me try a ratio of A/B" — but "A/B captures efficiency because absolute A is confounded by scale, and normalizing by B controls for that."

Sources of feature hypotheses:

• Domain knowledge (domain-research skill)
• EDA findings (eda skill)
• Diagnosis of model errors (diagnosis skill) — "the model fails on subgroup X, what feature captures X's behavior?"
• Feature interaction patterns — "A and B are both weak, but A*B would capture [specific phenomenon]"

2. Check for Redundancy

Before adding anything:

features(action="discover")

If an existing feature already captures the same signal (correlation >0.8 with your proposed feature), either skip yours or articulate what yours captures that the existing one doesn't.

3. Add the Feature

features(action="add", name="...", formula="...", type="...")

Feature types:

• instance: Computed per row from that row's columns
• grouped: Requires aggregation over a group (e.g., mean by category)
• interaction: Product or combination of two+ features
• ratio: Normalized comparison
• indicator: Binary flag for a condition
• regime: Flags different operating modes

4. Test in Isolation

One feature per experiment. If you batch-add five features, you learn nothing about which one helped.

experiments(action="create",
  description="Test [feature name]",
  hypothesis="[Feature] should improve [metric] because [mechanism]. Expected gain: [range]."
)
experiments(action="run", experiment_id="...")

5. Interpret the Result

Not just "did the metric go up?"

Result	What it means	What to do
Feature improves metric as predicted	Hypothesis confirmed. Domain reasoning is sound.	Look for related features capturing the same phenomenon from different angles.
Feature improves metric but for different reason than predicted	The signal exists but your explanation was wrong. Investigate the actual mechanism.	This often leads to better hypotheses.
Feature has no effect	Either redundant with existing features, or hypothesis is wrong.	Check correlation with existing features. If redundant, move on. If not, refine the hypothesis — maybe the functional form is wrong (continuous vs binned, linear vs interaction).
Feature hurts metric	It's adding noise that the model can't separate from signal.	The hypothesis may still be correct but the operationalization is wrong. Try a different formula or transformation.
Feature helps one model but not others	Signal exists but is model-dependent.	Trees might not need binned versions of continuous features. Linear models can't learn interactions natively. Match feature form to model type.

6. Log the Learning

The experiment conclusion should update your understanding of what drives the target. A feature that doesn't work teaches you something about the data — document what.

---

Feature Engineering Strategies

Interactions

When two features are individually weak but mechanically related:

features(action="add", name="leverage_x_rates", formula="leverage * rate_change", type="interaction")

Auto-search can help discover interactions you haven't thought of:

features(action="auto_search", features=[...], search_types=["interactions"])

But review auto-discovered interactions through a domain lens. If you can't explain why A*B should matter, it's probably a statistical artifact.

Temporal Features

For time-dependent data:

features(action="auto_search", features=[...], search_types=["lags", "rolling"])

Lag features capture momentum and reversion. Rolling features capture trends and volatility. But be careful about leakage — a lag of 0 periods is just the current value, and rolling windows must not include future data.

Grouped Aggregates

When row-level data needs context:

features(action="add", name="category_avg_price", formula="mean(price) group_by category", type="grouped")

These capture "how does this row compare to its group?" signals. But they require careful leakage protection — the aggregation must exclude the current row's target in supervised settings.

Binning

When a continuous feature has a non-linear relationship with the target:

features(action="add", name="age_group", formula="bin(age, edges=[18, 35, 50, 65])", type="instance")

Use domain knowledge for bin edges, not arbitrary quantiles. "Clinically meaningful age groups" is better than "quartiles."

---

Per-Model Feature Sets

Different models benefit from different features. This is a key source of ensemble diversity.

• Linear models: Need explicit interactions, bins, and polynomial terms because they can't learn non-linear relationships
• Tree models: Learn interactions natively but benefit from ratio features that make splits more efficient
• Neural networks: Can learn complex relationships but benefit from normalized inputs and meaningful encodings

Consider giving different feature subsets to different models:

models(action="update", name="lr_main", features=[...])  # linear-friendly features
models(action="update", name="xgb_main", features=[...])  # tree-friendly features

---

Anti-Patterns

• Batch-adding features: Adding 10 features at once makes it impossible to learn what works
• Features without hypotheses: "Let me try log(X)" without a reason is noise generation
• Ignoring negative results: A feature that doesn't work narrows the search space. Log it.
• Same features for every model: This reduces ensemble diversity. Model-specific feature sets are a powerful lever.
• Over-relying on auto-search: Automated discovery finds patterns. Domain reasoning finds signals. The distinction matters.