feature-engineering

Feature Engineering

When to Use

When preparing data for model training — selecting which signals to use, how to encode them, and what new features to derive.

Core Jobs

1. Understand the Data

Before engineering:

• Distribution of each feature (histogram, describe())
• Missing value rates
• Correlation with target (mutual information, Pearson)
• Correlation between features (avoid multicollinearity for linear models)

2. Handle Raw Features

Numeric:

• Scale: StandardScaler (normal dist), MinMaxScaler (bounded), RobustScaler (outliers)
• Transform: log(x+1) for skewed distributions

Categorical:

• Low cardinality (< 15 values): one-hot encoding
• High cardinality (> 50 values): target encoding, embedding
• Ordinal: label encode with meaningful order

Text:

• TF-IDF for sparse models
• Embeddings (sentence-transformers) for semantic similarity

Temporal:

• Extract: hour, day of week, month, is_weekend
• Cyclical: sin/cos encoding for hour/day
• Lag features: value at T-1, T-7, T-30

3. Create Derived Features

• Ratios: clicks/impressions, revenue/user
• Differences: current_price - avg_price
• Aggregations: user's avg purchase in last 30 days
• Interaction features: feature_A × feature_B (for linear models)

4. Feature Selection

• Remove zero-variance features
• Remove features with > 50% missing
• Use permutation importance or SHAP for post-hoc selection
• Regularization (L1/Lasso) for automatic selection

Key Outputs

• Feature engineering pipeline (sklearn Pipeline or similar)
• Feature importance report
• Missing value and encoding strategy doc

Anti-Patterns

• Feature leakage (using future data to predict past)
• Fitting scalers on the full dataset (fit on train only)
• Creating 200 features and not selecting — leads to overfitting
• Forgetting to apply same transformations at inference time