shap-model-explainability

SHAP Model Explainability

Overview

SHAP (SHapley Additive exPlanations) is a unified framework for explaining machine learning model predictions using Shapley values from cooperative game theory. It quantifies each feature's contribution to individual predictions and provides both local (per-instance) and global (dataset-level) explanations with theoretical guarantees of consistency and additivity.

When to Use

• Explaining which features drive a model's predictions (global importance)
• Understanding why a model made a specific prediction (local explanation)
• Debugging model behavior and identifying data leakage
• Analyzing model fairness across demographic groups
• Comparing feature importance across multiple models
• Generating interpretable model explanations for stakeholders
• For tree-based model interpretation, prefer SHAP over permutation importance or Gini importance (more accurate, instance-level)
• For deep learning interpretation on images, consider GradCAM; use SHAP for tabular/structured data

Prerequisites

pip install shap matplotlib
# Optional: xgboost lightgbm tensorflow torch (depending on model)

Quick Start

import shap
import xgboost as xgb
from sklearn.model_selection import train_test_split

# Load example data
X, y = shap.datasets.adult()
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)

# Train model
model = xgb.XGBClassifier(n_estimators=100).fit(X_train, y_train)

# Explain: select explainer → compute → visualize
explainer = shap.TreeExplainer(model)
shap_values = explainer(X_test)

shap.plots.beeswarm(shap_values)   # Global importance
shap.plots.waterfall(shap_values[0])  # Single prediction
print(f"Base value: {shap_values.base_values[0]:.3f}")
print(f"SHAP values shape: {shap_values.values.shape}")  # (n_samples, n_features)

Workflow

Step 1: Select the Right Explainer

Choose based on model type:

Model Type	Explainer	Speed	Exactness
Tree-based (XGBoost, LightGBM, RF, CatBoost)	TreeExplainer	Fast	Exact
Linear (LogReg, GLM, Ridge)	LinearExplainer	Instant	Exact
Deep learning (TensorFlow, PyTorch)	DeepExplainer	Fast	Approximate
Deep learning (gradient-based)	GradientExplainer	Fast	Approximate
Any model (black-box)	KernelExplainer	Slow	Approximate
Any model (permutation-based)	PermutationExplainer	Very slow	Exact
Unsure?	shap.Explainer	Auto	Auto

# Tree-based models (most common)
explainer = shap.TreeExplainer(model)

# Linear models
explainer = shap.LinearExplainer(model, X_train)

# Deep learning
explainer = shap.DeepExplainer(model, X_train[:100])

# Any model (model-agnostic, slower)
explainer = shap.KernelExplainer(model.predict, shap.kmeans(X_train, 50))

# Auto-select
explainer = shap.Explainer(model, X_train)

Step 2: Compute SHAP Values

shap_values = explainer(X_test)

# shap_values object contains:
# .values      — SHAP values array (n_samples, n_features)
# .base_values — Expected model output (baseline)
# .data        — Original feature values

# Verify additivity: prediction = base_value + sum(SHAP values)
print(f"  {shap_values.base_values[0]:.3f} + {shap_values.values[0].sum():.3f} = "
      f"{shap_values.base_values[0] + shap_values.values[0].sum():.3f}")

Step 3: Global Explanations

# Beeswarm: feature importance + value distributions (most informative)
shap.plots.beeswarm(shap_values, max_display=15)

# Bar: clean mean |SHAP| importance
shap.plots.bar(shap_values)

Step 4: Local Explanations (Individual Predictions)

# Waterfall: detailed breakdown of one prediction
shap.plots.waterfall(shap_values[0])

# Force: additive force visualization
shap.plots.force(shap_values[0])

Step 5: Feature Relationships

# Scatter: how a feature affects predictions
shap.plots.scatter(shap_values[:, "Age"])

# Colored by interaction feature
shap.plots.scatter(shap_values[:, "Age"], color=shap_values[:, "Education-Num"])

Step 6: Advanced Visualizations

# Heatmap: multi-sample SHAP grid
shap.plots.heatmap(shap_values[:100])

# Decision plot: cumulative SHAP paths
shap.plots.decision(shap_values.base_values[0], shap_values.values[:10],
                     feature_names=X_test.columns.tolist())

# Cohort comparison
import numpy as np
mask_a = X_test["Age"] < 40
shap.plots.bar({
    "Under 40": shap_values[mask_a],
    "40+": shap_values[~mask_a]
})

Key Parameters

Parameter	Explainer/Function	Default	Effect
feature_perturbation	TreeExplainer	"tree_path_dependent"	"interventional" for causal interpretation (requires background data)
model_output	TreeExplainer	"raw"	"probability" to explain probabilities instead of log-odds
data (background)	KernelExplainer, DeepExplainer	Required	100-1000 representative samples; use shap.kmeans(X, 50) for efficiency
nsamples	KernelExplainer	"auto"	Higher = more accurate but slower; minimum 2×features
max_display	All plot functions	10	Number of features shown in plots
alpha	scatter/beeswarm	1.0	Point transparency for dense datasets
show	All plot functions	True	Set False to get matplotlib figure for saving
clustering	beeswarm	None	shap.utils.hclust(...) to cluster correlated features

Key Concepts

SHAP Value Properties

SHAP values have three theoretical guarantees (unique among explanation methods):

• Additivity: prediction = base_value + sum(SHAP values) — exact decomposition
• Consistency: If a feature becomes more important in the model, its SHAP value increases
• Missingness: Features not present receive zero attribution

Interpretation: Positive SHAP → pushes prediction higher; Negative → lower; Magnitude → strength of impact.

Model Output Types

Understand what your model outputs — SHAP explains the output space:

• Regression: SHAP values in target units (e.g., dollars, temperature)
• Classification (log-odds): Default for tree classifiers. Use model_output="probability" for probability explanations
• Classification (probability): SHAP values sum to probability deviation from baseline

SHAP vs Other Methods

Method	Local	Global	Consistent	Model-agnostic
SHAP	Yes	Yes	Yes	Yes
Permutation importance	No	Yes	No	Yes
Gini/split importance	No	Yes	No	Trees only
LIME	Yes	No	No	Yes
Integrated Gradients	Yes	No	Partial	NN only

Interaction Values (TreeExplainer only)

shap_interaction = explainer.shap_interaction_values(X_test)
# Shape: (n_samples, n_features, n_features)
# Diagonal = main effects; off-diagonal = pairwise interactions

Background Data Selection

Background data establishes the baseline (expected model output). Selection affects SHAP magnitudes but not relative importance.

• Random sample from training data: 100-500 samples
• Use shap.kmeans(X_train, 50) for efficient summarization
• For TreeExplainer with tree_path_dependent: no background data needed (uses tree structure)
• For DeepExplainer/KernelExplainer: 100-1000 samples balance accuracy vs speed

Common Recipes

Recipe: Model Debugging

import numpy as np

# Find misclassified samples
predictions = model.predict(X_test)
errors = predictions != y_test
error_indices = np.where(errors)[0]

# Explain errors
for idx in error_indices[:3]:
    print(f"Sample {idx}: predicted={predictions[idx]}, actual={y_test.iloc[idx]}")
    shap.plots.waterfall(shap_values[idx])

# Check for data leakage: unexpected high-importance features
mean_abs_shap = np.abs(shap_values.values).mean(0)
top_features = X_test.columns[mean_abs_shap.argsort()[-5:]]
print(f"Top features (check for leakage): {list(top_features)}")

Recipe: Fairness Analysis

# Compare SHAP distributions across groups
group_a = shap_values[X_test["Sex"] == 0]
group_b = shap_values[X_test["Sex"] == 1]

shap.plots.bar({"Female": group_a, "Male": group_b})

# Check protected attribute importance
sex_importance = np.abs(shap_values[:, "Sex"].values).mean()
total_importance = np.abs(shap_values.values).mean()
print(f"Sex contribution: {sex_importance/total_importance:.1%} of total importance")

Recipe: Production Caching

import joblib

# Save explainer for reuse
joblib.dump(explainer, 'explainer.pkl')
explainer = joblib.load('explainer.pkl')

# Batch computation for API responses
def explain_batch(X_batch, explainer, top_n=5):
    sv = explainer(X_batch)
    results = []
    for i in range(len(X_batch)):
        top_idx = np.abs(sv.values[i]).argsort()[-top_n:]
        results.append({
            'prediction': sv.base_values[i] + sv.values[i].sum(),
            'top_features': {X_batch.columns[j]: sv.values[i][j] for j in top_idx}
        })
    return results

Recipe: MLflow Integration

import mlflow
import matplotlib.pyplot as plt

with mlflow.start_run():
    model = xgb.XGBClassifier().fit(X_train, y_train)
    explainer = shap.TreeExplainer(model)
    shap_values = explainer(X_test)

    shap.plots.beeswarm(shap_values, show=False)
    mlflow.log_figure(plt.gcf(), "shap_beeswarm.png")
    plt.close()

    for feat, imp in zip(X_test.columns, np.abs(shap_values.values).mean(0)):
        mlflow.log_metric(f"shap_{feat}", imp)

Expected Outputs

Output	Type	Description
shap_values	shap.Explanation	Object with .values (n_samples, n_features), .base_values (baseline), .data (input features)
Waterfall plot	matplotlib figure	Single-instance explanation showing feature contributions from base value to prediction
Beeswarm plot	matplotlib figure	Global summary: feature importance × direction for all samples
Bar plot	matplotlib figure	Mean absolute SHAP values per feature (global importance ranking)
Force plot	HTML/matplotlib	Interactive or static visualization of a single prediction
mean_abs_shap	pd.Series	Per-feature mean absolute SHAP value for ranking and reporting

Troubleshooting

Problem	Cause	Solution
Very slow computation	Using KernelExplainer for tree model	Use TreeExplainer for tree-based models
Slow on large dataset	Computing all samples at once	Sample subset: explainer(X_test[:1000]) or batch
SHAP values don't sum to prediction	Wrong model output type	Check model_output parameter; verify additivity
Log-odds vs probability confusion	Tree classifier defaults to log-odds	Use TreeExplainer(model, model_output="probability")
Plots too cluttered	Too many features shown	Set max_display=10 or use feature clustering
DeepExplainer error	Background data too small	Use 100-1000 background samples
Memory error	Large dataset + many features	Reduce background data with shap.kmeans(X, 50)
Force plot not rendering	Missing JS in notebook	Run shap.initjs() at notebook start
Inconsistent importance across runs	KernelExplainer sampling variance	Increase nsamples or use deterministic explainer
Negative importance for relevant feature	Feature interactions or correlations	Use feature_perturbation="interventional" or scatter plots

Bundled Resources

• references/theory.md — Mathematical foundations: Shapley value formula, key properties (additivity, symmetry, dummy, monotonicity), computation algorithms (Tree SHAP, Kernel SHAP, Deep SHAP, Linear SHAP), conditional expectations (interventional vs observational), comparison with LIME/DeepLIFT/LRP/Integrated Gradients, interaction values, theoretical limitations

Not migrated from original: references/explainers.md (340 lines) — detailed constructor parameters, methods, and performance benchmarks for each explainer class. Explainer selection guide and common usage are covered inline in Workflow Step 1 and Key Parameters.

Not migrated from original: references/plots.md (508 lines) — comprehensive parameter reference for all 9 plot types with advanced customization (violin, decision, feature clustering). Main plot types are covered inline in Workflow Steps 3-6.

Not migrated from original: references/workflows.md (606 lines) — detailed step-by-step workflows for feature engineering, model comparison, deep learning explanation, production deployment, and time series. Core patterns are covered in Common Recipes; consult original for extended workflows.

Best Practices

1. Choose specialized explainers first — TreeExplainer > LinearExplainer > DeepExplainer > KernelExplainer. Only use model-agnostic explainers when no specialized one exists
2. Start global, then go local — begin with beeswarm/bar for overall importance, then waterfall/scatter for individual predictions and feature relationships
3. Use multiple visualizations — different plots reveal different insights; combine global (beeswarm) + local (waterfall) + relationship (scatter)
4. Select appropriate background data — 100-500 representative samples from training data; use shap.kmeans() for efficiency
5. Validate with domain knowledge — unexpectedly high feature importance may indicate data leakage, not true predictive power
6. Remember SHAP shows association, not causation — a feature's high SHAP importance means the model uses it, not that it causally affects the outcome
7. Consider feature correlations — correlated features share SHAP importance; use feature_perturbation="interventional" for causal interpretation or feature clustering for grouped importance

References

• Lundberg & Lee (2017). "A Unified Approach to Interpreting Model Predictions" (NeurIPS)
• Lundberg et al. (2020). "From local explanations to global understanding with explainable AI for trees" (Nature Machine Intelligence)
• Official documentation: https://shap.readthedocs.io/
• GitHub: https://github.com/shap/shap

Related Skills

• scikit-learn-machine-learning — model training for SHAP analysis
• matplotlib-scientific-plotting — custom SHAP plot styling and export
• statistical-analysis — statistical testing to complement SHAP interpretation