npx claudepluginhub andikarachman/data-science-plugin --plugin dsThis skill uses the workspace's default tool permissions.
Detect data leakage that would inflate model performance during development but fail in production.
Enforces ML rigor: baseline comparisons vs dummy/linear models, cross-validation, interpretation, leakage prevention with sklearn templates.
Reviews data analysis pipelines for quality, correctness, and reproducibility. Assesses data quality, model validation, leakage detection, and verifies reproducibility. Use for pre-publication reviews, ML pipeline validation, or regulatory audits.
Provides systematic checklist for exploratory data analysis on tabular datasets: structure, missing data, duplicates, distributions, correlations, target analysis. Use when starting EDA.
Share bugs, ideas, or general feedback.
Detect data leakage that would inflate model performance during development but fail in production.
For each feature, verify it would be available at prediction time:
outcome_date when predicting outcome)Check: For each feature, ask: "At the moment we need to make a prediction, would this value already be known?" If no, it's leakage.
Check for features that are transformations of the target:
cancellation_reason when predicting churn)revenue_bucket when predicting revenue)Check: If removing this feature drops model performance by >50%, it may be a proxy for the target.
Flag when any of these occur:
| Signal | Classification Threshold | Regression Threshold |
|---|---|---|
| Single feature AUC | > 0.95 | N/A |
| Single feature R-squared | N/A | > 0.95 |
| Feature importance dominated by 1 feature | >50% of total importance | >50% of total importance |
| Train and test performance nearly identical | Gap < 0.5% | Gap < 0.5% |
Check: Run single-feature models. Any feature with AUC > 0.95 or R-squared > 0.95 warrants investigation.
Check for information leaking across the train/test boundary:
Check: Verify that train_ids.intersection(test_ids) is empty for all entity identifiers.
For each detected leakage:
import pandas as pd
from sklearn.metrics import roc_auc_score
def check_leakage(df, target_col, feature_cols):
"""Flag features with suspiciously high single-feature AUC."""
results = []
for col in feature_cols:
if df[col].dtype in ['float64', 'int64']:
try:
auc = roc_auc_score(df[target_col], df[col])
auc = max(auc, 1 - auc) # Handle inverse correlation
if auc > 0.95:
results.append({'feature': col, 'auc': auc, 'risk': 'HIGH'})
elif auc > 0.85:
results.append({'feature': col, 'auc': auc, 'risk': 'MEDIUM'})
except Exception:
pass
return pd.DataFrame(results).sort_values('auc', ascending=False)