/evaluate-model

/evaluate-model - Evaluate ML Model

Evaluate machine learning model performance with comprehensive metrics.

Steps

1. Ask the user for the model type: classification, regression, NLP, or generative
2. Load the model and test dataset from the specified paths
3. Run inference on the entire test dataset and collect predictions
4. For classification models, calculate: accuracy, precision, recall, F1-score, AUC-ROC
5. For regression models, calculate: MAE, MSE, RMSE, R-squared, MAPE
6. For NLP models, calculate: BLEU, ROUGE, perplexity, exact match
7. Generate a confusion matrix for classification tasks
8. Identify the worst-performing classes or data segments
9. Calculate calibration metrics: expected calibration error
10. Run performance profiling: inference time per sample, memory usage, throughput
11. Check for bias: evaluate performance across demographic subgroups if applicable
12. Generate a comprehensive evaluation report with all metrics and visualizations

Rules

• Use stratified sampling if the test set is imbalanced
• Report confidence intervals for all metrics when sample size allows
• Include both micro and macro averages for multi-class metrics
• Test on held-out data never seen during training
• Report inference latency percentiles (p50, p95, p99) not just averages
• Check for data leakage between train and test sets
• Include baseline comparisons (random, majority class, previous model version)