From agent-almanac
Sets up AutoML pipelines using Optuna or Ray Tune for hyperparameter optimization with Hyperband, ASHA, search spaces, and early stopping. For new ML projects, retraining, comparing models, or limited tuning expertise.
npx claudepluginhub pjt222/agent-almanacThis skill is limited to using the following tools:
> See [Extended Examples](references/EXAMPLES.md) for complete configuration files and templates.
Manages hyperparameter tuning operations for ML training, providing guidance, code generation, and configs for data prep, model training, and experiment tracking. Triggers on 'hyperparameter tuner' phrases.
Builds end-to-end AutoML pipelines with data checks, feature engineering, model selection, hyperparameter tuning, evaluation, and deployment artifacts for repeatable ML workflows.
Optimizes ML model hyperparameters via grid search, random search, Bayesian optimization, Optuna, and Hyperopt. Useful for tuning tree models, neural networks, SVMs, ensembles in Python.
Share bugs, ideas, or general feedback.
See Extended Examples for complete configuration files and templates.
Automate hyperparameter tuning and model selection using Optuna or Ray Tune with efficient search strategies.
Install Optuna or Ray Tune with appropriate backends.
# Create virtual environment
python -m venv venv
source venv/bin/activate # On Windows: venv\Scripts\activate
# Option 1: Optuna (simpler, good for single-machine)
pip install optuna optuna-dashboard
pip install scikit-learn xgboost lightgbm
# Option 2: Ray Tune (distributed, good for multi-machine/GPU)
pip install "ray[tune]" optuna hyperopt bayesian-optimization
pip install torch torchvision # if optimizing neural networks
# Visualization and tracking
pip install mlflow tensorboard plotly
Create project structure:
mkdir -p automl/{configs,experiments,models,results}
Expected: Clean environment with required packages installed, no dependency conflicts.
On failure: Use Python 3.8-3.11 (compatibility issues with 3.12+), if CUDA errors occur install CPU-only versions first, on M1/M2 Mac use conda instead of pip for scikit-learn.
Create configuration for hyperparameter search.
# automl/optuna_config.py
import optuna
from optuna.pruners import HyperbandPruner
from optuna.samplers import TPESampler
import xgboost as xgb
from sklearn.metrics import roc_auc_score, mean_squared_error
import numpy as np
# ... (see EXAMPLES.md for complete implementation)
Expected: Search space covers reasonable hyperparameter ranges, objective function runs without errors, pruning stops unpromising trials early.
On failure: If trials crash, reduce search space (e.g., lower max n_estimators), verify data has no NaN/inf values, check memory usage (reduce batch size if OOM), ensure eval_metric matches task type.
Execute hyperparameter search with efficient sampling strategies.
# automl/run_optimization.py
import optuna
from optuna.samplers import TPESampler, CmaEsSampler, NSGAIISampler
from optuna.pruners import HyperbandPruner, MedianPruner, SuccessiveHalvingPruner
import joblib
import pandas as pd
from pathlib import Path
# ... (see EXAMPLES.md for complete implementation)
Expected: Optimization completes with 50-70% of trials pruned early, best parameters found, visualization plots generated showing convergence.
On failure: If no pruning happens, verify objective reports intermediate values correctly, if optimization doesn't improve try different sampler (TPE → CmaES), if crashes with n_jobs>1 use n_jobs=1 for debugging.
Use Ray Tune for multi-GPU or multi-node optimization.
# automl/ray_tune_config.py
from ray import tune
from ray.tune.schedulers import ASHAScheduler, PopulationBasedTraining
from ray.tune.search.optuna import OptunaSearch
from ray.tune.search import ConcurrencyLimiter
import xgboost as xgb
from sklearn.metrics import roc_auc_score
import os
# ... (see EXAMPLES.md for complete implementation)
Expected: Ray Tune runs trials in parallel across CPUs/GPUs, ASHA scheduler stops bad trials early, best configuration found and logged.
On failure: If Ray crashes, start with ray.init(num_cpus=2, num_gpus=0) for debugging, reduce concurrent trials if OOM, check that train function doesn't modify shared data, use tune.report() not return for metrics.
Integrate with MLflow for experiment tracking and model registry.
# automl/mlflow_tracking.py
import mlflow
import mlflow.xgboost
from mlflow.tracking import MlflowClient
import optuna
from pathlib import Path
# ... (see EXAMPLES.md for complete implementation)
Expected: All trials logged to MLflow with parameters and metrics, best model registered in MLflow registry, experiments viewable in MLflow UI.
On failure: Start MLflow UI with mlflow ui --backend-store-uri file:./automl/mlruns, check write permissions to mlruns directory, if registration fails verify model registry is configured, ensure model artifact size < 2GB.
Save optimized model and set up monitoring.
# automl/deploy_model.py
import joblib
import json
from pathlib import Path
import optuna
import xgboost as xgb
# ... (see EXAMPLES.md for complete implementation)
Expected: Model saved in production-ready format, configuration documented, inference script created for deployment.
On failure: If model file too large (>100MB), consider model compression or feature selection, verify model loads correctly in fresh Python session, test inference script with sample data before deployment.
track-ml-experiments - MLflow experiment tracking and versioningorchestrate-ml-pipeline - Airflow/Kubeflow for production AutoML pipelines