From agent-almanac
Sets up MLflow tracking server, autologging for scikit-learn/PyTorch/TensorFlow/XGBoost, run comparisons with metrics/visuals, artifact management for reproducible ML workflows. For new projects, log migration, or CI/CD integration.
npx claudepluginhub pjt222/agent-almanacThis skill is limited to using the following tools:
> See [Extended Examples](references/EXAMPLES.md) for complete configuration files and templates.
Configures MLflow tracking setup for ML training workflows. Generates code, configurations, and best practices for experiment tracking with PyTorch, TensorFlow, scikit-learn.
Sets up ML experiment tracking with MLflow or Weights & Biases: installs packages, initializes tools, and provides logging code for parameters, metrics, and artifacts.
Onboards users to MLflow by analyzing codebase for GenAI (LLMs, LangChain) or traditional ML (sklearn, PyTorch) use cases and guiding through quickstart tutorials and integrations.
Share bugs, ideas, or general feedback.
See Extended Examples for complete configuration files and templates.
Set up MLflow tracking server and implement comprehensive experiment tracking with metrics, parameters, and artifacts.
pip install mlflow)Set up the MLflow tracking server with appropriate backend stores.
# Option 1: Local file-based tracking (development)
mkdir -p mlruns
export MLFLOW_TRACKING_URI="file:./mlruns"
# Option 2: SQLite backend with local artifacts
mlflow server \
--backend-store-uri sqlite:///mlflow.db \
--default-artifact-root ./mlartifacts \
# ... (see EXAMPLES.md for complete implementation)
Create a configuration file for team sharing:
# mlflow_config.py
import os
MLFLOW_TRACKING_URI = os.getenv(
"MLFLOW_TRACKING_URI",
"http://mlflow-server.company.com:5000"
)
# ... (see EXAMPLES.md for complete implementation)
Expected: MLflow UI accessible at specified host:port, showing empty experiments list. Server logs confirm successful startup without errors.
On failure: Check port availability with netstat -tulpn | grep 5000, verify database connection strings, ensure S3 credentials are configured (aws configure), check firewall rules for remote access.
Enable framework-specific autologging to capture metrics, parameters, and models automatically.
# training_script.py
import mlflow
from mlflow_config import MLFLOW_TRACKING_URI, MLFLOW_EXPERIMENT_NAME
# Set tracking URI
mlflow.set_tracking_uri(MLFLOW_TRACKING_URI)
mlflow.set_experiment(MLFLOW_EXPERIMENT_NAME)
# ... (see EXAMPLES.md for complete implementation)
For PyTorch:
import mlflow.pytorch
mlflow.pytorch.autolog(
log_every_n_epoch=1,
log_every_n_step=None,
log_models=True,
disable=False,
exclusive=False,
# ... (see EXAMPLES.md for complete implementation)
Expected: Run appears in MLflow UI with all hyperparameters, metrics (training/validation loss, accuracy), model artifacts, and input examples automatically logged.
On failure: Verify MLflow version compatibility with ML framework (mlflow.sklearn.autolog() requires MLflow ≥1.20), check if autologging is supported for your model type, disable autologging and use manual logging as fallback, inspect logs with mlflow.set_tracking_uri() for connection errors.
Add custom metrics, parameters, artifacts, and tags for complete experiment documentation.
# comprehensive_tracking.py
import mlflow
import numpy as np
import matplotlib.pyplot as plt
from pathlib import Path
def train_and_log_model(params, X_train, y_train, X_test, y_test):
"""
# ... (see EXAMPLES.md for complete implementation)
Expected: MLflow UI displays rich experiment information including step-by-step metrics, visualization artifacts, model signature, input examples, and comprehensive tags for filtering and searching.
On failure: Check artifact storage permissions (aws s3 ls s3://bucket/path), verify matplotlib backend for figure logging (plt.switch_backend('Agg')), ensure JSON-serializable data types for log_dict, check disk space for local artifact storage.
Use MLflow's comparison tools to analyze multiple experiments.
# compare_runs.py
import mlflow
from mlflow.tracking import MlflowClient
client = MlflowClient()
def compare_experiments(experiment_name, metric_name="test_accuracy", top_n=5):
"""
# ... (see EXAMPLES.md for complete implementation)
Command-line comparison:
# Compare runs using MLflow CLI
mlflow runs compare --experiment-name customer-churn \
--order-by "metrics.test_accuracy DESC" \
--max-results 10
# Export run data to CSV
mlflow experiments csv --experiment-name customer-churn \
--output experiments.csv
Expected: Console output shows sorted runs with key metrics, HTML report generated with formatted comparison table, CSV file contains all run data for further analysis.
On failure: Verify experiment exists with mlflow experiments list, check metric names match exactly (case-sensitive), ensure runs have completed successfully (check run status), verify file write permissions for output files.
Set up S3/Azure/GCS backends for scalable artifact management.
# artifact_storage_config.py
import mlflow
import os
def configure_s3_backend():
"""
Configure S3 for artifact storage.
"""
# ... (see EXAMPLES.md for complete implementation)
Docker Compose for MLflow with PostgreSQL and S3:
# docker-compose.yml
version: '3.8'
services:
postgres:
image: postgres:14
environment:
POSTGRES_DB: mlflow
# ... (see EXAMPLES.md for complete implementation)
Expected: Artifacts upload successfully to remote storage, MLflow UI shows artifact links pointing to S3/Azure/GCS URIs, downloading artifacts from UI works correctly.
On failure: Verify cloud credentials with aws s3 ls or az storage blob list, check bucket/container permissions (need write access), ensure MLflow installed with cloud extras (pip install mlflow[extras]), test network connectivity to storage endpoints, check CORS settings for browser access.
Set up automated cleanup, archival, and organization policies.
# lifecycle_management.py
import mlflow
from mlflow.tracking import MlflowClient
from datetime import datetime, timedelta
client = MlflowClient()
def archive_old_experiments(days_old=90):
# ... (see EXAMPLES.md for complete implementation)
Expected: Old experiments moved to deleted state, failed runs removed from active list, best runs tagged for easy filtering in UI, storage space reclaimed.
On failure: Check experiment permissions (must be owner to delete), verify runs are actually in FAILED status, ensure metric exists for all runs being ranked, check database connectivity for bulk operations, verify sufficient permissions for artifact deletion in remote storage.
MLFLOW_TRACKING_URI environment variable, check firewall rules, ensure server is runningexclusive=True or disable conflicting autologsregister-ml-model - Register tracked models in MLflow Model Registryversion-ml-data - Version datasets using DVC for reproducible experimentssetup-automl-pipeline - Integrate experiment tracking into automated ML pipelinesdeploy-ml-model-serving - Deploy best-performing tracked models to productionorchestrate-ml-pipeline - Combine experiment tracking with workflow orchestration