Skill

autoexperiment

From mlx

Autonomous time-budget experiment loop. Modify a training script, train for a fixed wall-clock budget, evaluate, record, repeat. Inspired by karpathy/autoresearch. Use for overnight architecture search, systematic hyperparameter sweeps, or any iterative model improvement workflow.

Install

npx claudepluginhub damionrashford/mlx --plugin mlx

Tool Access

This skill is limited to using the following tools:

Bash(uv run * scripts/time_budget_train.py *) BashReadWriteEditGlobGrep

Preview

Run autonomous time-budget experiment loops. Each iteration modifies `train.py`,

Supporting Assets

evals/evals.jsonreferences/EXPERIMENT.md.templatereferences/autoexperiment-guide.mdscripts/time_budget_train.py

SKILL.md

Similar Skills

design-system

Generates design tokens/docs from CSS/Tailwind/styled-components codebases, audits visual consistency across 10 dimensions, detects AI slop in UI.

team-skills-platform

163.7k

ui-demo

Records polished WebM UI demo videos of web apps using Playwright with cursor overlay, natural pacing, and three-phase scripting. Activates for demo, walkthrough, screen recording, or tutorial requests.

team-skills-platform

163.7k

kotlin-patterns

Delivers idiomatic Kotlin patterns for null safety, immutability, sealed classes, coroutines, Flows, extensions, DSL builders, and Gradle DSL. Use when writing, reviewing, refactoring, or designing Kotlin code.

team-skills-platform

163.7k

Stats

Stars1

Forks0

Last CommitApr 9, 2026

Actions

View Source View Plugin View on GitHub View README

Autoexperiment Skill

Run autonomous time-budget experiment loops. Each iteration modifies train.py, trains for a fixed wall-clock budget, evaluates, records in results.tsv, and repeats.

Setup

Ensure results.tsv exists with a baseline (exp000) before iterating

Create EXPERIMENT.md with your goal, baseline, hypothesis, and constraints

Run: /mlx:autoexperiment path/to/train.py

Protocol

Before each iteration

Read EXPERIMENT.md for the current hypothesis

Read results.tsv for experiment history

Identify one change to make (ONE variable only)

Iteration loop

Edit train.py with the single change

Run with TIME_BUDGET: timeout $BUDGET uv run train.py

Capture exit code and metrics

Record in results.tsv: KEEP / DISCARD / CRASH

If CRASH 3× in a row on the same error → stop, report diagnosis

After each iteration

Update EXPERIMENT.md "Next to try" section

Summarize: what changed, what happened, what's next

Templates

See references/EXPERIMENT.md.template for the hypothesis file format. See scripts/time_budget_train.py for a complete training script template with all patterns.

Key patterns

TIME_BUDGET: wall-clock seconds, not epochs. ~12 experiments/hour at 300s each

val_bpb: total_nats / (math.log(2) * total_bytes) — vocab-independent metric

GC freeze: after step 0 eliminates ~500ms stalls

Fast fail: if math.isnan(loss) or loss > 100: sys.exit(1)

Circuit breaker: 3 consecutive CRASHes on same error → escalate to user

See references/autoexperiment-guide.md for full documentation.

Autoexperiment Skill

Run autonomous time-budget experiment loops. Each iteration modifies train.py, trains for a fixed wall-clock budget, evaluates, records in results.tsv, and repeats.

Setup

Ensure results.tsv exists with a baseline (exp000) before iterating
Create EXPERIMENT.md with your goal, baseline, hypothesis, and constraints
Run: /mlx:autoexperiment path/to/train.py

Protocol

Before each iteration

Read EXPERIMENT.md for the current hypothesis
Read results.tsv for experiment history
Identify one change to make (ONE variable only)

Iteration loop

Edit train.py with the single change
Run with TIME_BUDGET: timeout $BUDGET uv run train.py
Capture exit code and metrics
Record in results.tsv: KEEP / DISCARD / CRASH
If CRASH 3× in a row on the same error → stop, report diagnosis

After each iteration

Update EXPERIMENT.md "Next to try" section
Summarize: what changed, what happened, what's next

Templates

See references/EXPERIMENT.md.template for the hypothesis file format. See scripts/time_budget_train.py for a complete training script template with all patterns.

Key patterns

TIME_BUDGET: wall-clock seconds, not epochs. ~12 experiments/hour at 300s each
val_bpb: total_nats / (math.log(2) * total_bytes) — vocab-independent metric
GC freeze: after step 0 eliminates ~500ms stalls
Fast fail: if math.isnan(loss) or loss > 100: sys.exit(1)
Circuit breaker: 3 consecutive CRASHes on same error → escalate to user

See references/autoexperiment-guide.md for full documentation.