autoresearch

Autoresearch

Autonomous experiment loop: try ideas, measure results, keep what works, discard what doesn't, never stop.

Works for any optimization target: test speed, bundle size, LLM training, build times, Lighthouse scores, binary size, latency, memory usage.

Setup

If autoresearch.md already exists in the working directory, skip setup and resume the loop — read autoresearch.md, autoresearch.jsonl, and git log, then continue experimenting.

Otherwise:

1. Gather context: Ask (or infer from $ARGUMENTS and conversation) the Goal, Command to benchmark, Primary metric (name + direction), Files in scope, and Constraints.
2. Create branch: git checkout -b autoresearch/<goal>-<date> (e.g. autoresearch/test-speed-2026-03-21).
3. Read source files: Understand the workload deeply before writing anything. Read every file in scope.
4. Write session files: Create autoresearch.md and autoresearch.sh (see templates below). If constraints require correctness validation (tests must pass, types must check), also create autoresearch.checks.sh. Commit all.
5. Run baseline: Execute the first experiment with no changes to establish the baseline metric.
6. Start looping: Begin the experiment loop immediately after the baseline is logged.

autoresearch.md

The heart of the session. A fresh agent with no context should be able to read this file alone and run the loop effectively. Invest time making it excellent.

# Autoresearch: <goal>

## Objective
<Specific description of what we're optimizing and the workload.>

## Metrics
- **Primary**: <name> (<unit>, lower/higher is better)
- **Secondary**: <name>, <name>, ...

## How to Run
`./autoresearch.sh` — outputs `METRIC name=value` lines.

## Files in Scope
<Every file the agent may modify, with a brief note on what it does.>

## Off Limits
<What must NOT be touched — evaluation harness, data prep, etc.>

## Constraints
<Hard rules: tests must pass, no new deps, fixed time budget, etc.>

## What's Been Tried
<Update this section as experiments accumulate. Note key wins, dead ends,
and architectural insights so the agent doesn't repeat failed approaches.>

Update autoresearch.md periodically — especially "What's Been Tried" — so resuming agents have full context.

autoresearch.sh

Bash script that runs the benchmark and outputs structured metrics.

#!/bin/bash
set -euo pipefail

# Pre-checks (fast, <1s — catch syntax errors early)
python3 -c "import ast; ast.parse(open('train.py').read())"

# Run benchmark
uv run train.py > /tmp/autoresearch-output.log 2>&1

# Extract and output metrics as METRIC lines
val_bpb=$(grep "^val_bpb:" /tmp/autoresearch-output.log | awk '{print $2}')
echo "METRIC val_bpb=$val_bpb"

Rules:

• Use set -euo pipefail.
• Output METRIC name=value lines to stdout (one per metric). The primary metric name must match what's documented in autoresearch.md.
• Metric names: word chars, dots, or µ (e.g. val_bpb, total_µs, bundle.size_kb).
• Keep the script fast — every second is multiplied by hundreds of runs.
• For fast/noisy benchmarks (<5s), run multiple times inside the script and report the median.
• Update the script during the loop as needed.

autoresearch.checks.sh (optional)

Backpressure checks: tests, types, lint. Only create when constraints require correctness validation.

#!/bin/bash
set -euo pipefail
pnpm test --run --reporter=dot 2>&1 | tail -50
pnpm typecheck 2>&1 | grep -i error || true

When this file exists:

• Run it after every passing benchmark (exit 0).
• If checks fail, log the experiment as checks_failed and revert.
• Check execution time does NOT affect the primary metric.
• Keep output minimal — suppress verbose progress, only show errors.

When this file does not exist, skip checks entirely.

The Experiment Loop

LOOP FOREVER. Never ask "should I continue?" — the user expects autonomous work.

Each iteration:

1. Formulate hypothesis: Based on prior results, source code understanding, and any ideas in autoresearch.ideas.md, choose what to try next.
2. Edit code: Modify the in-scope files. Make a single, focused change per experiment.
3. Commit: git add -A && git commit -m "<short description of what this experiment tries>"
4. Run benchmark:

   timeout 600 ./autoresearch.sh > run.log 2>&1
   

   If the command times out or crashes, treat it as a failure.
5. Parse metrics: Extract METRIC lines from the output:

   grep '^METRIC ' run.log
   

   If no METRIC lines found, the run crashed — read tail -50 run.log for the error.
6. Run checks (if autoresearch.checks.sh exists and benchmark passed):

   timeout 300 ./autoresearch.checks.sh > checks.log 2>&1
   

7. Evaluate and log:
   • Improved (primary metric better than best so far) → status keep. The commit stays.
   • Worse or equal → status discard. Revert: stage autoresearch files first, then reset.
   • Crash (benchmark failed) → status crash. Fix if trivial, otherwise revert and move on.
   • Checks failed → status checks_failed. Revert.
8. Log to JSONL: Append one line to autoresearch.jsonl:

   {"run":1,"commit":"a1b2c3d","metric":0.9979,"metrics":{"val_bpb":0.9979,"peak_vram_mb":45060.2},"status":"keep","description":"baseline","timestamp":1711036800000,"confidence":null}
   

9. On discard/crash/checks_failed — revert code changes:

   # Preserve autoresearch session files, revert everything else
   git add autoresearch.jsonl autoresearch.md autoresearch.sh autoresearch.ideas.md autoresearch.checks.sh 2>/dev/null || true
   git checkout -- .
   git clean -fd
   

10. Check confidence: After 3+ runs, run the confidence script from the skill's installation directory. On macOS, avoid readlink -f; resolve the skill directory once and invoke the script directly:

    skill_dir="$HOME/.agents/skills/autoresearch"
    bash "$skill_dir/scripts/confidence.sh"
    

    Interpret the score:
    • >= 2.0x: Improvement is likely real (green).
    • 1.0-2.0x: Above noise but marginal (yellow).
    • < 1.0x: Within noise — consider re-running to confirm (red).
11. Update session: Periodically update autoresearch.md "What's Been Tried" section and run the summary script to review progress.

Repeat forever until interrupted.

JSONL Schema

Each line in autoresearch.jsonl is a JSON object:

Field	Type	Description
run	number	1-indexed experiment count
commit	string	Short git SHA (7 chars)
metric	number	Primary metric value
metrics	object	All metrics dict (primary + secondary)
status	string	keep, discard, crash, or checks_failed
description	string	What this experiment tried
timestamp	number	Unix timestamp (ms)
confidence	number or null	MAD-based confidence score (null if <3 runs)

Resuming

When autoresearch.md exists in the working directory:

1. Read autoresearch.md for full context (objective, what's been tried, constraints).
2. Read autoresearch.jsonl to reconstruct state (best metric, run count, last segment).
3. Read git log --oneline -20 for recent commit history.
4. Check autoresearch.ideas.md if it exists — prune stale entries, experiment with promising ones.
5. Continue the loop from where it left off. Do not re-run the baseline.

Ideas Backlog

Append promising-but-complex ideas to autoresearch.ideas.md as bullets rather than losing them. Read Ideas Backlog Management for when to add, prune, or delete entries.

Loop Rules

Read Loop Rules when a keep/discard call is ambiguous, you're stuck, or thrashing on the same idea. Key rules used every run:

• Primary metric is king. Improved → keep. Worse/equal → discard.
• Simpler is better. Remove code for equal perf = keep. Ugly complexity for tiny gain = discard.
• Don't thrash. Repeatedly reverting the same idea? Try something structurally different.
• Think longer when stuck. Re-read source files, reason about what the CPU/compiler/runtime is actually doing. Deep understanding beats random variation.
• Crashes: fix if trivial (typo, missing import), otherwise log and move on. Don't over-invest.
• NEVER STOP. The user may be away for hours. Keep going until interrupted.

User Messages During Experiments

If the user sends a message while an experiment is running, finish the current run-evaluate-log cycle first, then incorporate their feedback in the next iteration.