autoresearch-ai-plugin

Autoresearch AI Plugin

Autonomous Experiment Loops for Claude Code — Let AI optimize while you sleep

Edit code → commit → run benchmark → measure metric → keep improvement or revert → repeat forever.

Works for any optimization target: LLM training loss, test speed, bundle size, build time, Lighthouse scores, and more.

Inspired by Karpathy's autoresearch, pi-autoresearch, and litesearch.

Skills

This plugin provides two skills that work together. Autoresearch is the core engine (works for any metric), and Autoresearch ML extends it with GPU-specific templates for LLM training.

1. Autoresearch (The Optimizer)

Domain-agnostic autonomous experiment loop.

• Edit → Measure → Keep/Discard: Autonomous cycle that edits code, runs benchmarks, and keeps only improvements.
• Context-Resilient: State persists in autoresearch.jsonl — survives context resets and session restarts.
• Confidence Scoring: MAD-based statistical analysis separates real improvements from measurement noise.
• ASI (Actionable Side Information): Structured annotations per experiment that survive git reverts — the only memory of discarded experiments.
• Secondary Metrics: Track tradeoff metrics (memory, compile time) alongside the primary optimization target.
• Segments: Multi-phase sessions — switch optimization targets mid-session without losing history.
• Cancel & Status: Check progress or stop the loop at any time while preserving experiment history.
• Any Metric: Test speed, bundle size, build time, Lighthouse scores, memory usage — if you can measure it, you can optimize it.

2. Autoresearch ML (The Researcher)

Specialized for LLM training with NVIDIA GPUs. Extends the core Autoresearch skill.

• Ready-to-Use Template: Complete LLM pretraining setup based on Karpathy's autoresearch (GPT + Flash Attention + MuonAdamW).
• Consumer to Datacenter GPUs: Supports NVIDIA GPUs from 4GB (GTX 1080 Ti) to 80GB (H100) with automatic VRAM scaling guidance.
• Fixed Time Budget: Every experiment runs for exactly 5 minutes — all results are directly comparable.
• Bits Per Byte: Vocab-size-independent metric (val_bpb) enables fair comparison across architectures.

---

Quick Start

Prerequisites

• Git — experiments use git commit/revert for state management
• For ML skill: NVIDIA GPU with 8GB+ VRAM, CUDA 12.0+, Python 3.10+, uv

Installation

Option 1: CLI Install (Recommended)

Use npx skills to install skills directly:

# Install all skills
npx skills add proyecto26/autoresearch-ai-plugin

# Install specific skills
npx skills add proyecto26/autoresearch-ai-plugin --skill autoresearch autoresearch-ml

# List available skills
npx skills add proyecto26/autoresearch-ai-plugin --list

This automatically installs to your .claude/skills/ directory.

Option 2: Claude Code Plugin

Install via Claude Code's built-in plugin system:

# Add the marketplace
/plugin marketplace add proyecto26/autoresearch-ai-plugin

# Install the plugin
/plugin install autoresearch-ai-plugin

Option 3: Clone and Copy

git clone https://github.com/proyecto26/autoresearch-ai-plugin.git
cp -r autoresearch-ai-plugin/skills/* .claude/skills/

Option 4: Git Submodule

Add as a submodule for easy updates:

git submodule add https://github.com/proyecto26/autoresearch-ai-plugin.git .claude/autoresearch-ai-plugin

Then reference skills from .claude/autoresearch-ai-plugin/skills/.

Option 5: Fork and Customize

1. Fork this repository
2. Customize skills for your specific needs (add new metrics, change templates)
3. Clone your fork into your projects

Usage Examples

"Run autoresearch to optimize my test suite"

Triggers Autoresearch to set up a benchmark loop, measure test runtime, and iteratively optimize your test configuration.

"Start an experiment loop to reduce bundle size"

Triggers Autoresearch to measure your build output and autonomously try tree-shaking, code splitting, and dependency optimizations.

"Set up ML autoresearch with my RTX 4090"

Triggers Autoresearch ML to copy the training assets, prepare data, and begin autonomous LLM pretraining experiments.

"Optimize val_bpb autonomously overnight"

Triggers Autoresearch ML to run 5-minute training experiments in a loop, keeping architecture and hyperparameter improvements.

"What's the autoresearch status?"

Shows a summary of the current session: total runs, kept improvements, best metric, confidence score.

---

How It Works