oh-my-evor

---

oh-my-evor turns Claude Code into a team of specialist AI agents that autonomously evolve a machine-learning model — proposing, critiquing, implementing, training, and evaluating candidates through an iterative mutation tree search. After a single setup conversation, it runs to your goal with zero human-in-the-loop.

But autonomy is cheap. The hard part is trust. So oh-my-evor is built around one non-negotiable idea:

Every reported gain must be provably real. No test-set leakage. No reward-hacking. No irreproducible flukes. No moving the goalposts.

---

⭐ Why oh-my-evor?

Most "AI improves your model" tools hand you a number and ask you to trust it. An autonomous agent optimizing a metric will happily — and invisibly — cheat: peek at the test set, overfit the validation split, silently swap the eval, or report a lucky seed. oh-my-evor is engineered so it structurally cannot:

The trap	How oh-my-evor closes it
🕵️ Test-set leakage	Frozen splits are chmod 444; a 13-check integrity gate flags near-perfect scores and per-step val spikes as leakage signatures
🎰 Reward hacking	Direction-aware anti-gaming checks (works for higher- and lower-is-better metrics); the gate may never auto-weaken its own fraud detection
🔒 Agents editing the referee	A hook-enforced capability governor + an always-on write-guard make it impossible for any agent to write the evaluator, touch frozen splits, hand-edit run state, or run training out of turn — enforced at the tool-call layer, not by prompt politeness
📉 Comparability drift	Eval-version is pinned; hardening the test bumps the version and re-scores comparably — never a silent swap
📎 Hand-wavy research	Every SOTA claim the research agent makes is anchored to a source URL — no citation, no claim

The result is an engine you can point at a real dataset and leave alone — and still defend the number it gives you.

---

🚀 Quick Start

Install in two commands:

/plugin marketplace add https://github.com/it-dainb/oh-my-evor
/plugin install oh-my-evor

The MCP server ships prebuilt (no Node build on your machine), and the bundled research MCPs provision their own isolated Python on first use. When you enable the plugin, Claude Code prompts once for an optional Hugging Face token (stored in your keychain) — leave it blank for anonymous access. The compute harness needs its Python deps once; ./install.sh installs them and pre-warms the research MCP environments:

./install.sh    # installs harness deps (pydantic, pyyaml, …) + pre-warms the bundled MCP venvs

If anything is missing, oh-my-evor tells you exactly what to run on your next session — it never fails cryptically.

Using a virtualenv or a non-default Python? The MCP server runs the compute harness through the interpreter named by the EVOR_PYTHON environment variable (default: python3). If you install the harness deps into a venv or any interpreter other than the default python3 (e.g. EVOR_PYTHON=/path/to/venv/bin/python ./install.sh), export that same EVOR_PYTHON in the environment where you launch Claude Code. Otherwise the server falls back to bare python3, which won't have the deps, and harness-backed tools (setup, freeze, preflight, run) will report a missing-dependency error.

Then start a mission — one setup conversation (a structured interview) locks the goal, metrics, and benchmark; after that it runs autonomously: