Ensemble — multi-model council

🗳️ Ensemble

Multi-model consensus debate via the filesystem — LLMs propose, peer-review, rebut, vote, and synthesize a group-confirmed answer. CLI + MCP.

Multi-round: propose → peer-review → rebut → vote → synthesize → converge. Not a one-shot poll — an auditable debate that runs until the models agree on a specific answer (or provably can't), with every step left on disk.

Multi-model consensus debate via the filesystem. Several top LLMs (OpenAI, Anthropic, DeepSeek) independently propose, review each other, rebut the critiques of their own proposal, and vote — and, once a majority agrees, synthesize a single merged answer that the group confirms. They never talk to each other directly: every contribution is a file in a shared folder, and a coordinator advances the debate phase by phase. Participants are anonymized to each other (shown only as "Participant A/B/C"), so they judge arguments on merit, not on brand.

%%{init: {'theme':'neutral', 'themeVariables': {'fontSize':'22px'}, 'flowchart': {'nodeSpacing': 55, 'rankSpacing': 70, 'padding': 16}}}%%
flowchart LR
    P[PROPOSING] --> R[REVIEWING] --> B[REBUTTAL] --> V{VOTING}
    V -- "revise / split<br/>(positions still moving)" --> P
    V -- "stable disagreement<br/>or safety fuse" --> D["Best-effort answer<br/>(plurality, Borda-broken tie)"]
    V -- "majority finalize" --> S["SYNTHESIS<br/>endorsed author merges<br/>(minority views kept)"]
    S --> C{"CONFIRM<br/>APPROVE majority?"}
    C -- yes --> A1["Synthesis = consensus answer"]
    C -- "no / stall / error" --> A2["Verbatim winning proposal<br/>(today's behaviour)"]
    A1 --> F[final.md]
    A2 --> F
    D --> F

Phases PROPOSING → REVIEWING → REBUTTAL → VOTING run every round. SYNTHESIS and CONFIRM run only after a majority finalizes; deadlocks skip them. Alongside its vote, each model may emit a Borda ranking of all proposals — a recorded signal that only ever decides a plurality tie on a deadlock, never a real majority.

Highlights

• 🗳️ Real consensus, not a poll — convergence means a majority endorses the same proposal; otherwise the debate keeps going or provably deadlocks.
• 🎭 Anonymized peer review — models see each other only as "Participant A/B/C", so arguments win on merit, not on brand.
• 🔀 Rebuttal phase — each model answers the critiques of its own proposal before anyone votes, so minds can actually change.
• 🧬 Group-confirmed synthesis — on consensus the endorsed author merges the best points (minority views kept) and the group ratifies it; any failure falls back to the verbatim winner, so the worst case is never worse than today.
• 📊 Borda ranking — a richer per-model signal that breaks deadlock ties deterministically (adapted, with synthesis, from karpathy/llm-council).
• 🗂️ Everything on disk — every proposal, review, rebuttal, vote, and synthesis is a Markdown file; debates are inspectable and fully resumable.
• 💸 Cost-aware — per-model token + USD accounting, prompt caching, and a hard --budget cap.
• 🌐 Grounding & roles — optional web-search citations and anti-groupthink stances (--ground, --roles diverse).
• 🔌 CLI + MCP — a rich terminal UI and a one-tool MCP server for Claude Code, Cursor, Cline, Kilo, Continue, and friends.
• 🧪 Measured, not asserted — a real eval harness (ensemble-eval) with a strong-model baseline and per-question audit logs.

Why filesystem?

Each model only reads and writes Markdown files. That makes every step of the debate a durable, inspectable artifact: you can open any round and read exactly what each model proposed, how it critiqued the others, and how it voted. A debate is fully resumable from disk.

Requirements

• Python ≥ 3.10
• API keys for at least two of three providers (below)
• (optional) a TAVILY_API_KEY for web-search grounding

Install

Ensemble isn't on PyPI yet, so install it from a clone. The [mcp] extra also installs the server used by the editor plugins — include it so you get everything in one go.

git clone https://github.com/raiyanyahya/ensemble.git
cd ensemble

python -m venv .venv
source .venv/bin/activate           # Windows: .venv\Scripts\activate

pip install -e ".[mcp]"             # CLI + MCP server
#   ".[mcp,dev]"  also installs pytest + ruff for development

This puts two commands on your PATH: