gaia-architecture-comparison

GAIA Architecture Comparison Skill

Compare swarmdo's GAIA benchmark harness against the Princeton HAL reference implementation and other open-source harnesses to understand capability gaps and prioritize improvements.

When to use

• Planning the next iteration of GAIA work
• Evaluating which architectural change has the highest pass-rate ROI
• Onboarding a new contributor to the benchmark codebase

Architecture overview

swarmdo harness (current)

gaia-bench run
  └─ gaia-loader.ts      — HF dataset download + cache
  └─ gaia-agent.ts       — multi-turn Anthropic Messages loop
       └─ gaia-tools/    — web_search, file_read, web_browse,
                           image_describe, python_exec
  └─ gaia-voting.ts      — Track A self-consistency (N attempts → majority vote)
  └─ gaia-hardness/      — Track Q difficulty predictor (ADR-136)
  └─ gaia-judge.ts       — two-stage LLM-as-judge scorer

HAL reference (Princeton)

HAL uses a similar loop but with:

• OpenAI function calling as the tool interface
• BrowserBase / Playwright for real browser automation
• Code interpreter sandbox (Jupyter kernel)
• Larger token budget per turn (4096+)
• Full 300-question evaluation set

Key differences

Dimension	swarmdo	HAL reference	Gap
Question count	53 (partial L1)	300 (full L1)	Use --limit 165 for full L1
Web search	DuckDuckGo / Google CSE	BrowserBase live	Add Playwright or Browserless
Code execution	python_exec stub	Real Jupyter kernel	Implement real sandbox
Image OCR	image_describe (Gemini)	GPT-4V / Gemini	Functionally equivalent
File handling	file_read	Full PDF/XLSX/ZIP parser	Expand file_read
Self-consistency	voting.ts (Track A)	Not in reference	swarmdo advantage
Hardness routing	predictor.ts (Track Q)	Not in reference	swarmdo advantage
Memory	AgentDB HNSW	None	swarmdo advantage
Pass-rate L1	~20.8% (iter 23)	74.6% (HAL Sonnet 4.5)	~54 pp gap

Gap analysis

Primary gaps (high impact)

1. Real code execution — many L2/L3 questions require running Python to compute a numerical answer. The current python_exec tool is a stub. Implementing a real sandbox (E2B, Pyodide, or subprocess) is the single highest-ROI change.

2. Full question set — running 53/300 L1 questions underestimates true pass-rate because the first 53 skew easier. Run --limit 165 (full L1) for a comparable HAL score.

3. Real browser — web_browse currently fetches raw HTML. Replacing it with Playwright/Browserless for JavaScript-rendered pages would unlock many web navigation questions.

Secondary gaps (medium impact)

4. Structured file parsing — PDF, XLSX, and ZIP attachments require dedicated parsers. file_read currently handles plain text and images only.

5. Turn budget — 12 turns may be insufficient for complex multi-step questions. HAL uses up to 20 turns for L3.

6. System prompt tuning — HAL's system prompt is more elaborate and explicitly instructs the model to use tools before answering.

swarmdo advantages

7. Self-consistency voting (Track A) — running N attempts per question and taking the majority answer reduces variance on borderline questions. HAL does not implement this.

8. Hardness routing (Track Q) — routing each question to an appropriate model and turn budget based on predicted difficulty. This reduces cost on easy questions while providing more resources for hard ones.

9. AgentDB memory — storing patterns across runs enables the agent to recall successful strategies for similar question types.

Improvement roadmap

Priority	Change	Expected Lift	Effort
P0	Real python_exec sandbox (E2B)	+15-25 pp	High
P0	Full 165-Q L1 evaluation	Accurate baseline	Low
P1	Playwright-based web_browse	+5-10 pp	Medium
P1	PDF/XLSX file parser	+3-8 pp	Medium
P2	Increase max-turns to 20 for L2/L3	+2-5 pp	Low
P2	System prompt tuning (iter 30 research)	+2-5 pp	Low
P3	Google Grounding via Gemini (iter 32)	+3-7 pp	Medium
P3	Multi-provider routing (Gemini Flash for cheap Q's)	Cost reduction	Medium

Loading context from past research

npx @swarmdo/cli@latest memory search \
  --namespace gaia-patterns \
  --query "architecture comparison HAL benchmark"

Storing comparison findings

npx @swarmdo/cli@latest memory store \
  --namespace gaia-patterns \
  --key "architecture-comparison-$(date +%Y%m%d)" \
  --value "HAL gap: 54pp. Primary: python_exec stub. Secondary: browser, file parsing."