deep-research

deep-research

Orchestrates the full pipeline. Owns the SQLite run log, the break points, and the sub-agent sequencing.

How to invoke

# new run
uv run python .claude/skills/deep-research/scripts/db.py init \
  --question "Your research question" \
  [--config /path/to/config.json] \
  [--overnight]

# returns a run_id — note it. Then run phases:
uv run python .claude/skills/deep-research/scripts/db.py resume --run-id <run_id>

Or use the slash command shortcut: /deep-research "your question".

Overnight mode

Pass --overnight to db.py init to run the full pipeline without blocking at break points.

What changes

Normal mode	Overnight mode
AskUserQuestion at Break 0, 1, 2	overnight.py queue-break auto-resolves each break
Pipeline pauses and waits for user	Pipeline continues immediately with a placeholder answer
User reads break prompt in real time	Prompts + auto-answers collected in digest.md

Workflow

# 1. Start overnight run
run_id=$(uv run python .claude/skills/deep-research/scripts/db.py init \
  --question "Your question" --overnight)

# 2. At each break point, queue instead of asking:
uv run python .claude/skills/deep-research/scripts/overnight.py queue-break \
  --run-id $run_id --break-number 0

# 3. When the pipeline completes, generate the digest:
uv run python .claude/skills/deep-research/scripts/overnight.py digest \
  --run-id $run_id
# → prints path to digest.md

# 4. Morning: read digest.md, then resume to redirect if needed:
uv run python .claude/skills/deep-research/scripts/db.py resume --run-id $run_id

digest.md contents

• Header: run question, started_at, overnight=true
• Section per break (0, 1, 2): the standard pipeline prompt, the overnight placeholder, and a note on how the pipeline continued
• Phase summary table (started/completed/error per phase)
• Final outputs section: paths to brief.md and understanding_map.md

Generated by overnight.py digest --run-id <rid>. Idempotent — safe to call multiple times; always overwrites.

After overnight — redirecting with --resume

--resume still works: it picks up from the first incomplete phase. If you want to change the direction set at a break, answer the next break interactively instead of calling queue-break again. The overnight.py status subcommand shows which breaks were auto-queued vs. answered by a human.

The pipeline

  Social
    │
  [BREAK 0] — user confirms/redirects source pool
    │
  Cartographer → Chronicler → Surveyor
    │
  [BREAK 1] — user validates foundation; uploads Phase 2 instructions
    │
  Synthesist → Architect → Inquisitor → Weaver
    │
  [BREAK 2] — user approves coherence; specifies final artifact format
    │
  Visionary → Steward
    │
  Research Brief + Understanding Map

Each sub-agent is invoked via Claude Code's native sub-agent mechanism — they run with independent context windows and return structured output that this orchestrator persists to the run DB.

Orchestrator responsibilities (agent-facing)

When you (the calling Claude agent) run this skill:

1. Create or resume the run — db.py init or db.py resume.

2. For each search-using persona, harvest MCP results into a shortlist file before invoking the persona. Sub-agents in this runtime do not inherit MCP tool access; the orchestrator (you) has it. Pipe MCP results through harvest.py write so the persona can read from disk:

   # Pseudo-code — call your MCPs, collect results into a JSON array, then:
   echo '<json-array-of-mcp-results>' | python .claude/skills/deep-research/scripts/harvest.py write \
     --run-id <run_id> --persona <persona-name> --phase <phaseN> \
     --query "<the run's research question>"
   

   Six personas need this: scout, cartographer, chronicler, surveyor, architect, visionary. The synthesist, inquisitor, weaver, and steward personas operate purely over in-run artifacts and need no harvest. Suggested per-persona MCP mapping (override via config.json):

   • scout (phase0, ex-social): consensus + semantic-scholar + paper-search + academic — broad sweep
   • cartographer (phase1, ex-grounder): consensus + semantic-scholar citation graph + paper-search — find seminal works
   • chronicler (phase1, ex-historian): consensus + paper-search — retrospectives and surveys
   • surveyor (phase1, ex-gaper): consensus + semantic-scholar — null-result probes
   • architect (phase2, ex-theorist): consensus + semantic-scholar — adjacent-field precedents
   • visionary (phase3, ex-thinker): consensus + semantic-scholar — cross-field analogues

   Source-priority rule (v0.50.2)

   Always try Consensus first, then Semantic Scholar, then Google Scholar (via paper-search MCP), then others. Reasons:

   • Consensus has stronger claim extraction + TLDRs out of the box
   • S2 fills the citation-graph + author-resolution gap Consensus doesn't cover
   • Google Scholar is widest coverage but slowest + most rate-limited; use as third-tier fallback
   • Other MCPs (academic, paper-search-mcp's arxiv/biorxiv search) are angle-specific (preprints, bio-only, etc.) — call after the top three

   When a higher-priority source rate-limits, log it as a notes line on the harvest write and fall through to the next tier — do not abort the harvest.

3. Invoke each sub-agent in order — via Claude Code's Task tool with subagent_type=<agent-name>. Pass the run_id and phase in the prompt so the persona can call harvest.py show. Do not inline their prompts here; they are defined in .claude/agents/.

4. After each agent completes, call db.py record-phase with the agent's structured output.

5. At break points, stop the pipeline. Use AskUserQuestion to prompt the user. Record their input via db.py record-break. Do not proceed until resolved.

6. If any agent errors or returns low-confidence output, record the error and prompt the user — do not silently skip a phase.

7. On completion, the final Scribe phase produces the artifacts; the orchestrator runs /research-eval automatically before marking the run complete.

Configuration (config.json)

Optional per-run config controlling which sources each agent uses. Schema:

{
  "enabled_mcps": {
    "scout":         ["consensus", "semantic-scholar", "paper-search", "academic"],
    "cartographer":  ["consensus", "semantic-scholar", "paper-search"],
    "chronicler":    ["consensus", "paper-search"],
    "surveyor":      ["consensus", "semantic-scholar"],
    "architect":     ["consensus", "semantic-scholar"],
    "visionary":     ["consensus", "semantic-scholar"]
  },
  "max_papers_per_phase": 50,
  "allow_institutional_access": true
}

Missing keys use sensible defaults (all MCPs enabled, 50 papers cap).

Resume semantics

db.py resume --run-id <id>:

1. Looks up the last phase with completed_at IS NULL
2. Invokes that agent fresh
3. Continues onward

A run can be interrupted and resumed across Claude Code sessions — the SQLite DB holds all state.

Outputs

Written to ~/.cache/coscientist/runs/run-<run_id>/:

• brief.md — the Research Brief
• understanding_map.md — the six-section Understanding Map
• eval.md — reference + claim audit report (from /research-eval)
• plus the SQLite DB (~/.cache/coscientist/runs/run-<run_id>.db)

Guardrails

• Never skip a break. The human-in-the-loop review is the whole point of this pipeline.
• Never bypass paper-acquire's triage gate, even when a sub-agent asks for full text urgently.
• Abort if /research-eval reports >30% unattributed claims — something went wrong upstream.