knowledge-fetch

Knowledge Fetch

Phase 3 of the inverted pipeline. The WebFetch body-pull happens inside the Phase-2 curators, so this skill no longer fetches by default — it reads each candidate's fetch sub-object from <project>/.metadata/candidates.json and assembles the canonical <project>/.metadata/fetch-manifest.json directly. Its remaining active job is opt-in cobrowse (--cobrowse or the interactive prompt): it walks the user through enabling the Claude-in-Chrome browser extension and dispatches source-fetcher sequentially to do two things — recover WebFetch misses (merging rescues into the manifest) and additively top up thin primary-tier sources (reading the fuller browser body beyond WebFetch's cap, superseding the cached body only if strictly longer, never degrading a usable one).

Read ${CLAUDE_PLUGIN_ROOT}/references/inverted-pipeline.md §"Phase 3 — knowledge-fetch" and references/fetch-cache-design.md once to anchor on the contract.

When to run

• candidates.json exists for the project (Phase 2 has run) AND either fetch-manifest.json does not yet exist OR the user explicitly wants to re-fetch (e.g., after evicting stale cache entries)
• User explicitly invokes /cogni-knowledge:knowledge-fetch

Never run when

• No candidates.json exists at <project_path>/.metadata/ — offer knowledge-curate first.
• No binding.json exists at the resolved knowledge root.

Parameters

Parameter	Required	Description
--knowledge-slug	Yes	Slug of the bound knowledge base.
--project-path	Yes	Absolute path to the project directory.
--knowledge-root	No	Override the default knowledge-base directory.
--max-age-days	No	Cache freshness window in days. Default: from binding.curator_defaults.fetch_cache_max_age_days (30). Forwarded to the cobrowse source-fetcher.
--cobrowse	No	Opt in to browser-assisted cobrowse — both recovery of WebFetch misses and additive top-up of thin primary-tier sources (reading the fuller body beyond WebFetch's cap). Default OFF (autonomous runs stay browser-free).
--no-cobrowse	No	Force cobrowse off (both recovery and top-up) even in an interactive session (suppresses the prompt).
--batch-size	No	Vestigial — the default WebFetch path no longer dispatches per batch (the curators already fetched). Cobrowse is per-URL sequential through the single shared browser tab. Kept for backward-compat; ignored.
--tier	No	Restrict the cobrowse sets (both misses and top-up candidates) to a single tier (primary, secondary, supporting). Default: all tiers (the top-up set is already primary-only by construction). No longer bounds WebFetch cost — that is the curator's max_candidates_per_sq (Phase 2).
--dry-run	No	Print the manifest plan (fetched / unavailable counts from the candidates' fetch sub-objects, cobrowse-eligible miss count, and cobrowse top-up candidate count) without dispatching cobrowse.

Workflow

0. Pre-flight

Required plugins. Probe only cogni-wiki (clean-break):

probe_plugin() {
  local plugin="$1" skill="$2"
  test -f "${CLAUDE_PLUGIN_ROOT}/../${plugin}/skills/${skill}/SKILL.md" && return 0
  for d in "${CLAUDE_PLUGIN_ROOT}/../../${plugin}/"*/skills/"${skill}"/SKILL.md; do
    [ -f "$d" ] && return 0
  done
  return 1
}
probe_plugin cogni-wiki wiki-setup && WIKI_OK=yes || WIKI_OK=no

Abort with the standard missing-plugin message on no.

Binding + candidates. Resolve knowledge_root. Read the binding (knowledge-binding.py read) and parse curator_defaults.fetch_cache_max_age_days (default 30 if absent — legacy bindings).

Read <project_path>/.metadata/candidates.json via:

python3 ${CLAUDE_PLUGIN_ROOT}/scripts/candidate-store.py read \
    --project-path <project_path>

Abort if success: false — offer knowledge-curate first.

1. Build the fetch manifest from the curators' results

The bodies were already fetched in Phase 2 — each candidate carries a fetch sub-object. Build <project_path>/.metadata/fetch-manifest.json directly from candidates.json; do not dispatch source-fetcher for the WebFetch results.

1. For each candidate with fetch.status == "ok" → a fetched[] entry: {url, cache_key, content_hash, fetch_method, fetched_at, from_cache} (carry pdf_pages_read / pdf_truncated if present; and carry cobrowse_topup_eligible / body_words if present — the additive flags a thin primary-tier survivor got from the curator).
2. For each candidate with fetch.status == "unavailable" (or no fetch at all — treat a missing sub-object as unavailable with reason: "unfetched") → an unavailable[] entry: {url, reason, attempted_at, fallback_attempted, from_cache}. unfetched is a manifest-only sentinel for a candidate the curator never fetched (a legacy/partial curate) — it is not written to the fetch-cache via fetch-cache.py store, so it is exempt from the closed VALID_REASONS vocabulary; re-run knowledge-curate to populate bodies. Such candidates are not cobrowse-eligible (no fetch.cobrowse_eligible), so they are never offered for cobrowse recovery.
3. Collect the cobrowse-eligible misses: candidates with fetch.status == "unavailable" and fetch.cobrowse_eligible == true. Apply the --tier filter to this set if set.
4. Collect the cobrowse top-up candidates (additive, opt-in enrichment — distinct from the misses): candidates with fetch.status == "ok" AND fetch.cobrowse_topup_eligible == true. These already have a usable body; cobrowse only attempts to supersede it with a fuller browser-rendered one (never recovers — they are not misses). Apply the user's --tier filter to this set too; no additional default is needed, because the curator only flags primary-tier survivors, so the set is already primary-only. A candidate is in at most one of the two cobrowse sets (ok vs unavailable are exclusive).
5. Write the manifest atomically (tempfile.mkstemp + os.replace; inline python3 -c is fine — same pattern fetch-cache.py / knowledge-binding.py use). If a manifest already exists, merge rather than overwrite (dedup each array by URL, keep the newer attempted_at).

If --dry-run: print fetched / unavailable counts, the cobrowse-eligible miss count, and the cobrowse top-up candidate count, and stop.

2. Cobrowse opt-in gate (default OFF)

Cobrowse is never auto-attempted — autonomous runs (knowledge-refresh --mode push) must stay deterministic and browser-free. The gate governs both cobrowse work-items: the misses (recover) and the top-up candidates (enrich). Read "cobrowse work" below as the union of the two sets.

• No cobrowse work (no eligible misses and no top-up candidates) → nothing to do; go to Step 4 (regardless of flags).
• --no-cobrowse → skip cobrowse entirely; go to Step 4.
• --cobrowse → opt in; go to Step 3.
• Neither flag, and there is cobrowse work, and the session is interactive → ask once via AskUserQuestion: "N source(s) failed WebFetch and M primary source(s) could be deepened. Run cobrowse? This opens your browser." (Yes → Step 3; No → Step 4.) Optionally persist the answer as binding.curator_defaults.cobrowse_enabled so future runs honour it without re-prompting (a persisted true/false is treated exactly like --cobrowse/--no-cobrowse).
• Otherwise (neither flag, cobrowse work exists, but the session is non-interactive — the autonomous path: knowledge-refresh --mode push) → default OFF: do NOT call AskUserQuestion (there is no one to answer), go to Step 4. The misses stay unavailable, the top-up candidates keep their usable bodies, and the summary prints the --cobrowse hint. This is the catch-all that keeps autonomous runs browser-free and non-blocking.

3. Cobrowse setup + recovery (when opted in)

Setup (mirror cogni-claims skills/claims/SKILL.md §"Pre-requisite: claude-in-chrome"). Probe mcp__claude-in-chrome__tabs_context_mcp.

• Succeeds → the extension is active; proceed to dispatch.
• Errors → tell the user (reuse cogni-claims' wording as the single source of truth): "Interactive cobrowsing requires the Claude-in-Chrome extension (your real Chrome browser). Please ensure it is active, then tell me to continue." Wait, then re-probe once. If it still errors → record each cobrowse-eligible miss as cobrowse_unavailable in the manifest (no agent dispatch) and go to Step 4.

claude-in-chrome is the browser extension, not a git/native MCP server — it is not in cogni-workspace/references/mcp-git-registry.json and does not route through cogni-workspace:install-mcp. Do not offer install-mcp here.

Recovery (misses). Dispatch source-fetcher in the default recover mode sequentially over the cobrowse-eligible miss URLs (single shared browser tab — no parallelism). Batch them into one or a few dispatches as convenient:

Task(source-fetcher,
     MODE=recover,
     CANDIDATES_PATH=<project_path>/.metadata/candidates.json,
     KNOWLEDGE_ROOT=<knowledge_root>,
     BATCH_URLS=<comma-separated cobrowse-eligible miss URLs>,
     MAX_AGE_DAYS=<max_age_days>,
     BATCH_OUTPUT_PATH=<project_path>/.metadata/.fetch.cobrowse.<NNN>.json)

After each returns, merge its batch JSON into fetch-manifest.json: a cobrowse fetched[] entry upgrades the matching unavailable[] entry (remove it from unavailable[], add to fetched[]); a cobrowse unavailable[] entry overwrites the prior reason. On a source-fetcher failure (no batch file, ok: false), leave the misses as-is and continue.

Top-up (additive enrichment). After recovery, dispatch source-fetcher in topup mode sequentially over the cobrowse top-up candidate URLs (same single shared browser tab). Pass each URL's baseline word count so the fetcher accepts a cobrowse body only if it strictly beats it:

Task(source-fetcher,
     MODE=topup,
     CANDIDATES_PATH=<project_path>/.metadata/candidates.json,
     KNOWLEDGE_ROOT=<knowledge_root>,
     BATCH_URLS=<comma-separated top-up candidate URLs>,
     BASELINE_WORDS=<url1=NNN,url2=NNN,...>,
     MAX_AGE_DAYS=<max_age_days>,
     BATCH_OUTPUT_PATH=<project_path>/.metadata/.fetch.topup.<NNN>.json)

Merge each top-up batch into fetch-manifest.json non-destructively — these rows are already in fetched[]:

• A superseded result → update the matching fetched[] entry's fetch_method to cobrowse_interactive (and its cache_key/content_hash/fetched_at to the cobrowse values). The richer body now backs the URL.
• A kept result (cobrowse body not strictly longer, or cobrowse failed) → leave the fetched[] entry unchanged. The original usable body stands.
• A top-up never moves a row into unavailable[] and never writes a negative-cache entry — a failed enrichment must not degrade a good source (the load-bearing non-degradation invariant). On a source-fetcher failure (no batch file, ok: false), leave every top-up row as-is and continue.

4. Final summary

Print ≤ 10 lines:

• Project: <topic> at <project_path>
• Fetched: <count> (<cache_hits> from cache) — from the Phase-2 curators
• Unavailable: <count> (<reason_top_3>)
• Cobrowse: <recovered> recovered of <eligible> eligible (or "skipped — run with --cobrowse to recover N misses" when off-path)
• Top-up: <superseded> deepened of <candidates> primary-tier candidate(s) (or "skipped — run with --cobrowse to deepen N primary source(s)" when off-path)
• Cost: $X.XX (sum of cost_estimate.estimated_usd across any cobrowse source-fetcher returns)
• Cache stats:

  python3 ${CLAUDE_PLUGIN_ROOT}/scripts/fetch-cache.py stat \
      --knowledge-root <knowledge_root>
  

  Print entries, total_bytes. (C1 check: entries == distinct normalized candidate URLs.)
• Next: run knowledge-ingest --knowledge-slug <slug> --project-path <project_path> (Phase 4) to deposit per-URL wiki pages.

If unavailable_count / total_candidates > 0.3, emit a non-blocking warning: "high unavailable rate (X%) — consider --cobrowse to recover misses, or evicting stale negative-cache entries via fetch-cache.py evict".

5. Record run metrics (phase-exit ledger)

Persist this phase's timing + cost to <project_path>/.metadata/run-metrics.json so the run leaves a durable per-phase ledger (read by knowledge-resume / knowledge-dashboard / a perf study). Capture PHASE_START=$(date -u +%FT%TZ) at the top of this skill's run (Step 0); then at exit:

python3 "${CLAUDE_PLUGIN_ROOT}/scripts/run-metrics.py" record \
    --project-path "<project_path>" --phase fetch \
    --started-at "$PHASE_START" --ended-at "$(date -u +%FT%TZ)" \
    --agent-count <cobrowse source-fetchers (recover + topup), else 0> --cost-usd <summed cobrowse cost, else 0>

The default (no-cobrowse) path dispatches no agents and costs nothing — record --agent-count 0 --cost-usd 0. Fail-soft — a record failure never blocks the phase. Full contract: ${CLAUDE_PLUGIN_ROOT}/references/run-metrics-wiring.md.

Edge cases

• Empty candidates list. Nothing to manifest. Skip to summary with a note. Often means knowledge-curate failed silently — direct the user to re-run curate.
• Candidates with no fetch sub-object. A legacy or partial curate left candidates unfetched (the curate ran before per-candidate body-pull existed, or was interrupted mid-run). Treat them as unavailable with reason: "unfetched"; suggest re-running knowledge-curate to populate bodies.
• Cache populated from a prior project. The curators' Phase-1 cache lookups short-circuited, so those candidates carry fetch.from_cache: true. Cache is shared per-knowledge-base; this is the cross-project compounding win.
• Re-fetch after eviction. fetch-cache.py evict --older-than-days N removes stale entries. Re-running knowledge-curate re-fetches everything missing from cache; knowledge-fetch only rebuilds the manifest + offers cobrowse.
• Cobrowse extension not enabled. When the user opts in but the probe fails after the re-prompt, the misses record cobrowse_unavailable (operator-actionable: enable the extension and re-run with --cobrowse). Off-path (default), misses keep their webfetch_* reason and the summary prints the --cobrowse hint.

Out of scope

• Does NOT WebFetch — the body-pull moved to Phase 2's source-curator. This skill only assembles the manifest and offers opt-in cobrowse recovery (of misses) and opt-in cobrowse top-up (additive deepening of thin primary-tier sources).
• Does NOT extract claims from fetched bodies — that is Phase 4 (source-ingester).
• Does NOT touch the wiki — Phase 4 (knowledge-ingest).
• Does NOT evict cache entries — that is fetch-cache.py evict (manual or via a future knowledge-refresh --vacuum).

Output

• <project_path>/.metadata/fetch-manifest.json (schema 0.1.0; built from the curators' fetch sub-objects + any cobrowse rescues)
• <project_path>/.metadata/.fetch.cobrowse.<NNN>.json for each cobrowse recovery dispatch, and <project_path>/.metadata/.fetch.topup.<NNN>.json for each cobrowse top-up dispatch (intermediate; only when --cobrowse opted in; kept for debugging)
• <knowledge_root>/.cogni-knowledge/fetch-cache/<sha256>.json — shared cache (written by the Phase-2 curators; cobrowse rescues overwrite negative entries here, and a top-up supersedes a thin webfetch/webfetch_fulltext entry with the fuller cobrowse_interactive body)

References

• ${CLAUDE_PLUGIN_ROOT}/references/inverted-pipeline.md — Phase 3 contract
• ${CLAUDE_PLUGIN_ROOT}/references/fetch-cache-design.md — cache mechanics
• ${CLAUDE_PLUGIN_ROOT}/agents/source-fetcher.md — dispatched agent
• ${CLAUDE_PLUGIN_ROOT}/scripts/fetch-cache.py --help
• ${CLAUDE_PLUGIN_ROOT}/scripts/knowledge-binding.py --help
• ${CLAUDE_PLUGIN_ROOT}/scripts/candidate-store.py --help