investigate

/investigate — TAC Investigation Skill

Version: 3.0.0 Architecture: MCP primary (call_tool) + native fallback (jq/grep/websearch.sh). CAPPY agent orchestrates Phases 2-7. Schema: inv_context.json v3.5.0 — see docs/inv_context_v3.5_schema.md

---

Usage

/investigate <symptom or case number>

Examples:

• /investigate SF-03845933
• /investigate XSOAR 8.5 Docker container OOM killed
• /investigate XSIAM correlation rules not matching bind variables

---

Phase Overview

Phase	Executor	Skill	Primary Tools	Fallback
0-1: Pre-Flight, Discovery	Main Claude	skills/initialize/SKILL.md	case-get + case-comments	Vision Direct (manual)
2: Triage	CAPPY Agent	skills/triage/SKILL.md	pattern-match + case-get	jq on local DB
3: Evidence	CAPPY Agent	skills/evidence/SKILL.md	analyze-evidence + har-forensics + log-analytics	tar/jq/grep
4: Hypothesis	CAPPY Agent	skills/sherlock/SKILL.md + skills/recon/SKILL.md	Reasoning + falsification	—
5: Validation	CAPPY Agent	skills/resolution/SKILL.md	cortex-docs + confluence-search + case-search	websearch.sh + JIRA gateway
6: Solution	CAPPY Agent	skills/synthesis/SKILL.md	Compile validated hypothesis	—
7: Deliverables	CAPPY Agent	skills/deliverables/SKILL.md	JIRA draft + customer response	JIRA gateway

Supporting skills (read at phase boundaries):

Skill	Purpose	Used In
skills/curator/SKILL.md	Claim registration, citation rules	Phases 2-7
skills/gate/SKILL.md	Phase gate thresholds (source of truth)	All phases
skills/validate/SKILL.md	Citation validation rules	Phases 5, 7
skills/handoff/SKILL.md	Inter-phase gate eval, context extraction	Between phases
skills/logging/SKILL.md	inv_context.json update rules	All phases
skills/escalation/SKILL.md	Escalation paths when blocked	When blocked
skills/initialize/SKILL.md	Phase 0 setup	Phase 0

---

Phase 0-1: Pre-Flight (Main Claude Direct)

SF Case Lookup

Primary (MCP available):

call_tool({ operation: "execute", tool: "case-get",
  params: { case_number: "{case_number}" }})

call_tool({ operation: "execute", tool: "case-comments",
  params: { case_number: "{case_number}" }})

Fallback (native): Open Vision Direct for manual lookup:

http://vision.paloaltonetworks.local:3002/tacopilot/{case_number}

Ask the user to paste the case context (product, version, symptom, customer).

Extract: customer name, product, version, build, symptom description, severity.

Case Directory Setup

case_dir="$HOME/Desktop/Investigations/01_Cases/00_ACTIVE/SF-{case_number}"
mkdir -p "$case_dir"/{evidence,extracted,analysis,deliverables}
cp templates/inv_context_template.json "$case_dir/inv_context.json"

Environment Detection

env_log=$(find "$case_dir" -name "env.log" -type f 2>/dev/null | head -1)
[ -n "$env_log" ] && grep -E "^(Product|Version|Build|Server Name|Deployment):" "$env_log"

# Extract bundles
for bundle in "$case_dir"/*.tar.gz "$case_dir"/evidence/*.tar.gz; do
  [ -f "$bundle" ] && tar -xzf "$bundle" -C "$case_dir/extracted/" 2>&1
done

Read skills/initialize/SKILL.md for full Phase 0-1 procedure.

HITL Checkpoint (Phase 0)

AskUserQuestion({
  questions: [{
    question: "Pre-flight complete. Product: {product}, Version: {version}. Found {N} evidence files. Proceed?",
    header: "Phase 0 — Pre-Flight",
    options: [
      { label: "Correct, proceed", description: "Start investigation" },
      { label: "Wrong product/version", description: "I'll provide corrections" },
      { label: "Missing files", description: "I have more evidence to add" }
    ]
  }]
})

---

Phases 2-7: CAPPY Agent Orchestration

Spawning CAPPY

After Phase 1 approval, spawn CAPPY using the spawn prompts in skills/investigate/spawn-prompts.md.

Phase 2 spawn (SP-1):

Task agent: CAPPY
Prompt: [SP-1 from spawn-prompts.md with {sf_number}, {product}, {version}, {symptom}, {severity}, {case_dir} substituted]

Each phase follows this cycle:

Main Claude spawns CAPPY (SP-N)
  ↓
CAPPY reads phase skill + supporting skills
  ↓
CAPPY executes tools (MCP primary → native fallback)
  ↓
CAPPY returns phase JSON + gate status
  ↓
Main Claude spawns CAPPY (SP-HANDOFF)
  ↓
CAPPY evaluates gate, extracts next-phase vars, writes audit trail
  ↓
Main Claude presents hitl_summary → user approves or chooses recovery
  ↓
Main Claude spawns CAPPY (SP-N+1) with next_spawn_vars

Full spawn sequence:

SP-1 (Phase 2 Triage)
  → SP-HANDOFF → HITL
SP-2 (Phase 3 Evidence)
  → SP-HANDOFF → HITL
SP-3 (Phase 4 Hypothesis)
  → SP-HANDOFF → LOOP DECISION ──────────────────────────────────┐
      ├── Loop exit met → SP-4 (Phase 5 Validation)              │
      └── Loop continues:                                         │
            ├── SP-LOOP-3 (re-evidence, targeted) ──→ SP-3 ──→ ──┘
            └── SP-LOOP-2 (re-triage, refined)   ──→ SP-2 → SP-3 → SP-3 → ──┘
SP-4 (Phase 5 Validation)
  → SP-HANDOFF → HITL
SP-SOLUTION (Phase 6 Solution)
  → SP-HANDOFF → HITL
SP-5 (Phase 7 Deliverables)
  → SP-HANDOFF → HITL → Investigation complete

On any 2nd gate failure: SP-ESCALATION

Loop exit conditions (first met wins — proceed to Phase 5):

1. Coherence ≥ 85% AND zero unverified claims
2. Issue definitively classified: customer_config | product_bug | known_issue
3. loop_count == 3 (hard cap — always proceed, document uncertainty)
4. Confidence delta < 5% from previous iteration (diminishing returns)
5. All evidence types exhausted (evidence_ceiling_hit: true)
6. User selects "Proceed anyway" at HITL checkpoint

Loop initialization in Phase 0 — add to inv_context.json:

{
  "investigation": {
    "loop_count": 0,
    "max_loops": 3,
    "loop_history": [],
    "evidence_ceiling": false
  }
}

CAPPY Responsibilities (Phases 2-7)

• Read phase-specific skill, invoke tools (MCP primary, native fallback)
• After each tool call: verify output against source files via jq/grep
• Register claims per skills/curator/SKILL.md (every claim needs file:line citation)
• Check phase gates per skills/gate/SKILL.md
• Return structured phase JSON to Main Claude

Main Claude Responsibilities

• Run Phases 0-1 directly
• After every SP-HANDOFF result: read loop_action and next_spawn before doing anything else
• Present HITL checkpoints to user at every phase boundary using the format below
• Never auto-proceed — always wait for user approval
• Spawn the correct SP based on HandoffResult and user choice

HITL Decision Protocol

After receiving HandoffResult from SP-HANDOFF, Main Claude calls AskUserQuestion then maps the user's response to a spawn action.

Normal phase transition (loop_action absent or null):

AskUserQuestion({
  questions: [{
    question: handoffResult.hitl_summary,
    header: "Phase {N} — {PASS | FAIL}",
    options: [
      { label: "Proceed to Phase {N+1}", description: handoffResult.recommendation },
      { label: "Choose recovery option", description: "Gate failed — show options" },
      { label: "Escalate", description: "Investigation deadlocked" }
    ]
  }]
})
// User: "Proceed" → spawn handoffResult.next_spawn with handoffResult.next_spawn_vars
// User: "Recovery" → present handoffResult.recovery_options, then re-spawn SP-N
// User: "Escalate" → spawn SP-ESCALATION

Loop continues (loop_action == "LOOP"):

AskUserQuestion({
  questions: [{
    question: handoffResult.hitl_summary,
    header: "Investigation Loop — Iteration {loop_count} of {max_loops}",
    options: [
      { label: "Continue loop",    description: "Run targeted {loop_target_phase == 3 ? 're-evidence' : 're-triage'} pass (recommended)" },
      { label: "Proceed anyway",   description: "Accept current findings with documented uncertainty" },
      { label: "Escalate",         description: "Investigation deadlocked — route to engineering" }
    ]
  }]
})
// User: "Continue loop"  → spawn handoffResult.next_spawn (SP-LOOP-3 or SP-LOOP-2)
//                          with handoffResult.next_spawn_vars (includes loop context)
// User: "Proceed anyway" → spawn SP-4 DIRECTLY (bypass further loop evaluation)
//                          pass current phase_result as Phase 4 context to SP-4
// User: "Escalate"       → spawn SP-ESCALATION

Loop converged (loop_action == "PROCEED", exit condition met):

AskUserQuestion({
  questions: [{
    question: handoffResult.hitl_summary,
    header: "Investigation Converged — {loop_exit_condition}",
    options: [
      { label: "Proceed to Phase 5 validation", description: "Classification: {issue_classification}" },
      { label: "Run one more loop pass",        description: "Not satisfied with current findings" },
      { label: "Escalate",                      description: "Investigation deadlocked" }
    ]
  }]
})
// User: "Proceed"     → spawn SP-4 with handoffResult.next_spawn_vars
// User: "One more"    → spawn handoffResult.loop_target SP-LOOP-3 with loop vars
//                       (this overrides the exit condition — force one more iteration)
// User: "Escalate"    → spawn SP-ESCALATION

---

Visual Feedback Protocol

Main Claude MUST output the following banners at the exact moments specified. CAPPY MUST output a status header at the start of each phase execution.

---

Phase Banners (Main Claude — before each spawn)

Print this BEFORE spawning each CAPPY agent:

─────────────────────────────────────────────────────
 CAPPY  Phase {N} — {PHASE_NAME}
 {PHASE_ICON}  {one-line description of what this phase does}
 Case: {sf_number}  |  {product} {version}
─────────────────────────────────────────────────────

Phase name, icon, and description by phase:

Phase	Name	Icon	Description
2	Triage	🔍	Searching pattern database and case history for known issues
3	Evidence	🔬	Analyzing logs, HAR files, and support bundles
4	Hypothesis	🧠	Building and falsifying root cause hypotheses
5	Validation	✅	Validating hypothesis against docs, JIRA, and precedent
6	Solution	📝	Compiling investigation narrative and remediation steps
7	Deliverables	📋	Generating customer response and JIRA escalation draft

Loop re-entry banner:

─────────────────────────────────────────────────────
 CAPPY  Investigation Loop — Iteration {N}
 🔄  {reason for loop}
 Strategy: {re-triage | re-evidence}
─────────────────────────────────────────────────────

Escalation banner:

─────────────────────────────────────────────────────
 CAPPY  Escalation Path Triggered
 ⚠️  Phase {N} failed twice — escalating
 Reason: {recommendation from handoff}
─────────────────────────────────────────────────────

---

HITL Checkpoint Format (Main Claude — after each HANDOFF result)

─────────────────────────────────────────────────────
 CHECKPOINT  Phase {N} — {PASS ✅ | FAIL ❌}
 Score: {score_value}%  (threshold: {threshold}%)
 {hitl_summary from handoff result}
─────────────────────────────────────────────────────

If PASS: ask "Proceed to Phase {next_phase}? [yes / no / details]" If FAIL (first): present recovery_options numbered list If FAIL (second): show escalation banner, spawn SP-ESCALATION

Never skip the checkpoint. Never auto-proceed.

---

Phase Gates

Phase	Gate	Threshold
2: Triage	Confidence	≥ 70%
3: Evidence	Completeness	≥ 80%
4: Hypothesis	Coherence	≥ 85%
5: Validation	Solution quality	≥ 85%
7: Deliverables	Deliverable quality + all claims verified	≥ 90%

Full threshold definitions and recovery options: skills/gate/SKILL.md

---

inv_context.json

State file tracking the full investigation. Initialize from template at Phase 0.

All claims start UNVERIFIED. Deliverables are blocked until verification rate meets threshold. Log all tool calls to tool_usage.mcpToolCalls[]. Log tool/manual discrepancies to tool_usage.toolFallbacks.

Context recovery if corrupted:

ls -la "$case_dir/inv_context.json.bak.*"
cp inv_context.json.bak.PHASE_2 inv_context.json
python3 -c "import json; json.load(open('inv_context.json'))"

---

Non-Negotiable Mandates

Manual Verification After Every Tool Call

After every tool call, spot-check the output against raw source files:

# Verify HAR claim
jq '.log.entries[] | select(.response.status >= 400)' file.har

# Verify log claim
grep -n "{claimed_error}" "$case_dir/extracted/{file}.log"

# Verify pattern match
jq --arg id "{pattern_id}" '.patterns[] | select(.id == $id)' \
  "${CLAUDE_PLUGIN_ROOT}/databases/cappy-cache_latest.json"

Log discrepancies to inv_context.json → tool_usage.toolFallbacks.

Citation Requirement

Every claim must have a source citation before registration:

• Log file: filename.log:line_234
• HAR: filename.har:entry_145
• Pattern: Pattern:P-XSOAR-042
• JIRA: XSUP-12345

No citation = claim is REJECTED. Rejected claims never enter inv_context.json.

Never Fabricate

If evidence is insufficient, request it from the customer:

Symptom	Request
UI errors	HAR file captured with DevTools open
Playbook not running	Incident export + d1.log
Container OOM	docker logs + memory stats
Agent connectivity	Agent logs + proxy config

---

Priority Routing

Priority	Severity	Strategy	Depth
P1	SEV-1	Immediate — parallel triage + evidence	Deep
P2	SEV-2	Expedited — sequential phases, fast checkpoints	Deep
P3	SEV-3	Standard single-agent workflow	Standard
P4	SEV-4	Pattern lookup only, async response acceptable	Quick

For P1/SEV-1: auto-escalate to skills/escalation/SKILL.md if no pattern match within Phase 2.

---

Evidence Hierarchy

Tier	Weight	Examples
PRIMARY	100%	Raw logs, HAR, PCAP, configs, screenshots
SECONDARY	80%	Parsed logs, TAC playbooks, JIRA tickets
TERTIARY	50%	AI suggestions, historical cases, patterns

---

Novel Issue Protocol

When pattern search returns zero results:

1. Mark "novelIssue": true in inv_context.json
2. Proceed to Phase 3 — do NOT block on zero patterns
3. Generate symptom-based hypothesis via skills/sherlock/SKILL.md

---

Cross-Verification

Vision HTML and TACO Pilot exports apply filtering. Always cross-verify counts against raw files:

Claim Type	Cross-Verify Command
Error count	grep -c '\berror\b' syslog syslog.1
HTTP 500s	jq '[.log.entries[] | select(.response.status==500)] | length' file.har
UFW blocks	grep -c 'UFW BLOCK' syslog*

Log discrepancies in inv_context.json → toolUsage.manualVerifications.

---

Phase 6.5: Final Verification Checkpoint

Mandatory before Phase 7. Run verification sweep:

Verification Sweep:
- Claims verified: X / Y  (must be 100%)
- Evidence types analyzed: X
- Hypotheses surviving: X  (must be ≥ 1)
- Gate: PASS / FAIL

Block Phase 7 if any claims are UNVERIFIED.

---

TACO Spot (Supplementary)

Use when investigation is stuck or confidence is low. These are supplementary — not primary.

Tool	When to Use
case-helper	Investigation deadlocked, need similar case context
topic-researcher	Novel issue, need background research
jira-aid	Phase 7, enrich JIRA with similar case precedent
tac-review	Pre-delivery, validate solution quality

---

Pattern Enrichment (Post-Investigation)

After a successful Phase 7, enrich the pattern database:

1. Was a pattern matched and confirmed? → Add required_evidence to that pattern
2. No pattern found but should exist? → Create new pattern entry
3. Evidence collection was difficult? → Document in required_evidence

# Add required_evidence to a pattern
jq '.patterns |= map(if .id == "P123" then . + {"required_evidence": ["log_bundle", "server_logs"]} else . end)' \
  "${CLAUDE_PLUGIN_ROOT}/databases/cappy-cache_latest.json" > temp.json \
  && mv temp.json "${CLAUDE_PLUGIN_ROOT}/databases/cappy-cache_latest.json"

---

Quick Reference: call_tool Tool Names

Salesforce

Tool	Purpose	Params
case-get	Case details	{ case_number: "03916568" }
case-comments	Comments + Tenant About Info	{ case_number: "03916568" }
case-search	Search by keyword	{ query: "EDL error XSIAM" }
case-similar	Find similar cases	{ case_number: "03916568" }
case-my	My assigned cases	{}
case-private-comment	Add private comment	{ case_number: "03916568", comment: "..." }

Investigation

Tool	Purpose	Params
pattern-match	Pattern DB search (Phase 2)	{ symptom: "EDL 500 error", product: "XSIAM" }
analyze-evidence	HAR/bundle/log analysis (Phase 3)	{ paths: [...], depth: "standard" }
har-forensics	Deep HAR analysis	{ action: "analyze", file: "/path/file.har" }
log-analytics	Log file analysis	{ file_path: "/path/file.log", patterns: ["error"] }
cortex-docs	Cortex documentation search	{ query: "EDL integration XSIAM" }
confluence-search	TAC playbooks, KB articles	{ query: "Broker VM troubleshooting" }
case-search	Closed case precedents	{ symptom: "...", product: "...", status: "closed" }

Utilities

Tool	Purpose	Params
smart-viewer	Images, PDFs, videos	{ path: "/path/to/file.png" }
taco-reader	TACO PILOT HTML exports	{ htmlPath: "/path/to/Vision.html" }
json-utils	JSON parsing	{ jsonPath: "/path/file.json", query: ".field" }

Tool Discovery

call_tool({ operation: "discover", query: "analyze HAR network trace" })
// Returns: tool name, description, inputSchema, examples

Pattern-Match Tips

• Symptom should be 10-100 chars, 3-7 concise keywords
• Verbose inputs return 0 patterns
• Include product name for better relevance