agent-architecture-audit

Agent Architecture Audit

Diagnose failures in agent systems by inspecting each layer between the raw model call and the user-visible output.

Contract

Inputs:

• Agent or LLM application repository, failing run, trace, log, or symptom
• Optional model/provider list, tool definitions, memory files, and UI output

Outputs:

• Severity-ranked findings with evidence references
• Layer-by-layer failure diagnosis
• Ordered fix plan that prefers code gates over prompt-only changes

Creates/Modifies:

• None in audit mode
• Follow-up fixes only when explicitly requested

External Side Effects:

• None by default
• External logs, observability tools, or production systems only when already authorized

Confirmation Required:

• Before changing prompts, memory, tool contracts, persistence, production config, or user data

Delegates To:

• debug for ordinary software defects
• evaluation or advanced-evaluation for benchmark design
• security-audit for prompt injection, secrets, auth, or privileged tool risk

When to Use

• An agent works in a model playground or direct API call but fails in the app.
• A wrapper, orchestration layer, memory system, or tool router was added and quality regressed.
• The model skips required tools, claims tool use that did not happen, or misreads tool output.
• Old conversation facts, stale memories, or compressed summaries leak into new tasks.
• Logs show a correct answer but the CLI, API, UI, streaming layer, or renderer shows a different result.
• Hidden retry, fallback, repair, or summarization loops may be mutating output.

Do not use for general code review — only when the failure is likely in the agent stack.

Layer Model

Audit the stack from source instructions to delivered output:

Layer	What Can Fail
System prompt	Conflicts, bloat, stale rules, unclear priorities
Session history	Old turns dominate or contradict the current task
Long-term memory	Cross-project leakage, stale preferences, agent-written facts
Distillation	Compaction turns guesses into pseudo-facts
Active recall	Redundant summaries waste context or revive old context
Tool selection	Required tools are optional in code, not enforced
Tool execution	Missing calls, failed calls, unvalidated arguments
Tool interpretation	Output is ignored, overtrusted, or read backward
Answer shaping	Format, schema, markdown, or JSON is changed after reasoning
Transport/rendering	Streaming, API, CLI, or UI mutates valid content
Hidden repair loops	A second model pass silently rewrites the answer
Persistence	Cached artifacts or expired state are reused as live evidence

Audit Process

1. Scope the System

Identify:

• user entrypoints and output surfaces
• model providers, model tiers, and retry/fallback behavior
• prompt assembly path and instruction sources
• memory, retrieval, compaction, and persistence paths
• tool schemas, tool routing, and execution validation
• known symptoms and the first release or commit where they appeared

2. Collect Evidence

Search for:

• prompt templates and instruction injection points
• tool schemas, routers, validators, and execution logs
• memory admission, retrieval, and compaction code
• provider calls outside the primary agent loop
• fallback, retry, repair, or output rewriting passes
• response serialization, markdown rendering, JSON parsing, and stream handling

Prefer file and line evidence. Compare raw model output, post-processed output, transported output, and rendered output when logs are available.

3. Map Failure Mechanisms

For each suspected issue, record:

• symptom: what the user sees
• layer: where the corruption enters
• mechanism: how the layer changes behavior
• root cause: code, config, prompt, memory, transport, or process
• evidence: file, line, trace id, log row, or reproduced run
• confidence: high, medium, or low

4. Prioritize Fixes

Prefer fixes in this order:

1. Enforce required tool use in code.
2. Remove or expose hidden repair and fallback agents.
3. Reduce duplicated context across prompt, memory, history, and summaries.
4. Tighten memory admission so user corrections outrank agent assertions.
5. Narrow compaction triggers and preserve uncertainty markers.
6. Pass through already-valid output instead of rewriting it.
7. Use typed envelopes for internal protocol boundaries.
8. Add trace tests that compare raw, processed, transported, and rendered output.

Do not solve tool discipline, memory safety, or transport corruption only by adding stronger prompt wording.

Severity

Severity	Meaning
Critical	The system can confidently perform wrong or unsafe operational behavior
High	Correctness or stability degrades frequently under normal use
Medium	The agent usually works, but the stack is fragile, wasteful, or hard to debug
Low	Maintainability, observability, or cosmetic output issues

Output Format

Lead with findings:

## Findings

| Severity | Layer | Finding | Evidence | Fix |
| --- | --- | --- | --- | --- |
| High | Tool selection | Required retrieval is prompt-only and can be skipped | `src/agent/router.ts:42` | Gate final answer on retrieval result |

## Diagnosis

[Explain which layer corrupts the behavior and why.]

## Fix Plan

1. [Code-first fix]
2. [Validation]
3. [Follow-up hardening]

If no issues are found, say so and list the evidence checked plus remaining blind spots.

Anti-Patterns

• Blaming the model before testing wrapper-layer behavior.
• Accepting "must use tool" prompt text when no code enforces the requirement.
• Treating compressed summaries as ground truth.
• Allowing hidden model passes to rewrite final output without traceability.
• Reporting green status without comparing internal output to delivered output.