hipocampus-compaction

Memory Compaction Tree

5-level hierarchical index over raw memory logs. Compaction nodes are search indices — originals are never deleted.

Hierarchy

memory/
  ROOT.md                      <- root node (topic index, ~3K tokens, Layer 1)
  2026-03-15.md                <- raw daily log (permanent, append-only)
  daily/2026-03-15.md          <- daily compaction node
  weekly/2026-W11.md           <- weekly compaction node
  monthly/2026-03.md           <- monthly compaction node

Compaction chain: Raw → Daily → Weekly → Monthly → Root

Tree traversal (search): Root → Monthly → Weekly → Daily → Raw

Fixed vs Tentative Nodes

Every compaction node has a status:

• tentative — period is still ongoing, regenerated when new data arrives
• fixed — period ended, never updated again

# Indicated in YAML frontmatter
---
type: weekly
status: tentative
period: 2026-W11
---

Key: tentative nodes are created immediately — ROOT.md is usable from day one.

When to Run

Called from hipocampus:core Session Start step 7, or directly by an external scheduler (e.g., OpenClaw heartbeat). Check trigger conditions below.

Trigger Conditions

Level	Tentative Create/Update	Fixed Transition
Raw → Daily	On each new raw addition	Date changes
Daily → Weekly	On daily add/change	ISO week ended + 7 days elapsed
Weekly → Monthly	On weekly add/change	Month ended + 7 days elapsed
Monthly → Root	On monthly add/change	Never (root accumulates forever)

Smart Thresholds

Below threshold: copy/concat verbatim (no information loss). Above threshold: generate LLM keyword-dense summary.

Level	Threshold	Above	Below
Raw → Daily	~200 lines	LLM keyword-dense summary	Copy raw verbatim
Daily → Weekly	~300 lines combined	LLM keyword-dense summary	Concat dailies
Weekly → Monthly	~500 lines combined	LLM keyword-dense summary	Concat weeklies
Monthly → Root	Always	Recursive recompaction	(N/A)

Type-Aware Compaction

Memory Types

Entries in daily logs are tagged: ## Topic [type]. Four types exist:

Type	Compaction behavior
user	Always preserve core content. Never compress to Historical Summary.
feedback	Always preserve rule + why + how-to-apply structure. Never compress to Historical Summary.
project	Completed → compress to Historical Summary. Active → keep in Active Context.
reference	Preserve pointer + 1-line description. Mark [?] if >30 days unverified.

Backward compat: Untagged entries (no [type] in heading) → treat as [project].

Topics Keyword Extraction

At every compaction level, extract topic keywords from content and write to frontmatter topics field:

• Scan headings (## Topic [type]) for keywords
• Scan Key Decisions for decision keywords
• Include type tag: topics: [hipocampus [project], terse-responses [feedback]]

Exclusion Filtering

When generating LLM summaries, strip:

• Code blocks (triple backtick) → replace with → filepath:lines
• Stack traces → 1-line error message
• Entries containing "임시", "테스트 중", "나중에 삭제", "temporary", "test run", "delete later" → remove entirely

Algorithm

CRITICAL — STRICT CHAIN ORDER: Steps 2→3→4→5 MUST execute in sequence. NEVER skip a level.

Each step feeds the next. Root reads from monthly. Monthly reads from weekly. Weekly reads from daily. If you skip a level, the chain breaks and data is lost or corrupted.

Raw → [Step 2] → Daily → [Step 3] → Weekly → [Step 4] → Monthly → [Step 5] → Root
         ↑                    ↑                     ↑                    ↑
     reads raw           reads daily            reads weekly        reads monthly
     writes daily/       writes weekly/         writes monthly/     writes ROOT.md

NEVER:

• Modify ROOT.md based on daily or weekly data (root reads ONLY from monthly)
• Modify monthly based on daily data (monthly reads ONLY from weekly)
• Skip Step 2 or 3 because "there's nothing new" — always verify by checking files
• Touch ROOT.md directly without going through the full chain

Step 0: Pre-Compaction Snapshot

Before starting the compaction chain, preserve current working state:

1. Read WORKING.md
2. If it contains active task content (not just the empty template):
   • Append a snapshot to today's daily log (memory/YYYY-MM-DD.md):

     ## Working State Snapshot [project]
     - context: pre-compaction automatic snapshot
     - state: [copy WORKING.md content]
     

   • This ensures in-progress work is captured before any context compression
3. If WORKING.md is empty or contains only the template, skip this step

Step 1: Discover Candidates

Scan memory/ for raw files. Group by date, ISO week, and month. Check each group against trigger conditions.

Step 2: Daily Compaction (max 1 per cycle)

Input: raw files (memory/YYYY-MM-DD.md) Output: daily nodes (memory/daily/YYYY-MM-DD.md)

For each date where raw exists and daily needs create/update:

1. Read raw file memory/YYYY-MM-DD.md
2. Count lines — compare against ~200 line threshold
3. Below threshold: copy raw verbatim to memory/daily/YYYY-MM-DD.md
4. Above threshold: generate keyword-dense summary
5. Write with frontmatter:

---
type: daily
status: tentative
period: YYYY-MM-DD
source-files: [memory/YYYY-MM-DD.md]
topics: [keyword1, keyword2, keyword3]
---

## Topics
## Key Decisions
## Tasks Completed
## Lessons Learned
## Open Items

6. If date has changed (raw is from a past date): set status: fixed

Secret scanning: The mechanical compaction (hipocampus compact) automatically redacts secrets in compaction nodes using regex patterns. When generating LLM summaries for above-threshold nodes, also avoid reproducing any API keys, tokens, passwords, or credentials from the source material. If you encounter a secret in the source, write [REDACTED] in its place.

CHECKPOINT: Verify memory/daily/ has the updated file before proceeding to Step 3.

Step 3: Weekly Compaction (max 1 per cycle)

Input: daily nodes (memory/daily/YYYY-MM-DD.md) — NEVER raw files Output: weekly nodes (memory/weekly/YYYY-WNN.md)

STOP-CHECK: Did Step 2 produce or update a daily node? If not, skip Steps 3-5 entirely — there's nothing new to propagate.

For each ISO week where dailies exist and weekly needs create/update:

1. Read all daily compaction files for that week (from memory/daily/, NOT from memory/)
2. Count combined lines — compare against ~300 line threshold
3. Below threshold: concat all dailies
4. Above threshold: generate keyword-dense weekly summary
5. Write to memory/weekly/YYYY-WNN.md with frontmatter
6. If ISO week ended + 7 days elapsed: set status: fixed

CHECKPOINT: Verify memory/weekly/ has the updated file before proceeding to Step 4.

Step 4: Monthly Compaction (max 1 per cycle)

Input: weekly nodes (memory/weekly/YYYY-WNN.md) — NEVER daily or raw files Output: monthly nodes (memory/monthly/YYYY-MM.md)

STOP-CHECK: Did Step 3 produce or update a weekly node? If not, skip Steps 4-5 — there's nothing new to propagate.

For each month where weeklies exist and monthly needs create/update:

1. Read all weekly compaction files for that month (from memory/weekly/, NOT from memory/daily/)
2. Count combined lines — compare against ~500 line threshold
3. Below threshold: concat all weeklies
4. Above threshold: generate keyword-dense monthly summary
5. Write to memory/monthly/YYYY-MM.md with frontmatter
6. If month ended + 7 days elapsed: set status: fixed

CHECKPOINT: Verify memory/monthly/ has the updated file before proceeding to Step 5.

Step 5: Root Compaction

Input: monthly nodes (memory/monthly/YYYY-MM.md) — NEVER weekly, daily, or raw files Output: memory/ROOT.md

STOP-CHECK: Did Step 4 produce or update a monthly node? If not, DO NOT touch ROOT.md.

When a monthly node is created or updated:

1. Read existing memory/ROOT.md (if exists)
2. Read the new/updated monthly node (from memory/monthly/, NOT from any other directory)
3. Recursive compaction: root = recompact(existing_root + monthly_changes)
   • Active Context: replace with current week's highlights — what's in progress, immediate priorities
   • Recent Patterns: update with newly emerged cross-cutting insights
   • Historical Summary: append/compress older context — merge periods, keep brief summaries
   • Topics Index: merge new topics, update existing entries with new sub-keywords and references
4. Write to memory/ROOT.md
5. If root exceeds size cap (compaction.rootMaxTokens in config, default 3000 tokens / ~100 lines): self-compress — compress Historical Summary first, keep Active Context and Topics Index intact

---
type: root
status: tentative
last-updated: YYYY-MM-DD
---

## Active Context (recent ~7 days)
- topic: current state, what's happening now

## Recent Patterns
- pattern: cross-cutting insight that emerged recently

## Historical Summary
- YYYY-MM~MM: high-level summary of that period
- YYYY-MM: key events

## Topics Index
- topic-keyword [type, Nd]: sub-keywords, references → knowledge/file.md
- topic-keyword [type]: sub-keywords

Age calculation: For each topic, find the most recent source-file date that mentions it. Compute days since that date. Write as Nd (e.g., 2d, 30d).

Type-specific root rules:

• user/feedback topics: always in Topics Index, never in Historical Summary only
• project topics: active → Active Context + Topics Index; completed >90d → Historical Summary only (remove from Topics Index if root exceeds size cap)
• reference topics: mark [?] if >30 days since last mention

Step 6: OpenClaw ROOT.md Sync

OpenClaw only: Sync ROOT.md content into the "Compaction Root" section of MEMORY.md:

• Read MEMORY.md, find ## Compaction Root section
• Replace everything between ## Compaction Root and the next ## heading (or EOF) with the Active Context, Recent Patterns, and Topics Index sections from ROOT.md
• This keeps the auto-loaded MEMORY.md in sync with the canonical ROOT.md

Step 7: Re-index

After writing any compaction files:

qmd update

If vector search is enabled (search.vector: true in hipocampus.config.json):

qmd embed

Guards

• CHAIN ORDER IS MANDATORY: Daily→Weekly→Monthly→Root. Never skip a level. Never read from a wrong source directory.
• Each level reads ONLY from its immediate predecessor: Root←Monthly←Weekly←Daily←Raw
• Raw files: never delete (permanent leaf nodes)
• Max 1 daily + 1 weekly + 1 monthly + 1 root per compaction cycle
• No empty summaries (minimum 50 bytes)
• Skip failed file reads — never abort entire compaction
• qmd update failure: warning only, not fatal
• Root self-compresses when exceeding size cap (shrink older topics first)
• Keyword-dense format only — no prose, no narrative. Optimized for BM25 recall.
• If you feel tempted to "just update ROOT.md quickly" — STOP. Run the full chain.

Agent Memory (Optional)

If memory/agents/compaction/AGENT.md exists, read it before starting compaction. Use learned patterns to inform decisions (e.g., typical raw log size, common threshold behavior).

After compaction completes, if you observed a new pattern worth remembering:

• Append to memory/agents/compaction/AGENT.md
• Keep under ~30 lines
• Example patterns: "this project averages ~80 raw lines/day — daily threshold rarely hit", "weekly nodes frequently need LLM summary (>300 lines)"

If no new patterns were observed, skip the update.

Edge Cases

• Empty days: No daily compaction node is generated for days without raw logs. Weekly naturally skips those days.
• First day: Create the full tentative tree immediately (daily → weekly → monthly → root). ROOT.md is usable from day one.
• Lifecycle example:
  • Day 1: raw created → daily(tentative) → weekly(tentative) → monthly(tentative) → ROOT
  • Day 2: daily(tentative) updated, weekly(tentative) updated, monthly(tentative) updated, ROOT updated
  • Week ends + 7 days: weekly → fixed, new weekly(tentative) starts
  • Month ends + 7 days: monthly → fixed, new monthly(tentative) starts