claude-token-reducer

---

The Problem

Every time you use Claude with a large codebase, you're paying for thousands of tokens that aren't relevant to your query. Most context management tools either:

• Send everything (expensive)
• Truncate blindly (loses important context)
• Require heavy Language Servers (slow, resource-intensive)

The Solution

Token Reducer is a local-first, intelligent context compression pipeline that:

• Reduces tokens by 90-98% while preserving semantic relevance
• Runs entirely locally — no API calls, no data leaving your machine
• Works in milliseconds — faster than Language Server alternatives
• Understands code semantically — AST parsing, not just text matching

┌─────────────────┐     ┌───────────────┐     ┌──────────────────┐
│  Your Codebase  │────▶│ Token Reducer │────▶│  Compressed      │
│  (50,000 tokens)│     │   Pipeline    │     │  Context (500t)  │
└─────────────────┘     └───────────────┘     └──────────────────┘
                              │
                    ┌─────────┴─────────┐
                    │  - AST Chunking   │
                    │  - BM25 + Vector  │
                    │  - TextRank       │
                    │  - Import Graph   │
                    │  - 2-Hop Symbols  │
                    └───────────────────┘

---

Easy Install

Option 1 — Claude Code /plugin Command (Recommended)

Step 1: Register the marketplace (one-time setup):

/plugin marketplace add Madhan230205/token-reducer

This registers the marketplace as Madhan230205-token-reducer.

Step 2: Install:

/plugin install token-reducer@Madhan230205-token-reducer

For project-scoped install:

/plugin install token-reducer@Madhan230205-token-reducer --scope project

Already ran Step 1 before? Just run /plugin install token-reducer@Madhan230205-token-reducer — no need to add the marketplace again.

---

Option 2 — Git Clone (Manual)

# 1. Clone into your Claude plugins folder
git clone https://github.com/Madhan230205/token-reducer.git ~/.claude/plugins/token-reducer

# 2. Install dependencies (optional but recommended for best results)
pip install -r ~/.claude/plugins/token-reducer/requirements-optional.txt

Windows users: Replace ~/.claude/plugins/ with %USERPROFILE%\.claude\plugins\

Then open ~/.claude/settings.json and add:

{
  "plugins": ["~/.claude/plugins/token-reducer"]
}

Restart Claude Code. Done.

---

What requirements-optional.txt installs:

Package	Purpose
sentence-transformers	Neural embeddings for smarter retrieval
hnswlib / faiss-cpu	Fast approximate nearest-neighbor search
tree-sitter + language grammars	AST-based code chunking (Python, JS, TS, Go, Rust, Java, C/C++, Ruby)

If you skip this step, Token Reducer still works using hash embeddings and regex chunking — no ML libraries required.

---

Option 3 — Zero-Dependency Quick Start

No pip, no ML libs — runs immediately after cloning:

git clone https://github.com/Madhan230205/token-reducer.git
cd token-reducer
python scripts/context_pipeline.py run \
  --inputs ./src \
  --query "Find auth logic" \
  --embedding-backend hash \
  --db .cache/index.db

---

Features

Core Pipeline

• Hybrid Retrieval — BM25 + semantic vector search with intelligent fallback
• AST-Based Chunking — Tree-sitter parsing for Python, TypeScript, Go, Rust, Java, and more
• TextRank Compression — Graph-based sentence scoring for intelligent summarization
• Sub-100ms Queries — SQLite FTS5 + HNSW indexes for instant results
• Local-First — Everything runs on your machine, no external APIs

LSP-Killer Features

• Import Graph — Automatically maps file dependencies without Language Server
• 2-Hop Symbol Expansion — Auto "go-to-definition" for referenced functions
• Diff Protocol — SEARCH/REPLACE edit format with automatic application
• Semantic Clustering — Groups similar chunks to avoid redundancy