The Problem
Every time you use Claude with a large codebase, you're paying for thousands of tokens that aren't relevant to your query. Most context management tools either:
- Send everything (expensive)
- Truncate blindly (loses important context)
- Require heavy Language Servers (slow, resource-intensive)
The Solution
Token Reducer is a local-first, intelligent context compression pipeline that:
- Reduces tokens by 90-98% while preserving semantic relevance
- Runs entirely locally — no API calls, no data leaving your machine
- Works in milliseconds — faster than Language Server alternatives
- Understands code semantically — AST parsing, not just text matching
┌─────────────────┐ ┌───────────────┐ ┌──────────────────┐
│ Your Codebase │────▶│ Token Reducer │────▶│ Compressed │
│ (50,000 tokens)│ │ Pipeline │ │ Context (500t) │
└─────────────────┘ └───────────────┘ └──────────────────┘
│
┌─────────┴─────────┐
│ - AST Chunking │
│ - BM25 + Vector │
│ - TextRank │
│ - Import Graph │
│ - 2-Hop Symbols │
└───────────────────┘
Easy Install
Option 1 — Claude Code /plugin Command (Recommended)
Step 1: Register the marketplace (one-time setup):
/plugin marketplace add Madhan230205/token-reducer
This registers the marketplace as Madhan230205-token-reducer.
Step 2: Install:
/plugin install token-reducer@Madhan230205-token-reducer
For project-scoped install:
/plugin install token-reducer@Madhan230205-token-reducer --scope project
Already ran Step 1 before? Just run /plugin install token-reducer@Madhan230205-token-reducer — no need to add the marketplace again.
Option 2 — Git Clone (Manual)
# 1. Clone into your Claude plugins folder
git clone https://github.com/Madhan230205/token-reducer.git ~/.claude/plugins/token-reducer
# 2. Install dependencies (optional but recommended for best results)
pip install -r ~/.claude/plugins/token-reducer/requirements-optional.txt
Windows users: Replace ~/.claude/plugins/ with %USERPROFILE%\.claude\plugins\
Then open ~/.claude/settings.json and add:
{
"plugins": ["~/.claude/plugins/token-reducer"]
}
Restart Claude Code. Done.
What requirements-optional.txt installs:
| Package | Purpose |
|---|
sentence-transformers | Neural embeddings for smarter retrieval |
hnswlib / faiss-cpu | Fast approximate nearest-neighbor search |
tree-sitter + language grammars | AST-based code chunking (Python, JS, TS, Go, Rust, Java, C/C++, Ruby) |
If you skip this step, Token Reducer still works using hash embeddings and regex chunking — no ML libraries required.
Option 3 — Zero-Dependency Quick Start
No pip, no ML libs — runs immediately after cloning:
git clone https://github.com/Madhan230205/token-reducer.git
cd token-reducer
python scripts/context_pipeline.py run \
--inputs ./src \
--query "Find auth logic" \
--embedding-backend hash \
--db .cache/index.db
Features
Core Pipeline
- Hybrid Retrieval — BM25 + semantic vector search with intelligent fallback
- AST-Based Chunking — Tree-sitter parsing for Python, TypeScript, Go, Rust, Java, and more
- TextRank Compression — Graph-based sentence scoring for intelligent summarization
- Sub-100ms Queries — SQLite FTS5 + HNSW indexes for instant results
- Local-First — Everything runs on your machine, no external APIs
LSP-Killer Features
- Import Graph — Automatically maps file dependencies without Language Server
- 2-Hop Symbol Expansion — Auto "go-to-definition" for referenced functions
- Diff Protocol — SEARCH/REPLACE edit format with automatic application
- Semantic Clustering — Groups similar chunks to avoid redundancy
Madhan230205
