embeddings-search-skill

Embeddings Search Skill

A Claude Code plugin that provides hybrid document search combining grep (literal matching) with semantic embeddings for comprehensive search across project documents.

Part of The-Focus-AI/claude-marketplace.

Features

• Hybrid Search: Combines literal text matching (grep) with semantic similarity (embeddings)
• Chunking Strategies: Sliding window, paragraph, or sentence-based chunking
• Metadata Extraction: Automatically extracts collections, document types, authors, and topics
• Taxonomy Discovery: Browse discovered collections, document types, authors, and topics
• Unified Index: Single .embeddings/index.json file for fast loading
• Date & Metadata Filtering: Search within specific time ranges and filter by metadata
• Auto-indexing: Automatically indexes documents on first search
• Incremental Updates: Only re-indexes changed files

Installation

Install via Claude Code plugin marketplace or add directly:

claude --plugin-dir /path/to/embeddings-search-skill

Usage

The skill activates automatically when searching for documents by meaning:

• "Find documents about budget planning"
• "Search for discussions about the Q4 review"
• "Look for files related to authentication"

Manual CLI Usage

# Index a directory (default: sliding window strategy)
npx tsx src/cli.ts index .

# Index with specific strategy
npx tsx src/cli.ts index . --strategy paragraph
npx tsx src/cli.ts index . --strategy sentence
npx tsx src/cli.ts index . --strategy sliding --window 500 --overlap 150

# Search documents
npx tsx src/cli.ts search "budget concerns"

# Search with filters
npx tsx src/cli.ts search "love" --collection "jane austin"
npx tsx src/cli.ts search "meeting notes" --type report
npx tsx src/cli.ts search "quarterly review" --after 2024-01-01 --before 2024-04-01

# View discovered taxonomy
npx tsx src/cli.ts taxonomy .

# List files matching filters
npx tsx src/cli.ts files --collection "jane austin" --type novel

# View index statistics
npx tsx src/cli.ts stats .

Chunking Strategies

• sliding (default): Fixed-size windows with overlap. Best for precise retrieval.
• paragraph: Respects document structure (headers, paragraphs). Best for structured docs.
• sentence: Groups sentences up to target size. Best for natural language queries.

Metadata Extraction

The indexer automatically extracts metadata from:

• File paths: Collection names from directory structure
• Content patterns: Document type (novel, letter, report, essay, documentation)
• Headers: Title, author, date
• Text analysis: Topics/themes, mentioned people

Filtering Options

• --collection <name> - Filter by collection (derived from directory names)
• --type <type> - Filter by document type (novel, letter, report, essay, documentation, notes)
• --author <name> - Filter by author
• --topic <topic> - Filter by topic (love, marriage, society, money, etc.)
• --path <pattern> - Filter by file path pattern
• --after <date> - Only documents modified after date
• --before <date> - Only documents modified before date

Supported File Types

Directly indexed:

• .txt - Plain text
• .md - Markdown

Requires text extraction first:

• .pdf - PDF documents
• .doc, .docx - Word documents
• .ppt, .pptx - PowerPoint presentations

For binary files, create a .txt sidecar with extracted text (e.g., report.pdf.txt).

Data Storage

Index is stored in .embeddings/index.json containing:

• File metadata (path, size, mtime, content hash)
• Document chunks with embeddings
• Extracted metadata (title, author, collection, topics, people)
• Taxonomy aggregations

Technical Details

• Embedding Model: Xenova/all-MiniLM-L6-v2 (~23MB, downloads on first use)
• Vector Dimensions: 384
• Similarity Metric: Cosine similarity
• Index Version: 5
• Storage: Single JSON index file

Dependencies

• @xenova/transformers - Local embedding model
• fast-glob - File discovery
• tsx - TypeScript execution

License

MIT