Arcaneum

Arcaneum

CLI tools and Claude Code plugins for semantic and full-text search.

pipx install "https://github.com/cwensel/arcaneum/releases/download/v0.7.0/arcaneum-0.7.0-py3-none-any.whl"

Overview

Arcaneum helps you discover and understand project dependencies, documentation, and reference implementations. By indexing libraries, frameworks, and technical papers, you can semantically search for patterns, APIs, and concepts when building new projects. Works especially well with the RDR (Recommendation Data Record) model for AI-assisted development planning.

The system supports PDF documents and source code with git-aware, AST-based chunking.

Currently Available:

• Semantic search with Qdrant (vector embeddings)
• Full-text search with MeiliSearch (exact phrase matching)
• Dual indexing workflow for comprehensive search

Features

Search Capabilities

• Semantic Search (Qdrant): Find conceptually similar content using vector embeddings
• Full-Text Search (MeiliSearch): Exact phrase matching, keyword search, and typo-tolerant queries

Indexing

• PDF Indexing: OCR support for scanned documents, page-level metadata, parallel processing
• Source Code Indexing: Git-aware with AST chunking, multi-branch support, 165+ languages
• Markdown Indexing: YAML frontmatter extraction, semantic chunking, incremental sync
• Dual Indexing: Single command to index to both search engines
• Performance Tuning: Granular control over workers, batch sizes, and process priority via arc corpus sync --max-embedding-batch, --text-workers, --cpu-workers, and single-system indexing flags such as --embedding-batch-size and --process-priority

Multiple Embedding Models

• arctic-m (768D) - DEFAULT for PDFs/markdown - stable FastEmbed retrieval model
• stella (1024D) - High-quality opt-in document model, requires arcaneum[sentence-transformers]
• mxbai-large (1024D) - High-quality FastEmbed document model
• jina-code (768D) - DEFAULT for code - stable FastEmbed code model
• jina-code-st (768D) - Legacy SentenceTransformers code path, requires arcaneum[sentence-transformers]
• jina-code-0.5b (896D) - Higher-quality opt-in code model, 32K context, requires arcaneum[sentence-transformers]
• jina-code-1.5b (1536D) - Highest quality code embeddings, SOTA Sept 2025, requires arcaneum[sentence-transformers]
• codesage-large (1024D) - CodeSage V2, 9 programming languages, requires arcaneum[sentence-transformers]
• bge-large (1024D) - BGE large embeddings, balanced performance
• jina-v3 (1024D) - Multilingual embeddings with extended 8K context
• bge-base (768D) - BGE base embeddings, balanced performance and speed
• bge-small (384D) - BGE small embeddings, fastest for size-constrained scenarios

See arc models list for complete model information and recommendations. Use arc models list --json for the LLM-readable catalog: it includes backend, recommended and default corpus uses, support/risk tier, prompt policy, context limit, hardware support, runtime-aligned batch guidance, and reindex warnings.

Arcaneum records each collection's embedding prompt policy when it is indexed. Reindex a corpus after changing model query/document prompts, tasks, or prompt-aware model defaults; semantic search rejects collections whose stored prompt policy no longer matches the current model registry.

GPU Acceleration

• CPU is the default for the most stable indexing behavior
• Supports Apple Silicon (MPS) and NVIDIA GPUs (CUDA)
• Use --gpu to opt into accelerator embedding
• FastEmbed/CoreML on Apple Silicon is experimental and requires ARC_EXPERIMENTAL_COREML=1

CLI-First Design

• All operations via command-line interface
• JSON output mode for automation
• Structured error messages with exit codes
• Python >= 3.12 required

Claude Code Integration

• Slash commands for all operations (/arc:search, /arc:index, /arc:collection, etc.)
• Discoverable via /help or /commands in Claude Code
• No MCP overhead - direct CLI execution

Quick Start

Get started with Arcaneum in just a few commands:

# 1. Install
pipx install "https://github.com/cwensel/arcaneum/releases/download/v0.7.0/arcaneum-0.7.0-py3-none-any.whl"

# 2. Install Claude Code plugin (optional, in Claude Code)
# /plugin install cwensel/arcaneum

# 3. Verify and start services
arc doctor
arc container start

# 4. Create a corpus and sync content (indexes to both Qdrant and MeiliSearch)
arc corpus create Frameworks --type code
arc corpus sync Frameworks ~/libs/fastapi ~/libs/sqlalchemy