From rag-cag
Provides chunking strategies for RAG document processing pipelines: fixed-size, semantic, recursive methods with Python examples, document-type recommendations, and best practices.
npx claudepluginhub jpoutrin/product-forge --plugin rag-cagThis skill uses the workspace's default tool permissions.
This skill provides chunking strategies for RAG document processing.
Generates chunking strategies for RAG systems: 256-1024 token sizes, 10-20% overlaps, semantic boundaries; validates coherence and evaluates precision/recall metrics. For vector DBs and large documents.
Designs Retrieval-Augmented Generation (RAG) systems with expertise in embedding models, vector databases, chunking strategies, retrieval optimization, and hybrid search for LLM apps.
Suggests manual /compact at logical task boundaries in long Claude Code sessions and multi-phase tasks to avoid arbitrary auto-compaction losses.
Share bugs, ideas, or general feedback.
This skill provides chunking strategies for RAG document processing.
def fixed_size_chunk(text: str, chunk_size: int = 500, overlap: int = 50):
chunks = []
start = 0
while start < len(text):
end = start + chunk_size
chunks.append(text[start:end])
start = end - overlap
return chunks
Split on natural boundaries (sentences, paragraphs).
def semantic_chunk(text: str, max_tokens: int = 500):
paragraphs = text.split("\n\n")
chunks = []
current_chunk = []
current_tokens = 0
for para in paragraphs:
para_tokens = count_tokens(para)
if current_tokens + para_tokens > max_tokens:
chunks.append("\n\n".join(current_chunk))
current_chunk = [para]
current_tokens = para_tokens
else:
current_chunk.append(para)
current_tokens += para_tokens
if current_chunk:
chunks.append("\n\n".join(current_chunk))
return chunks
Hierarchical splitting on multiple separators.
SEPARATORS = ["\n\n", "\n", ". ", " "]
def recursive_chunk(text: str, max_size: int, separators: list[str]):
if len(text) <= max_size:
return [text]
sep = separators[0] if separators else ""
chunks = []
parts = text.split(sep)
for part in parts:
if len(part) <= max_size:
chunks.append(part)
elif len(separators) > 1:
chunks.extend(recursive_chunk(part, max_size, separators[1:]))
else:
chunks.append(part[:max_size])
return chunks
| Document Type | Recommended Strategy | Chunk Size |
|---|---|---|
| Technical docs | Semantic (headers) | 500-1000 tokens |
| Legal documents | Semantic (sections) | 1000-2000 tokens |
| Code | Function/class based | 200-500 tokens |
| Conversations | Message boundaries | 100-300 tokens |
| General text | Recursive | 300-500 tokens |
@dataclass
class EnrichedChunk:
content: str
metadata: dict
summary: str # LLM-generated
keywords: list[str]
parent_id: str # For hierarchical retrieval