categorize-corpus

Categorize Corpus

Route each document to a category from a fixed list.

When to use

• "I have these 15 categories — sort every note into one."
• "Tag each repo README with one of my 12 topic labels."
• "Multi-label: give each document up to 3 tags from this list."

Prereqs

• A list of category labels, ideally with 1-line definitions. Ambiguous categories cause garbage labels — push back if the user's taxonomy overlaps heavily.

Procedure

1. Pick approach (consult choose-approach):
   • Zero-shot classifier (facebook/bart-large-mnli via HF, or cross-encoder/nli-deberta-v3-base) — free, local, fast, good baseline. Often sufficient for 10-20 well-defined categories.
   • Embedding + centroid: embed each category definition, embed each doc, assign by cosine similarity to nearest category centroid. Very cheap at scale.
   • Local LLM (Ollama llama3.1:8b or qwen2.5:7b) for semantic judgment on <5k docs.
   • Cloud LLM (cheap tier: DeepSeek/Gemini Flash/Haiku) for corpora where local quality isn't enough.
2. Build the prompt (LLM path):

   You are a classifier. Choose the single best category for the document.
   Categories:
   - <label>: <definition>
   - ...
   Document: <text, truncated to budget>
   Respond with JSON: {"category": "<label>", "confidence": 0-1, "reasoning": "<one sentence>"}
   

   Use structured output / JSON mode. Cache the system prompt (Anthropic/OpenAI prompt caching) — same taxonomy for every doc.
3. Calibrate on a sample: label 50-100 docs manually or with a premium model, run the chosen method, compute agreement. If <70%, refine category definitions and retry.
4. Batch the run:
   • Concurrent requests (10-50 in flight) for cloud APIs.
   • Truncate docs to a token budget — for classification, first 500-1000 tokens is usually enough.
   • Two-pass cheap→premium: run everything on cheap model, re-run only confidence<0.7 on premium.
5. Output:
   • categorized.jsonl: {doc_id, category, confidence, reasoning}
   • category-distribution.csv: counts per category.
   • Flag docs where top-1 and top-2 are close → manual review.

Cost control

• Prompt caching: with a 15-category system prompt of ~500 tokens, caching saves ~90% of input cost after the first call.
• Truncate docs: don't send 10k-token documents to classify into 15 buckets — the first paragraph is enough.
• For a 10k-doc corpus with avg 500 tokens/doc, Gemini Flash ≈ $0.05-0.15 total. DeepSeek V3 similar. Sonnet/GPT-4o would be $15-30+.

Multi-label variant

If categories aren't mutually exclusive, ask the model for {"categories": [...]} with max-k constraint, or use zero-shot with a threshold on independent scores.