suggest-categories

Suggest Categories

Bottom-up category derivation. Given a corpus and a target N, produce a workable categorization scheme.

When to use

• "What are the 10 main themes in these 500 GitHub repo READMEs?"
• "Give me 15 categories that would cover my voice notes."
• "I need to extend my taxonomy from 12 to 18 — propose the 6 new ones."

Procedure

1. Sample, don't sweep. A stratified sample of 300-1000 documents gives essentially the same themes as the full corpus. For large corpora (>2k), always sample.
2. Embed + cluster (choose-approach recommends this):
   • Embeddings: all-MiniLM-L6-v2 (local, free) or text-embedding-3-small (cloud, cheap).
   • Cluster: HDBSCAN if N is flexible, k-means with k=N if exact count is required, or BERTopic for topic-word output.
3. Label clusters with an LLM (cheap model — one call per cluster):
   • Input: top-10 c-TF-IDF terms + 3-5 exemplar docs.
   • Ask for: {label, 1-sentence definition, distinguishing examples}.
4. Coverage check: run the proposed categories through categorize-corpus on the sample with a confidence threshold. Anything that lands in "none" or "low-confidence" tells you where the scheme has gaps.
5. Iterate: merge overlapping categories, split oversized ones, add catch-alls sparingly. Aim for roughly balanced cluster sizes unless the user explicitly wants long-tail granularity.
6. Output:

   {
     "proposed_categories": [
       {
         "label": "Infrastructure & DevOps",
         "definition": "Notes about servers, containers, CI/CD, deployment, monitoring.",
         "estimated_share": 0.14,
         "exemplars": ["doc_id_123", "doc_id_456"],
         "top_terms": ["docker", "k8s", "deploy", ...]
       }
     ],
     "uncovered_share": 0.07,
     "notes": "..."
   }
   

Extending an existing taxonomy (N → N+k)

1. Run categorize-corpus with current N categories.
2. Isolate low-confidence + "none" docs.
3. Run this skill on that residue with target k.

Cost control

• One LLM call per cluster (typically 10-30 clusters) — trivially cheap.
• Embeddings are the bulk of cost. Batch 100+/request. For 10k docs × 500 tokens: ~$0.10 at cloud prices, free locally.