topic-analysis

Topic Analysis

Find the latent topics in a corpus and (if timestamps exist) chart their evolution.

When to use

• "What are my notes about?"
• "What themes show up in this voice notepad export?"
• "How have the topics in my journal shifted over the last year?"

Pre-flight

Run choose-approach first if the corpus is >1k documents.

Procedure

1. Ingest: load corpus into {id, text, timestamp?, metadata?} records. Accept JSONL, CSV, folder of text/md files, or a transcript dump.
2. Clean: lowercase, strip URLs, remove stopwords, lemmatize. For voice-note corpora, run synonym-cluster first to collapse transcription variants.
3. Choose method:
   • BERTopic (preferred default): sentence-transformer embeddings + UMAP + HDBSCAN + class-based TF-IDF. Handles short docs well, gives readable topic words, supports dynamic/temporal topics out of the box.
   • LDA via gensim: classic, fast, lower quality on short text.
   • Embedding + k-means when the user wants exactly k topics.
   • LLM topic labeling: only to name the clusters produced above — not to do the clustering itself.
4. Temporal view (if timestamps present): bucket by month/quarter/year, compute topic share per bucket, produce a stacked-area or line chart of topic prevalence over time.
5. Label: for each cluster, either present top-10 c-TF-IDF terms, or send those terms + 3 exemplar docs to a cheap LLM ("Give a 3-5 word human-readable label for a cluster whose top terms are … and examples are …").
6. Output:
   • topics.json: [{topic_id, label, keywords, size, exemplar_ids}]
   • topic-evolution.csv: date_bucket, topic_id, count, share
   • Optional topics.png plot.

Cost control

• BERTopic runs locally — embedding generation is the expensive step. Use sentence-transformers/all-MiniLM-L6-v2 (fast, free, local) unless quality is failing.
• If using cloud embeddings (OpenAI text-embedding-3-small, Voyage), batch to 100+ per request; est. $0.02 per 1M tokens.
• LLM labeling: one call per cluster (~20-50 clusters total), not per document. Cheap.

Pitfalls

• Voice-note corpora: short docs kill LDA. Use BERTopic.
• Single-topic corpora (e.g. all notes about one project): HDBSCAN will dump everything into one cluster. Lower min_cluster_size or switch to k-means with explicit k.
• Time-evolution plots mislead on small corpora — require ≥30 docs per time bucket or don't plot it.