ner-extraction

NER Extraction

Pull named entities out of a corpus.

When to use

• "Who is mentioned in my notes?"
• "List all the places referenced."
• "Extract every company name from this transcript dump."

Procedure

1. Pick the tool by scale:
   • spaCy (en_core_web_trf for quality, en_core_web_sm for speed) — free, local, handles millions of docs.
   • GLiNER or flair — better on domain-specific entities than spaCy's defaults, still local.
   • LLM-based NER (cloud or local) — use only when entity types are unusual (e.g. "extract every drug name", "extract every cited paper") or when spans are nested.
2. Run extraction: for each doc, produce [{text, label, start, end, doc_id}].
3. Post-process:
   • Deduplicate across the corpus (case-fold, strip punctuation).
   • Coreference/alias resolution: "Daniel", "Dan", "D. Rosehill" → one canonical entity. Use fuzzy match (RapidFuzz) + manual review or an LLM pass on the unique set only (never per-doc).
   • Run synonym-cluster on the deduped entity list to catch transcription variants.
4. Output:
   • entities.jsonl: one record per mention.
   • entities-summary.csv: canonical_name, label, mention_count, first_seen, last_seen.

Cost control

• Never send the raw corpus to an LLM for NER — use spaCy first. Only escalate to an LLM for ambiguous spans or specialized entity types.
• If you must LLM: deduplicate and batch. A corpus of 10k docs with 50k unique sentences containing entities can often be reduced to a few thousand unique sentence-templates.

Integration

• Feed output to trend-analysis for "mentions of X over time".
• Feed output to correlation-analysis to test whether entity mentions correlate with metadata.