linguistic-semantic-algorithms

pproenca Linguistic and Semantic Algorithms Best Practices

Reference of 40 algorithms an agent should reach for when extracting structure, meaning, history, or risk signals from source code and commit data. Categories are ordered by insight-per-effort — how much non-obvious truth the technique exposes relative to how easy it is to apply. The first two categories target the highest-leverage questions: what business entities live in this code? and where else does this concept already exist? — questions that grep and intuition cannot answer.

When to Apply

Reach for these algorithms when:

• Orienting in an unfamiliar codebase: PageRank the import graph to find the core, run LDA over identifier tokens to discover business themes, mine change coupling to surface hidden architectural couplings.
• Hunting a bug from a description: BM25 + history prior + embedding re-rank produces a ranked file shortlist far better than grep.
• Scoping a feature: find prior PRs that did similar work via embedding similarity; map the feature's vocabulary against the codebase's domain via TF-IDF and noun-phrase mining.
• Reviewing a refactor: AST-level GumTree diff reveals semantic impact text diff hides; PDG isomorphism finds the "same logic, different code" twin you should also update.
• Auditing risk: hotspots (churn × complexity), bus factor, defect-magnet density, dead-code candidates — together they direct attention to the parts of the codebase that pay back attention.
• Identifying domain entities and bounded contexts: noun-phrase mining + TF-IDF rare-term extraction + Louvain communities + Jensen-Shannon divergence on per-cluster vocabulary.

Rule Categories by Priority

Priority	Category	Impact	Prefix	Question answered
1	Concept & Domain Extraction	CRITICAL	concept-	What business entities live in this code?
2	Semantic Similarity & Feature Mapping	CRITICAL	sim-	Where else does this concept already exist?
3	Architectural Topology	HIGH	graph-	What is the shape of this codebase?
4	Co-Change & Temporal Mining	HIGH	mine-	What hidden couplings does history reveal?
5	Clone & Duplication Detection	MEDIUM-HIGH	clone-	Where are we repeating ourselves?
6	Bug & Feature Localization	MEDIUM-HIGH	local-	Given a description, where in code?
7	Identifier Linguistics	MEDIUM	ling-	How to prepare tokens so the other algorithms work?
8	Complexity & Risk Metrics	MEDIUM	risk-	Where is the danger concentrated?

Quick Reference

1. Concept & Domain Extraction (CRITICAL)

• concept-lda-topic-modeling — LDA over identifier tokens surfaces latent business themes
• concept-noun-phrase-mining — POS-tag + chunk identifiers to extract entity candidates
• concept-tfidf-rare-terms — IDF against a generic corpus isolates domain vocabulary from framework noise
• concept-identifier-cooccurrence-network — PMI-weighted co-occurrence graph reveals conceptual neighborhoods
• concept-entity-name-resolution — Cluster name variants (user/usr/u/userAccount) via embedding + edit distance
• concept-bounded-context-detection — Louvain + Jensen-Shannon divergence detects DDD bounded contexts

2. Semantic Similarity & Feature Mapping (CRITICAL)

• sim-codebert-embeddings — CodeBERT + cosine for semantic code search across renames
• sim-pdg-semantic-clones — Program Dependence Graph isomorphism finds Type-4 clones
• sim-cross-pr-feature-mapping — Embed merged PRs once, retrieve precedent at feature-design time
• sim-cosine-vsm-files — TF-IDF VSM file similarity when no GPU is available
• sim-call-pattern-similarity — N-grams on call-sequence find behavioral twins
• sim-doc-code-alignment — Joint code-doc embedding flags drift between docs and code

3. Architectural Topology (HIGH)

• graph-pagerank-core — PageRank the import graph to find the codebase core
• graph-betweenness-bottlenecks — Betweenness centrality surfaces bottleneck modules
• graph-louvain-modules — Louvain community detection reveals natural module boundaries
• graph-scc-cycle-tangles — Tarjan's SCC algorithm exposes circular-dependency tangles
• graph-feedback-arcs — Eades-Lin-Smyth FAS chooses the smallest cycle-breaking cut

4. Co-Change & Temporal Mining (HIGH)

• mine-change-coupling — Conditional probability over commit history exposes hidden coupling
• mine-hotspots-churn-complexity — Churn × complexity = canonical hotspot score (Tornhill)
• mine-bus-factor — Per-file authorship Gini coefficient surfaces knowledge concentration
• mine-commit-topic-modeling — LDA on commit messages reveals quarterly themes
• mine-bug-fix-density — Classify commits, rank files by fix-density to find defect magnets
• mine-codebase-aging — Last-modified age + reachability splits stable code from dead code

5. Clone & Duplication Detection (MEDIUM-HIGH)

• clone-minhash-lsh — MinHash + LSH for sub-linear near-duplicate retrieval
• clone-simhash — SimHash 64-bit fingerprints for O(1) Hamming-distance lookups
• clone-suffix-array-cpd — Token-level suffix array (PMD CPD) for precise clone boundaries
• clone-ast-gumtree — GumTree algorithm for fine-grained AST differencing
• clone-zhang-shasha-ted — Zhang-Shasha tree edit distance for exact subtree similarity

6. Bug & Feature Localization (MEDIUM-HIGH)

• local-tfidf-bug-reports — TF-IDF rank source files against bug report tokens
• local-bm25-saturation — BM25 handles length normalization and TF saturation
• local-history-prior-localization — Bayesian fusion of IR score with bug-history prior
• local-embedding-bug-text — Two-stage BM25 + embedding re-rank for semantic localization

7. Identifier Linguistics (MEDIUM)

• ling-camel-snake-split — Split camelCase, snake_case, digit-boundaries before any analysis
• ling-abbreviation-expansion — Expand idx→index, mgr→manager via dictionary + mining
• ling-porter-stemming — Apply Porter stemmer to unify singular/plural forms
• ling-pos-tagging-identifiers — POS-tag identifier heads to flag misnamed functions/classes

8. Complexity & Risk Metrics (MEDIUM)

• risk-cyclomatic-mccabe — McCabe cyclomatic complexity for branch-test surface
• risk-cognitive-complexity — SonarSource Cognitive Complexity for readability gates
• risk-halstead-volume — Halstead volume for language-agnostic size and effort
• risk-shannon-entropy-naming — Per-token entropy flags overloaded names

How to Use

Pick the category that matches the user's question, then read one or two specific rules from that category. Most rules cite combinable partners ("Combine with mine-change-coupling...") that compound the signal — read the partner rule when you need higher precision.

For unfamiliar repos, the highest-ROI starting sequence is:

1. graph-pagerank-core → read the top-20 most central files
2. concept-lda-topic-modeling + concept-tfidf-rare-terms → identify the business themes
3. mine-hotspots-churn-complexity → find where the bugs concentrate
4. mine-change-coupling → uncover hidden architectural couplings

For a single-task bug or feature, the pipeline is:

1. local-bm25-saturation (broad candidates) → local-embedding-bug-text (semantic re-rank) → local-history-prior-localization (fix-history boost)
2. sim-cross-pr-feature-mapping for prior precedent on new features
3. mine-change-coupling to surface partner files that historically move together

Always preprocess identifier tokens via ling-camel-snake-split → ling-abbreviation-expansion → ling-porter-stemming before any vocabulary-based algorithm. Skipping this step silently degrades every downstream signal.

Cross-language parsing. Most rule code examples use Python's built-in ast module for brevity. For real cross-language work (Go, Rust, Java, TS, C++ in the same repo), use tree-sitter — it provides robust parsers for 40+ languages with a uniform API. Every AST-based rule in this skill (PDG clones, GumTree, Zhang-Shasha, POS-tag heads, identifier co-occurrence) maps cleanly onto tree-sitter ASTs.

Reference Files

File	Description
references/_sections.md	Category definitions and impact ordering
assets/templates/_template.md	Template for adding new algorithm rules
metadata.json	Version and reference information