TestVDB - Vector Database Defect Miner

TestVDB

English | 中文

Automated Defect Mining for Vector Databases

TestVDB is an LLM-powered Claude Code plugin that automatically discovers compliance defects in vector databases. It reverse-engineers structured contracts from official documentation, generates targeted attack scripts through multi-agent debate, executes them in Docker sandboxes, and produces verified defect reports with full evidence chains.

Currently supports Milvus, Qdrant, Weaviate, and pgvector.

---

What's New in v2.2.0 — Command Decoupling

The monolithic /testvdb:mine pipeline is now split into three independently-triggerable, intelligently-collaborating commands:

Command	Stage	Output
/testvdb:contract <db> <version> [--force]	Doc extraction + contract generation	structured_contract.json
/testvdb:intel <db> [--max-issues N] [--max-commits N] [--force]	Intelligence gathering + threat modeling	threat_model.json
/testvdb:mine <db> <version> [--intel | --contract] [...]	Attack mining (intelligently consumes intel/contract cache)	defects + reports

Smart cache reuse (D-judgment) — scripts/check_cache.py decides whether to reuse cached intel/contract via four conditions: exists → TTL-fresh → valid → target/version match. Any miss → regenerate; all hit → pure mining (skip generation, save time).

--intel/--contract parameter control with C-boundary semantics:

Cache state	--xxx false behavior
MISSING (no cache)	Error exit ("missing, run /testvdb:xxx first")
STALE / INVALID	Use existing + warning (no refresh)
USABLE	Use as-is

This distinguishes "I have it but want the old one" from "I don't have it at all" — preventing silent mining without prerequisites.

End-to-end verified (CC 2.1.165): 5 agent types dispatched successfully with zero unknown — knowledge-extractor, contract-formalizer, issue-miner, bug-shape-extractor, threat-modeler.

Full Changelog →

---

What's New in v2.1.3

• Anti-Shortcut Enforcement: Stop-hook pipeline gate (scripts/hooks/pipeline_gate.py) validates three LLM shortcut symptoms at session end — (1) document analysis coverage below threshold, (2) unjustified fallback without documented reason, (3) pipeline phase not reaching DONE. Gate performs exact string matching (not fuzzy) — generic or placeholder URLs result in exit 2 interception.
• Agent Contract Hardening: All three attack agents now include mandatory step-by-step contracts: Read raw_knowledge.md → locate ## Document Sources table → copy URLs character-by-character.
• Gate Path Bug Fix: _resolve_round_dir() correctly resolves timestamp_dir against project_root (pipeline v3 convention) with fallback to session_dir-relative paths.
• Configurable Gate Thresholds: TESTVDB_GATE_ACTIVE_THRESHOLD (default 600s) and TESTVDB_DOC_COVERAGE_THRESHOLD (default 0.6) configurable via environment variables.

---

What's New in v2.1.2

• Cross-Turn State Machine: pipeline_state.json v3 — phase-level checkpoint recovery across context compaction.
• ScheduleWakeup Loop: Multi-round mining uses ScheduleWakeup-driven cross-turn iteration; reconstruct_context.py rebuilds full pipeline context from disk state at each loop turn.
• Executor Reliability Fix: Template variable substitution moved from embedded bash to explicit Step 0 shell assignments — zero-byte log bug eliminated.

---

What's New in v2.1.1

• Quality Hardening: All attack scripts use safe_request() pattern — zero bare API calls.
• AST-based API Format Validation: validate_api_format.py in Stage 1 debate.
• Target Neutrality Validation: validate_target_neutrality.py ensures attack scripts don't leak DB-specific signatures (e.g. Qdrant port 6333 when target is Weaviate).
• Reporter Split: reporter.md (defect reports) split from reporter-mre.md (MRE scripts).

---

Table of Contents