kibana-testing-tools

Auto-generate comprehensive Scout API tests from Kibana route definitions, including valid/invalid request tests, RBAC tests, and schema-driven test data generation.

Strategic test suite optimizer for Cypress→Scout migration. Analyzes entire test coverage, identifies blind spots, refactors for performance and DRY principles, then generates optimized Scout suites. Not a 1:1 converter.

Identify, debug, and fix flaky tests systematically by detecting root causes, applying targeted fixes, and verifying stability to eliminate intermittent CI failures.

End-to-end debugging and optimization for @kbn/evals framework suites. Auto-detects failures from Elasticsearch (OTEL traces) or LangSmith, pulls traces, identifies root causes (tool schema, prompt alignment, evaluator logic), reasons about evaluator choices, suggests fixes, runs adaptive audit loops (2 clean passes), auto-applies evaluator adjustments. Use when debugging @kbn/evals failures, improving pass rates, calibrating thresholds, analyzing traces, optimizing eval suites, or reviewing evaluator selection.

Review @kbn/evals code changes for alignment with the "Future of @kbn/evals" vision document. Checks that changes follow the trace-first, Elastic-native direction, use correct ownership boundaries, respect the data model and evaluation entry points, and avoid deepening Phoenix coupling. Use when reviewing PRs, commits, or planned work touching @kbn/evals, evaluation suites, evaluators, the evaluation data model, CI eval pipelines, or the golden cluster. Also triggers on "review evals", "check evals alignment", "evals vision review".