qa-audit

QA Audit

When to Use

• Assessing whether team is following defined QA processes
• Reviewing quality of existing test suite (not just coverage, but test quality)
• Preparing for external audit or certification (ISO, SOC2)
• Onboarding a new project and understanding current quality practices
• Investigating why defect escape rate suddenly increased

Core Jobs

1. Process Audit Framework

## QA Process Audit Checklist

### Requirements & Planning
[ ] User stories have acceptance criteria
[ ] Test approach defined before development starts
[ ] Testability considered in design decisions
[ ] Risk-based testing prioritization applied

### Development Phase
[ ] Code reviewed before merge (evidence in PR history)
[ ] Unit tests written alongside code (not after)
[ ] Static analysis enforced (CI scan results)
[ ] Security scanning in CI (Snyk/Dependabot active)

### Testing Phase
[ ] Test cases trace to requirements
[ ] Negative/boundary test cases present (not just happy path)
[ ] Defects logged with complete STR (steps to reproduce)
[ ] Regression suite maintained and green

### Release Phase
[ ] Release readiness criteria defined and checked
[ ] QA sign-off obtained before production
[ ] Rollback plan defined
[ ] Post-release monitoring plan in place

### Metrics & Improvement
[ ] Defect escape rate tracked monthly
[ ] Root cause analysis conducted for P1 defects
[ ] Process retrospectives including quality topics
[ ] QA metrics visible to stakeholders

2. Test Quality Audit

# Heuristics for reviewing test quality (not just coverage)

def audit_test_quality(test_file: str) -> List[Issue]:
    issues = []
    content = read_file(test_file)
    tests = parse_tests(content)

    for test in tests:
        # Check: does test have assertions?
        if not has_assertions(test):
            issues.append(Issue("NO_ASSERTION", test.name,
                "Test has no assertions — always passes"))

        # Check: is assertion testing something meaningful?
        if has_only_truthy_assertion(test):
            issues.append(Issue("WEAK_ASSERTION", test.name,
                "assert response == True — what does True mean here?"))

        # Check: does test name describe behavior?
        if test.name.startswith("test_") and len(test.name) < 15:
            issues.append(Issue("POOR_NAME", test.name,
                "test_foo doesn't describe what's being tested"))

        # Check: is test independent?
        if calls_other_tests(test):
            issues.append(Issue("DEPENDENCY", test.name,
                "Test calls another test function — hidden dependency"))

        # Check: is setup/teardown balanced?
        if has_setup(test) and not has_teardown(test):
            issues.append(Issue("MISSING_CLEANUP", test.name,
                "Setup without teardown — state leaks between tests"))

    return issues

Test quality anti-patterns to look for:

• Tests that always pass (no assertions or assert True trivially)
• Tests testing implementation details instead of behavior
• Tests with misleading names (test_foo tests bar functionality)
• Duplicate tests testing the same thing in different ways
• Tests coupled to specific data values that will change

3. Code Review Quality Audit

# Review code review effectiveness using git history
git log --format="%H %ae" | while read sha author; do
    # Count reviewers per PR
    reviewers=$(git log $sha -1 --format="%B" | grep "Reviewed-by:" | wc -l)
    if [ "$reviewers" -eq 0 ]; then
        echo "PR $sha by $author: NO REVIEW (merged without review)"
    fi
done

# Check review turnaround time via PR creation -> approval date
# (run via GitHub/GitLab API)

Code review audit questions:

• Are PRs reviewed before merging? (check git log for merge without approval)
• What % of PRs have substantive comments vs rubber-stamp approvals?
• Are security-sensitive changes getting extra review?
• Is review turnaround within SLA?

4. Audit Report Structure

## QA Process Audit Report
**Date:** 2026-04-12
**Team:** Platform Engineering
**Auditor:** QA Lead
**Period:** Q1 2026

## Executive Summary
Overall compliance: 67% (Needs Improvement)
Critical gaps: 3 | High gaps: 5 | Medium gaps: 4

## Findings

### CRITICAL: No test cases for negative/boundary scenarios
**Evidence:** Reviewed 47 test files — 41 test only happy path
**Impact:** High defect escape rate for edge cases (confirmed: 4 production bugs in Q1 from boundaries)
**Recommendation:** Mandate BVA/EP in test design; add to PR checklist

### HIGH: Release readiness checklist not followed
**Evidence:** 6 of 8 releases in Q1 had no formal sign-off documented
**Impact:** 2 of those releases had production incidents within 48 hours
**Recommendation:** Gate pipeline deployment on sign-off document creation

### MEDIUM: Static analysis enabled but findings ignored
**Evidence:** SonarQube shows 134 open issues (oldest: 6 months)
**Impact:** Technical debt accumulating; security issues unaddressed
**Recommendation:** Review backlog; set threshold: no new CRITICAL allowed

## Recommendations (Prioritized)
1. [Critical] Implement boundary/negative test requirement in DoD — 2 weeks
2. [High] Add release sign-off gate to deployment pipeline — 1 week
3. [High] Clear SonarQube backlog: Critical/Blocker items — 4 weeks
4. [Medium] Static analysis policy enforcement in CI — 2 weeks

## Next Audit: Q2 2026 (June)

Key Concepts

• Process audit — systematic review of whether defined processes are being followed
• Test quality review — evaluation of test effectiveness beyond coverage (assertions, independence, naming)
• Gap analysis — difference between defined process (how it should be) and actual practice (how it is)
• Finding — specific observation with evidence, impact, and recommendation
• Compliance rate — % of audit criteria being met; tracks process health over time

Checklist

• Audit scope and criteria defined before starting?
• Evidence collected objectively (git history, CI logs, not just interviews)?
• Findings include evidence + impact + recommendation (not just observations)?
• Critical findings have specific remediation timeline?
• Audit report shared with team and management?
• Follow-up audit scheduled to verify remediation?

Key Outputs

• Audit checklist with compliance rate (% meeting each criterion)
• Findings report: critical/high/medium gaps with evidence and impact
• Remediation plan with owners and timelines
• Follow-up audit schedule

Output Format

• 🔴 Critical — auditing without evidence (interviews only), no follow-up on findings, same issues recurring across audits
• 🟡 Warning — audit that only checks documentation (not actual practice), findings without recommendations, no compliance rate measurement
• 🟢 Suggestion — automate compliance checks where possible (PR review rate, coverage trend), quarterly audits, share audit results publicly within team for transparency

Anti-Patterns

• Audit as blame exercise (creates fear; people hide problems)
• Checklist audit without judgment (tick boxes without assessing actual quality)
• Audit findings that go into a drawer (no follow-up = wasted effort)
• Auditing process that doesn't exist yet (define process first, then audit compliance)

Integration

• qa-process-design — audits compliance with the process designed there
• qc-metrics — metrics provide evidence for audit findings
• qa-risk-management — audit findings feed risk assessment