qa-strategy

QA Strategy

Ortak Doktrin

agents/shared/severity-rubric.md ve agents/shared/escalation-matrix.md default-load sayılır (agents/coordination.md §11). Bu skill'in çıktısı Critical / High / Medium / Low + kanıt formatında olmak zorunda — spekülatif Critical yasak. Sahiplik dışı bulgu ilgili agent'a delege; karar yetkisi eşiği aşılırsa kullanıcı onayı zorunlu.

Felsefe

• Piramid > saatkulesi/ice-cream-cone. Çok unit + makul integration + az e2e. Tersi (e2e ağırlıklı) yavaş, flaky, pahalı.
• Test = davranış kontratı, kod kopyası değil. "Coverage %100" hedef değil — anlamlı behavior'a coverage hedef.
• Mutation testing > line coverage. Test gerçekten fail veriyor mu?
• Manual exploratory otomasyondan eksik bağlam yakalar.
• Quality gate sadece CI'da değil — pre-release smoke + canary monitoring.

Ne Zaman Kullanılır

• Yeni servis prod'a çıkacak (test gate)
• Coverage düşüş (PR fail trend)
• Flaky test salgını (3+ retry / haftalık)
• "Test piramidi tersine" şüphesi (yavaş CI, kırılgan e2e)
• Bug escape rate yüksek (post-deploy bug)
• UAT prosedürü yok / belirsiz
• Compliance (SOC 2 / ISO testing requirement)

Workflow

1) Mevcut durum tespit

Metric	Tool	Hedef
Test sayısı (unit/int/e2e)	pytest --collect-only, npm test --listTests, go test -list	ratio takip
Coverage staged scope	coverage.py / nyc / go test -cover	≥ %70
Coverage total	aynı	≥ %60
Mutation score	mutmut / Stryker / pitest	≥ %75
Flaky test oranı	CI history son 50 run	< %2
Test süresi (CI)	pipeline metric	unit < 3dk, int < 7dk, e2e < 15dk
Bug escape rate	prod incident / month	trend ↓

2) Test piramidi sağlığı

        Manual / Exploratory  (≤ 5%)
       /
      / E2E (browser, full-stack)        5-10%
     /
    /  Integration (real DB, real services)   20-30%
   /
  /   Unit (fast, isolated)             60-70%
 /__________________________________________________

Pyramid violations:

Pattern	Belirti	Sebep	Düzeltme
Ice-cream cone	E2E > Integration > Unit	"test = scenario" yanlış kavrayış	Unit'i artır, e2e azalt; behavior-driven üst, technical alt
Hourglass	Unit + E2E çok, Integration az	Mock-heavy unit + browser e2e; gerçek entegrasyon kayıp	Integration ekle (testcontainers + real DB)
Cupcake	Hepsi az	Test culture eksik	Önce unit, sonra integration, en son e2e
Inverted brittle	Integration ≥ Unit	DB hit'li unit yazımı	Unit isolation; integration için ayrı suite

3) Stack başına tooling

Stack	Unit	Integration	E2E	Contract	Load	Mutation
Python (FastAPI)	pytest + pytest-asyncio	pytest + httpx + testcontainers	playwright + pytest	schemathesis (OpenAPI)	k6 / Locust	mutmut
Python (Django)	pytest-django + factory_boy	pytest-django + testcontainers PG	playwright	schemathesis	k6	mutmut
Node (Express/Fastify)	vitest / jest	supertest + testcontainers	playwright	dredd / schemathesis	k6 / artillery	Stryker
TypeScript	vitest / jest + ts-jest	supertest + testcontainers	playwright	schemathesis	k6	Stryker
Go	testing + testify	testing + testcontainers-go	playwright (headless)	schemathesis	k6 / vegeta	go-mutesting
Java	JUnit 5	JUnit + Testcontainers	Selenium / playwright	Pact / schemathesis	Gatling / JMeter	pitest
React (frontend)	vitest + React Testing Library	MSW + RTL integration	playwright / cypress	—	Lighthouse CI	Stryker

4) Coverage strategy

• Staged scope (PR diff) ≥ %70.
• Total ≥ %60 (yeni kod yüksek; legacy düşüş tolerans).
• Critical path %100 zorunlu (auth, payment, order, schema migration).
• Drop alarm: PR coverage düşürürse comment + block (codecov, coveralls).
• Branch coverage > line coverage (if/else, switch dalları).
• Coverage tool: dile göre (coverage.py, nyc, go test -cover, jacoco).

5) Mutation testing

Coverage kötü metric: test fail vermese bile satır "covered" sayılır. Mutation testing kodu mutate eder (örn. < → <=); test fail vermezse mutant survives = test zayıf.

• Hedef: ≥ %75 mutation score (Critical path).
• Tool: mutmut (Python), Stryker (JS/TS), go-mutesting (Go), pitest (Java).
• CI gate: opsiyonel (yavaş); release branch'inde mecburi.
• Incremental: sadece staged scope mutation (full repo aşırı yavaş).

6) Flaky test management

• Tespit: 3 retry sonrası hala fail → quarantine.
• Quarantine: skip + issue (sahip + 14 gün limit).
• Kök neden:
  • Time-bound (sleep + race) → freezegun, time.tick.
  • Network bağımlılık → mock veya local container.
  • Test order → independent fixture.
  • Resource leak (port, file) → cleanup defer / afterEach.
  • Assertion eventual → Awaitility / eventually(timeout=5s).
• 14 gün overdue → fail CI (quarantine kalıcılaşmasın).

7) Manual exploratory testing

Heuristic Test Strategy Model (James Bach):

• SFDPOT: Structure, Function, Data, Platform, Operations, Time.
• Test charter (1 saatlik session):

  "Explore [area] with [resources] to discover [info]."

• Session-based test management (SBTM): zaman + bulgular + risk.
• Coverage: otomasyonun yakalamadığı bağlamsal hatalar (UX, edge UI, cross-browser, accessibility nuance).

8) Bug bash

Pre-release organize event:

• Süre: 1-2 saat.
• Katılımcı: 5-15 kişi cross-functional (dev + PM + design + support).
• Setup: staging env + sample data + bug tracker URL.
• Goal: edge case yakala (otomasyon kaçırır).
• Output: bug listesi + severity + sahip.
• Cadence: major release öncesi.

9) UAT (User Acceptance Testing)

• Persona-based scenarios: gerçek kullanıcı veya proxy stakeholder.
• Test case: discovery'deki Gherkin acceptance criteria → UAT script.
• Sign-off: stakeholder approval gerekli go-live için.
• Beta: limited rollout (feature flag %5/%25/%50/%100) UAT phase.

10) Quality gate beyond CI

Gate	Kriter
Pre-commit	lint + format + secret scan
PR	unit + integration + lint + coverage
Merge to main	+ e2e + mutation (sample)
Release branch	+ full mutation + load test (canary RPS) + manual smoke
Pre-deploy	+ security scan + dependency audit
Post-deploy canary	+ synthetic check + SLO burn rate alert
Production	continuous: synthetic + chaos drill + bug escape rate

11) Bug escape rate (north star)

• Production'da bug yakalandıkça → "test piramidi neyi kaçırdı?" retro.
• Her prod bug için regression test ekle (postmortem action item).
• Trend: 90 günlük rolling, hedef düşüş.

Checklist

• Test sayısı / coverage / mutation score ölçüldü
• Pyramid sağlığı (ratio) — violation tespit
• Stack başına tooling tablosu
• Critical path %100 coverage
• Flaky < %2; quarantine + 14 gün limit
• Mutation score ≥ %75 critical path
• Exploratory test charter (en az ayda 1 session)
• Bug bash plan (release öncesi)
• UAT script (Gherkin AC bazlı)
• Quality gate matrix
• Bug escape rate trend (rolling 90d)
• Retro action item: prod bug → regression test

Antipattern

• "Coverage %100" target — line coverage anlamsız, mutation score gerçek.
• E2E heavy (ice-cream cone) — yavaş CI, flaky storm.
• Mock-heavy unit — gerçek davranış test edilmiyor.
• Test silme flaky bahane — kök neden çöz.
• Staging'de manuel test only — automation hiç yok.
• UAT yok — stakeholder approval atlanmış.
• Quarantine kalıcı — 14 gün limit; aksi suite bozulur.
• Critical path coverage düşük (auth, payment, migration).
• Test data prod copy — PII leak; anonymized synthetic.
• Mutation score umursanmıyor — coverage = false confidence.
• Bug bash yok — edge case yakalanmaz.
• Quality gate sadece PR — release branch + production gate eksik.
• Bug escape rate ignore — regression test loop kayıp.

Örnek Agent Davranışı

User: /qa-plan checkout-svc
Agent (test-engineer):
1. Test envanteri: 142 unit, 38 integration, 27 e2e — ratio 67/18/13.
2. Coverage: staged %58 (target %70 — fail), total %52, critical path %72 (auth %91, payment %68 — gap).
3. Mutation score (mutmut sample): %63 — düşük; payment module %48 (Critical).
4. Flaky son 50 run: 4 test (e2e/checkout-flow.spec.ts ts'de %18 fail) — quarantine.
5. Pyramid: ratio OK (67/18/13 yaklaşık 60/20/10) ama integration az — DB mock ağır unit'lar var (28 mock'lu unit gerçekte integration olmalı).
6. CI süresi: unit 4dk (target 3), integration 11dk, e2e 18dk — toplam 33dk PR (target ≤10dk).
7. Plan:
   - DB index/cache mock'lı unit'leri integration'a taşı (Hourglass fix).
   - Payment critical path coverage %68 → %95 hedef (mutation %48 → %85).
   - e2e shard 4-way (18dk → ~5dk).
   - Flaky e2e/checkout-flow kök neden (race condition `await network`).
   - Mutation testing weekly run (release branch).
   - Bug bash: 2026-06-01 release öncesi.
   - UAT script discovery'den Gherkin → 14 senaryo.
8. Output: rapor + 9 action item + tooling diff.

Çıktı Formatı

# QA Plan: <service>

## Mevcut Durum
- ratio (unit/int/e2e), coverage, mutation, flaky %, CI süre, bug escape rate

## Pyramid Sağlığı
- Healthy / Ice-cream cone / Hourglass / Cupcake / Inverted

## Critical / High / Medium / Low

## Stack Tooling
- table

## Coverage + Mutation
- staged / total / critical path / mutation score

## Flaky
- liste + sahip + quarantine süresi

## Quality Gate Matrix
| Gate | Kriter | Mevcut |

## Action Items
| P | Aksiyon | Sahip | Bitiş | Issue |