Harness Workflow Audit

Harness Workflow Audit

Audit .github/workflows/*.yml for quality and hygiene: stale path filters, missing gates, permission scoping, script injection, action pinning, self-trigger loops, and ratchet calibration. Every check taxonomy entry is grounded in defect patterns observed in downstream repos, where each defect silently broke a gate and was only caught by manual review.

When to Use

• Auditing a repo's CI workflow files for correctness, completeness, and hygiene
• After a directory rename or restructure (path filters go stale silently)
• As the CI-hygiene dimension of a full codebase audit (composes into harness:audit)
• Before declaring a repo's quality gates trustworthy (e.g., during onboarding or adoption)
• When on_milestone fires as part of a release-readiness sweep
• NOT for auditing application code security (use harness-security-scan / harness-security-review)
• NOT for authoring new workflows (this skill audits; it proposes patches but does not design pipelines)
• NOT for dependency risk in actions' own supply chains beyond pinning (use harness-supply-chain-audit for package dependencies)

Iron Law

A gate that never fires is worse than no gate — it manufactures false confidence. Every check in this skill exists because a workflow that looked correct silently stopped enforcing anything. Never mark a workflow "healthy" based on its YAML looking reasonable; verify each filter, permission, and guard against the actual state of the repository.

---

Process

Phase 1: INVENTORY — Enumerate Workflows and Documented Gates

1. Resolve project root. Use the path argument or default to the current directory. If workflow was passed, restrict the audit to that single file (all phases still run).

2. List workflow files. Glob .github/workflows/*.{yml,yaml}. If none exist, report "No workflow files found" and stop.

3. Parse each workflow. For every file record: triggers (on: block with any paths: / paths-ignore: / branches: filters), jobs and their steps, permissions: blocks (workflow- and job-level), concurrency: groups, all uses: action references with their ref, and every run: script.

4. Build the documented-gate list. Collect what the repo claims to enforce:

   • package.json / pyproject.toml / Makefile scripts: build, typecheck, lint, test, coverage commands
   • Lint/typecheck configs present (eslint.config.*, tsconfig.json, .ruff.toml, etc.)
   • Claims in README.md / CONTRIBUTING.md / AGENTS.md ("CI runs X", "all PRs must pass Y")

5. Snapshot the file tree (git ls-files) for path-filter resolution in Phase 2.

6. Proceed to MECHANICAL.

---

Phase 2: MECHANICAL — Deterministic Checks

Run every check against every workflow. Record each finding as {file, line, check, severity, evidence, suggested_patch}.

Check M1: Path-filter correctness

1. For each glob in paths: / paths-ignore:, match it against the git ls-files snapshot.
2. Error: a paths: glob that matches zero tracked files — the trigger is dead and the gate never fires (the signature failure after a directory rename).
3. Warning: a workflow that gates a tool/directory but whose paths: filter misses files the job actually operates on (e.g., a lint gate filtered to src/** while the linter also covers scripts/**). Compare filter coverage against the paths the job's commands touch.
4. Warning: enforcement configs referenced by workflows (architecture rules, lint scopes) whose own internal path patterns match zero files — the job runs but governs nothing.
5. Suggested patch: the corrected glob (verify the replacement matches ≥1 tracked file before proposing it).

Check M2: Permission scoping

1. Warning: no top-level permissions: block — the workflow inherits the repo default, which is often write-all.
2. Error: permissions: write-all, or contents: write / pull-requests: write on jobs whose steps only read (no push, no comment, no release step).
3. Info: a job that pushes/comments but relies on workflow-level write when a job-level grant would scope it tighter.
4. Suggested patch: the minimal permissions: block derived from what the steps actually do.

Check M3: Action pinning

1. Error: third-party (non-actions/, non-github/) action pinned to a floating branch (@main, @master) — the action's author can change the code you execute at any time.
2. Warning: third-party action pinned to a mutable tag when the repo's own convention is SHA pinning; note SHA pinning (@<40-char-sha> # vX.Y.Z) as the strongest form.
3. Info: first-party actions/* on a floating branch.
4. Suggested patch: the pinned form with the current SHA (resolve via git ls-remote <action-repo> <ref> when network access allows; otherwise state the command for the user to run).

Check M4: Self-trigger and concurrency safety

1. Identify workflows that git push, commit, or otherwise write back to a branch.
2. Error: push-back with no re-trigger guard — require at least one of: [skip ci] in the commit message, an actor guard (if: github.actor != '<bot>'), or a paths-ignore: covering the generated file.
3. Error: push-back to a PR head branch with no existence check — the branch can be deleted mid-run (PR merged with branch auto-delete). Require a git ls-remote --exit-code --heads origin <branch> guard before the push, exiting cleanly when the branch is gone.
4. Warning: workflows that commit generated files (ledgers, baselines, reports) with per-run content (timestamps, run IDs) back to PR branches — concurrent PRs will conflict on the generated file rather than on real work. Suggest moving the artifact to a post-merge job on the default branch, or making the content deterministic.
5. Warning: no concurrency: group on workflows where overlapping runs race (deploy, push-back, cache-refresh jobs).

Check M5: Secret handling (mechanical slice)

1. Error: a run: step that echos / prints a value derived from ${{ secrets.* }}.
2. Warning: secrets passed via command-line arguments (visible in process lists / logs) instead of env:.

Check M6: Dead and stale references

1. Warning: run: steps invoking scripts/paths that do not exist in the tree; workflow_call / uses: references to local workflows or composite actions that are missing.
2. Info: references to removed identities, project boards, or branches that no longer exist.

Proceed to JUDGMENT. Do not skip Phase 3 because Phase 2 found nothing — the judgment checks catch the failures mechanical checks cannot.

---

Phase 3: JUDGMENT — Context-Dependent Checks

These require reading the workflow's intent against the repo's reality.

Check J1: Script injection (untrusted interpolation)

1. Flag ${{ github.event.* }}, ${{ github.head_ref }}, and other attacker-controlled values (PR titles, bodies, branch names, commit messages, issue comments) interpolated directly into run: scripts. Severity error — a crafted PR title becomes shell code.
2. Suggested patch: route the value through env: (env: TITLE: ${{ github.event.pull_request.title }} then "$TITLE" in the script), which makes it data instead of code.
3. Flag pull_request_target combined with a checkout of the PR head (ref: github.event.pull_request.head.sha) as error — untrusted code with secrets access.
4. Judge, don't grep: ${{ github.event.inputs.* }} on workflow_dispatch is operator-supplied and usually fine; the same pattern on issue_comment is not.

Check J2: Gate completeness

1. Diff the Phase 1 documented-gate list against what the workflow set actually runs.
2. Error: a gate documented or configured but wired to no CI step — e.g., a typecheck script in package.json for a TypeScript repo with no CI job running it, tests documented in CONTRIBUTING but never executed, coverage thresholds configured but unenforced.
3. Warning: the four-gate set (build / typecheck / lint / test) incomplete for the repo's language, with no documented reason.
4. Warning: a ratchet/baseline gate with no refresh job on the default branch — the baseline goes stale after merges, which is how gates end up disabled "temporarily" forever. If a sibling gate has a refresh job and this one does not, flag the asymmetry.
5. Suggested patch: the missing job/step, matching the repo's existing workflow style.

Check J3: Ratchet and severity calibration

1. For each "no new findings" ratchet gate, determine which severities it blocks on.
2. Warning: a ratchet that blocks PRs on info-severity findings while the error-severity gate passes — false-blocking teaches people to bypass the gate.
3. Warning: ratchet ledgers/baselines auto-committed back to PR branches (see M4.4) — calibration and conflict machinery interact; cross-reference the M4 finding rather than double-counting.
4. Suggested patch: the severity threshold change, or moving ledger writes post-merge.

Check J4: Fork-PR degradation

1. Warning: workflows that push, comment, or label on pull_request events without handling fork PRs, where GITHUB_TOKEN is read-only — the job fails or silently no-ops for outside contributors.
2. Suggested patch: an if: github.event.pull_request.head.repo.full_name == github.repository guard with a degraded read-only path, or a workflow_run split.

Proceed to REPORT.

---

Phase 4: REPORT — Ranked Findings

1. Rank findings by severity (error → warning → info), then by blast radius (a dead path filter on the security gate outranks one on a docs job).

2. Emit one entry per finding:

   [ERROR] path-filter-dead  .github/workflows/security.yml:7
     paths: 'lib/scanner/**' matches 0 tracked files (directory renamed to packages/scanner in <commit>)
     Effect: security gate has not run on any PR since the rename.
     Patch:
       -      - 'lib/scanner/**'
       +      - 'packages/scanner/**'
   

   Every entry must carry: severity, check id, file:line, evidence (what was observed and why it matters), and a concrete suggested patch (diff or exact YAML). No finding ships without a patch or an explicit "requires human decision: ".

3. Summary block:

   WORKFLOW AUDIT: <repo>
   Workflows audited: N   Findings: E error, W warning, I info
   Gates that never fire: <list or none>
   Documented-but-unwired gates: <list or none>
   

4. Filter by the severity argument if provided (default: report everything).

5. Composition note: when invoked as a dimension of harness:audit, return the findings list and summary block to the orchestrating skill instead of terminating — workflow hygiene is one axis of the full-repo audit.

---

Gates

• No "healthy" verdict without resolving every path filter. M1 must run against the real git ls-files output for every glob. A filter you did not resolve is a filter you did not audit.
• No skipping Phase 3 on a clean Phase 2. The costliest defects (documented-but-unwired gates, injection) are judgment checks. Mechanical-clean is not audit-clean.
• No finding without file:line and a suggested patch. A finding the user cannot act on in one step is not done.
• Do not modify workflow files. This skill audits and proposes patches; applying them is the user's decision. Auto-applying CI changes from an audit is how gates get broken twice.

Harness Integration

• harness skill run harness-workflow-audit — Run the audit (args: path, workflow, severity).
• Composes into harness:audit — the full-codebase audit orchestrator consumes this skill's findings as its CI-hygiene dimension.
• Complements harness-security-scan — that skill scans application code mechanically; this one audits the workflow files that decide whether that scan (and every other gate) actually runs.
• harness verify — after the user applies suggested patches, the quick gate confirms the repo's own checks still pass.

Evidence Requirements

Cite the observation behind every finding:

• Path filters: the glob, and the git ls-files match count
• Permissions: the step(s) proving write is / is not needed
• Pinning: the action ref and its owner (first- vs third-party)
• Self-trigger: the push/commit step and the absent guard, quoted
• Injection: the interpolated expression and the run: line it lands in
• Gate completeness: the config/doc claiming the gate, and the workflow set lacking it

Never assert "this gate does not run" without showing the zero-match glob or the missing step.

Success Criteria

• Every paths: / paths-ignore: glob in every workflow was resolved against the tracked file tree, and every zero-match glob is reported as an error
• Every gate documented in repo config/docs is either found wired in CI or reported as missing
• Every push-back workflow was checked for re-trigger guards, branch-existence guards, and concurrency groups
• Every finding has severity, check id, file:line, evidence, and a concrete suggested patch
• Findings are ranked, and the summary block states how many gates never fire
• No workflow file was modified

Escalation

• If a dead gate has been dead for a long time: Do not just fix the filter. Report that the gate has not run since <date/commit>, and recommend running the gated check once against the current tree before re-enabling — re-arming a long-dead gate usually surfaces a backlog of real findings.
• If YAML fails to parse: Report the file and parse error as its own error-severity finding and continue with the remaining workflows. A malformed workflow is itself a hygiene defect.
• If a permission's necessity cannot be determined (e.g., a composite action's internals are opaque): flag as "requires human decision" with both the tight and current grants — do not guess in either direction.
• If the repo intentionally runs no CI (archived, mirror, docs-only): report the inventory and stop; do not manufacture findings against a deliberate choice.
• If a finding implicates a security-sensitive workflow (pull_request_target, deploy credentials): surface it at the top of the report regardless of rank order.

Rationalizations to Reject

Rationalization	Reality
"The workflow is green on every PR, so it must be working"	A workflow whose path filter matches nothing is green because it never runs. Green means "did not fail", not "enforced something". Resolve the filters.
"The path globs look right — they match the directory names I can see"	Filters go stale exactly when directories are renamed, which is when they still look right. Only a match against git ls-files counts as verification.
"It's a private repo, so injection and pinning findings don't matter"	Private repos have contractors, compromised accounts, and dependency-of-dependency actions. Report the finding with severity intact; let the human accept the risk explicitly.
"This interpolation is fine because the value comes from our own team"	Branch names and PR titles are attacker-controlled the moment an outside contributor (or a compromised account) opens a PR. Route it through env: — the fix costs two lines.
"The push-back workflow has run for months without a loop, so it doesn't need guards"	It hasn't looped yet because timing has been kind. The branch-deleted race and the re-trigger loop are both timing-dependent; absence of incident is not presence of a guard.
"I'll report the finding without a patch — the maintainer will know what to do"	A finding without a concrete patch gets triaged to "later" and dies there. The patch is the deliverable; the finding is its justification.

Examples

Example: Auditing a TypeScript monorepo

$ harness skill run harness-workflow-audit --path .

WORKFLOW AUDIT: example-monorepo
Workflows audited: 5   Findings: 3 error, 3 warning, 1 info
Gates that never fire: security-scan.yml (dead path filter)
Documented-but-unwired gates: typecheck

[ERROR] path-filter-dead  .github/workflows/security-scan.yml:9
  paths: 'src/services/**' matches 0 tracked files (tree uses packages/*/src since the workspace migration)
  Effect: the security gate has not executed on any PR touching service code.
  Patch:
    -      - 'src/services/**'
    +      - 'packages/*/src/**'

[ERROR] gate-missing  .github/workflows/ci.yml
  package.json declares "typecheck": "tsc -b" and the repo is TypeScript, but no workflow step runs it.
  Effect: type errors land on main; the gate is assumed but absent.
  Patch: add to the ci job, after install:
    +      - name: Typecheck
    +        run: pnpm typecheck

[ERROR] injection  .github/workflows/label.yml:24
  ${{ github.event.pull_request.title }} interpolated into run:. A crafted title executes as shell.
  Patch: pass via env:
    +        env:
    +          PR_TITLE: ${{ github.event.pull_request.title }}
    -        run: echo "Title: ${{ github.event.pull_request.title }}"
    +        run: echo "Title: $PR_TITLE"

[WARNING] pushback-race  .github/workflows/ledger.yml:41
  Job pushes to the PR head branch with no existence check; branch auto-delete on merge races this run.
  Patch: guard the push:
    +      - run: git ls-remote --exit-code --heads origin "$HEAD" || { echo "branch gone, skipping"; exit 0; }

[WARNING] ratchet-calibration  .github/workflows/ledger.yml:18
  "New findings" ratchet blocks on info severity while the error gate passes — false-blocks PRs on fixture noise.
  Patch: gate on --severity error, report info findings as a non-blocking comment.

[WARNING] permissions-broad  .github/workflows/ci.yml:5
  permissions: contents: write at workflow level; no step pushes. Read suffices.
  Patch:
    -permissions:
    -  contents: write
    +permissions:
    +  contents: read

[INFO] pinning  .github/workflows/ci.yml:31
  third-party/setup-tool@main is a floating branch. Pin to a SHA:
    -        uses: third-party/setup-tool@main
    +        uses: third-party/setup-tool@4f2c3a1b… # v2.3.1

Next steps: fix the two dead/missing gates first — every other gate's meaning depends on them.