skill-security-auditor

Skill Security Auditor

Static analysis for AI agent skills. Run it on any skill from outside your own trust boundary before letting an agent execute its scripts. The auditor walks the entire skill directory and emits findings across ten threat categories, then collapses them into a single verdict that CI can gate on.

This skill is paranoid by design. It assumes a skill author may be trying to compromise the host system, exfiltrate credentials, persist beyond the session, or inject hostile instructions into the model. Most skills will trip a couple of LOW or MEDIUM findings — that's expected. The signal worth acting on is CRITICAL and HIGH.

When to use

Trigger this skill when the user:

• Is about to install a third-party skill, plugin, or agent extension.
• Asks "is this skill safe?", "audit it", "scan it", "is there anything sketchy in here?", or similar pre-install review questions.
• Wants a security gate on a skill marketplace, internal skill registry, or CI pipeline that ingests user-submitted skills.
• Just cloned a skill repo and wants a one-shot review before running it.

Do not use this skill when the user is auditing their own first-party code — there are better static-analysis tools (Semgrep, Bandit, CodeQL) for that. This skill specializes in the specific attack surface of AI agent skills: SKILL.md prompt injection, malicious tool helpers, supply-chain hooks in dependency manifests, and persistence mechanisms that survive the agent session.

Quick start

# Audit a local directory
python3 scripts/audit.py /path/to/skill

# Audit a git repo (cloned to a temp dir, optionally cleaned up after)
python3 scripts/audit.py https://github.com/example/some-skill.git --cleanup

# Strict mode — treat HIGH findings as blocking (recommended for CI)
python3 scripts/audit.py ./skill --strict

# Machine-readable output for CI
python3 scripts/audit.py ./skill --json --output report.json

# Markdown report suitable for pasting into a PR comment
python3 scripts/audit.py ./skill --markdown --output review.md

# Audit a sub-skill inside a repo containing several
python3 scripts/audit.py ./repo --skill skills/my-skill

Exit codes: 0 PASS · 1 FAIL · 2 WARN · 3 usage/IO error.

What gets scanned

Every file under the skill root is classified and routed to the appropriate scanner. The categories below summarize what each scanner looks for; references/threat-model.md covers the why in depth.

Category	Severity span	Examples
Code execution	HIGH–CRITICAL	eval, exec, os.system, subprocess(shell=True), dynamic getattr
Network exfil	MEDIUM–CRITICAL	hardcoded IPs, webhook sinks (webhook.site, ngrok, interact.sh), requests.post to runtime URLs
Credential harvest	HIGH–CRITICAL	reads from ~/.ssh, ~/.aws, ~/.gnupg, sensitive env vars, browser cookie DBs
Persistence	HIGH–CRITICAL	cron, systemd, launchctl, shell rc files, git hooks, authorized_keys, registry Run keys
Prompt injection	MEDIUM–CRITICAL	"ignore previous instructions", role markers, <|im_start|>system, hidden HTML comments
Supply chain	MEDIUM–CRITICAL	unpinned deps, typosquats (Levenshtein 1–2 from popular packages), npm postinstall hooks
Obfuscation	MEDIUM–HIGH	base64-decoded code, chr(...) chains, hex-escape blobs, Bidi / Trojan Source
Filesystem	LOW–CRITICAL	binaries, symlinks escaping the skill, SUID bits, writes to /etc, rm -rf patterns
Secrets	HIGH–CRITICAL	AWS keys, GitHub tokens, OpenAI/Anthropic keys, private key headers, JWTs
CI workflow	HIGH–CRITICAL	unescaped ${{ github.event.* }} in run:, pull_request_target with checkout

The scanners run in this order: structure → filesystem → supply-chain → workflows → prompts → code. A malformed skill surfaces the structural issue first instead of drowning the reviewer in cascade findings.

Verdict criteria

The verdict is a three-state collapse of the findings:

• PASS — no CRITICAL, no HIGH, fewer than 5 MEDIUM. LOW findings are informational and do not affect the verdict.
• WARN — at least one HIGH (in default mode), or 5+ MEDIUM. Means: a human should look at this before installing, but nothing is definitely malicious.
• FAIL — at least one CRITICAL, or any HIGH in --strict mode. Means: do not install. Either there's a clear malicious pattern, or there's something that needs explanation from the skill author before it can be trusted.

For automated gates, run with --strict. The default (non-strict) mode is for interactive review where a human is in the loop.

Reading a finding

Each finding has the same shape:

🔴 CRITICAL  (3 findings)
────────────────────────────────────────────────────────────────────────
scripts/install.py
  scripts/install.py:42  [CODE-EXEC]
    │ os.system(base64.b64decode(payload).decode())
    Risk: Decodes and executes a base64-encoded payload at runtime
    Fix:  Remove. Skills must not execute arbitrary decoded strings.

• The category tag (CODE-EXEC) is stable across runs — use it to build baseline rules or filter in CI.
• The snippet is the trimmed source line (max 140 chars). Long lines are truncated with ....
• The risk and fix fields are one sentence each, deliberately short so a reviewer can triage 20 findings in under a minute.

Suppression

Two suppression mechanisms exist, intentionally orthogonal:

1. Line-level (# noqa: SEC-AUDITOR) — when a legitimate skill genuinely needs a flagged pattern. Add the marker as a trailing comment on the line:

import pickle  # noqa: SEC-AUDITOR — internal-only cache, never deserialized from network

Equivalent markers auditor:ignore-line and audit-skip also work. Lines with any of these markers are skipped during scanning.

2. Baseline file (fingerprint suppression) — for accepting findings without modifying source. Each finding has a stable 16-char fingerprint derived from sha256(file + snippet + pattern + category) — line numbers don't matter, so the fingerprint survives reformatting and reordering.

Get fingerprints from a JSON run:

python3 scripts/audit.py ./skill --json | jq '.findings[] | {fingerprint, category, file}'

Then commit a baseline.yml:

suppressions:
  - fingerprint: a1b2c3d4e5f60718
    reason: reviewed 2025-04-30, internal HTTP call to corporate API
  - fingerprint: 0123456789abcdef
    reason: false positive in vendored library, see ticket SEC-441

And run with --baseline baseline.yml. See assets/baseline.example.yml for the full schema.

CI integration

GitHub Actions, gated on CRITICAL findings, opening a PR comment on WARN:

name: Skill security audit
on: [pull_request]
jobs:
  audit:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: { python-version: '3.11' }
      - name: Clone auditor
        run: git clone https://github.com/your-org/skill-security-auditor.git /tmp/auditor
      - name: Run audit
        id: audit
        run: |
          python3 /tmp/auditor/scripts/audit.py . \
            --strict \
            --markdown \
            --output audit-report.md
        continue-on-error: true
      - name: Post review comment
        if: always()
        uses: actions/github-script@v7
        with:
          script: |
            const fs = require('fs');
            const body = fs.readFileSync('audit-report.md', 'utf8');
            github.rest.issues.createComment({
              issue_number: context.issue.number,
              owner: context.repo.owner,
              repo: context.repo.repo,
              body
            });
      - name: Fail on critical
        if: steps.audit.outcome == 'failure'
        run: exit 1

For pre-commit, just call the auditor directly:

# .pre-commit-config.yaml
repos:
  - repo: local
    hooks:
      - id: skill-audit
        name: skill-security-auditor
        entry: python3 scripts/audit.py
        args: [--strict, --quiet]
        language: system
        pass_filenames: false
        always_run: true

References

Detailed reading lives in the references/ directory:

• references/threat-model.md — Full taxonomy of attacks a malicious skill can mount, with concrete examples per category. Read this if you're writing a new pattern or evaluating whether the auditor covers a given attack scenario.
• references/pattern-catalog.md — Complete list of every pattern the auditor recognizes, grouped by category, with example malicious source and the regex that catches it.
• references/remediation-guide.md — How to fix each finding category. Linked from the fix field when guidance doesn't fit in one line.

Limitations

Stated honestly because security tools that pretend to be exhaustive are worse than ones that admit gaps:

• No taint tracking. The auditor sees requests.post(url, data=secret) and flags it, but it doesn't trace whether url came from a trusted config. False positives are accepted as the cost of catching real exfiltration.
• No dynamic analysis. A skill that downloads its payload at runtime from a CDN with no suspicious-looking hardcoded URL won't be caught here. Pair this with sandboxed execution and network monitoring.
• Pattern arms race. Obfuscation techniques evolve; a determined attacker will find ways around any static check. CRITICAL findings are reliable; HIGH and MEDIUM are pattern matches that benefit from human eyes.
• Python AST coverage only. JS/TS/shell get regex coverage but no AST pass, so aliased-import attacks in those languages are weaker checks.
• No reputation data. This is a pure static analyzer — no calls to package registries or threat-intel feeds. Combine with pip-audit / npm audit for known-CVE coverage.

Why this exists

Skills run with the agent's full execution capability. A compromised skill can read your credentials, exfiltrate your data, persist on your system, or inject hostile instructions into the model that survive across conversations. Software that ingests untrusted code without static review is software that gets owned. This skill is the static review.