Skaffold Testing

Testing Lifecycle

Build → Test → Deploy → Verify
         ↑               ↑
    Pre-deploy      Post-deploy

Stage	Purpose	Runs During
test	Validate images before deployment	dev, run, test
verify	Validate deployment works correctly	dev, run, verify

Failed tests block deployment. Use --skip-tests to bypass.

Test Stage Overview

Two mechanisms for pre-deploy validation:

Type	Purpose	Tool Required
structureTests	Validate image contents	container-structure-test binary
custom	Run arbitrary commands	None (uses $IMAGE env var)

Container Structure Tests

Validate image contents without running the container.

Configuration

apiVersion: skaffold/v4beta11
kind: Config
test:
  - image: my-app
    structureTests:
      - ./tests/structure/*.yaml
    structureTestsArgs:
      - --driver=tar      # Faster, no Docker daemon needed
      - -q                # Quiet output

Test Types

1. Command Tests

Verify binaries work and produce expected output:

schemaVersion: '2.0.0'
commandTests:
  - name: "Python version"
    command: "python"
    args: ["--version"]
    expectedOutput: ["Python 3.12"]
    exitCode: 0

  - name: "App starts without error"
    command: "/app/bin/server"
    args: ["--help"]
    exitCode: 0

  - name: "No root shell access"
    command: "sh"
    args: ["-c", "whoami"]
    excludedOutput: ["root"]

Multi-line commands:

commandTests:
  - name: "Health check passes"
    command: "bash"
    args:
      - -c
      - |
        /app/bin/server &
        sleep 2
        curl -f http://localhost:8080/health

2. File Existence Tests

Verify files present with correct permissions:

fileExistenceTests:
  # Required files exist
  - name: "Config file present"
    path: /app/config.yaml
    shouldExist: true
    permissions: "-rw-r--r--"

  # Security: Sensitive files removed
  - name: "No .env file shipped"
    path: /app/.env
    shouldExist: false

  - name: "No git history shipped"
    path: /app/.git
    shouldExist: false

  # Correct ownership (non-root)
  - name: "App owned by appuser"
    path: /app
    shouldExist: true
    uid: 1000
    gid: 1000

3. File Content Tests

Validate file contents with regex:

fileContentTests:
  - name: "Logging configured correctly"
    path: /app/config.yaml
    expectedContents:
      - "level: info"
      - "format: json"
    excludedContents:
      - "level: debug"    # No debug in prod images
      - "password:"       # No hardcoded secrets

  - name: "Terraform checkpoint disabled"
    path: /root/.terraformrc
    expectedContents:
      - "disable_checkpoint = true"

4. Metadata Test

Validate image configuration:

metadataTest:
  # Environment variables set
  envVars:
    - key: NODE_ENV
      value: production
    - key: TZ
      value: UTC

  # Security: Non-root user
  user: appuser

  # Correct entrypoint
  entrypoint: ["/app/bin/server"]
  cmd: ["--config", "/app/config.yaml"]

  # Expected ports exposed
  exposedPorts: ["8080", "9090"]

  # Working directory set
  workdir: /app

  # Labels present
  labels:
    - key: org.opencontainers.image.source
      value: "https://github.com/.*"
      isRegex: true

Custom Tests

Run arbitrary commands with access to built image.

Security Scanning

test:
  - image: my-app
    custom:
      # Vulnerability scanning with Grype
      - command: grype $IMAGE --fail-on high --only-fixed
        timeoutSeconds: 300

      # Alternative: Trivy
      - command: trivy image --exit-code 1 --severity HIGH,CRITICAL $IMAGE
        timeoutSeconds: 300

      # SBOM generation (doesn't fail, just generates)
      - command: syft $IMAGE -o spdx-json > sbom.json
        timeoutSeconds: 120

Unit Tests Against Image

test:
  - image: my-app
    custom:
      - command: docker run --rm $IMAGE npm test
        timeoutSeconds: 300
        dependencies:
          paths:
            - "src/**/*.ts"
            - "test/**/*.ts"
            - "package.json"

Linting and Validation

test:
  - image: my-app
    custom:
      # Dockerfile linting
      - command: hadolint Dockerfile
        dependencies:
          paths:
            - "Dockerfile"

      # Kubernetes manifest validation
      - command: kubeval k8s/*.yaml
        dependencies:
          paths:
            - "k8s/*.yaml"

Dependencies Configuration

Control when tests re-run:

custom:
  - command: ./scripts/integration-test.sh
    timeoutSeconds: 600
    dependencies:
      # Static paths - re-run when these change
      paths:
        - "src/**/*.go"
        - "go.mod"
        - "go.sum"
      # Ignore patterns
      ignore:
        - "**/*_test.go"

  # Dynamic dependencies from command
  - command: ./scripts/e2e-test.sh
    dependencies:
      command: echo '["test/e2e/**/*.ts"]'

Verify Stage (Post-Deploy)

Run integration tests after deployment succeeds.

Execution Modes

Mode	Environment	Use Case
local (default)	Docker on host	Quick tests, local dev
kubernetesCluster	K8s Job	Integration tests needing cluster access

Basic Configuration

apiVersion: skaffold/v4beta11
kind: Config
verify:
  # Health check with external image
  - name: health-check
    container:
      name: curl-test
      image: curlimages/curl:latest
      command: ["/bin/sh"]
      args: ["-c", "curl -f http://my-app.default.svc:8080/health"]
    executionMode:
      kubernetesCluster: {}

  # Integration tests with built image
  - name: integration-tests
    container:
      name: integration-tests
      image: my-app-tests  # Built by Skaffold
      command: ["npm", "run", "test:integration"]
    executionMode:
      kubernetesCluster: {}

Kubernetes Job Customization

verify:
  - name: integration-tests
    container:
      name: tests
      image: my-app-tests
    executionMode:
      kubernetesCluster:
        # Inline overrides (kubectl run --overrides style)
        overrides: |
          {
            "spec": {
              "serviceAccountName": "test-runner",
              "activeDeadlineSeconds": 600
            }
          }

  - name: e2e-tests
    container:
      name: e2e
      image: my-e2e-tests
    executionMode:
      kubernetesCluster:
        # Use custom Job manifest
        jobManifestPath: ./k8s/e2e-job.yaml

E2E Job Manifest Example

# k8s/e2e-job.yaml
apiVersion: batch/v1
kind: Job
metadata:
  name: e2e-tests
spec:
  backoffLimit: 0
  activeDeadlineSeconds: 900
  template:
    spec:
      restartPolicy: Never
      serviceAccountName: e2e-runner
      containers: []  # Skaffold replaces this
      volumes:
        - name: test-config
          configMap:
            name: e2e-config

Profile-Based Testing

Quick vs Thorough

test:
  - image: my-app
    structureTests:
      - ./tests/structure/security.yaml
      - ./tests/structure/config.yaml
    custom:
      - command: grype $IMAGE --fail-on critical
        timeoutSeconds: 120

profiles:
  # Fast tests for dev loop
  - name: quick
    test:
      - image: my-app
        structureTests:
          - ./tests/structure/security.yaml  # Essential only
        # No vulnerability scan - too slow

  # Thorough tests for CI
  - name: ci
    test:
      - image: my-app
        structureTests:
          - ./tests/structure/*.yaml
        custom:
          - command: grype $IMAGE --fail-on high --only-fixed
            timeoutSeconds: 600
          - command: trivy image --scanners vuln,secret $IMAGE
            timeoutSeconds: 300
    verify:
      - name: full-integration
        container:
          name: integration
          image: my-app-tests
        executionMode:
          kubernetesCluster: {}

Essential Security Tests

Every production image should validate:

Security Structure Test Template

# tests/structure/security.yaml
schemaVersion: '2.0.0'

# 1. Non-root user
metadataTest:
  user: "appuser"  # or numeric UID like "1000"

# 2. No sensitive files shipped
fileExistenceTests:
  - name: "No .env file"
    path: /app/.env
    shouldExist: false
  - name: "No .git directory"
    path: /app/.git
    shouldExist: false
  - name: "No private keys"
    path: /app/id_rsa
    shouldExist: false
  - name: "No credentials files"
    path: /app/credentials.json
    shouldExist: false

# 3. No secrets in config files
fileContentTests:
  - name: "No hardcoded passwords"
    path: /app/config.yaml
    excludedContents:
      - "password:"
      - "secret:"
      - "api_key:"
      - "BEGIN RSA PRIVATE KEY"
      - "BEGIN OPENSSH PRIVATE KEY"

# 4. Correct file permissions
fileExistenceTests:
  - name: "Config not world-writable"
    path: /app/config.yaml
    shouldExist: true
    permissions: "-rw-r--r--"

# 5. Shell access removed (distroless)
commandTests:
  - name: "No shell available"
    command: "/bin/sh"
    exitCode: 127  # Command not found

Directory Structure

project/
├── skaffold.yaml
├── Dockerfile
├── tests/
│   ├── structure/
│   │   ├── security.yaml      # Security validations
│   │   ├── config.yaml        # Configuration checks
│   │   └── runtime.yaml       # Runtime requirements
│   └── integration/
│       └── run.sh             # Integration test script
└── k8s/
    ├── deployment.yaml
    └── e2e-job.yaml           # Verify stage job manifest

Agentic Optimizations

Context	Command
Quick structure test	container-structure-test test --driver=tar -q --image $IMAGE --config tests/structure/security.yaml
Security scan (critical only)	grype $IMAGE --fail-on critical -q
Skip tests in dev	skaffold dev --skip-tests
Run only tests	skaffold test
Run only verify	skaffold verify
CI with JUnit output	container-structure-test test --image $IMAGE --config test.yaml --test-report junit.xml

Quick Reference

Container Structure Test Flags

Flag	Description
--driver=tar	Use tar driver (faster, no Docker daemon)
--driver=docker	Use Docker driver (default)
-q	Quiet output
--test-report FILE	Generate test report
--output json	JSON output format

Skaffold Test Flags

Flag	Description
--skip-tests	Skip test phase
-p PROFILE	Use specific profile
--build-artifacts FILE	Use pre-built artifacts

Custom Test Environment

Variable	Description
$IMAGE	Built image with tag/digest

Common Patterns

Fail Fast in CI

test:
  - image: my-app
    custom:
      # Security gate first - fastest to fail
      - command: grype $IMAGE --fail-on critical -q
        timeoutSeconds: 60
    structureTests:
      # Then structure tests
      - ./tests/structure/security.yaml

Multi-Architecture Testing

test:
  - image: my-app-amd64
    structureTests:
      - ./tests/structure/*.yaml
  - image: my-app-arm64
    structureTests:
      - ./tests/structure/*.yaml

Test Image Separately from App

build:
  artifacts:
    - image: my-app
    - image: my-app-tests
      docker:
        dockerfile: Dockerfile.test

test:
  - image: my-app
    structureTests:
      - ./tests/structure/*.yaml

verify:
  - name: integration
    container:
      name: tests
      image: my-app-tests
    executionMode:
      kubernetesCluster: {}

Troubleshooting

container-structure-test not found

# Install on macOS
brew install container-structure-test

# Install on Linux
curl -LO https://storage.googleapis.com/container-structure-test/latest/container-structure-test-linux-amd64
chmod +x container-structure-test-linux-amd64
sudo mv container-structure-test-linux-amd64 /usr/local/bin/container-structure-test

Tests Pass Locally, Fail in CI

Check:

1. Docker daemon running in CI
2. Use --driver=tar if no daemon available
3. Image exists (not just built, but accessible)

Verify Tests Can't Reach Services

In Kubernetes mode:

1. Verify test pod can resolve service DNS
2. Check NetworkPolicies allow test pod traffic
3. Ensure services are ready before tests run (use statusCheck: true)