pipeline-patterns

Pipeline Patterns

Design robust CI/CD pipelines with clear stage separation, proper artifact flow, environment promotion, quality gates, and failure handling.

When to Use

• User designs a new CI/CD pipeline
• User asks about build, test, deploy stage organization
• User needs environment promotion (dev -> staging -> prod)
• User wants to optimize pipeline speed with parallelism
• User asks about quality gates or approval workflows

Core Patterns

Pipeline Stage Architecture

Structure pipelines in clear stages with explicit dependencies and artifact passing.

Build --> Lint --|
                |--> Integration Tests --> Deploy Staging --> E2E Tests --> Deploy Prod
         Unit --|

# GitHub Actions implementation
jobs:
  build:
    runs-on: ubuntu-latest
    outputs:
      image-tag: ${{ steps.meta.outputs.tags }}
    steps:
      - uses: actions/checkout@v4
      - id: meta
        uses: docker/metadata-action@v5
        with:
          images: ghcr.io/${{ github.repository }}
          tags: type=sha
      - uses: docker/build-push-action@v5
        with:
          push: true
          tags: ${{ steps.meta.outputs.tags }}

  lint:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: npm ci && npm run lint

  unit-test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: npm ci && npm test

  integration-test:
    needs: [build, lint, unit-test]
    runs-on: ubuntu-latest
    services:
      postgres:
        image: postgres:16
        env: { POSTGRES_DB: test, POSTGRES_PASSWORD: test }
        options: --health-cmd pg_isready --health-interval 5s --health-retries 5
    steps:
      - uses: actions/checkout@v4
      - run: npm ci && npm run test:integration

  deploy-staging:
    needs: integration-test
    runs-on: ubuntu-latest
    environment: staging
    steps:
      - run: echo "Deploy ${{ needs.build.outputs.image-tag }} to staging"

  e2e-test:
    needs: deploy-staging
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: npx playwright test --project=staging

  deploy-production:
    needs: e2e-test
    runs-on: ubuntu-latest
    environment:
      name: production
      url: https://myapp.com
    steps:
      - run: echo "Deploy ${{ needs.build.outputs.image-tag }} to production"

Artifact Management

Build once, deploy everywhere. Pass artifacts between stages to ensure consistency.

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: npm ci && npm run build
      - uses: actions/upload-artifact@v4
        with:
          name: app-build-${{ github.sha }}
          path: dist/
          retention-days: 7

  deploy:
    needs: build
    runs-on: ubuntu-latest
    steps:
      - uses: actions/download-artifact@v4
        with:
          name: app-build-${{ github.sha }}
          path: dist/
      - run: ./scripts/deploy.sh dist/

Quality Gates

Enforce quality standards before promotion to the next environment.

jobs:
  quality-gate:
    needs: [unit-test, lint, security-scan]
    runs-on: ubuntu-latest
    steps:
      - name: Check test coverage
        run: |
          COVERAGE=$(cat coverage/coverage-summary.json | jq '.total.lines.pct')
          if (( $(echo "$COVERAGE < 80" | bc -l) )); then
            echo "Coverage $COVERAGE% is below 80% threshold"
            exit 1
          fi

      - name: Check lint results
        run: |
          if [ "${{ needs.lint.result }}" != "success" ]; then
            echo "Lint check failed"
            exit 1
          fi

      - name: Check security scan
        run: |
          if [ "${{ needs.security-scan.result }}" != "success" ]; then
            echo "Security vulnerabilities found"
            exit 1
          fi

Environment Promotion with Approvals

Use GitHub environments to enforce manual approvals for production deployments.

jobs:
  deploy-staging:
    environment: staging
    runs-on: ubuntu-latest
    steps:
      - run: deploy --env staging

  smoke-test:
    needs: deploy-staging
    runs-on: ubuntu-latest
    steps:
      - run: curl -f https://staging.myapp.com/health

  deploy-production:
    needs: smoke-test
    environment:
      name: production         # Requires manual approval in GitHub settings
      url: https://myapp.com
    runs-on: ubuntu-latest
    steps:
      - run: deploy --env production

Parallel Test Splitting

Split large test suites across parallel runners to reduce pipeline duration.

jobs:
  test:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        shard: [1, 2, 3, 4]
    steps:
      - uses: actions/checkout@v4
      - run: npm ci
      - run: npx jest --shard=${{ matrix.shard }}/4

  merge-coverage:
    needs: test
    runs-on: ubuntu-latest
    steps:
      - run: echo "Merge coverage from all shards"

Anti-Patterns

• Build per environment: Build the artifact once and promote it. Rebuilding per environment risks deploying different code to staging and production.
• No quality gates: Deploying without coverage, lint, or security checks allows regressions to reach production.
• Sequential everything: Independent jobs like lint, unit tests, and security scans should run in parallel. Only use needs for true dependencies.
• Skipping staging: Always deploy to a staging environment before production. E2E tests against staging catch integration issues that unit tests miss.
• No rollback plan: Every deploy pipeline must have a documented and tested rollback path. If the pipeline cannot roll back, it is not production-ready.
• Secrets in logs: Never echo secrets. Use add-mask to redact sensitive values in output.

Quick Reference

Pipeline Design Checklist:
[ ] Build once, deploy many (single artifact)
[ ] Parallel independent jobs (lint, test, scan)
[ ] Quality gates before promotion
[ ] Environment-specific configs (not rebuilt code)
[ ] Manual approval for production
[ ] Rollback procedure documented and tested
[ ] Notifications on failure
[ ] Artifact retention policy defined
[ ] Concurrency controls to prevent duplicate runs
[ ] Timeout limits on all jobs