tag-based-production-deploys

Dev Deploys on Push, Prod Deploys on Tag -- No Exceptions

There is a critical difference between "code was merged" and "we decided to release." Branch-based production deploys collapse this distinction. A merge to main is a code integration event -- it means the code passed review and tests. It does not mean someone decided the system is ready for production traffic. A git tag is a deliberate, auditable act. Someone looked at the state of main, decided it was ready, and explicitly said "this is a release." That intentionality is the difference between controlled releases and accidental deployments.

Development environments should deploy continuously from branch pushes -- fast feedback, low ceremony. Production environments deploy only from git tags -- explicit, deliberate, with approval gates. This separation provides an audit trail (tags are immutable references), intentionality (creating a tag is a conscious decision), rollback clarity (deploy the previous tag), and compliance evidence (every production change has a traceable trigger).

The Environment/Trigger Matrix

Environment	Trigger	Branch/Tag	Approval Required	Purpose
Dev	Push to main	Branch	None	Fast feedback, continuous integration
Production	Git tag creation	Semver (e.g., 1.2.3)	Manual approval gate	Deliberate release with audit trail

Why This Matrix Works

Dev deploys are cheap. A broken dev environment wastes 30 minutes. The team fixes it and moves on. Speed matters more than ceremony. Push to the branch, deploy automatically, see results immediately.

Production deploys are expensive. A broken production environment wastes revenue, trust, and sleep. Ceremony matters more than speed. The extra 60 seconds to create a tag and approve a gate is trivially cheap insurance against accidental deployments.

Tag Naming Convention

Use semantic versioning. The v prefix is optional -- both v1.2.3 and 1.2.3 work with every major CI/CD platform's tag-based triggers. Pick one convention and stick with it.

Format:  <major>.<minor>.<patch>  or  v<major>.<minor>.<patch>
Example: 1.2.3

Hotfix:       1.2.4 (increment patch)

# Pre-release tags (1.2.3-rc.1) can be added when a staging environment is introduced.

Creating a Release

# Ensure you are on the correct commit
git log --oneline -5

# Create an annotated tag (includes author and message)
git tag -a 1.2.3 -m "Release 1.2.3: Add payment retry logic, fix timeout handling"

# Push the tag to trigger the production pipeline
git push origin 1.2.3

Annotated tags over lightweight tags: Annotated tags (git tag -a) store the tagger's name, email, date, and a message. Lightweight tags (git tag) are just pointers. Annotated tags provide the audit trail that compliance requires -- who released, when, and why.

The Full Pipeline Flow

The pipeline has distinct stages that separate validation from deployment. No stage is optional. The manual approval gate for production is non-negotiable.

Developer pushes code
    |
    v
[Pre-commit Hooks] -- Local, before code enters the repository
    | terraform_fmt, terraform_validate, tflint, checkov
    |
    v
[CI: Validate] -- Triggered on every push / pull request
    | Lint, test, build, terraform plan
    | Plan output posted as PR comment
    |
    v
[Merge to main] -- Code integration, not a release
    | Dev environment auto-deploys (branch trigger)
    |
    v
[Create git tag] -- Deliberate release decision
    | 1.2.3
    |
    v
[CD: Plan] -- Triggered by tag, runs terraform plan
    | Plan output stored as artifact
    |
    v
[CD: Approval Gate] -- Manual human approval
    | Reviewer inspects plan output
    | Approves or rejects
    |
    v
[CD: Apply] -- terraform apply with the approved plan
    | Resources created/modified/destroyed
    |
    v
[CD: Verify] -- Post-deployment validation
    | Health checks, smoke tests, monitoring check

Pre-Commit Hooks: The First Gate

Pre-commit hooks catch problems before code enters the repository. They are the cheapest form of validation -- failures cost seconds, not minutes.

# .pre-commit-config.yaml
repos:
  - repo: https://github.com/antonbabenko/pre-commit-terraform
    rev: v1.96.1
    hooks:
      - id: terraform_fmt
        # Enforces consistent formatting (no style debates in reviews)

      - id: terraform_validate
        # Catches syntax errors and invalid references

      - id: terraform_tflint
        args:
          - --args=--config=__GIT_WORKING_DIR__/.tflint.hcl
        # Catches provider-specific errors (invalid instance types,
        # deprecated resources, naming violations)

      - id: terraform_checkov  # Optional but recommended
        args:
          - --args=--quiet
          - --args=--compact
        # Security scanning: detects unencrypted resources, public access,
        # missing logging, over-permissive IAM policies

  - repo: https://github.com/pre-commit/pre-commit-hooks
    rev: v4.6.0
    hooks:
      - id: trailing-whitespace
      - id: end-of-file-fixer
      - id: check-merge-conflict
      - id: detect-private-key
        # Catches accidentally committed private keys

Every developer installs pre-commit hooks. It is not optional. Add pre-commit install to the repository's onboarding documentation and verify it in CI by running pre-commit run --all-files as a CI step.

CI Stage: Plan as PR Comment

When a pull request is opened, the CI pipeline runs terraform plan and posts the output as a PR comment. Reviewers see exactly what infrastructure changes the code will produce -- before it is merged.

# .github/workflows/terraform-ci.yml
name: Terraform CI
on:
  pull_request:
    paths: ["infrastructure/**"]

permissions:
  id-token: write
  contents: read
  pull-requests: write

jobs:
  plan:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::role/GithubActionsPlanRole
          aws-region: eu-west-1

      - uses: hashicorp/setup-terraform@v3

      - name: Terraform Plan
        id: plan
        run: |
          cd infrastructure/dev
          terraform init
          terraform plan -no-color -out=tfplan 2>&1 | tee plan-output.txt

      - name: Post Plan to PR
        uses: actions/github-script@v7
        with:
          script: |
            const fs = require('fs');
            const plan = fs.readFileSync('infrastructure/dev/plan-output.txt', 'utf8');
            const body = `#### Terraform Plan
            \`\`\`
            ${plan.substring(0, 60000)}
            \`\`\`
            *Triggered by commit: \`${context.sha.substring(0, 8)}\`*`;
            github.rest.issues.createComment({
              issue_number: context.issue.number,
              owner: context.repo.owner,
              repo: context.repo.repo,
              body: body
            });

CD Stage: Tag-Triggered with Approval

The production deployment pipeline triggers only on git tag creation. It includes a mandatory manual approval gate that blocks terraform apply until a human reviews and approves the plan.

# .github/workflows/terraform-cd-prod.yml
name: Terraform CD (Production)
on:
  push:
    tags: ["[0-9]*"]  # Matches semver tags (1.2.3); adjust to "v*" if using v-prefix

permissions:
  id-token: write
  contents: read

jobs:
  plan:
    runs-on: ubuntu-latest
    environment: production-plan
    steps:
      - uses: actions/checkout@v4

      - uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::role/GithubActionsDeployRole
          aws-region: eu-west-1

      - uses: hashicorp/setup-terraform@v3

      - name: Terraform Plan
        run: |
          cd infrastructure/prod
          terraform init
          terraform plan -no-color -out=tfplan

      - name: Upload Plan
        uses: actions/upload-artifact@v4
        with:
          name: tfplan
          path: infrastructure/prod/tfplan

  apply:
    needs: plan
    runs-on: ubuntu-latest
    environment: production  # Requires manual approval in GitHub settings
    steps:
      - uses: actions/checkout@v4

      - uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::role/GithubActionsDeployRole
          aws-region: eu-west-1

      - uses: hashicorp/setup-terraform@v3

      - name: Download Plan
        uses: actions/download-artifact@v4
        with:
          name: tfplan
          path: infrastructure/prod/

      - name: Terraform Apply
        run: |
          cd infrastructure/prod
          terraform init
          terraform apply tfplan

  verify:
    needs: apply
    runs-on: ubuntu-latest
    steps:
      - name: Health Check
        run: |
          for i in $(seq 1 10); do
            STATUS=$(curl -s -o /dev/null -w "%{http_code}" https://api.myapp.com/health)
            if [ "$STATUS" = "200" ]; then
              echo "Health check passed"
              exit 0
            fi
            echo "Attempt $i: status $STATUS, retrying..."
            sleep 10
          done
          echo "Health check failed after 10 attempts"
          exit 1

The Manual Approval Gate Is Non-Negotiable

For production terraform apply, a human must review the plan output and explicitly approve. This is not about distrust of automation -- it is about catching the plan that says "3 resources to destroy" when you expected "1 resource to modify." Automated pipelines are excellent at doing exactly what they are told. A human reviewer catches when what-they-are-told is wrong.

Configure the approval gate in your CI/CD platform's environment protection rules:

• Required reviewers: At least 1 (recommend 2 for infrastructure changes)
• Wait timer: Optional, useful for scheduled maintenance windows
• Branch restrictions: Only allow deployments from protected branches or tags

Good vs. Bad Deployment Patterns

BAD: Branch-based production deploys
- Merge to main auto-deploys to production
- Accidental merges cause accidental deployments
- No distinction between "code ready" and "release ready"
- Rollback means reverting commits and re-merging
- No audit trail of who decided to release

GOOD: Tag-based production deploys
- Merge to main deploys to dev only
- Production requires explicit git tag creation
- Clear distinction: merge = integration, tag = release
- Rollback means deploying the previous tag (instant)
- Full audit trail: who tagged, when, why (annotated tag message)

BAD: No approval gate for terraform apply
- CI/CD auto-applies infrastructure changes to production
- A plan that destroys a database executes without human review
- "We didn't mean to delete that" becomes a 4am incident

GOOD: Mandatory approval gate
- Plan output visible to reviewer before apply
- Reviewer verifies resource counts (create, modify, destroy)
- Unexpected destroys are caught before execution
- Apply only proceeds after explicit human approval

Cloud Provider Translation

Concept	AWS	GCP	Azure
Approval gate	GitHub Environments (protection rules)	Cloud Build Approval	Azure DevOps Approvals and Checks
Tag-based trigger	GitHub Actions on: push: tags	Cloud Build Trigger (tag)	Azure Pipelines trigger: tags
Plan artifact storage	GitHub Actions Artifacts	Cloud Storage	Azure Artifacts
Deployment environment	GitHub Environment	Cloud Deploy Target	Azure DevOps Environment
State locking	S3 native locking (use_lockfile)	GCS (native locking)	Azure Blob lease
OIDC for pipeline auth	STS AssumeRoleWithWebIdentity	Workload Identity Federation	Federated Credentials

Examples

Working implementations in examples/:

• examples/full-pipeline-workflow.md -- Complete GitHub Actions workflow set: CI validation on PR, dev auto-deploy on merge, production tag-triggered deploy with approval gate and post-deploy verification
• examples/pre-commit-config.md -- Pre-commit configuration with terraform_fmt, terraform_validate, tflint, checkov, and private key detection

Review Checklist

When designing or reviewing deployment pipelines:

• Development environments deploy automatically from branch pushes (fast feedback)
• Production environments deploy only from git tag creation (explicit release)
• Git tags use semantic versioning (1.2.3 or v1.2.3 -- pick one convention)
• Tags are annotated (git tag -a) with a message describing the release
• A manual approval gate blocks terraform apply in production
• The approval gate requires at least one reviewer (two recommended for infrastructure)
• terraform plan output is posted as a PR comment for code review
• terraform plan output is stored as an artifact and used by the apply step (plan and apply use the same plan file)
• Pre-commit hooks are configured: terraform_fmt, terraform_validate, tflint (checkov optional but recommended)
• Pre-commit hooks are verified in CI (pre-commit run --all-files)
• Post-deployment verification (health checks, smoke tests) runs after every production deploy
• Rollback procedure is documented: deploy the previous tag
• No branch push ever triggers a production terraform apply
• CI/CD authenticates via OIDC federation, not stored credentials (see zero-static-credentials skill)