Terraform Operations - Multi-Cloud

Overview

All HOLE Foundation infrastructure is managed via Terraform with state stored in Terraform Cloud.

Terraform Root: /Volumes/HOLE-RAID-DRIVE/Projects/hole-terraformer/terraform/ Terraform Cloud Org: theholetruth Backend: Remote (Terraform Cloud) Providers: Cloudflare, Azure, AWS

---

Workspace Organization

Directory Structure

/Volumes/HOLE-RAID-DRIVE/Projects/hole-terraformer/terraform/
├── cloudflare/
│   ├── foundation/          # Foundation account (387 resources) ✅
│   │   ├── main.tf
│   │   ├── backend.tf
│   │   ├── tf-with-dotenvx.sh ⭐
│   │   └── [24 .tf files]
│   └── personal/            # Personal account (~20 resources) ⏳
│       ├── main.tf
│       ├── backend.tf
│       └── [3 .tf files]
│
├── production/              # Azure (alias: azure/)
│   ├── main.tf              # Entra DS infrastructure
│   ├── backend.tf
│   └── outputs.tf
│
├── aws/
│   └── foundation/          # AWS Bedrock (~5 resources)
│       ├── main.tf
│       ├── backend.tf
│       ├── bedrock.tf
│       └── outputs.tf
│
├── dev/                     # Development workspace (all providers)
└── shared/                  # Shared modules (future)

Terraform Cloud Workspaces

Workspace	Provider	Local Path	State
cloudflare-foundation-prod	Cloudflare	cloudflare/foundation/	387 resources
cloudflare-personal-prod	Cloudflare	cloudflare/personal/	~20 resources
azure-hole-general-services-prod	Azure	production/	~10 resources
aws-foundation-prod	AWS	aws/foundation/	~5 resources

Organization: theholetruth Total Resources: ~422 across 4 workspaces

---

Provider-Specific Workflows

Cloudflare Workflow

Location: terraform/cloudflare/foundation/

Secrets: Auto-loaded via wrapper script

Commands:

cd /Volumes/HOLE-RAID-DRIVE/Projects/hole-terraformer/terraform/cloudflare/foundation

# Use wrapper script (auto-loads CLOUDFLARE_API_TOKEN from dotenvx)
./tf-with-dotenvx.sh plan
./tf-with-dotenvx.sh apply
./tf-with-dotenvx.sh state list
./tf-with-dotenvx.sh import <resource> <id>

Why wrapper? Automatically loads CF_API_TOKEN and TF_VAR_cloudflare_api_token from dotenvx.

Azure Workflow

Location: terraform/production/

Authentication: Azure CLI (no secrets needed)

Commands:

cd /Volumes/HOLE-RAID-DRIVE/Projects/hole-terraformer/terraform/production

# Verify Azure CLI authentication
az account show

# Standard Terraform commands
terraform plan
terraform apply
terraform state list

Why no wrapper? Azure uses Azure CLI authentication (joe@theholetruth.org already logged in).

AWS Workflow

Location: terraform/aws/foundation/

Secrets: Manual loading from dotenvx

Commands:

cd /Volumes/HOLE-RAID-DRIVE/Projects/hole-terraformer/terraform/aws/foundation

# Load AWS credentials from dotenvx
export AWS_ACCESS_KEY_ID=$(cd /Volumes/HOLE-RAID-DRIVE/dotenvx && dotenvx get AWS_ACCESS_KEY_ID)
export AWS_SECRET_ACCESS_KEY=$(cd /Volumes/HOLE-RAID-DRIVE/dotenvx && dotenvx get AWS_SECRET_ACCESS_KEY)
export TF_VAR_aws_access_key="$AWS_ACCESS_KEY_ID"
export TF_VAR_aws_secret_key="$AWS_SECRET_ACCESS_KEY"

# Standard Terraform commands
terraform plan
terraform apply
terraform state list

---

Standard Terraform Workflow (All Providers)

1. Edit Configuration

Make changes to .tf files:

• Add new resources
• Modify existing resources
• Delete resources

2. Format and Validate

terraform fmt -recursive    # Format all .tf files
terraform validate          # Check syntax

3. Plan Changes

terraform plan

What plan shows:

• + Resources to create
• ~ Resources to modify
• - Resources to delete
• → Resources to replace

4. Show User and Get Approval

ALWAYS:

• Show the complete plan output
• Explain what will change
• Get explicit approval

5. Apply Changes

terraform apply

With auto-approve (use caution):

terraform apply -auto-approve

6. Verify Changes

Provider-specific verification:

• Cloudflare: Check dashboard or use API
• Azure: Use az CLI to verify
• AWS: Use aws CLI to verify

---

Common Terraform Commands

View State

# List all resources
terraform state list

# Show specific resource
terraform state show <resource-address>

# Pull state file (backup)
terraform state pull > state-backup.json

Import Resources

# Import existing resource into Terraform
terraform import <resource-type>.<name> <resource-id>

# Example: Import Cloudflare zone
terraform import cloudflare_zone.example 1a2b3c4d5e6f7g8h9i0j

Remove Resources

# Remove from state only (doesn't delete in provider)
terraform state rm <resource-address>

# Destroy resource (deletes in provider)
terraform destroy -target=<resource-address>

Workspace Management

# Show current workspace
terraform workspace show

# List all workspaces
terraform workspace list

# Switch workspace (if using local workspaces)
terraform workspace select <name>

Note: HOLE Foundation uses Terraform Cloud workspaces (separate directories), not local workspaces.

---

Multi-Provider Operations

Scenario: Deploy Across Multiple Providers

User: "Deploy this AI-enhanced web app"

Required Changes:

1. Cloudflare (terraform/cloudflare/foundation/)

   • Pages project
   • Worker for API
   • KV namespace for cache
   • DNS records

2. AWS (terraform/aws/foundation/)

   • Lambda for AI processing
   • IAM role for Bedrock
   • Bedrock permissions

Workflow:

# 1. Update Cloudflare workspace
cd /Volumes/HOLE-RAID-DRIVE/Projects/hole-terraformer/terraform/cloudflare/foundation
# Edit .tf files
./tf-with-dotenvx.sh plan
./tf-with-dotenvx.sh apply

# 2. Update AWS workspace
cd /Volumes/HOLE-RAID-DRIVE/Projects/hole-terraformer/terraform/aws/foundation
# Load AWS credentials
export AWS_ACCESS_KEY_ID=$(cd /Volumes/HOLE-RAID-DRIVE/dotenvx && dotenvx get AWS_ACCESS_KEY_ID)
export AWS_SECRET_ACCESS_KEY=$(cd /Volumes/HOLE-RAID-DRIVE/dotenvx && dotenvx get AWS_SECRET_ACCESS_KEY)
export TF_VAR_aws_access_key="$AWS_ACCESS_KEY_ID"
export TF_VAR_aws_secret_key="$AWS_SECRET_ACCESS_KEY"
# Edit .tf files
terraform plan
terraform apply

# 3. Verify both providers
# Test Cloudflare Worker
# Test AWS Lambda
# Test integration

---

Terraform Cloud Integration

Remote State Benefits

✅ Team collaboration - Multiple people can work safely ✅ State locking - Prevents concurrent modifications ✅ Version history - Can rollback to previous states ✅ Secure storage - State never stored locally ✅ Access control - Manage who can apply changes

Triggering Runs

Via CLI (local execution):

terraform plan   # Creates speculative plan
terraform apply  # Creates and applies run

Via Terraform Cloud UI (optional):

• Queue plan manually
• Monitor run progress
• View run history

Via API (if using Terraform MCP):

• Use create_run tool
• Automated via Claude

---

Version Control

Git Workflow

Recommended workflow:

# 1. Make Terraform changes
vim cloudflare_zone.tf

# 2. Format
terraform fmt

# 3. Test locally
terraform plan

# 4. Commit
git add cloudflare_zone.tf
git commit -m "feat: Add new zone for newdomain.org"

# 5. Push
git push

# 6. Apply (could be automated via CI/CD)
terraform apply

What to Commit

✅ DO commit:

• All .tf files
• .terraform.lock.hcl (provider versions)
• README.md, documentation

❌ NEVER commit:

• *.auto.tfvars (secrets)
• terraform.tfvars (if contains secrets)
• .terraform/ directory
• terraform.tfstate (state is remote)
• .env files

---

Troubleshooting

State Lock Issues

Problem: "workspace already locked"

Solution:

# Force unlock (use with caution)
terraform force-unlock -force "<lock-id>"

# For Terraform Cloud workspaces
terraform force-unlock -force "theholetruth/<workspace-name>"

Provider Authentication Fails

Cloudflare:

# Verify token is loaded
echo "Token length: ${#CF_API_TOKEN}"
# Should be 40

# Reload from dotenvx
export CF_API_TOKEN=$(cd /Volumes/HOLE-RAID-DRIVE/dotenvx && dotenvx get CLOUDFLARE_API_TOKEN)

Azure:

# Re-authenticate
az login
az account set --subscription "de602062-dafa-4c8b-91b7-98a75bcd7cff"

AWS:

# Reload credentials
export AWS_ACCESS_KEY_ID=$(cd /Volumes/HOLE-RAID-DRIVE/dotenvx && dotenvx get AWS_ACCESS_KEY_ID)
export AWS_SECRET_ACCESS_KEY=$(cd /Volumes/HOLE-RAID-DRIVE/dotenvx && dotenvx get AWS_SECRET_ACCESS_KEY)

Plan Shows Drift

Problem: Plan shows changes you didn't make

Causes:

• Manual changes in provider dashboard
• Provider API updates
• Terraform provider version changes

Solutions:

1. Accept drift - Update .tf files to match reality
2. Enforce Terraform - Apply to force provider to match Terraform
3. Ignore changes - Use lifecycle { ignore_changes = [...] }

---

Best Practices (All Providers)

Before Making Changes

1. ✅ Navigate to correct workspace
2. ✅ Load appropriate secrets (provider-specific)
3. ✅ Run terraform plan (never skip!)
4. ✅ Review plan output carefully
5. ✅ Understand what will change

During Changes

6. ✅ Show user the plan
7. ✅ Explain impact (especially for Azure SSO)
8. ✅ Get explicit approval
9. ✅ Apply changes
10. ✅ Monitor for errors

After Changes

11. ✅ Verify in provider (optional - trust Terraform)
12. ✅ Test functionality (especially SSO after Azure changes)
13. ✅ Commit to git (version control)
14. ✅ Document changes (if significant)

---

Using Terraform MCP Tools (Future)

When HashiCorp Terraform MCP server is loaded (Phase 2), use these tools:

Workspace Management

• list_workspaces - See all 4 workspaces
• get_workspace - Get workspace details
• create_workspace - Create new workspace
• update_workspace - Modify workspace settings
• delete_workspace - Remove workspace

Variable Management

• list_workspace_variables - See all variables
• create_workspace_variable - Add secret/config
• update_workspace_variable - Modify variable
• delete_workspace_variable - Remove variable

Run Management

• create_run - Execute plan or apply
• get_run - Check run status
• list_runs - View run history
• cancel_run - Stop running operation

State Operations

• list_state_versions - View state history
• get_current_state - Download current state
• rollback_state - Restore previous state

Key Benefit: MCP tools work for ALL providers - same tools for Cloudflare, Azure, AWS!

---

Multi-Cloud Coordination

Cross-Provider Dependencies

When deploying across multiple providers:

Pattern 1: Cloudflare Worker → AWS Lambda

# Cloudflare (terraform/cloudflare/foundation/)
resource "cloudflare_worker_script" "api" {
  # ...
  # Environment variable points to AWS Lambda
  plain_text_binding {
    name = "AWS_LAMBDA_URL"
    text = "<lambda-function-url>"  # From AWS outputs
  }
}

# AWS (terraform/aws/foundation/)
resource "aws_lambda_function" "ai_processor" {
  # ...
}

resource "aws_lambda_function_url" "ai_processor" {
  function_name = aws_lambda_function.ai_processor.function_name
  authorization_type = "NONE"  # Or AWS_IAM with Worker auth
}

output "lambda_url" {
  value = aws_lambda_function_url.ai_processor.function_url
}

Workflow:

1. Deploy AWS Lambda first
2. Get Lambda URL from outputs
3. Update Cloudflare Worker with URL
4. Deploy Cloudflare Worker

Pattern 2: Multi-Provider Application

For full-stack apps requiring Cloudflare + AWS:

1. Plan both workspaces separately
2. Show user combined changes
3. Apply in dependency order:
   • AWS first (backend)
   • Cloudflare second (frontend, needs AWS URLs)
4. Verify integration works

---

Quick Reference by Provider

Cloudflare

cd terraform/cloudflare/foundation
./tf-with-dotenvx.sh plan
./tf-with-dotenvx.sh apply

Azure

cd terraform/production
terraform plan
terraform apply

AWS

cd terraform/aws/foundation
export AWS_ACCESS_KEY_ID=$(cd /Volumes/HOLE-RAID-DRIVE/dotenvx && dotenvx get AWS_ACCESS_KEY_ID)
export AWS_SECRET_ACCESS_KEY=$(cd /Volumes/HOLE-RAID-DRIVE/dotenvx && dotenvx get AWS_SECRET_ACCESS_KEY)
export TF_VAR_aws_access_key="$AWS_ACCESS_KEY_ID"
export TF_VAR_aws_secret_key="$AWS_SECRET_ACCESS_KEY"
terraform plan
terraform apply

---

Safety Checklist

Before running terraform apply on ANY provider:

• Correct workspace? (verify with pwd)
• Secrets loaded? (verify env vars)
• Ran terraform plan? (NEVER skip!)
• Reviewed plan output? (understand changes)
• User approved? (explicit confirmation)
• Backup state? (optional, TF Cloud has backups)

Special for Azure:

• Understand SSO impact? (can affect 22 services)
• Have backup login method?
• Plan to test SSO after?

---

Emergency Procedures

Rollback Changes

Option 1: Revert Terraform config

git revert <commit-hash>
terraform plan
terraform apply

Option 2: Restore previous state (via Terraform Cloud UI)

1. Go to workspace in Terraform Cloud
2. View state versions
3. Rollback to previous version

Unlock Stuck Workspace

terraform force-unlock -force "<lock-id>"

# For Terraform Cloud workspaces
terraform force-unlock -force "theholetruth/<workspace-name>"

Provider-Specific Rollback

Cloudflare: Changes are usually instant - rollback via Terraform Azure: Critical for SSO - test rollback procedure in advance AWS: Lambda/resources can be rolled back via Terraform

---

Performance Tips

Reduce Plan Time

• Use -target for specific resources:

  terraform plan -target=cloudflare_record.api
  

• Use -refresh=false to skip state refresh:

  terraform plan -refresh=false
  

Parallel Execution

Terraform automatically parallelizes operations where possible. For multi-provider deployments, run workspaces in parallel:

# Terminal 1: Cloudflare
cd terraform/cloudflare/foundation
./tf-with-dotenvx.sh apply &

# Terminal 2: AWS
cd terraform/aws/foundation
terraform apply &

# Wait for both
wait

---

Version History

• 1.0.0 (2026-01-09) - Initial Terraform operations skill
  • Multi-cloud workflow documented
  • All 3 providers covered
  • Provider-specific authentication methods
  • Cross-provider coordination patterns