Vault & Secrets Management

This skill provides HashiCorp Vault best practices for secrets management.

When to Use

Activates when:

• Working with secrets in Kubernetes
• Managing Vault initialization and unsealing
• Using External Secrets Operator
• Setting up new environments

Vault Lifecycle

Initialization

Command: make vault-init

This initializes a new Vault instance and saves keys to vault-init/ directory.

# Initialize Vault (first time only)
make vault-init

# Keys are saved to vault-init/ directory:
# - vault-init/unseal-key-1
# - vault-init/unseal-key-2
# - vault-init/unseal-key-3
# - vault-init/root-token

Unsealing

Command: make vault-unseal

Required after pod restarts. Vault starts in a sealed state and must be unsealed with keys.

# Unseal Vault (after pod restart)
make vault-unseal

Why unsealing is needed:

• Vault pods restart (manual restart, node failure, updates)
• Vault starts sealed for security
• Must provide unseal keys to make Vault operational

Security Best Practices

Never Commit Vault Keys or Tokens

❌ Don't:

# Don't commit these files
git add vault-init/
git commit -m "vault keys"  # NEVER DO THIS

✅ Do:

# Ensure vault-init/ is in .gitignore
echo "vault-init/" >> .gitignore

# Store keys securely offline
# - Password manager
# - Encrypted backup
# - Secure key management system

Secrets in Code

❌ Don't:

// Don't hardcode secrets
const apiKey = "sk-1234567890abcdef"
password := "supersecret123"

✅ Do:

// Use environment variables from Vault
apiKey := os.Getenv("API_KEY")

// Or use External Secrets Operator
// which automatically creates k8s Secrets from Vault

External Secrets Operator

External Secrets Operator manages Kubernetes Secret creation from Vault.

How It Works

1. Secrets stored in Vault
2. ExternalSecret CR references Vault path
3. ESO fetches secret from Vault
4. ESO creates/updates Kubernetes Secret automatically

Example ExternalSecret

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: app-secrets
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: vault-backend
    kind: SecretStore
  target:
    name: app-secrets
    creationPolicy: Owner
  data:
    - secretKey: api-key
      remoteRef:
        key: secret/data/app
        property: api_key

This creates a Kubernetes Secret named app-secrets with data from Vault.

Common Workflows

Storing a New Secret

# 1. Write secret to Vault
vault kv put secret/myapp/config \
  api_key=abc123 \
  db_password=secretpass

# 2. Create ExternalSecret CR
kubectl apply -f external-secret.yaml

# 3. Verify Kubernetes Secret was created
kubectl get secret app-secrets
kubectl describe secret app-secrets

Rotating a Secret

# 1. Update secret in Vault
vault kv put secret/myapp/config api_key=newkey123

# 2. ESO automatically syncs (based on refreshInterval)
# Or manually trigger sync by deleting the k8s Secret
kubectl delete secret app-secrets

# 3. ESO recreates Secret with new value
kubectl get secret app-secrets -o yaml

Debugging Secrets

# Check Vault status
kubectl exec -it vault-0 -- vault status

# Check if Vault is sealed
kubectl logs vault-0 | grep sealed

# Check External Secrets Operator
kubectl logs -n external-secrets deployment/external-secrets

# Check ExternalSecret status
kubectl describe externalsecret app-secrets

Vault Initialization Checklist

When setting up Vault in a new environment:

• Run make vault-init to initialize Vault
• Securely save unseal keys and root token from vault-init/
• Add vault-init/ to .gitignore
• Run make vault-unseal to unseal Vault
• Configure Vault policies and auth methods
• Set up External Secrets Operator integration
• Test secret retrieval and rotation

Pod Restart Recovery

When Vault pods restart:

# 1. Check Vault status
kubectl get pods -l app=vault

# 2. Vault will be sealed - check logs
kubectl logs vault-0 | tail -20

# 3. Unseal Vault
make vault-unseal

# 4. Verify Vault is operational
kubectl exec vault-0 -- vault status

Best Practices Summary

✅ Do:

• Initialize Vault with make vault-init
• Unseal Vault after restarts with make vault-unseal
• Use External Secrets Operator for Kubernetes secrets
• Store unseal keys securely offline
• Never commit Vault keys to git
• Rotate secrets regularly

❌ Don't:

• Commit vault-init/ directory to git
• Hardcode secrets in code
• Manually create Kubernetes Secrets for Vault-backed data
• Leave Vault sealed
• Share root token unnecessarily