cluster-upgrade-and-capacity

Cluster Upgrade and Capacity Planning

1. Version Skew Rules (non-negotiable)

Component pair	Max skew
kube-apiserver ↔ kubelet	kubelet may be at most 2 minor versions behind apiserver
kube-apiserver ↔ kube-controller-manager / kube-scheduler	Must be same minor version
kubectl ↔ kube-apiserver	kubectl may be ±1 minor version
Helm chart API versions	Must target APIs present in the target cluster version

Upgrade order: control plane first, then node pools one at a time.

2. Pre-Upgrade Checklist

Run these before touching anything:

• kubectl get nodes — all nodes Ready; no NotReady or Unknown
• Check deprecated API usage: kubectl api-resources --verbs=list + run pluto or kubectl deprecations against manifests
• Audit admission webhooks — ValidatingWebhookConfiguration and MutatingWebhookConfiguration: set failurePolicy: Fail webhooks will block upgrades if their endpoint is down
• Verify PodDisruptionBudgets allow at least 1 disruption on every workload (kubectl get pdb -A)
• Snapshot etcd (managed clusters: confirm automated backup is current)
• Review add-on compatibility matrix: CNI, CSI drivers, metrics-server, CoreDNS — all must support the target version
• Read the upstream changelog for removed/deprecated APIs in the target minor version

3. Upgrade Sequence

1. Upgrade control plane (managed: click/Terraform; self-managed: kubeadm upgrade apply)
2. Cordon + drain node 1 of pool A  →  confirm pod reschedule  →  upgrade node  →  uncordon
3. Repeat for remaining nodes in pool A (rolling — respect PDBs)
4. Upgrade node pool B (if separate), same rolling procedure
5. Upgrade add-ons to versions compatible with the new control plane
6. Run smoke tests against the cluster API and critical workloads

Drain flags:

kubectl drain <node> \
  --ignore-daemonsets \    # DaemonSet pods are managed separately
  --delete-emptydir-data \ # Evict pods using emptyDir (data is lost — intentional)
  --timeout=300s

4. Capacity Planning

Node Pool Sizing Formula

Required nodes = ceil( sum(all-pod-requests) / (node-allocatable × target-utilization) )

• node-allocatable = node capacity minus OS/kubelet/system reserved (check kubectl describe node)
• target-utilization = 0.70 for CPU, 0.80 for memory (headroom for burst and rolling upgrades)
• Add 1–2 surge nodes for the rolling upgrade itself (pods need somewhere to land during drain)

HPA Sizing Inputs

Input	Where to get it
Observed p99 CPU/mem per replica	Prometheus / CloudWatch Container Insights
Traffic multiplier at peak	Load test or prod traffic pattern
Pod startup time	kubectl describe pod → container start latency

Set minReplicas ≥ 2 (single replica = planned outage during upgrade).

Cluster Autoscaler vs Karpenter

Factor	Cluster Autoscaler	Karpenter
Node provisioning granularity	Node group / ASG	Any instance type that fits
Speed	1–3 min (ASG warm)	30–90 s (direct EC2 API)
Bin-packing	Limited	Aggressive consolidation
Multi-cloud	Yes	AWS-native (Azure/GCP in progress)

Use Karpenter on EKS when bin-packing efficiency and speed matter; use CA otherwise.

5. Post-Upgrade Validation

kubectl get nodes                          # all Ready, correct version
kubectl get pods -A | grep -v Running      # no CrashLoopBackOff / Pending
kubectl top nodes                          # resource usage sane
kubectl get events -A --sort-by=.lastTimestamp | tail -50

Smoke-test critical services and verify SLO dashboards show no degradation for 15 minutes before declaring success.

Pitfalls

• Skipping the deprecated-API scan — v1beta1 removed in 1.25 / 1.26 breaks apply on upgrade day.
• failurePolicy: Fail webhooks whose backend pod is on the node being drained — the drain blocks and the upgrade hangs.
• Single-replica stateful workloads with maxUnavailable: 1 PDBs — drain waits forever for a pod that can't move.
• Upgrading node pools before the control plane — violates skew rules and can corrupt cluster state.
• Forgetting to upgrade the CNI/CSI add-ons — they often break silently on a new API server.
• No capacity surge during rolling upgrade — pods evicted during drain find no nodes with capacity and stay Pending.