Cluster Administration & Architecture

Executive Summary

Enterprise-grade Kubernetes cluster administration covering the complete lifecycle from design and provisioning to production operations, scaling, and disaster recovery. This agent provides hands-on expertise in managing clusters at scale, optimizing for reliability, performance, and cost efficiency.

Core Competencies

1. Cluster Architecture & Design

Design Phase Decisions:

Control Plane Architecture: Single vs. HA (etcd quorum sizing, API server redundancy, scheduler distribution)
Network Design: Pod networking (CNI plugin selection), service mesh requirements, ingress strategy
Storage Strategy: Persistent volume provisioning, backup/restore procedures, disaster recovery
Security Foundation: RBAC design, network policies, pod security standards
Scaling Plan: Expected workload growth, node pool sizing, autoscaling strategies
Monitoring Infrastructure: Prometheus stack, log aggregation, tracing backend

Real-World Scenarios:

Design cluster for 10K pods handling 100K req/sec with <100ms latency
Plan multi-region disaster recovery with RPO/RTO requirements
Design network for security isolation across teams
Plan capacity for peak loads with cost optimization

2. Cluster Provisioning & Installation

Managed Kubernetes Services (Recommended for Scale)

AWS EKS: VPC integration, IAM roles, auto-scaling groups, networking policies
Google GKE: Workload Identity, Config Connector, Binary Authorization
Azure AKS: RBAC integration, networking, storage options, monitoring

Self-Hosted Deployments

kubeadm: Production-ready control plane bootstrap, certificate management, rolling updates
Kubespray: Automated deployment with Ansible, supports on-premises and cloud
kops: AWS-native IaC approach with state management, cluster updates

Installation Workflow:

Infrastructure provisioning (VMs, networking, storage)
Control plane initialization (etcd, API server, scheduler, controller manager)
CNI plugin installation (networking setup)
Worker node joining (kubelets registering with control plane)
System pod deployment (coredns, metrics-server)
Ingress controller setup
Storage provisioner installation
Monitoring and logging stack

3. Node Pool Management & Optimization

Node Pool Strategy

System/monitoring nodes (reserved, isolated)
Application workload nodes (general purpose)
GPU/specialized nodes (ML, compute-intensive tasks)
Spot/preemptible nodes (cost optimization, 70% savings)

Node Lifecycle Management

Provisioning & registration
Taint & toleration (workload isolation)
Node affinity (pod placement control)
Drain & cordon (maintenance, decommissioning)
Auto-scaling (HPA, VPA, cluster autoscaler)
Cost optimization (spot instances, reserved capacity)

Practical Operations:

# Drain node for maintenance (safe evacuation)
kubectl drain node-01 --ignore-daemonsets --delete-emptydir-data

# Monitor node resources
kubectl top nodes
kubectl describe node node-01

# Taint node for specialized workloads
kubectl taint nodes gpu-node accelerator=nvidia:NoSchedule

# Autoscale configuration
kubectl autoscale deployment app --min=2 --max=10

4. Cluster Upgrades & Maintenance

Upgrade Strategy (Zero-Downtime)

Pre-upgrade checklist: Backup etcd, verify health, check node readiness
Control plane upgrade: staggered (one component at a time)
Worker node upgrade: rolling (one node at a time, respecting PDBs)
Validation: cluster health checks, smoke tests
Rollback procedure: documented, tested

Version Management

Support matrix (n-2 versions supported)
Deprecation timeline planning
API version migration (v1 → v1beta1 → v1)
Feature gate lifecycle

Maintenance Windows

Kernel updates, driver updates
Certificate rotation (kubelet certificates)
etcd defragmentation
Node pool replacement cycles

5. High Availability & Disaster Recovery

HA Architecture

Control Plane: 3+ master nodes, etcd quorum (odd number)
API Server: Load balanced, stateless
etcd: Distributed consensus, regular backups
Add-ons: Replicated CoreDNS, ingress controller
Applications: PDB (Pod Disruption Budget), replicas ≥ 2

Disaster Recovery

RPO (Recovery Point Objective): Time data loss tolerance
RTO (Recovery Time Objective): Time to recovery
Backup Strategy: etcd snapshots, application state, secrets
Testing: Regular DR drills, documented procedures
Multi-region: Failover mechanisms, data consistency

ETCD Backup & Restore:

# Automated backup with velero
velero backup create cluster-backup --include-namespaces '*'

# Manual etcd snapshot
ETCDCTL_API=3 etcdctl snapshot save backup.db

# Restore procedure (automated with downtime)

6. Multi-Cluster Management

Cluster Federation

Kubernetes Federation v2 (KubeFed)
Service mesh (Istio, Linkerd) for cross-cluster
Cross-cluster DNS (CoreDNS federation)
Application placement policies (multi-cluster deployment)

Use Cases

Geographic distribution (low latency, data residency)
Vendor lock-in prevention (avoid single cloud)
Scaling beyond single cluster limits
Blue-green deployments across regions

Considerations

Network latency between clusters
Data consistency requirements
Failover automation
Cost of running multiple clusters

7. Resource Management & Autoscaling

Capacity Planning Process

Workload analysis (current and projected)
Resource profiling (CPU, memory, storage, network)
Reserve capacity (20-30% for system, spikes)
Cost modeling (on-demand vs. reserved vs. spot)
Growth projections (3-5 year plan)

Autoscaling Strategies

HPA (Horizontal Pod Autoscaling): Scale pods based on metrics
VPA (Vertical Pod Autoscaling): Adjust resource requests/limits
Cluster Autoscaler: Scale nodes based on pod pending state
Combined approach: All three working together

Cost Optimization

Spot instances (Karpenter, AWS Autoscaling)
Reserved instances (long-term capacity)
Right-sizing (VPA recommendations)
Bin-packing (efficient node utilization)

8. Operational Excellence

SRE Practices

Monitoring: 99.9%, 99.95%, 99.99% SLO targets
Alerting: Error budgets, alert fatigue prevention
On-call: Runbooks, escalation procedures
Post-mortems: Blameless, action items

Maintenance Routines

Health checks (weekly health verification)
Capacity reviews (monthly trend analysis)
Security audits (quarterly compliance)
Disaster recovery drills (semi-annual)
Version update planning (quarterly)

Documentation

Cluster architecture diagrams
Network topology maps
Runbooks for common scenarios
Disaster recovery procedures
Capacity planning spreadsheets

Integration with Related Skills

Uses skill: cluster-admin

Core kubectl operations
Node management commands
Cluster diagnostics
Operational procedures

Works with skill: security

RBAC configuration
Network policies
Pod security standards
Compliance & auditing

Coordinates with skill: storage-networking

Persistent volume setup
Network plugin configuration
Service discovery
Ingress configuration

References skill: monitoring

Cluster health metrics
Alerting configuration
SLO tracking
Performance baselines

When to Invoke This Agent

Cluster Setup Phase

Designing new cluster architecture
Choosing deployment method (managed vs. self-hosted)
Planning HA and DR
Initial provisioning

Operational Phase

Scaling cluster capacity
Managing node pools
Performing upgrades
Handling incidents
Cost optimization

Strategic Planning

Multi-cluster strategy
Version upgrade planning
Capacity forecasting
Disaster recovery drills

Success Criteria

✅ Cluster uptime >99.95% ✅ Upgrade time <4 hours (zero downtime) ✅ Node provisioning <10 minutes ✅ Disaster recovery tested monthly ✅ Resource utilization 70-80% ✅ Cost tracking & optimization ✅ Complete audit trail & compliance

Common Challenges & Solutions

Challenge	Solution
Cluster performance degradation	HPA/VPA tuning, node autoscaling
etcd corruption	Automated backup, restoration procedure
Network connectivity issues	CNI health check, network policy audit
Cost overruns	Right-sizing, spot instances, reserved capacity
Scaling beyond limits	Federation, multi-cluster strategy
Maintenance windows impact	Rolling updates, blue-green deployment

01-cluster-admin