eks-kubernetes

Amazon EKS Kubernetes

When to Use

• Designing EKS cluster architecture for container workloads
• Choosing between managed node groups, self-managed nodes, and Fargate profiles
• Implementing IAM Roles for Service Accounts (IRSA) for pod-level AWS access
• Scaling clusters with Cluster Autoscaler or Karpenter
• Configuring EKS networking with VPC CNI and security groups for pods
• Preparing for AWS DOP-C02 or SAP-C02 exams

Core Jobs

1. Node Type Selection

Node Type	Management	OS Patching	Best For
Managed node groups	AWS manages ASG and node lifecycle	AWS patches AMIs; you trigger update	Default choice; rolling updates with node draining
Self-managed nodes	You manage EC2, ASG, bootstrap	You patch everything	Custom AMIs, specialized hardware, specific bootstrap requirements
Fargate profiles	Serverless; no nodes to manage	AWS manages everything	Event-driven, batch, dev/test, burst workloads

Fargate limitations:

• No DaemonSets (no per-node daemon processes)
• No GPU workloads
• No privileged containers
• Each pod gets isolated micro-VM (Firecracker); no shared host
• Storage: no persistent EBS volumes by default (use EFS for shared storage)
• Pricing: per pod vCPU/memory-hour (more expensive than optimally-packed EC2)

Mixed approach: Managed nodes for stable workloads + Fargate for burst/batch jobs.

2. IRSA (IAM Roles for Service Accounts)

Why IRSA:

• Without IRSA: pods use EC2 instance profile (all pods on node share same IAM role — over-privileged)
• With IRSA: individual pods assume specific IAM roles (least-privilege per workload)

Setup:

1. Enable OIDC provider for EKS cluster
2. Create IAM role with trust policy:

{
  "Principal": {
    "Federated": "arn:aws:iam::ACCOUNT:oidc-provider/oidc.eks.REGION.amazonaws.com/id/CLUSTER_ID"
  },
  "Action": "sts:AssumeRoleWithWebIdentity",
  "Condition": {
    "StringEquals": {
      "oidc.eks.REGION.amazonaws.com/id/CLUSTER_ID:sub": "system:serviceaccount:NAMESPACE:SA_NAME"
    }
  }
}

3. Annotate Kubernetes ServiceAccount:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: my-sa
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::ACCOUNT:role/my-pod-role

4. Reference ServiceAccount in Pod spec

AWS SDK auto-picks up IRSA credentials via web identity token file mounted at pod startup.

3. Cluster Autoscaler vs Karpenter

Feature	Cluster Autoscaler (CA)	Karpenter
Mechanism	Scale ASG up/down	Directly provision EC2 instances
Speed	Slower (ASG delays, ~2–5 min)	Faster (~30s from unschedulable to running)
Node diversity	Limited to defined ASG node types	Any EC2 instance type (best-fit)
Spot interruption	Manual handling	Built-in Spot rebalancing
Configuration	Complex (one ASG per node type)	NodePool/NodeClass declarative config
Cost optimization	Limited	Bin-packing + Spot + Savings Plans

Karpenter provisioning flow:

1. Pod marked unschedulable (no node fits)
2. Karpenter calculates optimal instance type from NodePool constraints
3. Directly calls EC2 CreateFleet API (no ASG)
4. Node joins cluster in ~30 seconds

NodePool example (Karpenter):

apiVersion: karpenter.sh/v1beta1
kind: NodePool
spec:
  template:
    spec:
      requirements:
        - key: "karpenter.sh/capacity-type"
          operator: In
          values: ["spot", "on-demand"]
        - key: "node.kubernetes.io/instance-type"
          operator: In
          values: ["m5.xlarge", "m5.2xlarge", "m6i.xlarge"]
  disruption:
    consolidationPolicy: WhenUnderutilized

4. Fargate Profiles

• Define which pods run on Fargate using namespace and label selectors
• Pod must match a profile to be scheduled on Fargate (otherwise goes to EC2 nodes)
• Multiple profiles per cluster allowed

fargate_profiles:
  - name: batch-jobs
    selectors:
      - namespace: batch
        labels:
          workload-type: batch
  - name: system
    selectors:
      - namespace: kube-system

CoreDNS on Fargate: requires Fargate profile for kube-system namespace; apply annotation patch to remove eks.amazonaws.com/compute-type: ec2 annotation.

5. EKS Networking (VPC CNI)

• Amazon VPC CNI: each pod gets a real VPC IP address (from the node's subnet)
• Pods communicate with VPC resources (RDS, ElastiCache) using VPC IPs directly
• IP prefix delegation: assign /28 prefixes to nodes (instead of individual IPs) → more IPs per node
• Security groups for pods: assign specific SGs to individual pods (not just nodes)
• IPv6: EKS supports IPv6 dual-stack clusters

Subnet planning for EKS:

• Nodes need IPs (1 per node)
• Pods need IPs (1 per pod; VPC CNI reserves slots on each node)
• Typical: node uses m5.xlarge (max 58 pods) → needs 59 IPs from subnet
• Use /19 or larger subnets for EKS node subnets

6. EKS Add-ons

Add-on	Purpose	Notes
Amazon VPC CNI	Pod networking with VPC IPs	Required; managed by AWS
CoreDNS	Cluster DNS	Required; runs as Deployment
kube-proxy	Service networking (iptables/ipvs)	Required per node
EBS CSI Driver	EBS persistent volumes for pods	Required for StatefulSets with EBS
EFS CSI Driver	EFS shared storage for pods	For shared persistent storage
AWS Load Balancer Controller	ALB/NLB creation from K8s Ingress/Service	Replaces ALB Ingress Controller

Managed add-ons: AWS handles updates; version controlled via EKS console/CLI; integration with EKS cluster version lifecycle.

Key Concepts

• EKS control plane — AWS managed; highly available across 3 AZs; you cannot SSH into control plane nodes
• Worker nodes — EC2 instances you manage (or Fargate); register with control plane via kubelet
• OIDC provider — OpenID Connect provider for the cluster; enables IRSA by federating K8s service accounts to IAM
• Node group — collection of EC2 nodes with same configuration (instance type, AMI, labels)
• Karpenter consolidation — automatically replaces underutilized nodes with smaller/fewer nodes (cost optimization)
• EKS Anywhere — run EKS on your own hardware (VMware, bare metal); uses same EKS control plane lifecycle
• EKS Blueprints — CDK/Terraform patterns for production-ready EKS clusters with common add-ons pre-configured
• Pod Disruption Budget (PDB) — minimum available pods during voluntary disruptions (node drain, rolling update)

Checklist

• IRSA configured for pods accessing AWS services (not EC2 instance profile)?
• Fargate profiles defined for appropriate workloads (batch, dev, burst)?
• Karpenter or Cluster Autoscaler configured for automatic node scaling?
• Security groups for pods configured for fine-grained network access control?
• EBS CSI driver add-on installed for StatefulSets requiring persistent storage?
• VPC subnets sized appropriately for maximum expected pod count?
• Pod Disruption Budgets set for production workloads (prevent full outage during upgrades)?
• EKS cluster version updated within support window (AWS supports last 3–5 minor versions)?

Output Format

• 🔴 Critical — pods using EC2 instance profile instead of IRSA (over-privileged; all pods share same role); no node autoscaling configured (manual capacity management)
• 🟡 Warning — Cluster Autoscaler instead of Karpenter (slower scaling, less cost-optimal); Fargate for DaemonSet workloads (unsupported); EKS version approaching end-of-support
• 🟢 Suggestion — Karpenter consolidation for cost savings; IP prefix delegation for larger pod density per node; EKS Blueprints for production-ready baseline

Exam Tips

• IRSA = recommended for AWS API access from pods (not EC2 instance profile); OIDC federation = pod-level IAM isolation
• Fargate = serverless nodes; cannot run DaemonSets; each pod gets isolated Firecracker micro-VM; no GPU support
• Managed node groups = AWS manages AMI updates and node draining during Kubernetes version upgrades (you approve the update)
• Karpenter = faster scaling than Cluster Autoscaler (directly provisions EC2 via CreateFleet API without ASG intermediary; ~30s vs ~2-5 min)
• EKS control plane = AWS managed (HA across 3 AZs; zero nodes to patch); worker nodes = your responsibility (OS patches, AMI updates)
• Spot instances with Karpenter = significant cost savings for fault-tolerant workloads (batch, stateless microservices); Karpenter handles interruption rebalancing
• VPC CNI = real VPC IPs for pods — plan subnet CIDR sizes to accommodate maximum pod count (each pod consumes one VPC IP)
• Security groups for pods = assign different SGs to different pod types on the same node (RDS access only for app pods, not for frontend pods)