microservices-patterns

Microservices Patterns

A comprehensive skill for building, deploying, and managing production-grade microservices architectures. This skill covers service mesh patterns, traffic management, resilience engineering, observability, security, and modern microservices best practices using Istio and Kubernetes.

When to Use This Skill

Use this skill when:

• Architecting microservices-based applications with distributed systems
• Implementing service mesh infrastructure for service-to-service communication
• Adding resilience patterns like circuit breakers, retries, and timeouts
• Managing traffic routing, load balancing, and canary deployments
• Implementing distributed tracing and observability across microservices
• Securing microservices with mTLS and authorization policies
• Troubleshooting cascading failures and service degradation
• Building fault-tolerant distributed systems
• Implementing blue-green deployments and A/B testing
• Managing multi-cluster microservices deployments
• Implementing chaos engineering and fault injection
• Migrating from monolithic to microservices architecture

Core Concepts

Microservices Architecture

Microservices architecture structures an application as a collection of loosely coupled services:

• Service Independence: Each service is independently deployable and scalable
• Domain-Driven Design: Services align with business capabilities
• Decentralized Data: Each service owns its data store
• API-First: Services communicate via well-defined APIs
• Polyglot Persistence: Different services can use different databases
• Failure Isolation: Service failures don't cascade across the system

Service Mesh Fundamentals

A service mesh is an infrastructure layer for handling service-to-service communication:

• Data Plane: Sidecar proxies (Envoy) deployed alongside each service
• Control Plane: Manages and configures proxies (Istio, Linkerd, Consul)
• Service Discovery: Automatic service registration and discovery
• Load Balancing: Intelligent traffic distribution across service instances
• Observability: Built-in metrics, logs, and distributed tracing
• Security: mTLS, authentication, and authorization

Istio Architecture

Istio is the most popular service mesh implementation:

Control Plane Components:

• Istiod: Unified control plane for service discovery, configuration, and certificate management
• Pilot: Traffic management and service discovery
• Citadel: Certificate authority for mTLS
• Galley: Configuration validation and distribution

Data Plane:

• Envoy Proxy: High-performance sidecar proxy for each service
• Iptables Rules: Transparent traffic interception
• Service Proxy: Handles all network traffic for the service

Key Service Mesh Patterns

1. Sidecar Pattern: Proxy deployed alongside application container
2. Service Discovery: Automatic registration and discovery of services
3. Traffic Splitting: Route percentage of traffic to different versions
4. Circuit Breaker: Prevent cascading failures
5. Retry Logic: Automatic retry with exponential backoff
6. Timeout Policies: Request timeout configuration
7. Fault Injection: Chaos testing in production
8. Rate Limiting: Protect services from overload
9. mTLS: Mutual TLS for service-to-service encryption
10. Distributed Tracing: Request flow across services

Traffic Management

Virtual Services

Virtual services define routing rules for traffic within the mesh:

Key Features:

• HTTP/TCP/TLS Routing: Protocol-specific routing rules
• Match Conditions: Route based on headers, URIs, methods
• Weighted Routing: Traffic splitting across versions
• Redirects and Rewrites: URL manipulation
• Fault Injection: Delay and abort injection
• Retries: Automatic retry configuration
• Timeouts: Request timeout policies

Virtual Service Structure:

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: reviews
spec:
  hosts:
  - reviews
  http:
  - match:
    - headers:
        end-user:
          exact: jason
    route:
    - destination:
        host: reviews
        subset: v2
  - route:
    - destination:
        host: reviews
        subset: v3

Destination Rules

Destination rules configure policies for traffic after routing:

Key Features:

• Load Balancing: Round robin, random, least request
• Connection Pools: Connection limits and timeouts
• Outlier Detection: Circuit breaker configuration
• TLS Settings: mTLS mode configuration
• Subset Definitions: Version-based service subsets

Common Load Balancing Strategies:

• ROUND_ROBIN: Default, distributes evenly
• LEAST_REQUEST: Routes to instances with fewest requests
• RANDOM: Random distribution
• PASSTHROUGH: Use original destination

Traffic Splitting

Traffic splitting enables gradual rollouts and A/B testing:

Use Cases:

• Canary Deployments: Route small percentage to new version
• Blue-Green Deployments: Switch traffic between versions
• A/B Testing: Split traffic for experimentation
• Dark Launches: Shadow traffic to new version

Progressive Delivery Pattern:

v1: 100% → 90% → 70% → 50% → 20% → 0%
v2:   0% → 10% → 30% → 50% → 80% → 100%

Gateway Configuration

Gateways manage ingress and egress traffic:

Ingress Gateway:

• External traffic entry point
• TLS termination
• Protocol-specific routing
• Virtual hosting

Egress Gateway:

• Control outbound traffic
• Security policies for external services
• Traffic monitoring and logging

Resilience Patterns

Circuit Breaker Pattern

Circuit breakers prevent cascading failures by detecting and isolating failing services:

States:

• Closed: Normal operation, requests flow through
• Open: Service failing, requests fail immediately
• Half-Open: Testing if service recovered

Configuration Parameters:

• Consecutive Errors: Errors before opening circuit
• Interval: Time window for error counting
• Base Ejection Time: How long to eject failing instances
• Max Ejection Percentage: Maximum percentage of pool to eject

Benefits:

• Prevents resource exhaustion
• Fails fast instead of waiting for timeouts
• Gives failing services time to recover
• Monitors service health automatically

Retry Logic

Automatic retry with intelligent backoff strategies:

Retry Strategies:

• Fixed Delay: Constant delay between retries
• Exponential Backoff: Increasing delay between retries
• Jittered Backoff: Random jitter to prevent thundering herd

Configuration:

• Attempts: Maximum number of retries
• Per Try Timeout: Timeout for each attempt
• Retry On: Conditions triggering retry (5xx, timeout, refused-stream)
• Backoff: Base interval and maximum interval

Best Practices:

• Only retry idempotent operations
• Use exponential backoff with jitter
• Set maximum retry attempts
• Monitor retry rates

Timeout Policies

Timeout policies prevent indefinite waiting:

Timeout Types:

• Request Timeout: End-to-end request timeout
• Per Try Timeout: Timeout for each retry attempt
• Idle Timeout: Connection idle timeout
• Connection Timeout: Initial connection timeout

Timeout Hierarchy:

Overall Request Timeout
├─ Retry 1 (Per Try Timeout)
├─ Retry 2 (Per Try Timeout)
└─ Retry 3 (Per Try Timeout)

Best Practices:

• Set timeouts based on SLA requirements
• Use shorter timeouts for critical paths
• Configure per-try timeouts lower than overall timeout
• Monitor timeout rates and adjust

Bulkhead Pattern

Bulkheads isolate resources to prevent complete system failure:

Implementation:

• Thread Pools: Separate thread pools per service
• Connection Pools: Limited connections per upstream
• Queue Limits: Bounded queues to prevent memory issues
• Semaphores: Limit concurrent requests

Configuration:

• Max Connections: Maximum concurrent connections
• Max Requests Per Connection: HTTP/2 concurrent requests
• Max Pending Requests: Queue size for pending requests
• Connection Timeout: Time to establish connection

Rate Limiting

Rate limiting protects services from overload:

Rate Limit Types:

• Global Rate Limiting: Across all instances
• Local Rate Limiting: Per instance
• User-Based: Per user or API key
• Endpoint-Based: Per API endpoint

Algorithms:

• Token Bucket: Allows bursts while maintaining average rate
• Leaky Bucket: Smooths out traffic spikes
• Fixed Window: Simple time-window based limiting
• Sliding Window: More accurate than fixed window

Load Balancing

Service-Level Load Balancing

Istio provides intelligent Layer 7 load balancing:

Load Balancing Algorithms:

1. Round Robin

   • Default algorithm
   • Equal distribution across instances
   • Simple and predictable
   • Good for homogeneous instances

2. Least Request

   • Routes to instance with fewest active requests
   • Better for heterogeneous instances
   • Adapts to varying response times
   • Requires request tracking overhead

3. Random

   • Random instance selection
   • No state required
   • Good for large pools
   • Statistical distribution over time

4. Consistent Hash

   • Hash-based routing (sticky sessions)
   • Same client → same backend
   • Good for caching scenarios
   • Uses headers, cookies, or source IP

Connection Pool Management

Connection pools control resource usage:

TCP Settings:

• Max Connections: Total connections to upstream
• Connect Timeout: Connection establishment timeout
• TCP Keep Alive: Keep-alive probe configuration

HTTP Settings:

• HTTP1 Max Pending Requests: Queue size
• HTTP2 Max Requests: Concurrent streams
• Max Requests Per Connection: Connection reuse limit
• Max Retries: Outstanding retry budget

Health Checking

Active and passive health checking:

Passive Health Checking (Outlier Detection):

• Monitors actual traffic
• No additional probe overhead
• Detects failures automatically
• Ejects unhealthy instances

Active Health Checking:

• Explicit health probe requests
• Independent of traffic
• Configurable intervals
• Custom health endpoints

Security

Mutual TLS (mTLS)

mTLS provides encryption and authentication for service-to-service communication:

mTLS Benefits:

• Encryption: All traffic encrypted in transit
• Authentication: Services authenticate to each other
• Authorization: Service identity for policy enforcement
• Certificate Rotation: Automatic certificate management

mTLS Modes:

• STRICT: Require mTLS for all traffic
• PERMISSIVE: Accept both mTLS and plaintext (migration mode)
• DISABLE: No mTLS enforcement

Certificate Management:

• Automatic certificate issuance via Citadel
• Short-lived certificates (24 hours default)
• Automatic rotation
• SPIFFE-compliant identities

Authorization Policies

Fine-grained access control between services:

Policy Types:

• ALLOW: Explicitly allow traffic
• DENY: Explicitly deny traffic
• CUSTOM: Custom authorization logic

Match Conditions:

• Source: Source service identity, namespace, IP
• Destination: Target service, port, path
• Request: HTTP methods, headers, parameters
• JWT Claims: Token-based authorization

Policy Hierarchy:

Namespace-level default → Service-level → Specific paths

Authentication Policies

Configure authentication requirements:

Peer Authentication:

• Service-to-service authentication
• mTLS mode configuration
• Per-port settings

Request Authentication:

• End-user authentication
• JWT validation
• Custom authentication providers
• Token forwarding

Observability

Distributed Tracing

Track requests across microservices:

Key Concepts:

• Trace: Complete request journey
• Span: Individual service operation
• Parent-Child Relationships: Service call hierarchy
• Trace Context: Propagated metadata

Tracing Backends:

• Jaeger: CNCF distributed tracing
• Zipkin: Twitter's distributed tracing
• Tempo: Grafana's tracing backend
• AWS X-Ray: AWS distributed tracing

Trace Sampling:

• Always Sample: 100% sampling (development)
• Probabilistic: Sample percentage (e.g., 1%)
• Rate Limiting: Maximum traces per second
• Adaptive: Dynamic sampling based on traffic

Metrics Collection

Istio provides rich metrics automatically:

Service Metrics:

• Request Rate: Requests per second
• Error Rate: Percentage of failed requests
• Duration: Request latency (p50, p95, p99)
• Request Size: Request/response payload sizes

Infrastructure Metrics:

• CPU/Memory: Resource utilization
• Connection Pool: Pool statistics
• Circuit Breaker: Circuit state and events
• Retry/Timeout: Retry and timeout rates

Golden Signals:

1. Latency: How long requests take
2. Traffic: Request rate
3. Errors: Error rate
4. Saturation: Resource utilization

Logging

Structured logging for microservices:

Log Types:

• Access Logs: Request/response logging
• Application Logs: Service-specific logs
• Proxy Logs: Envoy sidecar logs
• Control Plane Logs: Istio component logs

Access Log Format:

{
  "timestamp": "2025-10-18T10:30:00Z",
  "method": "GET",
  "path": "/api/users",
  "status": 200,
  "duration_ms": 45,
  "upstream_service": "user-service-v2",
  "trace_id": "abc123",
  "user_agent": "mobile-app/2.1"
}

Kiali Visualization

Kiali provides service mesh observability:

Features:

• Service Graph: Visual topology of services
• Traffic Flow: Request flow visualization
• Health Status: Service health indicators
• Configuration Validation: Istio config validation
• Distributed Tracing: Integrated Jaeger traces

Best Practices

Service Design

1. Single Responsibility: Each service does one thing well
2. API-First Design: Define APIs before implementation
3. Idempotency: Design idempotent operations for safety
4. Versioning: Support multiple API versions
5. Backward Compatibility: Don't break existing clients

Deployment Strategies

1. Blue-Green Deployment

   • Maintain two identical environments
   • Switch traffic atomically
   • Easy rollback
   • Higher resource cost

2. Canary Deployment

   • Gradual rollout to subset of users
   • Monitor metrics before full rollout
   • Lower risk than big-bang
   • More complex orchestration

3. Rolling Update

   • Gradual replacement of instances
   • No additional resources needed
   • Kubernetes native support
   • Temporary version coexistence

4. Dark Launch

   • Route shadow traffic to new version
   • Test with production traffic
   • No user impact
   • Validate before real traffic

Resilience Engineering

1. Design for Failure: Assume services will fail
2. Fail Fast: Don't wait for timeouts
3. Graceful Degradation: Partial functionality better than none
4. Idempotent Retries: Safe to retry operations
5. Bulkhead Isolation: Isolate failure domains
6. Circuit Breakers: Prevent cascading failures
7. Timeouts Everywhere: Never wait indefinitely
8. Chaos Engineering: Test failure scenarios

Configuration Management

1. Namespace Isolation: Separate environments (dev, staging, prod)
2. GitOps: Store configs in Git
3. Validation: Validate configs before applying
4. Incremental Rollout: Test configs in dev first
5. Version Control: Track all config changes
6. Documentation: Document configuration decisions

Security Best Practices

1. mTLS by Default: Always encrypt service traffic
2. Least Privilege: Minimal authorization policies
3. Network Segmentation: Isolate services by namespace
4. Secret Management: Never hardcode secrets
5. Regular Updates: Keep Istio and Envoy updated
6. Audit Logging: Log all authorization decisions

Monitoring and Alerting

1. SLI/SLO/SLA: Define service level objectives
2. Dashboard Design: Focus on actionable metrics
3. Alert Fatigue: Only alert on actionable items
4. Error Budgets: Balance reliability and velocity
5. Runbooks: Document incident response
6. Post-Mortems: Learn from failures

Performance Optimization

1. Connection Pooling: Reuse connections
2. Request Batching: Batch when possible
3. Caching: Cache at multiple levels
4. Compression: Enable response compression
5. Protocol Selection: HTTP/2 or gRPC for efficiency
6. Resource Limits: Set appropriate limits
7. Horizontal Scaling: Scale out, not up

Migration Strategy

Strangler Pattern for monolith migration:

Phase 1: Route some traffic to microservices
Monolith (90%) + Microservices (10%)

Phase 2: Gradually increase microservice traffic
Monolith (70%) + Microservices (30%)

Phase 3: Continue migration
Monolith (40%) + Microservices (60%)

Phase 4: Complete migration
Monolith (0%) + Microservices (100%)

Common Patterns

Pattern 1: API Gateway Pattern

Single entry point for all client requests:

Components:

• External gateway (Istio Ingress)
• Virtual services for routing
• Rate limiting and authentication
• TLS termination

Benefits:

• Simplified client interface
• Centralized authentication
• Protocol translation
• Request aggregation

Pattern 2: Backend for Frontend (BFF)

Dedicated backend for each frontend type:

Use Cases:

• Mobile app has different needs than web
• Different data aggregation per client
• Client-specific optimization
• Reduced over-fetching

Implementation:

• Separate BFF service per client type
• Route by user-agent or subdomain
• Optimize responses per client
• Independent scaling

Pattern 3: Saga Pattern

Distributed transaction management:

Choreography-Based:

• Services publish events
• Other services react to events
• No central coordinator
• Loose coupling

Orchestration-Based:

• Central orchestrator
• Explicit transaction flow
• Easier to understand
• Single point of coordination

Pattern 4: CQRS (Command Query Responsibility Segregation)

Separate read and write models:

Benefits:

• Optimized read and write paths
• Independent scaling
• Different data models
• Event sourcing compatibility

Implementation:

• Write service updates data
• Read service queries optimized views
• Event bus for synchronization
• Eventually consistent reads

Pattern 5: Service Registry Pattern

Dynamic service discovery:

Components:

• Service registry (Kubernetes DNS)
• Service registration (automatic)
• Service discovery (Istio pilot)
• Health checking

Benefits:

• Dynamic scaling
• Automatic failover
• No hardcoded endpoints
• Location transparency

Pattern 6: Sidecar Pattern

Deploy auxiliary functionality alongside service:

Common Sidecars:

• Envoy proxy (traffic management)
• Log shipper (centralized logging)
• Metric collector (monitoring)
• Secret manager (credential injection)

Benefits:

• Separation of concerns
• Polyglot support
• Consistent functionality
• Independent updates

Pattern 7: Ambassador Pattern

Proxy for external service access:

Use Cases:

• Legacy system integration
• External API rate limiting
• Protocol translation
• Caching external responses

Implementation:

• Sidecar for external calls
• Circuit breaker for external service
• Retry logic and timeouts
• Monitoring and logging

Pattern 8: Anti-Corruption Layer

Isolate legacy system complexity:

Purpose:

• Translate between domain models
• Protect new architecture
• Gradual migration support
• Legacy system abstraction

Implementation:

• Adapter service layer
• Model translation
• Protocol conversion
• Versioning support

Advanced Techniques

Multi-Cluster Service Mesh

Extend service mesh across multiple clusters:

Use Cases:

• Multi-region deployment
• High availability
• Disaster recovery
• Compliance requirements

Implementation:

• Single control plane or multi-primary
• Service discovery across clusters
• Cross-cluster load balancing
• Consistent policies

Service Mesh Federation

Connect multiple independent service meshes:

Scenarios:

• Multiple teams/organizations
• Merger and acquisition
• Legacy mesh migration
• Different mesh implementations

Chaos Engineering

Proactively test system resilience:

Chaos Experiments:

• Service failures (pods deleted)
• Network latency injection
• Error injection (HTTP 503)
• Resource constraints (CPU/memory)
• DNS failures
• Certificate expiration

Tools:

• Istio fault injection
• Chaos Mesh
• Litmus Chaos
• Gremlin

GitOps for Service Mesh

Declarative configuration management:

Workflow:

1. Config changes in Git
2. Automated validation
3. Review and approval
4. Automated deployment
5. Continuous monitoring

Benefits:

• Version control
• Audit trail
• Disaster recovery
• Consistency

Troubleshooting

Common Issues

Issue 1: Service Not Accessible

• Check sidecar injection
• Verify VirtualService configuration
• Check DestinationRule subsets
• Validate service discovery
• Review authorization policies

Issue 2: High Latency

• Check retry and timeout settings
• Review connection pool limits
• Analyze distributed traces
• Check resource constraints
• Review load balancing algorithm

Issue 3: Circuit Breaker Not Working

• Verify outlier detection config
• Check error thresholds
• Review consecutive errors setting
• Validate base ejection time
• Monitor ejection metrics

Issue 4: mTLS Failures

• Check PeerAuthentication mode
• Verify certificate validity
• Review authorization policies
• Check namespace mesh config
• Validate Citadel operation

Issue 5: Traffic Routing Issues

• Validate VirtualService hosts
• Check subset definitions
• Review match conditions
• Verify gateway configuration
• Check service selector labels

Debugging Tools

1. istioctl: CLI for Istio management

   • istioctl analyze: Validate configuration
   • istioctl proxy-status: Check proxy sync status
   • istioctl proxy-config: View proxy configuration
   • istioctl dashboard: Access dashboards

2. kubectl: Kubernetes management

   • Check pod status
   • View logs
   • Port forwarding
   • Resource inspection

3. Kiali: Service mesh visualization

   • Service graph
   • Traffic flow
   • Configuration validation
   • Distributed tracing

4. Jaeger: Distributed tracing

   • Request traces
   • Latency analysis
   • Service dependencies
   • Error identification

5. Prometheus/Grafana: Metrics and visualization

   • Service metrics
   • Custom dashboards
   • Alerting rules
   • Historical analysis

Example Scenarios

Scenario 1: E-Commerce Microservices

Architecture:

• Frontend (React SPA)
• API Gateway
• Product Service
• Cart Service
• Order Service
• Payment Service
• Inventory Service
• Notification Service

Traffic Management:

• Canary deployment for new product search
• Circuit breaker on payment service
• Retry logic for inventory checks
• Timeout policies for external payment API
• Rate limiting on API gateway

Resilience:

• Graceful degradation if recommendations fail
• Bulkhead isolation for payment processing
• Fallback to cached product data
• Queue for async notifications

Scenario 2: Streaming Platform

Architecture:

• Video Service (transcoding)
• Metadata Service (content info)
• Recommendation Service (ML-based)
• User Service (profiles)
• CDN Integration
• Analytics Service

Traffic Management:

• A/B testing for recommendation algorithm
• Geographic routing to edge services
• Load balancing based on server capacity
• Traffic splitting for new video player

Performance:

• HTTP/2 for reduced latency
• Connection pooling for database
• Caching at multiple levels
• Adaptive bitrate streaming

Scenario 3: Financial Services Platform

Architecture:

• Account Service
• Transaction Service
• Fraud Detection Service
• Reporting Service
• External Bank Integration
• Audit Service

Security:

• Strict mTLS for all services
• Fine-grained authorization policies
• Audit logging for compliance
• Network segmentation by sensitivity

Resilience:

• Circuit breaker for external banks
• Idempotent transaction processing
• Saga pattern for distributed transactions
• Event sourcing for audit trail

Integration Patterns

Database per Service

Each microservice owns its database:

Benefits:

• Independent scaling
• Technology choice freedom
• Failure isolation
• Clear ownership

Challenges:

• Distributed transactions
• Data consistency
• Query complexity
• Data duplication

Solutions:

• Event-driven architecture
• Saga pattern
• CQRS
• API composition

Event-Driven Architecture

Asynchronous communication via events:

Components:

• Event producers
• Event bus (Kafka, RabbitMQ)
• Event consumers
• Event store

Patterns:

• Event notification
• Event-carried state transfer
• Event sourcing
• CQRS

API Composition

Aggregate data from multiple services:

Implementation:

• API Gateway queries services
• Parallel service calls
• Response aggregation
• Error handling

Optimization:

• Caching
• Request batching
• Partial responses
• Timeout management

Resources and References

Official Documentation

• Istio Documentation: https://istio.io/docs
• Kubernetes Documentation: https://kubernetes.io/docs
• Envoy Proxy: https://www.envoyproxy.io/docs
• CNCF Service Mesh Landscape: https://landscape.cncf.io/card-mode?category=service-mesh

Books and Papers

• "Building Microservices" by Sam Newman
• "Microservices Patterns" by Chris Richardson
• "Production-Ready Microservices" by Susan Fowler
• "The Art of Scalability" by Martin Abbott

Tools and Platforms

• Istio: Service mesh control plane
• Linkerd: Lightweight service mesh
• Consul Connect: HashiCorp service mesh
• AWS App Mesh: AWS-native service mesh
• Kiali: Service mesh observability
• Jaeger: Distributed tracing
• Prometheus: Metrics collection
• Grafana: Visualization

Community Resources

• Istio Blog: https://istio.io/blog
• CNCF Slack: #istio channel
• Stack Overflow: [istio] tag
• GitHub: istio/istio repository

---

Skill Version: 1.0.0 Last Updated: October 2025 Skill Category: Microservices, Service Mesh, Cloud Native, DevOps Compatible With: Istio 1.20+, Kubernetes 1.28+, Envoy Proxy Prerequisites: Kubernetes knowledge, containerization, networking basics