Mutual TLS Design

Mutual TLS Design

Both sides prove their identity with certificates -- the server authenticates to the client and the client authenticates to the server, establishing a cryptographically verified service-to-service channel

When to Use

• Authenticating services to each other in a microservices architecture
• Implementing zero trust networking where network location is not sufficient for trust
• Replacing API keys or shared secrets for service-to-service communication
• Deploying a service mesh (Istio, Linkerd, Cilium) with automatic mTLS
• Building internal PKI for workload identity
• Meeting compliance requirements for encrypted and authenticated internal traffic

Threat Context

Standard TLS authenticates only the server -- the client verifies the server's certificate, but the server has no cryptographic proof of the client's identity. In service-to-service communication, this means any service on the network can call any other service. API keys or bearer tokens add authentication but are static, shared, and vulnerable to theft -- a stolen API key grants access until someone notices and rotates it, which can be weeks or months. mTLS provides mutual cryptographic authentication: each service has its own certificate issued by a trusted CA, and both sides verify the other's identity during the TLS handshake. The 2017 Equifax breach exploited flat internal networking with no service authentication -- internal services trusted any connection from the internal network, allowing the attacker to move laterally through 48 databases after exploiting a single Apache Struts vulnerability.

Instructions

1. Deploy an internal CA. Use Vault PKI, step-ca (Smallstep), cfssl, or a service mesh's built-in CA. Do not use Let's Encrypt for internal mTLS -- Let's Encrypt certificates are publicly trusted, intended for public-facing services, and require public domain names. Internal CAs issue certificates trusted only within your organization's trust store. This limits the blast radius of CA compromise and allows you to issue certificates for internal service names that are not publicly resolvable.

2. Issue short-lived certificates. 24-72 hour certificate lifetimes eliminate the need for revocation infrastructure. If a certificate is compromised, it expires within hours rather than remaining valid for months. Automate certificate rotation with cert-manager (Kubernetes), Vault Agent, or the service mesh's automatic certificate rotation. The operational cost of short-lived certificates is higher, but the security benefit is decisive: revocation (CRLs, OCSP) is complex, unreliable, and can be bypassed. Short lifetimes make revocation unnecessary.

3. Use SPIFFE for workload identity. SPIFFE (Secure Production Identity Framework for Everyone) standardizes workload identity as a URI: spiffe://cluster.local/ns/production/sa/payment-service. SPIRE (the reference implementation) acts as a workload attestation agent that verifies workload identity using platform-specific selectors (Kubernetes service account, AWS instance metadata, Docker container ID) and issues SVID (SPIFFE Verifiable Identity Document) certificates. Service meshes (Istio, Linkerd) implement SPIFFE-compatible identity natively.

4. Configure certificate validation correctly. Both client and server must validate the peer's certificate: verify the certificate chain (signed by a trusted CA), verify the certificate is not expired, verify the SAN (Subject Alternative Name) matches the expected service identity. Do not disable certificate verification in production -- ever. InsecureSkipVerify: true (Go), rejectUnauthorized: false (Node.js), verify=False (Python requests) all defeat the entire purpose of mTLS by accepting any certificate, including attacker-generated ones.

5. Enforce mTLS at the network level. In Kubernetes, use PeerAuthentication policies (Istio) or NetworkPolicy with identity selectors (Cilium) to ensure that only mTLS connections are accepted. Istio's STRICT mode rejects any plaintext connection to the service. This prevents accidental fallback to unencrypted communication and ensures that no service can communicate without presenting a valid identity certificate.

6. Handle the migration from plaintext to mTLS. In brownfield environments with existing services communicating over plaintext, deploy mTLS in PERMISSIVE mode first (accept both plaintext and mTLS). Monitor which services are still sending plaintext using service mesh telemetry. Migrate them one by one, verifying each migration. Once all services use mTLS, switch to STRICT mode. Istio's PeerAuthentication supports per-namespace and per-service PERMISSIVE/STRICT configuration for gradual rollout.

Details

mTLS Handshake

The full TLS handshake with mutual authentication proceeds as follows:

1. Client sends ClientHello (supported cipher suites, TLS version)
2. Server responds with ServerHello, Server Certificate, and CertificateRequest
3. Client verifies the server's certificate against its trust store
4. Client sends its own Client Certificate and CertificateVerify message
5. The CertificateVerify contains a signature over the handshake transcript, proving the client possesses the private key corresponding to its certificate
6. Server verifies the client's certificate against its trust store
7. Both sides derive session keys and the encrypted channel is established

The critical difference from standard TLS is step 2 (CertificateRequest) and steps 4-6. Without the CertificateRequest, the client never presents a certificate and the server has no cryptographic proof of the client's identity.

Service Mesh mTLS

In Istio, Envoy sidecar proxies handle mTLS transparently. The application sends plaintext to localhost on a designated port. The local Envoy sidecar intercepts the outbound connection, establishes an mTLS connection to the destination's Envoy sidecar, and forwards the traffic. The destination sidecar decrypts and delivers plaintext to the destination application. No application code changes are required. Linkerd uses a similar sidecar architecture with its own proxy (linkerd2-proxy, written in Rust). Cilium uses eBPF to implement mTLS more efficiently without sidecars, reducing the per-pod resource overhead.

SPIFFE/SPIRE Deep Dive

SPIRE has two components: the SPIRE Server (central authority that maintains the signing CA, registration entries, and trust bundles) and the SPIRE Agent (runs on each node, attests workloads, caches and distributes SVIDs). Workload attestation uses platform-specific selectors: Kubernetes service account name and namespace, Docker container labels, AWS instance ID and IAM role, bare-metal process UID and binary path. This ensures that only the legitimate workload receives the certificate for its identity. An attacker who compromises the host but not the workload's attestation properties cannot obtain a valid SVID.

Debugging mTLS Failures

Common failure modes and their diagnostics:

Symptom	Likely Cause	Diagnostic Command
Connection refused	STRICT mode, client has no certificate	Check PeerAuthentication policy
TLS handshake error	Certificate not signed by trusted CA	openssl s_client -cert ... -key ...
Identity mismatch	SAN does not match expected identity	openssl x509 -text -in cert.pem
Intermittent failures	Certificate expired, clock skew	Check system time, certificate validity
Works from one pod, not another	Missing sidecar injection	istioctl authn tls-check

Anti-Patterns

1. InsecureSkipVerify: true in production. This disables certificate validation, meaning any certificate (self-signed, expired, wrong identity) is accepted. This provides encryption but zero authentication. A man-in-the-middle with any certificate can intercept all traffic. This single line of code negates the entire mTLS deployment.

2. Long-lived client certificates. Certificates with 1-year lifetimes require revocation infrastructure (CRLs, OCSP) that is complex and unreliable in practice. CRL checking is best-effort in most TLS implementations, and OCSP has soft-fail behavior. Use short-lived certificates (24-72 hours) with automatic rotation instead.

3. Sharing client certificates across services. If all services use the same client certificate and private key, you cannot distinguish between services at the authorization layer and cannot implement per-service access control. Each service must have its own unique identity certificate. Sharing certificates also means that compromising one service compromises the identity of all services.

4. mTLS without authorization. mTLS proves identity but does not enforce authorization. After verifying the client's certificate identity (e.g., payment-service), you still need to check whether payment-service is authorized to access the requested endpoint. Use service mesh authorization policies (Istio AuthorizationPolicy, Cilium NetworkPolicy) to define which services can call which endpoints.

5. Manual certificate distribution. Copying certificates to servers via SSH, baking them into Docker images, or storing them in environment variables does not scale and leads to expiration outages. Automate certificate lifecycle with SPIRE, cert-manager, Vault Agent, or service mesh automatic rotation. If a human is involved in certificate distribution, it will eventually fail.