Kubernetes Probes Done Right

The Problem#

Kubernetes probes decide when traffic reaches a pod and when a pod gets restarted. Bad probes can create outages during deploys or restart healthy services under temporary load.

Why It Matters#

Probes are not documentation; they are control signals. Readiness tells Kubernetes whether a pod should receive traffic. Liveness tells Kubernetes whether a process should be restarted. Startup gives slow applications time to boot.

Kubernetes is used by companies such as Spotify, Airbnb, Shopify, and many cloud platform teams, and probe mistakes are a common source of rollout instability in cluster-based systems.

Project Example#

A Java API takes 45 seconds to warm caches and connect to dependencies. If the liveness probe starts after 10 seconds and checks a slow dependency, Kubernetes may kill the pod in a loop. Use a startup probe for boot, readiness for traffic, and liveness only for unrecoverable deadlock.

Implementation Example#

startupProbe:
  httpGet:
    path: /health/startup
    port: 8080
  failureThreshold: 30
  periodSeconds: 2

readinessProbe:
  httpGet:
    path: /health/readiness
    port: 8080
  periodSeconds: 5

livenessProbe:
  httpGet:
    path: /health/liveness
    port: 8080
  periodSeconds: 10

The readiness endpoint can fail when the app is draining, warming, or unable to serve traffic. The liveness endpoint should be simpler and mostly local.

Implementation Checklist#

• Use startup probes for slow initialization.
• Make readiness fail when the pod should stop receiving traffic.
• Keep liveness checks local and cheap.
• Give probes realistic timeouts and failure thresholds.
• Test probes during rolling deploys and dependency outages.
• Use different endpoints for different probe meanings.
• Ensure readiness fails during graceful shutdown.
• Avoid dependency-heavy liveness checks.

Production Notes#

If the database is down, restarting every API pod usually does not fix the database. Make readiness fail if the service cannot serve useful traffic, but keep liveness focused on deadlocks and unrecoverable process state.

Common Mistakes#

• Checking the database in liveness.
• Making readiness pass before warmup finishes.
• Using the same endpoint for all probes.
• Setting aggressive timeouts on JVM or cold-starting apps.
• Forgetting startup probes for apps with slow boot.

Summary#

Probes are operational policy. Separate startup, readiness, and liveness so Kubernetes helps availability instead of fighting your app.