Health check

Health checks answer different questions depending on who asks. Kubernetes separates them into three probe types run by the kubelet. A liveness probe asks whether the process is stuck beyond recovery; after failureThreshold consecutive failures the container is killed and restarted according to the pod's restart policy. A readiness probe asks whether the instance should receive traffic now; failure marks the pod not ready and removes its address from the Service's EndpointSlices without restarting anything. A startup probe, when defined, suspends the other two until it succeeds, so slow-initialising applications are not killed by liveness checks during boot. Probes can be httpGet (any status from 200 up to 399 counts as success), tcpSocket, exec (exit code 0) or grpc using the standard gRPC Health Checking Protocol, stable since Kubernetes 1.27. Defaults are initialDelaySeconds 0, periodSeconds 10, timeoutSeconds 1, successThreshold 1 and failureThreshold 3.

Load balancers run their own checks independently: cloud load balancers, HAProxy and Envoy probe backends at an interval and use separate healthy and unhealthy thresholds, which act as hysteresis against flapping. Envoy and similar proxies add passive health checking (outlier detection), ejecting a backend after a run of real request failures without any synthetic probe.

The most common design error is a deep liveness check that calls the database or downstream services. When the shared dependency degrades, every replica fails liveness at once and Kubernetes restarts the whole fleet, turning a partial outage into a total one and adding cold-start load. The usual guidance is to keep liveness shallow (the process can serve an HTTP request and its event loop is not deadlocked), let readiness reflect whether the instance itself can serve useful work, and handle dependency failures through timeouts, circuit breakers and degraded responses rather than restarts. Some applications also fail readiness deliberately on receiving SIGTERM while continuing to serve for a few seconds, so that load balancers which learn about endpoint changes with a delay stop routing to them before the process exits.

Other pitfalls are timeouts shorter than garbage-collection pauses, check endpoints that are expensive or unauthenticated yet expose version and dependency details, and treating a green health check as proof of correctness: a service can answer 200 on /healthz while returning wrong data. External synthetic monitoring, which exercises a real user journey from outside the network, complements internal probes, and health-check results feed monitoring and alerting but are not a substitute for symptom-based SLO alerts.