Switch language한국어
Back to the list

War Story: How a Kubernetes 1.32 Node OOM Kill Cascaded Into a 2-Hour Outage for Our Video Streaming Service

TL;DR AI

Key summary

2 min read
  1. A Kubernetes 1.32 kubelet memory-accounting bug in cgroups v2 triggered a node OOM kill at a major video streaming service.

  2. The incident cascaded across 18 availability zones, cutting traffic by 92% and affecting up to 4.2 million concurrent viewers.

  3. Root cause was underreported memory usage in sidecar-heavy pods, especially with istio-proxy and linkerd-proxy containers.

  4. Adding pod-level memory limits with extra headroom significantly reduced failures and improved resilience.

Read the original