# Your Kubernetes Pod Keeps Getting OOMKilled. Why? *By Devang Pandya · Cloud Tech by Devang · Cloud • DevOps • AI — Made Simple* "Your pod keeps getting OOMKilled. Why?" is one of the questions I ask most in Kubernetes interviews. The usual answer is "increase the memory limit". Sometimes that's right, but on its own it's a junior answer. A senior engineer treats OOMKilled as a symptom and goes looking for the cause. Here's the full breakdown. --- ## 1. Read the evidence first Start with `kubectl describe pod `. Under the container's **Last State** you'll see `Terminated`, `Reason: OOMKilled` and **exit code 137** (128 + SIGKILL 9). Also check the restart count and `kubectl get events`. Separate two cases: - **Container OOMKilled**: the container went over its own memory **limit**, and the kernel's cgroup OOM killer ended it. - **Pod evicted**: the **node** ran low on memory and the kubelet evicted pods to protect itself. The reason here is `Evicted`, and the message mentions memory pressure. They have different fixes, so don't mix them up. ## 2. Requests vs limits - **Requests** are what the scheduler uses to place the pod. They're a reservation. - **Limits** are the hard ceiling the kernel enforces. If you set a small request and a large limit, the scheduler packs many pods onto a node assuming they're small. When they all grow at once, the node runs out of memory. Memory isn't compressible like CPU. CPU gets throttled, but memory gets killed. For most production services, setting the memory request equal to the limit (or close to it) gives predictable behaviour. ## 3. Make the runtime aware of the container Many runtimes size their heap from what they *think* the machine has: - **JVM**: modern JDKs are container-aware, but older ones weren't. Set `-XX:MaxRAMPercentage` (for example 70–75) so the heap leaves room for metaspace, threads and native memory. - **Node.js**: set `--max-old-space-size` below the container limit. Leave headroom, because the V8 heap isn't the process's only memory. - **Python, Gunicorn, Celery**: memory scales with the number of workers. Four workers × per-worker memory has to fit inside the limit. A common pattern: the heap limit is set equal to the container limit, so off-heap memory pushes the process over and the kernel kills it. ## 4. Leak or real growth? Graph **working-set memory** (`container_memory_working_set_bytes`), which is what the kubelet and OOM decisions care about, over hours or days. - A **sawtooth** that plateaus is normal garbage-collection behaviour. - A **steady climb** that only a restart resets is a leak. Raising the limit only stretches the time between crashes. - **Spikes** that line up with specific requests (large uploads, unbounded queries, batch jobs) point to a code path that needs streaming or pagination. Heap dumps, profilers and per-endpoint memory metrics will find the cause faster than trial-and-error limit changes. ## 5. Know your QoS class Kubernetes assigns a QoS class to each pod: - **Guaranteed**: requests equal limits for every container. These are the last to be evicted under node pressure. - **Burstable**: requests are set below limits. - **BestEffort**: nothing set. These are evicted first. Your critical workloads should be Guaranteed. Batch and dev workloads can be Burstable. ## 6. Right-size with data Use real usage data. Check percentiles in your metrics, or run Vertical Pod Autoscaler in recommendation mode, and set limits with sensible headroom above the observed peak. Review them after major releases. --- ## The senior angle OOMKilled isn't a sizing problem. It's information. Raising the limit without knowing *why* memory grows just moves the crash to 3 AM on a bigger, more expensive node. A senior answer explains the evidence, tells the container kill apart from node eviction, ties the runtime to the cgroup, and fixes the cause. In my 14+ years in Cloud and DevOps, the engineers I trust most ask "what is using the memory?" before "how much more can we give it?" --- > **Let's connect: Free 15-min Cloud & DevOps Interview Guidance Call 👆 link in bio** > ([topmate.io/cloudtechbydevang](https://topmate.io/cloudtechbydevang/2358399))