Before you start
Linux-node Kubernetes clusters; Bash examples. Optional cgroup evidence requires the node owner and the correct cgroup version. Documentation-reviewed on 2026-09-07; no cluster execution.
Commands are examples, not actions run by this page. Replace the sample names and paths. Run related blocks in the same shell. Read the explanations before copying. Logs and configuration can contain private data; keep credentials and unredacted evidence out of public posts.
Validation: documentation-reviewed guidance, not a tested live-cluster repair.
Replace the placeholders and retain one shell for these cards. Identify the exact container, Pod UID, and node. Application owners can inspect their authorized namespace; node logs and cgroup files require separate operator access. This guide does not authorize entering a node or changing kernel memory settings.
1. Establish what actually terminated
Linux SIGKILL is signal 9. Bash conventionally reports a command killed by signal N as 128 plus N, giving 137 for SIGKILL. That convention does not identify who sent the signal, and an application can explicitly exit with the same number. Correlate the runtime-reported reason and timestamps instead of diagnosing from arithmetic alone.
HELP_NS='REPLACE_NAMESPACE'
HELP_POD='REPLACE_POD'
HELP_CONTAINER='REPLACE_CONTAINER'
kubectl config current-context
kubectl -n "$HELP_NS" get pod "$HELP_POD" -o jsonpath='{.metadata.uid}{"\n"}{.spec.nodeName}{"\n"}{.status.containerStatuses}{"\n"}{.status.initContainerStatuses}{"\n"}'
kubectl -n "$HELP_NS" get pod "$HELP_POD" -o jsonpath='{range .spec.containers[*]}{.name}{"\t"}{.resources}{"\n"}{end}'
kubectl -n "$HELP_NS" describe pod "$HELP_POD"
kubectl -n "$HELP_NS" get pod "$HELP_POD" -o jsonpath='{.spec.resources}{"\n"}'Compare reason, exit code, finish time, events, and resources for the same container. Include init-container resources separately if that is the failing component. The final query checks optional Pod-level resources; an empty field does not prove the cluster supports that feature.
kubectl -n "$HELP_NS" logs "$HELP_POD" -c "$HELP_CONTAINER" --previous --timestamps --tail=100
kubectl -n "$HELP_NS" top pod "$HELP_POD" --containersTop needs a working Metrics API and permission. It reports a recent working-set sample, not the previous process's maximum usage. A metrics error is missing evidence, not a healthy reading.
2. Separate container, Pod, and node boundaries
Memory limits are enforced reactively through the Linux kernel. Requests instead participate in scheduling; increasing a limit alone does not reserve equivalent node capacity. Review memory-backed emptyDir use, workload concurrency, cache growth, and application heap settings alongside ordinary process memory.
Where the installed release and feature configuration support PodLevelResources, inspect the optional Pod-level memory budget too. A container-only view can miss that shared boundary. Ask the platform owner to confirm which budgets are active before changing a container limit.
| What you observe | What it suggests | Next step |
|---|---|---|
| OOMKilled coincides with the container approaching its configured limit. | A container-level memory boundary is a strong candidate. | Check incident-time monitoring and application allocation behavior. Reduce unnecessary concurrency or cache growth, fix a leak, or size an approved request/limit pair using measured demand and node headroom. |
| The Pod declares a memory budget shared by its containers. | A supported Pod-level boundary may be relevant in addition to individual container limits. | Correlate aggregate Pod usage, individual container usage, and runtime evidence with the active budgets; do not assume a per-container limit increase resolves a shared budget. |
| Several workloads fail on the same node with memory-pressure or kernel OOM evidence. | The incident may involve node-wide contention, not just one undersized limit. | Escalate to the node owner to compare allocatable capacity, reservations, and competing usage. Do not increase every limit on an already constrained node. |
| Only exit 137 is known, with no matching OOM reason or memory evidence. | The cause remains unresolved; forced termination is also possible. | Review termination timing, shutdown grace periods, runtime events, and authorized node logs before labeling the incident an OOM. |
3. Correct the cause and retain a rollback path
On cgroup v2, an operator can read the verified container cgroup's memory.events and memory.events.local. The former includes descendant events; counters are cumulative, and oom_kill can reflect any OOM killer. Compare the same cgroup over the relevant time window. A replaced container may have a different cgroup, so a zero counter there does not erase the earlier incident.
Choose one approved workload change and record its previous configuration. A limit increase needs capacity and admission checks; application tuning needs a realistic load test. If reverting, restore a compatible configuration without deliberately forcing the old failure under production load. Keep kernel OOM protection enabled.
Verify the fix
- Observe memory across startup, peak traffic, and scheduled work; confirm no new matching OOM terminations or restart increments.
- Check that the revised request still schedules and the node has headroom; absence of one container kill is not enough if neighboring workloads deteriorate.
- Verify correct application results and latency. If demand keeps rising, treat capacity changes as mitigation while the allocation defect is investigated.
Record the evidence, approved change, result, and rollback plan so the next administrator can follow your reasoning.
Avoid these shortcuts
- Do not diagnose OOM solely from 137 or a single kubectl top sample.
- Do not disable the OOM killer, remove every limit, or raise limits without reviewing capacity.
- Do not guess cgroup paths, expose node logs publicly, or confuse memory-backed emptyDir with ordinary disk usage.
Primary sources and version checks
Check the documentation for your installed release. Provider, runtime, CNI, storage-driver, and distribution details can differ.