Before you start
Linux-node Kubernetes clusters; Bash examples. Check your installed Kubernetes minor version and workload controller. Documentation-reviewed on 2026-09-07; not executed against a cluster.
Commands are examples, not actions run by this page. Replace the sample names and paths. Run related blocks in the same shell. Read the explanations before copying. Logs and configuration can contain private data; keep credentials and unredacted evidence out of public posts.
Validation: documentation-reviewed guidance, not a tested live-cluster repair.
Replace every REPLACE value before running these Bash examples. Keep the same shell between cards, confirm the context, and stop if access is denied. A multi-container Pod can have one failing container while another works. Record the Pod UID and incident time so a replacement Pod is not mistaken for recovery.
1. Capture the failing container
Record namespace, Pod, container, UID, restart count, last termination, and failure time. Save sanitized evidence. Empty previous logs may mean no retained previous instance exists; useful evidence may remain elsewhere.
HELP_NS='REPLACE_NAMESPACE'
HELP_POD='REPLACE_POD'
HELP_CONTAINER='REPLACE_CONTAINER'
kubectl config current-context
kubectl version --output=yaml
kubectl -n "$HELP_NS" get pod "$HELP_POD" -o wide
kubectl -n "$HELP_NS" get pod "$HELP_POD" -o jsonpath='{.metadata.uid}{"\n"}{.metadata.ownerReferences}{"\n"}{.status.containerStatuses}{"\n"}{.status.initContainerStatuses}{"\n"}'Owner references help identify the managed workload. Inspect init-container status too when initialization is failing. This reads API state without replacing a Pod.
kubectl -n "$HELP_NS" logs "$HELP_POD" -c "$HELP_CONTAINER" --previous --timestamps --tail=100
kubectl -n "$HELP_NS" logs "$HELP_POD" -c "$HELP_CONTAINER" --timestamps --tail=100
kubectl -n "$HELP_NS" describe pod "$HELP_POD"Match log times with termination and probe events. Describe output and logs can contain sensitive configuration; do not post them unredacted.
2. Match the evidence to the failure
The same status can hide different incidents. Treat a branch below as a hypothesis until the timestamps and the affected container agree. Inspect the reviewed workload manifest for its entrypoint, mounted configuration, dependency settings, and probes; do not print Secret values to investigate a missing reference.
| What you observe | What it suggests | Next step |
|---|---|---|
| Previous logs show a configuration parse error immediately before termination. | The application started, then rejected its input. | Have the application owner correct the specific configuration or reference through the managed manifest. Confirm that the replacement value is supported by the deployed application version. |
| Events report failed startup or liveness checks followed by a restart. | A probe may be killing an unhealthy process, or its configuration may not match valid startup behavior. | Compare endpoint, port, timeout, and measured startup time. Use an approved startup-probe budget for genuinely slow initialization; do not merely disable health checks. |
| The container exits successfully but is repeatedly restarted. | The command may finish normally while its restart policy expects a continuing service. | Confirm the intended job versus service design. Review a Job for finite work or correct the service entrypoint; a long sleep only hides the mismatch. |
| The last termination reason is OOMKilled. | Memory investigation is needed, not just a longer restart delay. | Follow the OOM guide and correlate limits, workload demand, and node evidence before resizing. |
3. Make one attributable correction
Readiness failure alone does not restart a container; it changes readiness and normal Service traffic eligibility. Startup and liveness checks have different purposes. A failed lifecycle hook is another possible cause, so retain hook-related events instead of assuming every kill came from liveness.
Apply an approved correction to the owning Deployment, StatefulSet, Job, or GitOps source, not an expendable Pod. Record the exact old and new revision, expected recovery window, and rollback owner. For a Deployment, an approved previous image or template may be recoverable; assess database/schema compatibility before rolling an application backward. Backoff timing can vary with version and kubelet configuration.
Verify the fix
- Repeat the same state and log checks against the current Pod UID; verify the intended image and configuration, not merely a changed Pod name.
- Observe restart counts over a representative startup and request cycle. A momentary Running phase is insufficient.
- Confirm readiness and the application's expected behavior. If the correction fails, restore the documented compatible revision through the controller owner and investigate the new evidence.
Record the evidence, approved change, result, and rollback plan so the next administrator can follow your reasoning.
Avoid these shortcuts
- Do not repeatedly delete Pods, hide the problem with sleep, or run an unreviewed rollout restart.
- Do not treat exit code 137 as conclusive OOM evidence.
- Do not add an ephemeral debug container as if it were a read-only query; it changes the Pod.
Primary sources and version checks
Check the documentation for your installed release. Provider, runtime, CNI, storage-driver, and distribution details can differ.