Before you start
Kubernetes Linux nodes; Bash and GNU df/du examples. Host inspection requires an authorized node session. Filesystem layout, eviction thresholds, and runtime support vary by installation. Documentation-reviewed on 2026-09-07.
Commands are examples, not actions run by this page. Replace the sample names and paths. Run related blocks in the same shell. Read the explanations before copying. Logs and configuration can contain private data; keep credentials and unredacted evidence out of public posts.
Validation: documentation-reviewed guidance, not a tested live-cluster repair.
First use authorized API reads. The later filesystem card belongs on the identified Linux node, not your laptop or inside an arbitrary container. Obtain node-owner approval and select a verified path on the pressured filesystem. No cleanup, drain, or disk expansion is performed by these examples.
1. Establish scope and symptoms
Record when the condition changed and which workloads became unavailable. Compare node conditions with Pod eviction or scheduling messages. A disk-pressure taint can explain why another Pod stays Pending, but manually removing that taint does not remove the underlying storage shortage.
HELP_NODE='REPLACE_AFFECTED_NODE'
kubectl config current-context
kubectl get node "$HELP_NODE" -o jsonpath='{.status.conditions}{"\n"}{.spec.taints}{"\n"}'
kubectl describe node "$HELP_NODE"
kubectl get pods --all-namespaces --field-selector "spec.nodeName=$HELP_NODE" -o wideCross-namespace and node reads require suitable permission. If denied, share the incident target with the platform owner rather than obtaining broader access yourself.
2. Measure the correct filesystem
The kubelet monitors nodefs and can monitor image or container filesystem signals, depending on the runtime and version. Discover the actual layout and thresholds. Different signal names need not mean separate disks.
HELP_FS_PATH='REPLACE_ABSOLUTE_PATH_ON_VERIFIED_PRESSURED_FILESYSTEM'
HELP_REVIEW_DIR='REPLACE_APPROVED_NARROW_DIRECTORY_ON_THAT_FILESYSTEM'
df -hT -- "$HELP_FS_PATH"
df -i -- "$HELP_FS_PATH"
du -x -h --max-depth=1 -- "$HELP_REVIEW_DIR"Run only in the authorized node session. df distinguishes block usage from inodes; du estimates a scoped directory and stays on one filesystem. A large directory walk can add I/O, and permission failures make its result incomplete.
| What you observe | What it suggests | Next step |
|---|---|---|
| Available blocks are low while inode availability remains reasonable. | Large logs, images, writable layers, or other files may be consuming capacity. | Identify the growing owner and retention requirements. Correct excessive generation or approved rotation, or plan a supported capacity increase; do not guess which runtime files are disposable. |
| Inodes are exhausted even though substantial byte capacity remains. | Many filesystem objects can prevent new file creation without filling the disk by bytes. | Investigate the responsible application's small-file behavior and retention. Approve any exact cleanup set with its data owner, or plan suitable filesystem capacity. |
| One filesystem looks healthy but DiskPressure remains true. | You may be inspecting the wrong filesystem, stale evidence, or a condition transition delay. | Match kubelet/runtime signals to actual mounts and timestamps. Check the configured transition behavior before assuming a stuck condition or disabling thresholds. |
3. Recover capacity without corrupting the runtime
Kubelet attempts node-level reclamation before evicting end-user Pods, with behavior depending on the filesystem layout. If reclamation is not sufficient, a node owner should review why: ongoing growth, constrained capacity, failed garbage collection, or unrelated host consumers. Do not manipulate container-runtime storage behind its back.
Container stdout/stderr rotation is managed through kubelet and runtime integration; arbitrary application files and every host log are not automatically covered by that policy. Reduce excessive logging at its source and review retention with the data owner. Keep necessary incident evidence before an approved cleanup.
Record configuration changes and prior values. Capacity expansion may add cost and may not support shrinking. For approved cleanup, define exact targets and recovery separately. Drain or replacement requires workload-disruption and storage checks; it is not a harmless diagnostic shortcut.
Verify the fix
- Recheck both available bytes and inodes on the previously pressured filesystem; maintain headroom above the actual configured thresholds.
- Observe DiskPressure clearing after the configured transition behavior, and confirm new evictions and related scheduling failures stop.
- Check workload readiness, application data, and the growth rate over representative operation. Revert an unsuccessful logging/configuration change through the node owner; retain evidence of any irreversible action.
Record the evidence, approved change, result, and rollback plan so the next administrator can follow your reasoning.
Avoid these shortcuts
- Do not recursively delete /var/lib/kubelet, containerd, CRI-O, Docker, volume, or unknown log directories.
- Do not run blanket image pruning, forced drains, or PVC deletion to make an alert disappear.
- Do not disable eviction thresholds or remove a pressure taint while the cause persists.
Primary sources and version checks
Check the documentation for your installed release. Provider, runtime, CNI, storage-driver, and distribution details can differ.