Before you start
GNU/Linux with GNU coreutils; lsof is optional. Inspect only systems you administer.
Commands are examples, not actions run by this page. Replace the sample names and paths. Run related blocks in the same shell. Read the explanations before copying. Logs and configuration can contain private data; keep credentials and unredacted evidence out of public posts.
Validation: documentation-reviewed guidance, not a tested live-cluster repair.
An application cannot write its report. Free space on your home filesystem does not help if its output directory lives on a different, full mount. Start with the actual destination, not the largest disk in the server.
1. Identify the exhausted resource
Replace the example with the existing directory where writes fail. A path-specific check avoids confusing the root filesystem with a separate data volume. A quota failure can also require investigation even when the filesystem has room.
TARGET="/var"
df -hT -- "$TARGET"
df -i -- "$TARGET"The first report shows capacity and filesystem type. The second shows inode usage. Keep errors visible; an inaccessible path is not evidence of free space.
| What you observe | What it suggests | Next step |
|---|---|---|
| Block usage is near capacity | Allocated data is consuming the filesystem. | Locate the growing owner directory and agree on retention or expansion. |
| Inodes are exhausted, but bytes remain | Very many filesystem objects may be exhausting the inode supply. | Investigate small-file caches, queues, and retention using the owning application. |
| Neither measurement is full | You may have the wrong path, a quota, or a different mount namespace. | Check the application identity and its actual mount before changing storage. |
2. Follow the space without crossing other filesystems
A directory scan can generate substantial I/O on a busy server. Start narrow, run once, and expand only if needed. du and df measure different things; do not expect every total to match. Permission errors also make a du result incomplete.
sudo du -xhd1 -- "$TARGET"GNU du stays on the same filesystem and reports one level of directories. Descend into the relevant directory rather than scanning every mount.
sudo lsof +L1Optional lsof lists open files with link counts below one. Match the device and process to the affected mount; the list may include other filesystems.
3. Recover through the owner, not a deletion recipe
For a runaway log, use the service’s documented rotation and retention process. For an open deleted file, coordinate a supported log reopen or a controlled restart of the owning service; a restart has availability impact and is not automatically safe.
For growing application data, confirm a tested backup and retention requirements before archiving or removing records through its supported tooling. Expanding a cloud disk, partition, volume, and filesystem are separate operations; use the procedure for the actual storage layout.
Record the selected files, process, mount, and recovery plan. Deletion has no universal undo. If the data owner cannot authorize removal, stop and provision capacity through the normal change process.
Verify the fix
- Repeat df for the same path and check both blocks and inodes.
- Have the application owner verify the original write operation, without overwriting an existing important file.
- Watch whether capacity stabilizes; document the growth source and retention change.
Record the evidence, approved change, result, and rollback plan so the next administrator can follow your reasoning.
Avoid these shortcuts
- Do not run a broad rm command against /var, a database directory, or container storage.
- Do not truncate a process file descriptor to reclaim space.
- A successful restart is not proof that the underlying growth problem is fixed.
Primary sources and version checks
Check the documentation for your installed release. Provider, runtime, CNI, storage-driver, and distribution details can differ.