Before you start
Linux hosts using systemd. Containers without systemd as PID 1 need their container or application logs instead.
Commands are examples, not actions run by this page. Replace the sample names and paths. Run related blocks in the same shell. Read the explanations before copying. Logs and configuration can contain private data; keep credentials and unredacted evidence out of public posts.
Validation: documentation-reviewed guidance, not a tested live-cluster repair.
A service works from a shell but fails at boot. A service manager does not necessarily inherit your shell’s working directory, environment, permissions, or startup sequence. Repeated restarts can obscure the first useful clue.
1. Capture state and the relevant journal
Choose the exact unit name; sshd.service and ssh.service are not interchangeable across every distribution. Service status can return a nonzero exit code because the service is inactive or failed. Read its output rather than treating that status as a broken diagnostic.
UNIT="example.service"
systemctl --failed --no-pager
systemctl status "$UNIT" --no-pager -l
systemctl show "$UNIT" -p Result -p ExecMainCode -p ExecMainStatus
sudo journalctl -u "$UNIT" -b --no-pager -n 100The unit result and the application’s journal messages answer different questions. Use an incident time window when the last 100 messages omit the original failure.
systemctl cat "$UNIT"Review the unit and drop-ins locally. Unit files and logs can contain credentials; redact before sharing.
2. Match the error to a narrow hypothesis
Look immediately before the final failed state. Use the application’s own configuration validator where one exists; do not invent a generic syntax-test flag.
| What you observe | What it suggests | Next step |
|---|---|---|
| Executable or execution failure | The configured executable, path access, interpreter, or execution restrictions need inspection. | Compare ExecStart with the installed file and service identity; use the permission guide for path access. |
| Application reports an invalid setting | The program started far enough to reject configuration. | Validate a proposed correction with the application’s supported test command before restarting. |
| Address already in use | Another listener may own the required socket. | Identify the listener and intended ownership before changing a port or stopping a process. |
| Start-limit or repeated-start warning | systemd has limited repeated starts. | Fix the earlier failure first; clearing the failed state alone does not repair it. |
3. Apply a reviewed fix and verify recovery
Keep a known-good configuration and change only the failing setting. Use a drop-in rather than editing a vendor unit in place where appropriate. A unit-file change needs daemon-reload; an application configuration change may instead need the application’s documented reload or restart.
Once the fix is approved, arrange any required maintenance window. The following are separate administrative actions, not an automatic recipe. Reset the failure counter only after correcting the cause. Retain a rollback to the previous valid configuration.
sudo systemctl daemon-reloadMakes systemd reread unit configuration. It does not restart the application.
sudo systemctl reset-failed "$UNIT"
sudo systemctl restart "$UNIT"Restart can interrupt users and work. Do not run during an incident without authorization and a recovery plan.
Verify the fix
- Check the unit’s state and fresh journal entries after the change.
- Test the application’s actual endpoint or job, not only whether its process is running.
- If startup at boot matters, review dependencies and validate a reboot only in an approved test window.
Record the evidence, approved change, result, and rollback plan so the next administrator can follow your reasoning.
Avoid these shortcuts
- Do not disable start limits just to hide recurring failures.
- Do not paste full environment values or unit secrets into a public issue.
- Do not kill an unknown process to free a port.
Primary sources and version checks
Check the documentation for your installed release. Provider, runtime, CNI, storage-driver, and distribution details can differ.