Linux troubleshooting: the commands to run when a server misbehaves

When a server misbehaves, many people panic and restart things. A calmer way is to look first: a short sequence of read-only commands tells you whether the problem is CPU, memory, disk, a crashed process, a closed port or the network. All commands below only read information, so they are safe to run on a live server. The sample outputs are illustrative: your numbers will differ. The commands themselves were checked on a Linux server before publishing (lsof is not installed by default on every system).

The order that saves time

1. Is the server up, and is it overloaded? (uptime)
2. Is it out of memory? (free -h)
3. Is the disk full? (df -h, du)
4. What is using the resources? (top, ps)
5. Is the app listening on the port? (ss -ltnp)
6. What do the logs say? (journalctl)
7. Is the network the problem? (ping, curl, dig)

1. Load: uptime

$ uptime
 03:08:23 up 30 days, 18:00,  4 users,  load average: 0.72, 0.61, 0.42

The three load averages are for the last 1, 5 and 15 minutes. Compare them to the number of CPU cores (nproc). On a 4-core machine, a load of 4 means fully busy, and a load far above 4 means processes are queuing. If the 1-minute number is much higher than the 15-minute one, the problem is new. High load can also come from processes waiting on a slow disk, not only from CPU.

2. Memory: free -h

$ free -h
               total        used        free      shared  buff/cache   available
Mem:           5.8Gi       3.0Gi       493Mi        46Mi       2.6Gi       2.8Gi

Do not panic about a small free number. Linux uses spare memory as a cache (buff/cache) and gives it back when needed. Look at available: it is the estimate of memory new programs can use. If available is close to zero and swap is heavily used, you have a real memory problem. Check whether the kernel killed something: journalctl -k | grep -i "out of memory" (the OOM killer).

3. Disk: df -h and du

$ df -h /
Filesystem      Size  Used Avail Use% Mounted on
/dev/vda2        79G   56G   19G  75% /

$ sudo du -xh /var --max-depth=1 | sort -h | tail -5

A full disk causes strange failures: databases refuse writes, logs stop, deployments fail. df shows each filesystem; du finds which folder is big (-x stays on one filesystem, sort -h sorts human-readable sizes). Common culprits: logs, Docker images and old backups. Also check inodes with df -i: you can run out of file entries while space remains.

4. Processes: top and ps

$ ps aux --sort=-%mem | head -5
$ ps aux --sort=-%cpu | head -5
$ top          # press M to sort by memory, P by CPU, q to quit

These list the top memory and CPU users. Note the PID (process id) and the command. In top, the wa value in the CPU line is time spent waiting for disk. A process in state D is stuck waiting on I/O, and one in state Z is a zombie. Do not kill -9 first. Try a normal kill PID (SIGTERM) so the program can shut down cleanly, and use -9 only if it does not respond.

5. Ports: ss

$ sudo ss -ltnp
State  Recv-Q Send-Q Local Address:Port  Peer Address:Port  Process
LISTEN 0      4096         0.0.0.0:443         0.0.0.0:*    users:(("nginx",pid=812))
LISTEN 0      4096       127.0.0.1:3000        0.0.0.0:*    users:(("node",pid=2056))

-l is listening sockets, -t TCP, -n numeric, -p the process. If your app should be on port 3000 and is missing, it is not running or it crashed. Also read the address: 127.0.0.1:3000 is reachable only from this machine, while 0.0.0.0:3000 is reachable from outside. A database listening on 0.0.0.0 should be protected by a firewall or bound to localhost. This quick check is also a security habit: know what is listening on your server.

6. Logs: journalctl

$ journalctl -u myapp -n 100 --no-pager     # last 100 lines for one service
$ journalctl -u myapp -f                    # follow live
$ journalctl -p err --since "1 hour ago"    # only errors in the last hour
$ systemctl status myapp                    # state, exit code and recent log lines

Logs usually name the real cause: a missing environment variable, a refused database connection, a permission error. Read the first error after a crash, not only the last lines. For files outside systemd, use tail -f /var/log/nginx/error.log.

7. Network: ping, curl, dig

$ curl -I https://example.com               # headers and status code
$ curl -sS -o /dev/null -w "%{http_code} %{time_total}s\n" https://example.com
$ dig +short example.com                     # does DNS resolve?
$ ip -brief addr                             # interfaces and addresses

Separate the layers: if dig fails, it is DNS; if DNS works but curl times out, it is routing or a firewall; if curl connects but returns 502, the app behind the proxy is down. ping can be blocked on purpose, so a failed ping alone proves little.

A simple decision guide

Symptom	Check first
Whole server slow	uptime, top, free -h
Site returns 502 or 503	systemctl status, journalctl -u, ss -ltnp
Cannot write files or database fails	df -h, df -i
App randomly killed	journalctl -k for out of memory
Works locally, not from outside	ss -ltnp address, firewall, curl from outside
Slow after a deploy	logs, then compare with the previous version

Habits that help

• Change one thing at a time and note it.
• Do not restart as the first step. A restart erases evidence such as memory state and the process list. Read logs first.
• Save useful output before you fix: journalctl -u myapp --since "1 hour ago" > issue.log.
• Set up monitoring and alerts so you find problems before users do.
• Write a short runbook for each service: how to check it, restart it and roll it back.

Takeaway: troubleshoot in layers, read-only first: load, memory, disk, processes, ports, logs, network. The answer is usually in the logs, and a calm checklist beats a panicked restart.

---

Linux & DevOps series, Part 3 (everyday operations): 2 of 5. Previous: Docker Compose: run a multi-container app with one file.