Log management on Linux: journalctl, logrotate and keeping disks safe

Logs tell you what happened when something broke. But logs have two problems: they are hard to read when there are too many, and they grow until the disk is full. Log management means three things: knowing where logs are, limiting how much space they use, and writing logs that are useful. The logrotate demo below was run (logrotate 3.22, in a temporary container, nothing kept). The journalctl commands are standard and the read-only ones were checked on a Linux server.

Where logs live

• systemd journal (journald): logs from services started by systemd, stored in a binary format and read with journalctl.
• Plain files in /var/log: for example web server logs (/var/log/nginx/access.log, error.log), auth.log and syslog on Debian and Ubuntu.
• Your own app's files, wherever you configured them.
• Containers: docker logs <container> reads what a container wrote to standard output.

A good rule for apps: write logs to standard output and let the platform (systemd, Docker) collect them. Fewer files to manage.

Reading the journal: journalctl

journalctl -u nginx -n 50 --no-pager      # last 50 lines of one service
journalctl -u myapp -f                    # follow live
journalctl -p err -b                      # errors since this boot
journalctl --since "2026-10-28 02:00" --until "2026-10-28 03:00"
journalctl -k                             # kernel messages (out of memory kills, disk errors)
journalctl -u myapp -o json-pretty -n 1   # structured output

-p filters by priority (err, warning, info...). -b means "this boot". -u picks a unit (service). Combine them to narrow a problem quickly.

Limiting the journal's size

The journal can grow large. Check it:

$ journalctl --disk-usage
Archived and active journals take up 311.7M in the file system.

(That is the number from one real server; yours will differ.) To cap it, edit /etc/systemd/journald.conf:

[Journal]
SystemMaxUse=500M
MaxRetentionSec=1month

Then sudo systemctl restart systemd-journald. To free space right now: sudo journalctl --vacuum-size=200M or --vacuum-time=2weeks. Note that the journal is only kept across reboots if /var/log/journal exists or Storage=persistent is set, and this default differs between distributions, so check yours. Also decide before you need it how long you must keep logs, because security investigations often need weeks or months of history.

Rotating plain-text logs: logrotate

Files such as nginx or your own app logs are not handled by the journal. logrotate renames the current log, starts a new one, compresses old ones and deletes the oldest. A typical config, such as /etc/logrotate.d/myapp:

/var/log/myapp/*.log {
    daily               # or: size 100M
    rotate 7            # keep 7 old files
    compress            # gzip old files
    delaycompress       # compress from the 2nd rotation (safe for apps still writing)
    missingok           # no error if the log is missing
    notifempty          # do not rotate empty files
    copytruncate        # copy then empty the file; use if the app cannot reopen its log
}

Proof it works

I ran logrotate in a throwaway container with a deliberately tiny size limit, adding 500 lines before each run:

(size 1k, rotate 3, compress, delaycompress, copytruncate; each run adds 500 lines first)

after run 1:  app.log  app.log.1
after run 2:  app.log  app.log.1  app.log.2.gz
after run 3:  app.log  app.log.1  app.log.2.gz  app.log.3.gz
after run 4:  app.log  app.log.1  app.log.2.gz  app.log.3.gz      <- the oldest was deleted
after run 5:  app.log  app.log.1  app.log.2.gz  app.log.3.gz

The newest old file stays uncompressed for one cycle (delaycompress), older ones are gzipped, and after the third rotation the oldest file is deleted, so the total number of files stays fixed at rotate 3. This is how space stays under control.

Choices that matter

• copytruncate vs signalling the app. Many programs keep the log file open. If logrotate renames it, the program keeps writing to the renamed file. Either tell the app to reopen its log in a postrotate script (nginx: kill -USR1 on the master process, or nginx -s reopen), or use copytruncate, which copies the file and then empties it. copytruncate is simpler, but a few lines written during the copy can be lost.
• daily or size: time-based for steady traffic, size-based for bursty logs.
• Test your config without changing anything: logrotate -d /etc/logrotate.d/myapp (debug, dry run) and force a real run with logrotate -f.
• Most distributions run logrotate daily from a systemd timer or cron. A config in /etc/logrotate.d/ is picked up automatically if logrotate is installed. Confirm that on your server with logrotate --version and systemctl list-timers | grep logrotate. Package defaults often assume it is present, and on a minimal server it may not be.

Writing logs worth reading

• Use levels: DEBUG, INFO, WARNING, ERROR. Log at INFO in production and turn on DEBUG only when investigating.
• Include context: a timestamp (UTC), the service name, a request id, the user id (not personal data), and what failed. "Error" alone is useless.
• One event per line, or better, structured logs (JSON) that tools can search by field.
• Never log secrets or sensitive data: passwords, tokens, full card numbers, session cookies. Logs get copied, shipped and read by many people.
• Do not log in tight loops without limits. One noisy line can fill a disk.
• Log errors where you handle them, with the stack trace, not at every layer. Duplicate logs hide the real one.

Centralising logs

On one server, journalctl and grep are enough. With several servers or containers, ship logs to a central place (Loki with Grafana, an ELK/OpenSearch stack, or a hosted service), so a crash does not take the logs down with the machine, and you can search across services. Pair it with alerts on error rates, as in the monitoring article.

A quick weekly checklist

1. df -h and journalctl --disk-usage: is space shrinking?
2. ls -lhS /var/log | head: is any file huge or never rotated?
3. Is logrotate installed and running (systemctl list-timers)?
4. Do logs contain secrets? Search a sample for password and token.
5. Can you find yesterday's error in under a minute?

Takeaway: read logs with journalctl, cap their size, rotate plain-text logs with logrotate (and check that it is actually installed), and write structured, secret-free logs that include context.

---

Linux & DevOps series, Part 3 (everyday operations): 5 of 5. Previous: Git branching strategies: trunk-based, GitHub Flow and Git Flow.