Linux Logs: journalctl, /var/log, rsyslog and logrotate

When a service crashes at night, the answer is almost always in the logs — the only question is where to look. In modern distributions (Ubuntu, Debian, RHEL, Fedora, Arch) the events are collected by the journald service and read with the journalctl command. In parallel, many programs write ordinary text files into /var/log, and logrotate keeps an eye on their size. Below is how to work with journalctl logs in Linux in practice, where the other journals live, and how not to lose important data during rotation.

Where logs are stored in Linux

There are three sources, and it helps to keep them in mind as a map:

• journald — the binary systemd journal. It accepts stdout and stderr of all services, kernel messages and syslog. It is stored in /var/log/journal/ (persistent) or /run/log/journal/ (until reboot).
• /var/log — text files. They are written by rsyslog (for example /var/log/syslog, /var/log/auth.log) or by the applications themselves: nginx, PostgreSQL, MySQL.
• The kernel ring buffer — messages from drivers and hardware, shown by dmesg or journalctl -k.

Most files in /var/log are accessible only to root or to the adm group, so reading commands often have to be run with sudo.

journalctl: basic commands

Without arguments journalctl prints the whole journal from the oldest entries, through the less pager. This is rarely needed — usually the selection is narrowed right away.

journalctl              # whole journal, from old to new
journalctl -e           # open right at the end
journalctl -n 50        # last 50 lines
journalctl -r           # reverse order, newest on top
journalctl --no-pager   # without less, handy for scripts and grep

Inside the pager the familiar less keys work: / — search, G — to the end, q — quit. Long lines are cut to the screen width; the left and right arrows scroll them.

Logs of a specific service: journalctl -u

The most common scenario is looking at what one systemd unit wrote. There is no need to find the file and grep it:

$ sudo journalctl -u nginx -n 20
Oct 14 03:12:41 web1 systemd[1]: Starting nginx.service - A high performance web server...
Oct 14 03:12:41 web1 nginx[1834]: nginx: [emerg] bind() to 0.0.0.0:80 failed (98: Address already in use)
Oct 14 03:12:41 web1 systemd[1]: nginx.service: Control process exited, code=exited, status=1/FAILURE
Oct 14 03:12:41 web1 systemd[1]: nginx.service: Failed with result 'exit-code'.

The -u flag can be repeated: journalctl -u nginx -u php8.2-fpm shows both services interleaved by time, which is handy when the error occurs at the boundary between them. For user units (systemctl --user) use journalctl --user -u name.

Logs in real time: journalctl -f

sudo journalctl -u nginx -f

This is the equivalent of tail -f: the command prints the last 10 entries and waits for new ones. It is convenient to keep it in a separate terminal while you restart the service or reproduce the error. Exit with Ctrl+C.

Filters by time and boot

If you know when the incident happened, narrow the selection by time with the --since and --until flags. They understand both exact dates and relative values:

journalctl --since "2026-10-14 02:50" --until "2026-10-14 03:30"
journalctl --since "1 hour ago"
journalctl --since yesterday --until today
journalctl -u nginx --since "10 min ago"

The second useful slice is by system boots. If the server rebooted by itself, the cause is looked for at the end of the previous session:

$ journalctl --list-boots
 -2 8f1c...  Sat 2026-10-12 09:01:12 UTC  Sun 2026-10-13 22:47:03 UTC
 -1 a7d2...  Sun 2026-10-13 22:48:30 UTC  Tue 2026-10-14 03:05:51 UTC
  0 c3e9...  Tue 2026-10-14 03:07:02 UTC  Tue 2026-10-14 09:15:40 UTC

$ journalctl -b -1 -e      # end of the previous boot
$ journalctl -b            # only the current boot

The list of boots is available only if the journal is stored on disk. How to enable that is described below, in the section on configuring journald.

Filter by severity: journalctl -p

Each entry has a syslog priority from 0 to 7. The -p flag shows the given level and everything more severe:

journalctl -p err -b              # all errors since the boot
journalctl -p warning..err -u ssh # a range of levels for one service

The command journalctl -p err -b is a good first check after a reboot or an update: within seconds it shows what went wrong.

Search and other filters

journald stores structured fields for each entry, and you can filter by them without grep:

journalctl _PID=1834                 # by process number
journalctl _COMM=sshd               # by program name
journalctl _UID=1000                 # by user
journalctl /usr/sbin/sshd            # by the path to the executable
journalctl -k                        # kernel messages only (like dmesg)
journalctl -t CRON                   # by syslog identifier
journalctl -g "Out of memory"        # text search, regular expression

The -g (--grep) flag appeared in systemd 237; on older systems use the usual pipeline journalctl --no-pager | grep .... Filters on different fields are combined with AND, so journalctl -u nginx -p err --since today returns only today's nginx errors.

Output formats

The -o flag changes the format. The most useful variants:

• -o short-iso — time in ISO 8601, convenient to compare with application logs.
• -o verbose — all fields of the entry, including _SYSTEMD_UNIT, _PID, _BOOT_ID.
• -o json or json-pretty — for processing by a script or jq.
• -o cat — only the message text, without date and host.

journalctl -u myapp -o json --since today | jq -r '.MESSAGE'

Configuring journald: storage and size

Settings are in /etc/systemd/journald.conf (overrides are better placed in /etc/systemd/journald.conf.d/*.conf). The main parameter is Storage:

• persistent — the journal is written to disk in /var/log/journal and survives a reboot.
• volatile — only in memory, in /run/log/journal.
• auto (default) — to disk if the directory /var/log/journal exists.

On some minimal images the directory does not exist, and after a reboot the history disappears. To enable permanent storage:

sudo mkdir -p /var/log/journal
sudo systemd-tmpfiles --create --prefix /var/log/journal
sudo systemctl restart systemd-journald

To keep the journal from taking over the whole disk, set limits:

# /etc/systemd/journald.conf.d/size.conf
[Journal]
Storage=persistent
SystemMaxUse=500M
MaxRetentionSec=1month

By default SystemMaxUse is 10% of the filesystem size, but not more than 4 GB.

How to clear the journalctl journal

$ journalctl --disk-usage
Archived and active journals take up 1.8G in the file system.

sudo journalctl --vacuum-size=300M   # keep no more than 300 MB
sudo journalctl --vacuum-time=2weeks # delete entries older than two weeks
sudo journalctl --rotate             # close the active files so they can be deleted too

It is not advisable to delete files from /var/log/journal with rm: vacuum does the same thing safely and does not touch the active file.

rsyslog: classic text logs

rsyslog is a daemon that accepts syslog messages (including those forwarded from journald) and sorts them into files according to rules. The configuration is in /etc/rsyslog.conf and /etc/rsyslog.d/*.conf. A rule consists of a selector source.level and an action:

# /etc/rsyslog.d/50-default.conf (excerpt)
auth,authpriv.*        /var/log/auth.log
*.*;auth,authpriv.none -/var/log/syslog
kern.*                 -/var/log/kern.log

A minus sign before the path means writing without syncing to disk after every line — faster, but in a crash the last lines may be lost. After changing the configuration, check the syntax and restart the service:

sudo rsyslogd -N1
sudo systemctl restart rsyslog

The main reason rsyslog is kept alongside journald is forwarding logs to a central server. One line sends everything over TCP:

# /etc/rsyslog.d/90-remote.conf
*.* @@logs.example.com:514

A single @ means UDP, two @@ mean TCP. If the server dies completely, a copy of the logs remains on another machine. To write your own message into the system log from a script, use logger:

logger -t backup -p user.err "Database backup failed"
journalctl -t backup -n 5

logrotate: log rotation

Text logs grow without limit until someone trims them. logrotate takes care of this: once a day (through cron or the logrotate.timer timer) it renames the current file, compresses the old ones and deletes the oldest. Global settings are in /etc/logrotate.conf, rules for programs are in /etc/logrotate.d/.

# /etc/logrotate.d/myapp
/var/log/myapp/*.log {
    daily
    rotate 14
    compress
    delaycompress
    missingok
    notifempty
    create 0640 myapp adm
    postrotate
        systemctl reload myapp > /dev/null 2>&1 || true
    endscript
}

Check a rule without changes and force a rotation:

sudo logrotate -d /etc/logrotate.d/myapp   # dry run with debug output
sudo logrotate -f /etc/logrotate.d/myapp   # rotate right now

A typical mistake is forgetting postrotate or copytruncate. Then the application keeps writing to the renamed app.log.1 through its old descriptor, while the new app.log stays empty.

Reading text logs

For files in /var/log the standard utilities are enough:

sudo tail -n 100 /var/log/syslog          # last lines
sudo tail -F /var/log/nginx/error.log     # follow, surviving rotation
sudo less +G /var/log/auth.log            # open at the end
sudo zgrep "Failed password" /var/log/auth.log.*.gz   # search in compressed archives
sudo grep -c " 500 " /var/log/nginx/access.log        # how many 500 responses

Note tail -F with a capital letter: unlike -f, it reopens the file when logrotate replaces it.

Step-by-step analysis of an incident

The scenario "the service crashed at night" is easiest to analyze in a fixed order:

1. systemctl status myapp — the current state and the last journal lines.
2. journalctl -u myapp --since "03:00" --until "03:30" — what the service wrote at the moment of the crash.
3. journalctl -p err -b — whether there are other errors in the system at the same time.
4. journalctl -k --since "03:00" -g "Out of memory|Killed process" — whether the kernel's OOM killer terminated the process.
5. The application's own logs in /var/log/myapp/, including the .gz archives.
6. If the server was rebooted — journalctl -b -1 -e.

Summary

For everyday work with logs in Linux a few commands are enough: journalctl -u for a service, -f for the stream, --since and -p err to narrow the selection, -b -1 to analyze a reboot. Text journals live in /var/log, rsyslog sorts and forwards them, and logrotate and journalctl --vacuum-* keep logs from filling the disk. Working with journalctl, rotation and load diagnostics is covered in practice in the Linux course lecture.