Journalctl Cheat Sheet

Configuration

Permissions

To see messages from other users and the system as well as performing various log operations from a regular user add it to the group:

sudo usermod -aG systemd-journal $USER

Persisting logs

If the directory /var/log/journal is present the systemd journal will be stored there, thus enabling a persistent journal across reboots.

sudo mkdir -p /var/log/journal
sudo chown root:systemd-journal /var/log/journal
sudo chmod 2755 /var/log/journal
sudo killall -USR1 systemd-journald

Example:

tree /var/log/journal

Should output something like this:

/var/log/journal
└── fe82e4aed51e4c1db953293110eca983
    ├── system.journal
    └── user-1000.journal

N.B. The following command should show current storage configuration:

# By deafault it is “Storage=auto”
cat /etc/systemd/journald.conf | grep Storage

Another way to enable persisten logging is to set the Storage option to persistent:

[Journal]
Storage=persistent

Base usage

Show all journal entries:

journalctl

To display the timestamps in UTC, you can use the --utc flag:

journalctl --utc

See boot-related messages from the current boot:

journalctl -b

See boot messages from N boots ago use -N, e.g.:

journalctl -b -2

This requires a persistent journal to be configured. For more see “Persisting the journal”.

To see the boots that systemd journal knows about, use the --list-boots option:

journalctl --list-boots

You can also use the boot ID to call back the data from a boot:

journalctl -b 7e4e564534f5477d8eabe7b4886d42a0

Jump to the end of the systemd journal (-e), and enable “follow” mode (-f):

journalctl -ef

Show all fields stored in the systemd journal with their field name and contents:

journalctl -o verbose

To print most recent M journal entries use -nM, e.g.:

journalctl -n10

Examples of querying

Displaying kernel messages:

journalctl -k

Display all messages in the systemd journal with a priority in the range emerg up to and including err:

journalctl -p emerg..err

It is possible to use either the priority name or its corresponding numeric value. In order of highest to lowest priority, these are:

0: emerg
1: alert
2: crit
3: err
4: warning
5: notice
6: info
7: debug

Display all messages filtered by the executable path (/usr/bin/gnome-shell):

journalctl /usr/bin/gnome-shell

Display all messages generated by the avahi-daemon.service systemd unit:

journalctl _SYSTEMD_UNIT=avahi-daemon.service

Display all messages generated by current user ID (id -u):

journalctl _UID=$(id -u)

Display all messages generated by process ID:

journalctl _PID=1115

Display all of the entries that involve the executable in question:

journalctl /bin/bash

Display all messages generated by, and about, the avahi-daemon.service systemd unit:

journalctl -u avahi-daemon.service

The next command “follows” the mysql daemon:

journalctl -u mysql.service -f

Specify more than one unit source:

journalctl -u openvpn.service -u sshd.service

Displays all messages between two dates:

journalctl --since "2019-07-05 21:30:01" --until "2019-07-05 21:30:02"

Printing the last 50 messages logged within the last hour:

journalctl -n50 --since "1 hour ago"

To show ournal entries in reverse chronological order use -r. For example next command shows the last 15 messages from the sshd daemon, listed in reverse order:

journalctl -u openvpn.service -r -n 15

Output Formats

The -o parameter specefies format of the journalctl output.

short-precise: The default format with microsecond precision
short: Is the default output format. It shows messages in syslog style
short-monotonic: Is similar to short, but the time stamp second value is shown with precision
short-iso: The default format augmented to show ISO 8601 wallclock timestamps
json: Will show each journal entry in json format in one long line
json-pretty: Will show each log entry in easy-to-read json format
json-sse: JSON formatted output wrapped to make add server-sent event compatible
export: A binary format suitable for transferring or backing up
verbose: Will show very detailed information for each journal record with all fields listed
cat: Shows messages in very short form, without any date/time or source server names

Example 1:

journalctl -u avahi-daemon.service -n 1 -o json-pretty

Output 1:

{
       "__CURSOR" : "s=f4eb8c1ac5174e75b701df381cb15c52;i=710b;b=3ee8dc71cb1b442c9ad479288f2acf42;m=16ee39e;t=58cf60b544c83;x=7ddbe5fb422a6a43",
       "__REALTIME_TIMESTAMP" : "1562363263470723",
       "__MONOTONIC_TIMESTAMP" : "24044446",
       "_BOOT_ID" : "3ee8dc71cb1b442c9ad479288f2acf42",
       "_MACHINE_ID" : "fe82e4aed51e4c1db953293110eca983",
       "_HOSTNAME" : "tower",
       "PRIORITY" : "6",
       "SYSLOG_FACILITY" : "3",
       "_SYSTEMD_SLICE" : "system.slice",
       "_TRANSPORT" : "syslog",
       "SYSLOG_IDENTIFIER" : "avahi-daemon",
       "_COMM" : "avahi-daemon",
       "_EXE" : "/usr/sbin/avahi-daemon",
       "_SYSTEMD_CGROUP" : "/system.slice/avahi-daemon.service",
       "_SYSTEMD_UNIT" : "avahi-daemon.service",
       "_UID" : "113",
       "_GID" : "119",
       "_CAP_EFFECTIVE" : "0",
       "_CMDLINE" : "avahi-daemon: running [tower.local]",
       "_PID" : "1115",
       "MESSAGE" : "Service \"FTP file sharing on tower\" (/services/ftp.service) successfully established.",
       "SYSLOG_PID" : "1115",
       "_SYSTEMD_INVOCATION_ID" : "42bac8d872d84693b33128f1efa9c034",
       "_SOURCE_REALTIME_TIMESTAMP" : "1562363263470149"
}

Example 2:

journalctl -u avahi-daemon.service -n 10 -o json | jq .MESSAGE

Output 2:

"Network interface enumeration completed."
"Registering new address record for fe80::9e5c:8eff:fe00:e5a3 on eno1.*."
"Registering new address record for 192.168.1.241 on eno1.IPv4."
"Started Avahi mDNS/DNS-SD Stack."
"Joining mDNS multicast group on interface docker0.IPv4 with address 172.17.0.1."
"New relevant interface docker0.IPv4 for mDNS."
"Registering new address record for 172.17.0.1 on docker0.IPv4."
"Server startup complete. Host name is tower.local. Local service cookie is 1451721814."
"Service \"tower\" (/services/ssh.service) successfully established."
"Service \"FTP file sharing on tower\" (/services/ftp.service) successfully established."

Truncate output (ellipsize fields)

journalctl --no-full

Output to standard output:

journalctl --no-pager

Maintenance

Disk usage

Current disk usage:

journalctl --disk-usage

Output example:

Archived and active journals take up 1.3G in the file system.

Use the --vacuum-size option to shrink journal by indicating a size:

journalctl --vacuum-size=1G

Output example:

Deleted archived journal /var/log/journal/5811f5275de24e19ac3c1a62f2b2e27d/[email protected]~ (24.0M).
Deleted archived journal /var/log/journal/5811f5275de24e19ac3c1a62f2b2e27d/user-1000@93efc46bbe3e4af2a0fc718a4564a79d-0000000000000b54-00058d33d3695c54.journal (16.0M).
Deleted archived journal /var/log/journal/5811f5275de24e19ac3c1a62f2b2e27d/system@522e24407a374023974d0083e68724cf-0000000000000001-00058d3e56f917ea.journal (8.0M).
Deleted archived journal /var/log/journal/5811f5275de24e19ac3c1a62f2b2e27d/[email protected]~ (24.0M).
Deleted archived journal /var/log/journal/5811f5275de24e19ac3c1a62f2b2e27d/user-1000@93efc46bbe3e4af2a0fc718a4564a79d-0000000000006285-00058d3e5ed217bf.journal (16.0M).
Deleted archived journal /var/log/journal/5811f5275de24e19ac3c1a62f2b2e27d/system@4f347a49f0d241ea9cb17432b605681d-0000000000000001-00058d6162219f66.journal (16.0M).
Deleted archived journal /var/log/journal/5811f5275de24e19ac3c1a62f2b2e27d/system@4f347a49f0d241ea9cb17432b605681d-0000000000010833-00058d6b1308d58a.journal (16.0M).
Deleted archived journal /var/log/journal/5811f5275de24e19ac3c1a62f2b2e27d/user-1000@93efc46bbe3e4af2a0fc718a4564a79d-000000000001083a-00058d6b131a116d.journal (8.0M).
Deleted archived journal /var/log/journal/5811f5275de24e19ac3c1a62f2b2e27d/system@4f347a49f0d241ea9cb17432b605681d-0000000000013342-00058d6c7d131fc4.journal (88.0M).
Deleted archived journal /var/log/journal/5811f5275de24e19ac3c1a62f2b2e27d/[email protected]~ (24.0M).
Deleted archived journal /var/log/journal/5811f5275de24e19ac3c1a62f2b2e27d/user-1000@7639a87d761f474ab3433bedcd1e1c02-0000000000028bf9-00058de308fa47f6.journal (56.0M).
Deleted archived journal /var/log/journal/5811f5275de24e19ac3c1a62f2b2e27d/system@4f347a49f0d241ea9cb17432b605681d-0000000000036f7e-00058e19ba8e5817.journal (16.0M).
Deleted archived journal /var/log/journal/5811f5275de24e19ac3c1a62f2b2e27d/user-1000@7639a87d761f474ab3433bedcd1e1c02-0000000000036f87-00058e19bba7343d.journal (8.0M).
Deleted archived journal /var/log/journal/5811f5275de24e19ac3c1a62f2b2e27d/[email protected]~ (32.0M).
Vacuuming done, freed 352.0M of archived journals from /var/log/journal/5811f5275de24e19ac3c1a62f2b2e27d.

Keep entries from the last year, you can type:

journalctl --vacuum-time=1years

Manage journal service

To control running journal service with systemd, use the systemctl utility. This utility is similar to the service utility provided by SysVinit and Upstart. Among others:

systemctl status systemd-journald indicates whether the service is running and additional information if it is
systemctl start systemd-journald starts the service (systemd unit)
systemctl stop systemd-journald stops the service
systemctl restart systemd-journald restarts the service
systemctl reload systemd-journald reloads the service's configuration if possible, but will not kill it (so no risk of a service interruption or of disrupting processing in progress, but the service may keep running with a stale configuration)
systemctl force-reload systemd-journald reloads the service's configuration if possible, and if not restarts the service (so the service is guaranteed to use the current configuration, but this may interrupt something)

References

man 1 journalctl - Query the systemd journal
man 7 systemd.journal-fields - Special journal fields
man 5 journald.conf - Journal service configuration files

sergeyklay/journalctl-cheatsheet.md