puppetctl "Unix syslog deliery error"

Getting this when running puppetctl status:

[db160024.bwi701:/root]# puppetctl status
2025/01/27 09:44:55 Unix syslog delivery error

Rsyslogd is running:

● rsyslog.service - System Logging Service
   Loaded: loaded (/usr/lib/systemd/system/rsyslog.service; enabled; vendor preset: enabled)
   Active: active (running) since Mon 2024-10-21 14:38:23 PDT; 3 months 6 days ago
     Docs: man:rsyslogd(8)
           https://www.rsyslog.com/doc/
 Main PID: 70437 (rsyslogd)
    Tasks: 4 (limit: 3298408)
   Memory: 45.1M
   CGroup: /system.slice/rsyslog.service
           └─70437 /usr/sbin/rsyslogd -n

There is a rule which directs output from puppet-agent to /var/log/puppet/puppet-agent.log:

[db160024.bwi701:/root]# cat /etc/rsyslog.d/puppet-agent.conf
# This file is managed by puppet. Local edits will be overwritten.
if $programname == 'puppet-agent' then /var/log/puppet/puppet-agent.log
& stop

However, /var/log/puppet/puppet-agent.log doesn't exist:

[db160024.bwi701:/root]# ls -l /var/log/puppet/puppet-agent.log
ls: cannot access '/var/log/puppet/puppet-agent.log': No such file or directory

Created the log file:

[db160024.bwi701:/root]# touch /var/log/puppet/puppet-agent.log
[db160024.bwi701:/root]# chmod 0600 /var/log/puppet/puppet-agent.log
[db160024.bwi701:/root]# ls -l /var/log/puppet/puppet-agent.log
-rw-------. 1 root root 0 Jan 27 09:52 /var/log/puppet/puppet-agent.log

puppetctl status still fails.

The config file /etc/rsyslog.d/listen.conf specifies the socket that rsyslog is listening on. This exists and seems to have the right permissions:

[db160024.bwi701:/root]# cat /etc/rsyslog.d/listen.conf
$SystemLogSocketName /run/systemd/journal/syslog
[db160024.bwi701:/root]# ls -l /run/systemd/journal/syslog
srw-rw-rw-. 1 root root 0 Oct 21 14:38 /run/systemd/journal/syslog

All signs point to rsyslog working as expected.

Running strace against puppetctl and looking for related syscalls:

...
connect(3, {sa_family=AF_UNIX, sun_path="/dev/log"}, 11) = -1 ENOENT (No such file or directory)
close(3)                                = 0
socket(AF_UNIX, SOCK_DGRAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0) = 3
connect(3, {sa_family=AF_UNIX, sun_path="/var/run/syslog"}, 18) = -1 ENOENT (No such file or directory)
close(3)                                = 0
socket(AF_UNIX, SOCK_DGRAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0) = 3
connect(3, {sa_family=AF_UNIX, sun_path="/var/run/log"}, 15) = -1 ECONNREFUSED (Connection refused)
close(3)                                = 0
socket(AF_UNIX, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0) = 3
connect(3, {sa_family=AF_UNIX, sun_path="/dev/log"}, 11) = -1 ENOENT (No such file or directory)
close(3)                                = 0
socket(AF_UNIX, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0) = 3
connect(3, {sa_family=AF_UNIX, sun_path="/var/run/syslog"}, 18) = -1 ENOENT (No such file or directory)
close(3)                                = 0
socket(AF_UNIX, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0) = 3
connect(3, {sa_family=AF_UNIX, sun_path="/var/run/log"}, 15) = -1 ECONNREFUSED (Connection refused)
close(3)                                = 0
...

On other similar systems /dev/log exists and is a symlink to /run/systemd/journal/dev-log. On this host, the former does not exist, but the latter does:

[db160024.bwi701:/root]# ls -l /dev/log
ls: cannot access '/dev/log': No such file or directory
[db160024.bwi701:/root]# ls -l /run/systemd/journal/dev-log
srw-rw-rw-. 1 root root 0 Oct 21 14:00 /run/systemd/journal/dev-log

It turns out that the /dev/log symlink is managed by the systemd-journald-dev-log.socket unit. And if this is restarted, followed by also restarting systemd-journald, the symlink is recreated. Note that the systemd-journald-dev-log.socket unit shows as failed until systemd-journald is restarted as well. (It is possible that stopping systemd-journald-dev-log.socket is enough but I have not tested that.)

[db160024.bwi701:/root]# systemctl restart systemd-journald-dev-log.socket
Job for systemd-journald-dev-log.socket failed.
See "systemctl status systemd-journald-dev-log.socket" and "journalctl -xe" for details.
[db160024.bwi701:/root]# systemctl status systemd-journald-dev-log.socket
● systemd-journald-dev-log.socket - Journal Socket (/dev/log)
   Loaded: loaded (/usr/lib/systemd/system/systemd-journald-dev-log.socket; static; vendor preset: disabled)
   Active: inactive (dead) since Mon 2025-01-27 12:38:11 PST; 3s ago
     Docs: man:systemd-journald.service(8)
           man:journald.conf(5)
   Listen: /run/systemd/journal/dev-log (Datagram)

Jan 27 12:38:11 db160024.bwi701.example.com systemd[1]: systemd-journald-dev-log.socket: Succeeded.
Jan 27 12:38:11 db160024.bwi701.example.com systemd[1]: Closed Journal Socket (/dev/log).
Jan 27 12:38:11 db160024.bwi701.example.com systemd[1]: Stopping Journal Socket (/dev/log).
Jan 27 12:38:11 db160024.bwi701.example.com systemd[1]: systemd-journald-dev-log.socket: Socket service systemd-journald.service already active, refu>
Jan 27 12:38:11 db160024.bwi701.example.com systemd[1]: Failed to listen on Journal Socket (/dev/log).
Warning: Journal has been rotated since unit was started. Log output is incomplete or unavailable.

[db160024.bwi701:/root]# systemctl restart systemd-journald
[db160024.bwi701:/root]# systemctl status systemd-journald-dev-log.socket
● systemd-journald-dev-log.socket - Journal Socket (/dev/log)
   Loaded: loaded (/usr/lib/systemd/system/systemd-journald-dev-log.socket; static; vendor preset: disabled)
   Active: active (running) since Mon 2025-01-27 12:38:58 PST; 2s ago
     Docs: man:systemd-journald.service(8)
           man:journald.conf(5)
   Listen: /run/systemd/journal/dev-log (Datagram)
    Tasks: 0 (limit: 3298408)
   Memory: 0B
   CGroup: /system.slice/systemd-journald-dev-log.socket

The /dev/log symlink now exists:

[db160024.bwi701:/root]# ls -l /dev/log
lrwxrwxrwx. 1 root root 28 Jan 27 12:38 /dev/log -> /run/systemd/journal/dev-log

And puppetctl is working now:

[db160024.bwi701:/root]# puppetctl status

 Puppet Agent:       Enabled
 Puppet Cron Job:    Enabled

cu/puppetctl-unix-syslog-delivery-error.md