Rsyslog & TLS & Letsencrypt

README.md

Rsyslog & TLS & Letsencrypt

rsyslog can send logs to a remote server, encrypted with TLS (to avoid tampering and MIM attacks); but generating certificates can be difficult; enters letsencrypt to save the day.

Currently the module imtcp seems to have a bug [1] so we use relp

We assume Ubuntu or Debian below. All commands are to be done as root, or using sudo

On each box (both client and server)

User

If you do not have a 'syslog' user, create it.

addgroup --system syslog
adduser --system --home /tmp --no-create-home  --gid ??? syslog
adduser syslog adm

(in the second command replace ??? with the GID of group syslog)

Software

Install needed software.

apt install rsyslog-gnutls rsyslog-relp certbot

(You will need a fairly recent version of certbot, 0.31 is too old.)

Restricted access

Then you may want to set these in '/etc/rsyslog.conf' (some parts may be present)

$FileOwner root
$FileGroup adm
$FileCreateMode 0640
$DirCreateMode 0755
$Umask 0022
$PrivDropToUser syslog
$PrivDropToGroup adm

Then check that any file in /var/log where rsyslogd should write to has permissions, e.g.

cd /var/log/
chmod -c g+w auth.log kern.log syslog user.log messages daemon.log
chgrp -c adm auth.log kern.log syslog user.log messages daemon.log

and also any file related to TLS (if already created with certbot, see later section)

Certificates

If the box has a web server, and this uses letsencrypt certificates, then you are fine; otherwise you can generate the needed certificates using certbot. You must edit for values appropriate for your server.

certbot --standalone --cert-name rsyslog -m [email protected] -d server.exeample.com  certonly

(if you are running a webserver in that host, you may need to stop it temporarily, or use options such as --http-01-port )

The above command will save certificates in /etc/letsencrypt/live/rsyslog/

Then download the CA cert. Check the URL using

openssl x509 -text -noout  -in /etc/letsencrypt/live/rsyslog/chain.pem  | grep CA

In my case it is http://x1.i.lencr.org/ . Download it with

wget -O -   http://x1.i.lencr.org/ | openssl x509 -inform DER -outform PEM   > /etc/letsencrypt/letsencrypt_ca.pem

Check that it downloaded fine

openssl x509 -text -noout -in  /etc/letsencrypt/letsencrypt_ca.pem 

Check that rsyslog can read the certificates (and in particular the private key); recent versions of rsyslog run as user syslog which is part of group adm so one way is to

chgrp -c adm -R /etc/letsencrypt/archive/rsyslog /etc/letsencrypt/live/rsyslog
chmod -c g+rx /etc/letsencrypt/ /etc/letsencrypt/archive/ /etc/letsencrypt/live  \
              /etc/letsencrypt/archive/rsyslog/ /etc/letsencrypt/live/rsyslog/
chmod -c g+r  /etc/letsencrypt/live/rsyslog/privkey.pem
chgrp -c adm /etc/letsencrypt/ /etc/letsencrypt/archive/ /etc/letsencrypt/live \
             /etc/letsencrypt/archive/rsyslog/ /etc/letsencrypt/live/rsyslog/

(but make sure that this change does not prohibit your webserver from accessing its certs)

Renewal

You may need to install a cron job, there are two examples files below.

Apparmor

Add permission to read the above files, edit '/etc/apparmor.d/usr.sbin.rsyslogd' and add

  /etc/letsencrypt/ r,
  /etc/letsencrypt/** r,

then reload the profile

apparmor_parser -r /etc/apparmor.d/usr.sbin.rsyslogd

If you use LXC, do this in both the main OS, and in the containerized OS

Checks

Check that file are readable.

sudo  -u syslog md5sum /etc/letsencrypt/live/rsyslog/fullchain.pem /etc/letsencrypt/live/rsyslog/privkey.pem /etc/letsencrypt/letsencrypt_ca.pem /var/log/syslog

In the server

Install and customize tls-server.conf in /etc/rsyslog.d/tls-server.conf

Restart rsyslog

systemctl restart rsyslog

In each client

Install and customize tls-client.conf in /etc/rsyslog.d/tls-client.conf

Restart rsyslog

systemctl restart rsyslog

Troubleshooting

Stop rsyslog

systemctl stop rsyslog

Start it in debug mode in a terminal or console.

rsyslogd -d -n -iNONE

or

systemctl stop rsyslog  ; script -c 'rsyslogd -d -n -iNONE' /tmp/rsyslog.log

Look carefully for errors

Other useful commands

To split a chain in the separate components and transform them into text (with fingerprint)

SERVER=server.example.com
# change to a temporary directory
cd `mktemp -d`
# split the full chain
csplit -b '%02d.pem' -z -f fullchain- /etc/letsencrypt/live/${SERVER}/fullchain.pem '/-----BEGIN CERTIFICATE-----/' '{*}'
# convert each part to readable text
for j in *pem ; do openssl x509 -fingerprint -text -noout  -in $j > $j.txt ; done

note that the first fullchain-00.pem is the certificate of the server.

References

[1] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=944591

Further reading

Install documentation...

apt install rsyslog-doc

... and read at least

• file:///usr/share/doc/rsyslog-doc/html/configuration/modules/imrelp.html

• file:///usr/share/doc/rsyslog-doc/html/configuration/modules/omrelp.html

• file:///usr/share/doc/rsyslog-doc/html/tutorials/tls.html

• file:///usr/share/doc/rsyslog-doc/html/tutorials/tls_cert_server.html

Some other interesting info:

• https://www.golinuxcloud.com/secure-remote-logging-rsyslog-tls-certificate/

• https://gist.github.com/drmalex07/bb178d61f800488446d22de4301160f1

• https://blog.lumy.me/computer-science/rsyslog.html

• https://gist.github.com/ReK42/610f5392f638b8cb19470aa0d347a0ea

• https://www.feistyduck.com/library/bulletproof-tls-guide/online/

• https://blogs.oracle.com/scoter/post/secure-logserver-with-rsyslog-oracle-linux

my_certbot

#!/bin/bash

# install in /etc/cron.weekly/my_certbot

# this cron script will stop apache,
# renew the certificate, and then start apache;
# it is useful in a host that running an apache web server

## see https://gist.github.com/mennucc/dca1df1741c0db993d54b9e3de1bae33

systemctl stop apache2
certbot  renew
systemctl start apache2

my_certbot_iptables

#!/usr/bin/python3

## install into /etc/cron.weekly/my_certbot_iptables

# this cron script will delete a rule that forwards the port 80 to another machine,
# renew the certificate, and then reinstate the rule;
# it is useful in a host that contains a container running a web server

## see https://gist.github.com/mennucc/dca1df1741c0db993d54b9e3de1bae33

import os, subprocess

line = None
cmd = ['iptables', '-t', 'nat', '-S'  , 'PREROUTING']
p=subprocess.run(cmd,capture_output=True, text=True)
for l in p.stdout.splitlines():
    if '--dport 80 ' in l:
        line = l.strip()
        print('iptables rule:', l)
if not line:
    print('no matching iptables rule')

if line:
    cmd = ['iptables', '-t', 'nat', '-D'] + line.split(' ')[1:]
    p=subprocess.run(cmd)


cmd = ['certbot', 'renew']
p=subprocess.run(cmd)

if line:
    cmd = ['iptables', '-t', 'nat'] + line.split(' ')
    p=subprocess.run(cmd)

tls-client.conf

# provides support for sending RELP messages (over TCP)

#https://gist.github.com/mennucc/dca1df1741c0db993d54b9e3de1bae33                                                                                         

module(load="omrelp")

action(
type="omrelp"
# CUSTOMIZE SERVER NAME
target="server.example.com"
TLS.PermittedPeer=["server.example.com"]
port="6514"
TLS="on"
TLS.Authmode="name"
# CUSTOMIZE : change this if certificates are stored somewhere else
TLS.CaCert="/etc/letsencrypt/letsencrypt_ca.pem"
TLS.MyCert="/etc/letsencrypt/live/rsyslog/fullchain.pem"
TLS.MyPrivKey="/etc/letsencrypt/live/rsyslog/privkey.pem"
)

# provides --MARK-- message, to ensure that channel is working
module(load="immark" interval="180")  

tls-server.conf

#https://gist.github.com/mennucc/dca1df1741c0db993d54b9e3de1bae33                                                                                         

module(load="imrelp" )

# start up listener at port 6514
input(
type="imrelp"
port="6514"
TLS="on"
TLS.Authmode="name"
# CUSTOMIZE : change this if certificates are stored somewhere else
TLS.CaCert="/etc/letsencrypt/letsencrypt_ca.pem"
TLS.MyCert="/etc/letsencrypt/live/rsyslog/fullchain.pem"
TLS.MyPrivKey="/etc/letsencrypt/live/rsyslog/privkey.pem"
# CUSTOMIZE : list here all names of all hosts that are allowed to connect here
TLS.PermittedPeer=["client1.example.com","client2.example.com","client3.example.com"]
KeepAlive="on"
)