Configuring logging clients for Logit with TLS mutual auth

Logit supports mutual auth in two ways:

Logit generate client certificates on our behalf by generating a private key and signing it with their CA.
We sign client certificates with our own CA and provide that CA's certificate to Logit.

This document covers the first example.

Request mutual auth from Logit

By default, the Logit endpoints have no authentication.

Send an email to [email protected] requesting the following changes to your stack. Include the stack ID in your request.

TLS ports for the Lumberjack and the Syslog protocols be opened
Unencrypted ports for the Lumberjack and Syslog protocols be closed

The port numbers vary from stack to stack; see your Stack Settings page for details of port numbers for your stack.

Request mutual auth be enabled and that a key and certificate be generated for your stack.

Note that this will be a single shared key and certificate, but you may wish to request more than one.

This may also be a good time to request IP address whitelisting and provide your list of IP addresses that you know you will be sourcing logging traffic from.

Logit PKI architecture

If you were to inspect the certificate presented by Logit's TLS endpoints on a freshly provisioned stack using:

$ openssl s_client -connect <your stack ID here>-ls.logit.io:<port> </dev/null

You would see that subject of the server certificate presented is CN=*.logit.io issued by /C=US/O=GeoTrust Inc./CN=RapidSSL SHA256 CA in turn issued by /C=US/O=GeoTrust Inc./CN=GeoTrust Global CA - a fairly standard wildcard certificate issued by a publicly trusted commercial CA.

This is the certificate referred to by the docs at https://logit.io/sources/configure/syslog.

When mutual auth is enabled, Logit will switch this out for a server certificate managed by their own PKI, and their own root CA, making the above documentation incorrect.

When mutual auth is enabled, the CA chain for the server certificate becomes:

Logit.io Ltd Root CA
Logit.io Intermediate CA
<YOUR STACK ID> Intermediate CA - an intermediate CA generated for your stack
<YOUR STACK ID>-ls.logit.io - a server certificate generated for your stack

The stack-specific intermediate CA is also the signing CA for the client certificate.

Certificate bundle provided by Logit

Logit will send you a zip file containing:

<YOUR STACK ID>-<x>.cert.pem - where x is an 8-digit hex number (the significance of this number is currently unclear). This is the client certificate, signed by the stack-specific intermediate CA.
<YOUR STACK ID>-<x>.key.encrypted.pem The corresponding private key for the above certificate. This private key needs to be provided to any process that wants to authenticate to the Logit service to initiate a log stream (Filebeat, Rsyslog). The decryption password is your stack's API key, available through the Logit settings portal.
<YOUR STACK ID>-ca-chain.pem - three certificates forming the CA chain, both from the Logit Root CA to the above certificate, and from the Logit Root CA to the server certificate.

Although it is permissible in the TLS spec for the client and the server certificates to be signed by different CAs, not all implemenations will do this. GnuTLS is used by Rsyslog and has this limitation, documented here:

Even in x509/fingerprint mode, both the client and sever certificate currently must be signed by the same root CA. This is an artifact of the underlying GnuTLS library and the way we use it. It is expected that we can resolve this issue in the future.

We have not done any further testing around this. There may be some possible configuration where the Logit server certificate is signed by the GeoTrust public CA, and the client certifiate is signed by a GDS CA that Logit trusts.

Install client key, client certificates, and CA chain certificates on your system

<YOUR STACK ID>-ca-chain.pem install to /etc/ssl/certs/logit-ca.pem; mode 0644
<YOUR STACK ID>-<x>.cert.pem install to /etc/ssl/certs/logit-client.pem; mode 0644
<YOUR STACK ID>-<x>.key.encrypted.pem install to /etc/ssl/private/logit-client.key; mode 0440

Set the group owner on the client key to ssl-cert to grant access to non-root processes whose UIDs are members of the ssl-cert group.

The client key will need to be decrypted before being installed.

Configuring Filebeat

Filebeat configuration fragment example:

output.logstash:
  enabled: true
  hosts:
    - 699bfc41-f38b-49c7-83a7-1c2666653c8d-ls.logit.io:14303
  ssl:
    enabled: true
    certificate_authorities:
      - /etc/ssl/certs/logstash-ca.crt
    certificate: "/etc/ssl/certs/logstash-client.crt"
    key: "/etc/ssl/private/logstash-client.key"

Puppet example with optional mutual auth - via the pcfens-filebeat module:

$logstash_output_config = {
  'logstash'                    => {
    'enabled'                   => 'true',
    'hosts'                     => ["${logstash_host}:${logstash_port}"],
    'ssl'                       => {
      'enabled'                 => true,
      'certificate_authorities' => [ $logstash_ca_cert_path ],
    },
  }
}

if $mutual_auth {
  $logstash_mutual_auth_config = {
    'logstash'        => {
      'ssl'           => {
        'key'         => $logstash_client_key_path,
        'certificate' => $logstash_client_cert_path,
      }
    }
  }
}
else {
  $logstash_mutual_auth_config = {}
}


$filebeat_outputs = deep_merge($logstash_output_config, $logstash_mutual_auth_config)

class { '::filebeat':
  manage_repo   => false,
  major_version => 5,
  outputs       => $filebeat_outputs,
}

Configuring Rsyslog

Rsyslog version 8.x is known to work. Earlier versions haven't been tried. Ubuntu Trusty ships with 7.x, but packages of 8.x for Trusty are available on the Rsyslog vendor's PPA.

The optional rsyslog-gnutls package must be installed.

Load the gtls "stream driver" and configure it for operation with Logit:

$DefaultNetstreamDriver gtls
$DefaultNetstreamDriverCAFile /etc/ssl/certs/logstash-ca.crt
$ActionSendStreamDriverAuthMode x509/name
$ActionSendStreamDriverPermittedPeer 699bfc41-f38b-49c7-83a7-1c2666653c8d-ls.logit.io
$ActionSendStreamDriverMode 1
$DefaultNetstreamDriverCertFile /etc/ssl/certs/logstash-client.crt
$DefaultNetstreamDriverKeyFile /etc/ssl/private/logstash-client.key

Declare an action on matching messages that uses the omfwd module to forward them to Logit:

*.*;local2.none action (
  type="omfwd"
  target="699bfc41-f38b-49c7-83a7-1c2666653c8d-ls.logit.io"
  name="logstash"
  template="ForwardFormat"
  protocol="TCP"
  port="14305"
  streamdriver="gtls"
  streamdrivermode="1"
  queue.spoolDirectory="/var/spool/rsyslog"
  queue.filename="logstash.queue"
  queue.type="LinkedList"
  queue.saveOnShutdown="on"
  queue.timeOutEnqueue="1"
  queue.maxDiskSpace="1g"
  action.resumeRetryCount="-1"
)

Note the streamdriver="gtls" and streamdrivermode="1" are required here, despite the earlier config fragment implying that the default behaviour has been changed.

dougneal/logit-with-mutual-auth.md