Using TLS between Beats and Logstash

The purpose of this document is to help with configuring and troubleshooting using TLS on the connection between Beats and Logstash.

Configuration

You must configure TLS on both the client and server to make this work. This configuration will ensure that Beats do not leak data by sending data to an untrusted Logstash server. This configuration does not provide client authentication (for that we require logstash-input-beats [issue #8] (logstash-plugins/logstash-input-beats#8) to be fixed).

logstash-input-beats configuration (reference docs):

input {
  beats {
    port => 5044
    ssl => true
    ssl_certificate => "/etc/pki/logging/logs.andrewkroh.com.crt"
    ssl_key => "/etc/pki/logging/logs.andrewkroh.com.key"
  }
}

Beats Logstash output configuration (reference docs):

output:
  logstash:
    hosts: ["logs.andrewkroh.com:5044"]
    ssl: # In 5.x this is ssl, prior versions this was tls.
      certificate_authorities:
        - /etc/pki/logging/ca.crt

Certificates

You should create a certificate authority (CA) and then sign the server certificate used by Logstash with the CA certificate. Then you configure Beats to trust any certificate signed by your CA certificate.

This document does not address how to create the certificates. There are many online resources describing the process.

Testing Logstash Independent of Beats

NOTE: The protocol between Beats and Logstash is NOT based on HTTP!

You can use curl to validate the Logstash server's certificate even though the protocol is not based on HTTP. You should receive an empty response error as opposed to a SSL error.

First let's test the connection using using the IP address in the URL.

curl -v --cacert ca.crt https://192.168.99.100:5044
* Rebuilt URL to: https://192.168.99.100:5044/
*   Trying 192.168.99.100...
* Connected to 192.168.99.100 (192.168.99.100) port 5044 (#0)
* WARNING: using IP address, SNI is being disabled by the OS.
* SSL: certificate verification failed (result: 5)
* Closing connection 0
curl: (51) SSL: certificate verification failed (result: 5)

It failed. The reason is because the certificate being used in not valid for the IP address, it is valid only for the hostname logs.andrewkroh.com. For work-arounds to this problem see the troubleshooting section.

Try the test again but use the hostname in the request URL.

$ curl -v --cacert ca.crt https://logs.andrewkroh.com:5044
* Rebuilt URL to: https://logs.andrewkroh.com:5044/
*   Trying 192.168.99.100...
* Connected to logs.andrewkroh.com (192.168.99.100) port 5044 (#0)
* TLS 1.2 connection using TLS_DHE_RSA_WITH_AES_256_CBC_SHA
* Server certificate: logs.andrewkroh.com
* Server certificate: andrewkroh.com
> GET / HTTP/1.1
> Host: logs.andrewkroh.com:5044
> User-Agent: curl/7.43.0
> Accept: */*
> 
* Empty reply from server
* Connection #0 to host logs.andrewkroh.com left intact
curl: (52) Empty reply from server

It works!

Testing Beats to Logstash

Test your Beat setup by running the Beat in the foreground so you can quickly see any errors that occur. If you have the Beat running as a service, stop it first.

$ <beatname> -c <beatname>.yml -e -v

Any errors will be printed to the console.

Troubleshooting

Your SSL connection fails when using the IP address of the server

This happens because your certificate is only valid for the hostname present in the Subject field.

The possible works-arounds are:

Create a DNS entry for the hostname mapping it to the server's IP.
Create an entry in /etc/hosts for the hostname. Or on Windows add an entry to C:\Windows\System32\drivers\etc\hosts.
Re-create the server certificate and add a SubjectAltName (SAN) for the IP address of the server. This make the server's certificate valid for both the hostname and the IP address.

"I want to use TLS between Beats and Logstash, and I don't care if someone can spoof my server.""

OK, so you just want to get up and running fast. Use the insecure: true option in you beat configuration files. This will cause Beats to ignore any TLS security issues.

output:
  logstash:
    hosts: ["logs.andrewkroh.com:5044"]
    tls:
      insecure: true

x509: cannot validate certificate for 192.168.99.100 because it doesn't contain any IP SANs

2016/02/25 19:56:31.441724 transport.go:125: ERR SSL client failed to connect with: x509: cannot validate certificate for 192.168.99.100 because it doesn't contain any IP SANs
2016/02/25 19:56:31.441748 single.go:126: INFO Connecting error publishing events (retrying): x509: cannot validate certificate for 192.168.99.100 because it doesn't contain any IP SANs

See Your SSL connection fails when using the IP address of the server.

getsockopt: no route to host

2016/02/25 17:37:08.496727 transport.go:125: ERR SSL client failed to connect with: dial tcp logs.andrewkroh.com:5044: getsockopt: no route to host

This is not a TLS problem. It's a networking problem. Make sure the two hosts can communicate.

getsockopt: connection refused

2016/02/25 22:04:06.115207 transport.go:125: ERR SSL client failed to connect with: dial tcp 192.168.99.100:5044: getsockopt: connection refused

This is not a TLS problem. Make sure that Logstash is running and that there is no firewall blocking the traffic.

No connection could be made because the target machine actively refused it.

A firewall is refusing the connection. Check if a firewall is blocking the traffic on the client, the network, or the destination host.

How do I view the contents of my certificate?

$ openssl x509 -text -noout -in logs.andrewkroh.com.crt
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 4 (0x4)
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN=andrewkroh.com
        Validity
            Not Before: Feb 25 20:12:53 2016 GMT
            Not After : Feb 22 20:12:53 2026 GMT
        Subject: CN=logs.andrewkroh.com
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
            RSA Public Key: (2048 bit)
                Modulus (2048 bit):
                    00:bd:b8:0c:2b:d0:09:df:fd:ad:76:ae:53:8f:4c:
                    47:56:2b:98:5b:bf:98:9a:7e:5d:b9:9a:fb:f0:e8:
                    62:bc:53:83:ef:90:a9:ba:c5:2e:cb:49:97:89:d7:
                    45:ae:4a:8d:7b:38:a4:ce:30:02:88:ac:a9:01:06:
                    38:90:d4:93:45:2d:eb:7d:15:cc:16:04:cf:2e:dc:
                    a0:a9:e6:11:90:0e:37:d8:12:be:21:8d:2d:74:c3:
                    b1:da:40:04:a3:57:17:51:8c:d2:ac:2f:53:2a:c4:
                    78:4a:97:6a:5c:54:79:37:ad:92:8a:a9:69:ac:92:
                    97:98:e7:68:8d:47:ed:81:4e:0e:7f:f1:0b:25:5e:
                    4f:92:19:d4:b0:03:77:0e:12:6e:76:59:81:3e:fb:
                    5c:05:eb:12:b1:63:3a:31:b2:a9:78:fb:c5:08:00:
                    8f:4f:80:5c:48:d5:0e:8b:d3:11:60:df:99:f4:43:
                    bb:8b:21:49:f9:2a:3b:84:4c:ad:92:49:17:8b:76:
                    f0:92:3b:eb:7e:e8:9d:20:1c:65:06:24:a8:87:06:
                    3f:0b:9e:ec:92:05:60:43:42:10:c2:5e:5a:a1:d7:
                    95:80:c8:e1:57:2f:8a:e8:54:92:02:fe:38:3c:f6:
                    ac:46:33:53:c9:48:b5:6e:a2:5c:aa:bc:2c:db:8c:
                    e1:5f
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Basic Constraints: 
                CA:FALSE
            X509v3 Subject Key Identifier: 
                94:F4:2A:C9:E5:8A:68:78:33:45:5F:1A:E9:F8:19:3B:54:E4:0B:3D
            X509v3 Authority Key Identifier: 
                keyid:03:47:B8:E1:45:97:DD:B7:F4:C3:0B:31:B4:0A:F8:76:6E:99:3B:5B
                DirName:/CN=andrewkroh.com
                serial:99:4D:A2:8A:02:99:9C:34

            X509v3 Extended Key Usage: 
                TLS Web Server Authentication
            X509v3 Key Usage: 
                Digital Signature, Key Encipherment
            X509v3 Subject Alternative Name: 
                IP Address:192.168.99.100
    Signature Algorithm: sha256WithRSAEncryption
        9d:7d:d3:6f:6b:9b:64:08:d9:21:e9:cc:48:c0:51:7e:a9:84:
        6e:8a:b6:4b:c3:63:40:a6:3a:fc:0b:4a:bc:3b:a6:db:3e:2e:
        18:b5:35:c3:ab:9f:87:30:ef:ef:46:21:ab:44:fc:bc:b9:93:
        62:6a:b8:7c:47:31:2f:4e:82:39:f1:bb:de:58:f6:aa:e0:01:
        ff:1f:28:1d:cf:17:55:2a:0d:4a:97:2f:ee:63:97:3d:f5:fe:
        4b:11:db:2d:85:9a:0b:04:b2:1e:63:8a:dc:a4:5c:6a:c0:c5:
        f6:ec:e6:22:90:ea:fd:a6:a5:9e:29:fc:e1:6d:f2:8f:db:4b:
        b7:dc:58:37:6a:fe:c2:e5:25:7b:ae:f6:57:b0:e7:f4:73:c8:
        3a:d9:bb:65:5c:e3:e1:53:9c:fd:79:39:88:18:a9:f9:1b:26:
        58:8a:55:42:5a:f3:d2:c5:d4:f3:7c:88:57:d0:81:15:40:eb:
        31:38:3e:ad:65:ff:5a:1a:c4:c6:42:fc:2a:cc:a2:d8:d3:f8:
        a1:33:a2:da:00:9b:71:91:c5:bf:7e:70:53:ea:81:89:51:9f:
        4f:04:00:cf:cb:f1:49:bd:4d:6e:8f:4a:2b:57:4c:12:34:3a:
        c0:1a:87:79:aa:e3:47:55:cd:d7:ab:fc:06:e9:65:b9:a2:8e:
        11:a4:1f:53

The example above shows a certificate that is valid for both logs.andrewkroh.com and 192.168.99.100.

The hostname is part of the Subject field:

Subject: CN=logs.andrewkroh.com

The IP address is included as a X509 v3 Subject Alternative Name extension field.

X509v3 Subject Alternative Name: 
    IP Address:192.168.99.100

How do I generate a quick cert for testing?

openssl req -x509  -batch -nodes -newkey rsa:2048 -keyout lumberjack.key -out lumberjack.crt \
  -subj /CN=logstash.example.com \
  -addext "subjectAltName = DNS:logstash.example.com,IP:192.168.99.100"

hello, i have a problem with filebeat and logstash,when using ssl => true, it not works, the beat logs is only
2016/04/08 08:19:31.928773 client.go:90: DBG connect
2016/04/08 08:19:31.929104 client.go:136: DBG Try to publish 2 events to logstash with window size 5
2016/04/08 08:19:31.946976 client.go:95: DBG close connection
2016/04/08 08:19:31.947039 client.go:114: DBG 0 events out of 2 events sent to logstash. Continue sending ...
2016/04/08 08:19:31.947055 single.go:76: INFO Error publishing events (retrying): EOF
2016/04/08 08:19:31.947063 single.go:152: INFO send fail
2016/04/08 08:19:31.947073 single.go:159: INFO backoff retry: 2s

other info is :

Rebuilt URL to: https://localhost:5044/
Hostname was NOT found in DNS cache
Trying 127.0.0.1...
Connected to localhost (127.0.0.1) port 5044 (#0)
successfully set certificate verify locations:
CAfile: ./certs/logstash-forwarder.crt
CApath: none
SSLv3, TLS Unknown, Unknown (22):
SSLv3, TLS handshake, Client hello (1):
SSLv2, Unknown (22):
SSLv3, TLS handshake, Server hello (2):
SSLv3, TLS handshake, CERT (11):
SSLv3, TLS handshake, Server key exchange (12):
SSLv3, TLS handshake, Server finished (14):
SSLv2, Unknown (22):
SSLv3, TLS handshake, Client key exchange (16):
SSLv2, Unknown (20):
SSLv3, TLS change cipher, Client hello (1):
SSLv2, Unknown (22):
SSLv3, TLS handshake, Finished (20):
SSLv2, Unknown (20):
SSLv3, TLS change cipher, Client hello (1):
SSLv2, Unknown (22):
SSLv3, TLS handshake, Finished (20):
SSL connection using TLSv1.2 / ECDHE-RSA-AES256-SHA
Server certificate:
subject: CN=localhost
start date: 2016-04-08 08:18:56 GMT
expire date: 2026-04-06 08:18:56 GMT
common name: localhost (matched)
issuer: CN=localhost
SSL certificate verify ok.
SSLv2, Unknown (23):

GET / HTTP/1.1
User-Agent: curl/7.36.0
Host: localhost:5044
Accept: /
SSLv2, Unknown (21):
SSLv3, TLS alert, Client hello (1):
Empty reply from server
Connection #0 to host localhost left intact
curl: (52) Empty reply from server

filebeath.cnf

filebeat:
prospectors:
-
paths:
- /home/work/tmp/test.log

  input_type: log

  document_type: log

registry_file: /home/work/logstash/filebeat/registry

output:

logstash:
    hosts: ["localhost:5044"]

tls:
  #certificate: "/home/work/logstash/tls/certs/logstash-forwarder.crt"
  #certificate_key: "/home/work/logstash/tls/private/logstash-forwarder.key"
  certificate_authorities: ["/home/work/logstash/tls/certs/logstash-forwarder.crt"]
  #insecure: true

shipper:

logging:
  to_files: true
  level: debug
  files:
    path: "/home/work/logstash/filebeat"
name: filebeat.log
rotateeverybytes: 10485760 # = 10MB

*my logstash conf is *
input {
stdin {}
beats {
port => 5044
type => "logs"
ssl_certificate => "/home/work/logstash/tls/certs/logstash-forwarder.crt"
ssl_key => "/home/work/logstash/tls/private/logstash-forwarder.key"
ssl => true
}
}

output {
stdout {}
}

can you help me to fix this problem? thank you very much.

andrewkroh/beats-logstash-tls.md