2022 high-level docs for logstash-to-logstash using the HTTP input/output pair

logstash-to-logstash-over-http.md

We have had some success using LS-to-LS over HTTP(S), which supports an HTTP(s) Load Balancer or Proxy in the middle, and can be secured with TLS/SSL. It can be made to be quite performant, but doing so requires some specific tuning.

Upstream (HTTP Output)

The upstream pipelie would contain a single HTTP output plugin aimed either directly at a downstream Logstash or at a Load Balancer, importantly configured with:

• format => json_batch (for performance; without this one event will be sent at a time) and
• retry_non_idempotent => true (for resilience; without this, some failures cannot be safely retried).

Depending on whether we ar sending directly to another Logstash or through an SSL-terminating Load Balancer or proxy, the output may need to be configured

• with HTTP Basic credentials (user/password),
• with an X509 Certificate Authority (cacert), and/or
• with its own X509 Certificate keypair to assert its identity (client_cert/client_key).

output {
    http {
        id => 'ls1-output'
        url => 'https://load.balancer.address:8888'
        http_method => post

        #user => "logstash"
        #password => "SeCReT"

        retry_non_idempotent => true
        format => json_batch
        
        cacert => "./cert-bundle/ca/ca.crt"
        client_key => "./cert-bundle/upstream/upstream.key.pk8"
        client_cert => "./cert-bundle/upstream/upstream.crt"
    }
}

Downstream (HTTP Input)

The downstream pipeline would be configured with an HTTP input, importantly configured with:

• additional_codecs => { "application/json" => "json" } (to read the batched JSON payloads), and
• ecs_compatibility => disabled (to avoid adding ECS-related fields in LS 8+ where ECS is on-by-default).

If this downstream Logstash is publicly accessible it should be guarded to prevent unauthorized events from being injected by an adversary, which would require either SSL+Basic Auth or mutual-TLS.

If the customer is re-establishing TLS between the LB and the downstream Logstash, then the input would need to be configured

• with an X509 Certificate keypair to identify itself (ssl_certificate/ssl_key),
• possibly HTTP Basic auth (user/password), and
• possibly with an X509 Certificate Authority with which to trust the upstream or LB's certificate (if ssl_verify_mode => peer or ssl_verify_mode => force_peer).

input {
    http {
        id => 'ls2-input'
        host => '0.0.0.0'
        port => 8888
        #user => "logstash"
        #password => "SeCReT"
        additional_codecs => { "application/json" => "json_lines" }
        ecs_compatibility => disabled
        
        ssl => true
        ssl_key => "./cert-bundle/downstream.key.pk8"
        ssl_certificate => "./cert-bundle/logstash2.crt"
        ssl_certificate_authorities => "./cert-bundle/ca/ca.crt"
        ssl_verify_mode => peer # will need CN to match host
    }
}