SSH Through TLS Terminating Ingress

AKA ssh over ssl
AKA ssh over tls
AKA ssh over https

Background

Some firewalls are annoying and don't just block ports that aren't 443, they also block packets that aren't TLS/SSL. Fortunately there is of course a way around this, and that is to wrap the ssh communication in TLS.

ssh has an option ProxyCommand which accepts an arbitrary command to provide a pipe (write to stdin, receive from stdout). This lets you use any transport you want, ssh will send encrypted traffic through that transport. So we can use an ssl transport, which is virtually indistiguishable from https traffic.

So we get ssh to run over ssl with ssh -o 'ProxyCommand socat - openssl:hostname:443' user@hostname -- but this will only work with a server that receives the ssl connection, and terminates so that the ssh server can receive the standard ssh protocol. Fortunately a number of layer 4 load balancers are designed to do exactly this. Examples include haproxy and traefik. The former haproxy already has a tutorial which is how I figured out this could be done with traefik.

Socat

Socat itself can be used on both sides, to illustrate what's happening better. Notably this is not suitable for production, the alternative of using a proper load balancer with letsencrypt for valid SSL certificates is much more secure. Though if you have valid ssl keys you remove verify=0 from below and it might be reasonable enough.

# create a self signed certificate
openssl req -x509 -newkey rsa:4096 -sha256 -days 3650 -nodes \
  -keyout cert.key -out cert.crt -subj "/CN=ssh.example.com" \
  -addext "subjectAltName=DNS:ssh.example.com"
cat cert.key cert.crt > cert.pem

# server side -- listen on https, forward to ssh
socat openssl-listen:443,fork,reuseaddr,cert=cert.pem,cafile=cert.crt,verify=0 tcp:localhost:22

# client side -- ssh over ssl
ssh -o 'ProxyCommand socat - openssl:ssh.example.com:443,verify=0' [email protected]

Traefik

Consider the following files:

docker-compose.yml

version: '3'
services:
  ingress:
    image: traefik
    ports:
      - 80:80
      - 443:443
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - ./ingress/traefik.yaml:/etc/traefik/traefik.yaml
      - ./ingress/config:/config
      - ./ingress/data:/data

ingress/traefik.yaml

providers:
  file:
    directory: /config
    watch: true
  docker:
    exposedByDefault: false
entryPoints:
  web:
    address: :80
  websecure:
    address: :443

ingress/config/ssh-over-tls.yaml

tcp:
  routers:
    ssh:
      # change the hostname as necessary
      rule: HostSNI(`ssh.example.com`)
      entrypoints:
        - websecure
      service: ssh
      # this tells traefik to use tls termination
      #  (incoming packets are tls, they are decoded before being sent to load balancer)
      # also possible to use traefik's letsencrypt support to get a valid ssl cert
      tls: {}
  services:
    ssh:
      loadBalancer:
        servers:
          # out of docker "localhost" alternatively you can point somehwere else, like to a vm
          - address: 172.17.0.1:22

This can now be run with docker-compose up -d, traefik may already be in place or otherwise can be used for forwarding all types of other traefik like actual http/https to/from docker containers or other things.

Client Side

A connection can be made with

ssh -o 'ProxyCommand socat - openssl:ssh.example.com:443' [email protected]

Naturally this becomes simpler with a ssh config.

Host example.com
  ProxyCommand socat - openssl:ssh.example.com:443
  # probably you also want identity here..
  User user

ssh example.com

Conclusion

ssh's ProxyCommand makes piggybacking off of arbitrary transports pretty easy, since TLS-terminating ingresses are often already in place for other purposes, it's simple enough to reuse them for proxying ssh, allowing you to serve ssh over https. As this took me a while to figure out, I'm putting this here for someone else to find who needs ssh over ssl with traefik. This would also work in a kubernetes cluster.