Interactive Debugging on Github Actions

Did you ever run into some issue where a job would behave slightly different in you CI environment than on your local machine? Did you ever wish you could run just a few commands in a shell on your build machine?

These are, of course rhetorical questions. And if you're using Github Actions to run your CI jobs, you'll have noticed that this use case is not supported at all. There are some workarounds (e.g. https://github.com/nektos/act), but since they're not officially supported they can be a bit unstable. Also, even they usually don't reproduce the exact environment found on github's servers.

Minimal Reverse Shell

Anyways, here's a cool technique to investigate your CI failures interactively. It's creating a reverse shell from the build machine, with strict TLS certificate pinning to prevent any random internet person to just look around your build.

First, run this on any server connected to the internet (or at least connected to the build machine):

openssl req -x509 -newkey rsa:4096 -keyout key.pem -out cert.pem -days 365 -nodes
openssl s_server -quiet -key key.pem -cert cert.pem -port 2222

The cert.pem created from the first command here should be made available to the CI job.

In your workflow definition, add the following step:

      - name: Do regular CI stuff
        [...]

      - name: Spawn Reverse Shell on Failure
        if: failure()
        run: |
          sudo apt-get -qqy install openssl
          mkfifo /tmp/s; /bin/sh -i < /tmp/s 2>&1 | openssl s_client -quiet -no-CApath -strict -verify 1 -verify_return_error -CAfile cert.pem -connect $SERVER_IP:2222 > /tmp/s; rm /tmp/s

(credit to int0x33 for the original inspiration)

Of course, this technique is not unique to github, it can be used on any CI runner with network connectivity to the target host. Which these days is almost always the case. If the target port is changed from 2222 to 443, it will even go through almost any firewalls.

Deluxe Reverse Shell

The above solution works great, but after using it a few times it the limitations become more and more noticable: There is no auto-completion, no line editing and, worst of all, hitting CTRL-C immediately and irrevocably terminates the connection.

To remedy all of this, we can set up a ssh reverse shell. The basic idea is to start up a local ssh server on the CI machine, and then create a reverse ssh tunnel so remote users can connect to the local server.

First, generate a new SSH key pair:

ssh-keygen -t rsa -f rshell.id_rsa

The public and secret key generated by this should be made available to the CI job. We're using the keys in both directions below, i.e. both to log in to the middle-man machine and to log in one the CI machine.

We also need a middle-man host, which needs to be accessible from the internet, or at least from the CI machine. On this host, we create an rshell user that can be used for establishing the reverse tunnel. It is set up to be able to login only with the private key generated above. To prevent this user from doing anything else, its shell is set to /bin/true:

adduser --disabled-password --shell /bin/true rshell
mkdir -p /home/rshell/.ssh
cp ${PUBLIC_KEY} /home/rshell/.ssh/authorized_keys
chown -R rshell:rshell /home/rshell/.ssh

The final step is to set up the local ssh server and to create the tunnel: (it should also be possible to use sshd as the ssh server, but it has to be started as root and one needs to be careful with the configuration to stop it from interfering with the system.)

- name: Spawn Deluxe Reverse Shell on Failure
   if: failure()
   env:
     MIDDLEMAN: ${{ secrets.RSHELL_REMOTE_HOST }}
   run: |
     sudo apt-get -qqy install dropbear-bin openssh-client
     mkdir -p ~/.ssh
     echo "${{ secrets.RSHELL_USER_PUBLIC_KEY }}" >> ~/.ssh/authorized_keys
     echo "${{ secrets.RSHELL_USER_SECRET_KEY }}" >> ~/.ssh/rshell.id_rsa
     echo "${{ secrets.RSHELL_REMOTE_HOSTKEY }}" >> ~/.ssh/known_hosts
     chmod 0400 ~/.ssh/rshell.id_rsa ~/.ssh/authorized_keys
     chmod go-w ~
     dropbearkey -t rsa -f ~/.ssh/dropbear.rsa
     dropbearkey -t ecdsa -f ~/.ssh/dropbear.ecdsa
     dropbear -E -R -w -g -a -p 2222 -P ./dropbear.pid -r ~/.ssh/dropbear.rsa -r ~/.ssh/dropbear.ecdsa
     ssh -N -i ~/.ssh/rshell.id_rsa -R 2222:127.0.0.1:2222 ${MIDDLEMAN}

Now, it is possible to log in from the middle-man machine. (if this is a special-purpose machine, one might consider setting GatewayPorts yes in the local sshd config to allow

$ ssh -i ${PRIVATE_KEY_FILE} -p 2222  [email protected]

Voila. The connection will persist until the CI job is shut down, and even supports multiple users investigating the CI job in parallel. As above, this can also be obfuscated by using port 443 for environments with a restrictive firewall.

This also works on Mac, except that the apt-get in the first line has to be replaced by

brew install dropbear openssh

lava/tls_reverse_shell.md

Interactive Debugging on Github Actions

Minimal Reverse Shell

Deluxe Reverse Shell