Running Hashcat on Google Cloud's GPU-based VMs

In February 2017, Google announced the availability GPU-based VMs. I spun up a few of these instances, and ran some benchmarks. Along the way, I wrote down the steps taken to provision these VM instances, and install relevant drivers.

Update April 2019: Updated instructions to use instances with the Tesla T4 GPUs.

Some details:

Each VM supports up to four Tesla K80 boards, which, if I recall correctly, comprises two GK210 GPUs.
GPUs are billed per minute (10 min. minimum).

My only use case for GPU-based VMs is cracking (password) hashes, hence the test run with Hashcat.

Some remarks:

I should warn you beforehand that the K80s are known to be sub-optimal for cracking (password) hashes (using Hashcat). Low cost per hour per GPU doesn't necessarily mean it's the best choice in terms of performance/cost ratio.
For example, take a look at @jmgosney's GTX 1080 Hashcat benchmarks. If you compare these results to any K80 Hashcat benchmark, you will see that a single GTX 1080 is several times faster than a Tesla K80.
Also, at this time, you can't attach GPUs to low-cost preemptible instances (similar to Amazon EC2 spot instances).
Amazon, on the other hand, does support low-cost GPU-based spot instances, which makes cracking in the cloud a whole lot more interesting. See spot pricing.

Prerequisites

A Google Cloud Platform account.
You need to have the command-line tool gcloud installed.
If you already have gcloud installed, verify that the current version (gcloud version) is greater than (or equal) to 144.0.0.
You need to authorise (gcloud auth login) your local machine to access your Google Cloud account.
Make sure that you have enough GPUs available in both your project, and the desired region/zone. You can check the current limit using: gcloud compute regions describe europe-west4 | grep NVIDIA.*GPUS -B1 (replace europe-west4 with your desired region, and zone).

$ gcloud compute regions describe europe-west4 | grep NVIDIA.*GPU -B1

- limit: 8.0
  metric: NVIDIA_T4_GPUS
--
- limit: 8.0
  metric: PREEMPTIBLE_NVIDIA_T4_GPUS

If the metric 'NVIDIA_T4_GPUS' isn't listed, or the limit is set to 0.0, you need to ask your account manager to increase the limit, or submit an increase request.
Increase requests are usually granted in a matter of hours. However, you might be asked to make a prepayment of a few tens of dollars (Billing > select your billing account > 'Payment overview' > 'Make a payment').

Creating VM instance (and attaching GPUs)

Creating the virtual machine instance, and attaching GPUs, can be done in a single command:

$ gcloud compute instances create koenrh-gpu-instance-2 \
  --machine-type n1-standard-2 \
  --zone europe-west4-b \
  --accelerator type=nvidia-tesla-t4,count=4 \
  --image-family ubuntu-1804-lts \
  --image-project ubuntu-os-cloud \
  --maintenance-policy TERMINATE

Created [https://www.googleapis.com/compute/beta/projects/koenrh-test-drive/zones/europe-west4-b/instances/koenrh-gpu-instance-2].
NAME                   ZONE            MACHINE_TYPE   PREEMPTIBLE  INTERNAL_IP  EXTERNAL_IP    STATUS
koenrh-gpu-instance-2  europe-west4-b  n1-standard-2               10.132.0.2   35.xxx.xx.xxx  RUNNING

Besides the instance name, there are a few other arguments you might want to change:

--zone [zone]: Make sure that GPUs are actually supported in the specified zone.
--accelereator: type specifies the GPU model that you want to use. count specifies the number of processors (maximum is 8). You can attach up to four K80 boards, which means there's a maximum of 8 processors (K80 board houses two GK210 GPUs).

Authorising your SSH key

First, you need to authorise your SSH key by adding your SSH public key the just-provisioned machine. Use the following command, but use your own instance name, instance region, and path to your public key (after ssh-keys=).

# NOTE: Update the path to your SSH public key
$ gcloud compute instances add-metadata koenrh-gpu-instance-2 \
  --zone europe-west4-b \
  --metadata-from-file ssh-keys=/Users/koen/.ssh/id_ed25519.pub

Updated [https://www.googleapis.com/compute/v1/projects/koenrh-test-drive/zones/europe-west4-b/instances/koenrh-gpu-instance-2].

Installing drivers

Get the NVIDIA signing key that is used to sign packages.

curl -o cudatools.asc http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/7fa2af80.pub

Verify the integrity of the downloaded key. It should return with cudatools.asc: OK.

$ CUDA_KEY_CHECKSUM=47217c49dcb9e47a8728b354450f694c9898cd4a126173044a69b1e9ac0fba96
$ echo "$CUDA_KEY_CHECKSUM *cudatools.asc" | sha256sum --check --strict

cudatools.asc: OK

Or verify the fingerprint of the downloaded key. It should match AE09 FE4B BD22 3A84 B2CC FCE3 F60F 4B3D 7FA2 AF80.

$ gpg --with-fingerprint --with-colons cudatools.asc | grep "^fpr:" | cut -d: -f 10

AE09FE4BBD223A84B2CCFCE3F60F4B3D7FA2AF80

Import the CUDA signing key:

$ apt-key add cudatools.asc
OK

Download the CUDA installer, again, verify its integrity, and add the repository to the system using dpkg:

$ curl -sO http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/cuda-repo-ubuntu1804_10.1.105-1_amd64.deb

$ echo "2711e92b362972d6ba16dd1e1410f56ecc808831c3ae42b2b9fa98cea086f146 *cuda-repo-ubuntu1804_10.1.105-1_amd64.deb" | sha256sum --check --strict

cuda-repo-ubuntu1804_10.1.105-1_amd64.deb: OK

$ sudo dpkg -i cuda-repo-ubuntu1804_10.1.105-1_amd64.deb

...
OK

Finally, install CUDA, which includes the NVIDIA drivers:

$ sudo apt-get update && sudo apt-get install -y cuda

...

This could take a while.

Reboot the machine: sudo reboot

When the installation has finished, verify that the driver is installed, and initialised properly using the nvidia-smi tool.

$ nvidia-smi

...

Tuning the GPUs

A little performance tuning. Before you can change the application clocks you need to put the GPUs in 'persistence mode'.

Optionally, allow non-admin/root users to change the application clocks using the following command.

sudo nvidia-smi --applications-clocks-permission=UNRESTRICTED

Enable 'persistence mode' with the following command.

sudo nvidia-smi --persistence-mode=ENABLED

You must set both the memory and graphics clock, because the clock rates are tied to a specific memory clock rate. Set the clock rates using the following command (you can query the supported application clocks using nvidia-smi -q -i 0 -d SUPPORTED_CLOCKS).

sudo nvidia-smi --applications-clocks=2505,875

Running Hashcat

First, we need to get hold of Hashcat's signing key. I hate to use key servers, but Hashcat doesn't seem to have published the (ASCII) signing key anywhere else, which means we have to fetch the key from a key server.

$ gpg --recv-keys "A70833229D040B4199CC00523C17DA8B8A16544F"

gpg: key 8A16544F: "Hashcat signing key <[email protected]>" not changed
gpg: Total number processed: 1
gpg:              unchanged: 1

Just to be sure, verify that you have imported the correct key. The fingerprint should match A708 3322 9D04 0B41 99CC 0052 3C17 DA8B 8A16 544F.

$ gpg --list-keys --with-fingerprint A70833229D040B4199CC00523C17DA8B8A16544F

pub   rsa2048/8A16544F 2015-12-19 [SC]
      Key fingerprint = A708 3322 9D04 0B41 99CC  0052 3C17 DA8B 8A16 544F
uid         [ unknown] Hashcat signing key <[email protected]>
sub   rsa2048/911D7BDE 2015-12-19 [E]

Download the archive containing the pre-compiled binaries, and signature from Hashcat's website.

curl -OO https://hashcat.net/files/hashcat-5.1.0.7z{,.asc}

Verify the integrity of the downloaded Hashcat archive using the command depicted below. It should indicate that the signature is valid ("Good signature"). The warning indicates that the key that was used to sign this package doesn't have a trusted signature, therefore you should verify that the fingerprint of the key ('Primary key fingerprint' at the bottom) matches the fingerprint of the key that we have imported (and verified) in the previous steps.

$ gpg --verify hashcat-5.1.0.7z.asc hashcat-5.1.0.7z

gpg: Signature made Sun 02 Dec 2018 11:03:59 AM UTC
gpg:                using RSA key 3C17DA8B8A16544F
gpg: checking the trustdb
gpg: no ultimately trusted keys found
gpg: Good signature from "Hashcat signing key <[email protected]>" [unknown]
gpg: WARNING: This key is not certified with a trusted signature!
gpg:          There is no indication that the signature belongs to the owner.
Primary key fingerprint: A708 3322 9D04 0B41 99CC  0052 3C17 DA8B 8A16 544F

Unpack the archive with Hashcat binaries.

7z x hashcat-5.1.0.7z

...

Everything is Ok

Folders: 39
Files: 601
Size:       57418237
Compressed: 2658676

Run a full Hashcat benchmark.

cd ./hashcat-5.1.0/

./hashcat64.bin -b | tee hashcat-benchmark.txt

You can find the full results here (2x Tesla K80, 4x GPU).

Notes to self

Automate provisioning of GPU VMs (whether AWS EC2 or GCP VM) using shell scripts or, better, Ansible.
Figure out how to run hashcat in a distributed way, i.e. how to distribute password cracking jobs across servers or even clusters of servers.
- Look into hashtopus.
Look into management APIs or GUIs for remote management.
- For example: hashview (site)

koenrh/gcp-gpu-vm-hashcat.md