Skip to content

Instantly share code, notes, and snippets.

@analogist
Last active April 26, 2022 06:45
Show Gist options
  • Save analogist/f74d28b5f00ae3db0cd7f0870f7bad90 to your computer and use it in GitHub Desktop.
Save analogist/f74d28b5f00ae3db0cd7f0870f7bad90 to your computer and use it in GitHub Desktop.
Convert Google Cloud Storage gsutil ls -L base64-encoded md5 hash listings to md5sum-compatible hash format
#!/bin/bash
# This awk/sed script looks for "gs://bucket/path/filename" and "Hash (md5)",
# swaps the two (so the hash appears before the file, in the md5sum format),
# cuts out the "gs://bucket/" name, and finally converts the md5 hash
# from base64 to hex. This allows for later off-line file integrity check
# via the standard "md5sum -c".
#
# To use it, either save it as an alias or a shell script, and pipe the
# output of gsutil ls -L through it. For example, save it as "md5convert.sh"
# and then run "gsutil ls -L gs://bucket/path | bash md5convert.sh"
awk 'BEGIN { \
decodehash = "base64 -d | xxd -p | tr -d \"\\n\""; \
truncname = "sed \"s/gs:\/\/[a-z0-9_.\-]*\///\" | sed \"s/:$//\"" } \
/Hash \(md5\)/ { print $3 | decodehash; close(decodehash); \
printf " %s\n",fname | truncname; close(truncname) } \
/^gs:\/\// { fname = $0 }'
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment