Skip to content

Instantly share code, notes, and snippets.

@ElijahLynn
Last active February 10, 2025 22:27
Show Gist options
  • Save ElijahLynn/2882f122c6e83b9eefc5d1e3c85eeea7 to your computer and use it in GitHub Desktop.
Save ElijahLynn/2882f122c6e83b9eefc5d1e3c85eeea7 to your computer and use it in GitHub Desktop.
GCP: How to use HMAC credentials with gsutil

It isn't as easy as setting some ENV vars...

If you have an interactive TTY (local dev testing)
Generate a boto config, will prompt for access key ID and secret access token This will add gs_secret_access_key and gs_access_key_id and save to ~/.boto.

gsutil config -a

If you don't have an interactive TTY (CI/automation)
You can generate .boto manually with these lines:

[Credentials]
gs_access_key_id = GOOG1EJY545678ASDFGHJKL123456789FGHJKCVBN
gs_secret_access_key = xxxxxxxxxxxxxxxxxxxx

And then:

export BOTO_CONFIG=/path/to/.boto

Enable the boto creds - set to true to disable (it seems counterintuitive but that is the way it works)

gcloud config set pass_credentials_to_gsutil false

Test to see if it works with a bucket it should have read access to, should be able to confirm the access key ID is present

gsutil -D ls "gs://[[ BUCKET_NAME ]]"
...
DEBUG 0124 13:26:47.080310 provider.py] Using access key found in config file.
DEBUG 0124 13:26:47.080398 provider.py] Using secret key found in config file.
...
DEBUG 0124 13:26:47.132560 https_connection.py] validating server certificate: hostname=storage.googleapis.com, certificate hosts=['storage.googleapis.com']
send: b'GET /[[ BUCKET_NAME ]]/?delimiter=/ HTTP/1.1\r\nHost: storage.googleapis.com\r\nAccept-Encoding: identity\r\nx-goog-api-version: 1\r\nUser-Agent: Boto/2.49.0 Python/3.11.11 Darwin/23.5.0 gsutil/5.33 (darwin) analytics/enabled interactive/True command/ls google-cloud-sdk/507.0.0\r\nDate: Fri, 24 Jan 2025 21:26:47 GMT\r\nAuthorization: GOOG1 GOOG1EJY5PCQOI987FHYAF73MXDHJSYTEJFDJJDFJJDFDF:XXXXXXXXXXXXXXXXXX

Disable the boto creds when done, if needed

gcloud config set pass_credentials_to_gsutil true
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment