Download Logs From GCP in Bulk

download-logs.zsh

#!/bin/zsh

# This script will iterate over hourly intervals and download all logs matching a LQL query within each hour. At the end
# of this process you will end up with one JSON file containing all the logs.

# Adjust as needed; the following will download all logs matching the query between
#2025-05-12 09:00:00 and 2025-05-13 23:00:00
YEAR_MONTH='2025-05-'
DAY=12
HOUR=9
END_DAY=14

PROJECT_ID='<FILL IN YOUR PROJECT ID>'
OUTPUT_FILE='downloaded-logs.json'
TMP_FILE="$OUTPUT_FILE"_tmp

while ((DAY < END_DAY)); do
    START_TIME=$( printf '%s%02dT%02d:00:00.0Z' $YEAR_MONTH $DAY $HOUR )
    END_TIME=$( printf '%s%02dT%02d:00:00.0Z' $YEAR_MONTH $DAY $HOUR+1 )

    # Add in your LQL query, or just leave the timestamps if you'd like to get all logs
    # The example below will download all logs produced by a Cloud Run service called "my-cloud-run-service"
    QUERY='
      resource.type = "cloud_run_revision"
      AND resource.labels.service_name = "my-cloud-run-service"
      AND timestamp > "'"$START_TIME"'"
      AND timestamp < "'"$END_TIME"'"'

    echo "Getting logs from $START_TIME till $END_TIME"

    gcloud logging read "$QUERY" --project="$PROJECT_ID" --format=json >> "$TMP_FILE"

    # note: tail needed as jq cmd will give lengths of all arrays
    echo "Got $(jq '. | length' "$TMP_FILE" | tail -n 1) log records"
    echo "----\n"

    ((HOUR++))
    # note: 24:00:00 is not a valid time in Google's LQL
    if ((HOUR == 23)); then
        ((DAY++))
        HOUR=0
    fi
done

echo "Merging received logs and cleaning up"
# Each batch of logs is downloaded as JSON array [{...},{...},...], these are appended to the TMP_FILE; that isn't a
# valid JSON file, the tex file looks like this [{...}, {...}, ...][{...}, {...}, ...]; the following command will merge
# all the arrays into one
jq -s '[.[][]]' "$TMP_FILE" > "$OUTPUT_FILE"
rm "$TMP_FILE"

echo "Done. Got $(jq '. | length' $OUTPUT_FILE)"

This is useful when you hit the 10,000 records limit in the Google Cloud Log Explorer:

You can also copy your logs using the gcloud logging copy command: Batch and route logs retroactively