AWS CloudWatch Query Every Log Group Beyond AWS UI Maximum

aws-cw-query-all-log-groups.md

Get all log groups

aws2 logs describe-log-groups --page-size 50 --max-items 1000 --profile preprod | jq -r '.logGroups[].logGroupName'

---

Query every log group in the last 30 mins

aws2 logs describe-log-groups --page-size 50 --max-items 1000 --profile preprod | jq -r '.logGroups[].logGroupName' | while read line; do echo $line; aws2 logs start-query --profile preprod --log-group-name $line --query-string 'filter @message like /(?i)(mongoerror|staging1)/| fields @timestamp, @message | sort @timestamp desc' --start-time $(expr 1578699670 - 1800) --end-time 1578699670 | jq -r '.queryId' >> queryIds.txt; echo $line >> link.txt; tail -1 queryIds.txt >> link.txt; done

Get results from every query ID generated in previous command

cat queryIds.txt | while read line; do echo $line; echo $line >> results.txt ; aws2 logs get-query-results --profile preprod --query-id $line >> results.txt; done

---