tsellers-r7

The following:
 - decompresses the file into a pipe
 - transforms the base64 encoded data in the 'data' field using the DAP 'transform' filter
 - uses the DAT 'decode_http_reply' filter to decode the HTTP response
 - emits this as JSON
 - Uses jq to check the '.data.http_title' field for "Invalid URL" (sorry for the dots in the name)
 - uses pigz to compress the stream into an output file so that you don't have to store uncompressed data at any point.

pigz -dc <filename> | \
  dap json + transform data=base64decode + decode_http_reply data + json | \

tsellers-r7

aws ec2 describe-regions --output text --query 'Regions[*].[RegionName]' --region us-east-1 | while read region; do echo -e "$region "; aws --region $region ec2 describe-images --owners 099720109477 --filters Name=root-device-type,Values=ebs Name=architecture,Values=x86_64 Name=name,Values='*hvm-ssd/ubuntu-xenial-16.04*' --query 'sort_by(Images, &Name)[-1].ImageLocation'; done;