Skip to content

Instantly share code, notes, and snippets.

@jhyland87

jhyland87/get-header-val.md

Last active November 2, 2017 19:47
Show Gist options
  • Save jhyland87/95d88e9f32a77f8d6860f69c3332719d to your computer and use it in GitHub Desktop.
Save jhyland87/95d88e9f32a77f8d6860f69c3332719d to your computer and use it in GitHub Desktop.
Download ZIP
Raw
get-header-val.md

Awk Variables

Key Description Value Type Values Default Notes
item,items Header item(s) to look for String Set-Cookie SetCookie setcookie N/A This is the only required value
limit Maximum number of results to return Integer 0 (all) all 2 1 Limits the results to a single item unless this is changed
keys Determines if the results should include the header keys as well boolean 1 0 1 0 is treated the same as all
quote Determines if the output values should have the header value in quotes boolean/string 1 single double 0
format Format to print data in String %-30s : %s\n %s\n N/A

Retrieving HTTP Response Headers

The header output can be generated and saved using curl. the -I or --head flag will

  • -i, --include - Return the HTTP response header values only
  • -I, --head - Include the response headers in the output
  • -D, --dump-header - Write the HTTP response headers to the file specified

The HTTP response headers can be provided to get-header-val.awk either via STDIN, or by specifying a file containing the response headers.

For example, this:

$ curl https://www.google.com/ --silent --head --output /dev/null --dump-header ./response-header.txt && \
  ./get-header-val.awk -v item=Date ./response-header.txt

Thu, 02 Nov 2017 17:57:11 GMT

should yield the exact same result as either of these:

$ curl https://www.google.com/ --silent --head | ./get-header-val.awk -v item=Date
$ curl https://www.google.com/ --silent --head --output /dev/null --dump-header ./response-header.txt && \
  cat ./response-header.txt | ./get-header-val.awk -v item=Date

Thu, 02 Nov 2017 17:57:13 GMT

Example Usages

$ curl https://www.google.com/  \
  --compressed   \
  --silent   \
  --dump-header ./response-header.txt   \
  --output ./response-output.txt

The file ./response-header.txt is what would be processed using get-header-val.awk.

Basic Usage

Specify the desired header value

$ ./get-header-val.awk \
  -v item="Date" \
  ./response-header.txt

text/html; charset=UTF-8

The header value doesn't need to be exact. The item value specified and the header keys are simplified before the comparison is made. Everything except for alpha-numeric characters are removed and converted to lowercase. Thus, S eT!... Cook IE? == setcookie

$ ./get-header-val.awk \
  -v item="DATE" \
  ./response-header.txt

text/html; charset=UTF-8

$ ./get-header-val.awk \
  -v item="D a t e" \
  ./response-header.txt

text/html; charset=UTF-8

$ ./get-header-val.awk \
  -v item="setcookie" \
  -v limit=2 \
  ./response-header.txt

XSRF-TOKEN=eyJpdiI6Imo0SGJFOWJuVldBeUpMVFVIdnJpVmc9PSIsInZhbHVlIjoiWnhhZmc3ZEM5eTM1VUpvRDdkd0EwQW5hellZdlFcL0ZRQmtcL0xnMW9UTVVoZDZzeHkrcW4rYmNFZ0Q2akFvb2dMT2JxZjE3dzhybW1JMHRibDU5RnZ3QT09IiwibWFjIjoiYTc4MjAzNmY3OWMzNjM1ZDUxZjlmZTg5YWY0ZjJhOTcxMzVhNThkNjdjYjE0MmI0Zjk1OWRiYTNkZTY0ZjkwZSJ9; expires=Thu, 02-Nov-2017 17:50:38 GMT; Max-Age=7200; path=/ whatsmydns_session=eyJpdiI6Ik9ZTVh6TzFPdzE4bmIwOEI3NWdpQnc9PSIsInZhbHVlIjoiKzRtMFRVaEFTSWxuQkFHMVpudGV5U1BoWTZjU1Rocmh4b2pwelBSQ2dsWmJTSklFdmJnaEdCamlNQmN0VHRhTTU2b3NUa3ltXC9DOWxMUW5kekkzckV3PT0iLCJtYWMiOiIwMTk0NGMwYWUxZGM1NDQ1NjRlNTNhNDBjY2NlYTQ0MzAyMzY0YzkyYjY2OTU3NjZkZjQ5MWU3MzFlNTE0MTRkIn0%3D; expires=Thu, 02-Nov-2017 17:50:38 GMT; Max-Age=7200; path=/; HttpOnly

$ ./get-header-val.awk \
  -v item="setcookie" \
  -v limit=all \
  -v keys=1 \
  -v delim=" => " \
  ./response-header.txt

Set-Cookie => XSRF-TOKEN=eyJpdiI6Imo0SGJFOWJuVldBeUpMVFVIdnJpVmc9PSIsInZhbHVlIjoiWnhhZmc3ZEM5eTM1VUpvRDdkd0EwQW5hellZdlFcL0ZRQmtcL0xnMW9UTVVoZDZzeHkrcW4rYmNFZ0Q2akFvb2dMT2JxZjE3dzhybW1JMHRibDU5RnZ3QT09IiwibWFjIjoiYTc4MjAzNmY3OWMzNjM1ZDUxZjlmZTg5YWY0ZjJhOTcxMzVhNThkNjdjYjE0MmI0Zjk1OWRiYTNkZTY0ZjkwZSJ9; expires=Thu, 02-Nov-2017 17:50:38 GMT; Max-Age=7200; path=/ Set-Cookie => whatsmydns_session=eyJpdiI6Ik9ZTVh6TzFPdzE4bmIwOEI3NWdpQnc9PSIsInZhbHVlIjoiKzRtMFRVaEFTSWxuQkFHMVpudGV5U1BoWTZjU1Rocmh4b2pwelBSQ2dsWmJTSklFdmJnaEdCamlNQmN0VHRhTTU2b3NUa3ltXC9DOWxMUW5kekkzckV3PT0iLCJtYWMiOiIwMTk0NGMwYWUxZGM1NDQ1NjRlNTNhNDBjY2NlYTQ0MzAyMzY0YzkyYjY2OTU3NjZkZjQ5MWU3MzFlNTE0MTRkIn0%3D; expires=Thu, 02-Nov-2017 17:50:38 GMT; Max-Age=7200; path=/; HttpOnly

$ ./get-header-val.awk \
  -v item="setcookie" \
  -v limit=all \
  -v keys=1 \
  ./response-header.txt

Set-Cookie=XSRF-TOKEN=eyJpdiI6Imo0SGJFOWJuVldBeUpMVFVIdnJpVmc9PSIsInZhbHVlIjoiWnhhZmc3ZEM5eTM1VUpvRDdkd0EwQW5hellZdlFcL0ZRQmtcL0xnMW9UTVVoZDZzeHkrcW4rYmNFZ0Q2akFvb2dMT2JxZjE3dzhybW1JMHRibDU5RnZ3QT09IiwibWFjIjoiYTc4MjAzNmY3OWMzNjM1ZDUxZjlmZTg5YWY0ZjJhOTcxMzVhNThkNjdjYjE0MmI0Zjk1OWRiYTNkZTY0ZjkwZSJ9; expires=Thu, 02-Nov-2017 17:50:38 GMT; Max-Age=7200; path=/ Set-Cookie=whatsmydns_session=eyJpdiI6Ik9ZTVh6TzFPdzE4bmIwOEI3NWdpQnc9PSIsInZhbHVlIjoiKzRtMFRVaEFTSWxuQkFHMVpudGV5U1BoWTZjU1Rocmh4b2pwelBSQ2dsWmJTSklFdmJnaEdCamlNQmN0VHRhTTU2b3NUa3ltXC9DOWxMUW5kekkzckV3PT0iLCJtYWMiOiIwMTk0NGMwYWUxZGM1NDQ1NjRlNTNhNDBjY2NlYTQ0MzAyMzY0YzkyYjY2OTU3NjZkZjQ5MWU3MzFlNTE0MTRkIn0%3D; expires=Thu, 02-Nov-2017 17:50:38 GMT; Max-Age=7200; path=/; HttpOnly

$ ./get-header-val.awk \
  -v item="setcookie" \
  -v limit=all \
  -v keys=1 \
  -v quote=1 \
  ./response-header.txt

Set-Cookie="XSRF-TOKEN=eyJpdiI6Imo0SGJFOWJuVldBeUpMVFVIdnJpVmc9PSIsInZhbHVlIjoiWnhhZmc3ZEM5eTM1VUpvRDdkd0EwQW5hellZdlFcL0ZRQmtcL0xnMW9UTVVoZDZzeHkrcW4rYmNFZ0Q2akFvb2dMT2JxZjE3dzhybW1JMHRibDU5RnZ3QT09IiwibWFjIjoiYTc4MjAzNmY3OWMzNjM1ZDUxZjl"ZTg5YWY0ZjJhOTcxMzVhNThkNjdjYjE0MmI0Zjk1OWRiYTNkZTY0ZjkwZSJ9; expires=Thu, 02-Nov-2017 17:50:38 GMT; Max-Age=7200; path=/ Set-Cookie="whatsmydns_session=eyJpdiI6Ik9ZTVh6TzFPdzE4bmIwOEI3NWdpQnc9PSIsInZhbHVlIjoiKzRtMFRVaEFTSWxuQkFHMVpudGV5U1BoWTZjU1Rocmh4b2pwelBSQ2dsWmJTSklFdmJnaEdCamlNQmN0VHRhTTU2b3NUa3ltXC9DOWxMUW5kekkzckV3PT0iLCJtYWMiOiIwMTk0NGMwYWUxZGM1NDQ"NjRlNTNhNDBjY2NlYTQ0MzAyMzY0YzkyYjY2OTU3NjZkZjQ5MWU3MzFlNTE0MTRkIn0%3D; expires=Thu, 02-Nov-2017 17:50:38 GMT; Max-Age=7200; path=/; HttpOnly

$ ./get-header-val.awk \
  -v item="setcookie" \
  -v limit=all \
  -v keys=1 \
  -v quote="single" \
  ./response-header.txt

Set-Cookie='XSRF-TOKEN=eyJpdiI6Imo0SGJFOWJuVldBeUpMVFVIdnJpVmc9PSIsInZhbHVlIjoiWnhhZmc3ZEM5eTM1VUpvRDdkd0EwQW5hellZdlFcL0ZRQmtcL0xnMW9UTVVoZDZzeHkrcW4rYmNFZ0Q2akFvb2dMT2JxZjE3dzhybW1JMHRibDU5RnZ3QT09IiwibWFjIjoiYTc4MjAzNmY3OWMzNjM1ZDUxZjl'ZTg5YWY0ZjJhOTcxMzVhNThkNjdjYjE0MmI0Zjk1OWRiYTNkZTY0ZjkwZSJ9; expires=Thu, 02-Nov-2017 17:50:38 GMT; Max-Age=7200; path=/ Set-Cookie='whatsmydns_session=eyJpdiI6Ik9ZTVh6TzFPdzE4bmIwOEI3NWdpQnc9PSIsInZhbHVlIjoiKzRtMFRVaEFTSWxuQkFHMVpudGV5U1BoWTZjU1Rocmh4b2pwelBSQ2dsWmJTSklFdmJnaEdCamlNQmN0VHRhTTU2b3NUa3ltXC9DOWxMUW5kekkzckV3PT0iLCJtYWMiOiIwMTk0NGMwYWUxZGM1NDQ'NjRlNTNhNDBjY2NlYTQ0MzAyMzY0YzkyYjY2OTU3NjZkZjQ5MWU3MzFlNTE0MTRkIn0%3D; expires=Thu, 02-Nov-2017 17:50:38 GMT; Max-Age=7200; path=/; HttpOnly

$ ./get-header-val.awk \
  -v item="setcookie" \
  -v limit=all \
  -v format="%s\n" \
  ./response-header.txt

jY2NlYTQ0MzAyMzY0YzkyYjY2OTU3NjZkZjQ5MWU3MzFlNTE0MTRkIn0%3D; expires=Thu, 02-Nov-2017 17:50:38 GMT; Max-Age=7200; path=/; HttpOnly

$ ./get-header-val.awk \
  -v item="setcookie" \
  -v limit=all \
  -v format="%-30s : %s\n" \
  ./response-header.txt

Set-Cookie : XSRF-TOKEN=eyJpdiI6Imo0SGJFOWJuVldBeUpMVFVIdnJpVmc9PSIsInZhbHVlIjoiWnhhZmc3ZEM5eTM1VUpvRDdkd0EwQW5hellZdlFcL0ZRQmtcL0xnMW9UTVVoZDZzeHkrcW4rYmNFZ0Q2akFvb2dMT2JxZjE3dzhybW1JMHRibDU5RnZ3QT09IiwibWFjIjoiYTc4MjAzNmY3OWMzNjM1ZDUxZjlmZTg5YWY0ZjJhOTcxMzVhNThkNjdjYjE0MmI0Zjk1OWRiYTNkZTY0ZjkwZSJ9; expires=Thu, 02-Nov-2017 17:50:38 GMT; Max-Age=7200; path=/ Set-Cookie : whatsmydns_session=eyJpdiI6Ik9ZTVh6TzFPdzE4bmIwOEI3NWdpQnc9PSIsInZhbHVlIjoiKzRtMFRVaEFTSWxuQkFHMVpudGV5U1BoWTZjU1Rocmh4b2pwelBSQ2dsWmJTSklFdmJnaEdCamlNQmN0VHRhTTU2b3NUa3ltXC9DOWxMUW5kekkzckV3PT0iLCJtYWMiOiIwMTk0NGMwYWUxZGM1NDQ1NjRlNTNhNDBjY2NlYTQ0MzAyMzY0YzkyYjY2OTU3NjZkZjQ5MWU3MzFlNTE0MTRkIn0%3D; expires=Thu, 02-Nov-2017 17:50:38 GMT; Max-Age=7200; path=/; HttpOnly

$ ./get-header-val.awk \
  -v format="%s\n" \
  -v keys=0  \
  -v item="setcookie" \
  -v limit=all \
  ./response-header.txt

XSRF-TOKEN=eyJpdiI6Imo0SGJFOWJuVldBeUpMVFVIdnJpVmc9PSIsInZhbHVlIjoiWnhhZmc3ZEM5eTM1VUpvRDdkd0EwQW5hellZdlFcL0ZRQmtcL0xnMW9UTVVoZDZzeHkrcW4rYmNFZ0Q2akFvb2dMT2JxZjE3dzhybW1JMHRibDU5RnZ3QT09IiwibWFjIjoiYTc4MjAzNmY3OWMzNjM1ZDUxZjlmZTg5YWY0ZjJhOTcxMzVhNThkNjdjYjE0MmI0Zjk1OWRiYTNkZTY0ZjkwZSJ9; expires=Thu, 02-Nov-2017 17:50:38 GMT; Max-Age=7200; path=/ whatsmydns_session=eyJpdiI6Ik9ZTVh6TzFPdzE4bmIwOEI3NWdpQnc9PSIsInZhbHVlIjoiKzRtMFRVaEFTSWxuQkFHMVpudGV5U1BoWTZjU1Rocmh4b2pwelBSQ2dsWmJTSklFdmJnaEdCamlNQmN0VHRhTTU2b3NUa3ltXC9DOWxMUW5kekkzckV3PT0iLCJtYWMiOiIwMTk0NGMwYWUxZGM1NDQ1NjRlNTNhNDBjY2NlYTQ0MzAyMzY0YzkyYjY2OTU3NjZkZjQ5MWU3MzFlNTE0MTRkIn0%3D; expires=Thu, 02-Nov-2017 17:50:38 GMT; Max-Age=7200; path=/; HttpOnly

$ ./get-header-val.awk \
  -v quote=single \
  -v delim=": " \
  -v keys=1 \
  -v limit=0 \
  -v item="setcookie" \
  ./response-header.txt
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment