Useful awk commands for extracting data from logs

awk-commands.md

$1 is the first column. awk separates columns by spaces by default. delimiters can be specified with -F. In the examples below '[: ]' means that columns are delimited by spaces or colons.

99.56.8.181 10.0.1.239 - - [16/Nov/2018:20:45:59 +0000] "GET /app/themes/finecooking/dist/img/marketing-hero-cover.jpg HTTP/1.0" 200 38808 "https://www.finecooking.com/sw.js" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:63.0) Gecko/20100101 Firefox/63.0"

Print IP and user agent for requests between the hours of 18 and 19

cat /var/log/httpd/access_log | awk -F'[: ]' '$6 >= 18 && $6 <= 19 { print }' | awk -F\" '{print $1,$6}' | more

Print entire log entry for requests between the hours of 18 and 19 and status code is 301

cat /var/log/httpd/access_log | awk -F'[: ]' '$6 >= 18 && $6 <= 19 && $13 == 301 { print }' | more

Find referrers matching a pattern

cat /var/log/httpd/access_log | awk -F'"' '$4~/(menshealth\.com|fitnessmagazine\.com|seriouseats\.com|giants\.com|soaphub\.com|tmz\.com|bleacherreport\.com)/ {print $1,$4}' | more

Find requests matching a pattern

cat /var/log/httpd/access_log | awk -F'"' '$2~/\/search\?q=/ {print}' | more

More links

• https://coderwall.com/p/_zkogw/awk-to-explore-your-access-log-files
• https://coderwall.com/p/ueazhw/parsing-logs-with-awk