Skip to content

Instantly share code, notes, and snippets.

@KarimJedda

KarimJedda/tail_to_sqlite.sh

Last active June 11, 2020 18:58
Show Gist options
  • Save KarimJedda/8ddb5391fc32dcf29fed0ca1b653a05d to your computer and use it in GitHub Desktop.
Save KarimJedda/8ddb5391fc32dcf29fed0ca1b653a05d to your computer and use it in GitHub Desktop.
Download ZIP
Tailing Nginx logs to SQLite
Raw
tail_to_sqlite.sh
tail -f /var/log/nginx/access.log | \
# parsing every line
gawk '{print $1 " " substr($4, 2, length($4) - 8) " " $6 " " $7 " " substr($5, 2, length($5) -2) " " substr($8, 2, length($8)-2) " " $9; system("")}' FPAT='[^ ]*|"[^"]*"|\\[[^]]*\\]' | \
# inserting every new line into sqlite
(while read ip timestamp status_code bytes_sent request_method request_url request_protocol referrer user_agent; do sqlite3 -batch /home/logger/logs.db "insert into todo (ip, timestamp, status_code, bytes_sent, request_method, request_url, request_protocol, referrer, user_agent) values (\"$ip\",\"$timestamp\", \"$status_code\", \"$bytes_sent\", \"$request_method\", \"$request_url\", \"$request_protocol\" ,\"$referrer\" , $user_agent);"; done )
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment