Finalized log format for logstash parsing of NGINX log using grok

nginx-log-format

NGINX reverse proxy sample access log: 11.22.33.44 [20/Nov/2016:15:31:06 +0600] GET "/depot/359551/chunk/9b49d7a7b2fb193e0eb406ed17a500d210844bd7" "OK" 200 651616 "-" "Valve/Steam HTTP Client 1.0" "HIT" "valve285.steamcontent.com" "-" "0.000" "-"
NGINX reverse-proxy log format: log_format main '$remote_addr [$time_local] $request_method "$uri" $request_completion $status $body_bytes_sent "$http_referer" "$http_user_agent" "$upstream_cache_status" "$host" "$http_range" "$request_time" "$upstream_response_time"';
GROK format: %{IP:clientip} \[%{HTTPDATE:date}\] %{WORD:req_method} \"%{URIPATHPARAM:request}\" \"%{WORD:req_complete}\" %{NUMBER:http_status_code} %{NUMBER:bytes_sent} %{QS:referrer} %{QS:agent} \"%{WORD:upstream_cache_status}\" \"%{HOSTNAME:host}\" %{QS:http_range} %{QS:req_time} %{QS:res_time}

Sample default NGINX access log: 195.211.148.195 - - [20/Nov/2016:15:30:37 +0600] "GET http://www.msftncsi.com/ncsi.txt HTTP/1.1" 404 580 "-" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0; WOW64; .NET CLR 1.1.4322; Media Center PC 5.0; .NET CLR 3.5.30729; .NET CLR 3.0.30729; .NET4.0C; .NET4.0E)"
GROK format: %{IP:clientip} \- \- \[%{HTTPDATE:date}\] \"%{WORD:action} http:%{URIPATHPARAM:request} HTTP/%{NUMBER:httpversion}\" %{NUMBER:response} %{NUMBER:bytes} %{QS:ignore} %{QS:referrer}


NGINX reverse proxy sample error log: 2016/11/20 17:17:42 [error] 25960#25960: *12540 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 10.27.211.6, server: *.cs.steampowered.com, request: "GET /depot/291481/chunk/92edc14f8fafc7d005cfe5b819feaff3a36a20e9?valid_until=1480170059&hash=4a781984bc8a71de6801d905d17f36f0 HTTP/1.1", upstream: "http://11.222.33.44:80/depot/291481/chunk/92edc14f8fafc7d005cfe5b819feaff3a36a20e9?valid_until=1480170059&hash=4a781984bc8a71de6801d905d17f36f0", host: "cdn.edgecast.cs.steampowered.com"
GROK format: %{DATE:date} %{TIME:time} \[%{WORD:log_level}\] (%{NUMBER:pid:int}#%{NUMBER}: \*%{NUMBER}|\*%{NUMBER}) %{DATA:err_message}(?:, client: (?<client_ip>%{IP}|%{HOSTNAME}))(?:, server: %{NOTSPACE:server})(?:, request: \"%{WORD:req_type} %{URIPATHPARAM:request} %{NOTSPACE:http_ver}\")(?:, upstream: \"%{NOTSPACE:upstream}\")(?:, host: \"%{NOTSPACE:host}\")