-
-
Save brandond/cc57062e4eb8628ec20c53f836882a67 to your computer and use it in GitHub Desktop.
| # VPC Flow Log fields | |
| # version account-id interface-id srcaddr dstaddr srcport dstport protocol packets bytes start end action log-status | |
| # http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/flow-logs.html#flow-log-records | |
| VPCFLOWLOG %{NUMBER:version:int} %{NUMBER:account-id} %{NOTSPACE:interface-id} %{NOTSPACE:srcaddr} %{NOTSPACE:dstaddr} %{NOTSPACE:srcport:int} %{NOTSPACE:dstport:int} %{NOTSPACE:protocol:int} %{NOTSPACE:packets:int} %{NOTSPACE:bytes:int} %{NUMBER:start:int} %{NUMBER:end:int} %{NOTSPACE:action} %{NOTSPACE:log-status} |
jonbrouse
commented
Nov 25, 2016
Great!!
Thank you very much
You've made me hate life a little less today. Thank you.
This works when all fields are populated, but when values are missing, the grok patterns do not match data in the input.
For example, given this line:
2 843709267542 eni-a1a3e46e - - - - - - - 1544040246 1544040811 - NODATA
the pattern matching fails. Is there a way account for dashes?
Hi
You can use following pattern for matching both patterns
(?<version>\S+)\s(?<account-id>\S+)\s(?<interface-id>\S+)\s(?<srcaddr>\S+)\s(?<dstaddr>\S+)\s(?<srcport>\S+)\s(?<dstport>\S+)\s(?<protocol>\S+)\s(?<packets>\S+)\s(?<bytes>\S+)\s(?<start>\S+)\s(?<end>\S+)\s(?<action>\S+)\s(?<log-status>\S+)
Thank you!
Gr8 job m8
for the latest vpc flow logs format, i am using below pattern
%{NUMBER:version:int} %{NOTSPACE:vpc-id} %{NOTSPACE:region} %{NOTSPACE:subnet-id} %{NOTSPACE:instance-id} %{NOTSPACE:interface-id} %{NUMBER:account-id} %{NOTSPACE:type} %{NOTSPACE:srcaddr} %{NOTSPACE:dstaddr} %{NOTSPACE:srcport:int} %{NOTSPACE:dstport:int} %{NOTSPACE:pkt-srcaddr} %{NOTSPACE:pkt-dstaddr} %{NOTSPACE:protocol:int} %{NOTSPACE:bytes:int} %{NOTSPACE:packets:int} %{NUMBER:start:int} %{NUMBER:end:int} %{NOTSPACE:action} %{NOTSPACE:tcp-flags} %{NOTSPACE:log-status}
an update to this for version 5:
%{NUMBER:version:int} %{NUMBER:account-id} %{NOTSPACE:interface-id} %{NOTSPACE:srcaddr} %{NOTSPACE:dstaddr} %{NOTSPACE:srcport:int} %{NOTSPACE:dstport:int} %{NOTSPACE:protocol:int} %{NOTSPACE:packets:int} %{NOTSPACE:bytes:int} %{NUMBER:start:int} %{NUMBER:end:int} %{NOTSPACE:action} %{NOTSPACE:log-status} %{NOTSPACE:vpc-id} %{NOTSPACE:subnet-id} %{NOTSPACE:instance-id} %{NOTSPACE:tcp-flags} %{NOTSPACE:type} %{NOTSPACE:pkt-srcaddr} %{NOTSPACE:pkt-dstaddr} %{NOTSPACE:region} %{NOTSPACE:az-id} %{NOTSPACE:sublocation-type} %{NOTSPACE:sublocation-id} %{NOTSPACE:flow-direction} %{GREEDYDATA:other}