logstash-grok-pattern-vpc-flow-log

vpc-flowlog

# VPC Flow Log fields
# version account-id interface-id srcaddr dstaddr srcport dstport protocol packets bytes start end action log-status
# http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/flow-logs.html#flow-log-records
VPCFLOWLOG %{NUMBER:version:int} %{NUMBER:account-id} %{NOTSPACE:interface-id} %{NOTSPACE:srcaddr} %{NOTSPACE:dstaddr} %{NOTSPACE:srcport:int} %{NOTSPACE:dstport:int} %{NOTSPACE:protocol:int} %{NOTSPACE:packets:int} %{NOTSPACE:bytes:int} %{NUMBER:start:int} %{NUMBER:end:int} %{NOTSPACE:action} %{NOTSPACE:log-status}

You've made me hate life a little less today. Thank you.

This works when all fields are populated, but when values are missing, the grok patterns do not match data in the input.

For example, given this line:

2 843709267542 eni-a1a3e46e - - - - - - - 1544040246 1544040811 - NODATA

the pattern matching fails. Is there a way account for dashes?

Hi
You can use following pattern for matching both patterns

(?<version>\S+)\s(?<account-id>\S+)\s(?<interface-id>\S+)\s(?<srcaddr>\S+)\s(?<dstaddr>\S+)\s(?<srcport>\S+)\s(?<dstport>\S+)\s(?<protocol>\S+)\s(?<packets>\S+)\s(?<bytes>\S+)\s(?<start>\S+)\s(?<end>\S+)\s(?<action>\S+)\s(?<log-status>\S+)

Thank you!

Gr8 job m8

for the latest vpc flow logs format, i am using below pattern

%{NUMBER:version:int} %{NOTSPACE:vpc-id} %{NOTSPACE:region} %{NOTSPACE:subnet-id} %{NOTSPACE:instance-id} %{NOTSPACE:interface-id} %{NUMBER:account-id} %{NOTSPACE:type} %{NOTSPACE:srcaddr} %{NOTSPACE:dstaddr} %{NOTSPACE:srcport:int} %{NOTSPACE:dstport:int} %{NOTSPACE:pkt-srcaddr} %{NOTSPACE:pkt-dstaddr} %{NOTSPACE:protocol:int} %{NOTSPACE:bytes:int} %{NOTSPACE:packets:int} %{NUMBER:start:int} %{NUMBER:end:int} %{NOTSPACE:action} %{NOTSPACE:tcp-flags} %{NOTSPACE:log-status}

an update to this for version 5:

%{NUMBER:version:int} %{NUMBER:account-id} %{NOTSPACE:interface-id} %{NOTSPACE:srcaddr} %{NOTSPACE:dstaddr} %{NOTSPACE:srcport:int} %{NOTSPACE:dstport:int} %{NOTSPACE:protocol:int} %{NOTSPACE:packets:int} %{NOTSPACE:bytes:int} %{NUMBER:start:int} %{NUMBER:end:int} %{NOTSPACE:action} %{NOTSPACE:log-status} %{NOTSPACE:vpc-id} %{NOTSPACE:subnet-id} %{NOTSPACE:instance-id} %{NOTSPACE:tcp-flags} %{NOTSPACE:type} %{NOTSPACE:pkt-srcaddr} %{NOTSPACE:pkt-dstaddr} %{NOTSPACE:region} %{NOTSPACE:az-id} %{NOTSPACE:sublocation-type} %{NOTSPACE:sublocation-id} %{NOTSPACE:flow-direction} %{GREEDYDATA:other}