pfsense2-2.grok

pfsense2-2.grok

# GROK match pattern for logstash.conf filter: %{LOG_DATA}%{IP_SPECIFIC_DATA}%{IP_DATA}%{PROTOCOL_DATA}

# GROK Custom Patterns (add to patterns directory and reference in GROK filter for pfSense events):

# GROK Patterns for pfSense 2.2 Logging Format
# 
# Created 27 Jan 2015 by J. Pisano (Handles TCP, UDP, and ICMP log entries)
# Edited 14 Feb 2015 by E. Paul
#
# Usage: Use with following GROK match pattern
#
# %{LOG_DATA}%{IP_SPECIFIC_DATA}%{IP_DATA}%{PROTOCOL_DATA}

LOG_DATA (%{INT:rule}),(%{INT:sub_rule}),,(%{INT:tracker}),(%{WORD:iface}),(%{WORD:reason}),(%{WORD:action}),(%{WORD:direction}),(%{INT:ip_ver}),

IP_SPECIFIC_DATA (%{IPv4_SPECIFIC_DATA}|%{IPv6_SPECIFIC_DATA})

IPv4_SPECIFIC_DATA (%{BASE16NUM:tos}),,(%{INT:ttl}),(%{INT:id}),(%{INT:offset}),(%{WORD:flags}),(%{INT:proto_id}),(%{WORD:proto}),

IPv6_SPECIFIC_DATA (%{BASE16NUM:class}),(%{DATA:flow_label}),(%{INT:hop_limit}),(%{WORD:proto}),(%{INT:proto_id}),

IP_DATA (%{INT:length}),(%{IP:src_ip}),(%{IP:dest_ip}),

PROTOCOL_DATA (%{TCP_DATA}|%{UDP_DATA}|%{ICMP_DATA})

TCP_DATA (%{INT:src_port}),(%{INT:dest_port}),(%{INT:data_length}),(%{WORD:tcp_flags}),(%{INT:sequence_number}),(%{INT:ack_number}),(%{INT:tcp_window}),(%{DATA:urg_data}),(%{DATA:tcp_options})

UDP_DATA (%{INT:src_port}),(%{INT:dest_port}),(%{INT:data_length})

ICMP_DATA (%{ICMP_TYPE}%{ICMP_RESPONSE})

ICMP_TYPE (?<icmp_type>(request|reply|unreachproto|unreachport|unreach|timeexceed|paramprob|redirect|maskreply|needfrag|tstamp|tstampreply)),

ICMP_RESPONSE (%{ICMP_ECHO_REQ_REPLY}|%{ICMP_UNREACHPORT}| %{ICMP_UNREACHPROTO}|%{ICMP_UNREACHABLE}|%{ICMP_NEED_FLAG}|%{ICMP_TSTAMP}|%{ICMP_TSTAMP_REPLY})

ICMP_ECHO_REQ_REPLY (%{INT:icmp_echo_id}),(%{INT:icmp_echo_sequence})

ICMP_UNREACHPORT (%{IP:icmp_unreachport_dest_ip}),(%{WORD:icmp_unreachport_protocol}),(%{INT:icmp_unreachport_port})

ICMP_UNREACHPROTO (%{IP:icmp_unreach_dest_ip}),(%{WORD:icmp_unreachproto_protocol})

ICMP_UNREACHABLE (%{GREEDYDATA:icmp_unreachable})

ICMP_NEED_FLAG (%{IP:icmp_need_flag_ip}),(%{INT:icmp_need_flag_mtu})

ICMP_TSTAMP (%{INT:icmp_tstamp_id}),(%{INT:icmp_tstamp_sequence})

ICMP_TSTAMP_REPLY (%{INT:icmp_tstamp_reply_id}),(%{INT:icmp_tstamp_reply_sequence}),(%{INT:icmp_tstamp_reply_otime}),(%{INT:icmp_tstamp_reply_rtime}),(%{INT:icmp_tstamp_reply_ttime})

Very nice!

Please take a look at my fork at https://gist.github.com/rudibroekhuizen/bb1fb0182fb860a1cfda. I've added a pattern for the pfsync protocol, which is used to sync two pfsense firewalls.

One line changed:
PROTOCOL_DATA (%{TCP_DATA}|%{UDP_DATA}|%{ICMP_DATA}|%{PFSYNC_DATA})

One line added:
PFSYNC_DATA (datalength=%{INT:data_length})

An pfsync event looks like this:
9,16777216,,1000000103,em0,match,block,in,4,0x10,,255,8086,0,DF,240,pfsync,636,192.168.1.1,192.168.1.2,datalength=616

I have tried a few tweaks but for th elife of me can not get fields to come in as INT or NUMBER.
I have been focusing on this line
IP_DATA (%{INT:length}),(%{IP:src_ip}),(%{IP:dest_ip}),
I have tried
IP_DATA (%{NUMBER:length}),(%{IP:src_ip}),(%{IP:dest_ip}),
IP_DATA (%{NUMBER:length:int}),(%{IP:src_ip}),(%{IP:dest_ip}),

But logstash always recognizes it as String anyone else having this issue.
Majority of fields are string the only number ones are the
geoip.latitude,
geoip.area_code,
geoip.longitude,
geoip.dma_code,
_score
fields.

I noticed that ack_number is not always present, so I had to change it to (%{INT:ack_number})?
See my fork

I added optional anchor/ecn fields in my fork after reviewing parse failures.