postfix grok patterns for logstash

postfix

# Syslog stuff
COMPONENT ([\w._\/%-]+)
COMPID postfix\/%{COMPONENT:component}(?:\[%{NUMBER:pid}\])?
POSTFIX (?:%{SYSLOGTIMESTAMP:timestamp}|%{TIMESTAMP_ISO8601:timestamp8601}) (?:%{SYSLOGFACILITY} )?%{SYSLOGHOST:logsource} %{COMPID}:

# Milter
HELO (?:\[%{IP:helo}\]|%{HOST:helo}|%{DATA:helo})

MILTERCONNECT %{QUEUEID:qid}: milter-reject: CONNECT from %{RELAY:relay}: %{GREEDYDATA:milter_reason}; proto=%{WORD:proto}
MILTERUNKNOWN %{QUEUEID:qid}: milter-reject: UNKNOWN from %{RELAY:relay}: %{GREEDYDATA:milter_reason}; proto=%{WORD:proto}
MILTEREHLO %{QUEUEID:qid}: milter-reject: EHLO from %{RELAY:relay}: %{GREEDYDATA:milter_reason}; proto=%{WORD:proto} helo=<%{HELO}>
MILTERMAIL %{QUEUEID:qid}: milter-reject: MAIL from %{RELAY:relay}: %{GREEDYDATA:milter_reason}; from=<%{EMAILADDRESS:from}> proto=%{WORD:proto} helo=<%{HELO}>
MILTERHELO %{QUEUEID:qid}: milter-reject: HELO from %{RELAY:relay}: %{GREEDYDATA:milter_reason}; proto=%{WORD:proto} helo=<%{HELO}>
MILTERRCPT %{QUEUEID:qid}: milter-reject: RCPT from %{RELAY:relay}: %{GREEDYDATA:milter_reason}; from=<%{EMAILADDRESS:from}> to=<%{EMAILADDRESS:to}> proto=%{WORD:proto} helo=<%{HELO}>
MILTERENDOFMESSAGE %{QUEUEID:qid}: milter-reject: END-OF-MESSAGE from %{RELAY:relay}: %{GREEDYDATA:milter_reason}; from=<%{EMAILADDRESS:from}> to=<%{EMAILADDRESS:to}> proto=%{WORD:proto} helo=<%{HELO}>

# Postfix stuff
HELO (?:\[%{IP:helo}\]|%{HOST:helo}|%{DATA:helo})
QUEUEID (?:[A-F0-9]+|NOQUEUE)
EMAILADDRESSPART [a-zA-Z0-9_.+-=:~]+
EMAILADDRESS %{EMAILADDRESSPART:local}@%{EMAILADDRESSPART:remote}
RELAY (?:%{HOSTNAME:relayhost}(?:\[%{IP:relayip}\](?::[0-9]+(.[0-9]+)?)?)?)
POSREAL [0-9]+(.[0-9]+)?
DSN %{NONNEGINT}.%{NONNEGINT}.%{NONNEGINT}
STATUS sent|deferred|bounced|expired
PERMERROR 5[0-9]{2}
MESSAGELEVEL reject|warning|error|fatal|panic

POSTFIXSMTPMESSAGE %{MESSAGELEVEL}: %{GREEDYDATA:reason}
POSTFIXACTION discard|dunno|filter|hold|ignore|info|prepend|redirect|replace|reject|warn

# postfix/smtp and postfix/lmtp, postfix/local and postfix/error
POSTFIXSMTP %{POSTFIXSMTPRELAY}|%{POSTFIXSMTPCONNECT}|%{POSTFIXSMTP5XX}|%{POSTFIXSMTPREFUSAL}|%{POSTFIXSMTPLOSTCONNECTION}|%{POSTFIXSMTPTIMEOUT}
POSTFIXSMTPRELAY %{QUEUEID:qid}: to=<%{DATA:to}>,(?:\sorig_to=<%{DATA:orig_to}>,)? relay=%{RELAY},(?: delay=%{POSREAL:delay},)?(?: delays=%{DATA:delays}?,)?(?: conn_use=%{POSREAL:conn_use},)?( %{WORD}=%{DATA},)+? dsn=%{DSN:dsn}, status=%{STATUS:result} %{GREEDYDATA:reason}
POSTFIXSMTPCONNECT connect to %{RELAY}: %{GREEDYDATA:reason}
POSTFIXSMTP5XX %{QUEUEID:qid}: to=<%{EMAILADDRESS:to}>,(?:\sorig_to=<%{EMAILADDRESS:orig_to}>,)? relay=%{RELAY}, (%{WORD}=%{DATA},)+ dsn=%{DSN:dsn}, status=%{STATUS:result} \(host %{HOSTNAME}\[%{IP}\] said: %{PERMERROR:responsecode} %{DATA:smtp_response} \(in reply to %{DATA:command} command\)\)
POSTFIXSMTPREFUSAL %{QUEUEID:qid}: host %{RELAY} refused to talk to me: %{GREEDYDATA:reason}
POSTFIXSMTPLOSTCONNECTION %{QUEUEID:qid}: lost connection with %{RELAY} while %{GREEDYDATA:reason}
POSTFIXSMTPTIMEOUT %{QUEUEID:qid}: conversation with %{RELAY} timed out while %{GREEDYDATA:reason}

# postfix/smtpd
POSTFIXSMTPD %{POSTFIXSMTPDCONNECTS}|%{POSTFIXSMTPDMILTER}|%{POSTFIXSMTPDACTIONS}|%{POSTFIXSMTPDTIMEOUTS}|%{POSTFIXSMTPDLOGIN}|%{POSTFIXSMTPDCLIENT}|%{POSTFIXSMTPDNOQUEUE}|%{POSTFIXSMTPDWARNING}|%{POSTFIXSMTPDLOSTCONNECTION}
POSTFIXSMTPDCONNECTS (?:dis)?connect from %{RELAY}
POSTFIXSMTPDMILTER %{MILTERCONNECT}|%{MILTERUNKNOWN}|%{MILTEREHLO}|%{MILTERMAIL}|%{MILTERHELO}|%{MILTERRCPT}
POSTFIXSMTPDACTIONS %{QUEUEID:qid}: %{POSTFIXACTION:postfix_action}: %{DATA:command} from %{RELAY}: %{PERMERROR:responsecode} %{DSN:dsn} %{DATA}: %{DATA:reason}; from=<%{EMAILADDRESS:from}> to=<%{EMAILADDRESS:to}> proto=%{DATA:proto} helo=<%{HELO}>
POSTFIXSMTPDTIMEOUTS timeout after %{DATA:command} from %{RELAY}
POSTFIXSMTPDLOGIN %{QUEUEID:qid}: client=%{DATA:client}, sasl_method=%{DATA:saslmethod}, sasl_username=%{GREEDYDATA:saslusername}
POSTFIXSMTPDCLIENT %{QUEUEID:qid}: client=%{GREEDYDATA:client}
POSTFIXSMTPDNOQUEUE NOQUEUE: %{POSTFIXACTION:postfix_action}: %{DATA:command} from %{RELAY}: %{GREEDYDATA:reason}
POSTFIXSMTPDWARNING warning:( %{IP}: | hostname %{HOSTNAME} )?%{GREEDYDATA:reason}
POSTFIXSMTPDLOSTCONNECTION (?:lost connection after %{DATA:smtp_response} from %{RELAY}|improper command pipelining after HELO from %{GREEDYDATA:reason})

# postfix/cleanup
POSTFIXCLEANUP %{POSTFIXCLEANUPMESSAGE}|%{POSTFIXCLEANUPMILTER}
POSTFIXCLEANUPMESSAGE %{QUEUEID:qid}: (resent-)?message-id=(<)?%{GREEDYDATA:messageid}(>)?
POSTFIXCLEANUPMILTER %{MILTERENDOFMESSAGE}

# postfix/bounce
POSTFIXBOUNCE %{QUEUEID:qid}: sender (non-)?delivery( status)? notification: %{QUEUEID:bouncequeueid}

# postfix/qmgr and postfix/pickup
POSTFIXQMGR %{QUEUEID:qid}: (?:removed|from=<(?:%{DATA:from})?>(?:, size=%{NUMBER:size}, nrcpt=%{NUMBER:nrcpt} \(%{GREEDYDATA:queuestatus}\))?)

# postfix/anvil
POSTFIXANVIL statistics: %{GREEDYDATA:reason}

# postfix/trivial-rewrite
POSTFIXREWRITE warning: do not list domain %{DATA:domain} in BOTH mydestination and virtual_alias_domains

# Dovecot
DOVEIMAP imap\(%{DATA:user}\): %{DATA:reason} in=%{NUMBER:inbytes} out=%{NUMBER:outbytes}
DOVECMD anvil|auth|config|log|master
DOVEMISC %{DOVECMD:command}: %{GREEDYDATA:reason}
DOVELOGIN imap-login: %{DATA:action}:(?: user=<(%{DATA:user})?>, (method=%{DATA:loginmethod}, )?rip=%{IP:rip}, lip=%{IP:lip},( mpid=%{NUMBER:mpid},( %{DATA:sectype},)?| %{DATA:securesession},)? session=<%{DATA:session}>| %{GREEDYDATA:reason})
DOVELDA lda\((%{DATA:user})?\):( %{DATA:action}:)? msgid=(?:<%{DATA:mesgid}@%{DATA:domain}>|%{DATA:mesgid}):( saved mail to| stored mail into mailbox) .*?%{DATA:folder}.*?
DOVEAUTH auth-worker\(%{NUMBER:pid}\): pam\((?:%{USERNAME:user}|%{EMAILADDRESS:user}),%{IP:ip}\): %{GREEDYDATA:reason}

# Dovecot wrap em up
DOVECOT (?:%{SYSLOGTIMESTAMP:timestamp}|%{TIMESTAMP_ISO8601:timestamp8601}) (?:%{SYSLOGFACILITY} )?%{SYSLOGHOST:logsource} dovecot: (%{DOVEIMAP}|%{DOVELOGIN}|%{DOVELDA}|%{DOVEAUTH}|%{DOVEMISC})

# Postfix wrap em up
PF %{POSTFIX} (?:%{POSTFIXSMTP}|%{POSTFIXANVIL}|%{POSTFIXQMGR}|%{POSTFIXBOUNCE}|%{POSTFIXCLEANUP}|%{POSTFIXSMTPD}|%{POSTFIXREWRITE})

Hi,

I have put the logstash.conf in the /etc/logstash folder and grok pattern in /etc/logstash/patterns folder

When I start the logstash server I get this error in the logstash.log

oose\nto restart a running system."}
{:timestamp=>"2016-02-24T12:39:12.053000-0500", :message=>"Error: Expected one of #, input, filter, output at line 45, column 1 (byte 878) after "}
{:timestamp=>"2016-02-24T12:39:12.072000-0500", :message=>"You may be interested in the '--configtest' flag which you can\nuse to validate logstash's configuration before you choose\nto restart a running system."}
{:timestamp=>"2016-02-24T12:46:28.668000-0500", :message=>"Error: Expected one of #, input, filter, output at line 45, column 1 (byte 878) after "}
{:timestamp=>"2016-02-24T12:46:28.689000-0500", :message=>"You may be interested in the '--configtest' flag which you can\nuse to validate logstash's configuration before you ch

However

/opt/logstash/bin/logstash -f /etc/logstash/conf.d/logstash.conf --configtest
Configuration OK

I am a beginner and I am trying to use filebeat. Am I doing something wrong here ?

Regards,

Where is MILTERENDOFMESSAGE supposed to be defined? When testing on http://grokconstructor.appspot.com/do/match the app throws this error:

This grok pattern has an unknown name MILTERENDOFMESSAGE : %{MILTERENDOFMESSAGE}

I was testing the %{PF} filter using some logs from mail.log I pasted in.

@athreyavc1

If you haven't figured it out yet, I'd suggest checking all files in the /etc/logstash/conf.d directory. Not just logstash.conf. Logstash concatnates all the files in there together. So if you're just testing one file, you won't be testing everything.

I think you can just do -f /etc/logstash/conf.d --configtest as well.

@jerrac

I've updated the above with the milter patterns.

@athreyavc1

You can put the patterns file anywhere. Then specify its location in the grok filter.

e.g.

        grok {
            patterns_dir => ["/home/logstash/config/patterns"]
            match => { "message" => ["%{PF}", "%{DOVECOT}" ] }
        }

This is defined twice and incorrect:

HELO (?:\[%{IP:helo}\]|%{HOST:helo}|%{DATA:helo})
It should be:
HELO (?:\[%{IP:helo}\]|%{HOSTNAME:helo}|%{DATA:helo})

Hi,

I am facing below error while running logstash

I am using logstash-5.4.1

Sending Logstash's logs to /export/workspace/elk/logstash-5.4.1/logs which is now configured via log4j2.properties
[2017-06-16T16:07:46,249][ERROR][logstash.pipeline ] Error registering plugin {:plugin=>"#<LogStash::FilterDelegator:0x5741236e @id="4c47f094fbc2b16c560f40e2a4d5aa22733d4937-2", @klass=LogStash::Filters::Grok, @metric_events=#<LogStash::Instrument::NamespacedMetric:0x46e91c42 @Metric=#<LogStash::Instrument::Metric:0x114e8494 @collector=#<LogStash::Instrument::Collector:0x74c40b5b @agent=nil, @metric_store=#<LogStash::Instrument::MetricStore:0x2a401cfe @store=#<Concurrent::Map:0x24451ef8 @default_proc=nil>, @structured_lookup_mutex=#Mutex:0x414a54c0, @fast_lookup=#<Concurrent::Map:0x15001161 @default_proc=nil>>>>, @namespace_name=[:stats, :pipelines, :main, :plugins, :filters, :"4c47f094fbc2b16c560f40e2a4d5aa22733d4937-2", :events]>, @logger=#<LogStash::Logging::Logger:0x33333304 @logger=#Java::OrgApacheLoggingLog4jCore::Logger:0x50370495>, @filter=<LogStash::Filters::Grok patterns_dir=>["/home/apptivo/Downloads/grok_pattern"], match=>{"message"=>["%{PF}"]}, id=>"4c47f094fbc2b16c560f40e2a4d5aa22733d4937-2", enable_metric=>true, periodic_flush=>false, patterns_files_glob=>"*", break_on_match=>true, named_captures_only=>true, keep_empty_captures=>false, tag_on_failure=>["_grokparsefailure"], timeout_millis=>30000, tag_on_timeout=>"_groktimeout">>", :error=>"pattern %{HOST:helo} not defined"}

[2017-06-16T16:07:46,259][ERROR][logstash.agent ] Pipeline aborted due to error {:exception=>#<Grok::PatternError: pattern %{HOST:helo} not defined>, :backtrace=>["/export/workspace/elk/logstash-5.4.1/vendor/bundle/jruby/1.9/gems/jls-grok-0.11.4/lib/grok-pure.rb:123:in compile'", "org/jruby/RubyKernel.java:1479:in loop'", "/export/workspace/elk/logstash-5.4.1/vendor/bundle/jruby/1.9/gems/jls-grok-0.11.4/lib/grok-pure.rb:93:in compile'", "/export/workspace/elk/logstash-5.4.1/vendor/bundle/jruby/1.9/gems/logstash-filter-grok-3.4.0/lib/logstash/filters/grok.rb:286:in register'", "org/jruby/RubyArray.java:1613:in each'", "/export/workspace/elk/logstash-5.4.1/vendor/bundle/jruby/1.9/gems/logstash-filter-grok-3.4.0/lib/logstash/filters/grok.rb:280:in register'", "org/jruby/RubyHash.java:1342:in each'", "/export/workspace/elk/logstash-5.4.1/vendor/bundle/jruby/1.9/gems/logstash-filter-grok-3.4.0/lib/logstash/filters/grok.rb:275:in register'", "/export/workspace/elk/logstash-5.4.1/logstash-core/lib/logstash/pipeline.rb:268:in register_plugin'", "/export/workspace/elk/logstash-5.4.1/logstash-core/lib/logstash/pipeline.rb:279:in register_plugins'", "org/jruby/RubyArray.java:1613:in each'", "/export/workspace/elk/logstash-5.4.1/logstash-core/lib/logstash/pipeline.rb:279:in register_plugins'", "/export/workspace/elk/logstash-5.4.1/logstash-core/lib/logstash/pipeline.rb:289:in start_workers'", "/export/workspace/elk/logstash-5.4.1/logstash-core/lib/logstash/pipeline.rb:214:in run'", "/export/workspace/elk/logstash-5.4.1/logstash-core/lib/logstash/agent.rb:398:in `start_pipeline'"]}

Check the pattern file and see whether {HOST:helo} has been misspelt or not defined in the patterns file for that is where the error emanates from.

Is there any postfix filter for multiple lines.

Sep 23 18:11:48 share postfix/pickup[11067]: E1010623160E: uid=0 from=[email protected]
Sep 23 18:11:48 share postfix/cleanup[17185]: E1010623160E: warning: header Subject: Review and update IT assets - US from local; from=[email protected] to=[email protected]
Sep 23 18:11:48 share postfix/cleanup[17185]: E1010623160E: message-id=5d88bd8c.53hwfoOxAPVNmQpT%[email protected]
Sep 23 18:11:48 share postfix/cleanup[17185]: E1010623160E: warning: header Content-Disposition: attachment;? filename="ConfIncomingPostfixForTM.sh" from local; from=[email protected] to=[email protected]
Sep 23 18:11:48 share postfix/qmgr[11068]: E1010623160E: from=[email protected], size=13835, nrcpt=1 (queue active)
Sep 23 18:11:49 share postfix/smtp[17187]: E1010623160E: to=[email protected], relay=abc.cde.com[xx.xx.xx.xx]:587, delay=0.82, delays=0.12/0.01/0.23/0.47, dsn=2.0.0, status=sent (250 ok 1569242509 qp 55892 by mail.com)
Sep 23 18:11:49 share postfix/qmgr[11068]: E1010623160E: removed

I want to filter following content from these lines.
from=[email protected], Subject: Review and update IT assets - US, to=[email protected], message-id=5d88bd8c.53hwfoOxAPVNmQpT%[email protected], attachment;? filename="ConfIncomingPostfixForTM.sh", size=13835, relay=abc.cde.com[xx.xx.xx.xx]:587, delay=0.82, delays=0.12/0.01/0.23/0.47, dsn=2.0.0, status=sent

Postfix pattern causes error in latest Logstash:

[2020-10-01T11:28:30,027][ERROR][logstash.agent ] Failed to execute action {:id=>:main, :action_type=>LogStash::ConvergeResult::FailedAction, :message=>"Could not execute action: PipelineAction::Create<main>, action_result: false", :backtrace=>nil}

HELO (?:[%{IP:helo}]|%{HOSTNAME:helo}|%{DATA:helo})

This was the cause of the error. Thanks.