Required jars:
- log4j2 (comes with flink)
- log4j2-logstash-layout (https://github.com/vy/log4j2-logstash-layout)
- jackson-core (my version is 2.12.0)
- jackson-databind (my version is 2.12.0)
- jackson-annotations (my version is 2.12.0)
Note: It seems that since Log4j 2.14.0, we do not need to add jackson dependencies according to log4j documentation. However I need to stick with log4j2 version comes with Flink for now, so I have not tested it yet.
log4j2 logstash config example properties file for flink 1.11.1:
rootLogger.level = WARN
rootLogger.appenderRef.logstash.ref = LogstashAppender
appender.main.name = LogstashAppender
appender.main.type = Socket
appender.main.host = logstash.myhost.com
appender.main.port = 5044
appender.main.logstash.type = LogstashLayout
appender.main.logstash.dateTimeFormatPattern = yyyy-MM-dd'T'HH:mm:ss.SSSZZZ
appender.main.logstash.eventTemplateUri = classpath:LogstashJsonEventLayoutV1.json
appender.main.logstash.prettyPrintEnabled = false
appender.main.logstash.stackTraceEnabled = true
# adds custom field as 'source=flink'
appender.main.logstash.eventTemplateAdditionalFields.type = EventTemplateAdditionalFields
appender.main.logstash.eventTemplateAdditionalFields.sourceField.type = KeyValuePair
appender.main.logstash.eventTemplateAdditionalFields.sourceField.key = source
appender.main.logstash.eventTemplateAdditionalFields.sourceField.value = flink
logger.myhost.name = myhost.com
logger.myhost.level = DEBUG
logger.netty.name = org.apache.flink.shaded.akka.org.jboss.netty.channel.DefaultChannelPipeline
logger.netty.level = OFF