rsyslog conf.d file to handle FortiAnalyzer's malformed CEF format and ingest to Sentinel via AMA

40-fortianalyzer-filtered-cef.conf

#####################################################################
# FortiAnalyzer CEF Integration with Severity-Based Noise Reduction
#####################################################################
#
# PURPOSE:
# --------
# This configuration solves two critical problems with FortiAnalyzer
# CEF log forwarding to Microsoft Sentinel:
#
# 1. PROTOCOL COMPLIANCE: FortiAnalyzer sends raw CEF messages without
#    RFC3164/RFC5424 compliant syslog PRI headers. Azure Monitor Agent
#    (AMA) requires properly formatted syslog messages and will reject
#    or misprocess headerless CEF data.
#
# 2. LOG VOLUME REDUCTION: FortiAnalyzer generates high volumes of
#    informational and notice-level logs that create noise in Sentinel,
#    increase ingestion costs, and reduce alert-to-noise ratio.
#
# SOLUTION:
# ---------
# - Receives raw CEF messages from FortiAnalyzer on UDP port 1514
# - Adds mandatory syslog PRI header <164> (local4.warning) for AMA compliance
# - Implements severity-based filtering at rsyslog level (before AMA)
# - Only forwards security-relevant events (critical/high/medium severity)
# - Filters out noise (notice/info/debug severity) before network transmission
#
# BENEFITS:
# ---------
# - 60-80% reduction in Sentinel ingestion volume and costs
# - Improved security alert-to-noise ratio
# - Reduced bandwidth usage between log collector and Azure
# - Maintains full compliance with Azure Monitor Agent requirements
# - Preserves all security-critical events while eliminating operational noise
#
# FILTERING LOGIC:
# ----------------
# FORWARDED: deviceSeverity=critical, high, medium
# DROPPED: deviceSeverity=notice, info, debug
#
# IMPORTANT - FORTIANALYZER RELIABLE CONNECTION MODE:
# --------------------------------------------------
# By default, FortiAnalyzer sends logs via UDP (unreliable, best-effort delivery).
# When "Reliable Connection" is enabled in FortiAnalyzer, it switches to TCP.
# 
# This configuration ONLY listens on UDP port 1514 by default.
# If you enable "Reliable Connection" in FortiAnalyzer, logs will stop flowing!
#
# To support TCP mode, uncomment the TCP INPUT SECTION below.
# You can safely enable both UDP and TCP listeners simultaneously.
#
# DEPLOYMENT:
# -----------
# 1. Configure FortiAnalyzer to forward CEF logs to this server:1514
# 2. Ensure Azure Monitor Agent is configured with appropriate DCR
# 3. This setup assumes that the DCR is listening for LOG_LOCAL4:LOG_WARNING
# 4. If FortiAnalyzer "Reliable Connection" is enabled, uncomment TCP section
# 5. If debugging, uncomment the line that starts with #action(type="omfile"
#    and monitor /var/log/forti-force-pri.log for all received messages
# 6. Validate filtered logs appear in Sentinel CommonSecurityLog table
#
# VALIDATION TESTED: 2025-05-23
#####################################################################

# Template that ALWAYS adds PRI header - no conditional logic
template(name="FortiCEF_ForcePRI" type="string"
  string="<164>%TIMESTAMP% %HOSTNAME% %rawmsg-after-pri%\n")

# Main processing ruleset - handles both UDP and TCP inputs
ruleset(name="forti-force-pri-cef") {
    # Debug logging
    #action(type="omfile" file="/var/log/forti-force-pri.log")
    
    # CEF filtering
    if not ($rawmsg contains "CEF:") then {
        stop
    }
    
    # SEVERITY FILTERING - Only forward Warning+ severity messages
    if ($rawmsg contains "deviceSeverity=critical" or $rawmsg contains "deviceSeverity=high" or $rawmsg contains "deviceSeverity=medium") then {
        # FORCE PRI header addition - unconditional
        action(type="omfwd"
               template="FortiCEF_ForcePRI"
               target="127.0.0.1"
               port="28330"
               protocol="tcp"
               queue.type="LinkedList"
               queue.filename="omfwd-forti-force"
               queue.maxFileSize="32m"
               queue.maxDiskSpace="1g"
               action.resumeRetryCount="-1"
               action.resumeInterval="5"
               queue.size="25000"
               queue.workerThreads="100"
               queue.saveonshutdown="on")
    }
    stop
}

# UDP INPUT (Default - always enabled)
input(type="imudp" port="1514" ruleset="forti-force-pri-cef")

# TCP INPUT SECTION - UNCOMMENT IF FORTIANALYZER "RELIABLE CONNECTION" IS ENABLED
# -------------------------------------------------------------------------------
# FortiAnalyzer switches from UDP to TCP when "Reliable Connection" is checked.
# TCP provides guaranteed delivery with retransmission but adds overhead.
# You can have both UDP and TCP listeners active simultaneously.
#
#module(load="imtcp")  # Load TCP input module
#input(type="imtcp" port="1514" ruleset="forti-force-pri-cef")