rsyslog conf.d file to handle FortiAnalyzer's malformed CEF format and ingest to Sentinel via AMA

40-fortianalyzer-filtered-cef.conf

#####################################################################
# FortiAnalyzer CEF Integration - Production Configuration
#####################################################################
#
# PURPOSE:
# --------
# This configuration solves the critical protocol compliance issue with 
# FortiAnalyzer CEF log forwarding to Microsoft Sentinel via Azure Monitor Agent.
#
# PROBLEM:
# --------
# FortiAnalyzer sends CEF messages in partial syslog format:
#   SENDS:     "MMM dd HH:mm:ss hostname CEF:0|..."
#   EXPECTED:  "<164>MMM dd HH:mm:ss hostname CEF:0|..."
# 
# Azure Monitor Agent (AMA) strictly requires RFC3164/RFC5424 compliant 
# syslog messages with PRI headers and will reject headerless messages entirely.
#
# SOLUTION:
# ---------
# - Receives CEF messages from FortiAnalyzer on TCP port 1514
# - Adds mandatory syslog PRI header <164> (local4.warning) for AMA compliance
# - Forwards ALL UTM logs to ensure complete security visibility
# - No severity-based filtering to prevent missing critical security events
#
# KEY LEARNINGS:
# --------------
# Initial attempts at severity-based filtering (deviceSeverity=critical|high|medium)
# resulted in missing logs in Sentinel. Customer requirements prioritised log 
# completeness over volume reduction. The simple type="utm" filter ensures all
# Unified Threat Management logs are captured.
#
# FILTERING LOGIC:
# ----------------
# FORWARDED: ALL messages containing cat="utm"
# DROPPED:   Non-CEF messages and non-UTM CEF messages
#
# IMPORTANT - FORTIANALYZER CONNECTION MODE:
# ------------------------------------------
# FortiAnalyzer can send logs via UDP (default) or TCP ("Reliable Connection" mode).
# This configuration is set up for TCP on port 1514, which is recommended for
# production environments to ensure reliable log delivery.
#
# To support UDP mode (if needed), uncomment the UDP INPUT line at the bottom.
# You can safely enable both UDP and TCP listeners simultaneously.
#
# DEPLOYMENT:
# -----------
# 1. Configure FortiAnalyzer to forward CEF logs to this server:1514
#    - Enable "Reliable Connection" for TCP mode (recommended)
# 2. Ensure Azure Monitor Agent is configured with appropriate DCR
# 3. DCR should be configured for LOG_LOCAL4:LOG_WARNING
# 4. For debugging, uncomment the action(type="omfile" line and monitor
#    /var/log/forti-force-pri.log to see all received messages
# 5. Validate logs appear in Sentinel CommonSecurityLog table
#    - Note: Ingestion delay can be 10-15 minutes
#
# VALIDATION COMPLETED: 2025-05-23
# UPDATED: 2025-08-04 - Changed to accept all UTM logs based on production feedback
#####################################################################

# Template that ALWAYS adds PRI header - no conditional logic needed
template(name="FortiCEF_ForcePRI" type="string" 
         string="<164>%TIMESTAMP% %HOSTNAME% %rawmsg-after-pri%\n")

# Main processing ruleset - handles both UDP and TCP inputs
ruleset(name="forti-force-pri-cef") {
    # Debug logging - uncomment to troubleshoot
    #action(type="omfile" file="/var/log/forti-force-pri.log")
    
    # CEF filtering - only process CEF formatted messages
    if not ($rawmsg contains "CEF:") then {
        stop
    }
    
    # Forward ALL UTM logs regardless of severity level
    # This ensures complete security visibility in Sentinel
    if ($rawmsg contains 'cat="utm"') then {
        # FORCE PRI header addition for AMA compliance
        action(type="omfwd"
               template="FortiCEF_ForcePRI"
               target="127.0.0.1"
               port="28330"
               protocol="tcp"
               queue.type="LinkedList"
               queue.filename="omfwd-forti-force"
               queue.maxFileSize="32m"
               queue.maxDiskSpace="1g"
               action.resumeRetryCount="-1"
               action.resumeInterval="5"
               queue.size="25000" 
               queue.workerThreads="100"
               queue.saveonshutdown="on")
    }
    
    stop
}

# TCP INPUT (Recommended for production - reliable delivery)
module(load="imtcp")  # Load TCP input module
input(type="imtcp" port="1514" ruleset="forti-force-pri-cef")

# UDP INPUT (Optional - uncomment if FortiAnalyzer uses default UDP mode)
# Note: UDP is best-effort delivery and may lose logs under heavy load
#input(type="imudp" port="1514" ruleset="forti-force-pri-cef")