Some things to keep in mind with the queue are:
- This queue only takes place where you use 'call ForwardToQRadar' as your action for a filter, such as:
if ($msg contains 'Teardown' )
then {
call ForwardToQRadar
}
- If you have another queue set up, that will have a completely separate set of dequeueing parameters. Keep that in mind as you're choosing your values so you don't overload your sending system.
- The queue also serves as a way to rate limit the data being forwarded, and the rate limiting will take place even if data isn't building up in the queue. I think the way the documentation describes this is that ALL incoming logs end up passing through the queue, and therefore will only be forwarded as fast as your dequeueing settings allow. Keep that in mind as you are choosing the settings. If the data's bursty the queue will smooth out the delivery (and also start delivering out of order, see the next point), but if you start sustaining log rates higher than your dequeue rate, your queue will keep growing in size and never empty.
- Messages could/will be sent out of order, even during normal operation, but also guaranteed while the buffer is emptying. Incoming messages to the queue will be sent out before the older messages are pulled off the top of the buffer. For us this isn't a problem, because I'd rather have the data available and out of order, than not have the data at all. Keep this in mind if you're using the data in SIEM though, as alerts might get generated when they normally wouldn't have, or vice versa.
Hopefully that's clear enough. I'm sure one of the people who are much more familiar and better versed in rsyslog will chime in if I've inadvertently lied about anything. This has been working for me for about 3 months now and has helped me through a few QRadar upgrades with (as far as I can tell) no message loss.
template(name="QradarForwardMsgFormat" type="string"
string="<%pri%>%timestamp% %fromhost-ip% %syslogtag%%msg%")
Then, I have a ruleset that uses the template to omfwd to QRadar. This is
where the queue is defined:
ruleset(name="ForwardToQRadar") {
action(
name="forwardToQRadar" # useful when you're looking at
statistics/troubleshooting impstats output
type="omfwd"
Template="QradarForwardMsgFormat"
Target="<destination hostname or IP address"
Port="514"
Protocol="tcp"
queue.type = "LinkedList"
queue.spoolDirectory = "/var/log/queue" # start writing in
messages to disk under this directory
queue.filename = "forwardToQRadar_q" # this base
filename that will be created under spoolDirectory
queue.size = "300000000" # Sized for my environment.
This will let me buffer about 6 hours' worth of data
queue.highwatermark = "500000" # num messages in memory
queue before it starts writing to disk queue
queue.lowwatermark = "400000" # until memory queue is back
to 400k, at which point the memory queue is used again
queue.maxdiskspace="200g" # the queue will be full when
it occupies 200GB of space on disk. Should not be reached. queue.size=300mil *
512 bytes/msg = ~153GB
queue.saveonshutdown="on" # save queue contents to disk
at shutdown
# simple rate limiting - real max EPS to QRadar for this queue
is 1,000,000 / dequeueslowdown * dequeuebatchsize
queue.dequeueslowdown = "275" # wait 275 microseconds
between dequeuing each batch of messages. if batch size = 2 and this is 275,
effective rate is 7272 msg/sec (or slightly less)
queue.dequeuebatchsize = "2" # clear queue in max of 2 msg
batches. 1,000,000 microseconds per second / 275 = 3636 * 2 batch size = 7272
EPS
action.resumeRetryCount="-1" # retry forever
action.reportSuspension="on" # report errors to log
action.reportSuspensionContinuation="on"
)
}