Skip to content

Instantly share code, notes, and snippets.

@richm
Created April 9, 2020 19:12
Show Gist options
  • Save richm/8ddaebd6b6a824bd0269c047de1411dd to your computer and use it in GitHub Desktop.
Save richm/8ddaebd6b6a824bd0269c047de1411dd to your computer and use it in GitHub Desktop.
{% if rsyslog_flows | length > 0 %}
{% for flow in rsyslog_flows %}
{% for input_name in flow.inputs %}
{% if rsyslog_input_basics.0.name == input_name %}
{% if __basics_first_time == true %}
if $inputname == "imjournal" then {
{% endif %}
{% for flow_output in flow.outputs %}
call {{ flow_output }}
{% endfor %}
{% if __basics_first_time == true %}
stop
}
{% set __basics_first_time = false %}
{% endif %}
{% endif %}
{% endfor %}
{% endfor %}
{% else %}
if $inputname == "imjournal" then {
call files
stop
}
{% endif %}
@richm
Copy link
Author

richm commented Apr 10, 2020

inputs:

  • name: A
    type: a
  • name: B
    type: b
  • name: C
    type: a
    outputs:
  • name: X
    type: x
  • name: Y
    type: y
  • name: Z
    type: x
    flows:
  • name: M
    inputs: [A, B]
    outputs: [X, Y]
  • name: N
    inputs: [A, C]
    outputs: [X, Z]

This is bad - does "call X" 3 times:

if $input == A then {
call X
call Y
call Z
}
if $input == B then {
call X
call Y
}
if $input == C then {
call X
call Z
}

what we want is a mapping of unique outputs to a unique list of inputs for each output
X: A, B, C
Y: A, B
Z: A, C

Each input has some sort of condition associated with it that uniquely identifies records from that input.

For imfile, we can use the "syslogtag" message property https://www.rsyslog.com/doc/v8-stable/configuration/properties.html#message-properties which is set from the imfile "Tag" property https://www.rsyslog.com/doc/v8-stable/configuration/modules/imfile.html#tag
the rsyslog config condition would be $syslogtag == "tag value from imfile input config" e.g.

    if $syslogtag == "files0" then ...

For imjournal, use the "inputname" message property which has the value "imjournal"

    if $inputname == "imjournal" ...

For imtcp, use the "inputname" message property which gets its value from the "Name" property in the imtcp input configuration: https://www.rsyslog.com/doc/v8-stable/configuration/modules/imtcp.html#name - note that Name is also used for the stats

    if $inputname == "forward0" ...

For imrelp, use the "inputname" message property which gets its value from the "Name" property in the imrelp input configuration: https://www.rsyslog.com/doc/v8-stable/configuration/modules/imrelp.html#name - NOTE: This isn't specified in the documentation, but that's what the source code does. Also note that the "Name" is used to construct the name used in stats.

    if $inputname == "relp0" ...

Other inputs will have other ways to denote which input this is from. One notable exception may be imkafka - I cannot figure out how to distinguish between two different kafka inputs - it sets syslogtag based on some "key" field in the message, but I don't know where that is set. So the condition may be compound e.g. ($inputname == "imkafka" and something else). Similarly with imuxsock - there's $inputname == "imuxsock" but may have to override "Hostname" https://www.rsyslog.com/doc/v8-stable/configuration/modules/imuxsock.html#hostname and use ($inputname == "imuxsock" and $hostname == "sock0").

The point is that the input must have a condition which can be rendered into an rsyslog if statement condition and concatenated with conditions from other inputs:

    if (condition_for_input1) or (condition_for_input2) ... then {
        call outputA
        call outputB
        ....
    }

assuming "outputA" and "outputB" have the exact same inputs.

for name,inputs,outputs in flows:
for each outputname in outputs:
uniqueoutputs[outputname] = uniqueorderedset(inputs + uniqueoutputs.get(outputname, [])

for each outputname,inputnamelist in uniqueoutputs.items():
output = logging_outputs[outputname]

maybe handle errors here if user specified an output in a flow that is not defined

in logging_outputs

{{ "if " }}
for inputname in inputnamelist:
input = logging_inputs[inputname]
if not loop.first:
{{ " or " }}
{{ input.rsyslog_condition }}
# https://jinja.palletsprojects.com/en/2.11.x/templates/#for
{{ " then { " }}
{{ "call " + output.rsyslog_output }}
{{ "}" }}

{% for each input in inputs %}
{% if there is at least one output for this input %}

if $input in outputs["X"] then {
call X
}
if $input in outputs["Y"] then {
call Y
}
if $input in outputs["Z"] then {
call Z
}

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment