-
-
Save mbierman/f3d184b65e0f4de6fa75a4a5d5145426 to your computer and use it in GitHub Desktop.
#!/bin/bash | |
# v 2.1.0 | |
syslog=/etc/rsyslog.d/09-externalserver.conf | |
# this logs notice and above. use *.* log everything. | |
filter=*.notice | |
server=192.168.0.19 # Change the server to the IP of your syslog server. | |
port=514 | |
hostname=firewalla | |
valid=$(grep "$server:$port" $syslog 2>/dev/null) | |
create () { | |
# To use TCP uncomment line 13 to use TCP and comment line 15 | |
# echo -e "# remote syslog server (TCP):\n$filter @@$server:$port" | sudo tee $syslog | |
# Line 15 assumes UDP: to use TCP, comment the line 15 and uncomment line 13 | |
echo -e "# remote syslog server (UDP):\n\$LocalHostName $hostname\nfilter @$server:$port" | sudo tee $syslog | |
echo "Restarting rsyslog..." | |
sudo systemctl restart rsyslog | |
echo "remote syslog added" | |
exit | |
} | |
cleanup () { | |
sudo rm -f $syslog | |
sudo systemctl restart rsyslog | |
} | |
if [ -f "$syslog" ] ; then | |
if [ -n "$valid" ] ; then | |
echo "remote syslog already in place with $server:$port specified" | |
case $1 in | |
-c) | |
echo -e "\nrecreating syslog configuration..." | |
cleanup | |
create | |
;; | |
-r|-restart|-force|-f) | |
echo "Restarting rsyslog..." | |
sudo systemctl restart rsyslog | |
exit | |
;; | |
-u|-update) | |
read -p "Are you sure you want to remove the syslog forwarder? type 'y' " -n 1 -r | |
echo | |
if [[ $REPLY =~ ^[Yy]$ ]] ; then | |
ls $syslog 2>/dev/null && cleanup || echo -e "\n\nNo log found.\n" | |
fi | |
exit | |
;; | |
-h) | |
echo -e "You can use:\n - \`$0 -c\` recreate forwarding\n - \`$0 -r\` restart the syslog service\ | |
\n - \`$0 -u\` uninstall the settings to send to the remote syslog server\n\n" | |
exit | |
;; | |
esac | |
else | |
echo "The server is not configured correctly. On it." | |
cleanup | |
create | |
fi | |
else | |
echo "There was no syslog forwarder in place." | |
create | |
fi |
saiful0190
commented
May 11, 2023
•
- Do you have a working syslog server?
- Did you set the IP of that syslog server in the script?
Thank you very much for this. I do have one minor tweak to offer: As it currently transmits the log entries, they are all originating from the hostname "localhost". This gets confusing on the receiving end - my little tweak let's me define the hostname I want to use (such as "firewalla" and then add an additional line to the 09-externalserver.conf file:
- Add this line up where filter, server and port are defined:
hostname=firewalla
- Modify line 15 in the script (which echos the config material into the conf file for UDP transmission) to the following:
echo -e "# remote syslog server (UDP):\n\$LocalHostName $hostname\nfilter @$server:$port" | sudo tee $syslog
Notice I injected the "$LocalHostName $hostname\n" portion resulting in the file looking like the following:
# remote syslog server (UDP): $LocalHostName firewalla filter @192.168.1.12:514
(Notice the "" before the $ is needed to escape it so an actual $ is echoed out and variable substitution isn't performed.)
Integrated. Great ideas! Thank you.
Hello, is it possible to send /log/blog to the remote syslog server? I would like to see the src and dst IP from the /log/blog/current/conn.log
Hi All!
While I do appreciate great scripting abilities as it makes any process more simple, I decided that the *.* @server:port
is the legacy way of collecting logs, where adopting the new rsyslog parms is more powerful and gives us greater flexibility.
The problem with the current scripting method is that it doesn't capture all the logs that we actually really want from Firewalla. The *.*
captures everything that's written in the /var/log/syslog
directory. Very useful but lacks quite a bit of networking data that we want to see from the /bspool/manager
and the /alog/firewalla/
dir (ie conn.log, conn_long, etc). The reason we don't see the connection logs is because they aren't written to the /var/log/syslog
dir.
By abandoning the legacy rsyslog parameters we can adopt the new system which is more powerful and flexible.
################
Prerequisite
################
Did you setup the syslog inputs on the synology in log center?
If not go to Log Center
> Log Receiving
> Create
> Give your connection a name
, then specify whether you want to use TCP or UDP
on port 514
. BSD
format is fine as well.
######################
New Firewalla Syslog Config
######################
# deifne global workDirectory for saving the state file of log messages.
global(workDirectory="/var/spool/rsyslog")
# enable the Rsyslog imfile module processing text files or logs.
module(load="imfile" PollingInterval="10")
# define template for StandardSyslogFormat for processing log messages.
# that will be forwarded to rsyslog server
template(
name="StandardSyslogFormat"
type="string"
string="<%PRI%>%TIMESTAMP:::date-rfc3339% %HOSTNAME% %syslogtag:1:32%%msg:::sp-if-no-1st-sp%%msg%"
)
# define ruleset "forwardSysLogs" with action object to send logs to rsyslog server
# define the queue
ruleset(name="forwardSysLogs") {
action(
type="omfwd"
target="172.16.2.20" # set your Synology Syslog Server NAS IP
port="514" # Specify port number
protocol="tcp" # specify protocol UDP or TCP
template="StandardSyslogFormat" # specifies the template to use above
queue.SpoolDirectory="/var/spool/rsyslog"
queue.FileName="remote"
queue.MaxDiskSpace="1g"
queue.SaveOnShutdown="on"
queue.Type="LinkedList"
ResendLastMSGOnReconnect="on"
)
stop
}
# define input files forwardSysLogs logs to send to the rsyslog server
# and apply ruleset "forwardSysLogs"
# in /bspool/manager
input(type="imfile" ruleset="forwardSysLogs" Tag="ConnLog" File="/bspool/manager/conn.log")
input(type="imfile" ruleset="forwardSysLogs" Tag="ConnLongLog" File="/bspool/manager/conn_long.log")
input(type="imfile" ruleset="forwardSysLogs" Tag="DNS" File="/bspool/manager/dns.log")
input(type="imfile" ruleset="forwardSysLogs" Tag="Files" File="/bspool/manager/files.log")
input(type="imfile" ruleset="forwardSysLogs" Tag="HeartBeat" File="/bspool/manager/heartbeat.log")
input(type="imfile" ruleset="forwardSysLogs" Tag="NTP" File="/bspool/manager/ntp.log")
input(type="imfile" ruleset="forwardSysLogs" Tag="OSCP" File="/bspool/manager/oscp.log")
input(type="imfile" ruleset="forwardSysLogs" Tag="SSL" File="/bspool/manager/ssl.log")
input(type="imfile" ruleset="forwardSysLogs" Tag="StdErr" File="/bspool/manager/stderr.log")
input(type="imfile" ruleset="forwardSysLogs" Tag="StdOut" File="/bspool/manager/stdout.log")
input(type="imfile" ruleset="forwardSysLogs" Tag="HTTP" File="/bspool/manager/http.log")
input(type="imfile" ruleset="forwardSysLogs" Tag="Notice" File="/bspool/manager/notice.log")
input(type="imfile" ruleset="forwardSysLogs" Tag="Weird" File="/bspool/manager/weird.log")
# define input files forwardSysLogs logs to send to the rsyslog server
# and apply ruleset "forwardSysLogs"
# in /alog
input(type="imfile" ruleset="forwardSysLogs" Tag="ACL-Alarm" File="/alog/acl-alarm.log")
input(type="imfile" ruleset="forwardSysLogs" Tag="ACL-Audit" File="/alog/acl-audit.log")
input(type="imfile" ruleset="forwardSysLogs" Tag="DNS-Masq" File="/alog/dnsmasq-acl.log")
# define input files forwardSysLogs logs to send to the rsyslog server
# and apply ruleset "forwardSysLogs"
# in /alog/firewalla
input(type="imfile" ruleset="forwardSysLogs" Tag="FireApi" File="/alog/firewalla/FireApi.log")
input(type="imfile" ruleset="forwardSysLogs" Tag="FireKick" File="/alog/firewalla/FireKick.log")
input(type="imfile" ruleset="forwardSysLogs" Tag="FireMain" File="/alog/firewalla/FireMain.log")
input(type="imfile" ruleset="forwardSysLogs" Tag="FireMon" File="/alog/firewalla/FireMon.log")
input(type="imfile" ruleset="forwardSysLogs" Tag="FireRouter" File="/alog/firewalla/FireRouter.log")
input(type="imfile" ruleset="forwardSysLogs" Tag="Trace" File="/alog/firewalla/Trace.log")
input(type="imfile" ruleset="forwardSysLogs" Tag="CleanLog" File="/alog/firewalla/clean_log.log")
input(type="imfile" ruleset="forwardSysLogs" Tag="Firelog" File="/alog/firewalla/firelog.log")
input(type="imfile" ruleset="forwardSysLogs" Tag="Node" File="/alog/firewalla/node.log")
input(type="imfile" ruleset="forwardSysLogs" Tag="SyncTime" File="/alog/firewalla/sync_time.log")
# Sending all other Syslog logs to Server (Synology)
# @@IP is for TCP
# @IP is for UDP
*.* @@172.16.2.20:514
#########################
Modifying the config
########################
Be sure to change the following attributes in the new config file before pasting via VI
into the syslog conf file:
target="172.16.2.20" # set your Synology Syslog Server NAS IP
port="514" # Specify port number
protocol="tcp" # specify protocol UDP or TCP
AND
# Sending all other Syslog logs to Server (Synology)
# @@IP is for TCP
# @IP is for UDP
*.* @@172.16.2.20:514
#########################################
Setting up Syslog on Firewalla to send to Synology
#########################################
To collect the all of the important Firewalla modify the existing syslog conf file
or create a new one by doing the following:
-
Go to the following directory by running the following command:
cd /etc/rsyslog.d
-
Run
ls -lar
to find the 09-externalserver.conf file. -
If the conf file doesn't exist then you'll need to create the the Syslog conf file (skip this step if
09-externalserver.conf
does exist):
sudo touch /etc/rsyslog.d/09-externalserver.conf
sudo vi /etc/rsyslog.d/09-externalserver.conf
-
If the 09-externalserver.conf file exists because you already ran the script on this Github then you'll need to delete the script that created it (this step is extremely important or your new config will be overwritten by the script).
sudo rm -rf /home/pi/.firewalla/config/post_main.d/addremotesyslog.sh
-
Since deleting the script, we can keep the file name in place if it already exists but we want to
erase everything from that file
by running the following command (if you created the conf file using thetouch
command then skip this step since it will already be empty):
sudo sed -i d 09-externalserver.conf
-
Next we need to open the file in order to paste our new configs by running the following command:
sudo vi 09-externalserver.conf
-
Press the letter
i
on your keyboard for insert -
Copy the configs and paste them into the file by
right clicking
(this is how you paste using VIM) -
Once the configs are copied then
press escape
then type:wq!
on your kyboard and hitenter
-
Now run the following command to restart the syslog engine:
sudo systemctl restart rsyslog
That's it! You should now be grabbing all of the important Firewalla logs!
######################
Synology Log Center Results
######################
Here is a screenshot of my Synology Log Center
:
Hello, is it possible to send /log/blog to the remote syslog server? I would like to see the src and dst IP from the /log/blog/current/conn.log
Just made a solution for exactly what it is you're looking to do:
Also, since the packet headers are recived by the syslog server as localhost
(which really bothers me). Syslog will always send the events based on the entry of the hosts
file. So add a line above the default local
addresses with your devices assigned IP address
tabbed with a hostname of your liking (ie firewalla).
cd /etc
andsudo vi hosts
to create the following entry:- Press
i
for insert - Create the following entry with your own locally assigned static
IP address
:
172.16.2.1 firewalla firewalla.lan
127.0.0.1 localhost Firewalla
- Press
esc
after inserting your info then type:wq!
and hitenter
- Now run the following command to restart the syslog engine:
sudo systemctl restart rsyslog
Now you will see logs come in as firewalla
instead of localhost
!
Results:
Hello @mjaestewart
I appreciate this very much, but for some reason, it is not working for me. When doing a tcpdump I see no traffic using port 514 or if I do it by host I do not see it making a connection to my NAS.
Hello @mjaestewart I appreciate this very much, but for some reason, it is not working for me. When doing a tcpdump I see no traffic using port 514 or if I do it by host I do not see it making a connection to my NAS.
##########
Update
##########
- Original Post is now updated with this solution
Did you setup the syslog inputs on the synology in log center?
If not go to Log Center
> Log Receiving
> Create
> Give your connection a name
, then specify whether you want to use TCP or UDP
on port 514
. BSD
format is fine as well.
On thing that I found on my end as well is that the script was never removed /home/pi/.firewalla/config/post_main.d/addremotesyslog.sh
and will overwrite the sylog file that was created.
So we have to remove that script by doing the following:
-
sudo rm -rf /home/pi/.firewalla/config/post_main.d/addremotesyslog.sh
-
Then change the
IP
ANDProtocol
information in the config file I posted to reflect your environment Conf Here
target="172.16.2.20" # set your Synology Syslog Server NAS IP
port="514" # Specify port number
protocol="tcp" # specify protocol UDP or TCP
-
Recreate the the Syslog conf file:
sudo touch /etc/rsyslog.d/09-externalserver.conf
sudo vi /etc/rsyslog.d/09-externalserver.conf
-
Press the letter
i
on your keyboard for insert -
Copy the configs Conf Here and paste them into the file by right clicking (this is how you paste using VIM)
-
Once the configs are copied then
press escape
then type:wq!
on your kyboard and hitenter
-
Now run the following command to restart the syslog engine:
sudo systemctl restart rsyslog
Can this be used to get Firewalla gold networking blocks and IDS in to Wazuh?
@bn1980 I assume so. It looks like Wazuh supports syslog input.
@mjaestewart This has worked great, but inevitably the zeek events stop streaming because I think zeek rotates those logs and then rsyslog doesn't pick up the change and is reading from the wrong inode. Have you seen this same behavior? How were you able to handle it? I don't see a logrotate conf for the zeek logs, so I assume it's the builtin zeek functionality for rotation.
@mjaestewart This has worked great, but inevitably the zeek events stop streaming because I think zeek rotates those logs and then rsyslog doesn't pick up the change and is reading from the wrong inode. Have you seen this same behavior? How were you able to handle it? I don't see a logrotate conf for the zeek logs, so I assume it's the builtin zeek functionality for rotation.
I’ll put together a solution tomorrow and post it :-) Yes, I also see the same behavior.
@mjaestewart if you find a solution I'd love to test and incorporate it.
Here is my updated solution. I've tested all day, and so far so good. @mbierman I reused what you had already done, and built on that 👍
Script
#!/bin/bash
# v 2.1.0
script_location="/home/pi/.firewalla/config/post_main.d/" # script location
script="firewalla_rsyslog.sh" # script used to install firewalla syslog
cron_cmd="0 * * * * cd $script_location && sudo ./$script -c"
syslog="/etc/rsyslog.d/09-externalserver.conf" # rsyslog location
server="172.16.2.20" # Change the server to the IP of your syslog server.
port="514" # port used for forwarding logs to destination
protocol="tcp" #use tcp or udp
other_protocol="@@" # use @@ for TCP and @ for UDP
valid=$(grep "$server:$port" $syslog 2>/dev/null)
### Creating the syslog file
create() {
sudo touch $syslog
sudo cat > $syslog <<EOF
\$LocalHostName Firewalla
# deifne global workDirectory for saving the state file of log messages.
global(workDirectory="/var/spool/rsyslog")
# enable the Rsyslog imfile module processing text files or logs.
module(load="imfile" PollingInterval="10")
# define template for StandardSyslogFormat for processing log messages.
# that will be forwarded to rsyslog server
template(
name="StandardSyslogFormat"
type="string"
string="<%PRI%>%TIMESTAMP:::date-rfc3339% %HOSTNAME% %syslogtag:1:32%%msg:::sp-if-no-1st-sp%%msg%"
)
# define ruleset "forwardSysLogs" with action object to send logs to rsyslog server
# define the queue
ruleset(name="forwardSysLogs") {
action(
type="omfwd"
target="$server" # set your Synology Syslog Server NAS IP
port="$port" # Specify port number
protocol="$protocol" # specify protocol UDP or TCP
template="StandardSyslogFormat" # specifies the template to use above
queue.SpoolDirectory="/var/spool/rsyslog"
queue.FileName="remote"
queue.MaxDiskSpace="1g"
queue.SaveOnShutdown="on"
queue.Type="LinkedList"
ResendLastMSGOnReconnect="on"
)
stop
}
# define input files forwardSysLogs logs to send to the rsyslog server
# and apply ruleset "forwardSysLogs"
# in /bspool/manager
input(type="imfile" ruleset="forwardSysLogs" Tag="ConnLog" File="/bspool/manager/conn.log")
input(type="imfile" ruleset="forwardSysLogs" Tag="ConnLongLog" File="/bspool/manager/conn_long.log")
input(type="imfile" ruleset="forwardSysLogs" Tag="DNS" File="/bspool/manager/dns.log")
input(type="imfile" ruleset="forwardSysLogs" Tag="Files" File="/bspool/manager/files.log")
input(type="imfile" ruleset="forwardSysLogs" Tag="HeartBeat" File="/bspool/manager/heartbeat.log")
input(type="imfile" ruleset="forwardSysLogs" Tag="NTP" File="/bspool/manager/ntp.log")
input(type="imfile" ruleset="forwardSysLogs" Tag="OSCP" File="/bspool/manager/oscp.log")
input(type="imfile" ruleset="forwardSysLogs" Tag="SSL" File="/bspool/manager/ssl.log")
input(type="imfile" ruleset="forwardSysLogs" Tag="StdErr" File="/bspool/manager/stderr.log")
input(type="imfile" ruleset="forwardSysLogs" Tag="StdOut" File="/bspool/manager/stdout.log")
input(type="imfile" ruleset="forwardSysLogs" Tag="HTTP" File="/bspool/manager/http.log")
input(type="imfile" ruleset="forwardSysLogs" Tag="Notice" File="/bspool/manager/notice.log")
input(type="imfile" ruleset="forwardSysLogs" Tag="Weird" File="/bspool/manager/weird.log")
# define input files forwardSysLogs logs to send to the rsyslog server
# and apply ruleset "forwardSysLogs"
# in /alog
input(type="imfile" ruleset="forwardSysLogs" Tag="ACL-Alarm" File="/alog/acl-alarm.log")
input(type="imfile" ruleset="forwardSysLogs" Tag="ACL-Audit" File="/alog/acl-audit.log")
input(type="imfile" ruleset="forwardSysLogs" Tag="DNS-Masq" File="/alog/dnsmasq-acl.log")
# define input files forwardSysLogs logs to send to the rsyslog server
# and apply ruleset "forwardSysLogs"
# in /alog/firewalla
input(type="imfile" ruleset="forwardSysLogs" Tag="FireApi" File="/alog/firewalla/FireApi.log")
input(type="imfile" ruleset="forwardSysLogs" Tag="FireKick" File="/alog/firewalla/FireKick.log")
input(type="imfile" ruleset="forwardSysLogs" Tag="FireMain" File="/alog/firewalla/FireMain.log")
input(type="imfile" ruleset="forwardSysLogs" Tag="FireMon" File="/alog/firewalla/FireMon.log")
input(type="imfile" ruleset="forwardSysLogs" Tag="FireRouter" File="/alog/firewalla/FireRouter.log")
input(type="imfile" ruleset="forwardSysLogs" Tag="Trace" File="/alog/firewalla/Trace.log")
input(type="imfile" ruleset="forwardSysLogs" Tag="CleanLog" File="/alog/firewalla/clean_log.log")
input(type="imfile" ruleset="forwardSysLogs" Tag="Firelog" File="/alog/firewalla/firelog.log")
input(type="imfile" ruleset="forwardSysLogs" Tag="Node" File="/alog/firewalla/node.log")
input(type="imfile" ruleset="forwardSysLogs" Tag="SyncTime" File="/alog/firewalla/sync_time.log")
# Sending all other Syslog logs to Server (Synology)
# @@IP is for TCP
# @IP is for UDP
*.* $other_protocol$server:$port
EOF
echo "Restarting rsyslog..."
sudo systemctl restart rsyslog
echo "remote syslog added"
echo "adding cron job for reliability"
(crontab -u pi -l 2>/dev/null; echo "$cron_cmd") | crontab -u pi -
sudo systemctl restart cron
exit
}
cleanup() {
sudo rm -f $syslog
sudo systemctl restart rsyslog
(crontab -u pi -l | grep -vF "$cron_cmd" | crontab -u pi -)
}
if [ -f "$syslog" ] ; then
if [ -n "$valid" ] ; then
echo "remote syslog already in place with $server:$port specified"
case $1 in
-c)
echo -e "\nrecreating syslog configuration..."
cleanup
create
;;
-r|-restart|-force|-f)
echo "Restarting rsyslog..."
sudo systemctl restart rsyslog
exit
;;
-u|-update)
read -p "Are you sure you want to remove the syslog forwarder? type 'y' " -n 1 -r
echo
if [[ $REPLY =~ ^[Yy]$ ]] ; then
ls $syslog 2>/dev/null && cleanup || echo -e "\n\nNo log found.\n"
fi
exit
;;
-h)
echo -e "You can use:\n - \`$0 -c\` recreate forwarding\n - \`$0 -r\` restart the syslog service\
\n - \`$0 -u\` uninstall the settings to send to the remote syslog server\n\n"
exit
;;
esac
else
echo "The server is not configured correctly. On it."
cleanup
create
fi
else
echo "There was no syslog forwarder in place."
create
fi
Fixes
- Hostname is now set to
Firewalla
- Cron is now used to ensure persistent sending of all FW logs
- Implementation is now completely automated via script
Setting up the Directory
To send logs to a remote syslog server using UDP, do the following:
- ssh to the Firewalla box.
- Copy the script above.
- If
/home/pi/.firewalla/config/post_main.d/
doesn’t exist, create it first.
sudo mkdir /home/pi/.firewalla/config/post_main.d/
- Next, create the file:
sudo vi /home/pi/.firewalla/config/post_main.d/firewalla_rsyslog.sh
Modifying the Variables in the Script and Executing
- Edit the following variables in the script for your specific environment:
server
to the IP address of your syslog server.port
to the correct port being used for rsyslogprotocol
to specify TCP or UDPother_protocol
uses a single @ for UDP and a double @@ for TCP
- Paste this script into
firewalla_rsyslog.sh
. This is going to creatersyslog configs
and thecron job
that runs to ensure that the syslog setting remains in place, even if there's a firewalla update that wipes out the settings in the future. - Save the file
:wq!
- Give the script execute permissions.
sudo chmod +x /home/pi/.firewalla/config/post_main.d/firewalla_rsyslog.sh
- Execute the script.
sudo /home/pi/.firewalla/config/post_main.d/firewalla_rsyslog.sh -c
creates the file and restarts syslog
Additional Arguments
/home/pi/.firewalla/config/post_main.d/firewalla_rsyslog.sh -r
restarts syslog/home/pi/.firewalla/config/post_main.d/firewalla_rsyslog.sh -u
uninstalls the forwarder and restarts syslog.
@mjaestewart Do you end up with an endless supply of imfile state files in /var/spool/rsyslog? I ended up adding a cronjob to delete files older than 5 minutes in that directory because otherwise it just fills up indefinitely. I'm assuming it has something to do with zeek truncating/rotating the log files because I also end up with these messages from rsyslogd in /var/log/syslog: imfile: internal error? inotify provided watch descriptor 3745 which we could not find in our tables.
You may also want to consider this post about persisting cron through reboots/restarts: https://help.firewalla.com/hc/en-us/articles/360054056754-Customized-Scripting
I notice the cronjob disappeared after a reload so I added it to the location described by that article.