Skip to content

Instantly share code, notes, and snippets.

@timconradinc

timconradinc/beaver.conf

Last active December 16, 2015 23:29
Show Gist options
  • Save timconradinc/5513885 to your computer and use it in GitHub Desktop.
Save timconradinc/5513885 to your computer and use it in GitHub Desktop.
Download ZIP
Beaver Issue #135
Raw
beaver.conf
[beaver]
redis_url:redis://10.99.9.99:6379/0
[/syslog/logs/palo*/user-info*log]
type: PaloAltoTrafficLog
[/syslog/logs/palo*/local*log]
type: PaloAltoThreatLog
# this is for testing/troubleshooting
[/syslog/logs/palo*/traffic-log-door.log]
type: PaloAltoTrafficLog
# this is for testing/troubleshooting
[/syslog/logs/palo*/threat-log-door.log]
type: PaloAltoThreatLog
Raw
gistfile1.txt
Data missing 40 - should have 42 or 46 fields in the csv area - message post redis:
{"@source"=>
"file://logstash01/syslog/logs/palo-altoexample.org/local4-crit-2013-05-03.log",
"@source_host"=>"logstash01",
"@message"=>
"May 3 16:06:54 palo-altoexample.org 1,2013/05/03 16:06:54,0009C108314,THREAT,spyware,1,2013/05/03 16:06:49,184.254.253.254,10.12.17.0,0.0.0.0,0.0.0.0,rule3,,,unknown-udp,vsys1,MSTap,MSTap,eth
ernet1/5,ethernet1/5,syslog.example.com,2013/05/03 16:06:54,1815151,2,16465,65012,0,0,0x0,udp,alert,\"\",ZeroAccess.Gen Command and Control Traffic(13235),any,critical,server-to-client,2004995424,0x0,United Stat
es,1",
"@tags"=>[],
"@fields"=>{},
"@timestamp"=>"2013-05-03T20:07:03.361599Z",
"@source_path"=>
"/syslog/logs/palo-altoexample.org/local4-crit-2013-05-03.log",
"@type"=>"PaloAltoThreatLog"}
(this is put after the first file.readlines(4096) in readfile() in worker.py - note both the first aand last lines are truncated. the first is truncated at beginning and end. the previous WORKER_READFILE shows a complete last line.
WORKER_READFILE: [
u'.175.0,0.0.0.0,0.0.0.0,rule3,,,unknown-udp,vsys1,MSTap,MSTap,ethernet1/5,ethernet1/5,syslog.example.com,2013/05/03 16:06:51,1613660,1,16465,65012,0,0,0x0,udp,alert,"",ZeroAccess.Gen Command and Control Traffic(13235),any,critical,server-to-client,2004995422,0x0,Viet Nam,10.0.0.0-10.255.255.255,0,\n',
u'May 3 16:06:54 palo-altoexample.org 1,2013/05/03 16:06:54,0009C108314,THREAT,spyware,1,2013/05/03 16:06:48,190.254.253.254,10.12.17.0,0.0.0.0,0.0.0.0,rule3,,,unknown-udp,vsys1,MSTap,MSTap,ethernet1/5,ethernet1/5,syslog.example.com,2013/05/03 16:06:54,342284,1,16465,65012,0,0,0x0,udp,alert,"",ZeroAccess.Gen Command and Control Traffic(13235),any,critical,server-to-client,2004995423,0x0,Colombia,10.0.0.0-10.255.255.255,0,\n',
u'May 3 16:06:54 palo-altoexample.org 1,2013/05/03 16:06:54,0009C108314,THREAT,spyware,1,2013/05/03 16:06:48,190.254.253.254,10.12.17.0,0.0.0.0,0.0.0.0,rule3,,,unknown-udp,vsys1,MSTap,MSTap,ethernet1/5,ethernet1/5,syslog.example.com,2013/05/03 16:06:54,342284,1,16465,65012,0,0,0x0,udp,alert,"",ZeroAccess.Gen Command and Control Traffic(13235),any,critical,server-to-client,2004995423,0x0,Colombia,10.0.0.0-10.255.255.255,0,\n',
u'May 3 16:06:54 palo-altoexample.org 1,2013/05/03 16:06:54,0009C108314,THREAT,spyware,1,2013/05/03 16:06:49,184.254.253.254,10.12.17.0,0.0.0.0,0.0.0.0,rule3,,,unknown-udp,vsys1,MSTap,MSTap,ethernet1/5,ethernet1/5,syslog.example.com,2013/05/03 16:06:54,1815151,2,16465,65012,0,0,0x0,udp,alert,"",ZeroAccess.Gen Command and Control Traffic(13235),any,critical,server-to-client,2004995424,0x0,United States,10.0.0.0-10.255.255.255,0,\n',
u'May 3 16:06:54 palo-altoexample.org 1,2013/05/03 16:06:54,0009C108314,THREAT,spyware,1,2013/05/03 16:06:49,184.254.253.254,10.12.17.0,0.0.0.0,0.0.0.0,rule3,,,unknown-udp,vsys1,MSTap,MSTap,ethernet1/5,ethernet1/5,syslog.example.com,2013/05/03 16:06:54,1815151,2,16465,65012,0,0,0x0,udp,alert,"",ZeroAccess.Gen Command and Control Traffic(13235),any,critical,server-to-client,2004995424,0x0,United States,1']
raw syslogs:
May 3 16:06:54 palo-altoexample.org 1,2013/05/03 16:06:54,0009C108314,THREAT,spyware,1,2013/05/03 16:06:49,184.254.253.254,10.12.17.0,0.0.0.0,0.0.0.0,rule3,,,unknown-udp,vsys1,MSTap,MSTap,ethernet1/5,ethernet1/5,syslog.example.com,2013/05/03 16:06:54,1815151,2,16465,65012,0,0,0x0,udp,alert,"",ZeroAccess.Gen Command and Control Traffic(13235),any,critical,server-to-client,2004995424,0x0,United States,10.0.0.0-10.255.255.255,0,
May 3 16:06:54 palo-altoexample.org 1,2013/05/03 16:06:54,0009C108314,THREAT,spyware,1,2013/05/03 16:06:49,184.254.253.254,10.12.17.0,0.0.0.0,0.0.0.0,rule3,,,unknown-udp,vsys1,MSTap,MSTap,ethernet1/5,ethernet1/5,syslog.example.com,2013/05/03 16:06:54,1815151,2,16465,65012,0,0,0x0,udp,alert,"",ZeroAccess.Gen Command and Control Traffic(13235),any,critical,server-to-client,2004995424,0x0,United States,10.0.0.0-10.255.255.255,0,
Raw
gistfile2.txt
beaver --debug -c beaver-shipper.conf -t redis --experimental >beaver.out 2>&1
[2013-05-03 16:35:01,304] DEBUG Logger level is DEBUG
[2013-05-03 16:35:01,304] DEBUG Processing beaver portion of config file beaver-shipper.conf
[2013-05-03 16:35:01,307] DEBUG Skipping glob due to no files found: /syslog/logs/palo*/threat-log-door.log
[2013-05-03 16:35:01,309] DEBUG [CONFIG] "rabbitmq_vhost" => "/"
[2013-05-03 16:35:01,309] DEBUG [CONFIG] "sqs_aws_secret_key" => "None"
[2013-05-03 16:35:01,309] DEBUG [CONFIG] "respawn_delay" => "3"
[2013-05-03 16:35:01,309] DEBUG [CONFIG] "ssh_tunnel" => "None"
[2013-05-03 16:35:01,309] DEBUG [CONFIG] "mqtt_clientid" => "mosquitto"
[2013-05-03 16:35:01,309] DEBUG [CONFIG] "pid" => "None"
[2013-05-03 16:35:01,309] DEBUG [CONFIG] "redis_url" => "redis://10.99.9.99:6379/0"
[2013-05-03 16:35:01,309] DEBUG [CONFIG] "mqtt_host" => "localhost"
[2013-05-03 16:35:01,309] DEBUG [CONFIG] "rabbitmq_exchange_durable" => "False"
[2013-05-03 16:35:01,310] DEBUG [CONFIG] "max_failure" => "7"
[2013-05-03 16:35:01,310] DEBUG [CONFIG] "rabbitmq_exchange" => "logstash-exchange"
[2013-05-03 16:35:01,310] DEBUG [CONFIG] "globs" => "{}"
[2013-05-03 16:35:01,310] DEBUG [CONFIG] "transport" => "redis"
[2013-05-03 16:35:01,310] DEBUG [CONFIG] "redis_password" => "None"
[2013-05-03 16:35:01,310] DEBUG [CONFIG] "mqtt_topic" => "/logstash"
[2013-05-03 16:35:01,310] DEBUG [CONFIG] "rabbitmq_host" => "localhost"
[2013-05-03 16:35:01,310] DEBUG [CONFIG] "mqtt_port" => "1883"
[2013-05-03 16:35:01,310] DEBUG [CONFIG] "hostname" => "logstash01"
[2013-05-03 16:35:01,310] DEBUG [CONFIG] "ssh_tunnel_port" => "None"
[2013-05-03 16:35:01,310] DEBUG [CONFIG] "daemonize" => "False"
[2013-05-03 16:35:01,310] DEBUG [CONFIG] "max_queue_size" => "100"
[2013-05-03 16:35:01,310] DEBUG [CONFIG] "zeromq_address" => "tcp://localhost:2120"
[2013-05-03 16:35:01,310] DEBUG [CONFIG] "config" => "beaver-shipper.conf"
[2013-05-03 16:35:01,310] DEBUG [CONFIG] "files" => "None"
[2013-05-03 16:35:01,311] DEBUG [CONFIG] "zeromq_pattern" => "push"
[2013-05-03 16:35:01,311] DEBUG [CONFIG] "discover_interval" => "15.0"
[2013-05-03 16:35:01,311] DEBUG [CONFIG] "zeromq_hwm" => "None"
[2013-05-03 16:35:01,311] DEBUG [CONFIG] "rabbitmq_ha_queue" => "False"
[2013-05-03 16:35:01,311] DEBUG [CONFIG] "subprocess_poll_sleep" => "1"
[2013-05-03 16:35:01,311] DEBUG [CONFIG] "format" => "json"
[2013-05-03 16:35:01,311] DEBUG [CONFIG] "update_file_mapping_time" => "None"
[2013-05-03 16:35:01,311] DEBUG [CONFIG] "ssh_remote_port" => "None"
[2013-05-03 16:35:01,311] DEBUG [CONFIG] "udp_port" => "9999"
[2013-05-03 16:35:01,311] DEBUG [CONFIG] "redis_namespace" => "logstash:beaver"
[2013-05-03 16:35:01,311] DEBUG [CONFIG] "wait_timeout" => "5"
[2013-05-03 16:35:01,311] DEBUG [CONFIG] "rabbitmq_queue" => "logstash-queue"
[2013-05-03 16:35:01,311] DEBUG [CONFIG] "rabbitmq_key" => "logstash-key"
[2013-05-03 16:35:01,311] DEBUG [CONFIG] "path" => "/var/log"
[2013-05-03 16:35:01,312] DEBUG [CONFIG] "udp_host" => "127.0.0.1"
[2013-05-03 16:35:01,312] DEBUG [CONFIG] "sincedb_path" => "None"
[2013-05-03 16:35:01,312] DEBUG [CONFIG] "rabbitmq_exchange_type" => "direct"
[2013-05-03 16:35:01,312] DEBUG [CONFIG] "rabbitmq_username" => "guest"
[2013-05-03 16:35:01,312] DEBUG [CONFIG] "sqs_aws_region" => "us-east-1"
[2013-05-03 16:35:01,312] DEBUG [CONFIG] "ssh_remote_host" => "None"
[2013-05-03 16:35:01,312] DEBUG [CONFIG] "rabbitmq_queue_durable" => "False"
[2013-05-03 16:35:01,312] DEBUG [CONFIG] "queue_timeout" => "60"
[2013-05-03 16:35:01,312] DEBUG [CONFIG] "mqtt_keepalive" => "60"
[2013-05-03 16:35:01,312] DEBUG [CONFIG] "fqdn" => "False"
[2013-05-03 16:35:01,312] DEBUG [CONFIG] "zeromq_bind" => "connect"
[2013-05-03 16:35:01,312] DEBUG [CONFIG] "ssh_key_file" => "None"
[2013-05-03 16:35:01,312] DEBUG [CONFIG] "sqs_aws_queue" => "None"
[2013-05-03 16:35:01,312] DEBUG [CONFIG] "debug" => "True"
[2013-05-03 16:35:01,312] DEBUG [CONFIG] "output" => "None"
[2013-05-03 16:35:01,313] DEBUG [CONFIG] "rabbitmq_password" => "guest"
[2013-05-03 16:35:01,313] DEBUG [CONFIG] "sqs_aws_access_key" => "None"
[2013-05-03 16:35:01,313] DEBUG [CONFIG] "rabbitmq_port" => "5672"
[2013-05-03 16:35:01,313] DEBUG [CONFIG] "confd_path" => "/etc/beaver/conf.d"
[2013-05-03 16:35:01,318] INFO Starting worker...
[2013-05-03 16:35:01,318] WARNING [/var/log/system.log] - [Errno 2] No such file or directory: '/var/log/system.log'
[2013-05-03 16:35:01,318] INFO Working...
[2013-05-03 16:42:00,531] INFO SIGINT detected
[2013-05-03 16:42:00,531] INFO Shutting down. Please wait...
[2013-05-03 16:42:00,532] INFO Closing worker...
[2013-05-03 16:42:00,532] DEBUG [TailManager] - Closing all tail objects
[2013-05-03 16:42:00,533] INFO Shutdown complete.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment