narutaro

#Is NetFlow streaming data analysis possible with fluentd?

I wanted to do some analysis against NetFlow data that I receive everyday. The analyis that I wanna do is various. It is simple pattern matching for a specific IP address, detecting specifc traffic pattern, figuring out network graph and calicurate proximity of certain nodes and so on so forth. I might use norikura plug-in for that purpose later but not sure now.

I know storm, kafuka, spark streaming and even memSQL and VoltDB are good for this purpose but wanted to go with quick way for now.

#Captureing NetFlow with fluentd

I use this NetFlow plug-in on fluentd

narutaro

sudo apt-get update && apt-get upgrade
sudo apt-get install nodejs
sudo apt-get install npm

##Version

# nodejs -v
v0.10.25

narutaro

Dropbox

First Header	Second Header
Content Cell	Content Cell
Content Cell	Content Cell

Google Drive

Flickr

Copy

Amazon cloud drive

narutaro

I tried followings and picked up copy.com.

• dropbox
• google drive
• flickr
• compy.com
• amazon cloud drive

Here is why.

narutaro

#Thrift 0.9.3 on ubuntu 14.04

Thrift's official page looks not maitained well. I followed the instruction on the official page but did not work. I hope this page helps a person like me with saving time.

sudo apt-get install libboost-dev libboost-test-dev libboost-program-options-dev libboost-system-dev libboost-filesystem-dev libevent-dev automake libtool flex bison pkg-config g++ libssl-dev ant

Ubuntu 14.04 gets following errors.

E: Unable to locate package libboost-dev

narutaro

Settin up nfsen

Error

inoue@netflowc:~/nfsen-1.3.6p1$ ./install.pl 
Can't locate RRDs.pm in @INC (you may need to install the RRDs module) (@INC contains: ./libexec ./installer-items /etc/perl /usr/local/lib/perl/5.18.2 /usr/local/share/perl/5.18.2 /usr/lib/perl5 /usr/share/perl5 /usr/lib/perl/5.18 /usr/share/perl/5.18 /usr/local/lib/site_perl .) at libexec/NfSenRRD.pm line 38.
BEGIN failed--compilation aborted at libexec/NfSenRRD.pm line 38.
Compilation failed in require at libexec/NfSen.pm line 43.
BEGIN failed--compilation aborted at libexec/NfSen.pm line 43.
Compilation failed in require at ./install.pl line 46.

narutaro

#Every second, 500 times

$ sar 1 500
Linux 3.19.0-25-generic (netflowc) 	10/30/2015 	_x86_64_	(16 CPU)

02:00:09 PM     CPU     %user     %nice   %system   %iowait    %steal     %idle
02:00:10 PM     all     20.25      0.00      6.00     23.25      0.00     50.50
02:00:11 PM     all     19.02      0.00      6.21     32.71      0.00     42.06
02:00:12 PM all 18.47 0.00 5.69 33.21 0.00 42.63

narutaro

#How to delete a file starting with - or --

##NO.1 - NG

$ rm '--attack_metadata'
rm: unrecognized option '--attack_metadata'
Try 'rm ./--attack_metadata' to remove the file ‘--attack_metadata’.
Try 'rm --help' for more information.

narutaro

#Situation You have a running job which is taking longer time than expected and you gotta hit a road now. However, you don't want to stop the job cause it run long already.

#Solution This is what you can do.

1. Ctrl+Z and put the job back ground
2. Check the job id with jobs
3. disown the job

narutaro

#Welcome to Gistlog

##What is Gistlog? Gistlog makes your gist snippets readable like a blog. This page is the live sample of Gistlog. All the contents here come form your gist.

##Intallation This repositry consists of html, css and js so just clone this repogitry to any web server. I recommend to use github pages. Just push to your repositry as <your_name>.github.io

git clone https://github.com/narutaro/narutaro.github.io.git