narutaro

#Thrift 0.9.3 on ubuntu 14.04

Thrift's official page looks not maitained well. I followed the instruction on the official page but did not work. I hope this page helps a person like me with saving time.

sudo apt-get install libboost-dev libboost-test-dev libboost-program-options-dev libboost-system-dev libboost-filesystem-dev libevent-dev automake libtool flex bison pkg-config g++ libssl-dev ant

Ubuntu 14.04 gets following errors.

E: Unable to locate package libboost-dev

narutaro

I tried followings and picked up copy.com.

• dropbox
• google drive
• flickr
• compy.com
• amazon cloud drive

Here is why.

narutaro

Dropbox

First Header	Second Header
Content Cell	Content Cell
Content Cell	Content Cell

Google Drive

Flickr

Copy

Amazon cloud drive

narutaro

sudo apt-get update && apt-get upgrade
sudo apt-get install nodejs
sudo apt-get install npm

##Version

# nodejs -v
v0.10.25

narutaro

#Is NetFlow streaming data analysis possible with fluentd?

I wanted to do some analysis against NetFlow data that I receive everyday. The analyis that I wanna do is various. It is simple pattern matching for a specific IP address, detecting specifc traffic pattern, figuring out network graph and calicurate proximity of certain nodes and so on so forth. I might use norikura plug-in for that purpose later but not sure now.

I know storm, kafuka, spark streaming and even memSQL and VoltDB are good for this purpose but wanted to go with quick way for now.

#Captureing NetFlow with fluentd

I use this NetFlow plug-in on fluentd

narutaro

#Open source NetFlow collector ##Over ten years players ###nfsen family - nfpcapd nfsen has very stable and well-accumulated feature set. It provides full-stack of NetFlow feature. NetFlow collector daemon nfpcapd, flow dump file operation tool nfdump and Web front-end nfsen. Community is still avtive.

###ntop-ng family - nprobe

###pmacct family- pmacctd Momory table is something that I want to try.

narutaro

#Preparing a docker container

Use a docker container to setup pmacct. Port 5678/udp for receiving NetFlow.

$ docker run -d -h pmacct --name=pmacct -p 8889:8889 -p 5678:5678/udp -v /home/inoue/docker/pmacct:/pmacct -it ubuntu

Build essentials.

apt-get install build-essential checkinstall

narutaro

#Symptom

# gem install rest-client
Fetching: unf_ext-0.0.7.1.gem (100%)
Building native extensions.  This could take a while...
ERROR:  Error installing rest-client:
	ERROR: Failed to build gem native extension.

        /usr/bin/ruby1.9.1 extconf.rb

narutaro

Usage

root@tcpreplay:/tcpreplay# tcpreplay -i eth0 -K --mbps 10 smallFlows.pcap 
File Cache is enabled
Actual: 14261 packets (9216531 bytes) sent in 7.03 seconds.
Rated: 1249999.6 Bps, 9.99 Mbps, 1934.15 pps
Flows: 1209 flows, 163.97 fps, 14243 flow packets, 18 non-flow
Statistics for network device: eth0
	Attempted packets:         14261
	Successful packets: 14261

narutaro

The best NetFlow generator flowgen

me@vaio:~/NetFlow-Generator$ time ./flowgen -p 5555 -i 1 --dstaddr 1.1.1.1 -n 1000000 10.40.8.240 
collector = 10.40.8.240
count     = 1000000
spoof     = (none)
port      = 5555
wait      = 0 (msec)
interval = 1