danish-rehman

Rolling count of stats using pyspark

CQL Query

CREATE TABLE rollups_min (       
      event_min text,       
      time timestamp,       
      value int,       
 PRIMARY KEY (event_min, time)

Count stats for twitter stream and store in Cassandra

cd $SPARK_HOME

/bin/spark-submit --packages TargetHolding/pyspark-cassandra:0.3.5 /Users/drehman/Apps/workspace/spark_cassandra_stream_example.py

python twitter_rolling_count.py -q data -d data 2>&1 | nc -lk 10.0.0.235 9999

ISSUE: ReceiverTracker: Deregistered receiver for stream 0: Restarting receiver with delay
FIX: Whenever the driver stops, this message will be their. Increase the timeout to see if it happend pre-maturely.

ISSUE: Whenever the driver stops, this message will be their.
FIX: Increase the timeout to see if it happend pre-maturely.

ISSUE: Cannot obtain a new communication channel pyspark

Compile and install pyspark-cassandra connector.

#INSTALL JDK and EXPORT JAVA_HOME

brew install sbt
sbt compile 
sbt spPublishLocal
make dist

cd $SPARK_HOME
./bin/spark-submit --packages TargetHolding/pyspark-cassandra:0.3.5 /Users/drehman/Apps/workspace/spark_cassandra_example.py

Follow this official link

Download - Binary link

tar xzf apache-cassandra-<Version>.tgz

CQLSH shell

python bot_stream.py 2>&1 | nc -lk 127.0.0.1 9999

Introduction to Spark internals 2012

	consumer_key = 'your-consumer-key'
	consumer_secret = 'your-consumer-secret'
	access_token = 'your-access-token'
	access_secret = 'your-access-secret'

	log4j = sc._jvm.org.apache.log4j
	log4j.LogManager.getRootLogger().setLevel(log4j.Level.ERROR)