Related Setup: https://gist.github.com/hofmannsven/6814278
Related Pro Tips: https://ochronus.com/git-tips-from-the-trenches/
Mageswaran Mageswaran1989
Mageswaran1989
/ README.md
Last active
August 29, 2015 14:09
— forked from hofmannsven/README.md
Mageswaran1989
/ stackexchange_demo.scala
Last active
August 29, 2015 14:23
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| /** | |
| * Get the stackexchange data from https://archive.org/details/stackexchange | |
| * Data set used here : math.stackexchange.com | |
| **/ | |
| //Open the file. The text file is an RDD (Resilient Distributed Dataset) | |
| //of Strings, which are the lines of the file. | |
| val postXML = sc.textFile("Posts.xml") | |
| //Count the lines. Note: Run twice and see the difference ;) |
Mageswaran1989
/ StackExchange_ReadMe.txt
Created
June 25, 2015 18:02
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| - Format: 7zipped | |
| - Files: | |
| - **badges**.xml | |
| - UserId, e.g.: "420" | |
| - Name, e.g.: "Teacher" | |
| - Date, e.g.: "2008-09-15T08:55:03.923" | |
| - **comments**.xml | |
| - Id | |
| - PostId | |
| - Score |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| package org.aja.tej.tej.test.spark | |
| /** | |
| * Created by mageswaran on 9/8/15. | |
| */ | |
| import java.util.Random | |
| import org.apache.spark.{SparkConf, SparkContext} |
Mageswaran1989
/ HelloWorldSpark.scala
Created
November 24, 2015 12:17
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| package org.aja.tej.examples | |
| import java.io.File | |
| import org.aja.tej.utils.TejUtils | |
| import org.apache.spark.{SparkConf, SparkContext} | |
| /** |
Mageswaran1989
/ ANNet.scala
Created
November 24, 2015 12:22
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| package org.aja.tej.examples.ml | |
| import org.aja.tej.utils.TejUtils | |
| import org.apache.spark.ml.classification.MultilayerPerceptronClassifier | |
| import org.apache.spark.ml.evaluation.MulticlassClassificationEvaluator | |
| import org.apache.spark.mllib.util.MLUtils | |
| import org.apache.spark.sql.SQLContext | |
| /** | |
| * Created by mageswaran on 25/9/15. |
Mageswaran1989
/ GroupTest.scala
Created
November 24, 2015 12:26
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| package org.aja.tej.tej.test.spark | |
| /** | |
| * Created by mageswaran on 9/8/15. | |
| */ | |
| import java.util.Random | |
| import org.apache.spark.{SparkConf, SparkContext} |
Mageswaran1989
/ Neo4j + Spark Streaming.scala
Created
May 14, 2016 16:35
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| //For any updates check : https://github.com/Mageswaran1989/aja/blob/master/src/examples/scala/org/aja/tej/examples/streaming/twitter/TwitterWithNeo4j.scala | |
| package org.aja.tej.examples.streaming.twitter | |
| import com.google.gson.Gson | |
| import org.aja.tej.utils.{TejUtils, TejTwitterUtils} | |
| import org.anormcypher.{Cypher, Neo4jREST} | |
| import org.apache.spark.sql.{AnalysisException, Row, SQLContext} | |
| import org.apache.spark.streaming.twitter.TwitterUtils | |
| import org.apache.spark.streaming.{Seconds, StreamingContext} | |
| import play.api.libs.ws.ning |
Mageswaran1989
/ TwitterWithNeo4j.scala
Created
May 14, 2016 16:38
Successfully integrated Neo4j with Spark Streaming
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| package org.aja.tej.examples.streaming.twitter | |
| import com.google.gson.Gson | |
| import org.aja.tej.utils.{TejUtils, TejTwitterUtils} | |
| import org.anormcypher.{Cypher, Neo4jREST} | |
| import org.apache.spark.sql.{AnalysisException, Row, SQLContext} | |
| import org.apache.spark.streaming.twitter.TwitterUtils | |
| import org.apache.spark.streaming.{Seconds, StreamingContext} | |
| import play.api.libs.ws.ning |
Mageswaran1989
/ list_s3_folder.py
Last active
November 7, 2020 07:22
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| import os | |
| import boto3 | |
| from collections import defaultdict | |
| import botocore | |
| def get_matching_s3_objects(bucket, | |
| aws_access_key_id, | |
| aws_secret_access_key, | |
| region_name, | |
| prefix='', |
OlderNewer