Skip to content

Instantly share code, notes, and snippets.

View Mageswaran1989's full-sized avatar

Mageswaran Mageswaran1989

19 followers · 0 following
  • Hyderabad, India
View GitHub Profile
@Mageswaran1989
Mageswaran1989 / emr_monitor.py
Created April 2, 2020 19:25
"""
Copy this file to AIRFLOW_HOME/dags/
"""
import boto3
from datetime import datetime, timedelta
import logging
logging = logging.getLogger(__name__)
logging.setLevel("ERROR")
@Mageswaran1989
Mageswaran1989 / StarWars.csv
Last active January 30, 2020 04:23
name height weight eyecolor haircolor jedi species
Anakin Skywalker 188 84 blue blond jedi human
Padme Amidala 165 45 brown brown no_jedi human
Luke Skywalker 172 77 blue blond jedi human
Leia Skywalker 150 49 brown brown no_jedi human
Qui-Gon Jinn 193 89 blue brown jedi human
Obi-Wan Kenobi 182 77 bluegray auburn jedi human
Han Solo 180 80 brown brown no_jedi human
Sheev Palpatine 173 75 blue red no_jedi human
R2-D2 96 32 no_jedi droid
@Mageswaran1989
Mageswaran1989 / spark_dataset.ipynb
Created January 30, 2020 03:24
A gentele introduction to Spark Datasets
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
@Mageswaran1989
Mageswaran1989 / text_preprocessing.ipynb
Created December 26, 2019 04:53
Tensorflow flow Text Preprocessing
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
@Mageswaran1989
Mageswaran1989 / tensorflow_dataset_estimatort.py
Created August 28, 2019 15:16
Tensorflow 2.0 : Scalable Modleling with Estimator and Dataset
import glob
import os
import shutil
import tensorflow as tf
from tqdm import tqdm
import numpy as np
import matplotlib.pyplot as plt
import psutil
from absl import logging
@Mageswaran1989
Mageswaran1989 / plotly_test.json
Created June 28, 2019 08:04
{'data': [Scattergl({
'hoverlabel': {'bgcolor': '#EBEDF3', 'bordercolor': '#EBEDF3', 'font': {'color': 'black'}},
'hovertemplate': '%{y:d}',
'line': {'color': '#7252B6', 'width': 4},
'marker': {'size': 10},
'mode': 'lines+markers',
'name': 'Bytes Read',
'x': [1, 2, 3, 4, 5],
'xaxis': 'x',
'y': [1590002400, 6360009600, 4240006400, 530000800, 3180004800],
@Mageswaran1989
Mageswaran1989 / list_s3_folder.py
Last active November 7, 2020 07:22
import os
import boto3
from collections import defaultdict
import botocore
def get_matching_s3_objects(bucket,
aws_access_key_id,
aws_secret_access_key,
region_name,
prefix='',
@Mageswaran1989
Mageswaran1989 / TwitterWithNeo4j.scala
Created May 14, 2016 16:38
Successfully integrated Neo4j with Spark Streaming
package org.aja.tej.examples.streaming.twitter
import com.google.gson.Gson
import org.aja.tej.utils.{TejUtils, TejTwitterUtils}
import org.anormcypher.{Cypher, Neo4jREST}
import org.apache.spark.sql.{AnalysisException, Row, SQLContext}
import org.apache.spark.streaming.twitter.TwitterUtils
import org.apache.spark.streaming.{Seconds, StreamingContext}
import play.api.libs.ws.ning
@Mageswaran1989
Mageswaran1989 / Neo4j + Spark Streaming.scala
Created May 14, 2016 16:35
//For any updates check : https://github.com/Mageswaran1989/aja/blob/master/src/examples/scala/org/aja/tej/examples/streaming/twitter/TwitterWithNeo4j.scala
package org.aja.tej.examples.streaming.twitter
import com.google.gson.Gson
import org.aja.tej.utils.{TejUtils, TejTwitterUtils}
import org.anormcypher.{Cypher, Neo4jREST}
import org.apache.spark.sql.{AnalysisException, Row, SQLContext}
import org.apache.spark.streaming.twitter.TwitterUtils
import org.apache.spark.streaming.{Seconds, StreamingContext}
import play.api.libs.ws.ning
@Mageswaran1989
Mageswaran1989 / GroupTest.scala
Created November 24, 2015 12:26
package org.aja.tej.tej.test.spark
/**
* Created by mageswaran on 9/8/15.
*/
import java.util.Random
import org.apache.spark.{SparkConf, SparkContext}