Dedunu Dhananjaya dedunumax
dedunumax
/ EMR_with_ReleaseLabel_and_SubnetID.py
Last active
April 27, 2016 19:18
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| import boto.emr | |
| import boto.exception | |
| from boto.emr.instance_group import InstanceGroup | |
| __author__ = 'dedunu' | |
| connection = boto.emr.connect_to_region( | |
| region_name='us-east-1', | |
| aws_access_key_id='<Your AWS Access Key>', | |
| aws_secret_access_key='<You AWS Secred Key>', |
dedunumax
/ EMR_with_ReleaseLabel.py
Last active
January 5, 2016 18:26
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| import boto.emr | |
| import boto.exception | |
| from boto.emr.instance_group import InstanceGroup | |
| __author__ = 'dedunu' | |
| connection = boto.emr.connect_to_region( | |
| region_name='us-east-1', | |
| aws_access_key_id='<Your AWS Access Key>', | |
| aws_secret_access_key='<You AWS Secred Key>', |
dedunumax
/ rss_fetch_python3.py
Created
December 13, 2015 06:44
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| from urllib.request import urlopen | |
| from xml.dom import minidom | |
| __author__ = 'dedunu' | |
| class Reader: | |
| def get_url_list(self, url_string): | |
| data = urlopen(url_string) | |
| rss_string = b'' |
dedunumax
/ rss_fetch_python2.py
Created
December 13, 2015 06:42
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| import urllib2 | |
| from xml.dom import minidom | |
| __author__ = 'dedunu' | |
| class Reader: | |
| def get_url_list(self, url_string): | |
| data = urllib2.urlopen(url_string) | |
| rss_string = '' |
dedunumax
/ remove_all_followers.py
Last active
December 15, 2022 13:57
Remove all the followers from your twitter account.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| import tweepy | |
| __author__ = 'dedunumax' | |
| ''' | |
| This script will remove all the followers from your twitter account. For that first it will block user one by one and | |
| then unblock them. If you are following your followers, you won't be subscribed to them anymore once you run this job. | |
| Rub this script carefully. | |
| Install tweepy module using pip. To install tweepy run below command in your terminal. |
dedunumax
/ Hadoop Cluster ID mismatch error log
Last active
August 29, 2015 14:21
Hadoop Cluster ID mismatch error
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| FATAL org.apache.hadoop.hdfs.server.datanode.DataNode: Initialization failed for Block pool (Datanode Uuid unassigned) service to master/192.168.1.1:9000. Exiting. | |
| java.io.IOException: Incompatible clusterIDs in /home/hadoop/hadoop/data: namenode clusterID = CID-68a4c0d2-5524-486e-8bc9-e1fc3c5c2e29; datanode clusterID = CID-c6c3e9e5-be1c-4a3f-a4b2-bb9441a989c5 | |
| at org.apache.hadoop.hdfs.server.datanode.DataStorage.doTransition(DataStorage.java:646) | |
| at org.apache.hadoop.hdfs.server.datanode.DataStorage.addStorageLocations(DataStorage.java:320) | |
| at org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:403) | |
| at org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:422) | |
| at org.apache.hadoop.hdfs.server.datanode.DataNode.initStorage(DataNode.java:1311) | |
| at org.apache.hadoop.hdfs.server.datanode.DataNode.initBlockPool(DataNode.java:1276) | |
| at org.apache.hadoop.hdfs.server.datanode.BPOfferService.verifyAndSetNamespaceInfo( |
dedunumax
/ HadoopJobOutputForMultiInput
Created
May 21, 2015 10:04
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| 15/05/21 09:48:09 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032 | |
| 15/05/21 09:48:10 WARN mapreduce.JobSubmitter: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this. | |
| 15/05/21 09:48:10 INFO input.FileInputFormat: Total input paths to process : 1 | |
| 15/05/21 09:48:10 INFO input.FileInputFormat: Total input paths to process : 1 | |
| 15/05/21 09:48:10 INFO input.FileInputFormat: Total input paths to process : 1 | |
| 15/05/21 09:48:10 INFO mapreduce.JobSubmitter: number of splits:3 | |
| 15/05/21 09:48:10 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1432197111554_0002 | |
| 15/05/21 09:48:10 INFO impl.YarnClientImpl: Submitted application application_1432197111554_0002 | |
| 15/05/21 09:48:10 INFO mapreduce.Job: The url to track the job: http://hdp101.local:8088/proxy/application_1432197111554_0002/ | |
| 15/05/21 09:48:10 INFO mapreduce.Job: Running job: job_1432197111554_0002 |
dedunumax
/ MultiInputSampleHadoop.java
Created
May 21, 2015 09:50
Sample Main class for Hadoop MultiInputs.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| package org.dedunu.hadoop.muiltiinputsample; | |
| import org.apache.hadoop.conf.Configuration; | |
| import org.apache.hadoop.fs.Path; | |
| import org.apache.hadoop.io.Text; | |
| import org.apache.hadoop.mapreduce.Job; | |
| import org.apache.hadoop.mapreduce.lib.input.MultipleInputs; | |
| import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat; | |
| import org.apache.hadoop.mapreduce.lib.output.TextOutputFormat; |
dedunumax
/ AirlineInputFormat.java
Created
May 21, 2015 09:49
Sample Custom InputFormat class for Hadoop.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| package org.dedunu.hadoop.muiltiinputsample; | |
| import org.apache.hadoop.io.Text; | |
| import org.apache.hadoop.mapreduce.InputSplit; | |
| import org.apache.hadoop.mapreduce.RecordReader; | |
| import org.apache.hadoop.mapreduce.TaskAttemptContext; | |
| import org.apache.hadoop.mapreduce.lib.input.FileInputFormat; | |
| import org.apache.hadoop.mapreduce.lib.input.LineRecordReader; | |
| import java.io.IOException; |
dedunumax
/ Records.txt
Last active
August 29, 2015 14:21
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| ABC_America_Airline_Jan_2015.txt | |
| Date SSN From To Amount($) | |
| 01/15/15 12345678 CO TX 200 | |
| 01/16/15 23452345 NV UT 150 | |
| 01/16/15 34252454 CA CO 200 | |
| 01/16/15 56785678 CA TX 150 | |
| 01/17/15 43545666 LA UT 200 | |
| 01/17/15 67856783 TX CO 150 |