wavescholar
/ gist:6cc708de5f9bea623c86
Created
September 6, 2014 23:53
Fedora Setup on clean install
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| yum update | |
| yum groupinstall "Books and Guides" "C Development Tools and Libraries" "Development Tools" "Fedora Eclipse" "System Tools" "Editors" | |
| rpm -ivh jdk-7u67-linux-x64.rpm | |
| #as su | |
| alternatives --install /usr/bin/java java /usr/java/jdk1.7.0_67/jre/bin/java 2 | |
| alternatives --install /usr/bin/javaws javaws /usr/java/jdk1.7.0_67/jre/bin/javaws 2 | |
| alternatives --install /usr/bin/javac javac /usr/java/jdk1.7.0_67/bin/javac 2 | |
| alternatives --install /usr/bin/jar jar /usr/java/jdk1.7.0_67/bin/jar 2 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| First This | |
| https://gist.github.com/wavescholar/6cc708de5f9bea623c86 | |
| Get The RPM | |
| http://www.cloudera.com/content/cloudera-content/cloudera-docs/CDH5/latest/CDH5-Quick-Start/cdh5qs_yarn_pseudo.html | |
| sudo yum --nogpgcheck localinstall cloudera-cdh-5-0.x86_64.rpm |
wavescholar
/ gist:088fac6a275a3e44fb80
Created
September 7, 2014 20:18
running the hadoop grep example
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| hadoop fs -put /etc/hadoop/conf/*.xml input | |
| [bcampbell@localhost ~]$ hadoop fs -ls input | |
| Found 7 items | |
| -rw-r--r-- 1 bcampbell supergroup 507105 2014-09-07 15:55 input/Milton_ParadiseLost.txt | |
| -rw-r--r-- 1 bcampbell supergroup 246679 2014-09-07 15:55 input/WilliamYeats.txt | |
| -rw-r--r-- 1 bcampbell supergroup 2133 2014-09-07 15:58 input/core-site.xml | |
| -rw-r--r-- 1 bcampbell supergroup 2324 2014-09-07 15:58 input/hdfs-site.xml | |
| -rw-r--r-- 1 bcampbell supergroup 246679 2014-09-07 15:56 input/inputWC | |
| -rw-r--r-- 1 bcampbell supergroup 1549 2014-09-07 15:58 input/mapred-site.xml | |
| -rw-r--r-- 1 bcampbell supergroup 2375 2014-09-07 15:58 input/yarn-site.xml |
wavescholar
/ gist:1d7527840f386f567a40
Last active
August 29, 2015 14:06
Maven Pom file for generating eclipse workspace - CDH5 Fedora 20
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" | |
| xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"> | |
| <modelVersion>4.0.0</modelVersion> | |
| <groupId>com.bcampbell.hadoopproject</groupId> | |
| <artifactId>wordcount</artifactId> | |
| <version>0.0.1</version> | |
| <packaging>jar</packaging> | |
| <name>wordcount</name> |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Maven Eclipse Setup | |
| run this in the directory | |
| mvn archetype:generate \ | |
| -DarchetypeGroupId=org.apache.maven.archetypes \ | |
| -DarchetypeArtifactId=maven-archetype-quickstart \ | |
| -DgroupId=com.bcampbell.hadoopproject \ | |
| -DartifactId=wordcount | |
| Many things will be downloaded from cloudera. |
wavescholar
/ gist:bd185d798e94a025446a
Last active
August 29, 2015 14:15
HDFS Compression Code Snippet
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| //Other options for typeare record and block. I'n not sure that block will work with any other codec than bzip2 | |
| CompressionType compressionType= CompressionType.NONE; | |
| compressionCodecEnum {gzip, bzip2,none}; | |
| compressionCodecEnum compressionCodecType = compressionCodecEnum.bzip2; | |
| if( compressionCodecType==compressionCodecEnum.bzip2) | |
| { | |
| org.apache.hadoop.io.SequenceFile.Writer.Option compressionClass = SequenceFile.Writer.valueClass(GzipCodec.class); | |
| CompressionCodec Codec = new BZip2Codec(); | |
| org.apache.hadoop.io.SequenceFile.Writer.Option optCom = SequenceFile.Writer.compression(CompressionType.BLOCK, Codec); |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Writable key = (Writable) ReflectionUtils.newInstance( | |
| sequenceFileReader.getKeyClass(), conf); | |
| Writable value = (Writable) ReflectionUtils.newInstance( | |
| sequenceFileReader.getValueClass(), conf); | |
| boolean next = true; | |
| while (next) { | |
| try { | |
| if (sequenceFileReader.next(key, value)) { | |
| next = true; | |
| } else { |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| ./bin/spark-submit --class org.apache.spark.examples.SparkPi --master yarn-cluster --num-executors 3 --driver-memory 4g --executor-memory 2g --executor-cores 1 --queue thequeue lib/spark-examples*.jar 10 | |
| ./bin/spark-submit --class org.apache.spark.examples.SparkPi --master local[2] --num-executors 3 --driver-memory 4g --executor-memory 2g --executor-cores 1 --queue thequeue lib/spark-examples*.jar 10 | |
| val lines = sc.textFile("README.md") | |
| val rdd2 = sc.textFile("hdfs:///some/path.txt") | |
| Spark UI | |
| http://10.22.7.183:4040/jobs/ |
OlderNewer