Brad Karels bradkarels
bradkarels
/ hdfsCmds
Created
January 21, 2015 18:41
A brief history setting up Hadoop (HDFS) locally (psuedo distributed)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| 890 cd ~/Downloads/ | |
| 891 wget http://mirror.cc.columbia.edu/pub/software/apache/hadoop/common/hadoop-2.6.0/hadoop-2.6.0.tar.gz | |
| 915 cd ~ | |
| 917 tar xzf Downloads/hadoop-2.6.0.tar.gz | |
| 920 mv hadoop-2.6.0/ hadoop/ | |
| 923 cd hadoop/ | |
| 924 vim ~/.bashrc | |
| Set HADOOP_HOME and add HADOOP_HOME/bin to PATH | |
| 925 . ~/.bashrc | |
| 926 ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa |
bradkarels
/ embeddedH2.groovy
Created
December 17, 2014 19:48
Simple example: Embed H2 database within a groovy script (file based persistence)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| @GrabConfig(systemClassLoader=true) | |
| @Grab(group='com.h2database', module='h2', version='1.3.176') | |
| import java.sql.* | |
| import groovy.sql.Sql | |
| import org.h2.jdbcx.JdbcConnectionPool | |
| println("More groovy...") | |
| def sql = Sql.newInstance("jdbc:h2:things", "sa", "sa", "org.h2.Driver") // DB files for 'things' in current directory (./hello.h2.db) |
bradkarels
/ spark-local-environment-vars
Created
December 3, 2014 18:49
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| # Spark local environment variables | |
| export SPARK_HOME=/home/bkarels/spark/current | |
| export SPARK_MASTER_IP=127.0.0.1 | |
| export SPARK_MASTER_PORT=7077 | |
| export SPARK_MASTER_WEBUI_PORT=8080 | |
| #SPARK_MASTER-OPTS= | |
| export SPARK_LOCAL_DIRS=$SPARK_HOME/work | |
| export SPARK_WORKER_CORES=2 | |
| export SPARK_WORKER_MEMORY=4G | |
| #export SPARK_WORKER_WEBUI_PORT=8081 |
NewerOlder