winse winse
winse
/ zeppelin-0.7.1-CMD.sh
Last active
April 19, 2017 03:30
zeppelin-0.7.1 build on docker Centos6. Ref: http://zeppelin.apache.org/docs/0.7.1/install/build.html
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| tar zxvf zeppelin-0.7.1.tgz | |
| cd zeppelin-0.7.1 | |
| ./dev/change_scala_version.sh 2.11 | |
| # fix some network problems | |
| MAVEN_REPO=/opt/repo | |
| rm -rf $MAVEN_REPO/org/apache/pig/pig/0.16.0/* | |
| rm -rf $MAVEN_REPO/org/apache/tez/tez-api/0.7.0/* | |
| wget -P $MAVEN_REPO/org/apache/pig/pig/0.16.0/ http://nexus-private.hortonworks.com:8081/nexus/content/shadows/central-m1/org.apache.pig/jars/pig-0.16.0-h2.jar | |
| wget -P $MAVEN_REPO/com/github/eirslett/yarn/0.18.1/yarn-0.18.1./ https://github.com/yarnpkg/yarn/releases/download/v0.18.1/yarn-v0.18.1.tar.gz |
winse
/ hadoop-2.6.5-CMD.sh
Created
April 19, 2017 07:38
build hadoop-2.6.5 with snappy on docker Centos6
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| tar zxvf snappy-1.1.3.tar.gz | |
| cd snappy-1.1.3 | |
| ./configure | |
| make && make install | |
| cd hadoop-2.6.5-src | |
| mvn clean package -Pdist,native -Dmaven.javadoc.skip=true -DskipTests -Drequire.snappy=true -Dbundle.snappy=true -Dsnappy.lib=/usr/local/lib |
winse
/ spark-sql.CMD
Last active
April 25, 2017 07:57
build spark-2.1.0 and hive-2.3-snapshot on docker Centos6. Ref: http://spark.apache.org/docs/latest/building-spark.html#building-a-runnable-distribution
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| screen # 以防万一SSH断了 | |
| docker exec -ti dev bash | |
| # 1. build spark | |
| export MAVEN_OPTS="-Xmx2g -XX:ReservedCodeCacheSize=512m" | |
| ./dev/make-distribution.sh --name spark-2.1.0-hadoop2.6.5 --tgz --mvn "$(which mvn)" \ | |
| -Psparkr -Phadoop-2.6 -Dhadoop.version=2.6.5 -Phive -Phive-thriftserver -Pyarn -DskipTests -Dmaven.javadoc.skip=true | |
| [root@680108a35829 spark-2.1.0]# ll spark-2.1.0-bin-hadoop2.6.5.tgz | |
| -rw-r--r-- 1 root root 185594381 Apr 21 09:48 spark-2.1.0-bin-hadoop2.6.5.tgz |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| # Re-install openclaw on pi-4 |
OlderNewer