Adam adam-phillipps
adam-phillipps
/ gist:04af1de0c1e383a795f2956534fc114f
Created
September 1, 2018 02:11
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| cat << WPR >> wrapper.py | |
| import boto3 | |
| import os | |
| # create clients | |
| s3 = boto3.resource('s3') | |
| sqs = boto3.client('sqs', 'us-west-2') | |
| # count the backlog messages | |
| def msg_count(): |
adam-phillipps
/ logs from crawler
Last active
August 31, 2018 15:10
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| [root@ip-172-31-24-37 /]# /var/lib/cloud/instance/scripts/part-001 | |
| Loaded plugins: priorities, update-motd, upgrade-helper | |
| amzn-main | 2.1 kB 00:00:00 | |
| amzn-updates | 2.5 kB 00:00:00 | |
| No packages marked for update | |
| Loaded plugins: priorities, update-motd, upgrade-helper | |
| Package 1:java-1.8.0-openjdk-1.8.0.181-8.b13.39.39.amzn1.x86_64 already installed and latest version | |
| Nothing to do | |
| Loaded plugins: priorities, update-motd, upgrade-helper | |
| No Match for argument: java-1.7.0-openjdk |
adam-phillipps
/ gist:dcbf561ce5e5d291f89d71d991c6e3ac
Created
August 30, 2018 05:01
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Running jar with 'java -jar SmashEndurance.jar 1120000000000 http://www.rabo.org.ro/'... | |
| Sending 'GET' request to URL : https://www.googleapis.com/pagespeedonline/v4/runPagespeed?key=AIzaSyCasztV2Q743eTjhm_RrR4Io6KCAZ_pOB0&url=http://www.rabo.org.ro/ | |
| Response Code : 200 | |
| Aug 30, 2018 4:59:02 AM org.openqa.selenium.phantomjs.PhantomJSDriverService <init> | |
| INFO: executable: /root/phantomjs | |
| Aug 30, 2018 4:59:02 AM org.openqa.selenium.phantomjs.PhantomJSDriverService <init> | |
| INFO: port: 7316 | |
| Aug 30, 2018 4:59:02 AM org.openqa.selenium.phantomjs.PhantomJSDriverService <init> | |
| INFO: arguments: [--webdriver=7316, --webdriver-logfile=/root/phantomjsdriver.log] |
adam-phillipps
/ userdata
Created
August 28, 2018 23:50
gets run when an instance is provisioned (one time)
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| #!/bin/bash | |
| export $QUEUE=https://sqs.us-west-2.amazonaws.com/088617881078/backlog_crawlBot | |
| export $WIP_Q=https://sqs.us-west-2.amazonaws.com/088617881078/wip_crawlBot | |
| export $ART_NAME=SmashEndurance.jar | |
| export $ART_BKT=endurance-crawl-artifacts | |
| export $OUT_BKT=endurance-crawl-output | |
| # get docker and start the daemon | |
| sudo yum update -y | |
| sudo yum install docker -y | |
| sudo usermod -a -G docker ec2-user |
adam-phillipps
/ gist:9b10cc759b4b6b23f6d486736c22b669
Created
August 7, 2018 17:22
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| --- | |
| Description: Instances for the cluster via LaunchConfig through Autoscaling | |
| Outputs: | |
| LearningLockerLaunchConfig: | |
| Value: | |
| Ref: LearningLockerLaunchConfig | |
| LearningLockerLaunchConfigName: | |
| Value: LearningLockerLaunchConfig | |
| Resources: | |
| LearningLockerLaunchConfig: |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| --- | |
| Description: Instances for the cluster via LaunchConfig through Autoscaling | |
| Outputs: | |
| MongoLaunchConfig: | |
| Value: | |
| Ref: MongoLaunchConfig | |
| MongoLaunchConfigName: | |
| Value: MongoLaunchConfig | |
| Resources: | |
| MongoLaunchConfig: |
adam-phillipps
/ mystuff
Created
June 14, 2018 07:20
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Getting self-IP | |
| No metadata, found 'hostname -i': 172.17.0.2 | |
| Using ip: 172.17.0.2 | |
| Modifying configuration | |
| Creating the /usr/spark-2.3.0/conf/spark-env.sh file... | |
| Adding to spark-env.sh: | |
| export SPARK_LOCAL_IP=172.17.0.2 | |
| export PYSPARK_PYTHON=python | |
| Creating the /spark/conf/spark-defaults.conf file... |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| root@cdee95838ee0:/usr/spark-2.3.0# spark-submit examples/src/main/python/kmeans.py data/mllib/kmeans_data.txt 10 5 | |
| Spark Command: /usr/jdk1.8.0_131/bin/java -cp /usr/spark-2.3.0/conf/:/usr/spark-2.3.0/jars/*:/usr/hadoop-2.8.3/etc/hadoop/:/usr/hadoop-2.8.3/etc/hadoop/*:/usr/hadoop-2.8.3/share/hadoop/common/lib/*:/usr/hadoop-2.8.3/share/hadoop/common/*:/usr/hadoop-2.8.3/share/hadoop/hdfs/*:/usr/hadoop-2.8.3/share/hadoop/hdfs/lib/*:/usr/hadoop-2.8.3/share/hadoop/yarn/lib/*:/usr/hadoop-2.8.3/share/hadoop/yarn/*:/usr/hadoop-2.8.3/share/hadoop/mapreduce/lib/*:/usr/hadoop-2.8.3/share/hadoop/mapreduce/*:/usr/hadoop-2.8.3/share/hadoop/tools/lib/* -Xmx1g org.apache.spark.deploy.SparkSubmit examples/src/main/python/kmeans.py data/mllib/kmeans_data.txt 10 5 | |
| ======================================== | |
| 2018-06-14 06:35:29 WARN NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable | |
| WARN: This is a naive implementation of KMeans Clustering and is given | |
adam-phillipps
/ gist:43d9c3e7d1caa8af1500ab56c8eb2bdf
Created
June 14, 2018 06:26
spark-submit config fail
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| $ python bin/smash_planner.py | |
| Getting self-IP | |
| No metadata, found 'hostname -i': 172.17.0.2 | |
| Using ip: 172.17.0.2 | |
| Modifying configuration | |
| Creating the /usr/spark-2.3.0/conf/spark-env.sh file... | |
| Adding to spark-env.sh: | |
| export SPARK_LOCAL_IP=172.17.0.2 | |
| export PYSPARK_PYTHON=python |
adam-phillipps
/ added_SPARK_MASTER_IP
Created
June 9, 2018 22:28
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Getting self-IP | |
| No metadata, using 'hostname -i': 172.17.0.2 | |
| Created Spark command: su ubuntu -c '/usr/spark-2.3.0/bin/spark-submit --master spark://spark-master:7077 /usr/local/smash_planner/build_phase.py' | |
| Modifying configuration | |
| Creating the /usr/spark-2.3.0/conf/spark-env.sh file... | |
| Adding to spark-env.sh: | |
| export SPARK_LOCAL_IP=172.17.0.2 |