Skip to content

Instantly share code, notes, and snippets.

@adam-phillipps

adam-phillipps/logs from crawler

Last active August 31, 2018 15:10
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save adam-phillipps/09fd80eeded7e80db8491a1d27a87d40 to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save adam-phillipps/09fd80eeded7e80db8491a1d27a87d40 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
gistfile1.txt
[root@ip-172-31-24-37 /]# /var/lib/cloud/instance/scripts/part-001
Loaded plugins: priorities, update-motd, upgrade-helper
amzn-main | 2.1 kB 00:00:00
amzn-updates | 2.5 kB 00:00:00
No packages marked for update
Loaded plugins: priorities, update-motd, upgrade-helper
Package 1:java-1.8.0-openjdk-1.8.0.181-8.b13.39.39.amzn1.x86_64 already installed and latest version
Nothing to do
Loaded plugins: priorities, update-motd, upgrade-helper
No Match for argument: java-1.7.0-openjdk
No Packages marked for removal
Loaded plugins: priorities, update-motd, upgrade-helper
Package docker-18.03.1ce-5.0.amzn1.x86_64 already installed and latest version
Nothing to do
Usage: /etc/init.d/docker {start|stop|status|restart|condrestart|try-restart|reload|force-reload}
Stopping docker: [ OK ]
Starting docker: . [ OK ]
/var/lib/cloud/instance/scripts/part-001: line 16: pip: command not found
/var/lib/cloud/instance/scripts/part-001: line 17: pip: command not found
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 50994 0 50994 0 0 180k 0 --:--:-- --:--:-- --:--:-- 181k
{"urls":["https://www.wappalyzer.com/"],"applications":[{"name":"Amazon Cloudfront","confidence":"100","version":"","icon":"Amazon-Cloudfront.svg","website":"http://aws.amazon.com/cloudfront/","categories":[{"31":"CDN"}]},{"name":"Google Analytics","confidence":"100","version":"UA","icon":"Google Analytics.svg","website":"http://google.com/analytics","categories":[{"10":"Analytics"}]},{"name":"Nginx","confidence":"100","version":"1.14.0","icon":"Nginx.svg","website":"http://nginx.org/en","categories":[{"22":"Web Servers"}]},{"name":"Snap.svg","confidence":"100","version":"0.5.1","icon":"Snap.svg.png","website":"http://snapsvg.io","categories":[{"12":"JavaScript Frameworks"}]},{"name":"Stripe","confidence":"100","version":"3","icon":"Stripe.png","website":"http://stripe.com","categories":[{"41":"Payment Processors"}]},{"name":"Swiftlet","confidence":"100","version":"","icon":"Swiftlet.png","website":"http://swiftlet.org","categories":[{"18":"Web Frameworks"}]},{"name":"Ubuntu","confidence":"100","version":"","icon":"Ubuntu.png","website":"http://www.ubuntu.com/server","categories":[{"28":"Operating Systems"}]},{"name":"PHP","confidence":"0","version":"","icon":"PHP.svg","website":"http://php.net","categories":[{"27":"Programming Languages"}]}],"meta":{"language":null}}
Getting jar file from S3
Working...
Getting message from queue
Running jar with 'java -jar SmashEndurance.jar 1150000000000 http://www.carolinacolortones.com'...
Sending 'GET' request to URL : https://www.googleapis.com/pagespeedonline/v4/runPagespeed?key=AIzaSyCasztV2Q743eTjhm_RrR4Io6KCAZ_pOB0&url=http://www.carolinacolortones.com
Response Code : 200
Current page:https://www.facebook.com/carolinacolortones
Aug 31, 2018 3:06:13 PM org.openqa.selenium.phantomjs.PhantomJSDriverService <init>
INFO: executable: /phantomjs
Aug 31, 2018 3:06:13 PM org.openqa.selenium.phantomjs.PhantomJSDriverService <init>
INFO: port: 32524
Aug 31, 2018 3:06:13 PM org.openqa.selenium.phantomjs.PhantomJSDriverService <init>
INFO: arguments: [--webdriver=32524, --webdriver-logfile=/phantomjsdriver.log]
Aug 31, 2018 3:06:13 PM org.openqa.selenium.phantomjs.PhantomJSDriverService <init>
INFO: environment: {}
/phantomjs: line 1: syntax error near unexpected token `newline'
/phantomjs: line 1: `<!doctype html>'
Aug 31, 2018 3:06:34 PM org.openqa.selenium.os.OsProcess checkForError
SEVERE: org.apache.commons.exec.ExecuteException: Process exited with an error: 2 (Exit value: 2)
org.openqa.selenium.WebDriverException: Timed out waiting for driver server to start.
Build info: version: 'unknown', revision: 'unknown', time: 'unknown'
System info: host: 'ip-172-31-24-37', ip: '172.31.24.37', os.name: 'Linux', os.arch: 'amd64', os.version: '4.14.62-65.117.amzn1.x86_64', java.version: '1.8.0_181'
Driver info: driver.version: PhantomJSDriver
at org.openqa.selenium.remote.service.DriverService.waitUntilAvailable(DriverService.java:193)
at org.openqa.selenium.remote.service.DriverService.start(DriverService.java:179)
at org.openqa.selenium.phantomjs.PhantomJSCommandExecutor.execute(PhantomJSCommandExecutor.java:78)
at org.openqa.selenium.remote.RemoteWebDriver.execute(RemoteWebDriver.java:548)
at org.openqa.selenium.remote.RemoteWebDriver.startSession(RemoteWebDriver.java:212)
at org.openqa.selenium.remote.RemoteWebDriver.<init>(RemoteWebDriver.java:130)
at org.openqa.selenium.phantomjs.PhantomJSDriver.<init>(PhantomJSDriver.java:116)
at org.openqa.selenium.phantomjs.PhantomJSDriver.<init>(PhantomJSDriver.java:105)
at ai.smash.endurance.services.APIDemo.initPhantomJS(APIDemo.java:74)
at ai.smash.endurance.services.APIDemo.getSeleniumDriver(APIDemo.java:491)
at ai.smash.endurance.services.APIDemo.parseFacebookPage(APIDemo.java:184)
at ai.smash.endurance.services.APIDemo.main(APIDemo.java:138)
Caused by: org.openqa.selenium.net.UrlChecker$TimeoutException: Timed out waiting for [http://localhost:32524/status] to be available after 20005 ms
at org.openqa.selenium.net.UrlChecker.waitUntilAvailable(UrlChecker.java:100)
at org.openqa.selenium.remote.service.DriverService.waitUntilAvailable(DriverService.java:188)
... 11 more
Caused by: java.util.concurrent.TimeoutException
at java.util.concurrent.FutureTask.get(FutureTask.java:205)
at com.google.common.util.concurrent.SimpleTimeLimiter.callWithTimeout(SimpleTimeLimiter.java:156)
at org.openqa.selenium.net.UrlChecker.waitUntilAvailable(UrlChecker.java:75)
... 12 more
Aug 31, 2018 3:06:34 PM org.openqa.selenium.phantomjs.PhantomJSDriverService <init>
INFO: executable: /phantomjs
Aug 31, 2018 3:06:34 PM org.openqa.selenium.phantomjs.PhantomJSDriverService <init>
INFO: port: 6708
Aug 31, 2018 3:06:34 PM org.openqa.selenium.phantomjs.PhantomJSDriverService <init>
INFO: arguments: [--webdriver=6708, --webdriver-logfile=/phantomjsdriver.log]
Aug 31, 2018 3:06:34 PM org.openqa.selenium.phantomjs.PhantomJSDriverService <init>
INFO: environment: {}
/phantomjs: line 1: syntax error near unexpected token `newline'
/phantomjs: line 1: `<!doctype html>'
Aug 31, 2018 3:06:54 PM org.openqa.selenium.os.OsProcess checkForError
SEVERE: org.apache.commons.exec.ExecuteException: Process exited with an error: 2 (Exit value: 2)
Successfully ran jar file. Saving results to S3
Deleting finished message
Counted number of messages in queue: 0
Working...
Getting message from queue
Traceback (most recent call last):
File "wrapper.py", line 39, in <module>
work()
File "wrapper.py", line 22, in work
msg = sqs.receive_message(QueueUrl=os.getenv('QUEUE'), WaitTimeSeconds=20, MaxNumberOfMessages=1, VisibilityTimeout=120)['Messages'][0]
KeyError: 'Messages'
Raw
logs from crawler
[root@ip-172-31-24-37 /]# /var/lib/cloud/instance/scripts/part-001
Loaded plugins: priorities, update-motd, upgrade-helper
amzn-main | 2.1 kB 00:00:00
amzn-updates | 2.5 kB 00:00:00
No packages marked for update
Loaded plugins: priorities, update-motd, upgrade-helper
Package 1:java-1.8.0-openjdk-1.8.0.181-8.b13.39.39.amzn1.x86_64 already installed and latest version
Nothing to do
Loaded plugins: priorities, update-motd, upgrade-helper
No Match for argument: java-1.7.0-openjdk
No Packages marked for removal
Loaded plugins: priorities, update-motd, upgrade-helper
Package docker-18.03.1ce-5.0.amzn1.x86_64 already installed and latest version
Nothing to do
Usage: /etc/init.d/docker {start|stop|status|restart|condrestart|try-restart|reload|force-reload}
Stopping docker: [ OK ]
Starting docker: . [ OK ]
/var/lib/cloud/instance/scripts/part-001: line 16: pip: command not found
/var/lib/cloud/instance/scripts/part-001: line 17: pip: command not found
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 50994 0 50994 0 0 754k 0 --:--:-- --:--:-- --:--:-- 766k
{"urls":["https://www.wappalyzer.com/"],"applications":[{"name":"Amazon Cloudfront","confidence":"100","version":"","icon":"Amazon-Cloudfront.svg","website":"http://aws.amazon.com/cloudfront/","categories":[{"31":"CDN"}]},{"name":"Google Analytics","confidence":"100","version":"UA","icon":"Google Analytics.svg","website":"http://google.com/analytics","categories":[{"10":"Analytics"}]},{"name":"Nginx","confidence":"100","version":"1.14.0","icon":"Nginx.svg","website":"http://nginx.org/en","categories":[{"22":"Web Servers"}]},{"name":"Snap.svg","confidence":"100","version":"0.5.1","icon":"Snap.svg.png","website":"http://snapsvg.io","categories":[{"12":"JavaScript Frameworks"}]},{"name":"Stripe","confidence":"100","version":"3","icon":"Stripe.png","website":"http://stripe.com","categories":[{"41":"Payment Processors"}]},{"name":"Swiftlet","confidence":"100","version":"","icon":"Swiftlet.png","website":"http://swiftlet.org","categories":[{"18":"Web Frameworks"}]},{"name":"Ubuntu","confidence":"100","version":"","icon":"Ubuntu.png","website":"http://www.ubuntu.com/server","categories":[{"28":"Operating Systems"}]},{"name":"PHP","confidence":"0","version":"","icon":"PHP.svg","website":"http://php.net","categories":[{"27":"Programming Languages"}]}],"meta":{"language":null}}
Getting jar file from S3
Working...
Getting message from queue
Running jar with 'java -jar SmashEndurance.jar 1150000000000 http://www.carolinacolortones.com'...
Sending 'GET' request to URL : https://www.googleapis.com/pagespeedonline/v4/runPagespeed?key=AIzaSyCasztV2Q743eTjhm_RrR4Io6KCAZ_pOB0&url=http://www.carolinacolortones.com
Response Code : 200
Current page:https://www.facebook.com/carolinacolortones
Aug 31, 2018 3:23:51 AM org.openqa.selenium.phantomjs.PhantomJSDriverService <init>
INFO: executable: /phantomjs
Aug 31, 2018 3:23:51 AM org.openqa.selenium.phantomjs.PhantomJSDriverService <init>
INFO: port: 18318
Aug 31, 2018 3:23:51 AM org.openqa.selenium.phantomjs.PhantomJSDriverService <init>
INFO: arguments: [--webdriver=18318, --webdriver-logfile=/phantomjsdriver.log]
Aug 31, 2018 3:23:51 AM org.openqa.selenium.phantomjs.PhantomJSDriverService <init>
INFO: environment: {}
/phantomjs: line 1: syntax error near unexpected token `newline'
/phantomjs: line 1: `<!doctype html>'
Aug 31, 2018 3:24:11 AM org.openqa.selenium.os.OsProcess checkForError
SEVERE: org.apache.commons.exec.ExecuteException: Process exited with an error: 2 (Exit value: 2)
org.openqa.selenium.WebDriverException: Timed out waiting for driver server to start.
Build info: version: 'unknown', revision: 'unknown', time: 'unknown'
System info: host: 'ip-172-31-24-37', ip: '172.31.24.37', os.name: 'Linux', os.arch: 'amd64', os.version: '4.14.62-65.117.amzn1.x86_64', java.version: '1.8.0_181'
Driver info: driver.version: PhantomJSDriver
at org.openqa.selenium.remote.service.DriverService.waitUntilAvailable(DriverService.java:193)
at org.openqa.selenium.remote.service.DriverService.start(DriverService.java:179)
at org.openqa.selenium.phantomjs.PhantomJSCommandExecutor.execute(PhantomJSCommandExecutor.java:78)
at org.openqa.selenium.remote.RemoteWebDriver.execute(RemoteWebDriver.java:548)
at org.openqa.selenium.remote.RemoteWebDriver.startSession(RemoteWebDriver.java:212)
at org.openqa.selenium.remote.RemoteWebDriver.<init>(RemoteWebDriver.java:130)
at org.openqa.selenium.phantomjs.PhantomJSDriver.<init>(PhantomJSDriver.java:116)
at org.openqa.selenium.phantomjs.PhantomJSDriver.<init>(PhantomJSDriver.java:105)
at ai.smash.endurance.services.APIDemo.initPhantomJS(APIDemo.java:74)
at ai.smash.endurance.services.APIDemo.getSeleniumDriver(APIDemo.java:491)
at ai.smash.endurance.services.APIDemo.parseFacebookPage(APIDemo.java:184)
at ai.smash.endurance.services.APIDemo.main(APIDemo.java:138)
Caused by: org.openqa.selenium.net.UrlChecker$TimeoutException: Timed out waiting for [http://localhost:18318/status] to be available after 20004 ms
at org.openqa.selenium.net.UrlChecker.waitUntilAvailable(UrlChecker.java:100)
at org.openqa.selenium.remote.service.DriverService.waitUntilAvailable(DriverService.java:188)
... 11 more
Caused by: java.util.concurrent.TimeoutException
at java.util.concurrent.FutureTask.get(FutureTask.java:205)
at com.google.common.util.concurrent.SimpleTimeLimiter.callWithTimeout(SimpleTimeLimiter.java:156)
at org.openqa.selenium.net.UrlChecker.waitUntilAvailable(UrlChecker.java:75)
... 12 more
Aug 31, 2018 3:24:11 AM org.openqa.selenium.phantomjs.PhantomJSDriverService <init>
INFO: executable: /phantomjs
Aug 31, 2018 3:24:11 AM org.openqa.selenium.phantomjs.PhantomJSDriverService <init>
INFO: port: 25411
Aug 31, 2018 3:24:11 AM org.openqa.selenium.phantomjs.PhantomJSDriverService <init>
INFO: arguments: [--webdriver=25411, --webdriver-logfile=/phantomjsdriver.log]
Aug 31, 2018 3:24:11 AM org.openqa.selenium.phantomjs.PhantomJSDriverService <init>
INFO: environment: {}
/phantomjs: line 1: syntax error near unexpected token `newline'
/phantomjs: line 1: `<!doctype html>'
^[[1;5BAug 31, 2018 3:24:31 AM org.openqa.selenium.os.OsProcess checkForError
SEVERE: org.apache.commons.exec.ExecuteException: Process exited with an error: 2 (Exit value: 2)
Successfully ran jar file. Saving results to S3
Deleting finished message
Counted number of messages in queue: 0
Working...
Getting message from queue
Traceback (most recent call last):
File "wrapper.py", line 39, in <module>
work()
File "wrapper.py", line 22, in work
msg = sqs.receive_message(QueueUrl=os.getenv('QUEUE'), WaitTimeSeconds=20, MaxNumberOfMessages=1, VisibilityTimeout=120)['Messages'][0]
KeyError: 'Messages'
[root@ip-172-31-24-37 /]#
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment