Skip to content

Instantly share code, notes, and snippets.

@rickychilcott

rickychilcott/apify.log

Created March 5, 2019 19:51
Show Gist options
  • Save rickychilcott/e7978fcc11e5a52f3d263de10f651b2b to your computer and use it in GitHub Desktop.
Save rickychilcott/e7978fcc11e5a52f3d263de10f651b2b to your computer and use it in GitHub Desktop.
Download ZIP
Apify Issue with Thirty One
Raw
apify.log
[2019-03-05 19:48:59.998: EXECUTOR] Starting crawler (actId: Te8MHFMj393SFQZ2c, actExecutionId: 7zxGNNb2qHYucQKeE)
[2019-03-05 19:49:00.038: EXECUTOR] DEBUG: CrawlerExecutor._spawnSlave(): isBootstrapper=true
[2019-03-05 19:49:00.060: EXECUTOR] Slave process spawned (slaveId: 1, proxy: default)
[2019-03-05 19:49:00.128: S0000001] Loading crawler configuration from: /tmp/actExec_7zxGNNb2qHYucQKeE_1857wF1jZYDQFJ3Q/config.json
[2019-03-05 19:49:00.129: S0000001] DEBUG: crawlerUtils.prepareConfig()
[2019-03-05 19:49:00.130: S0000001] Starting crawler using RemoteRequestManager (URL: http://localhost:36203/slave/1, bootstrap: true)...
[2019-03-05 19:49:00.132: S0000001] DEBUG: Scheduling periodic PING to server
[2019-03-05 19:49:00.134: S0000001] DEBUG: ON URL CHANGED | targetUrl:
[2019-03-05 19:49:00.136: S0000001] DEBUG: ON LOAD STARTED
[2019-03-05 19:49:00.137: S0000001] ON LOAD FINISHED | status: success, url: N/A
[2019-03-05 19:49:00.138: S0000001] DEBUG: CrawlerUtils.injectClientUtils(): injected client-side utilities to page
[2019-03-05 19:49:00.151: S0000001] DEBUG: CrawlerUtils.injectJQuery(): injected jQuery to page
[2019-03-05 19:49:00.151: S0000001] DEBUG: Initial about:blank page was loaded
[2019-03-05 19:49:00.152: S0000001] DEBUG: Crawler.onNavigationRequested({"type":"StartUrl","isMainFrame":true,"contentType":"","label":"product","url":"https://www.mythirtyone.com/us/en/collection/storage-organization?page=4","postData":"","method":"GET"})
[2019-03-05 19:49:00.153: S0000001] DEBUG: Crawler.onNavigationRequested(): Invoking user-provided 'interceptRequest' function.
[2019-03-05 19:49:00.154: S0000001] DEBUG: Crawler.onNavigationRequested(): User-provided 'interceptRequest' function returned a result:
{
"url": "https://www.mythirtyone.com/us/en/collection/storage-organization?page=4",
"uniqueKey": "https://www.mythirtyone.com/us/en/collection/storage-organization?page=4",
"label": "product",
"willLoad": true,
"method": "GET",
"postData": null,
"contentType": null,
"queuePosition": "LAST"
}
[2019-03-05 19:49:00.154: S0000001] DEBUG: Captured new request ({null:https://www.mythirtyone.com/us/en/collection/storage-organization?page=4})
[2019-03-05 19:49:00.155: S0000001] DEBUG: Crawler.handleNextRequest()
[2019-03-05 19:49:00.155: S0000001] DEBUG: RemoteRequestManager.fetchNextRequest()
[2019-03-05 19:49:00.155: S0000001] DEBUG: RemoteRequestManager._enqueueMessage(): messageType=fetchNextRequest
[2019-03-05 19:49:00.155: S0000001] DEBUG: RemoteRequestManager._sendNextMessage(): webPageIsBusy=false, enqueuedMessages.length=1
[2019-03-05 19:49:00.159: EXECUTOR] DEBUG: Received message from slave (URL: /slave/1, messageType: fetchNextRequest)
[2019-03-05 19:49:00.234: EXECUTOR] Add new request ({xcm7aYkiWKhKO9h:https://www.mythirtyone.com/us/en/collection/storage-organization?page=4}): page enqueued (url: https://www.mythirtyone.com/us/en/collection/storage-organization?page=4, label: "product")
[2019-03-05 19:49:00.243: EXECUTOR] Bootstrapping is finished (the bootstrapping slave process invoked 'fetchNextRequest')
[2019-03-05 19:49:00.250: EXECUTOR] DEBUG: PageManager.fetchNextRequest() results: requestId=xcm7aYkiWKhKO9h, statusMessage='null'
[2019-03-05 19:49:00.268: S0000001] DEBUG: RemoteRequestManager._sendNextMessage(): message sent (messageType=fetchNextRequest, status=success)
[2019-03-05 19:49:00.269: S0000001] DEBUG: RemoteRequestManager.fetchNextRequest() results: request={xcm7aYkiWKhKO9h:https://www.mythirtyone.com/us/en/collection/storage-organization?page=4}, statusMessage='Request was fetched successfully', requestsFetchedCount='1'
[2019-03-05 19:49:00.269: S0000001] DEBUG: Waiting 0 ms before loading next page
[2019-03-05 19:49:00.269: S0000001] DEBUG: RemoteRequestManager._sendNextMessage(): webPageIsBusy=false, enqueuedMessages.length=0
[2019-03-05 19:49:00.273: S0000001] OPEN | https://www.mythirtyone.com/us/en/collection/storage-organization?page=4
[2019-03-05 19:49:00.273: S0000001] DEBUG: Crawler.onNavigationRequested({"url":"https://www.mythirtyone.com/us/en/collection/storage-organization?page=4","type":"Other","isMainFrame":true,"postData":"","contentType":"","method":""}): Page requested by the crawler, it will open.
[2019-03-05 19:49:00.274: S0000001] DEBUG: ON LOAD STARTED
[2019-03-05 19:49:00.732: S0000001] DEBUG: ON URL CHANGED | targetUrl: https://www.mythirtyone.com/us/en/collection/storage-organization?page=4
[2019-03-05 19:49:00.970: S0000001] ON CONSOLE MESSAGE | configuring require
[2019-03-05 19:49:00.970: S0000001] ON CONSOLE MESSAGE | cache busting with lng=12&bust=201903050503337139
[2019-03-05 19:49:00.971: S0000001] ON CONSOLE MESSAGE | after config
[2019-03-05 19:49:00.973: S0000001] DEBUG: Crawler.onError(): JavaScript on page threw an exception | msg: ReferenceError: Can't find variable: jQuery, trace:
-> https://www.mythirtyone.com/us/en/collection/storage-organization?page=4: 270 (in function global code)
[2019-03-05 19:49:00.979: S0000001] DEBUG: Crawler.onError(): JavaScript on page threw an exception | msg: ReferenceError: Can't find variable: Map, trace:
-> https://apps.bazaarvoice.com/deployments/thirtyonegifts/main_site/production/en_US/bv.js: 19
-> https://apps.bazaarvoice.com/deployments/thirtyonegifts/main_site/production/en_US/bv.js: 19
-> https://apps.bazaarvoice.com/deployments/thirtyonegifts/main_site/production/en_US/bv.js: 19 (in function n)
-> https://apps.bazaarvoice.com/deployments/thirtyonegifts/main_site/production/en_US/bv.js: 19
-> https://apps.bazaarvoice.com/deployments/thirtyonegifts/main_site/production/en_US/bv.js: 19
-> https://apps.bazaarvoice.com/deployments/thirtyonegifts/main_site/production/en_US/bv.js: 19 (in function n)
-> https://apps.bazaarvoice.com/deployments/thirtyonegifts/main_site/production/en_US/bv.js: 19
-> https://apps.bazaarvoice.com/deployments/thirtyonegifts/main_site/production/en_US/bv.js: 19
-> https://apps.bazaarvoice.com/deployments/thirtyonegifts/main_site/production/en_US/bv.js: 19 (in function n)
-> https://apps.bazaarvoice.com/deployments/thirtyonegifts/main_site/production/en_US/bv.js: 19
-> https://apps.bazaarvoice.com/deployments/thirtyonegifts/main_site/production/en_US/bv.js: 19 (in function n)
-> https://apps.bazaarvoice.com/deployments/thirtyonegifts/main_site/production/en_US/bv.js: 19
-> https://apps.bazaarvoice.com/deployments/thirtyonegifts/main_site/production/en_US/bv.js: 19 (in function n)
-> https://apps.bazaarvoice.com/deployments/thirtyonegifts/main_site/production/en_US/bv.js: 19
-> https://apps.bazaarvoice.com/deployments/thirtyonegifts/main_site/production/en_US/bv.js: 19 (in function n)
-> https://apps.bazaarvoice.com/deployments/thirtyonegifts/main_site/production/en_US/bv.js: 19
-> https://apps.bazaarvoice.com/deployments/thirtyonegifts/main_site/production/en_US/bv.js: 19
-> https://apps.bazaarvoice.com/deployments/thirtyonegifts/main_site/production/en_US/bv.js: 19 (in function n)
-> https://apps.bazaarvoice.com/deployments/thirtyonegifts/main_site/production/en_US/bv.js: 19
[2019-03-05 19:49:01.001: S0000001] DEBUG: Crawler.onNavigationRequested({"url":"about:blank","type":"Other","isMainFrame":false,"postData":"","contentType":"","method":""}): Page requested by the crawler, it will open.
[2019-03-05 19:49:01.079: S0000001] DEBUG: Crawler.onNavigationRequested({"url":"https://assets.adobedtm.com/48ec9783bdd2727fc8717dd5e1b50a591f825c7f/scripts/satellite-594aa5be64746d3fce00ca07.html","type":"Other","isMainFrame":false,"postData":"","contentType":"","method":""}): Page requested by the crawler, it will open.
[2019-03-05 19:49:01.081: S0000001] DEBUG: Crawler.onError(): JavaScript on page threw an exception | msg: TypeError: undefined is not an object (evaluating 'document.getElementsByTagName('meta')['description']'), trace:
-> https://assets.adobedtm.com/48ec9783bdd2727fc8717dd5e1b50a591f825c7f/scripts/satellite-5ade02de64746d5f7e0091b4.js: 3
-> https://assets.adobedtm.com/48ec9783bdd2727fc8717dd5e1b50a591f825c7f/satelliteLib-c495c1f9d31ccd9456dda9b250fa1b93b7153967.js: 3 (in function i)
-> https://assets.adobedtm.com/48ec9783bdd2727fc8717dd5e1b50a591f825c7f/satelliteLib-c495c1f9d31ccd9456dda9b250fa1b93b7153967.js: 2 (in function i)
-> https://assets.adobedtm.com/48ec9783bdd2727fc8717dd5e1b50a591f825c7f/satelliteLib-c495c1f9d31ccd9456dda9b250fa1b93b7153967.js: 2 (in function onload)
[2019-03-05 19:49:01.186: S0000001] DEBUG: Crawler.onError(): JavaScript on page threw an exception | msg: TypeError: undefined is not an object (evaluating 'indexModule.setup'), trace:
-> https://www.mythirtyone.com/Scripts/require.js?lng=12&bust=201903050503337139: 900 (in function check)
[2019-03-05 19:49:01.389: S0000001] ON RESOURCE ERROR | resourceError: {"errorCode":5,"errorString":"Operation canceled","id":76,"status":403,"statusText":"Forbidden","url":"https://d16bpg3kvlhleg.cloudfront.net/pp/js/moment-with-locales.min.js?lng=12&bust=201903050503337139"}
[2019-03-05 19:49:01.613: S0000001] DEBUG: Crawler.onNavigationRequested({"url":"https://assets.adobedtm.com/48ec9783bdd2727fc8717dd5e1b50a591f825c7f/scripts/satellite-5a67af4964746d7cf100181c.html","type":"Other","isMainFrame":false,"postData":"","contentType":"","method":""}): Page requested by the crawler, it will open.
[2019-03-05 19:49:01.614: S0000001] DEBUG: Crawler.onNavigationRequested({"url":"https://assets.adobedtm.com/48ec9783bdd2727fc8717dd5e1b50a591f825c7f/scripts/satellite-5ab50c2864746d4a0200103f.html","type":"Other","isMainFrame":false,"postData":"","contentType":"","method":""}): Page requested by the crawler, it will open.
[2019-03-05 19:49:01.642: S0000001] ON CONSOLE MESSAGE | fancyBox3 global loaded!
[2019-03-05 19:49:01.673: S0000001] ON CONSOLE MESSAGE | [object HTMLImageElement] has been loaded
[2019-03-05 19:49:01.674: S0000001] ON CONSOLE MESSAGE | [object HTMLImageElement] has been loaded
[2019-03-05 19:49:01.674: S0000001] ON CONSOLE MESSAGE | [object HTMLImageElement] has been loaded
[2019-03-05 19:49:01.674: S0000001] ON CONSOLE MESSAGE | [object HTMLImageElement] has been loaded
[2019-03-05 19:49:01.674: S0000001] ON CONSOLE MESSAGE | [object HTMLImageElement] has been loaded
[2019-03-05 19:49:01.674: S0000001] ON CONSOLE MESSAGE | [object HTMLImageElement] has been loaded
[2019-03-05 19:49:01.675: S0000001] ON CONSOLE MESSAGE | Loaded Orders
[2019-03-05 19:49:01.704: S0000001] ON CONSOLE MESSAGE | back to top loaded
[2019-03-05 19:49:01.746: S0000001] ON LOAD FINISHED | status: success, url: https://www.mythirtyone.com/us/en/collection/storage-organization?page=4
[2019-03-05 19:49:01.747: S0000001] DEBUG: CrawlerUtils.injectClientUtils(): injected client-side utilities to page
[2019-03-05 19:49:01.752: S0000001] DEBUG: CrawlerUtils.injectJQuery(): injected jQuery to page
[2019-03-05 19:49:01.753: S0000001] DEBUG: Crawler.invokePageFunction()
[2019-03-05 19:49:01.753: S0000001] Capturing snapshots to: screenshot_2019-03-05T19-49-01.753_reqxcm7aYkiWKhKO9h.(png|html)
[2019-03-05 19:49:02.578: S0000001] DEBUG: RemoteRequestManager.saveSnapshot()
[2019-03-05 19:49:02.578: S0000001] DEBUG: RemoteRequestManager._enqueueMessage(): messageType=saveSnapshot
[2019-03-05 19:49:02.579: S0000001] DEBUG: RemoteRequestManager._sendNextMessage(): webPageIsBusy=false, enqueuedMessages.length=1
[2019-03-05 19:49:02.581: S0000001] DEBUG: Crawler.invokePageFunction(): Invoking user-provided 'pageFunction'.
[2019-03-05 19:49:02.582: S0000001] Page function will asynchronously finish later (if the crawler hangs here, make sure context.finish() is really called in pageFunction!).
[2019-03-05 19:49:02.648: EXECUTOR] DEBUG: Received message from slave (URL: /slave/1, messageType: saveSnapshot)
[2019-03-05 19:49:02.649: EXECUTOR] DEBUG: Received crawler snapshots in files (screenshot: screenshot_2019-03-05T19-49-01.753_reqxcm7aYkiWKhKO9h.png, HTML: undefined)
[2019-03-05 19:49:02.651: S0000001] DEBUG: RemoteRequestManager._sendNextMessage(): message sent (messageType=saveSnapshot, status=success)
[2019-03-05 19:49:02.651: S0000001] DEBUG: RemoteRequestManager._sendNextMessage(): webPageIsBusy=false, enqueuedMessages.length=0
[2019-03-05 19:49:07.587: S0000001] ON CONSOLE MESSAGE | done waiting
[2019-03-05 19:49:07.588: S0000001] DEBUG: Saving snapshot (customName: 'N/A')
[2019-03-05 19:49:07.589: S0000001] Capturing snapshots to: screenshot_2019-03-05T19-49-07.589_reqxcm7aYkiWKhKO9h.(png|html)
[2019-03-05 19:49:08.731: S0000001] DEBUG: RemoteRequestManager.saveSnapshot()
[2019-03-05 19:49:08.731: S0000001] DEBUG: RemoteRequestManager._enqueueMessage(): messageType=saveSnapshot
[2019-03-05 19:49:08.732: S0000001] DEBUG: RemoteRequestManager._sendNextMessage(): webPageIsBusy=false, enqueuedMessages.length=1
[2019-03-05 19:49:08.740: EXECUTOR] DEBUG: Received message from slave (URL: /slave/1, messageType: saveSnapshot)
[2019-03-05 19:49:08.741: EXECUTOR] DEBUG: Received crawler snapshots in files (screenshot: screenshot_2019-03-05T19-49-07.589_reqxcm7aYkiWKhKO9h.png, HTML: undefined)
[2019-03-05 19:49:08.743: S0000001] DEBUG: RemoteRequestManager._sendNextMessage(): message sent (messageType=saveSnapshot, status=success)
[2019-03-05 19:49:08.743: S0000001] DEBUG: RemoteRequestManager._sendNextMessage(): webPageIsBusy=false, enqueuedMessages.length=0
[2019-03-05 19:49:10.132: S0000001] DEBUG: Sending periodic PING to server
[2019-03-05 19:49:10.133: S0000001] DEBUG: RemoteRequestManager._sendBufferedRequests(): length=0
[2019-03-05 19:49:10.133: S0000001] DEBUG: RemoteRequestManager._enqueueMessage(): messageType=dummy
[2019-03-05 19:49:10.133: S0000001] DEBUG: RemoteRequestManager._sendNextMessage(): webPageIsBusy=false, enqueuedMessages.length=1
[2019-03-05 19:49:10.173: EXECUTOR] DEBUG: Received message from slave (URL: /slave/1, messageType: dummy)
[2019-03-05 19:49:10.176: S0000001] DEBUG: RemoteRequestManager._sendNextMessage(): message sent (messageType=dummy, status=success)
[2019-03-05 19:49:10.177: S0000001] DEBUG: RemoteRequestManager._sendNextMessage(): webPageIsBusy=false, enqueuedMessages.length=0
[2019-03-05 19:49:20.132: S0000001] DEBUG: Sending periodic PING to server
[2019-03-05 19:49:20.132: S0000001] DEBUG: RemoteRequestManager._sendBufferedRequests(): length=0
[2019-03-05 19:49:20.132: S0000001] DEBUG: RemoteRequestManager._enqueueMessage(): messageType=dummy
[2019-03-05 19:49:20.132: S0000001] DEBUG: RemoteRequestManager._sendNextMessage(): webPageIsBusy=false, enqueuedMessages.length=1
[2019-03-05 19:49:20.134: EXECUTOR] DEBUG: Received message from slave (URL: /slave/1, messageType: dummy)
[2019-03-05 19:49:20.136: S0000001] DEBUG: RemoteRequestManager._sendNextMessage(): message sent (messageType=dummy, status=success)
[2019-03-05 19:49:20.136: S0000001] DEBUG: RemoteRequestManager._sendNextMessage(): webPageIsBusy=false, enqueuedMessages.length=0
[2019-03-05 19:49:30.132: S0000001] DEBUG: Sending periodic PING to server
[2019-03-05 19:49:30.133: S0000001] DEBUG: RemoteRequestManager._sendBufferedRequests(): length=0
[2019-03-05 19:49:30.133: S0000001] DEBUG: RemoteRequestManager._enqueueMessage(): messageType=dummy
[2019-03-05 19:49:30.133: S0000001] DEBUG: RemoteRequestManager._sendNextMessage(): webPageIsBusy=false, enqueuedMessages.length=1
[2019-03-05 19:49:30.134: EXECUTOR] DEBUG: Received message from slave (URL: /slave/1, messageType: dummy)
[2019-03-05 19:49:30.136: S0000001] DEBUG: RemoteRequestManager._sendNextMessage(): message sent (messageType=dummy, status=success)
[2019-03-05 19:49:30.136: S0000001] DEBUG: RemoteRequestManager._sendNextMessage(): webPageIsBusy=false, enqueuedMessages.length=0
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment