Created
March 29, 2021 19:08
-
-
Save aktech/97b28fbfa2e77be045b21fc54ee06e9a to your computer and use it in GitHub Desktop.
coiled-scheduler-logs
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| [ 160.964880] cloud-init[1391]: ng dask ... extracting (42.7%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (42.8%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (43.6%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (44.3%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (45.1%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (45.8%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (46.4%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (47.1%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (47.9%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (48.9%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (49.4%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (49.8%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (50.4%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (50.9%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (51.4%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (51.7%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (52.2%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (52.7%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (53.1%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (53.4%) | |
| [2021-03-29T18:33:29.039179][1B[1A[2K | |
| Pulling dask ... extracting (54.1%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (54.7%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (55.3%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (56.3%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (57.2%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (58.1%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (59.1%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (59.8%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (60.8%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (61.7%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (62.3%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (63.1%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (63.7%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (64.3%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (64.8%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (65.3%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (65.7%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (66.2%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (66.6%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (66.9%) | |
| [1B[1A[2K | |
| Pu[2021-03-29T18:33:29.039203]lling dask ... extracting (67.5%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (68.0%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (68.3%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (68.6%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (68.9%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (69.2%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (69.5%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (69.9%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (70.0%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (70.2%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (70.3%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (70.4%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (70.6%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (70.7%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (71.1%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (71.4%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (71.8%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (72.0%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (72.2%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (72.3%) | |
| [1B[1A[2K | |
| Pulling dask ... [2021-03-29T18:33:29.039239]extracting (72.5%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (72.7%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (73.0%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (73.3%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (73.5%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (73.7%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (73.8%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (74.0%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (74.2%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (74.3%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (74.5%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (74.6%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (74.8%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (75.0%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (75.2%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (75.4%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (75.6%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (75.7%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (75.9%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (76.0%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (76.[2021-03-29T18:33:29.039263]3%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (76.6%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (76.9%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (77.3%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (77.8%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (78.2%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (78.5%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (78.8%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (79.2%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (79.4%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (79.6%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (79.7%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (79.9%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (80.0%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (80.3%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (80.5%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (80.6%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (80.8%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (81.0%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (81.2%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (81.4%) | |
| [1B[1A[2[2021-03-29T18:33:29.039283]K | |
| Pulling dask ... extracting (81.7%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (81.9%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (82.2%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (82.6%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (82.9%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (83.1%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (83.3%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (83.4%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (83.6%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (83.8%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (84.0%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (84.2%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (84.3%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (84.4%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (84.6%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (84.9%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (85.3%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (85.7%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (86.0%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (86.3%) | |
| [1B[1A[2K | |
| Pulling dask [2021-03-29T18:33:29.139169]... extracting (86.7%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (87.1%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (87.3%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (87.5%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (87.8%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (88.1%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (88.4%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (88.6%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (88.9%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (89.1%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (89.3%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (89.5%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (89.8%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (90.2%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (90.5%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (90.8%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (90.9%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (91.1%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (91.4%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (91.6%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting [2021-03-29T18:33:29.139212](91.7%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (91.7%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (91.9%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (92.0%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (92.1%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (92.2%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (92.3%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (92.3%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (92.4%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (92.5%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (92.6%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (92.7%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (92.8%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (93.0%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (93.1%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (93.4%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (93.6%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (93.7%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (93.7%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (93.9%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (94.2%) | |
| [1B[1[2021-03-29T18:33:29.139235]A[2K | |
| Pulling dask ... extracting (94.4%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (94.6%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (95.0%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (95.1%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (95.3%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (95.4%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (95.6%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (95.7%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (95.9%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (96.2%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (96.6%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (96.9%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (97.2%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (97.3%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (97.3%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (97.4%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (97.4%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (97.6%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (97.7%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (97.7%) | |
| [1B[1A[2K | |
| Pulling d[2021-03-29T18:33:29.139255]ask ... extracting (97.8%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (97.9%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (98.0%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (98.5%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (98.8%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (99.1%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (99.6%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (99.6%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (99.8%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (100.0%) | |
| [1B[1A[2K | |
| Pulling dask ... pull complete | |
| [1B[1A[2K | |
| Pulling dask ... extracting (100.0%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (100.0%) | |
| [1B[1A[2K | |
| Pulling dask ... pull complete | |
| [1B[1A[2K | |
| Pulling dask ... extracting (11.9%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (100.0%) | |
| [1B[1A[2K | |
| Pulling dask ... extracting (100.0%) | |
| [1B[1A[2K | |
| Pulling dask ... pull complete | |
| [1B[1A[2K | |
| Pulling dask ... digest: sha256:9c44f783992303c3a4... | |
| [1B[1A[2K | |
| Pulling dask ... status: downloaded newer image fo... | |
| [1B[1A[2[2021-03-29T18:33:29.139274]K | |
| Pulling dask ... [32mdone[0m | |
| [1BRecreating tmp_dask_1 ... | |
| [ 165.792451] cloud-init[1391]: [1A[2K | |
| Recreating tmp_dask_1 ... [32mdone[0m | |
| [1BAttaching to tmp_dask_1 | |
| [ 165.793580] cloud-init[1391]: [36mdask_1 |[0m distributed.scheduler - INFO - Clear task state | |
| [ 165.798830] cloud-init[1391]: [36mdask_1 |[0m distributed.scheduler - INFO - Scheduler at: tls://10.0.0.134:8786 | |
| [ 165.800369] cloud-init[1391]: [36mdask_1 |[0m distributed.scheduler - INFO - dashboard at: :8787 | |
| [ 165.802028] cloud-init[1391]: [36mdask_1 |[0m distributed.preloading - INFO - Downloading preload at https://cloud.coiled.io/preloads/insights.py | |
| [ 165.944466] cloud-init[1391]: [36mdask_1 |[0m distributed.preloading - INFO - Run preload setup function: https://cloud.coiled.io/preloads/insights.py | |
| [ 165.946262] cloud-init[1391]: [36mdask_1 |[0m distributed.preloading - INFO - Downloading preload at https://cloud.coiled.io/preloads/aws-credentials.py | |
| [ 166.007134] cloud-init[1391]: [36mdask_1 |[0m distributed.preloading - INFO - Run preload setup function: https://cloud.coiled.io/preloads/aws-credentials.py | |
| [ 166.009351] cloud-init[1391]: [36mdask_1 |[0m distributed.scheduler - INFO - Connecting to coiled | |
| [ 166.148337] cloud-init[1391]: [36mdask_1 |[0m distributed.scheduler - INFO - Registering scheduler to coiled | |
| [ 166.654268] cloud-init[1391]: [36mdask_1 |[0m distributed.scheduler - INFO - Registered successfully to coiled. | |
| [ 169.544874] cloud-init[1391]: [36mdask_1 |[0m distributed.comm.tcp - INFO - Connection closed before handshake completed | |
| [ 205.666975] cloud-init[1391]: [36mdask_1 |[0m distributed.scheduler - INFO - Register worker <Worker 'tls://10.0.1.91:45717', name: coiled-dask-8936-worker-bc747e, memory: 0, processing: 0> | |
| [ 205.670085] cloud-init[1391]: [36mdask_1 |[0m distributed.scheduler - INFO - Starting worker compute stream, tls://10.0.1.91:45717 | |
| [ 205.671346] cloud-init[1391]: [36mdask_1 |[0m distributed.core - INFO - Starting established connection | |
| [ 207.397169] cloud-init[1391]: [36mdask_1 |[0m distributed.scheduler - INFO - Register worker <Worker 'tls://10.0.1.22:45743', name: coiled-dask-8936-worker-3668b2, memory: 0, processing: 0> | |
| [ 207.399421] cloud-init[1391]: [36mdask_1 |[0m distributed.scheduler - INFO - Starting worker compute stream, tls://10.0.1.22:45743 | |
| [ 207.399976] cloud-init[1391]: [36mdask_1 |[0m distributed.core - INFO - Starting established connection | |
| [ 1365.455156] cloud-init[1391]: [36mdask_1 |[0m distributed.scheduler - INFO - Scheduler closing after being idle for 1200.00 s | |
| [ 1365.460812] cloud-init[1391]: [36mdask_1 |[0m distributed.scheduler - INFO - Scheduler closing... | |
| [ 1365.463667] cloud-init[1391]: [36mdask_1 |[0m distributed.scheduler - INFO - Closing websocket connection to coiled | |
| [ 1365.466205] cloud-init[1391]: [36mdask_1 |[0m distributed.scheduler - INFO - Scheduler closing all comms | |
| [ 1365.471384] cloud-init[1391]: [36mdask_1 |[0m distributed.scheduler - INFO - Remove worker <Worker 'tls://10.0.1.91:45717', name: coiled-dask-8936-worker-bc747e, memory: 0, processing: 0> | |
| [ 1365.477897] cloud-init[1391]: [36mdask_1 |[0m distributed.core - INFO - Removing comms to tls://10.0.1.91:45717 | |
| [ 1365.483295] cloud-init[1391]: [36mdask_1 |[0m distributed.scheduler - ERROR - 'NoneType' object has no attribute 'write_message' | |
| [ 1365.486087] cloud-init[1391]: [36mdask_1 |[0m Traceback (most recent call last): | |
| [ 1365.487739] cloud-init[1391]: [36mdask_1 |[0m File "/opt/conda/envs/coiled/lib/python3.8/site-packages/distributed/scheduler.py", line 4409, in remove_worker | |
| [ 1365.487875] cloud-init[1391]: [36mdask_1 |[0m await result | |
| [ 1365.488005] cloud-init[1391]: [36mdask_1 |[0m File "https://cloud.coiled.io/preloads/insights.py", line 170, in remove_worker | |
| [ 1365.488159] cloud-init[1391]: [36mdask_1 |[0m File "/opt/conda/envs/coiled/lib/python3.8/site-packages/tornado/websocket.py", line 1508, in write_message | |
| [ 1365.488320] cloud-init[1391]: [36mdask_1 |[0m return self.protocol.write_message(message, binary=binary) | |
| [ 1365.488494] cloud-init[1391]: [36mdask_1 |[0m AttributeError: 'NoneType' object has no attribute 'write_message' | |
| [ 1365.921810] cloud-init[1391]: [36mtmp_dask_1 exited with code 0 | |
| [ 1366.288922] cloud-init[1391]: [0mCloud-init v. 20.3-2-g371b392c-0ubuntu1~20.04.1 running 'modules:final' at Mon, 29 Mar 2021 18:31:49 +0000. Up 61.71 seconds. | |
| [[0;32m OK [0m] Stopped [0;1;39mInitial cloud-init…ob (metadata service crawler)[0m. | |
| [[0;32m OK [0m] Stopped [0;1;39mInitial cloud-init job (pre-networking)[0m. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment