Skip to content

Instantly share code, notes, and snippets.

@edraizen

edraizen/worker_cachingFileStoreError.out

Created December 29, 2021 22:40
Show Gist options
  • Save edraizen/84e45cc72fd004cb1b12f56dad9e0bf4 to your computer and use it in GitHub Desktop.
Save edraizen/84e45cc72fd004cb1b12f56dad9e0bf4 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
worker_cachingFileStoreError.out
INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
INFO:toil:Running Toil version 5.6.0a1-458a26c04268e57ebace072e1d64b6a087e99e51-dirty on host default-toil-c49a786e-6899-4cc4-ae90-f49e2c9cfaaf-0-j5pbg.
DEBUG:toil:Configuration: {'workflowID': 'be4ac964-d6a0-4efa-84e3-220a4b14a690', 'workflowAttemptNumber': 0, 'jobStore': 's3hsds:us-east-1:/home/ed4bu/hello-world', 'logLevel': 'DEBUG', 'workDir': None, 'noStdOutErr': False, 'stats': False, 'clean': 'always', 'clusterStats': None, 'restart': False, 'batchSystem': 'kubernetes', 'disableAutoDeployment': False, 'maxLocalJobs': 2, 'manualMemArgs': False, 'coalesceStatusCalls': False, 'statePollingWait': None, 'parasolCommand': 'parasol', 'parasolMaxBatches': 10000, 'scale': 1.0, 'mesos_endpoint': '10.42.0.155:5050', 'allocate_mem': False, 'tes_endpoint': 'http://10.42.0.155:8000', 'tes_user': None, 'tes_password': None, 'tes_bearer_token': None, 'kubernetes_host_path': '/media/smb-rivanna/ed4bu/UrfoldServer/urfold_runs/cath_pipeline/h5_restructure/k8/', 'kubernetes_owner': 'default', 'runCwlInternalJobsOnWorkers': False, 'disableCaching': False, 'linkImports': True, 'moveExports': False, 'provisioner': None, 'nodeTypes': [], 'minNodes': None, 'maxNodes': [10], 'targetTime': 1800, 'betaInertia': 0.1, 'scaleInterval': 60, 'preemptableCompensation': 0.0, 'nodeStorage': 50, 'nodeStorageOverrides': [], 'metrics': False, 'maxPreemptableServiceJobs': 9223372036854775807, 'maxServiceJobs': 9223372036854775807, 'deadlockWait': 60, 'deadlockCheckInterval': 30, 'defaultMemory': 2147483648, 'defaultCores': 1, 'defaultDisk': 2147483648, 'readGlobalFileMutableByDefault': False, 'defaultPreemptable': False, 'maxCores': 9223372036854775807, 'maxMemory': 9223372036854775807, 'maxDisk': 9223372036854775807, 'retryCount': 1, 'enableUnlimitedPreemptableRetries': False, 'doubleMem': False, 'maxJobDuration': 9223372036854775807, 'rescueJobsFrequency': 60, 'environment': {'TOIL_S3_HOST': 'minio.minio-tenant-1.svc.cluster.local', 'TOIL_S3_PORT': '80', 'TOIL_S3_USE_SSL': 'False', 'HS_ENDPOINT': 'http://hsds.default.svc.cluster.local:5101', 'HS_USERNAME': '[email protected]', 'HS_PASSWORD': '13e571b7f8e41419c63aa7a0f3a33e29'}, 'disableChaining': False, 'disableJobStoreChecksumVerification': False, 'maxLogFileSize': 64000, 'writeLogs': None, 'writeLogsGzip': None, 'writeLogsFromAllJobs': False, 'sseKey': None, 'servicePollingInterval': 60, 'useAsync': True, 'forceDockerAppliance': False, 'statusWait': 3600, 'disableProgress': False, 'debugWorker': False, 'disableWorkerOutputCapture': False, 'badWorker': 0.0, 'badWorkerFailInterval': 0.01, 'cwl': False, 'cleanWorkDir': 'always', 'toilS3Host': 'minio.minio-tenant-1.svc.cluster.local', 'toilS3Port': '80', 'toilS3UseSSL': 'False', 'hs_endpoint': 'http://hsds.default.svc.cluster.local:5101', 'hs_username': '[email protected]', 'hs_password': '13e571b7f8e41419c63aa7a0f3a33e29'}
DEBUG:toil.deferred:Running for file /tmp/scratch/cba7e1a4fdb051db80b0192033cfe42e/deferred/funcvp54z3g6
DEBUG:root:group.__getitem__(jobs)
DEBUG:root:__get_link_json(jobs)
DEBUG:root:getObjByUuid(d-e96f8bb6-63b15792-52a3-290639-28fe64)
DEBUG:root:use basic auth with username: [email protected]
DEBUG:root:GET: /datasets/d-e96f8bb6-63b15792-52a3-290639-28fe64 [/home/ed4bu/hello-world-jobstore.h5] bucket: None
INFO:root:GET: http://hsds.default.svc.cluster.local:5101/datasets/d-e96f8bb6-63b15792-52a3-290639-28fe64 [/home/ed4bu/hello-world-jobstore.h5]
INFO:root:status: 200
DEBUG:root:rsp_json: {'id': 'd-e96f8bb6-63b15792-52a3-290639-28fe64', 'root': 'g-e96f8bb6-63b15792-61e7-033eeb-39df1a', 'shape': {'class': 'H5S_SIMPLE', 'dims': [1], 'maxdims': [0]}, 'type': {'class': 'H5T_COMPOUND', 'fields': [{'name': 'job_id', 'type': {'class': 'H5T_STRING', 'length': 'H5T_VARIABLE', 'charSet': 'H5T_CSET_UTF8', 'strPad': 'H5T_STR_NULLTERM'}}, {'name': 'overlargeID', 'type': {'class': 'H5T_STRING', 'length': 'H5T_VARIABLE', 'charSet': 'H5T_CSET_UTF8', 'strPad': 'H5T_STR_NULLTERM'}}, {'name': 'numChunks', 'type': {'class': 'H5T_INTEGER', 'base': 'H5T_STD_I32LE'}}, {'name': 'chunkName', 'type': {'class': 'H5T_STRING', 'length': 'H5T_VARIABLE', 'charSet': 'H5T_CSET_UTF8', 'strPad': 'H5T_STR_NULLTERM'}}, {'name': 'chunkValue', 'type': {'class': 'H5T_STRING', 'length': 'H5T_VARIABLE', 'charSet': 'H5T_CSET_UTF8', 'strPad': 'H5T_STR_NULLTERM'}}]}, 'creationProperties': {'filters': [{'class': 'H5Z_FILTER_DEFLATE', 'id': 1, 'level': 9, 'name': 'gzip'}]}, 'layout': {'class': 'H5D_CHUNKED', 'dims': [8192]}, 'attributeCount': 0, 'created': 1640815506, 'lastModified': 1640815506, 'domain': '/home/ed4bu/hello-world-jobstore.h5', 'hrefs': [{'rel': 'self', 'href': 'http://hsds.hdf.test/datasets/d-e96f8bb6-63b15792-52a3-290639-28fe64?domain=/home/ed4bu/hello-world-jobstore.h5'}, {'rel': 'root', 'href': 'http://hsds.hdf.test/groups/g-e96f8bb6-63b15792-61e7-033eeb-39df1a?domain=/home/ed4bu/hello-world-jobstore.h5'}, {'rel': 'home', 'href': 'http://hsds.hdf.test/?domain=/home/ed4bu/hello-world-jobstore.h5'}, {'rel': 'attributes', 'href': 'http://hsds.hdf.test/datasets/d-e96f8bb6-63b15792-52a3-290639-28fe64/attributes?domain=/home/ed4bu/hello-world-jobstore.h5'}, {'rel': 'data', 'href': 'http://hsds.hdf.test/datasets/d-e96f8bb6-63b15792-52a3-290639-28fe64/value?domain=/home/ed4bu/hello-world-jobstore.h5'}]}
INFO:root:req - cursor: 0 page_size: 1
DEBUG:root:query param: [0:1]
DEBUG:root:params: {'query': "job_id=='9fc7c5e7-37cd-4806-b36e-fc95a4f42054'", 'select': '[0:1]'}
DEBUG:root:use basic auth with username: [email protected]
DEBUG:root:GET: /datasets/d-e96f8bb6-63b15792-52a3-290639-28fe64/value [/home/ed4bu/hello-world-jobstore.h5] bucket: None
INFO:root:GET: http://hsds.default.svc.cluster.local:5101/datasets/d-e96f8bb6-63b15792-52a3-290639-28fe64/value [/home/ed4bu/hello-world-jobstore.h5]
DEBUG:root:GET params query:job_id=='9fc7c5e7-37cd-4806-b36e-fc95a4f42054'
DEBUG:root:GET params select:[0:1]
INFO:root:status: 200
DEBUG:root:rsp_json: {'index': [0], 'value': [['9fc7c5e7-37cd-4806-b36e-fc95a4f42054', 0, 1, '000', 'Q0JaaDkxQVkmU1mimYD5AADo/+R33z4ARGP/6A05jIK///+iYASOEAAQMAFqQgaIKYmNQmnqPSPTU0eo2poxNDRp6aho0BsSMgap+ppNDUn6TGomI9E00BoGmgAZNGjJoaAOAYRhNMQwCAZADCNMmTCMBDRBlgwAxaTUKTAIbDLLo8NCthdCsGfFhOevPJ74snU4utkYjC/EoKmTlwz0I+Wo9tSKFznLTj/RpzCLtAe7RdnzrFIwZYShcVDzKFga1rD4dcsvtljk7O7N6PXjiuyfVGr2cJO+yrlEsBBlh9xQF+VQNlwN55oaJo7tvptodJ6Z5TS8cUKzr3w0kRH8BAULIOkTV2xNDGrEe4JVSHBVtU29V1GOvcQWYXY9dK1gyyCzeS70uUuw4yXmawN2abeyxteHSrQkUwUG3kIiS5PkRcESn50ZC1CAjCg0BuT5JgIKmi1S0kULCHCPhc/vIFV1z1s2o+ioRGKLTCsWJ1QRkhilLN83ppITVBhgYb4TYSHX9TMdIinc79eXnbgjqkf3l2DRdN4uaqmvUF2zqMl2kAzMyFKEAvwbadLOO5Heh0rUP1GxPltzJlSGkWUKDtYOB71/hAP8XckU4UJCimYD5A==']], 'hrefs': [{'rel': 'self', 'href': 'http://hsds.hdf.test/datasets/d-e96f8bb6-63b15792-52a3-290639-28fe64/value?domain=/home/ed4bu/hello-world-jobstore.h5'}, {'rel': 'root', 'href': 'http://hsds.hdf.test/groups/g-e96f8bb6-63b15792-61e7-033eeb-39df1a?domain=/home/ed4bu/hello-world-jobstore.h5'}, {'rel': 'home', 'href': 'http://hsds.hdf.test/?domain=/home/ed4bu/hello-world-jobstore.h5'}, {'rel': 'owner', 'href': 'http://hsds.hdf.test/datasets/d-e96f8bb6-63b15792-52a3-290639-28fe64?domain=/home/ed4bu/hello-world-jobstore.h5'}]}
INFO:root:got 1 rows
INFO:root:completed iteration, returning: 1 rows
INFO:toil.jobStores.hsdsJobStore:Loading JOB [('9fc7c5e7-37cd-4806-b36e-fc95a4f42054', 0, 1, '000', 'Q0JaaDkxQVkmU1mimYD5AADo/+R33z4ARGP/6A05jIK///+iYASOEAAQMAFqQgaIKYmNQmnqPSPTU0eo2poxNDRp6aho0BsSMgap+ppNDUn6TGomI9E00BoGmgAZNGjJoaAOAYRhNMQwCAZADCNMmTCMBDRBlgwAxaTUKTAIbDLLo8NCthdCsGfFhOevPJ74snU4utkYjC/EoKmTlwz0I+Wo9tSKFznLTj/RpzCLtAe7RdnzrFIwZYShcVDzKFga1rD4dcsvtljk7O7N6PXjiuyfVGr2cJO+yrlEsBBlh9xQF+VQNlwN55oaJo7tvptodJ6Z5TS8cUKzr3w0kRH8BAULIOkTV2xNDGrEe4JVSHBVtU29V1GOvcQWYXY9dK1gyyCzeS70uUuw4yXmawN2abeyxteHSrQkUwUG3kIiS5PkRcESn50ZC1CAjCg0BuT5JgIKmi1S0kULCHCPhc/vIFV1z1s2o+ioRGKLTCsWJ1QRkhilLN83ppITVBhgYb4TYSHX9TMdIinc79eXnbgjqkf3l2DRdN4uaqmvUF2zqMl2kAzMyFKEAvwbadLOO5Heh0rUP1GxPltzJlSGkWUKDtYOB71/hAP8XckU4UJCimYD5A==')]
DEBUG:root:group.__getitem__(jobs)
DEBUG:root:__get_link_json(jobs)
DEBUG:root:getObjByUuid(d-e96f8bb6-63b15792-52a3-290639-28fe64)
DEBUG:root:use basic auth with username: [email protected]
DEBUG:root:GET: /datasets/d-e96f8bb6-63b15792-52a3-290639-28fe64 [/home/ed4bu/hello-world-jobstore.h5] bucket: None
INFO:root:GET: http://hsds.default.svc.cluster.local:5101/datasets/d-e96f8bb6-63b15792-52a3-290639-28fe64 [/home/ed4bu/hello-world-jobstore.h5]
INFO:root:status: 200
DEBUG:root:rsp_json: {'id': 'd-e96f8bb6-63b15792-52a3-290639-28fe64', 'root': 'g-e96f8bb6-63b15792-61e7-033eeb-39df1a', 'shape': {'class': 'H5S_SIMPLE', 'dims': [1], 'maxdims': [0]}, 'type': {'class': 'H5T_COMPOUND', 'fields': [{'name': 'job_id', 'type': {'class': 'H5T_STRING', 'length': 'H5T_VARIABLE', 'charSet': 'H5T_CSET_UTF8', 'strPad': 'H5T_STR_NULLTERM'}}, {'name': 'overlargeID', 'type': {'class': 'H5T_STRING', 'length': 'H5T_VARIABLE', 'charSet': 'H5T_CSET_UTF8', 'strPad': 'H5T_STR_NULLTERM'}}, {'name': 'numChunks', 'type': {'class': 'H5T_INTEGER', 'base': 'H5T_STD_I32LE'}}, {'name': 'chunkName', 'type': {'class': 'H5T_STRING', 'length': 'H5T_VARIABLE', 'charSet': 'H5T_CSET_UTF8', 'strPad': 'H5T_STR_NULLTERM'}}, {'name': 'chunkValue', 'type': {'class': 'H5T_STRING', 'length': 'H5T_VARIABLE', 'charSet': 'H5T_CSET_UTF8', 'strPad': 'H5T_STR_NULLTERM'}}]}, 'creationProperties': {'filters': [{'class': 'H5Z_FILTER_DEFLATE', 'id': 1, 'level': 9, 'name': 'gzip'}]}, 'layout': {'class': 'H5D_CHUNKED', 'dims': [8192]}, 'attributeCount': 0, 'created': 1640815506, 'lastModified': 1640815506, 'domain': '/home/ed4bu/hello-world-jobstore.h5', 'hrefs': [{'rel': 'self', 'href': 'http://hsds.hdf.test/datasets/d-e96f8bb6-63b15792-52a3-290639-28fe64?domain=/home/ed4bu/hello-world-jobstore.h5'}, {'rel': 'root', 'href': 'http://hsds.hdf.test/groups/g-e96f8bb6-63b15792-61e7-033eeb-39df1a?domain=/home/ed4bu/hello-world-jobstore.h5'}, {'rel': 'home', 'href': 'http://hsds.hdf.test/?domain=/home/ed4bu/hello-world-jobstore.h5'}, {'rel': 'attributes', 'href': 'http://hsds.hdf.test/datasets/d-e96f8bb6-63b15792-52a3-290639-28fe64/attributes?domain=/home/ed4bu/hello-world-jobstore.h5'}, {'rel': 'data', 'href': 'http://hsds.hdf.test/datasets/d-e96f8bb6-63b15792-52a3-290639-28fe64/value?domain=/home/ed4bu/hello-world-jobstore.h5'}]}
DEBUG:root:dataset.__getitem__
DEBUG:root:arg: Ellipsis type: <class 'ellipsis'>
DEBUG:root:new_dtype: [('job_id', 'O'), ('overlargeID', 'O'), ('numChunks', '<i4'), ('chunkName', 'O'), ('chunkValue', 'O')]
DEBUG:root:selection_constructor
DEBUG:root:dataset shape: (1,)
DEBUG:root:mshape: (1,)
DEBUG:root:single_element: False
DEBUG:root:not settng nonstrict
DEBUG:root:selection._sel: ((0,), (1,), (1,), (False,))
INFO:root:selection: start (0,) stop [1] step (1,)
DEBUG:root:split_dim: 0
DEBUG:root:chunks_per_page: 1
DEBUG:root:mshape_split_dim: 0
DEBUG:root:chunk size for split_dim: 8192
DEBUG:root:num_rows: 8192
DEBUG:root:paged read, chunks_per_page: 1 max_chunks: 1, num_pages: 1
DEBUG:root:page_number: 0
DEBUG:root:start: [0] stop: [1]
INFO:root:page_stop: 1
INFO:root:page_mshape: (1,)
DEBUG:root:use basic auth with username: [email protected]
DEBUG:root:GET: /datasets/d-e96f8bb6-63b15792-52a3-290639-28fe64/value [/home/ed4bu/hello-world-jobstore.h5] bucket: None
INFO:root:GET: http://hsds.default.svc.cluster.local:5101/datasets/d-e96f8bb6-63b15792-52a3-290639-28fe64/value [/home/ed4bu/hello-world-jobstore.h5]
DEBUG:root:GET params select:[0:1:1]
INFO:root:status: 200
INFO:root:returning binary content, length: 683
INFO:root:binary response, 683 bytes
DEBUG:root:slices: [slice(0, 1, None)]
DEBUG:root:new page_start: [1]
DEBUG:root:done = True
INFO:root:got arr: (1,), cleaning up shape!
INFO:toil.jobStores.hsdsJobStore:ALL JOBS: [('9fc7c5e7-37cd-4806-b36e-fc95a4f42054', 0, 1, '000', 'Q0JaaDkxQVkmU1mimYD5AADo/+R33z4ARGP/6A05jIK///+iYASOEAAQMAFqQgaIKYmNQmnqPSPTU0eo2poxNDRp6aho0BsSMgap+ppNDUn6TGomI9E00BoGmgAZNGjJoaAOAYRhNMQwCAZADCNMmTCMBDRBlgwAxaTUKTAIbDLLo8NCthdCsGfFhOevPJ74snU4utkYjC/EoKmTlwz0I+Wo9tSKFznLTj/RpzCLtAe7RdnzrFIwZYShcVDzKFga1rD4dcsvtljk7O7N6PXjiuyfVGr2cJO+yrlEsBBlh9xQF+VQNlwN55oaJo7tvptodJ6Z5TS8cUKzr3w0kRH8BAULIOkTV2xNDGrEe4JVSHBVtU29V1GOvcQWYXY9dK1gyyCzeS70uUuw4yXmawN2abeyxteHSrQkUwUG3kIiS5PkRcESn50ZC1CAjCg0BuT5JgIKmi1S0kULCHCPhc/vIFV1z1s2o+ioRGKLTCsWJ1QRkhilLN83ppITVBhgYb4TYSHX9TMdIinc79eXnbgjqkf3l2DRdN4uaqmvUF2zqMl2kAzMyFKEAvwbadLOO5Heh0rUP1GxPltzJlSGkWUKDtYOB71/hAP8XckU4UJCimYD5A==')]
INFO:toil.jobStores.hsdsJobStore:ADDING CHUNK 000=Q0JaaDkxQVkmU1mimYD5AADo/+R33z4ARGP/6A05jIK///+iYASOEAAQMAFqQgaIKYmNQmnqPSPTU0eo2poxNDRp6aho0BsSMgap+ppNDUn6TGomI9E00BoGmgAZNGjJoaAOAYRhNMQwCAZADCNMmTCMBDRBlgwAxaTUKTAIbDLLo8NCthdCsGfFhOevPJ74snU4utkYjC/EoKmTlwz0I+Wo9tSKFznLTj/RpzCLtAe7RdnzrFIwZYShcVDzKFga1rD4dcsvtljk7O7N6PXjiuyfVGr2cJO+yrlEsBBlh9xQF+VQNlwN55oaJo7tvptodJ6Z5TS8cUKzr3w0kRH8BAULIOkTV2xNDGrEe4JVSHBVtU29V1GOvcQWYXY9dK1gyyCzeS70uUuw4yXmawN2abeyxteHSrQkUwUG3kIiS5PkRcESn50ZC1CAjCg0BuT5JgIKmi1S0kULCHCPhc/vIFV1z1s2o+ioRGKLTCsWJ1QRkhilLN83ppITVBhgYb4TYSHX9TMdIinc79eXnbgjqkf3l2DRdN4uaqmvUF2zqMl2kAzMyFKEAvwbadLOO5Heh0rUP1GxPltzJlSGkWUKDtYOB71/hAP8XckU4UJCimYD5A==
INFO:toil.jobStores.hsdsJobStore:DICT TO JOB {'job_id': '9fc7c5e7-37cd-4806-b36e-fc95a4f42054', 'overlargeID': 0, 'numChunks': 1, '000': 'Q0JaaDkxQVkmU1mimYD5AADo/+R33z4ARGP/6A05jIK///+iYASOEAAQMAFqQgaIKYmNQmnqPSPTU0eo2poxNDRp6aho0BsSMgap+ppNDUn6TGomI9E00BoGmgAZNGjJoaAOAYRhNMQwCAZADCNMmTCMBDRBlgwAxaTUKTAIbDLLo8NCthdCsGfFhOevPJ74snU4utkYjC/EoKmTlwz0I+Wo9tSKFznLTj/RpzCLtAe7RdnzrFIwZYShcVDzKFga1rD4dcsvtljk7O7N6PXjiuyfVGr2cJO+yrlEsBBlh9xQF+VQNlwN55oaJo7tvptodJ6Z5TS8cUKzr3w0kRH8BAULIOkTV2xNDGrEe4JVSHBVtU29V1GOvcQWYXY9dK1gyyCzeS70uUuw4yXmawN2abeyxteHSrQkUwUG3kIiS5PkRcESn50ZC1CAjCg0BuT5JgIKmi1S0kULCHCPhc/vIFV1z1s2o+ioRGKLTCsWJ1QRkhilLN83ppITVBhgYb4TYSHX9TMdIinc79eXnbgjqkf3l2DRdN4uaqmvUF2zqMl2kAzMyFKEAvwbadLOO5Heh0rUP1GxPltzJlSGkWUKDtYOB71/hAP8XckU4UJCimYD5A=='}
INFO:toil.jobStores.aws.utils:GOT ATTRIBUTES: {'job_id': '9fc7c5e7-37cd-4806-b36e-fc95a4f42054', 'overlargeID': 0, 'numChunks': 1, '000': 'Q0JaaDkxQVkmU1mimYD5AADo/+R33z4ARGP/6A05jIK///+iYASOEAAQMAFqQgaIKYmNQmnqPSPTU0eo2poxNDRp6aho0BsSMgap+ppNDUn6TGomI9E00BoGmgAZNGjJoaAOAYRhNMQwCAZADCNMmTCMBDRBlgwAxaTUKTAIbDLLo8NCthdCsGfFhOevPJ74snU4utkYjC/EoKmTlwz0I+Wo9tSKFznLTj/RpzCLtAe7RdnzrFIwZYShcVDzKFga1rD4dcsvtljk7O7N6PXjiuyfVGr2cJO+yrlEsBBlh9xQF+VQNlwN55oaJo7tvptodJ6Z5TS8cUKzr3w0kRH8BAULIOkTV2xNDGrEe4JVSHBVtU29V1GOvcQWYXY9dK1gyyCzeS70uUuw4yXmawN2abeyxteHSrQkUwUG3kIiS5PkRcESn50ZC1CAjCg0BuT5JgIKmi1S0kULCHCPhc/vIFV1z1s2o+ioRGKLTCsWJ1QRkhilLN83ppITVBhgYb4TYSHX9TMdIinc79eXnbgjqkf3l2DRdN4uaqmvUF2zqMl2kAzMyFKEAvwbadLOO5Heh0rUP1GxPltzJlSGkWUKDtYOB71/hAP8XckU4UJCimYD5A=='}
DEBUG:toil.jobStores.aws.utils:Creating binary from 1 chunks: [(0, 'Q0JaaDkxQVkmU1mimYD5AADo/+R33z4ARGP/6A05jIK///+iYASOEAAQMAFqQgaIKYmNQmnqPSPTU0eo2poxNDRp6aho0BsSMgap+ppNDUn6TGomI9E00BoGmgAZNGjJoaAOAYRhNMQwCAZADCNMmTCMBDRBlgwAxaTUKTAIbDLLo8NCthdCsGfFhOevPJ74snU4utkYjC/EoKmTlwz0I+Wo9tSKFznLTj/RpzCLtAe7RdnzrFIwZYShcVDzKFga1rD4dcsvtljk7O7N6PXjiuyfVGr2cJO+yrlEsBBlh9xQF+VQNlwN55oaJo7tvptodJ6Z5TS8cUKzr3w0kRH8BAULIOkTV2xNDGrEe4JVSHBVtU29V1GOvcQWYXY9dK1gyyCzeS70uUuw4yXmawN2abeyxteHSrQkUwUG3kIiS5PkRcESn50ZC1CAjCg0BuT5JgIKmi1S0kULCHCPhc/vIFV1z1s2o+ioRGKLTCsWJ1QRkhilLN83ppITVBhgYb4TYSHX9TMdIinc79eXnbgjqkf3l2DRdN4uaqmvUF2zqMl2kAzMyFKEAvwbadLOO5Heh0rUP1GxPltzJlSGkWUKDtYOB71/hAP8XckU4UJCimYD5A==')]
DEBUG:toil.jobStores.aws.utils:Creating binary from b'Q0JaaDkxQVkmU1mimYD5AADo/+R33z4ARGP/6A05jIK///+iYASOEAAQMAFqQgaIKYmNQmnqPSPTU0eo2poxNDRp6aho0BsSMgap+ppNDUn6TGomI9E00BoGmgAZNGjJoaAOAYRhNMQwCAZADCNMmTCMBDRBlgwAxaTUKTAIbDLLo8NCthdCsGfFhOevPJ74snU4utkYjC/EoKmTlwz0I+Wo9tSKFznLTj/RpzCLtAe7RdnzrFIwZYShcVDzKFga1rD4dcsvtljk7O7N6PXjiuyfVGr2cJO+yrlEsBBlh9xQF+VQNlwN55oaJo7tvptodJ6Z5TS8cUKzr3w0kRH8BAULIOkTV2xNDGrEe4JVSHBVtU29V1GOvcQWYXY9dK1gyyCzeS70uUuw4yXmawN2abeyxteHSrQkUwUG3kIiS5PkRcESn50ZC1CAjCg0BuT5JgIKmi1S0kULCHCPhc/vIFV1z1s2o+ioRGKLTCsWJ1QRkhilLN83ppITVBhgYb4TYSHX9TMdIinc79eXnbgjqkf3l2DRdN4uaqmvUF2zqMl2kAzMyFKEAvwbadLOO5Heh0rUP1GxPltzJlSGkWUKDtYOB71/hAP8XckU4UJCimYD5A=='
DEBUG:toil.jobStores.hsdsJobStore:Loaded job 9fc7c5e7-37cd-4806-b36e-fc95a4f42054
DEBUG:toil.worker:Parsed job description
INFO:toil.worker:Working on job 'helloWorld' 9fc7c5e7-37cd-4806-b36e-fc95a4f42054 v1
DEBUG:toil.worker:Got a command to run: _toil 0fbcffac-76f5-49b9-b159-d60bba3a730a /tmp/work tutorial_helloworld False
DEBUG:toil.job:Loading user module ModuleDescriptor(dirPath='/tmp/work', name='tutorial_helloworld', fromVirtualEnv=False).
DEBUG:root:group.__getitem__(files)
DEBUG:root:__get_link_json(files)
DEBUG:root:getObjByUuid(d-e96f8bb6-63b15792-c4d0-b633a9-53f8c7)
DEBUG:root:use basic auth with username: [email protected]
DEBUG:root:GET: /datasets/d-e96f8bb6-63b15792-c4d0-b633a9-53f8c7 [/home/ed4bu/hello-world-jobstore.h5] bucket: None
INFO:root:GET: http://hsds.default.svc.cluster.local:5101/datasets/d-e96f8bb6-63b15792-c4d0-b633a9-53f8c7 [/home/ed4bu/hello-world-jobstore.h5]
INFO:root:status: 200
DEBUG:root:rsp_json: {'id': 'd-e96f8bb6-63b15792-c4d0-b633a9-53f8c7', 'root': 'g-e96f8bb6-63b15792-61e7-033eeb-39df1a', 'shape': {'class': 'H5S_SIMPLE', 'dims': [9], 'maxdims': [0]}, 'type': {'class': 'H5T_COMPOUND', 'fields': [{'name': 'file_id', 'type': {'class': 'H5T_STRING', 'length': 'H5T_VARIABLE', 'charSet': 'H5T_CSET_UTF8', 'strPad': 'H5T_STR_NULLTERM'}}, {'name': 'ownerID', 'type': {'class': 'H5T_STRING', 'length': 'H5T_VARIABLE', 'charSet': 'H5T_CSET_UTF8', 'strPad': 'H5T_STR_NULLTERM'}}, {'name': 'encrypted', 'type': {'class': 'H5T_INTEGER', 'base': 'H5T_STD_I8LE'}}, {'name': 'version', 'type': {'class': 'H5T_STRING', 'length': 'H5T_VARIABLE', 'charSet': 'H5T_CSET_UTF8', 'strPad': 'H5T_STR_NULLTERM'}}, {'name': 'checksum', 'type': {'class': 'H5T_STRING', 'length': 'H5T_VARIABLE', 'charSet': 'H5T_CSET_UTF8', 'strPad': 'H5T_STR_NULLTERM'}}, {'name': 'numChunks', 'type': {'class': 'H5T_INTEGER', 'base': 'H5T_STD_I32LE'}}, {'name': 'chunkName', 'type': {'class': 'H5T_STRING', 'length': 'H5T_VARIABLE', 'charSet': 'H5T_CSET_UTF8', 'strPad': 'H5T_STR_NULLTERM'}}, {'name': 'chunkValue', 'type': {'class': 'H5T_STRING', 'length': 'H5T_VARIABLE', 'charSet': 'H5T_CSET_UTF8', 'strPad': 'H5T_STR_NULLTERM'}}]}, 'creationProperties': {'filters': [{'class': 'H5Z_FILTER_DEFLATE', 'id': 1, 'level': 9, 'name': 'gzip'}]}, 'layout': {'class': 'H5D_CHUNKED', 'dims': [8192]}, 'attributeCount': 0, 'created': 1640815506, 'lastModified': 1640815506, 'domain': '/home/ed4bu/hello-world-jobstore.h5', 'hrefs': [{'rel': 'self', 'href': 'http://hsds.hdf.test/datasets/d-e96f8bb6-63b15792-c4d0-b633a9-53f8c7?domain=/home/ed4bu/hello-world-jobstore.h5'}, {'rel': 'root', 'href': 'http://hsds.hdf.test/groups/g-e96f8bb6-63b15792-61e7-033eeb-39df1a?domain=/home/ed4bu/hello-world-jobstore.h5'}, {'rel': 'home', 'href': 'http://hsds.hdf.test/?domain=/home/ed4bu/hello-world-jobstore.h5'}, {'rel': 'attributes', 'href': 'http://hsds.hdf.test/datasets/d-e96f8bb6-63b15792-c4d0-b633a9-53f8c7/attributes?domain=/home/ed4bu/hello-world-jobstore.h5'}, {'rel': 'data', 'href': 'http://hsds.hdf.test/datasets/d-e96f8bb6-63b15792-c4d0-b633a9-53f8c7/value?domain=/home/ed4bu/hello-world-jobstore.h5'}]}
DEBUG:toil.jobStores.hsdsJobStore:query 9 jobs for 'file_id==0fbcffac-76f5-49b9-b159-d60bba3a730a'
DEBUG:root:group.__getitem__(files)
DEBUG:root:__get_link_json(files)
DEBUG:root:getObjByUuid(d-e96f8bb6-63b15792-c4d0-b633a9-53f8c7)
DEBUG:root:use basic auth with username: [email protected]
DEBUG:root:GET: /datasets/d-e96f8bb6-63b15792-c4d0-b633a9-53f8c7 [/home/ed4bu/hello-world-jobstore.h5] bucket: None
INFO:root:GET: http://hsds.default.svc.cluster.local:5101/datasets/d-e96f8bb6-63b15792-c4d0-b633a9-53f8c7 [/home/ed4bu/hello-world-jobstore.h5]
INFO:root:status: 200
DEBUG:root:rsp_json: {'id': 'd-e96f8bb6-63b15792-c4d0-b633a9-53f8c7', 'root': 'g-e96f8bb6-63b15792-61e7-033eeb-39df1a', 'shape': {'class': 'H5S_SIMPLE', 'dims': [9], 'maxdims': [0]}, 'type': {'class': 'H5T_COMPOUND', 'fields': [{'name': 'file_id', 'type': {'class': 'H5T_STRING', 'length': 'H5T_VARIABLE', 'charSet': 'H5T_CSET_UTF8', 'strPad': 'H5T_STR_NULLTERM'}}, {'name': 'ownerID', 'type': {'class': 'H5T_STRING', 'length': 'H5T_VARIABLE', 'charSet': 'H5T_CSET_UTF8', 'strPad': 'H5T_STR_NULLTERM'}}, {'name': 'encrypted', 'type': {'class': 'H5T_INTEGER', 'base': 'H5T_STD_I8LE'}}, {'name': 'version', 'type': {'class': 'H5T_STRING', 'length': 'H5T_VARIABLE', 'charSet': 'H5T_CSET_UTF8', 'strPad': 'H5T_STR_NULLTERM'}}, {'name': 'checksum', 'type': {'class': 'H5T_STRING', 'length': 'H5T_VARIABLE', 'charSet': 'H5T_CSET_UTF8', 'strPad': 'H5T_STR_NULLTERM'}}, {'name': 'numChunks', 'type': {'class': 'H5T_INTEGER', 'base': 'H5T_STD_I32LE'}}, {'name': 'chunkName', 'type': {'class': 'H5T_STRING', 'length': 'H5T_VARIABLE', 'charSet': 'H5T_CSET_UTF8', 'strPad': 'H5T_STR_NULLTERM'}}, {'name': 'chunkValue', 'type': {'class': 'H5T_STRING', 'length': 'H5T_VARIABLE', 'charSet': 'H5T_CSET_UTF8', 'strPad': 'H5T_STR_NULLTERM'}}]}, 'creationProperties': {'filters': [{'class': 'H5Z_FILTER_DEFLATE', 'id': 1, 'level': 9, 'name': 'gzip'}]}, 'layout': {'class': 'H5D_CHUNKED', 'dims': [8192]}, 'attributeCount': 0, 'created': 1640815506, 'lastModified': 1640815506, 'domain': '/home/ed4bu/hello-world-jobstore.h5', 'hrefs': [{'rel': 'self', 'href': 'http://hsds.hdf.test/datasets/d-e96f8bb6-63b15792-c4d0-b633a9-53f8c7?domain=/home/ed4bu/hello-world-jobstore.h5'}, {'rel': 'root', 'href': 'http://hsds.hdf.test/groups/g-e96f8bb6-63b15792-61e7-033eeb-39df1a?domain=/home/ed4bu/hello-world-jobstore.h5'}, {'rel': 'home', 'href': 'http://hsds.hdf.test/?domain=/home/ed4bu/hello-world-jobstore.h5'}, {'rel': 'attributes', 'href': 'http://hsds.hdf.test/datasets/d-e96f8bb6-63b15792-c4d0-b633a9-53f8c7/attributes?domain=/home/ed4bu/hello-world-jobstore.h5'}, {'rel': 'data', 'href': 'http://hsds.hdf.test/datasets/d-e96f8bb6-63b15792-c4d0-b633a9-53f8c7/value?domain=/home/ed4bu/hello-world-jobstore.h5'}]}
DEBUG:root:dataset.__getitem__
DEBUG:root:arg: Ellipsis type: <class 'ellipsis'>
DEBUG:root:new_dtype: [('file_id', 'O'), ('ownerID', 'O'), ('encrypted', 'i1'), ('version', 'O'), ('checksum', 'O'), ('numChunks', '<i4'), ('chunkName', 'O'), ('chunkValue', 'O')]
DEBUG:root:selection_constructor
DEBUG:root:dataset shape: (9,)
DEBUG:root:mshape: (9,)
DEBUG:root:single_element: False
DEBUG:root:not settng nonstrict
DEBUG:root:selection._sel: ((0,), (9,), (1,), (False,))
INFO:root:selection: start (0,) stop [9] step (1,)
DEBUG:root:split_dim: 0
DEBUG:root:chunks_per_page: 1
DEBUG:root:mshape_split_dim: 0
DEBUG:root:chunk size for split_dim: 8192
DEBUG:root:num_rows: 8192
DEBUG:root:paged read, chunks_per_page: 1 max_chunks: 1, num_pages: 1
DEBUG:root:page_number: 0
DEBUG:root:start: [0] stop: [9]
INFO:root:page_stop: 9
INFO:root:page_mshape: (9,)
DEBUG:root:use basic auth with username: [email protected]
DEBUG:root:GET: /datasets/d-e96f8bb6-63b15792-c4d0-b633a9-53f8c7/value [/home/ed4bu/hello-world-jobstore.h5] bucket: None
INFO:root:GET: http://hsds.default.svc.cluster.local:5101/datasets/d-e96f8bb6-63b15792-c4d0-b633a9-53f8c7/value [/home/ed4bu/hello-world-jobstore.h5]
DEBUG:root:GET params select:[0:9:1]
INFO:root:status: 200
INFO:root:returning binary content, length: 1832
INFO:root:binary response, 1832 bytes
DEBUG:root:slices: [slice(0, 9, None)]
DEBUG:root:new page_start: [9]
DEBUG:root:done = True
INFO:root:got arr: (9,), cleaning up shape!
DEBUG:toil.jobStores.hsdsJobStore:all files: [('a77581f6-ad82-5142-acb7-5a6b01eab3fb', '891f7db6-e4d9-4221-a58e-ab6cc4395f94', 0, '603d581f-b96b-4d21-a753-484416015310', 'sha1$c007123652d864967a5f0536f20d19f6084a4947', 0, '000', 0)
('79bad63b-c331-5212-a0d2-d06a788c4b3c', '891f7db6-e4d9-4221-a58e-ab6cc4395f94', 0, 0, 0, 1, '000', 'VTE4')
('6681340b-9a30-5117-9d8c-f805c4e9ebd8', '891f7db6-e4d9-4221-a58e-ab6cc4395f94', 0, 0, 0, 1, '000', 'VYAFlU0AAAAAAAAAjA10b2lsLnJlc291cmNllIwQTW9kdWxlRGVzY3JpcHRvcpSTlIwJL3RtcC93b3JrlIwTdHV0b3JpYWxfaGVsbG93b3JsZJSJh5SBlC4=')
('679e4679-210a-5a36-baf7-f7f6bacbca0b', '891f7db6-e4d9-4221-a58e-ab6cc4395f94', 0, '01eaadf0-7413-49ec-b4dc-28b9b1a33eae', 'sha1$647cbbc1812f6ff4951963f3d3e56fafaa2fe066', 0, '000', 0)
('01c37f76-e635-5f9c-ad18-6ed1ae1b2e2a', '891f7db6-e4d9-4221-a58e-ab6cc4395f94', 0, '9c6a4d46-6c17-42fd-9e29-44911359ca1e', 'sha1$3f416b1bfe2e3a208c9af2e184069d8e6a0850a4', 0, '000', 0)
('6a87fe18-4ece-468b-8b5c-40b725a90300', 0, 0, 0, 0, 1, '000', 'VYAFlbwAAAAAAAAAjAh0b2lsLmpvYpSMGlVuZnVsZmlsbGVkUHJvbWlzZVNlbnRpbmVslJOUKYGUfZQojBFmdWxmaWxsaW5nSm9iTmFtZZSMQSdoZWxsb1dvcmxkJyBUZW1wb3JhcnlJRCg0YjhjNTZiMS0yZTg4LTRmYTMtOGJlMC0xYzU4YmFjOTA5YjApIHYwlIwHZmlsZV9pZJSMJDZhODdmZTE4LTRlY2UtNDY4Yi04YjVjLTQwYjcyNWE5MDMwMJR1Yi4=')
('de90cdf4-4065-5763-8ae2-b71758c6f931', '891f7db6-e4d9-4221-a58e-ab6cc4395f94', 0, 0, 0, 1, '000', 'VYAFlW4AAAAAAAAAjAh0b2lsLmpvYpSMB1Byb21pc2WUk5SMKHMzaHNkczp1cy1lYXN0LTE6L2hvbWUvZWQ0YnUvaGVsbG8td29ybGSUjCQ2YTg3ZmUxOC00ZWNlLTQ2OGItOGI1Yy00MGI3MjVhOTAzMDCUhpRSlC4=')
('0fbcffac-76f5-49b9-b159-d60bba3a730a', '9fc7c5e7-37cd-4806-b36e-fc95a4f42054', 0, '13a7e36f-e502-4120-93bb-e193ced59d36', 'sha1$434d7d7e7db838c77a6d3adcf390a3681fb54b34', 0, '000', 0)
('a09ece78-f7f5-580e-bcf3-59b8e4b7c1fa', '891f7db6-e4d9-4221-a58e-ab6cc4395f94', 0, 0, 0, 1, '000', 'VTlmYzdjNWU3LTM3Y2QtNDgwNi1iMzZlLWZjOTVhNGY0MjA1NA==')]
DEBUG:root:group.__getitem__(files)
DEBUG:root:__get_link_json(files)
DEBUG:root:getObjByUuid(d-e96f8bb6-63b15792-c4d0-b633a9-53f8c7)
DEBUG:root:use basic auth with username: [email protected]
DEBUG:root:GET: /datasets/d-e96f8bb6-63b15792-c4d0-b633a9-53f8c7 [/home/ed4bu/hello-world-jobstore.h5] bucket: None
INFO:root:GET: http://hsds.default.svc.cluster.local:5101/datasets/d-e96f8bb6-63b15792-c4d0-b633a9-53f8c7 [/home/ed4bu/hello-world-jobstore.h5]
INFO:root:status: 200
DEBUG:root:rsp_json: {'id': 'd-e96f8bb6-63b15792-c4d0-b633a9-53f8c7', 'root': 'g-e96f8bb6-63b15792-61e7-033eeb-39df1a', 'shape': {'class': 'H5S_SIMPLE', 'dims': [9], 'maxdims': [0]}, 'type': {'class': 'H5T_COMPOUND', 'fields': [{'name': 'file_id', 'type': {'class': 'H5T_STRING', 'length': 'H5T_VARIABLE', 'charSet': 'H5T_CSET_UTF8', 'strPad': 'H5T_STR_NULLTERM'}}, {'name': 'ownerID', 'type': {'class': 'H5T_STRING', 'length': 'H5T_VARIABLE', 'charSet': 'H5T_CSET_UTF8', 'strPad': 'H5T_STR_NULLTERM'}}, {'name': 'encrypted', 'type': {'class': 'H5T_INTEGER', 'base': 'H5T_STD_I8LE'}}, {'name': 'version', 'type': {'class': 'H5T_STRING', 'length': 'H5T_VARIABLE', 'charSet': 'H5T_CSET_UTF8', 'strPad': 'H5T_STR_NULLTERM'}}, {'name': 'checksum', 'type': {'class': 'H5T_STRING', 'length': 'H5T_VARIABLE', 'charSet': 'H5T_CSET_UTF8', 'strPad': 'H5T_STR_NULLTERM'}}, {'name': 'numChunks', 'type': {'class': 'H5T_INTEGER', 'base': 'H5T_STD_I32LE'}}, {'name': 'chunkName', 'type': {'class': 'H5T_STRING', 'length': 'H5T_VARIABLE', 'charSet': 'H5T_CSET_UTF8', 'strPad': 'H5T_STR_NULLTERM'}}, {'name': 'chunkValue', 'type': {'class': 'H5T_STRING', 'length': 'H5T_VARIABLE', 'charSet': 'H5T_CSET_UTF8', 'strPad': 'H5T_STR_NULLTERM'}}]}, 'creationProperties': {'filters': [{'class': 'H5Z_FILTER_DEFLATE', 'id': 1, 'level': 9, 'name': 'gzip'}]}, 'layout': {'class': 'H5D_CHUNKED', 'dims': [8192]}, 'attributeCount': 0, 'created': 1640815506, 'lastModified': 1640815506, 'domain': '/home/ed4bu/hello-world-jobstore.h5', 'hrefs': [{'rel': 'self', 'href': 'http://hsds.hdf.test/datasets/d-e96f8bb6-63b15792-c4d0-b633a9-53f8c7?domain=/home/ed4bu/hello-world-jobstore.h5'}, {'rel': 'root', 'href': 'http://hsds.hdf.test/groups/g-e96f8bb6-63b15792-61e7-033eeb-39df1a?domain=/home/ed4bu/hello-world-jobstore.h5'}, {'rel': 'home', 'href': 'http://hsds.hdf.test/?domain=/home/ed4bu/hello-world-jobstore.h5'}, {'rel': 'attributes', 'href': 'http://hsds.hdf.test/datasets/d-e96f8bb6-63b15792-c4d0-b633a9-53f8c7/attributes?domain=/home/ed4bu/hello-world-jobstore.h5'}, {'rel': 'data', 'href': 'http://hsds.hdf.test/datasets/d-e96f8bb6-63b15792-c4d0-b633a9-53f8c7/value?domain=/home/ed4bu/hello-world-jobstore.h5'}]}
INFO:root:req - cursor: 0 page_size: 9
DEBUG:root:query param: [0:9]
DEBUG:root:params: {'query': "file_id=='0fbcffac-76f5-49b9-b159-d60bba3a730a'", 'select': '[0:9]'}
DEBUG:root:use basic auth with username: [email protected]
DEBUG:root:GET: /datasets/d-e96f8bb6-63b15792-c4d0-b633a9-53f8c7/value [/home/ed4bu/hello-world-jobstore.h5] bucket: None
INFO:root:GET: http://hsds.default.svc.cluster.local:5101/datasets/d-e96f8bb6-63b15792-c4d0-b633a9-53f8c7/value [/home/ed4bu/hello-world-jobstore.h5]
DEBUG:root:GET params query:file_id=='0fbcffac-76f5-49b9-b159-d60bba3a730a'
DEBUG:root:GET params select:[0:9]
INFO:root:status: 200
DEBUG:root:rsp_json: {'index': [7], 'value': [['0fbcffac-76f5-49b9-b159-d60bba3a730a', '9fc7c5e7-37cd-4806-b36e-fc95a4f42054', 0, '13a7e36f-e502-4120-93bb-e193ced59d36', 'sha1$434d7d7e7db838c77a6d3adcf390a3681fb54b34', 0, '000', 0]], 'hrefs': [{'rel': 'self', 'href': 'http://hsds.hdf.test/datasets/d-e96f8bb6-63b15792-c4d0-b633a9-53f8c7/value?domain=/home/ed4bu/hello-world-jobstore.h5'}, {'rel': 'root', 'href': 'http://hsds.hdf.test/groups/g-e96f8bb6-63b15792-61e7-033eeb-39df1a?domain=/home/ed4bu/hello-world-jobstore.h5'}, {'rel': 'home', 'href': 'http://hsds.hdf.test/?domain=/home/ed4bu/hello-world-jobstore.h5'}, {'rel': 'owner', 'href': 'http://hsds.hdf.test/datasets/d-e96f8bb6-63b15792-c4d0-b633a9-53f8c7?domain=/home/ed4bu/hello-world-jobstore.h5'}]}
INFO:root:got 1 rows
INFO:root:completed iteration, returning: 1 rows
INFO:toil.jobStores.hsdsJobStore:JOBITEM: {'name': '0fbcffac-76f5-49b9-b159-d60bba3a730a', 'ownerID': '9fc7c5e7-37cd-4806-b36e-fc95a4f42054', 'encrypted': 'False', 'version': '13a7e36f-e502-4120-93bb-e193ced59d36', 'checksum': 'sha1$434d7d7e7db838c77a6d3adcf390a3681fb54b34', 'numChunks': 0, 'chunks': None, '000': 0} [('0fbcffac-76f5-49b9-b159-d60bba3a730a', '9fc7c5e7-37cd-4806-b36e-fc95a4f42054', 0, '13a7e36f-e502-4120-93bb-e193ced59d36', 'sha1$434d7d7e7db838c77a6d3adcf390a3681fb54b34', 0, '000', 0)]
INFO:toil.jobStores.aws.utils:GOT ATTRIBUTES: {'name': '0fbcffac-76f5-49b9-b159-d60bba3a730a', 'ownerID': '9fc7c5e7-37cd-4806-b36e-fc95a4f42054', 'encrypted': 'False', 'version': '13a7e36f-e502-4120-93bb-e193ced59d36', 'checksum': 'sha1$434d7d7e7db838c77a6d3adcf390a3681fb54b34', 'numChunks': 0, 'chunks': None, '000': 0}
DEBUG:toil.jobStores.aws.utils:Creating binary from 0 chunks: [(0, 0)]
DEBUG:toil.jobStores.aws.baseS3JobStore:Reading FileInfo(fileID='0fbcffac-76f5-49b9-b159-d60bba3a730a', ownerID='9fc7c5e7-37cd-4806-b36e-fc95a4f42054', encrypted=False, version='13a7e36f-e502-4120-93bb-e193ced59d36', previousVersion='13a7e36f-e502-4120-93bb-e193ced59d36', content=None, checksum='sha1$434d7d7e7db...cf390a3681fb54b34', _numContentChunks=0) bound to <toil.jobStores.hsdsJobStore.HSDSJobStore object at 0x7f040d83e370> into stream.
DEBUG:toil.jobStores.aws.baseS3JobStore:Checking checsum for FileInfo(fileID='0fbcffac-76f5-49b9-b159-d60bba3a730a', ownerID='9fc7c5e7-37cd-4806-b36e-fc95a4f42054', encrypted=False, version='13a7e36f-e502-4120-93bb-e193ced59d36', previousVersion='13a7e36f-e502-4120-93bb-e193ced59d36', content=None, checksum='sha1$434d7d7e7db...cf390a3681fb54b34', _numContentChunks=0) bound to <toil.jobStores.hsdsJobStore.HSDSJobStore object at 0x7f040d83e370>: sha1$434d7d7e7db838c77a6d3adcf390a3681fb54b34
DEBUG:toil.jobStores.aws.baseS3JobStore:Starting sha1 checksum to match 434d7d7e7db838c77a6d3adcf390a3681fb54b34
DEBUG:toil.jobStores.aws.baseS3JobStore:Completed checksum with hash 434d7d7e7db838c77a6d3adcf390a3681fb54b34 vs. expected 434d7d7e7db838c77a6d3adcf390a3681fb54b34
INFO:toil.worker:Loaded body Job('helloWorld' 9fc7c5e7-37cd-4806-b36e-fc95a4f42054 v1) from description 'helloWorld' 9fc7c5e7-37cd-4806-b36e-fc95a4f42054 v1
DEBUG:toil.fileStores.cachingFileStore:Starting job ('helloWorld' 9fc7c5e7-37cd-4806-b36e-fc95a4f42054 v1) with ID (9fc7c5e7-37cd-4806-b36e-fc95a4f42054).
ERROR:root:Error talking to caching database: database is locked
WARNING:toil.lib.retry:Error in <function CachingFileStore._staticWrite at 0x7f03fdb645e0>: database is locked. Retrying after 1 s...
ERROR:root:Error talking to caching database: database is locked
WARNING:toil.lib.retry:Error in <function CachingFileStore._staticWrite at 0x7f03fdb645e0>: database is locked. Retrying after 1 s...
ERROR:root:Error talking to caching database: database is locked
WARNING:toil.lib.retry:Error in <function CachingFileStore._staticWrite at 0x7f03fdb645e0>: database is locked. Retrying after 2 s...
ERROR:root:Error talking to caching database: database is locked
WARNING:toil.lib.retry:Error in <function CachingFileStore._staticWrite at 0x7f03fdb645e0>: database is locked. Retrying after 4 s...
ERROR:root:Error talking to caching database: database is locked
WARNING:toil.lib.retry:Error in <function CachingFileStore._staticWrite at 0x7f03fdb645e0>: database is locked. Retrying after 8 s...
ERROR:root:Error talking to caching database: database is locked
WARNING:toil.lib.retry:Error in <function CachingFileStore._staticWrite at 0x7f03fdb645e0>: database is locked. Retrying after 16 s...
ERROR:root:Error talking to caching database: database is locked
WARNING:toil.lib.retry:Error in <function CachingFileStore._staticWrite at 0x7f03fdb645e0>: database is locked. Retrying after 1 s...
ERROR:root:Error talking to caching database: database is locked
WARNING:toil.lib.retry:Error in <function CachingFileStore._staticWrite at 0x7f03fdb645e0>: database is locked. Retrying after 1 s...
ERROR:root:Error talking to caching database: database is locked
WARNING:toil.lib.retry:Error in <function CachingFileStore._staticWrite at 0x7f03fdb645e0>: database is locked. Retrying after 2 s...
ERROR:root:Error talking to caching database: database is locked
WARNING:toil.lib.retry:Error in <function CachingFileStore._staticWrite at 0x7f03fdb645e0>: database is locked. Retrying after 4 s...
ERROR:root:Error talking to caching database: database is locked
WARNING:toil.lib.retry:Error in <function CachingFileStore._staticWrite at 0x7f03fdb645e0>: database is locked. Retrying after 8 s...
ERROR:root:Error talking to caching database: database is locked
WARNING:toil.lib.retry:Error in <function CachingFileStore._staticWrite at 0x7f03fdb645e0>: database is locked. Retrying after 16 s...
ERROR:root:Error talking to caching database: database is locked
WARNING:toil.lib.retry:Error in <function CachingFileStore._staticWrite at 0x7f03fdb645e0>: database is locked. Retrying after 1 s...
ERROR:root:Error talking to caching database: database is locked
WARNING:toil.lib.retry:Error in <function CachingFileStore._staticWrite at 0x7f03fdb645e0>: database is locked. Retrying after 1 s...
ERROR:root:Error talking to caching database: database is locked
WARNING:toil.lib.retry:Error in <function CachingFileStore._staticWrite at 0x7f03fdb645e0>: database is locked. Retrying after 2 s...
ERROR:root:Error talking to caching database: database is locked
WARNING:toil.lib.retry:Error in <function CachingFileStore._staticWrite at 0x7f03fdb645e0>: database is locked. Retrying after 4 s...
ERROR:root:Error talking to caching database: database is locked
WARNING:toil.lib.retry:Error in <function CachingFileStore._staticWrite at 0x7f03fdb645e0>: database is locked. Retrying after 8 s...
ERROR:root:Error talking to caching database: database is locked
WARNING:toil.lib.retry:Error in <function CachingFileStore._staticWrite at 0x7f03fdb645e0>: database is locked. Retrying after 16 s...
ERROR:root:Error talking to caching database: database is locked
WARNING:toil.lib.retry:Error in <function CachingFileStore._staticWrite at 0x7f03fdb645e0>: database is locked. Retrying after 1 s...
ERROR:root:Error talking to caching database: database is locked
WARNING:toil.lib.retry:Error in <function CachingFileStore._staticWrite at 0x7f03fdb645e0>: database is locked. Retrying after 1 s...
ERROR:root:Error talking to caching database: database is locked
WARNING:toil.lib.retry:Error in <function CachingFileStore._staticWrite at 0x7f03fdb645e0>: database is locked. Retrying after 2 s...
ERROR:root:Error talking to caching database: database is locked
WARNING:toil.lib.retry:Error in <function CachingFileStore._staticWrite at 0x7f03fdb645e0>: database is locked. Retrying after 4 s...
ERROR:root:Error talking to caching database: database is locked
WARNING:toil.lib.retry:Error in <function CachingFileStore._staticWrite at 0x7f03fdb645e0>: database is locked. Retrying after 8 s...
ERROR:root:Error talking to caching database: database is locked
WARNING:toil.lib.retry:Error in <function CachingFileStore._staticWrite at 0x7f03fdb645e0>: database is locked. Retrying after 16 s...
ERROR:root:Error talking to caching database: database is locked
WARNING:toil.lib.retry:Error in <function CachingFileStore._staticWrite at 0x7f03fdb645e0>: database is locked. Retrying after 1 s...
ERROR:root:Error talking to caching database: database is locked
WARNING:toil.lib.retry:Error in <function CachingFileStore._staticWrite at 0x7f03fdb645e0>: database is locked. Retrying after 1 s...
ERROR:root:Error talking to caching database: database is locked
WARNING:toil.lib.retry:Error in <function CachingFileStore._staticWrite at 0x7f03fdb645e0>: database is locked. Retrying after 2 s...
ERROR:root:Error talking to caching database: database is locked
WARNING:toil.lib.retry:Error in <function CachingFileStore._staticWrite at 0x7f03fdb645e0>: database is locked. Retrying after 4 s...
ERROR:root:Error talking to caching database: database is locked
WARNING:toil.lib.retry:Error in <function CachingFileStore._staticWrite at 0x7f03fdb645e0>: database is locked. Retrying after 8 s...
ERROR:root:Error talking to caching database: database is locked
WARNING:toil.lib.retry:Error in <function CachingFileStore._staticWrite at 0x7f03fdb645e0>: database is locked. Retrying after 16 s...
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment