Created
April 19, 2023 23:13
-
-
Save bveeramani/809c5b24e7d7d8c3382618d6d6104f72 to your computer and use it in GitHub Desktop.
read_tfrecords_benchmark
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
[INFO 2023-04-13 16:18:26,072] anyscale_job_wrapper.py: 256 ### Starting ### | |
[INFO 2023-04-13 16:18:26,072] anyscale_job_wrapper.py: 280 ### Starting entrypoint ### | |
[INFO 2023-04-13 16:18:26,072] anyscale_job_wrapper.py: 162 Running command python read_tfrecords_benchmark.py | |
[INFO 2023-04-13 16:18:26,076] anyscale_job_wrapper.py: 176 Starting process 1561. | |
+ python read_tfrecords_benchmark.py | |
2023-04-13 16:18:27,133 INFO worker.py:1315 -- Using address 10.0.14.144:6379 set in the environment variable RAY_ADDRESS | |
2023-04-13 16:18:27,133 INFO worker.py:1432 -- Connecting to existing Ray cluster at address: 10.0.14.144:6379... | |
2023-04-13 16:18:27,139 INFO worker.py:1613 -- Connected to Ray cluster. View the dashboard at [1m[32mhttps://console.anyscale-staging.com/api/v2/sessions/ses_hapzispyymkvr3f7jsmrbi7txz/services?redirect_to=dashboard [39m[22m | |
Running benchmark: read-tfrecords | |
2023-04-13 16:18:28,874 INFO streaming_executor.py:87 -- Executing DAG InputDataBuffer[Input] -> TaskPoolMapOperator[ReadImage->MapBatches(images_to_bytes)->Write] | |
2023-04-13 16:18:28,875 INFO streaming_executor.py:88 -- Execution config: ExecutionOptions(resource_limits=ExecutionResources(cpu=None, gpu=None, object_store_memory=None), locality_with_output=False, preserve_order=False, actor_locality_enabled=True, verbose_progress=False) | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 1.98 MiB/4.29 GiB object_store_memory: 0%| | 0/32 [00:00<?, ?it/s] | |
[2m[36m(ReadImage->MapBatches(images_to_bytes)->Write pid=655)[0m To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags. | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 1.98 MiB/4.29 GiB object_store_memory: 0%| | 0/32 [00:00<?, ?it/s] | |
[2m[36m(ReadImage->MapBatches(images_to_bytes)->Write pid=655)[0m 2023-04-13 16:18:31.939072: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/nvidia/lib:/usr/local/nvidia/lib64 | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 1.98 MiB/4.29 GiB object_store_memory: 0%| | 0/32 [00:03<?, ?it/s] | |
[2m[36m(ReadImage->MapBatches(images_to_bytes)->Write pid=655)[0m 2023-04-13 16:18:31.939413: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/nvidia/lib:/usr/local/nvidia/lib64 | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 1.98 MiB/4.29 GiB object_store_memory: 0%| | 0/32 [00:03<?, ?it/s] | |
[2m[36m(ReadImage->MapBatches(images_to_bytes)->Write pid=655)[0m 2023-04-13 16:18:31.939426: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly. | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 1.98 MiB/4.29 GiB object_store_memory: 0%| | 0/32 [00:03<?, ?it/s] | |
Running: 0.0/16.0 CPU, 0.0/0.0 GPU, 0.0 MiB/4.29 GiB object_store_memory: 0%| | 0/32 [00:05<?, ?it/s] | |
Running: 0.0/16.0 CPU, 0.0/0.0 GPU, 0.0 MiB/4.29 GiB object_store_memory: 3%|▎ | 1/32 [00:05<02:51, 5.52s/it] | |
2023-04-13 16:18:34,453 WARNING plan.py:557 -- Warning: The Ray cluster currently does not have any available CPUs. The Dataset job will hang unless more CPUs are freed up. A common reason is that cluster resources are used by Actors or Tune trials; see the following link for more details: https://docs.ray.io/en/master/data/dataset-internals.html#datasets-and-tune | |
2023-04-13 16:20:02,769 INFO streaming_executor.py:87 -- Executing DAG InputDataBuffer[Input] -> TaskPoolMapOperator[ReadImage->MapBatches(images_to_bytes)->Write] | |
2023-04-13 16:20:02,769 INFO streaming_executor.py:88 -- Execution config: ExecutionOptions(resource_limits=ExecutionResources(cpu=None, gpu=None, object_store_memory=None), locality_with_output=False, preserve_order=False, actor_locality_enabled=True, verbose_progress=False) | |
Running 0: 0%| | 0/100 [00:00<?, ?it/s] | |
Running: 0.0/16.0 CPU, 0.0/0.0 GPU, 0.0 MiB/4.29 GiB object_store_memory: 0%| | 0/100 [00:00<?, ?it/s] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 38.47 MiB/4.29 GiB object_store_memory: 0%| | 0/100 [00:00<?, ?it/s] | |
Running: 4.0/16.0 CPU, 0.0/0.0 GPU, 9.62 MiB/4.29 GiB object_store_memory: 0%| | 0/100 [00:00<?, ?it/s] | |
Running: 4.0/16.0 CPU, 0.0/0.0 GPU, 9.62 MiB/4.29 GiB object_store_memory: 1%| | 1/100 [00:00<00:33, 2.98it/s] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 36.06 MiB/4.29 GiB object_store_memory: 12%|█▏ | 12/100 [00:00<00:29, 2.98it/s] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 36.06 MiB/4.29 GiB object_store_memory: 13%|█▎ | 13/100 [00:00<00:02, 36.71it/s] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 38.47 MiB/4.29 GiB object_store_memory: 13%|█▎ | 13/100 [00:00<00:02, 36.71it/s] | |
Running: 4.0/16.0 CPU, 0.0/0.0 GPU, 9.62 MiB/4.29 GiB object_store_memory: 13%|█▎ | 13/100 [00:00<00:02, 36.71it/s] | |
Running: 4.0/16.0 CPU, 0.0/0.0 GPU, 9.62 MiB/4.29 GiB object_store_memory: 20%|██ | 20/100 [00:00<00:02, 35.10it/s] | |
Running: 4.0/16.0 CPU, 0.0/0.0 GPU, 9.62 MiB/4.29 GiB object_store_memory: 26%|██▌ | 26/100 [00:00<00:01, 39.92it/s] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 36.06 MiB/4.29 GiB object_store_memory: 26%|██▌ | 26/100 [00:00<00:01, 39.92it/s] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 38.47 MiB/4.29 GiB object_store_memory: 26%|██▌ | 26/100 [00:00<00:01, 39.92it/s] | |
Running: 4.0/16.0 CPU, 0.0/0.0 GPU, 9.62 MiB/4.29 GiB object_store_memory: 26%|██▌ | 26/100 [00:00<00:01, 39.92it/s] | |
Running: 4.0/16.0 CPU, 0.0/0.0 GPU, 9.62 MiB/4.29 GiB object_store_memory: 32%|███▏ | 32/100 [00:00<00:01, 34.91it/s] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 36.07 MiB/4.29 GiB object_store_memory: 38%|███▊ | 38/100 [00:01<00:01, 34.91it/s] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 36.07 MiB/4.29 GiB object_store_memory: 39%|███▉ | 39/100 [00:01<00:01, 41.91it/s] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 38.47 MiB/4.29 GiB object_store_memory: 39%|███▉ | 39/100 [00:01<00:01, 41.91it/s] | |
Running: 4.0/16.0 CPU, 0.0/0.0 GPU, 9.62 MiB/4.29 GiB object_store_memory: 39%|███▉ | 39/100 [00:01<00:01, 41.91it/s] | |
Running: 4.0/16.0 CPU, 0.0/0.0 GPU, 9.62 MiB/4.29 GiB object_store_memory: 45%|████▌ | 45/100 [00:01<00:01, 36.64it/s] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 36.07 MiB/4.29 GiB object_store_memory: 51%|█████ | 51/100 [00:01<00:01, 36.64it/s] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 36.07 MiB/4.29 GiB object_store_memory: 52%|█████▏ | 52/100 [00:01<00:01, 41.78it/s] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 38.47 MiB/4.29 GiB object_store_memory: 52%|█████▏ | 52/100 [00:01<00:01, 41.78it/s] | |
Running: 4.0/16.0 CPU, 0.0/0.0 GPU, 9.62 MiB/4.29 GiB object_store_memory: 52%|█████▏ | 52/100 [00:01<00:01, 41.78it/s] | |
Running: 4.0/16.0 CPU, 0.0/0.0 GPU, 9.62 MiB/4.29 GiB object_store_memory: 57%|█████▋ | 57/100 [00:01<00:01, 35.74it/s] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 36.06 MiB/4.29 GiB object_store_memory: 64%|██████▍ | 64/100 [00:01<00:01, 35.74it/s] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 36.06 MiB/4.29 GiB object_store_memory: 65%|██████▌ | 65/100 [00:01<00:00, 43.54it/s] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 38.47 MiB/4.29 GiB object_store_memory: 65%|██████▌ | 65/100 [00:01<00:00, 43.54it/s] | |
[2m[36m(ReadImage->MapBatches(images_to_bytes)->Write pid=3024)[0m 2023-04-13 16:20:04.683674: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 AVX512F FMA[32m [repeated 16x across cluster] (Ray deduplicates logs by default. Set RAY_DEDUP_LOGS=0 to disable log deduplication, or see https://docs.ray.io/en/master/ray-observability/ray-logging.html#log-deduplication for more options.)[0m | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 38.47 MiB/4.29 GiB object_store_memory: 65%|██████▌ | 65/100 [00:01<00:00, 43.54it/s] | |
[2m[36m(ReadImage->MapBatches(images_to_bytes)->Write pid=3024)[0m To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.[32m [repeated 16x across cluster][0m | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 38.47 MiB/4.29 GiB object_store_memory: 65%|██████▌ | 65/100 [00:01<00:00, 43.54it/s] | |
[2m[36m(ReadImage->MapBatches(images_to_bytes)->Write pid=1694)[0m 2023-04-13 16:18:32.073948: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/nvidia/lib:/usr/local/nvidia/lib64[32m [repeated 30x across cluster][0m | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 38.47 MiB/4.29 GiB object_store_memory: 65%|██████▌ | 65/100 [00:01<00:00, 43.54it/s] | |
[2m[36m(ReadImage->MapBatches(images_to_bytes)->Write pid=1694)[0m 2023-04-13 16:18:32.073959: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.[32m [repeated 15x across cluster][0m | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 38.47 MiB/4.29 GiB object_store_memory: 65%|██████▌ | 65/100 [00:01<00:00, 43.54it/s] | |
Running: 4.0/16.0 CPU, 0.0/0.0 GPU, 9.62 MiB/4.29 GiB object_store_memory: 65%|██████▌ | 65/100 [00:01<00:00, 43.54it/s] | |
Running: 4.0/16.0 CPU, 0.0/0.0 GPU, 9.62 MiB/4.29 GiB object_store_memory: 70%|███████ | 70/100 [00:01<00:00, 36.33it/s] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 36.07 MiB/4.29 GiB object_store_memory: 77%|███████▋ | 77/100 [00:02<00:00, 36.33it/s] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 36.07 MiB/4.29 GiB object_store_memory: 78%|███████▊ | 78/100 [00:02<00:00, 44.06it/s] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 38.47 MiB/4.29 GiB object_store_memory: 78%|███████▊ | 78/100 [00:02<00:00, 44.06it/s] | |
Running: 4.0/16.0 CPU, 0.0/0.0 GPU, 9.62 MiB/4.29 GiB object_store_memory: 78%|███████▊ | 78/100 [00:02<00:00, 44.06it/s] | |
Running: 4.0/16.0 CPU, 0.0/0.0 GPU, 9.62 MiB/4.29 GiB object_store_memory: 84%|████████▍ | 84/100 [00:02<00:00, 38.41it/s] | |
Running: 9.0/16.0 CPU, 0.0/0.0 GPU, 21.64 MiB/4.29 GiB object_store_memory: 90%|█████████ | 90/100 [00:02<00:00, 38.41it/s] | |
Running: 9.0/16.0 CPU, 0.0/0.0 GPU, 21.64 MiB/4.29 GiB object_store_memory: 91%|█████████ | 91/100 [00:02<00:00, 44.72it/s] | |
Running: 5.0/16.0 CPU, 0.0/0.0 GPU, 12.02 MiB/4.29 GiB object_store_memory: 91%|█████████ | 91/100 [00:02<00:00, 44.72it/s] | |
Running: 3.0/16.0 CPU, 0.0/0.0 GPU, 7.21 MiB/4.29 GiB object_store_memory: 95%|█████████▌| 95/100 [00:02<00:00, 44.72it/s] | |
Running: 3.0/16.0 CPU, 0.0/0.0 GPU, 7.21 MiB/4.29 GiB object_store_memory: 97%|█████████▋| 97/100 [00:02<00:00, 39.16it/s] | |
Running: 0.0/16.0 CPU, 0.0/0.0 GPU, 0.0 MiB/4.29 GiB object_store_memory: 97%|█████████▋| 97/100 [00:04<00:00, 39.16it/s] | |
2023-04-13 16:20:15,642 INFO streaming_executor.py:87 -- Executing DAG InputDataBuffer[Input] -> TaskPoolMapOperator[ReadImage->MapBatches(images_to_bytes)->Write] | |
2023-04-13 16:20:15,642 INFO streaming_executor.py:88 -- Execution config: ExecutionOptions(resource_limits=ExecutionResources(cpu=None, gpu=None, object_store_memory=None), locality_with_output=False, preserve_order=False, actor_locality_enabled=True, verbose_progress=False) | |
Running 0: 0%| | 0/51 [00:00<?, ?it/s] | |
Running: 0.0/16.0 CPU, 0.0/0.0 GPU, 0.0 MiB/4.29 GiB object_store_memory: 0%| | 0/51 [00:00<?, ?it/s] | |
Running: 6.0/16.0 CPU, 0.0/0.0 GPU, 6.84 MiB/4.29 GiB object_store_memory: 0%| | 0/51 [00:00<?, ?it/s] | |
Running: 6.0/16.0 CPU, 0.0/0.0 GPU, 6.84 MiB/4.29 GiB object_store_memory: 2%|▏ | 1/51 [00:00<00:05, 8.70it/s] | |
Running: 3.0/16.0 CPU, 0.0/0.0 GPU, 3.33 MiB/4.29 GiB object_store_memory: 20%|█▉ | 10/51 [00:00<00:04, 8.70it/s] | |
Running: 3.0/16.0 CPU, 0.0/0.0 GPU, 3.33 MiB/4.29 GiB object_store_memory: 22%|██▏ | 11/51 [00:00<00:00, 58.15it/s] | |
Running: 5.0/16.0 CPU, 0.0/0.0 GPU, 5.05 MiB/4.29 GiB object_store_memory: 45%|████▌ | 23/51 [00:00<00:00, 58.15it/s] | |
Running: 5.0/16.0 CPU, 0.0/0.0 GPU, 5.05 MiB/4.29 GiB object_store_memory: 47%|████▋ | 24/51 [00:00<00:00, 84.89it/s] | |
Running: 3.0/16.0 CPU, 0.0/0.0 GPU, 3.33 MiB/4.29 GiB object_store_memory: 67%|██████▋ | 34/51 [00:00<00:00, 84.89it/s] | |
Running: 3.0/16.0 CPU, 0.0/0.0 GPU, 3.33 MiB/4.29 GiB object_store_memory: 69%|██████▊ | 35/51 [00:00<00:00, 92.03it/s] | |
Running: 3.0/16.0 CPU, 0.0/0.0 GPU, 3.33 MiB/4.29 GiB object_store_memory: 94%|█████████▍| 48/51 [00:00<00:00, 104.62it/s] | |
[2m[36m(ReadImage->MapBatches(images_to_bytes)->Write pid=3397)[0m 2023-04-13 16:20:16.952631: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 AVX512F FMA[32m [repeated 3x across cluster][0m | |
Running: 3.0/16.0 CPU, 0.0/0.0 GPU, 3.33 MiB/4.29 GiB object_store_memory: 94%|█████████▍| 48/51 [00:01<00:00, 104.62it/s] | |
[2m[36m(ReadImage->MapBatches(images_to_bytes)->Write pid=3397)[0m To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.[32m [repeated 3x across cluster][0m | |
Running: 3.0/16.0 CPU, 0.0/0.0 GPU, 3.33 MiB/4.29 GiB object_store_memory: 94%|█████████▍| 48/51 [00:01<00:00, 104.62it/s] | |
[2m[36m(ReadImage->MapBatches(images_to_bytes)->Write pid=3025)[0m 2023-04-13 16:20:05.756329: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/nvidia/lib:/usr/local/nvidia/lib64[32m [repeated 6x across cluster][0m | |
Running: 3.0/16.0 CPU, 0.0/0.0 GPU, 3.33 MiB/4.29 GiB object_store_memory: 94%|█████████▍| 48/51 [00:01<00:00, 104.62it/s] | |
[2m[36m(ReadImage->MapBatches(images_to_bytes)->Write pid=3025)[0m 2023-04-13 16:20:05.756336: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.[32m [repeated 3x across cluster][0m | |
Running: 3.0/16.0 CPU, 0.0/0.0 GPU, 3.33 MiB/4.29 GiB object_store_memory: 94%|█████████▍| 48/51 [00:01<00:00, 104.62it/s] | |
Running: 1.0/16.0 CPU, 0.0/0.0 GPU, 0.9 MiB/4.29 GiB object_store_memory: 94%|█████████▍| 48/51 [00:03<00:00, 104.62it/s] | |
Running: 0.0/16.0 CPU, 0.0/0.0 GPU, 0.0 MiB/4.29 GiB object_store_memory: 98%|█████████▊| 50/51 [00:03<00:00, 104.62it/s] | |
2023-04-13 16:20:19,142 INFO streaming_executor.py:87 -- Executing DAG InputDataBuffer[Input] -> TaskPoolMapOperator[ReadRange->MapBatches(generate_features)->Write] | |
2023-04-13 16:20:19,142 INFO streaming_executor.py:88 -- Execution config: ExecutionOptions(resource_limits=ExecutionResources(cpu=None, gpu=None, object_store_memory=None), locality_with_output=False, preserve_order=False, actor_locality_enabled=True, verbose_progress=False) | |
Running 0: 0%| | 0/80 [00:00<?, ?it/s] | |
Running: 0.0/16.0 CPU, 0.0/0.0 GPU, 0.0 MiB/4.29 GiB object_store_memory: 0%| | 0/80 [00:00<?, ?it/s] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.29 GiB object_store_memory: 0%| | 0/80 [00:00<?, ?it/s] | |
[2m[36m(ReadRange->MapBatches(generate_features)->Write pid=3396)[0m 2023-04-13 16:20:52.665776: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 AVX512F FMA[32m [repeated 3x across cluster][0m | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.29 GiB object_store_memory: 0%| | 0/80 [00:33<?, ?it/s] | |
[2m[36m(ReadRange->MapBatches(generate_features)->Write pid=3396)[0m To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.[32m [repeated 3x across cluster][0m | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.29 GiB object_store_memory: 0%| | 0/80 [00:33<?, ?it/s] | |
[2m[36m(ReadImage->MapBatches(images_to_bytes)->Write pid=3399)[0m 2023-04-13 16:20:17.954977: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/nvidia/lib:/usr/local/nvidia/lib64[32m [repeated 6x across cluster][0m | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.29 GiB object_store_memory: 0%| | 0/80 [00:33<?, ?it/s] | |
[2m[36m(ReadImage->MapBatches(images_to_bytes)->Write pid=3399)[0m 2023-04-13 16:20:17.954984: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.[32m [repeated 3x across cluster][0m | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.29 GiB object_store_memory: 0%| | 0/80 [00:33<?, ?it/s] | |
Running: 13.0/16.0 CPU, 0.0/0.0 GPU, 13.0 MiB/4.29 GiB object_store_memory: 0%| | 0/80 [01:03<?, ?it/s] | |
Running: 13.0/16.0 CPU, 0.0/0.0 GPU, 13.0 MiB/4.29 GiB object_store_memory: 1%|▏ | 1/80 [01:03<1:24:05, 63.87s/it] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.29 GiB object_store_memory: 4%|▍ | 3/80 [01:03<1:21:58, 63.87s/it] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.29 GiB object_store_memory: 5%|▌ | 4/80 [01:04<15:22, 12.14s/it] | |
Running: 13.0/16.0 CPU, 0.0/0.0 GPU, 13.0 MiB/4.29 GiB object_store_memory: 6%|▋ | 5/80 [01:04<15:10, 12.14s/it] | |
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 14.0 MiB/4.29 GiB object_store_memory: 8%|▊ | 6/80 [01:04<14:58, 12.14s/it] | |
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 14.0 MiB/4.29 GiB object_store_memory: 9%|▉ | 7/80 [01:04<06:52, 5.65s/it] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.29 GiB object_store_memory: 10%|█ | 8/80 [01:04<06:46, 5.65s/it] | |
Running: 13.0/16.0 CPU, 0.0/0.0 GPU, 13.0 MiB/4.29 GiB object_store_memory: 10%|█ | 8/80 [01:04<06:46, 5.65s/it] | |
Running: 13.0/16.0 CPU, 0.0/0.0 GPU, 13.0 MiB/4.29 GiB object_store_memory: 12%|█▎ | 10/80 [01:04<03:45, 3.23s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.29 GiB object_store_memory: 14%|█▍ | 11/80 [01:04<03:42, 3.23s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.29 GiB object_store_memory: 15%|█▌ | 12/80 [01:04<02:36, 2.30s/it] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.29 GiB object_store_memory: 15%|█▌ | 12/80 [01:04<02:36, 2.30s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.29 GiB object_store_memory: 15%|█▌ | 12/80 [01:08<02:36, 2.30s/it] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.29 GiB object_store_memory: 16%|█▋ | 13/80 [01:08<02:33, 2.30s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.29 GiB object_store_memory: 16%|█▋ | 13/80 [01:08<02:33, 2.30s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.29 GiB object_store_memory: 18%|█▊ | 14/80 [01:08<02:27, 2.24s/it] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.29 GiB object_store_memory: 18%|█▊ | 14/80 [01:08<02:27, 2.24s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.29 GiB object_store_memory: 19%|█▉ | 15/80 [01:09<02:25, 2.24s/it] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.29 GiB object_store_memory: 19%|█▉ | 15/80 [01:09<02:25, 2.24s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.29 GiB object_store_memory: 19%|█▉ | 15/80 [01:09<02:25, 2.24s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.29 GiB object_store_memory: 20%|██ | 16/80 [01:09<01:48, 1.69s/it] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.29 GiB object_store_memory: 20%|██ | 16/80 [01:09<01:48, 1.69s/it] | |
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 14.0 MiB/4.29 GiB object_store_memory: 20%|██ | 16/80 [02:07<01:48, 1.69s/it] | |
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 14.0 MiB/4.29 GiB object_store_memory: 21%|██▏ | 17/80 [02:07<12:02, 11.46s/it] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.29 GiB object_store_memory: 22%|██▎ | 18/80 [02:07<11:50, 11.46s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.29 GiB object_store_memory: 22%|██▎ | 18/80 [02:07<11:50, 11.46s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.29 GiB object_store_memory: 24%|██▍ | 19/80 [02:07<07:48, 7.68s/it] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.29 GiB object_store_memory: 24%|██▍ | 19/80 [02:07<07:48, 7.68s/it] | |
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 14.0 MiB/4.29 GiB object_store_memory: 24%|██▍ | 19/80 [02:07<07:48, 7.68s/it] | |
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 14.0 MiB/4.29 GiB object_store_memory: 25%|██▌ | 20/80 [02:07<06:14, 6.25s/it] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.29 GiB object_store_memory: 26%|██▋ | 21/80 [02:07<06:08, 6.25s/it] | |
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 14.0 MiB/4.29 GiB object_store_memory: 26%|██▋ | 21/80 [02:07<06:08, 6.25s/it] | |
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 14.0 MiB/4.29 GiB object_store_memory: 28%|██▊ | 22/80 [02:07<03:56, 4.07s/it] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.29 GiB object_store_memory: 29%|██▉ | 23/80 [02:07<03:51, 4.07s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.29 GiB object_store_memory: 29%|██▉ | 23/80 [02:07<03:51, 4.07s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.29 GiB object_store_memory: 30%|███ | 24/80 [02:07<02:33, 2.74s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.29 GiB object_store_memory: 31%|███▏ | 25/80 [02:08<02:02, 2.23s/it] | |
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 14.0 MiB/4.29 GiB object_store_memory: 31%|███▏ | 25/80 [02:08<02:02, 2.23s/it] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.29 GiB object_store_memory: 34%|███▍ | 27/80 [02:08<01:58, 2.23s/it] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.29 GiB object_store_memory: 35%|███▌ | 28/80 [02:08<01:05, 1.27s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.29 GiB object_store_memory: 35%|███▌ | 28/80 [02:08<01:05, 1.27s/it] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.29 GiB object_store_memory: 35%|███▌ | 28/80 [02:08<01:05, 1.27s/it] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.29 GiB object_store_memory: 36%|███▋ | 29/80 [02:11<01:25, 1.67s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.29 GiB object_store_memory: 36%|███▋ | 29/80 [02:11<01:25, 1.67s/it] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.29 GiB object_store_memory: 36%|███▋ | 29/80 [02:12<01:25, 1.67s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.29 GiB object_store_memory: 36%|███▋ | 29/80 [02:12<01:25, 1.67s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.29 GiB object_store_memory: 38%|███▊ | 30/80 [02:12<01:12, 1.44s/it] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.29 GiB object_store_memory: 38%|███▊ | 30/80 [02:12<01:12, 1.44s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.29 GiB object_store_memory: 38%|███▊ | 30/80 [02:12<01:12, 1.44s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.29 GiB object_store_memory: 39%|███▉ | 31/80 [02:12<00:57, 1.17s/it] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.29 GiB object_store_memory: 39%|███▉ | 31/80 [02:12<00:57, 1.17s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.29 GiB object_store_memory: 39%|███▉ | 31/80 [02:13<00:57, 1.17s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.29 GiB object_store_memory: 40%|████ | 32/80 [02:13<00:51, 1.06s/it] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.29 GiB object_store_memory: 40%|████ | 32/80 [02:13<00:51, 1.06s/it] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.29 GiB object_store_memory: 41%|████▏ | 33/80 [03:10<12:13, 15.60s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.29 GiB object_store_memory: 41%|████▏ | 33/80 [03:10<12:13, 15.60s/it] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.29 GiB object_store_memory: 41%|████▏ | 33/80 [03:11<12:13, 15.60s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.29 GiB object_store_memory: 41%|████▏ | 33/80 [03:11<12:13, 15.60s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.29 GiB object_store_memory: 42%|████▎ | 34/80 [03:11<08:48, 11.49s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.29 GiB object_store_memory: 44%|████▍ | 35/80 [03:11<06:14, 8.33s/it] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.29 GiB object_store_memory: 44%|████▍ | 35/80 [03:11<06:14, 8.33s/it] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.29 GiB object_store_memory: 45%|████▌ | 36/80 [03:11<04:25, 6.03s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.29 GiB object_store_memory: 45%|████▌ | 36/80 [03:11<04:25, 6.03s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.29 GiB object_store_memory: 46%|████▋ | 37/80 [03:11<03:05, 4.32s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.29 GiB object_store_memory: 48%|████▊ | 38/80 [03:11<02:09, 3.09s/it] | |
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 14.0 MiB/4.29 GiB object_store_memory: 48%|████▊ | 38/80 [03:11<02:09, 3.09s/it] | |
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 14.0 MiB/4.29 GiB object_store_memory: 49%|████▉ | 39/80 [03:11<01:30, 2.21s/it] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.29 GiB object_store_memory: 50%|█████ | 40/80 [03:12<01:28, 2.21s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.29 GiB object_store_memory: 50%|█████ | 40/80 [03:12<01:28, 2.21s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.29 GiB object_store_memory: 51%|█████▏ | 41/80 [03:12<00:48, 1.25s/it] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.29 GiB object_store_memory: 51%|█████▏ | 41/80 [03:12<00:48, 1.25s/it] | |
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 14.0 MiB/4.29 GiB object_store_memory: 51%|█████▏ | 41/80 [03:12<00:48, 1.25s/it] | |
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 14.0 MiB/4.29 GiB object_store_memory: 52%|█████▎ | 42/80 [03:12<00:37, 1.00it/s] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.29 GiB object_store_memory: 54%|█████▍ | 43/80 [03:12<00:36, 1.00it/s] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.29 GiB object_store_memory: 55%|█████▌ | 44/80 [03:12<00:21, 1.65it/s] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.29 GiB object_store_memory: 55%|█████▌ | 44/80 [03:12<00:21, 1.65it/s] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.29 GiB object_store_memory: 55%|█████▌ | 44/80 [03:15<00:21, 1.65it/s] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.29 GiB object_store_memory: 56%|█████▋ | 45/80 [03:15<00:21, 1.65it/s] | |
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 14.0 MiB/4.29 GiB object_store_memory: 56%|█████▋ | 45/80 [03:15<00:21, 1.65it/s] | |
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 14.0 MiB/4.29 GiB object_store_memory: 57%|█████▊ | 46/80 [03:15<00:34, 1.03s/it] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.29 GiB object_store_memory: 59%|█████▉ | 47/80 [03:16<00:33, 1.03s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.29 GiB object_store_memory: 59%|█████▉ | 47/80 [03:17<00:33, 1.03s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.29 GiB object_store_memory: 60%|██████ | 48/80 [03:17<00:28, 1.14it/s] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.29 GiB object_store_memory: 60%|██████ | 48/80 [03:17<00:28, 1.14it/s] | |
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 14.0 MiB/4.29 GiB object_store_memory: 60%|██████ | 48/80 [04:14<00:28, 1.14it/s] | |
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 14.0 MiB/4.29 GiB object_store_memory: 61%|██████▏ | 49/80 [04:14<06:16, 12.14s/it] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.29 GiB object_store_memory: 62%|██████▎ | 50/80 [04:14<06:04, 12.14s/it] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.29 GiB object_store_memory: 64%|██████▍ | 51/80 [04:15<03:45, 7.79s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.29 GiB object_store_memory: 64%|██████▍ | 51/80 [04:15<03:45, 7.79s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.29 GiB object_store_memory: 65%|██████▌ | 52/80 [04:15<02:53, 6.21s/it] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.29 GiB object_store_memory: 65%|██████▌ | 52/80 [04:15<02:53, 6.21s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.29 GiB object_store_memory: 65%|██████▌ | 52/80 [04:15<02:53, 6.21s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.29 GiB object_store_memory: 66%|██████▋ | 53/80 [04:15<02:11, 4.88s/it] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.29 GiB object_store_memory: 66%|██████▋ | 53/80 [04:15<02:11, 4.88s/it] | |
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 14.0 MiB/4.29 GiB object_store_memory: 66%|██████▋ | 53/80 [04:15<02:11, 4.88s/it] | |
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 14.0 MiB/4.29 GiB object_store_memory: 68%|██████▊ | 54/80 [04:15<01:37, 3.73s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.29 GiB object_store_memory: 69%|██████▉ | 55/80 [04:15<01:33, 3.73s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.29 GiB object_store_memory: 70%|███████ | 56/80 [04:15<00:53, 2.22s/it] | |
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 14.0 MiB/4.29 GiB object_store_memory: 70%|███████ | 56/80 [04:15<00:53, 2.22s/it] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.29 GiB object_store_memory: 72%|███████▎ | 58/80 [04:15<00:48, 2.22s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.29 GiB object_store_memory: 72%|███████▎ | 58/80 [04:16<00:48, 2.22s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.29 GiB object_store_memory: 74%|███████▍ | 59/80 [04:16<00:25, 1.23s/it] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.29 GiB object_store_memory: 74%|███████▍ | 59/80 [04:16<00:25, 1.23s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.29 GiB object_store_memory: 74%|███████▍ | 59/80 [04:16<00:25, 1.23s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.29 GiB object_store_memory: 75%|███████▌ | 60/80 [04:16<00:20, 1.04s/it] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.29 GiB object_store_memory: 75%|███████▌ | 60/80 [04:16<00:20, 1.04s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.29 GiB object_store_memory: 75%|███████▌ | 60/80 [04:18<00:20, 1.04s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.29 GiB object_store_memory: 76%|███████▋ | 61/80 [04:18<00:25, 1.36s/it] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.29 GiB object_store_memory: 76%|███████▋ | 61/80 [04:19<00:25, 1.36s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.29 GiB object_store_memory: 76%|███████▋ | 61/80 [04:19<00:25, 1.36s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.29 GiB object_store_memory: 78%|███████▊ | 62/80 [04:19<00:22, 1.23s/it] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.29 GiB object_store_memory: 78%|███████▊ | 62/80 [04:19<00:22, 1.23s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.29 GiB object_store_memory: 78%|███████▊ | 62/80 [04:19<00:22, 1.23s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.29 GiB object_store_memory: 79%|███████▉ | 63/80 [04:19<00:16, 1.02it/s] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.29 GiB object_store_memory: 79%|███████▉ | 63/80 [04:19<00:16, 1.02it/s] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.29 GiB object_store_memory: 79%|███████▉ | 63/80 [04:21<00:16, 1.02it/s] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.29 GiB object_store_memory: 80%|████████ | 64/80 [04:21<00:17, 1.12s/it] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.29 GiB object_store_memory: 80%|████████ | 64/80 [04:21<00:17, 1.12s/it] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.29 GiB object_store_memory: 81%|████████▏ | 65/80 [05:18<04:04, 16.30s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.29 GiB object_store_memory: 81%|████████▏ | 65/80 [05:18<04:04, 16.30s/it] | |
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 14.0 MiB/4.29 GiB object_store_memory: 81%|████████▏ | 65/80 [05:18<04:04, 16.30s/it] | |
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 14.0 MiB/4.29 GiB object_store_memory: 82%|████████▎ | 66/80 [05:18<02:45, 11.84s/it] | |
Running: 13.0/16.0 CPU, 0.0/0.0 GPU, 13.0 MiB/4.29 GiB object_store_memory: 82%|████████▎ | 66/80 [05:19<02:45, 11.84s/it] | |
Running: 13.0/16.0 CPU, 0.0/0.0 GPU, 13.0 MiB/4.29 GiB object_store_memory: 84%|████████▍ | 67/80 [05:19<01:50, 8.50s/it] | |
Running: 11.0/16.0 CPU, 0.0/0.0 GPU, 11.0 MiB/4.29 GiB object_store_memory: 84%|████████▍ | 67/80 [05:19<01:50, 8.50s/it] | |
Running: 11.0/16.0 CPU, 0.0/0.0 GPU, 11.0 MiB/4.29 GiB object_store_memory: 85%|████████▌ | 68/80 [05:19<01:12, 6.07s/it] | |
Running: 9.0/16.0 CPU, 0.0/0.0 GPU, 9.0 MiB/4.29 GiB object_store_memory: 86%|████████▋ | 69/80 [05:19<01:06, 6.07s/it] | |
Running: 9.0/16.0 CPU, 0.0/0.0 GPU, 9.0 MiB/4.29 GiB object_store_memory: 88%|████████▊ | 70/80 [05:19<00:33, 3.35s/it] | |
Running: 8.0/16.0 CPU, 0.0/0.0 GPU, 8.0 MiB/4.29 GiB object_store_memory: 89%|████████▉ | 71/80 [05:19<00:30, 3.35s/it] | |
Running: 8.0/16.0 CPU, 0.0/0.0 GPU, 8.0 MiB/4.29 GiB object_store_memory: 90%|█████████ | 72/80 [05:19<00:16, 2.05s/it] | |
Running: 6.0/16.0 CPU, 0.0/0.0 GPU, 6.0 MiB/4.29 GiB object_store_memory: 90%|█████████ | 72/80 [05:19<00:16, 2.05s/it] | |
Running: 6.0/16.0 CPU, 0.0/0.0 GPU, 6.0 MiB/4.29 GiB object_store_memory: 92%|█████████▎| 74/80 [05:19<00:08, 1.35s/it] | |
Running: 4.0/16.0 CPU, 0.0/0.0 GPU, 4.0 MiB/4.29 GiB object_store_memory: 92%|█████████▎| 74/80 [05:19<00:08, 1.35s/it] | |
Running: 4.0/16.0 CPU, 0.0/0.0 GPU, 4.0 MiB/4.29 GiB object_store_memory: 95%|█████████▌| 76/80 [05:19<00:03, 1.06it/s] | |
Running: 3.0/16.0 CPU, 0.0/0.0 GPU, 3.0 MiB/4.29 GiB object_store_memory: 95%|█████████▌| 76/80 [05:21<00:03, 1.06it/s] | |
Running: 2.0/16.0 CPU, 0.0/0.0 GPU, 2.0 MiB/4.29 GiB object_store_memory: 96%|█████████▋| 77/80 [05:21<00:02, 1.06it/s] | |
Running: 2.0/16.0 CPU, 0.0/0.0 GPU, 2.0 MiB/4.29 GiB object_store_memory: 98%|█████████▊| 78/80 [05:21<00:01, 1.09it/s] | |
Running: 1.0/16.0 CPU, 0.0/0.0 GPU, 1.0 MiB/4.29 GiB object_store_memory: 98%|█████████▊| 78/80 [05:21<00:01, 1.09it/s] | |
Running: 1.0/16.0 CPU, 0.0/0.0 GPU, 1.0 MiB/4.29 GiB object_store_memory: 99%|█████████▉| 79/80 [05:21<00:00, 1.31it/s] | |
Running: 0.0/16.0 CPU, 0.0/0.0 GPU, 0.0 MiB/4.29 GiB object_store_memory: 99%|█████████▉| 79/80 [05:22<00:00, 1.31it/s] | |
Running: 0.0/16.0 CPU, 0.0/0.0 GPU, 0.0 MiB/4.29 GiB object_store_memory: 100%|██████████| 80/80 [05:22<00:00, 1.22it/s] | |
2023-04-13 16:25:41,909 INFO streaming_executor.py:87 -- Executing DAG InputDataBuffer[Input] -> TaskPoolMapOperator[ReadRange->MapBatches(generate_features)->Write] | |
2023-04-13 16:25:41,909 INFO streaming_executor.py:88 -- Execution config: ExecutionOptions(resource_limits=ExecutionResources(cpu=None, gpu=None, object_store_memory=None), locality_with_output=False, preserve_order=False, actor_locality_enabled=True, verbose_progress=False) | |
Running 0: 0%| | 0/80 [00:00<?, ?it/s] | |
Running: 0.0/16.0 CPU, 0.0/0.0 GPU, 0.0 MiB/4.29 GiB object_store_memory: 0%| | 0/80 [00:00<?, ?it/s] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.29 GiB object_store_memory: 0%| | 0/80 [00:00<?, ?it/s] | |
[2m[36m(ReadRange->MapBatches(generate_features)->Write pid=7423)[0m 2023-04-13 16:25:54.395985: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 AVX512F FMA[32m [repeated 4x across cluster][0m | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.29 GiB object_store_memory: 0%| | 0/80 [00:12<?, ?it/s] | |
[2m[36m(ReadRange->MapBatches(generate_features)->Write pid=7423)[0m To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.[32m [repeated 4x across cluster][0m | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.29 GiB object_store_memory: 0%| | 0/80 [00:12<?, ?it/s] | |
[2m[36m(ReadRange->MapBatches(generate_features)->Write pid=3652)[0m 2023-04-13 16:20:55.354167: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/nvidia/lib:/usr/local/nvidia/lib64[32m [repeated 8x across cluster][0m | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.29 GiB object_store_memory: 0%| | 0/80 [00:12<?, ?it/s] | |
[2m[36m(ReadRange->MapBatches(generate_features)->Write pid=3652)[0m 2023-04-13 16:20:55.354178: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.[32m [repeated 4x across cluster][0m | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.29 GiB object_store_memory: 0%| | 0/80 [00:12<?, ?it/s] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.29 GiB object_store_memory: 1%|▏ | 1/80 [00:39<51:52, 39.40s/it] | |
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 14.0 MiB/4.29 GiB object_store_memory: 2%|▎ | 2/80 [00:39<51:13, 39.40s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.29 GiB object_store_memory: 2%|▎ | 2/80 [00:39<51:13, 39.40s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.29 GiB object_store_memory: 4%|▍ | 3/80 [00:39<13:09, 10.26s/it] | |
Running: 11.0/16.0 CPU, 0.0/0.0 GPU, 11.0 MiB/4.29 GiB object_store_memory: 5%|▌ | 4/80 [00:39<12:59, 10.26s/it] | |
Running: 11.0/16.0 CPU, 0.0/0.0 GPU, 11.0 MiB/4.29 GiB object_store_memory: 6%|▋ | 5/80 [00:39<06:17, 5.04s/it] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.29 GiB object_store_memory: 11%|█▏ | 9/80 [00:39<05:57, 5.04s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.29 GiB object_store_memory: 11%|█▏ | 9/80 [00:39<05:57, 5.04s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.29 GiB object_store_memory: 12%|█▎ | 10/80 [00:39<02:06, 1.80s/it] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.29 GiB object_store_memory: 12%|█▎ | 10/80 [00:40<02:06, 1.80s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.29 GiB object_store_memory: 12%|█▎ | 10/80 [00:40<02:06, 1.80s/it] | |
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 14.0 MiB/4.29 GiB object_store_memory: 14%|█▍ | 11/80 [00:40<02:04, 1.80s/it] | |
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 14.0 MiB/4.29 GiB object_store_memory: 15%|█▌ | 12/80 [00:40<01:32, 1.36s/it] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.29 GiB object_store_memory: 16%|█▋ | 13/80 [00:40<01:31, 1.36s/it] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.29 GiB object_store_memory: 18%|█▊ | 14/80 [00:43<01:38, 1.49s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.29 GiB object_store_memory: 18%|█▊ | 14/80 [00:43<01:38, 1.49s/it] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.29 GiB object_store_memory: 18%|█▊ | 14/80 [00:44<01:38, 1.49s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.29 GiB object_store_memory: 18%|█▊ | 14/80 [00:44<01:38, 1.49s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.29 GiB object_store_memory: 19%|█▉ | 15/80 [00:44<01:23, 1.28s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.29 GiB object_store_memory: 20%|██ | 16/80 [00:44<01:07, 1.05s/it] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.29 GiB object_store_memory: 20%|██ | 16/80 [00:44<01:07, 1.05s/it] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.29 GiB object_store_memory: 21%|██▏ | 17/80 [01:18<08:39, 8.24s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.29 GiB object_store_memory: 21%|██▏ | 17/80 [01:18<08:39, 8.24s/it] | |
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 14.0 MiB/4.29 GiB object_store_memory: 21%|██▏ | 17/80 [01:18<08:39, 8.24s/it] | |
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 14.0 MiB/4.29 GiB object_store_memory: 22%|██▎ | 18/80 [01:18<06:32, 6.33s/it] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.29 GiB object_store_memory: 24%|██▍ | 19/80 [01:18<06:26, 6.33s/it] | |
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 14.0 MiB/4.29 GiB object_store_memory: 24%|██▍ | 19/80 [01:19<06:26, 6.33s/it] | |
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 14.0 MiB/4.29 GiB object_store_memory: 25%|██▌ | 20/80 [01:19<03:49, 3.83s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.29 GiB object_store_memory: 26%|██▋ | 21/80 [01:19<03:46, 3.83s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.29 GiB object_store_memory: 28%|██▊ | 22/80 [01:19<02:22, 2.45s/it] | |
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 14.0 MiB/4.29 GiB object_store_memory: 28%|██▊ | 22/80 [01:19<02:22, 2.45s/it] | |
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 14.0 MiB/4.29 GiB object_store_memory: 30%|███ | 24/80 [01:19<01:31, 1.63s/it] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.29 GiB object_store_memory: 30%|███ | 24/80 [01:19<01:31, 1.63s/it] | |
Running: 12.0/16.0 CPU, 0.0/0.0 GPU, 12.0 MiB/4.29 GiB object_store_memory: 30%|███ | 24/80 [01:19<01:31, 1.63s/it] | |
Running: 12.0/16.0 CPU, 0.0/0.0 GPU, 12.0 MiB/4.29 GiB object_store_memory: 32%|███▎ | 26/80 [01:19<01:01, 1.15s/it] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.29 GiB object_store_memory: 35%|███▌ | 28/80 [01:19<00:59, 1.15s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.29 GiB object_store_memory: 35%|███▌ | 28/80 [01:19<00:59, 1.15s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.29 GiB object_store_memory: 36%|███▋ | 29/80 [01:19<00:35, 1.42it/s] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.29 GiB object_store_memory: 36%|███▋ | 29/80 [01:19<00:35, 1.42it/s] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.29 GiB object_store_memory: 36%|███▋ | 29/80 [01:22<00:35, 1.42it/s] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.29 GiB object_store_memory: 38%|███▊ | 30/80 [01:22<00:35, 1.42it/s] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.29 GiB object_store_memory: 38%|███▊ | 30/80 [01:22<00:35, 1.42it/s] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.29 GiB object_store_memory: 39%|███▉ | 31/80 [01:22<00:46, 1.05it/s] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.29 GiB object_store_memory: 40%|████ | 32/80 [01:23<00:38, 1.23it/s] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.29 GiB object_store_memory: 40%|████ | 32/80 [01:23<00:38, 1.23it/s] | |
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 14.0 MiB/4.29 GiB object_store_memory: 40%|████ | 32/80 [01:58<00:38, 1.23it/s] | |
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 14.0 MiB/4.29 GiB object_store_memory: 41%|████▏ | 33/80 [01:58<05:49, 7.44s/it] | |
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 14.0 MiB/4.29 GiB object_store_memory: 44%|████▍ | 35/80 [01:58<03:36, 4.81s/it] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.29 GiB object_store_memory: 45%|████▌ | 36/80 [01:58<03:31, 4.81s/it] | |
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 14.0 MiB/4.29 GiB object_store_memory: 45%|████▌ | 36/80 [01:58<03:31, 4.81s/it] | |
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 14.0 MiB/4.29 GiB object_store_memory: 46%|████▋ | 37/80 [01:58<02:18, 3.23s/it] | |
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 14.0 MiB/4.29 GiB object_store_memory: 49%|████▉ | 39/80 [01:58<01:30, 2.20s/it] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.29 GiB object_store_memory: 50%|█████ | 40/80 [01:58<01:27, 2.20s/it] | |
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 14.0 MiB/4.29 GiB object_store_memory: 50%|█████ | 40/80 [01:58<01:27, 2.20s/it] | |
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 14.0 MiB/4.29 GiB object_store_memory: 51%|█████▏ | 41/80 [01:58<00:59, 1.53s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.29 GiB object_store_memory: 52%|█████▎ | 42/80 [01:58<00:58, 1.53s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.29 GiB object_store_memory: 54%|█████▍ | 43/80 [01:58<00:39, 1.07s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.29 GiB object_store_memory: 56%|█████▋ | 45/80 [01:59<00:27, 1.29it/s] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.29 GiB object_store_memory: 56%|█████▋ | 45/80 [01:59<00:27, 1.29it/s] | |
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 14.0 MiB/4.29 GiB object_store_memory: 56%|█████▋ | 45/80 [02:01<00:27, 1.29it/s] | |
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 14.0 MiB/4.29 GiB object_store_memory: 59%|█████▉ | 47/80 [02:01<00:31, 1.05it/s] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.29 GiB object_store_memory: 59%|█████▉ | 47/80 [02:01<00:31, 1.05it/s] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.29 GiB object_store_memory: 59%|█████▉ | 47/80 [02:02<00:31, 1.05it/s] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.29 GiB object_store_memory: 60%|██████ | 48/80 [02:02<00:27, 1.17it/s] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.29 GiB object_store_memory: 60%|██████ | 48/80 [02:02<00:27, 1.17it/s] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.29 GiB object_store_memory: 61%|██████▏ | 49/80 [02:37<04:06, 7.96s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.29 GiB object_store_memory: 61%|██████▏ | 49/80 [02:37<04:06, 7.96s/it] | |
Running: 12.0/16.0 CPU, 0.0/0.0 GPU, 12.0 MiB/4.29 GiB object_store_memory: 61%|██████▏ | 49/80 [02:38<04:06, 7.96s/it] | |
Running: 12.0/16.0 CPU, 0.0/0.0 GPU, 12.0 MiB/4.29 GiB object_store_memory: 62%|██████▎ | 50/80 [02:38<03:05, 6.19s/it] | |
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 14.0 MiB/4.29 GiB object_store_memory: 66%|██████▋ | 53/80 [02:38<02:47, 6.19s/it] | |
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 14.0 MiB/4.29 GiB object_store_memory: 68%|██████▊ | 54/80 [02:38<01:10, 2.72s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.29 GiB object_store_memory: 69%|██████▉ | 55/80 [02:38<01:07, 2.72s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.29 GiB object_store_memory: 70%|███████ | 56/80 [02:38<00:46, 1.95s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.29 GiB object_store_memory: 72%|███████▎ | 58/80 [02:38<00:31, 1.42s/it] | |
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 14.0 MiB/4.29 GiB object_store_memory: 74%|███████▍ | 59/80 [02:38<00:29, 1.42s/it] | |
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 14.0 MiB/4.29 GiB object_store_memory: 75%|███████▌ | 60/80 [02:38<00:20, 1.03s/it] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.29 GiB object_store_memory: 76%|███████▋ | 61/80 [02:38<00:19, 1.03s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.29 GiB object_store_memory: 76%|███████▋ | 61/80 [02:41<00:19, 1.03s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.29 GiB object_store_memory: 78%|███████▊ | 62/80 [02:41<00:19, 1.09s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.29 GiB object_store_memory: 79%|███████▉ | 63/80 [02:41<00:15, 1.09it/s] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.29 GiB object_store_memory: 79%|███████▉ | 63/80 [02:41<00:15, 1.09it/s] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.29 GiB object_store_memory: 79%|███████▉ | 63/80 [02:42<00:15, 1.09it/s] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.29 GiB object_store_memory: 80%|████████ | 64/80 [02:42<00:14, 1.13it/s] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.29 GiB object_store_memory: 80%|████████ | 64/80 [02:42<00:14, 1.13it/s] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.29 GiB object_store_memory: 81%|████████▏ | 65/80 [03:16<02:06, 8.42s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.29 GiB object_store_memory: 81%|████████▏ | 65/80 [03:16<02:06, 8.42s/it] | |
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 14.0 MiB/4.29 GiB object_store_memory: 81%|████████▏ | 65/80 [03:17<02:06, 8.42s/it] | |
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 14.0 MiB/4.29 GiB object_store_memory: 82%|████████▎ | 66/80 [03:17<01:30, 6.47s/it] | |
Running: 13.0/16.0 CPU, 0.0/0.0 GPU, 13.0 MiB/4.29 GiB object_store_memory: 82%|████████▎ | 66/80 [03:17<01:30, 6.47s/it] | |
Running: 13.0/16.0 CPU, 0.0/0.0 GPU, 13.0 MiB/4.29 GiB object_store_memory: 84%|████████▍ | 67/80 [03:17<01:03, 4.87s/it] | |
Running: 12.0/16.0 CPU, 0.0/0.0 GPU, 12.0 MiB/4.29 GiB object_store_memory: 84%|████████▍ | 67/80 [03:17<01:03, 4.87s/it] | |
Running: 12.0/16.0 CPU, 0.0/0.0 GPU, 12.0 MiB/4.29 GiB object_store_memory: 85%|████████▌ | 68/80 [03:17<00:43, 3.59s/it] | |
Running: 9.0/16.0 CPU, 0.0/0.0 GPU, 9.0 MiB/4.29 GiB object_store_memory: 85%|████████▌ | 68/80 [03:17<00:43, 3.59s/it] | |
Running: 9.0/16.0 CPU, 0.0/0.0 GPU, 9.0 MiB/4.29 GiB object_store_memory: 86%|████████▋ | 69/80 [03:17<00:28, 2.63s/it] | |
Running: 8.0/16.0 CPU, 0.0/0.0 GPU, 8.0 MiB/4.29 GiB object_store_memory: 89%|████████▉ | 71/80 [03:17<00:23, 2.63s/it] | |
Running: 8.0/16.0 CPU, 0.0/0.0 GPU, 8.0 MiB/4.29 GiB object_store_memory: 90%|█████████ | 72/80 [03:17<00:09, 1.22s/it] | |
Running: 7.0/16.0 CPU, 0.0/0.0 GPU, 7.0 MiB/4.29 GiB object_store_memory: 90%|█████████ | 72/80 [03:17<00:09, 1.22s/it] | |
Running: 5.0/16.0 CPU, 0.0/0.0 GPU, 5.0 MiB/4.29 GiB object_store_memory: 91%|█████████▏| 73/80 [03:17<00:08, 1.22s/it] | |
Running: 5.0/16.0 CPU, 0.0/0.0 GPU, 5.0 MiB/4.29 GiB object_store_memory: 92%|█████████▎| 74/80 [03:17<00:05, 1.19it/s] | |
Running: 4.0/16.0 CPU, 0.0/0.0 GPU, 4.0 MiB/4.29 GiB object_store_memory: 94%|█████████▍| 75/80 [03:18<00:04, 1.19it/s] | |
Running: 4.0/16.0 CPU, 0.0/0.0 GPU, 4.0 MiB/4.29 GiB object_store_memory: 95%|█████████▌| 76/80 [03:18<00:02, 1.67it/s] | |
Running: 3.0/16.0 CPU, 0.0/0.0 GPU, 3.0 MiB/4.29 GiB object_store_memory: 95%|█████████▌| 76/80 [03:18<00:02, 1.67it/s] | |
Running: 1.0/16.0 CPU, 0.0/0.0 GPU, 1.0 MiB/4.29 GiB object_store_memory: 96%|█████████▋| 77/80 [03:19<00:01, 1.67it/s] | |
Running: 1.0/16.0 CPU, 0.0/0.0 GPU, 1.0 MiB/4.29 GiB object_store_memory: 98%|█████████▊| 78/80 [03:19<00:01, 1.76it/s] | |
Running: 0.0/16.0 CPU, 0.0/0.0 GPU, 0.0 MiB/4.29 GiB object_store_memory: 99%|█████████▉| 79/80 [03:19<00:00, 1.76it/s] | |
Running: 0.0/16.0 CPU, 0.0/0.0 GPU, 0.0 MiB/4.29 GiB object_store_memory: 100%|██████████| 80/80 [03:19<00:00, 2.08it/s] | |
2023-04-13 16:29:01,700 INFO streaming_executor.py:87 -- Executing DAG InputDataBuffer[Input] -> TaskPoolMapOperator[ReadRange->MapBatches(generate_features)->Write] | |
2023-04-13 16:29:01,700 INFO streaming_executor.py:88 -- Execution config: ExecutionOptions(resource_limits=ExecutionResources(cpu=None, gpu=None, object_store_memory=None), locality_with_output=False, preserve_order=False, actor_locality_enabled=True, verbose_progress=False) | |
Running 0: 0%| | 0/32 [00:00<?, ?it/s] | |
Running: 0.0/16.0 CPU, 0.0/0.0 GPU, 0.0 MiB/4.29 GiB object_store_memory: 0%| | 0/32 [00:00<?, ?it/s] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 4.0 MiB/4.29 GiB object_store_memory: 0%| | 0/32 [00:00<?, ?it/s] | |
[2m[36m(ReadRange->MapBatches(generate_features)->Write pid=9864)[0m 2023-04-13 16:29:10.396962: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 AVX512F FMA[32m [repeated 3x across cluster][0m | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 4.0 MiB/4.29 GiB object_store_memory: 0%| | 0/32 [00:08<?, ?it/s] | |
[2m[36m(ReadRange->MapBatches(generate_features)->Write pid=9864)[0m To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.[32m [repeated 3x across cluster][0m | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 4.0 MiB/4.29 GiB object_store_memory: 0%| | 0/32 [00:08<?, ?it/s] | |
[2m[36m(ReadRange->MapBatches(generate_features)->Write pid=7425)[0m 2023-04-13 16:25:56.075771: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/nvidia/lib:/usr/local/nvidia/lib64[32m [repeated 6x across cluster][0m | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 4.0 MiB/4.29 GiB object_store_memory: 0%| | 0/32 [00:08<?, ?it/s] | |
[2m[36m(ReadRange->MapBatches(generate_features)->Write pid=7425)[0m 2023-04-13 16:25:56.075783: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.[32m [repeated 3x across cluster][0m | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 4.0 MiB/4.29 GiB object_store_memory: 0%| | 0/32 [00:08<?, ?it/s] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 4.0 MiB/4.29 GiB object_store_memory: 3%|▎ | 1/32 [00:11<06:02, 11.70s/it] | |
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 3.5 MiB/4.29 GiB object_store_memory: 6%|▋ | 2/32 [00:11<05:50, 11.70s/it] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 4.0 MiB/4.29 GiB object_store_memory: 6%|▋ | 2/32 [00:11<05:50, 11.70s/it] | |
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 3.5 MiB/4.29 GiB object_store_memory: 6%|▋ | 2/32 [00:11<05:50, 11.70s/it] | |
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 3.5 MiB/4.29 GiB object_store_memory: 9%|▉ | 3/32 [00:11<01:30, 3.11s/it] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 4.0 MiB/4.29 GiB object_store_memory: 12%|█▎ | 4/32 [00:12<01:27, 3.11s/it] | |
Running: 12.0/16.0 CPU, 0.0/0.0 GPU, 3.0 MiB/4.29 GiB object_store_memory: 12%|█▎ | 4/32 [00:12<01:27, 3.11s/it] | |
Running: 12.0/16.0 CPU, 0.0/0.0 GPU, 3.0 MiB/4.29 GiB object_store_memory: 16%|█▌ | 5/32 [00:12<00:42, 1.57s/it] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 4.0 MiB/4.29 GiB object_store_memory: 25%|██▌ | 8/32 [00:12<00:37, 1.57s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 3.75 MiB/4.29 GiB object_store_memory: 25%|██▌ | 8/32 [00:12<00:37, 1.57s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 3.75 MiB/4.29 GiB object_store_memory: 28%|██▊ | 9/32 [00:12<00:15, 1.50it/s] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 4.0 MiB/4.29 GiB object_store_memory: 28%|██▊ | 9/32 [00:12<00:15, 1.50it/s] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 3.75 MiB/4.29 GiB object_store_memory: 28%|██▊ | 9/32 [00:12<00:15, 1.50it/s] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 3.75 MiB/4.29 GiB object_store_memory: 34%|███▍ | 11/32 [00:12<00:11, 1.88it/s] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 4.0 MiB/4.29 GiB object_store_memory: 34%|███▍ | 11/32 [00:12<00:11, 1.88it/s] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 3.75 MiB/4.29 GiB object_store_memory: 34%|███▍ | 11/32 [00:12<00:11, 1.88it/s] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 3.75 MiB/4.29 GiB object_store_memory: 38%|███▊ | 12/32 [00:12<00:09, 2.10it/s] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 4.0 MiB/4.29 GiB object_store_memory: 38%|███▊ | 12/32 [00:13<00:09, 2.10it/s] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 3.75 MiB/4.29 GiB object_store_memory: 38%|███▊ | 12/32 [00:13<00:09, 2.10it/s] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 3.75 MiB/4.29 GiB object_store_memory: 41%|████ | 13/32 [00:13<00:10, 1.89it/s] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 4.0 MiB/4.29 GiB object_store_memory: 41%|████ | 13/32 [00:13<00:10, 1.89it/s] | |
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 3.5 MiB/4.29 GiB object_store_memory: 41%|████ | 13/32 [00:16<00:10, 1.89it/s] | |
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 3.5 MiB/4.29 GiB object_store_memory: 44%|████▍ | 14/32 [00:16<00:19, 1.11s/it] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 4.0 MiB/4.29 GiB object_store_memory: 47%|████▋ | 15/32 [00:16<00:18, 1.11s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 3.75 MiB/4.29 GiB object_store_memory: 47%|████▋ | 15/32 [00:17<00:18, 1.11s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 3.75 MiB/4.29 GiB object_store_memory: 50%|█████ | 16/32 [00:17<00:11, 1.35it/s] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 4.0 MiB/4.29 GiB object_store_memory: 50%|█████ | 16/32 [00:17<00:11, 1.35it/s] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 3.75 MiB/4.29 GiB object_store_memory: 50%|█████ | 16/32 [00:23<00:11, 1.35it/s] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 3.75 MiB/4.29 GiB object_store_memory: 53%|█████▎ | 17/32 [00:23<00:29, 1.94s/it] | |
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 3.5 MiB/4.29 GiB object_store_memory: 53%|█████▎ | 17/32 [00:23<00:29, 1.94s/it] | |
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 3.5 MiB/4.29 GiB object_store_memory: 56%|█████▋ | 18/32 [00:23<00:21, 1.50s/it] | |
Running: 13.0/16.0 CPU, 0.0/0.0 GPU, 3.25 MiB/4.29 GiB object_store_memory: 56%|█████▋ | 18/32 [00:23<00:21, 1.50s/it] | |
Running: 13.0/16.0 CPU, 0.0/0.0 GPU, 3.25 MiB/4.29 GiB object_store_memory: 59%|█████▉ | 19/32 [00:23<00:14, 1.15s/it] | |
Running: 12.0/16.0 CPU, 0.0/0.0 GPU, 3.0 MiB/4.29 GiB object_store_memory: 59%|█████▉ | 19/32 [00:23<00:14, 1.15s/it] | |
Running: 12.0/16.0 CPU, 0.0/0.0 GPU, 3.0 MiB/4.29 GiB object_store_memory: 62%|██████▎ | 20/32 [00:23<00:10, 1.15it/s] | |
Running: 10.0/16.0 CPU, 0.0/0.0 GPU, 2.5 MiB/4.29 GiB object_store_memory: 62%|██████▎ | 20/32 [00:23<00:10, 1.15it/s] | |
Running: 10.0/16.0 CPU, 0.0/0.0 GPU, 2.5 MiB/4.29 GiB object_store_memory: 66%|██████▌ | 21/32 [00:23<00:07, 1.52it/s] | |
Running: 9.0/16.0 CPU, 0.0/0.0 GPU, 2.25 MiB/4.29 GiB object_store_memory: 69%|██████▉ | 22/32 [00:23<00:06, 1.52it/s] | |
Running: 9.0/16.0 CPU, 0.0/0.0 GPU, 2.25 MiB/4.29 GiB object_store_memory: 72%|███████▏ | 23/32 [00:23<00:03, 2.55it/s] | |
Running: 8.0/16.0 CPU, 0.0/0.0 GPU, 2.0 MiB/4.29 GiB object_store_memory: 72%|███████▏ | 23/32 [00:24<00:03, 2.55it/s] | |
Running: 7.0/16.0 CPU, 0.0/0.0 GPU, 1.75 MiB/4.29 GiB object_store_memory: 75%|███████▌ | 24/32 [00:24<00:03, 2.55it/s] | |
Running: 7.0/16.0 CPU, 0.0/0.0 GPU, 1.75 MiB/4.29 GiB object_store_memory: 78%|███████▊ | 25/32 [00:24<00:01, 3.58it/s] | |
Running: 5.0/16.0 CPU, 0.0/0.0 GPU, 1.25 MiB/4.29 GiB object_store_memory: 78%|███████▊ | 25/32 [00:24<00:01, 3.58it/s] | |
Running: 5.0/16.0 CPU, 0.0/0.0 GPU, 1.25 MiB/4.29 GiB object_store_memory: 84%|████████▍ | 27/32 [00:24<00:01, 3.97it/s] | |
Running: 4.0/16.0 CPU, 0.0/0.0 GPU, 1.0 MiB/4.29 GiB object_store_memory: 84%|████████▍ | 27/32 [00:24<00:01, 3.97it/s] | |
Running: 4.0/16.0 CPU, 0.0/0.0 GPU, 1.0 MiB/4.29 GiB object_store_memory: 88%|████████▊ | 28/32 [00:24<00:01, 3.82it/s] | |
Running: 3.0/16.0 CPU, 0.0/0.0 GPU, 0.75 MiB/4.29 GiB object_store_memory: 88%|████████▊ | 28/32 [00:25<00:01, 3.82it/s] | |
Running: 3.0/16.0 CPU, 0.0/0.0 GPU, 0.75 MiB/4.29 GiB object_store_memory: 91%|█████████ | 29/32 [00:25<00:00, 3.41it/s] | |
Running: 1.0/16.0 CPU, 0.0/0.0 GPU, 0.25 MiB/4.29 GiB object_store_memory: 91%|█████████ | 29/32 [00:26<00:00, 3.41it/s] | |
Running: 1.0/16.0 CPU, 0.0/0.0 GPU, 0.25 MiB/4.29 GiB object_store_memory: 94%|█████████▍| 30/32 [00:26<00:00, 2.04it/s] | |
Running: 0.0/16.0 CPU, 0.0/0.0 GPU, 0.0 MiB/4.29 GiB object_store_memory: 100%|██████████| 32/32 [00:26<00:00, 2.04it/s] | |
Running case: tfrecords-images-100-256 | |
Read progress 0: 0%| | 0/32 [00:00<?, ?it/s] | |
Result of case tfrecords-images-100-256: {'time': 0.06479635800008055} | |
Running case: tfrecords-images-100-2048 | |
Read progress 0: 0%| | 0/100 [00:00<?, ?it/s] | |
Read progress 0: 17%|█▋ | 17/100 [00:00<00:00, 160.91it/s] | |
Read progress 0: 34%|███▍ | 34/100 [00:00<00:00, 102.90it/s] | |
Read progress 0: 46%|████▌ | 46/100 [00:00<00:00, 101.94it/s] | |
Read progress 0: 57%|█████▋ | 57/100 [00:00<00:00, 89.00it/s] | |
Read progress 0: 67%|██████▋ | 67/100 [00:00<00:00, 89.29it/s] | |
Read progress 0: 77%|███████▋ | 77/100 [00:00<00:00, 91.52it/s] | |
Read progress 0: 92%|█████████▏| 92/100 [00:00<00:00, 103.56it/s] | |
[2m[36m(_execute_read_task_split pid=10387)[0m 2023-04-13 16:29:29.737974: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 AVX512F FMA[32m [repeated 3x across cluster][0m | |
Read progress 0: 97%|█████████▋| 97/100 [00:01<00:00, 103.56it/s] | |
[2m[36m(_execute_read_task_split pid=10387)[0m To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.[32m [repeated 3x across cluster][0m | |
Read progress 0: 97%|█████████▋| 97/100 [00:01<00:00, 103.56it/s] | |
[2m[36m(ReadRange->MapBatches(generate_features)->Write pid=9863)[0m 2023-04-13 16:29:11.921215: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/nvidia/lib:/usr/local/nvidia/lib64[32m [repeated 6x across cluster][0m | |
Read progress 0: 97%|█████████▋| 97/100 [00:01<00:00, 103.56it/s] | |
[2m[36m(ReadRange->MapBatches(generate_features)->Write pid=9863)[0m 2023-04-13 16:29:11.921228: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.[32m [repeated 3x across cluster][0m | |
Read progress 0: 97%|█████████▋| 97/100 [00:01<00:00, 103.56it/s] | |
Result of case tfrecords-images-100-2048: {'time': 3.71467114699999} | |
Running case: tfrecords-images-1000-mix | |
Read progress 0: 0%| | 0/32 [00:00<?, ?it/s] | |
Result of case tfrecords-images-1000-mix: {'time': 0.05040225200002624} | |
Running case: tfrecords-random-int-1g | |
Read progress 0: 0%| | 0/80 [00:00<?, ?it/s] | |
[2m[36m(_execute_read_task_split pid=11388)[0m 2023-04-13 16:29:33.910067: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 AVX512F FMA[32m [repeated 5x across cluster][0m | |
Read progress 0: 0%| | 0/80 [00:03<?, ?it/s] | |
[2m[36m(_execute_read_task_split pid=11388)[0m To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.[32m [repeated 5x across cluster][0m | |
Read progress 0: 0%| | 0/80 [00:03<?, ?it/s] | |
[2m[36m(_execute_read_task_split pid=11387)[0m 2023-04-13 16:29:35.346885: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/nvidia/lib:/usr/local/nvidia/lib64[32m [repeated 8x across cluster][0m | |
Read progress 0: 0%| | 0/80 [00:03<?, ?it/s] | |
[2m[36m(_execute_read_task_split pid=11387)[0m 2023-04-13 16:29:35.346899: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.[32m [repeated 4x across cluster][0m | |
Read progress 0: 0%| | 0/80 [00:03<?, ?it/s] | |
Read progress 0: 1%|▏ | 1/80 [00:54<1:11:15, 54.12s/it] | |
Read progress 0: 5%|▌ | 4/80 [00:54<13:09, 10.38s/it] | |
Read progress 0: 6%|▋ | 5/80 [00:54<09:24, 7.53s/it] | |
Read progress 0: 9%|▉ | 7/80 [00:55<05:11, 4.26s/it] | |
Read progress 0: 12%|█▎ | 10/80 [00:55<02:35, 2.22s/it] | |
Read progress 0: 15%|█▌ | 12/80 [00:55<01:45, 1.55s/it] | |
Read progress 0: 18%|█▊ | 14/80 [00:58<01:44, 1.58s/it] | |
Read progress 0: 19%|█▉ | 15/80 [00:59<01:26, 1.33s/it] | |
Read progress 0: 21%|██▏ | 17/80 [01:47<09:34, 9.11s/it] | |
Read progress 0: 22%|██▎ | 18/80 [01:47<07:36, 7.36s/it] | |
Read progress 0: 25%|██▌ | 20/80 [01:47<04:44, 4.74s/it] | |
Read progress 0: 26%|██▋ | 21/80 [01:47<03:44, 3.81s/it] | |
Read progress 0: 28%|██▊ | 22/80 [01:48<02:58, 3.08s/it] | |
Read progress 0: 29%|██▉ | 23/80 [01:48<02:19, 2.44s/it] | |
Read progress 0: 30%|███ | 24/80 [01:49<01:45, 1.88s/it] | |
Read progress 0: 32%|███▎ | 26/80 [01:49<00:59, 1.11s/it] | |
Read progress 0: 35%|███▌ | 28/80 [01:49<00:39, 1.33it/s] | |
Read progress 0: 36%|███▋ | 29/80 [01:50<00:37, 1.38it/s] | |
Read progress 0: 38%|███▊ | 30/80 [01:51<00:43, 1.15it/s] | |
Read progress 0: 39%|███▉ | 31/80 [01:52<00:41, 1.17it/s] | |
Read progress 0: 41%|████▏ | 33/80 [02:40<08:13, 10.49s/it] | |
Read progress 0: 44%|████▍ | 35/80 [02:40<04:58, 6.63s/it] | |
Read progress 0: 45%|████▌ | 36/80 [02:41<03:58, 5.43s/it] | |
Read progress 0: 46%|████▋ | 37/80 [02:41<03:02, 4.25s/it] | |
Read progress 0: 48%|████▊ | 38/80 [02:41<02:15, 3.23s/it] | |
Read progress 0: 49%|████▉ | 39/80 [02:42<01:40, 2.44s/it] | |
Read progress 0: 50%|█████ | 40/80 [02:42<01:17, 1.93s/it] | |
Read progress 0: 51%|█████▏ | 41/80 [02:43<01:00, 1.56s/it] | |
Read progress 0: 54%|█████▍ | 43/80 [02:43<00:33, 1.10it/s] | |
Read progress 0: 55%|█████▌ | 44/80 [02:43<00:25, 1.39it/s] | |
Read progress 0: 57%|█████▊ | 46/80 [02:44<00:21, 1.58it/s] | |
Read progress 0: 59%|█████▉ | 47/80 [02:45<00:19, 1.65it/s] | |
Read progress 0: 60%|██████ | 48/80 [02:45<00:19, 1.65it/s] | |
Read progress 0: 61%|██████▏ | 49/80 [03:33<06:33, 12.68s/it] | |
Read progress 0: 62%|██████▎ | 50/80 [03:33<04:38, 9.30s/it] | |
Read progress 0: 64%|██████▍ | 51/80 [03:33<03:18, 6.83s/it] | |
Read progress 0: 65%|██████▌ | 52/80 [03:34<02:25, 5.20s/it] | |
Read progress 0: 68%|██████▊ | 54/80 [03:35<01:17, 3.00s/it] | |
Read progress 0: 69%|██████▉ | 55/80 [03:36<01:01, 2.45s/it] | |
Read progress 0: 70%|███████ | 56/80 [03:36<00:44, 1.85s/it] | |
Read progress 0: 71%|███████▏ | 57/80 [03:36<00:34, 1.51s/it] | |
Read progress 0: 72%|███████▎ | 58/80 [03:37<00:25, 1.18s/it] | |
Read progress 0: 74%|███████▍ | 59/80 [03:37<00:18, 1.12it/s] | |
Read progress 0: 75%|███████▌ | 60/80 [03:37<00:13, 1.49it/s] | |
Read progress 0: 79%|███████▉ | 63/80 [03:38<00:07, 2.25it/s] | |
Read progress 0: 80%|████████ | 64/80 [03:38<00:07, 2.23it/s] | |
Read progress 0: 81%|████████▏ | 65/80 [04:26<02:48, 11.25s/it] | |
Read progress 0: 84%|████████▍ | 67/80 [04:26<01:29, 6.85s/it] | |
Read progress 0: 85%|████████▌ | 68/80 [04:28<01:08, 5.70s/it] | |
Read progress 0: 89%|████████▉ | 71/80 [04:29<00:28, 3.14s/it] | |
Read progress 0: 91%|█████████▏| 73/80 [04:29<00:15, 2.17s/it] | |
Read progress 0: 92%|█████████▎| 74/80 [04:29<00:10, 1.83s/it] | |
Read progress 0: 94%|█████████▍| 75/80 [04:30<00:07, 1.54s/it] | |
Read progress 0: 95%|█████████▌| 76/80 [04:30<00:04, 1.22s/it] | |
Read progress 0: 99%|█████████▉| 79/80 [04:30<00:00, 1.58it/s] | |
Result of case tfrecords-random-int-1g: {'time': 270.63967487499997} | |
Running case: tfrecords-random-float-1g | |
Read progress 0: 0%| | 0/80 [00:00<?, ?it/s] | |
[2m[36m(_execute_read_task_split pid=14580)[0m To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags. | |
Read progress 0: 0%| | 0/80 [00:01<?, ?it/s] | |
[2m[36m(_execute_read_task_split pid=14580)[0m To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags. | |
Read progress 0: 0%| | 0/80 [00:01<?, ?it/s] | |
[2m[36m(_execute_read_task_split pid=11388)[0m 2023-04-13 16:29:35.405076: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/nvidia/lib:/usr/local/nvidia/lib64[32m [repeated 4x across cluster][0m | |
Read progress 0: 0%| | 0/80 [00:01<?, ?it/s] | |
[2m[36m(_execute_read_task_split pid=11388)[0m 2023-04-13 16:29:35.405089: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.[32m [repeated 2x across cluster][0m | |
Read progress 0: 0%| | 0/80 [00:01<?, ?it/s] | |
[2m[36m(_execute_read_task_split pid=14579)[0m 2023-04-13 16:34:04.571833: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 AVX512F FMA | |
Read progress 0: 0%| | 0/80 [00:01<?, ?it/s] | |
[2m[36m(_execute_read_task_split pid=14579)[0m To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags. | |
Read progress 0: 0%| | 0/80 [00:01<?, ?it/s] | |
Read progress 0: 1%|▏ | 1/80 [00:50<1:06:59, 50.88s/it] | |
Read progress 0: 2%|▎ | 2/80 [00:51<28:00, 21.55s/it] | |
Read progress 0: 6%|▋ | 5/80 [00:52<07:43, 6.17s/it] | |
Read progress 0: 9%|▉ | 7/80 [00:52<04:30, 3.71s/it] | |
Read progress 0: 11%|█▏ | 9/80 [00:52<02:48, 2.38s/it] | |
Read progress 0: 15%|█▌ | 12/80 [00:52<01:33, 1.38s/it] | |
Read progress 0: 18%|█▊ | 14/80 [00:57<01:47, 1.63s/it] | |
Read progress 0: 19%|█▉ | 15/80 [00:57<01:35, 1.47s/it] | |
Read progress 0: 20%|██ | 16/80 [00:58<01:23, 1.31s/it] | |
Read progress 0: 21%|██▏ | 17/80 [01:42<11:19, 10.78s/it] | |
Read progress 0: 22%|██▎ | 18/80 [01:43<08:39, 8.37s/it] | |
Read progress 0: 24%|██▍ | 19/80 [01:43<06:22, 6.28s/it] | |
Read progress 0: 28%|██▊ | 22/80 [01:43<02:55, 3.02s/it] | |
Read progress 0: 29%|██▉ | 23/80 [01:44<02:20, 2.47s/it] | |
Read progress 0: 31%|███▏ | 25/80 [01:44<01:28, 1.62s/it] | |
Read progress 0: 32%|███▎ | 26/80 [01:44<01:10, 1.31s/it] | |
Read progress 0: 35%|███▌ | 28/80 [01:44<00:44, 1.16it/s] | |
Read progress 0: 36%|███▋ | 29/80 [01:44<00:36, 1.39it/s] | |
Read progress 0: 38%|███▊ | 30/80 [01:48<01:09, 1.39s/it] | |
Read progress 0: 39%|███▉ | 31/80 [01:48<00:52, 1.08s/it] | |
Read progress 0: 40%|████ | 32/80 [01:49<00:52, 1.10s/it] | |
Read progress 0: 41%|████▏ | 33/80 [02:33<10:01, 12.80s/it] | |
Read progress 0: 42%|████▎ | 34/80 [02:35<07:22, 9.62s/it] | |
Read progress 0: 45%|████▌ | 36/80 [02:35<03:58, 5.42s/it] | |
Read progress 0: 48%|████▊ | 38/80 [02:35<02:21, 3.36s/it] | |
Read progress 0: 51%|█████▏ | 41/80 [02:36<01:13, 1.88s/it] | |
Read progress 0: 52%|█████▎ | 42/80 [02:36<01:01, 1.61s/it] | |
Read progress 0: 54%|█████▍ | 43/80 [02:36<00:49, 1.35s/it] | |
Read progress 0: 55%|█████▌ | 44/80 [02:36<00:39, 1.09s/it] | |
Read progress 0: 56%|█████▋ | 45/80 [02:37<00:31, 1.12it/s] | |
Read progress 0: 57%|█████▊ | 46/80 [02:39<00:45, 1.33s/it] | |
Read progress 0: 60%|██████ | 48/80 [02:41<00:34, 1.09s/it] | |
Read progress 0: 61%|██████▏ | 49/80 [03:24<05:39, 10.96s/it] | |
Read progress 0: 62%|██████▎ | 50/80 [03:26<04:17, 8.59s/it] | |
Read progress 0: 64%|██████▍ | 51/80 [03:27<03:09, 6.53s/it] | |
Read progress 0: 65%|██████▌ | 52/80 [03:27<02:13, 4.78s/it] | |
Read progress 0: 66%|██████▋ | 53/80 [03:27<01:33, 3.48s/it] | |
Read progress 0: 69%|██████▉ | 55/80 [03:27<00:50, 2.03s/it] | |
Read progress 0: 70%|███████ | 56/80 [03:28<00:37, 1.58s/it] | |
Read progress 0: 71%|███████▏ | 57/80 [03:28<00:28, 1.26s/it] | |
Read progress 0: 72%|███████▎ | 58/80 [03:28<00:21, 1.05it/s] | |
Read progress 0: 74%|███████▍ | 59/80 [03:28<00:15, 1.34it/s] | |
Read progress 0: 75%|███████▌ | 60/80 [03:28<00:11, 1.70it/s] | |
Read progress 0: 76%|███████▋ | 61/80 [03:29<00:10, 1.76it/s] | |
Read progress 0: 78%|███████▊ | 62/80 [03:30<00:13, 1.35it/s] | |
Read progress 0: 79%|███████▉ | 63/80 [03:31<00:14, 1.20it/s] | |
Read progress 0: 80%|████████ | 64/80 [03:32<00:14, 1.12it/s] | |
Read progress 0: 81%|████████▏ | 65/80 [04:17<03:30, 14.00s/it] | |
Read progress 0: 82%|████████▎ | 66/80 [04:18<02:19, 9.95s/it] | |
Read progress 0: 84%|████████▍ | 67/80 [04:18<01:31, 7.05s/it] | |
Read progress 0: 85%|████████▌ | 68/80 [04:19<01:02, 5.19s/it] | |
Read progress 0: 86%|████████▋ | 69/80 [04:19<00:42, 3.84s/it] | |
Read progress 0: 90%|█████████ | 72/80 [04:19<00:13, 1.72s/it] | |
Read progress 0: 94%|█████████▍| 75/80 [04:20<00:04, 1.01it/s] | |
Read progress 0: 95%|█████████▌| 76/80 [04:20<00:03, 1.16it/s] | |
Read progress 0: 96%|█████████▋| 77/80 [04:21<00:02, 1.25it/s] | |
Read progress 0: 98%|█████████▊| 78/80 [04:21<00:01, 1.31it/s] | |
Read progress 0: 99%|█████████▉| 79/80 [04:21<00:00, 1.59it/s] | |
Read progress 0: 100%|██████████| 80/80 [04:22<00:00, 1.75it/s] | |
Result of case tfrecords-random-float-1g: {'time': 262.41669915} | |
Running case: tfrecords-random-bytes-1g | |
Read progress 0: 0%| | 0/32 [00:00<?, ?it/s] | |
[2m[36m(_execute_read_task_split pid=14580)[0m 2023-04-13 16:34:06.195826: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/nvidia/lib:/usr/local/nvidia/lib64[32m [repeated 6x across cluster][0m | |
Read progress 0: 0%| | 0/32 [00:01<?, ?it/s] | |
[2m[36m(_execute_read_task_split pid=14580)[0m 2023-04-13 16:34:06.195839: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.[32m [repeated 3x across cluster][0m | |
Read progress 0: 0%| | 0/32 [00:01<?, ?it/s] | |
[2m[36m(_execute_read_task_split pid=17699)[0m 2023-04-13 16:38:26.976407: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 AVX512F FMA[32m [repeated 2x across cluster][0m | |
Read progress 0: 0%| | 0/32 [00:01<?, ?it/s] | |
[2m[36m(_execute_read_task_split pid=17699)[0m To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.[32m [repeated 2x across cluster][0m | |
Read progress 0: 0%| | 0/32 [00:01<?, ?it/s] | |
Read progress 0: 3%|▎ | 1/32 [00:15<08:08, 15.76s/it] | |
Read progress 0: 6%|▋ | 2/32 [00:15<03:17, 6.58s/it] | |
Read progress 0: 9%|▉ | 3/32 [00:16<01:48, 3.74s/it] | |
Read progress 0: 12%|█▎ | 4/32 [00:16<01:05, 2.33s/it] | |
Read progress 0: 16%|█▌ | 5/32 [00:16<00:41, 1.53s/it] | |
Read progress 0: 28%|██▊ | 9/32 [00:16<00:12, 1.83it/s] | |
Read progress 0: 38%|███▊ | 12/32 [00:16<00:06, 2.96it/s] | |
Read progress 0: 44%|████▍ | 14/32 [00:21<00:16, 1.11it/s] | |
Read progress 0: 47%|████▋ | 15/32 [00:21<00:13, 1.29it/s] | |
Read progress 0: 50%|█████ | 16/32 [00:22<00:10, 1.49it/s] | |
Read progress 0: 53%|█████▎ | 17/32 [00:32<00:41, 2.79s/it] | |
Read progress 0: 59%|█████▉ | 19/32 [00:32<00:22, 1.75s/it] | |
Read progress 0: 62%|██████▎ | 20/32 [00:32<00:16, 1.40s/it] | |
Read progress 0: 69%|██████▉ | 22/32 [00:32<00:08, 1.12it/s] | |
Read progress 0: 72%|███████▏ | 23/32 [00:33<00:06, 1.35it/s] | |
Read progress 0: 81%|████████▏ | 26/32 [00:33<00:02, 2.40it/s] | |
Read progress 0: 91%|█████████ | 29/32 [00:33<00:00, 3.82it/s] | |
Read progress 0: 97%|█████████▋| 31/32 [00:36<00:00, 1.72it/s] | |
Result of case tfrecords-random-bytes-1g: {'time': 36.31453858500004} | |
Finish benchmark: read-tfrecords | |
[2m[36m(_execute_read_task_split pid=17698)[0m 2023-04-13 16:38:28.675456: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/nvidia/lib:/usr/local/nvidia/lib64[32m [repeated 6x across cluster][0m | |
[2m[36m(_execute_read_task_split pid=17698)[0m 2023-04-13 16:38:28.675470: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.[32m [repeated 3x across cluster][0m | |
[2m[36m(_execute_read_task_split pid=17698)[0m 2023-04-13 16:38:27.000738: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 AVX512F FMA[32m [repeated 2x across cluster][0m | |
[2m[36m(_execute_read_task_split pid=17698)[0m To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.[32m [repeated 2x across cluster][0m | |
Subprocess return code: 0 | |
[INFO 2023-04-13 16:39:05,205] anyscale_job_wrapper.py: 191 Process 1561 exited with return code 0. | |
[INFO 2023-04-13 16:39:05,206] anyscale_job_wrapper.py: 294 Finished with return code 0. Time taken: 1239.13400811 | |
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts. | |
boto3 1.26.112 requires botocore<1.30.0,>=1.29.112, but you have botocore 1.29.105 which is incompatible. | |
Completed 344 Bytes/344 Bytes (2.6 KiB/s) with 1 file(s) remaining | |
upload: ../../../../../release_test_out.json to s3://ray-release-automation-results/working_dirs/read_tfrecords_benchmark_single_node/shzzzxvcxu/tmp/release_test_out.json | |
Completed 374 Bytes/374 Bytes (3.3 KiB/s) with 1 file(s) remaining | |
upload: ../../../../../metrics_test_out.json to s3://ray-release-automation-results/working_dirs/read_tfrecords_benchmark_single_node/shzzzxvcxu/tmp/metrics_test_out.json | |
Completed 243 Bytes/243 Bytes (2.3 KiB/s) with 1 file(s) remaining | |
upload: ./output.json to s3://ray-release-automation-results/working_dirs/read_tfrecords_benchmark_single_node/shzzzxvcxu/tmp/output.json | |
[INFO 2023-04-13 16:40:00,389] anyscale_job_wrapper.py: 346 ### Finished ### | |
[INFO 2023-04-13 16:40:00,389] anyscale_job_wrapper.py: 349 ### JSON |{"collected_metrics":true,"last_prepare_time_taken":null,"prepare_return_codes":[],"return_code":0,"total_time_taken":1293.749378899,"uploaded_artifact":false,"uploaded_metrics":true,"uploaded_results":true,"workload_time_taken":1239.13400811}| ### |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
[INFO 2023-04-18 22:27:37,533] anyscale_job_wrapper.py: 256 ### Starting ### | |
[INFO 2023-04-18 22:27:37,533] anyscale_job_wrapper.py: 280 ### Starting entrypoint ### | |
[INFO 2023-04-18 22:27:37,534] anyscale_job_wrapper.py: 162 Running command python read_tfrecords_benchmark.py | |
[INFO 2023-04-18 22:27:37,538] anyscale_job_wrapper.py: 176 Starting process 1447. | |
+ python read_tfrecords_benchmark.py | |
2023-04-18 22:27:38,567 INFO worker.py:1315 -- Using address 10.138.0.17:6379 set in the environment variable RAY_ADDRESS | |
2023-04-18 22:27:38,567 INFO worker.py:1432 -- Connecting to existing Ray cluster at address: 10.138.0.17:6379... | |
2023-04-18 22:27:38,574 INFO worker.py:1613 -- Connected to Ray cluster. View the dashboard at [1m[32mhttps://console.anyscale-staging.com/api/v2/sessions/ses_g3nbd6p5xypuyvzw83tu6ssnk4/services?redirect_to=dashboard [39m[22m | |
Running benchmark: read-tfrecords | |
2023-04-18 22:27:40,301 INFO streaming_executor.py:87 -- Executing DAG InputDataBuffer[Input] -> TaskPoolMapOperator[ReadImage->MapBatches(images_to_bytes)->Write] | |
2023-04-18 22:27:40,302 INFO streaming_executor.py:88 -- Execution config: ExecutionOptions(resource_limits=ExecutionResources(cpu=None, gpu=None, object_store_memory=None), locality_with_output=False, preserve_order=False, actor_locality_enabled=True, verbose_progress=False) | |
2023-04-18 22:27:40,302 INFO streaming_executor.py:91 -- Tip: To enable per-operator progress reporting, set RAY_DATA_VERBOSE_PROGRESS=1. | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 1.98 MiB/4.28 GiB object_store_memory: 0%| | 0/32 [00:00<?, ?it/s] | |
[2m[36m(ReadImage->MapBatches(images_to_bytes)->Write pid=605)[0m To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags. | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 1.98 MiB/4.28 GiB object_store_memory: 0%| | 0/32 [00:00<?, ?it/s] | |
[2m[36m(ReadImage->MapBatches(images_to_bytes)->Write pid=605)[0m 2023-04-18 22:27:41.035336: I tensorflow/core/util/port.cc:104] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`. | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 1.98 MiB/4.28 GiB object_store_memory: 0%| | 0/32 [00:00<?, ?it/s] | |
[2m[36m(ReadImage->MapBatches(images_to_bytes)->Write pid=605)[0m 2023-04-18 22:27:42.763223: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/nvidia/lib:/usr/local/nvidia/lib64 | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 1.98 MiB/4.28 GiB object_store_memory: 0%| | 0/32 [00:02<?, ?it/s] | |
[2m[36m(ReadImage->MapBatches(images_to_bytes)->Write pid=605)[0m 2023-04-18 22:27:42.763596: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/nvidia/lib:/usr/local/nvidia/lib64 | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 1.98 MiB/4.28 GiB object_store_memory: 0%| | 0/32 [00:02<?, ?it/s] | |
[2m[36m(ReadImage->MapBatches(images_to_bytes)->Write pid=605)[0m 2023-04-18 22:27:42.763616: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly. | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 1.98 MiB/4.28 GiB object_store_memory: 0%| | 0/32 [00:02<?, ?it/s] | |
Running: 5.0/16.0 CPU, 0.0/0.0 GPU, 0.65 MiB/4.28 GiB object_store_memory: 0%| | 0/32 [00:04<?, ?it/s] | |
Running: 5.0/16.0 CPU, 0.0/0.0 GPU, 0.65 MiB/4.28 GiB object_store_memory: 3%|▎ | 1/32 [00:04<02:18, 4.48s/it] | |
Running: 1.0/16.0 CPU, 0.0/0.0 GPU, 0.12 MiB/4.28 GiB object_store_memory: 34%|███▍ | 11/32 [00:04<01:33, 4.48s/it] | |
Running: 1.0/16.0 CPU, 0.0/0.0 GPU, 0.12 MiB/4.28 GiB object_store_memory: 38%|███▊ | 12/32 [00:04<00:05, 3.60it/s] | |
Running: 1.0/16.0 CPU, 0.0/0.0 GPU, 0.12 MiB/4.28 GiB object_store_memory: 84%|████████▍ | 27/32 [00:04<00:00, 9.73it/s] | |
Running: 1.0/16.0 CPU, 0.0/0.0 GPU, 0.11 MiB/4.28 GiB object_store_memory: 97%|█████████▋| 31/32 [00:04<00:00, 9.73it/s] | |
Running: 0.0/16.0 CPU, 0.0/0.0 GPU, 0.0 MiB/4.28 GiB object_store_memory: 97%|█████████▋| 31/32 [00:04<00:00, 9.73it/s] | |
2023-04-18 22:29:07,543 INFO streaming_executor.py:87 -- Executing DAG InputDataBuffer[Input] -> TaskPoolMapOperator[ReadImage->MapBatches(images_to_bytes)->Write] | |
2023-04-18 22:29:07,543 INFO streaming_executor.py:88 -- Execution config: ExecutionOptions(resource_limits=ExecutionResources(cpu=None, gpu=None, object_store_memory=None), locality_with_output=False, preserve_order=False, actor_locality_enabled=True, verbose_progress=False) | |
2023-04-18 22:29:07,543 INFO streaming_executor.py:91 -- Tip: To enable per-operator progress reporting, set RAY_DATA_VERBOSE_PROGRESS=1. | |
Running 0: 0%| | 0/100 [00:00<?, ?it/s] | |
Running: 0.0/16.0 CPU, 0.0/0.0 GPU, 0.0 MiB/4.28 GiB object_store_memory: 0%| | 0/100 [00:00<?, ?it/s] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 38.47 MiB/4.28 GiB object_store_memory: 0%| | 0/100 [00:00<?, ?it/s] | |
Running: 3.0/16.0 CPU, 0.0/0.0 GPU, 7.21 MiB/4.28 GiB object_store_memory: 0%| | 0/100 [00:00<?, ?it/s] | |
Running: 3.0/16.0 CPU, 0.0/0.0 GPU, 7.21 MiB/4.28 GiB object_store_memory: 1%| | 1/100 [00:00<00:33, 2.98it/s] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 38.47 MiB/4.28 GiB object_store_memory: 13%|█▎ | 13/100 [00:00<00:29, 2.98it/s] | |
Running: 5.0/16.0 CPU, 0.0/0.0 GPU, 12.02 MiB/4.28 GiB object_store_memory: 13%|█▎ | 13/100 [00:00<00:29, 2.98it/s] | |
Running: 5.0/16.0 CPU, 0.0/0.0 GPU, 12.02 MiB/4.28 GiB object_store_memory: 14%|█▍ | 14/100 [00:00<00:03, 24.86it/s] | |
Running: 5.0/16.0 CPU, 0.0/0.0 GPU, 12.02 MiB/4.28 GiB object_store_memory: 25%|██▌ | 25/100 [00:00<00:01, 42.14it/s] | |
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 33.66 MiB/4.28 GiB object_store_memory: 26%|██▌ | 26/100 [00:00<00:01, 42.14it/s] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 38.47 MiB/4.28 GiB object_store_memory: 26%|██▌ | 26/100 [00:00<00:01, 42.14it/s] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 38.47 MiB/4.28 GiB object_store_memory: 32%|███▏ | 32/100 [00:00<00:01, 38.22it/s] | |
Running: 5.0/16.0 CPU, 0.0/0.0 GPU, 12.02 MiB/4.28 GiB object_store_memory: 33%|███▎ | 33/100 [00:00<00:01, 38.22it/s] | |
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 33.66 MiB/4.28 GiB object_store_memory: 37%|███▋ | 37/100 [00:01<00:01, 38.22it/s] | |
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 33.66 MiB/4.28 GiB object_store_memory: 38%|███▊ | 38/100 [00:01<00:01, 41.40it/s] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 38.47 MiB/4.28 GiB object_store_memory: 39%|███▉ | 39/100 [00:01<00:01, 41.40it/s] | |
Running: 5.0/16.0 CPU, 0.0/0.0 GPU, 12.02 MiB/4.28 GiB object_store_memory: 39%|███▉ | 39/100 [00:01<00:01, 41.40it/s] | |
Running: 5.0/16.0 CPU, 0.0/0.0 GPU, 12.02 MiB/4.28 GiB object_store_memory: 44%|████▍ | 44/100 [00:01<00:01, 36.32it/s] | |
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 33.66 MiB/4.28 GiB object_store_memory: 50%|█████ | 50/100 [00:01<00:01, 36.32it/s] | |
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 33.66 MiB/4.28 GiB object_store_memory: 51%|█████ | 51/100 [00:01<00:01, 41.93it/s] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 38.47 MiB/4.28 GiB object_store_memory: 52%|█████▏ | 52/100 [00:01<00:01, 41.93it/s] | |
Running: 5.0/16.0 CPU, 0.0/0.0 GPU, 12.02 MiB/4.28 GiB object_store_memory: 52%|█████▏ | 52/100 [00:01<00:01, 41.93it/s] | |
Running: 5.0/16.0 CPU, 0.0/0.0 GPU, 12.02 MiB/4.28 GiB object_store_memory: 56%|█████▌ | 56/100 [00:01<00:01, 35.17it/s] | |
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 33.66 MiB/4.28 GiB object_store_memory: 63%|██████▎ | 63/100 [00:01<00:01, 35.17it/s] | |
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 33.66 MiB/4.28 GiB object_store_memory: 64%|██████▍ | 64/100 [00:01<00:00, 42.87it/s] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 38.47 MiB/4.28 GiB object_store_memory: 65%|██████▌ | 65/100 [00:01<00:00, 42.87it/s] | |
[2m[36m(ReadImage->MapBatches(images_to_bytes)->Write pid=2901)[0m 2023-04-18 22:29:09.421447: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 AVX512F AVX512_VNNI FMA[32m [repeated 16x across cluster] (Ray deduplicates logs by default. Set RAY_DEDUP_LOGS=0 to disable log deduplication, or see https://docs.ray.io/en/master/ray-observability/ray-logging.html#log-deduplication for more options.)[0m | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 38.47 MiB/4.28 GiB object_store_memory: 65%|██████▌ | 65/100 [00:01<00:00, 42.87it/s] | |
[2m[36m(ReadImage->MapBatches(images_to_bytes)->Write pid=2901)[0m To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.[32m [repeated 16x across cluster][0m | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 38.47 MiB/4.28 GiB object_store_memory: 65%|██████▌ | 65/100 [00:01<00:00, 42.87it/s] | |
[2m[36m(ReadImage->MapBatches(images_to_bytes)->Write pid=1581)[0m 2023-04-18 22:27:42.496392: I tensorflow/core/util/port.cc:104] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.[32m [repeated 15x across cluster][0m | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 38.47 MiB/4.28 GiB object_store_memory: 65%|██████▌ | 65/100 [00:01<00:00, 42.87it/s] | |
[2m[36m(ReadImage->MapBatches(images_to_bytes)->Write pid=1581)[0m 2023-04-18 22:27:43.798676: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/nvidia/lib:/usr/local/nvidia/lib64[32m [repeated 30x across cluster][0m | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 38.47 MiB/4.28 GiB object_store_memory: 65%|██████▌ | 65/100 [00:01<00:00, 42.87it/s] | |
[2m[36m(ReadImage->MapBatches(images_to_bytes)->Write pid=1581)[0m 2023-04-18 22:27:43.798692: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.[32m [repeated 15x across cluster][0m | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 38.47 MiB/4.28 GiB object_store_memory: 65%|██████▌ | 65/100 [00:01<00:00, 42.87it/s] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 38.47 MiB/4.28 GiB object_store_memory: 70%|███████ | 70/100 [00:01<00:00, 37.14it/s] | |
Running: 5.0/16.0 CPU, 0.0/0.0 GPU, 12.02 MiB/4.28 GiB object_store_memory: 72%|███████▏ | 72/100 [00:01<00:00, 37.14it/s] | |
Running: 5.0/16.0 CPU, 0.0/0.0 GPU, 12.02 MiB/4.28 GiB object_store_memory: 77%|███████▋ | 77/100 [00:02<00:00, 42.80it/s] | |
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 33.66 MiB/4.28 GiB object_store_memory: 78%|███████▊ | 78/100 [00:02<00:00, 42.80it/s] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 38.47 MiB/4.28 GiB object_store_memory: 78%|███████▊ | 78/100 [00:02<00:00, 42.80it/s] | |
Running: 5.0/16.0 CPU, 0.0/0.0 GPU, 12.02 MiB/4.28 GiB object_store_memory: 78%|███████▊ | 78/100 [00:02<00:00, 42.80it/s] | |
Running: 5.0/16.0 CPU, 0.0/0.0 GPU, 12.02 MiB/4.28 GiB object_store_memory: 82%|████████▏ | 82/100 [00:02<00:00, 35.20it/s] | |
Running: 9.0/16.0 CPU, 0.0/0.0 GPU, 21.64 MiB/4.28 GiB object_store_memory: 89%|████████▉ | 89/100 [00:02<00:00, 35.20it/s] | |
Running: 9.0/16.0 CPU, 0.0/0.0 GPU, 21.64 MiB/4.28 GiB object_store_memory: 90%|█████████ | 90/100 [00:02<00:00, 43.94it/s] | |
Running: 8.0/16.0 CPU, 0.0/0.0 GPU, 19.23 MiB/4.28 GiB object_store_memory: 92%|█████████▏| 92/100 [00:02<00:00, 43.94it/s] | |
Running: 3.0/16.0 CPU, 0.0/0.0 GPU, 7.21 MiB/4.28 GiB object_store_memory: 92%|█████████▏| 92/100 [00:02<00:00, 43.94it/s] | |
Running: 3.0/16.0 CPU, 0.0/0.0 GPU, 7.21 MiB/4.28 GiB object_store_memory: 96%|█████████▌| 96/100 [00:02<00:00, 38.06it/s] | |
Running: 1.0/16.0 CPU, 0.0/0.0 GPU, 2.41 MiB/4.28 GiB object_store_memory: 97%|█████████▋| 97/100 [00:04<00:00, 38.06it/s] | |
Running: 0.0/16.0 CPU, 0.0/0.0 GPU, 0.0 MiB/4.28 GiB object_store_memory: 99%|█████████▉| 99/100 [00:04<00:00, 38.06it/s] | |
2023-04-18 22:29:20,463 INFO streaming_executor.py:87 -- Executing DAG InputDataBuffer[Input] -> TaskPoolMapOperator[ReadImage->MapBatches(images_to_bytes)->Write] | |
2023-04-18 22:29:20,463 INFO streaming_executor.py:88 -- Execution config: ExecutionOptions(resource_limits=ExecutionResources(cpu=None, gpu=None, object_store_memory=None), locality_with_output=False, preserve_order=False, actor_locality_enabled=True, verbose_progress=False) | |
2023-04-18 22:29:20,463 INFO streaming_executor.py:91 -- Tip: To enable per-operator progress reporting, set RAY_DATA_VERBOSE_PROGRESS=1. | |
Running 0: 0%| | 0/51 [00:00<?, ?it/s] | |
Running: 0.0/16.0 CPU, 0.0/0.0 GPU, 0.0 MiB/4.28 GiB object_store_memory: 0%| | 0/51 [00:00<?, ?it/s] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.78 MiB/4.28 GiB object_store_memory: 0%| | 0/51 [00:00<?, ?it/s] | |
Running: 3.0/16.0 CPU, 0.0/0.0 GPU, 3.33 MiB/4.28 GiB object_store_memory: 0%| | 0/51 [00:00<?, ?it/s] | |
Running: 3.0/16.0 CPU, 0.0/0.0 GPU, 3.33 MiB/4.28 GiB object_store_memory: 2%|▏ | 1/51 [00:00<00:11, 4.44it/s] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 18.37 MiB/4.28 GiB object_store_memory: 25%|██▌ | 13/51 [00:00<00:08, 4.44it/s] | |
Running: 3.0/16.0 CPU, 0.0/0.0 GPU, 3.33 MiB/4.28 GiB object_store_memory: 25%|██▌ | 13/51 [00:00<00:08, 4.44it/s] | |
Running: 3.0/16.0 CPU, 0.0/0.0 GPU, 3.33 MiB/4.28 GiB object_store_memory: 27%|██▋ | 14/51 [00:00<00:01, 36.54it/s] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 15.42 MiB/4.28 GiB object_store_memory: 51%|█████ | 26/51 [00:00<00:00, 36.54it/s] | |
Running: 3.0/16.0 CPU, 0.0/0.0 GPU, 3.33 MiB/4.28 GiB object_store_memory: 51%|█████ | 26/51 [00:00<00:00, 36.54it/s] | |
Running: 3.0/16.0 CPU, 0.0/0.0 GPU, 3.33 MiB/4.28 GiB object_store_memory: 53%|█████▎ | 27/51 [00:00<00:00, 47.83it/s] | |
Running: 12.0/16.0 CPU, 0.0/0.0 GPU, 10.95 MiB/4.28 GiB object_store_memory: 76%|███████▋ | 39/51 [00:00<00:00, 47.83it/s] | |
Running: 3.0/16.0 CPU, 0.0/0.0 GPU, 3.33 MiB/4.28 GiB object_store_memory: 76%|███████▋ | 39/51 [00:00<00:00, 47.83it/s] | |
Running: 3.0/16.0 CPU, 0.0/0.0 GPU, 3.33 MiB/4.28 GiB object_store_memory: 78%|███████▊ | 40/51 [00:00<00:00, 52.87it/s] | |
[2m[36m(ReadImage->MapBatches(images_to_bytes)->Write pid=3281)[0m 2023-04-18 22:29:21.867708: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 AVX512F AVX512_VNNI FMA[32m [repeated 3x across cluster][0m | |
Running: 3.0/16.0 CPU, 0.0/0.0 GPU, 3.33 MiB/4.28 GiB object_store_memory: 94%|█████████▍| 48/51 [00:01<00:00, 52.87it/s] | |
[2m[36m(ReadImage->MapBatches(images_to_bytes)->Write pid=3281)[0m To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.[32m [repeated 3x across cluster][0m | |
Running: 3.0/16.0 CPU, 0.0/0.0 GPU, 3.33 MiB/4.28 GiB object_store_memory: 94%|█████████▍| 48/51 [00:01<00:00, 52.87it/s] | |
[2m[36m(ReadImage->MapBatches(images_to_bytes)->Write pid=2903)[0m 2023-04-18 22:29:09.735154: I tensorflow/core/util/port.cc:104] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.[32m [repeated 3x across cluster][0m | |
Running: 3.0/16.0 CPU, 0.0/0.0 GPU, 3.33 MiB/4.28 GiB object_store_memory: 94%|█████████▍| 48/51 [00:01<00:00, 52.87it/s] | |
[2m[36m(ReadImage->MapBatches(images_to_bytes)->Write pid=2903)[0m 2023-04-18 22:29:10.622954: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/nvidia/lib:/usr/local/nvidia/lib64[32m [repeated 6x across cluster][0m | |
Running: 3.0/16.0 CPU, 0.0/0.0 GPU, 3.33 MiB/4.28 GiB object_store_memory: 94%|█████████▍| 48/51 [00:01<00:00, 52.87it/s] | |
[2m[36m(ReadImage->MapBatches(images_to_bytes)->Write pid=2903)[0m 2023-04-18 22:29:10.622968: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.[32m [repeated 3x across cluster][0m | |
Running: 3.0/16.0 CPU, 0.0/0.0 GPU, 3.33 MiB/4.28 GiB object_store_memory: 94%|█████████▍| 48/51 [00:01<00:00, 52.87it/s] | |
Running: 1.0/16.0 CPU, 0.0/0.0 GPU, 0.91 MiB/4.28 GiB object_store_memory: 94%|█████████▍| 48/51 [00:03<00:00, 52.87it/s] | |
Running: 1.0/16.0 CPU, 0.0/0.0 GPU, 0.91 MiB/4.28 GiB object_store_memory: 96%|█████████▌| 49/51 [00:03<00:00, 9.67it/s] | |
Running: 0.0/16.0 CPU, 0.0/0.0 GPU, 0.0 MiB/4.28 GiB object_store_memory: 98%|█████████▊| 50/51 [00:03<00:00, 9.67it/s] | |
2023-04-18 22:29:24,199 INFO streaming_executor.py:87 -- Executing DAG InputDataBuffer[Input] -> TaskPoolMapOperator[ReadRange->MapBatches(generate_features)->Write] | |
2023-04-18 22:29:24,199 INFO streaming_executor.py:88 -- Execution config: ExecutionOptions(resource_limits=ExecutionResources(cpu=None, gpu=None, object_store_memory=None), locality_with_output=False, preserve_order=False, actor_locality_enabled=True, verbose_progress=False) | |
2023-04-18 22:29:24,199 INFO streaming_executor.py:91 -- Tip: To enable per-operator progress reporting, set RAY_DATA_VERBOSE_PROGRESS=1. | |
Running 0: 0%| | 0/80 [00:00<?, ?it/s] | |
Running: 0.0/16.0 CPU, 0.0/0.0 GPU, 0.0 MiB/4.28 GiB object_store_memory: 0%| | 0/80 [00:00<?, ?it/s] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 0%| | 0/80 [00:00<?, ?it/s] | |
[2m[36m(ReadRange->MapBatches(generate_features)->Write pid=3279)[0m 2023-04-18 22:29:55.662969: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 AVX512F AVX512_VNNI FMA[32m [repeated 3x across cluster][0m | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 0%| | 0/80 [00:31<?, ?it/s] | |
[2m[36m(ReadRange->MapBatches(generate_features)->Write pid=3279)[0m To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.[32m [repeated 3x across cluster][0m | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 0%| | 0/80 [00:31<?, ?it/s] | |
[2m[36m(ReadImage->MapBatches(images_to_bytes)->Write pid=3280)[0m 2023-04-18 22:29:22.092111: I tensorflow/core/util/port.cc:104] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.[32m [repeated 3x across cluster][0m | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 0%| | 0/80 [00:31<?, ?it/s] | |
[2m[36m(ReadImage->MapBatches(images_to_bytes)->Write pid=3280)[0m 2023-04-18 22:29:22.951696: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/nvidia/lib:/usr/local/nvidia/lib64[32m [repeated 6x across cluster][0m | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 0%| | 0/80 [00:31<?, ?it/s] | |
[2m[36m(ReadImage->MapBatches(images_to_bytes)->Write pid=3280)[0m 2023-04-18 22:29:22.951704: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.[32m [repeated 3x across cluster][0m | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 0%| | 0/80 [00:31<?, ?it/s] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 0%| | 0/80 [01:01<?, ?it/s] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 1%|▏ | 1/80 [01:01<1:21:17, 61.74s/it] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 1%|▏ | 1/80 [01:01<1:21:17, 61.74s/it] | |
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 14.0 MiB/4.28 GiB object_store_memory: 1%|▏ | 1/80 [01:02<1:21:17, 61.74s/it] | |
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 14.0 MiB/4.28 GiB object_store_memory: 2%|▎ | 2/80 [01:02<33:17, 25.61s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 4%|▍ | 3/80 [01:02<32:52, 25.61s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 5%|▌ | 4/80 [01:02<12:08, 9.59s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 8%|▊ | 6/80 [01:02<06:17, 5.11s/it] | |
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 14.0 MiB/4.28 GiB object_store_memory: 9%|▉ | 7/80 [01:02<06:12, 5.11s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 9%|▉ | 7/80 [01:02<06:12, 5.11s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 10%|█ | 8/80 [01:02<03:41, 3.07s/it] | |
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 14.0 MiB/4.28 GiB object_store_memory: 11%|█▏ | 9/80 [01:02<03:38, 3.07s/it] | |
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 14.0 MiB/4.28 GiB object_store_memory: 12%|█▎ | 10/80 [01:02<02:19, 1.99s/it] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 14%|█▍ | 11/80 [01:02<02:17, 1.99s/it] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 15%|█▌ | 12/80 [01:03<01:32, 1.36s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 15%|█▌ | 12/80 [01:03<01:32, 1.36s/it] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 15%|█▌ | 12/80 [01:03<01:32, 1.36s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 15%|█▌ | 12/80 [01:06<01:32, 1.36s/it] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 16%|█▋ | 13/80 [01:06<01:31, 1.36s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 16%|█▋ | 13/80 [01:06<01:31, 1.36s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 18%|█▊ | 14/80 [01:06<01:43, 1.57s/it] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 18%|█▊ | 14/80 [01:07<01:43, 1.57s/it] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 19%|█▉ | 15/80 [01:07<01:29, 1.37s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 19%|█▉ | 15/80 [01:07<01:29, 1.37s/it] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 19%|█▉ | 15/80 [01:07<01:29, 1.37s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 19%|█▉ | 15/80 [01:07<01:29, 1.37s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 20%|██ | 16/80 [01:07<01:12, 1.13s/it] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 20%|██ | 16/80 [01:07<01:12, 1.13s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 20%|██ | 16/80 [02:03<01:12, 1.13s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 21%|██▏ | 17/80 [02:03<14:33, 13.87s/it] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 21%|██▏ | 17/80 [02:03<14:33, 13.87s/it] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 22%|██▎ | 18/80 [02:03<10:52, 10.53s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 22%|██▎ | 18/80 [02:03<10:52, 10.53s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 24%|██▍ | 19/80 [02:04<07:55, 7.79s/it] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 24%|██▍ | 19/80 [02:04<07:55, 7.79s/it] | |
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 14.0 MiB/4.28 GiB object_store_memory: 24%|██▍ | 19/80 [02:04<07:55, 7.79s/it] | |
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 14.0 MiB/4.28 GiB object_store_memory: 25%|██▌ | 20/80 [02:04<05:42, 5.72s/it] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 26%|██▋ | 21/80 [02:04<05:37, 5.72s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 26%|██▋ | 21/80 [02:04<05:37, 5.72s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 28%|██▊ | 22/80 [02:04<03:08, 3.26s/it] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 28%|██▊ | 22/80 [02:04<03:08, 3.26s/it] | |
Running: 13.0/16.0 CPU, 0.0/0.0 GPU, 13.0 MiB/4.28 GiB object_store_memory: 28%|██▊ | 22/80 [02:04<03:08, 3.26s/it] | |
Running: 13.0/16.0 CPU, 0.0/0.0 GPU, 13.0 MiB/4.28 GiB object_store_memory: 29%|██▉ | 23/80 [02:04<02:25, 2.55s/it] | |
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 14.0 MiB/4.28 GiB object_store_memory: 31%|███▏ | 25/80 [02:04<02:20, 2.55s/it] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 34%|███▍ | 27/80 [02:04<02:15, 2.55s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 34%|███▍ | 27/80 [02:05<02:15, 2.55s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 35%|███▌ | 28/80 [02:05<00:52, 1.01s/it] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 35%|███▌ | 28/80 [02:05<00:52, 1.01s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 35%|███▌ | 28/80 [02:08<00:52, 1.01s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 36%|███▋ | 29/80 [02:08<01:09, 1.37s/it] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 36%|███▋ | 29/80 [02:08<01:09, 1.37s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 36%|███▋ | 29/80 [02:09<01:09, 1.37s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 38%|███▊ | 30/80 [02:09<01:05, 1.32s/it] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 38%|███▊ | 30/80 [02:09<01:05, 1.32s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 38%|███▊ | 30/80 [02:10<01:05, 1.32s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 39%|███▉ | 31/80 [02:10<00:55, 1.14s/it] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 40%|████ | 32/80 [02:10<00:54, 1.14s/it] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 41%|████▏ | 33/80 [03:04<08:27, 10.79s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 41%|████▏ | 33/80 [03:04<08:27, 10.79s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 42%|████▎ | 34/80 [03:05<06:34, 8.57s/it] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 42%|████▎ | 34/80 [03:05<06:34, 8.57s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 42%|████▎ | 34/80 [03:06<06:34, 8.57s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 44%|████▍ | 35/80 [03:06<05:07, 6.84s/it] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 44%|████▍ | 35/80 [03:06<05:07, 6.84s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 44%|████▍ | 35/80 [03:06<05:07, 6.84s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 45%|████▌ | 36/80 [03:06<03:48, 5.20s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 46%|████▋ | 37/80 [03:06<02:46, 3.87s/it] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 48%|████▊ | 38/80 [03:06<02:42, 3.87s/it] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 49%|████▉ | 39/80 [03:06<01:33, 2.29s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 49%|████▉ | 39/80 [03:06<01:33, 2.29s/it] | |
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 14.0 MiB/4.28 GiB object_store_memory: 49%|████▉ | 39/80 [03:06<01:33, 2.29s/it] | |
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 14.0 MiB/4.28 GiB object_store_memory: 50%|█████ | 40/80 [03:06<01:11, 1.78s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 51%|█████▏ | 41/80 [03:06<01:09, 1.78s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 52%|█████▎ | 42/80 [03:06<00:41, 1.09s/it] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 52%|█████▎ | 42/80 [03:07<00:41, 1.09s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 52%|█████▎ | 42/80 [03:07<00:41, 1.09s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 55%|█████▌ | 44/80 [03:07<00:26, 1.34it/s] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 55%|█████▌ | 44/80 [03:07<00:26, 1.34it/s] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 56%|█████▋ | 45/80 [03:10<00:45, 1.31s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 56%|█████▋ | 45/80 [03:10<00:45, 1.31s/it] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 56%|█████▋ | 45/80 [03:10<00:45, 1.31s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 56%|█████▋ | 45/80 [03:11<00:45, 1.31s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 57%|█████▊ | 46/80 [03:11<00:37, 1.11s/it] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 57%|█████▊ | 46/80 [03:11<00:37, 1.11s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 57%|█████▊ | 46/80 [03:11<00:37, 1.11s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 59%|█████▉ | 47/80 [03:11<00:30, 1.09it/s] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 59%|█████▉ | 47/80 [03:11<00:30, 1.09it/s] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 59%|█████▉ | 47/80 [03:11<00:30, 1.09it/s] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 60%|██████ | 48/80 [03:11<00:24, 1.32it/s] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 60%|██████ | 48/80 [03:11<00:24, 1.32it/s] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 60%|██████ | 48/80 [04:06<00:24, 1.32it/s] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 61%|██████▏ | 49/80 [04:06<07:57, 15.40s/it] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 61%|██████▏ | 49/80 [04:06<07:57, 15.40s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 61%|██████▏ | 49/80 [04:06<07:57, 15.40s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 62%|██████▎ | 50/80 [04:06<05:35, 11.20s/it] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 62%|██████▎ | 50/80 [04:07<05:35, 11.20s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 62%|██████▎ | 50/80 [04:07<05:35, 11.20s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 64%|██████▍ | 51/80 [04:07<03:59, 8.25s/it] | |
Running: 13.0/16.0 CPU, 0.0/0.0 GPU, 13.0 MiB/4.28 GiB object_store_memory: 64%|██████▍ | 51/80 [04:07<03:59, 8.25s/it] | |
Running: 13.0/16.0 CPU, 0.0/0.0 GPU, 13.0 MiB/4.28 GiB object_store_memory: 65%|██████▌ | 52/80 [04:07<02:45, 5.90s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 68%|██████▊ | 54/80 [04:08<02:33, 5.90s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 69%|██████▉ | 55/80 [04:08<01:06, 2.65s/it] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 69%|██████▉ | 55/80 [04:08<01:06, 2.65s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 70%|███████ | 56/80 [04:08<01:03, 2.65s/it] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 70%|███████ | 56/80 [04:08<01:03, 2.65s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 70%|███████ | 56/80 [04:08<01:03, 2.65s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 71%|███████▏ | 57/80 [04:08<00:42, 1.85s/it] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 71%|███████▏ | 57/80 [04:08<00:42, 1.85s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 71%|███████▏ | 57/80 [04:09<00:42, 1.85s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 72%|███████▎ | 58/80 [04:09<00:34, 1.55s/it] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 72%|███████▎ | 58/80 [04:09<00:34, 1.55s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 72%|███████▎ | 58/80 [04:09<00:34, 1.55s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 74%|███████▍ | 59/80 [04:09<00:26, 1.25s/it] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 75%|███████▌ | 60/80 [04:09<00:25, 1.25s/it] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 76%|███████▋ | 61/80 [04:12<00:27, 1.46s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 76%|███████▋ | 61/80 [04:12<00:27, 1.46s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 78%|███████▊ | 62/80 [04:12<00:21, 1.17s/it] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 78%|███████▊ | 62/80 [04:13<00:21, 1.17s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 78%|███████▊ | 62/80 [04:13<00:21, 1.17s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 79%|███████▉ | 63/80 [04:13<00:17, 1.05s/it] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 79%|███████▉ | 63/80 [04:13<00:17, 1.05s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 79%|███████▉ | 63/80 [04:13<00:17, 1.05s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 80%|████████ | 64/80 [04:13<00:14, 1.12it/s] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 80%|████████ | 64/80 [04:14<00:14, 1.12it/s] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 80%|████████ | 64/80 [05:08<00:14, 1.12it/s] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 81%|████████▏ | 65/80 [05:08<03:45, 15.00s/it] | |
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 14.0 MiB/4.28 GiB object_store_memory: 81%|████████▏ | 65/80 [05:08<03:45, 15.00s/it] | |
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 14.0 MiB/4.28 GiB object_store_memory: 82%|████████▎ | 66/80 [05:08<02:32, 10.91s/it] | |
Running: 13.0/16.0 CPU, 0.0/0.0 GPU, 13.0 MiB/4.28 GiB object_store_memory: 82%|████████▎ | 66/80 [05:09<02:32, 10.91s/it] | |
Running: 13.0/16.0 CPU, 0.0/0.0 GPU, 13.0 MiB/4.28 GiB object_store_memory: 84%|████████▍ | 67/80 [05:09<01:45, 8.13s/it] | |
Running: 12.0/16.0 CPU, 0.0/0.0 GPU, 12.0 MiB/4.28 GiB object_store_memory: 84%|████████▍ | 67/80 [05:09<01:45, 8.13s/it] | |
Running: 12.0/16.0 CPU, 0.0/0.0 GPU, 12.0 MiB/4.28 GiB object_store_memory: 85%|████████▌ | 68/80 [05:09<01:10, 5.85s/it] | |
Running: 11.0/16.0 CPU, 0.0/0.0 GPU, 11.0 MiB/4.28 GiB object_store_memory: 85%|████████▌ | 68/80 [05:09<01:10, 5.85s/it] | |
Running: 11.0/16.0 CPU, 0.0/0.0 GPU, 11.0 MiB/4.28 GiB object_store_memory: 86%|████████▋ | 69/80 [05:09<00:46, 4.21s/it] | |
Running: 10.0/16.0 CPU, 0.0/0.0 GPU, 10.0 MiB/4.28 GiB object_store_memory: 86%|████████▋ | 69/80 [05:09<00:46, 4.21s/it] | |
Running: 10.0/16.0 CPU, 0.0/0.0 GPU, 10.0 MiB/4.28 GiB object_store_memory: 88%|████████▊ | 70/80 [05:09<00:30, 3.00s/it] | |
Running: 9.0/16.0 CPU, 0.0/0.0 GPU, 9.0 MiB/4.28 GiB object_store_memory: 88%|████████▊ | 70/80 [05:09<00:30, 3.00s/it] | |
Running: 9.0/16.0 CPU, 0.0/0.0 GPU, 9.0 MiB/4.28 GiB object_store_memory: 89%|████████▉ | 71/80 [05:09<00:19, 2.15s/it] | |
Running: 8.0/16.0 CPU, 0.0/0.0 GPU, 8.0 MiB/4.28 GiB object_store_memory: 89%|████████▉ | 71/80 [05:10<00:19, 2.15s/it] | |
Running: 8.0/16.0 CPU, 0.0/0.0 GPU, 8.0 MiB/4.28 GiB object_store_memory: 90%|█████████ | 72/80 [05:10<00:12, 1.54s/it] | |
Running: 8.0/16.0 CPU, 0.0/0.0 GPU, 8.0 MiB/4.28 GiB object_store_memory: 91%|█████████▏| 73/80 [05:10<00:07, 1.14s/it] | |
Running: 6.0/16.0 CPU, 0.0/0.0 GPU, 6.0 MiB/4.28 GiB object_store_memory: 92%|█████████▎| 74/80 [05:10<00:06, 1.14s/it] | |
Running: 5.0/16.0 CPU, 0.0/0.0 GPU, 5.0 MiB/4.28 GiB object_store_memory: 92%|█████████▎| 74/80 [05:10<00:06, 1.14s/it] | |
Running: 5.0/16.0 CPU, 0.0/0.0 GPU, 5.0 MiB/4.28 GiB object_store_memory: 94%|█████████▍| 75/80 [05:10<00:03, 1.50it/s] | |
Running: 4.0/16.0 CPU, 0.0/0.0 GPU, 4.0 MiB/4.28 GiB object_store_memory: 94%|█████████▍| 75/80 [05:10<00:03, 1.50it/s] | |
Running: 4.0/16.0 CPU, 0.0/0.0 GPU, 4.0 MiB/4.28 GiB object_store_memory: 95%|█████████▌| 76/80 [05:10<00:02, 1.90it/s] | |
Running: 3.0/16.0 CPU, 0.0/0.0 GPU, 3.0 MiB/4.28 GiB object_store_memory: 95%|█████████▌| 76/80 [05:12<00:02, 1.90it/s] | |
Running: 3.0/16.0 CPU, 0.0/0.0 GPU, 3.0 MiB/4.28 GiB object_store_memory: 96%|█████████▋| 77/80 [05:12<00:02, 1.19it/s] | |
Running: 3.0/16.0 CPU, 0.0/0.0 GPU, 3.0 MiB/4.28 GiB object_store_memory: 98%|█████████▊| 78/80 [05:12<00:01, 1.44it/s] | |
Running: 2.0/16.0 CPU, 0.0/0.0 GPU, 2.0 MiB/4.28 GiB object_store_memory: 98%|█████████▊| 78/80 [05:12<00:01, 1.44it/s] | |
Running: 1.0/16.0 CPU, 0.0/0.0 GPU, 1.0 MiB/4.28 GiB object_store_memory: 98%|█████████▊| 78/80 [05:13<00:01, 1.44it/s] | |
Running: 1.0/16.0 CPU, 0.0/0.0 GPU, 1.0 MiB/4.28 GiB object_store_memory: 99%|█████████▉| 79/80 [05:13<00:00, 1.38it/s] | |
Running: 0.0/16.0 CPU, 0.0/0.0 GPU, 0.0 MiB/4.28 GiB object_store_memory: 99%|█████████▉| 79/80 [05:13<00:00, 1.38it/s] | |
2023-04-18 22:34:37,666 INFO streaming_executor.py:87 -- Executing DAG InputDataBuffer[Input] -> TaskPoolMapOperator[ReadRange->MapBatches(generate_features)->Write] | |
2023-04-18 22:34:37,666 INFO streaming_executor.py:88 -- Execution config: ExecutionOptions(resource_limits=ExecutionResources(cpu=None, gpu=None, object_store_memory=None), locality_with_output=False, preserve_order=False, actor_locality_enabled=True, verbose_progress=False) | |
2023-04-18 22:34:37,666 INFO streaming_executor.py:91 -- Tip: To enable per-operator progress reporting, set RAY_DATA_VERBOSE_PROGRESS=1. | |
Running 0: 0%| | 0/80 [00:00<?, ?it/s] | |
Running: 0.0/16.0 CPU, 0.0/0.0 GPU, 0.0 MiB/4.28 GiB object_store_memory: 0%| | 0/80 [00:00<?, ?it/s] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 0%| | 0/80 [00:00<?, ?it/s] | |
[2m[36m(ReadRange->MapBatches(generate_features)->Write pid=7216)[0m 2023-04-18 22:34:49.228824: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 AVX512F AVX512_VNNI FMA[32m [repeated 4x across cluster][0m | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 0%| | 0/80 [00:11<?, ?it/s] | |
[2m[36m(ReadRange->MapBatches(generate_features)->Write pid=7216)[0m To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.[32m [repeated 4x across cluster][0m | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 0%| | 0/80 [00:11<?, ?it/s] | |
[2m[36m(ReadRange->MapBatches(generate_features)->Write pid=3547)[0m 2023-04-18 22:29:56.934307: I tensorflow/core/util/port.cc:104] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.[32m [repeated 4x across cluster][0m | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 0%| | 0/80 [00:11<?, ?it/s] | |
[2m[36m(ReadRange->MapBatches(generate_features)->Write pid=3547)[0m 2023-04-18 22:29:58.277619: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/nvidia/lib:/usr/local/nvidia/lib64[32m [repeated 8x across cluster][0m | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 0%| | 0/80 [00:11<?, ?it/s] | |
[2m[36m(ReadRange->MapBatches(generate_features)->Write pid=3547)[0m 2023-04-18 22:29:58.277633: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.[32m [repeated 4x across cluster][0m | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 0%| | 0/80 [00:11<?, ?it/s] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 0%| | 0/80 [00:38<?, ?it/s] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 1%|▏ | 1/80 [00:38<50:45, 38.55s/it] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 1%|▏ | 1/80 [00:38<50:45, 38.55s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 1%|▏ | 1/80 [00:38<50:45, 38.55s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 2%|▎ | 2/80 [00:38<20:52, 16.06s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 4%|▍ | 3/80 [00:38<11:15, 8.77s/it] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 4%|▍ | 3/80 [00:39<11:15, 8.77s/it] | |
Running: 13.0/16.0 CPU, 0.0/0.0 GPU, 13.0 MiB/4.28 GiB object_store_memory: 4%|▍ | 3/80 [00:39<11:15, 8.77s/it] | |
Running: 13.0/16.0 CPU, 0.0/0.0 GPU, 13.0 MiB/4.28 GiB object_store_memory: 5%|▌ | 4/80 [00:39<06:49, 5.39s/it] | |
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 14.0 MiB/4.28 GiB object_store_memory: 8%|▊ | 6/80 [00:39<06:38, 5.39s/it] | |
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 14.0 MiB/4.28 GiB object_store_memory: 9%|▉ | 7/80 [00:39<02:27, 2.03s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 10%|█ | 8/80 [00:39<02:25, 2.03s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 11%|█▏ | 9/80 [00:39<01:31, 1.29s/it] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 11%|█▏ | 9/80 [00:39<01:31, 1.29s/it] | |
Running: 13.0/16.0 CPU, 0.0/0.0 GPU, 13.0 MiB/4.28 GiB object_store_memory: 11%|█▏ | 9/80 [00:39<01:31, 1.29s/it] | |
Running: 13.0/16.0 CPU, 0.0/0.0 GPU, 13.0 MiB/4.28 GiB object_store_memory: 14%|█▍ | 11/80 [00:39<01:00, 1.14it/s] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 15%|█▌ | 12/80 [00:39<00:59, 1.14it/s] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 15%|█▌ | 12/80 [00:39<00:59, 1.14it/s] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 16%|█▋ | 13/80 [00:39<00:41, 1.61it/s] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 16%|█▋ | 13/80 [00:39<00:41, 1.61it/s] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 16%|█▋ | 13/80 [00:43<00:41, 1.61it/s] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 18%|█▊ | 14/80 [00:43<00:40, 1.61it/s] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 19%|█▉ | 15/80 [00:43<01:10, 1.09s/it] | |
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 14.0 MiB/4.28 GiB object_store_memory: 20%|██ | 16/80 [00:43<01:09, 1.09s/it] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 20%|██ | 16/80 [00:44<01:09, 1.09s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 20%|██ | 16/80 [01:18<01:09, 1.09s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 21%|██▏ | 17/80 [01:18<06:32, 6.24s/it] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 21%|██▏ | 17/80 [01:18<06:32, 6.24s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 21%|██▏ | 17/80 [01:19<06:32, 6.24s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 22%|██▎ | 18/80 [01:19<05:28, 5.31s/it] | |
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 14.0 MiB/4.28 GiB object_store_memory: 22%|██▎ | 18/80 [01:20<05:28, 5.31s/it] | |
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 14.0 MiB/4.28 GiB object_store_memory: 24%|██▍ | 19/80 [01:20<04:18, 4.23s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 25%|██▌ | 20/80 [01:20<04:13, 4.23s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 26%|██▋ | 21/80 [01:20<02:37, 2.67s/it] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 28%|██▊ | 22/80 [01:20<02:35, 2.67s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 28%|██▊ | 22/80 [01:20<02:35, 2.67s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 29%|██▉ | 23/80 [01:20<01:45, 1.85s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 30%|███ | 24/80 [01:20<01:24, 1.51s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 32%|███▎ | 26/80 [01:21<00:53, 1.00it/s] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 32%|███▎ | 26/80 [01:21<00:53, 1.00it/s] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 32%|███▎ | 26/80 [01:21<00:53, 1.00it/s] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 34%|███▍ | 27/80 [01:21<00:47, 1.11it/s] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 35%|███▌ | 28/80 [01:21<00:37, 1.39it/s] | |
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 14.0 MiB/4.28 GiB object_store_memory: 36%|███▋ | 29/80 [01:21<00:36, 1.39it/s] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 36%|███▋ | 29/80 [01:21<00:36, 1.39it/s] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 36%|███▋ | 29/80 [01:24<00:36, 1.39it/s] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 38%|███▊ | 30/80 [01:24<00:44, 1.12it/s] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 38%|███▊ | 30/80 [01:24<00:44, 1.12it/s] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 39%|███▉ | 31/80 [01:25<00:49, 1.02s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 39%|███▉ | 31/80 [01:25<00:49, 1.02s/it] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 39%|███▉ | 31/80 [01:25<00:49, 1.02s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 39%|███▉ | 31/80 [01:27<00:49, 1.02s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 40%|████ | 32/80 [01:27<01:01, 1.29s/it] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 40%|████ | 32/80 [01:27<01:01, 1.29s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 40%|████ | 32/80 [01:59<01:01, 1.29s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 41%|████▏ | 33/80 [01:59<07:02, 9.00s/it] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 41%|████▏ | 33/80 [01:59<07:02, 9.00s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 41%|████▏ | 33/80 [02:00<07:02, 9.00s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 42%|████▎ | 34/80 [02:00<05:16, 6.89s/it] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 42%|████▎ | 34/80 [02:00<05:16, 6.89s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 42%|████▎ | 34/80 [02:00<05:16, 6.89s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 44%|████▍ | 35/80 [02:00<03:47, 5.05s/it] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 44%|████▍ | 35/80 [02:00<03:47, 5.05s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 44%|████▍ | 35/80 [02:00<03:47, 5.05s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 45%|████▌ | 36/80 [02:00<02:43, 3.71s/it] | |
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 14.0 MiB/4.28 GiB object_store_memory: 45%|████▌ | 36/80 [02:01<02:43, 3.71s/it] | |
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 14.0 MiB/4.28 GiB object_store_memory: 46%|████▋ | 37/80 [02:01<01:54, 2.67s/it] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 48%|████▊ | 38/80 [02:01<01:52, 2.67s/it] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 49%|████▉ | 39/80 [02:01<01:02, 1.52s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 49%|████▉ | 39/80 [02:01<01:02, 1.52s/it] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 49%|████▉ | 39/80 [02:01<01:02, 1.52s/it] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 50%|█████ | 40/80 [02:01<00:49, 1.23s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 50%|█████ | 40/80 [02:01<00:49, 1.23s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 51%|█████▏ | 41/80 [02:01<00:36, 1.07it/s] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 52%|█████▎ | 42/80 [02:01<00:27, 1.40it/s] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 52%|█████▎ | 42/80 [02:01<00:27, 1.40it/s] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 52%|█████▎ | 42/80 [02:02<00:27, 1.40it/s] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 54%|█████▍ | 43/80 [02:02<00:21, 1.73it/s] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 54%|█████▍ | 43/80 [02:02<00:21, 1.73it/s] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 54%|█████▍ | 43/80 [02:02<00:21, 1.73it/s] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 55%|█████▌ | 44/80 [02:02<00:20, 1.78it/s] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 55%|█████▌ | 44/80 [02:02<00:20, 1.78it/s] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 55%|█████▌ | 44/80 [02:02<00:20, 1.78it/s] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 56%|█████▋ | 45/80 [02:02<00:17, 2.03it/s] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 56%|█████▋ | 45/80 [02:02<00:17, 2.03it/s] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 56%|█████▋ | 45/80 [02:03<00:17, 2.03it/s] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 57%|█████▊ | 46/80 [02:03<00:22, 1.51it/s] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 57%|█████▊ | 46/80 [02:04<00:22, 1.51it/s] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 59%|█████▉ | 47/80 [02:05<00:30, 1.07it/s] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 59%|█████▉ | 47/80 [02:05<00:30, 1.07it/s] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 59%|█████▉ | 47/80 [02:05<00:30, 1.07it/s] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 59%|█████▉ | 47/80 [02:08<00:30, 1.07it/s] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 60%|██████ | 48/80 [02:08<00:48, 1.52s/it] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 60%|██████ | 48/80 [02:08<00:48, 1.52s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 60%|██████ | 48/80 [02:41<00:48, 1.52s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 61%|██████▏ | 49/80 [02:41<05:36, 10.85s/it] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 61%|██████▏ | 49/80 [02:41<05:36, 10.85s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 61%|██████▏ | 49/80 [02:41<05:36, 10.85s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 62%|██████▎ | 50/80 [02:41<03:52, 7.74s/it] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 62%|██████▎ | 50/80 [02:41<03:52, 7.74s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 62%|██████▎ | 50/80 [02:42<03:52, 7.74s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 64%|██████▍ | 51/80 [02:42<02:40, 5.53s/it] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 64%|██████▍ | 51/80 [02:42<02:40, 5.53s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 64%|██████▍ | 51/80 [02:42<02:40, 5.53s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 65%|██████▌ | 52/80 [02:42<01:51, 3.98s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 66%|██████▋ | 53/80 [02:42<01:16, 2.82s/it] | |
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 14.0 MiB/4.28 GiB object_store_memory: 68%|██████▊ | 54/80 [02:42<01:13, 2.82s/it] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 68%|██████▊ | 54/80 [02:42<01:13, 2.82s/it] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 69%|██████▉ | 55/80 [02:42<00:39, 1.57s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 69%|██████▉ | 55/80 [02:42<00:39, 1.57s/it] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 69%|██████▉ | 55/80 [02:42<00:39, 1.57s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 69%|██████▉ | 55/80 [02:43<00:39, 1.57s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 70%|███████ | 56/80 [02:43<00:31, 1.32s/it] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 70%|███████ | 56/80 [02:43<00:31, 1.32s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 70%|███████ | 56/80 [02:43<00:31, 1.32s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 71%|███████▏ | 57/80 [02:43<00:24, 1.06s/it] | |
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 14.0 MiB/4.28 GiB object_store_memory: 71%|███████▏ | 57/80 [02:43<00:24, 1.06s/it] | |
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 14.0 MiB/4.28 GiB object_store_memory: 72%|███████▎ | 58/80 [02:43<00:17, 1.25it/s] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 74%|███████▍ | 59/80 [02:43<00:16, 1.25it/s] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 74%|███████▍ | 59/80 [02:44<00:16, 1.25it/s] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 75%|███████▌ | 60/80 [02:44<00:12, 1.63it/s] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 75%|███████▌ | 60/80 [02:44<00:12, 1.63it/s] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 75%|███████▌ | 60/80 [02:44<00:12, 1.63it/s] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 76%|███████▋ | 61/80 [02:44<00:10, 1.84it/s] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 76%|███████▋ | 61/80 [02:44<00:10, 1.84it/s] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 76%|███████▋ | 61/80 [02:46<00:10, 1.84it/s] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 78%|███████▊ | 62/80 [02:46<00:14, 1.23it/s] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 78%|███████▊ | 62/80 [02:46<00:14, 1.23it/s] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 79%|███████▉ | 63/80 [02:47<00:15, 1.07it/s] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 79%|███████▉ | 63/80 [02:47<00:15, 1.07it/s] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 79%|███████▉ | 63/80 [02:47<00:15, 1.07it/s] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 79%|███████▉ | 63/80 [02:51<00:15, 1.07it/s] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 80%|████████ | 64/80 [02:51<00:25, 1.61s/it] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 80%|████████ | 64/80 [02:51<00:25, 1.61s/it] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 81%|████████▏ | 65/80 [03:23<02:32, 10.17s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 81%|████████▏ | 65/80 [03:23<02:32, 10.17s/it] | |
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 14.0 MiB/4.28 GiB object_store_memory: 81%|████████▏ | 65/80 [03:23<02:32, 10.17s/it] | |
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 14.0 MiB/4.28 GiB object_store_memory: 82%|████████▎ | 66/80 [03:23<01:41, 7.28s/it] | |
Running: 13.0/16.0 CPU, 0.0/0.0 GPU, 13.0 MiB/4.28 GiB object_store_memory: 82%|████████▎ | 66/80 [03:23<01:41, 7.28s/it] | |
Running: 12.0/16.0 CPU, 0.0/0.0 GPU, 12.0 MiB/4.28 GiB object_store_memory: 84%|████████▍ | 67/80 [03:23<01:34, 7.28s/it] | |
Running: 12.0/16.0 CPU, 0.0/0.0 GPU, 12.0 MiB/4.28 GiB object_store_memory: 85%|████████▌ | 68/80 [03:23<00:48, 4.04s/it] | |
Running: 11.0/16.0 CPU, 0.0/0.0 GPU, 11.0 MiB/4.28 GiB object_store_memory: 85%|████████▌ | 68/80 [03:23<00:48, 4.04s/it] | |
Running: 11.0/16.0 CPU, 0.0/0.0 GPU, 11.0 MiB/4.28 GiB object_store_memory: 86%|████████▋ | 69/80 [03:23<00:34, 3.13s/it] | |
Running: 11.0/16.0 CPU, 0.0/0.0 GPU, 11.0 MiB/4.28 GiB object_store_memory: 88%|████████▊ | 70/80 [03:23<00:23, 2.35s/it] | |
Running: 10.0/16.0 CPU, 0.0/0.0 GPU, 10.0 MiB/4.28 GiB object_store_memory: 88%|████████▊ | 70/80 [03:23<00:23, 2.35s/it] | |
Running: 9.0/16.0 CPU, 0.0/0.0 GPU, 9.0 MiB/4.28 GiB object_store_memory: 88%|████████▊ | 70/80 [03:24<00:23, 2.35s/it] | |
Running: 9.0/16.0 CPU, 0.0/0.0 GPU, 9.0 MiB/4.28 GiB object_store_memory: 89%|████████▉ | 71/80 [03:24<00:16, 1.80s/it] | |
Running: 8.0/16.0 CPU, 0.0/0.0 GPU, 8.0 MiB/4.28 GiB object_store_memory: 89%|████████▉ | 71/80 [03:24<00:16, 1.80s/it] | |
Running: 8.0/16.0 CPU, 0.0/0.0 GPU, 8.0 MiB/4.28 GiB object_store_memory: 90%|█████████ | 72/80 [03:24<00:11, 1.38s/it] | |
Running: 7.0/16.0 CPU, 0.0/0.0 GPU, 7.0 MiB/4.28 GiB object_store_memory: 90%|█████████ | 72/80 [03:24<00:11, 1.38s/it] | |
Running: 7.0/16.0 CPU, 0.0/0.0 GPU, 7.0 MiB/4.28 GiB object_store_memory: 91%|█████████▏| 73/80 [03:24<00:07, 1.05s/it] | |
Running: 6.0/16.0 CPU, 0.0/0.0 GPU, 6.0 MiB/4.28 GiB object_store_memory: 91%|█████████▏| 73/80 [03:24<00:07, 1.05s/it] | |
Running: 6.0/16.0 CPU, 0.0/0.0 GPU, 6.0 MiB/4.28 GiB object_store_memory: 92%|█████████▎| 74/80 [03:24<00:04, 1.24it/s] | |
Running: 5.0/16.0 CPU, 0.0/0.0 GPU, 5.0 MiB/4.28 GiB object_store_memory: 92%|█████████▎| 74/80 [03:24<00:04, 1.24it/s] | |
Running: 5.0/16.0 CPU, 0.0/0.0 GPU, 5.0 MiB/4.28 GiB object_store_memory: 94%|█████████▍| 75/80 [03:24<00:02, 1.67it/s] | |
Running: 4.0/16.0 CPU, 0.0/0.0 GPU, 4.0 MiB/4.28 GiB object_store_memory: 94%|█████████▍| 75/80 [03:24<00:02, 1.67it/s] | |
Running: 4.0/16.0 CPU, 0.0/0.0 GPU, 4.0 MiB/4.28 GiB object_store_memory: 95%|█████████▌| 76/80 [03:24<00:01, 2.21it/s] | |
Running: 3.0/16.0 CPU, 0.0/0.0 GPU, 3.0 MiB/4.28 GiB object_store_memory: 95%|█████████▌| 76/80 [03:25<00:01, 2.21it/s] | |
Running: 3.0/16.0 CPU, 0.0/0.0 GPU, 3.0 MiB/4.28 GiB object_store_memory: 96%|█████████▋| 77/80 [03:25<00:01, 2.87it/s] | |
Running: 2.0/16.0 CPU, 0.0/0.0 GPU, 2.0 MiB/4.28 GiB object_store_memory: 96%|█████████▋| 77/80 [03:25<00:01, 2.87it/s] | |
Running: 2.0/16.0 CPU, 0.0/0.0 GPU, 2.0 MiB/4.28 GiB object_store_memory: 98%|█████████▊| 78/80 [03:25<00:00, 2.34it/s] | |
Running: 1.0/16.0 CPU, 0.0/0.0 GPU, 1.0 MiB/4.28 GiB object_store_memory: 98%|█████████▊| 78/80 [03:27<00:00, 2.34it/s] | |
Running: 1.0/16.0 CPU, 0.0/0.0 GPU, 1.0 MiB/4.28 GiB object_store_memory: 99%|█████████▉| 79/80 [03:27<00:00, 1.39it/s] | |
Running: 0.0/16.0 CPU, 0.0/0.0 GPU, 0.0 MiB/4.28 GiB object_store_memory: 99%|█████████▉| 79/80 [03:28<00:00, 1.39it/s] | |
Running: 0.0/16.0 CPU, 0.0/0.0 GPU, 0.0 MiB/4.28 GiB object_store_memory: 100%|██████████| 80/80 [03:28<00:00, 1.28it/s] | |
2023-04-18 22:38:05,765 INFO streaming_executor.py:87 -- Executing DAG InputDataBuffer[Input] -> TaskPoolMapOperator[ReadRange->MapBatches(generate_features)->Write] | |
2023-04-18 22:38:05,765 INFO streaming_executor.py:88 -- Execution config: ExecutionOptions(resource_limits=ExecutionResources(cpu=None, gpu=None, object_store_memory=None), locality_with_output=False, preserve_order=False, actor_locality_enabled=True, verbose_progress=False) | |
2023-04-18 22:38:05,765 INFO streaming_executor.py:91 -- Tip: To enable per-operator progress reporting, set RAY_DATA_VERBOSE_PROGRESS=1. | |
Running 0: 0%| | 0/32 [00:00<?, ?it/s] | |
Running: 0.0/16.0 CPU, 0.0/0.0 GPU, 0.0 MiB/4.28 GiB object_store_memory: 0%| | 0/32 [00:00<?, ?it/s] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 4.0 MiB/4.28 GiB object_store_memory: 0%| | 0/32 [00:00<?, ?it/s] | |
[2m[36m(ReadRange->MapBatches(generate_features)->Write pid=9742)[0m 2023-04-18 22:38:14.549368: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 AVX512F AVX512_VNNI FMA[32m [repeated 3x across cluster][0m | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 4.0 MiB/4.28 GiB object_store_memory: 0%| | 0/32 [00:08<?, ?it/s] | |
[2m[36m(ReadRange->MapBatches(generate_features)->Write pid=9742)[0m To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.[32m [repeated 3x across cluster][0m | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 4.0 MiB/4.28 GiB object_store_memory: 0%| | 0/32 [00:08<?, ?it/s] | |
[2m[36m(ReadRange->MapBatches(generate_features)->Write pid=7218)[0m 2023-04-18 22:34:49.600254: I tensorflow/core/util/port.cc:104] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.[32m [repeated 3x across cluster][0m | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 4.0 MiB/4.28 GiB object_store_memory: 0%| | 0/32 [00:08<?, ?it/s] | |
[2m[36m(ReadRange->MapBatches(generate_features)->Write pid=7218)[0m 2023-04-18 22:34:50.854900: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/nvidia/lib:/usr/local/nvidia/lib64[32m [repeated 6x across cluster][0m | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 4.0 MiB/4.28 GiB object_store_memory: 0%| | 0/32 [00:08<?, ?it/s] | |
[2m[36m(ReadRange->MapBatches(generate_features)->Write pid=7218)[0m 2023-04-18 22:34:50.854914: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.[32m [repeated 3x across cluster][0m | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 4.0 MiB/4.28 GiB object_store_memory: 0%| | 0/32 [00:08<?, ?it/s] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 4.0 MiB/4.28 GiB object_store_memory: 3%|▎ | 1/32 [00:12<06:26, 12.45s/it] | |
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 3.5 MiB/4.28 GiB object_store_memory: 6%|▋ | 2/32 [00:12<06:13, 12.45s/it] | |
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 3.5 MiB/4.28 GiB object_store_memory: 9%|▉ | 3/32 [00:12<01:34, 3.27s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 3.75 MiB/4.28 GiB object_store_memory: 16%|█▌ | 5/32 [00:12<01:28, 3.27s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 3.75 MiB/4.28 GiB object_store_memory: 19%|█▉ | 6/32 [00:12<00:34, 1.31s/it] | |
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 3.5 MiB/4.28 GiB object_store_memory: 19%|█▉ | 6/32 [00:12<00:34, 1.31s/it] | |
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 3.5 MiB/4.28 GiB object_store_memory: 25%|██▌ | 8/32 [00:12<00:20, 1.18it/s] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 4.0 MiB/4.28 GiB object_store_memory: 25%|██▌ | 8/32 [00:13<00:20, 1.18it/s] | |
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 3.5 MiB/4.28 GiB object_store_memory: 25%|██▌ | 8/32 [00:13<00:20, 1.18it/s] | |
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 3.5 MiB/4.28 GiB object_store_memory: 31%|███▏ | 10/32 [00:13<00:13, 1.63it/s] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 4.0 MiB/4.28 GiB object_store_memory: 31%|███▏ | 10/32 [00:13<00:13, 1.63it/s] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 3.75 MiB/4.28 GiB object_store_memory: 31%|███▏ | 10/32 [00:13<00:13, 1.63it/s] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 3.75 MiB/4.28 GiB object_store_memory: 38%|███▊ | 12/32 [00:13<00:09, 2.15it/s] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 4.0 MiB/4.28 GiB object_store_memory: 38%|███▊ | 12/32 [00:13<00:09, 2.15it/s] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 4.0 MiB/4.28 GiB object_store_memory: 41%|████ | 13/32 [00:13<00:07, 2.41it/s] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 3.75 MiB/4.28 GiB object_store_memory: 41%|████ | 13/32 [00:13<00:07, 2.41it/s] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 4.0 MiB/4.28 GiB object_store_memory: 41%|████ | 13/32 [00:13<00:07, 2.41it/s] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 3.75 MiB/4.28 GiB object_store_memory: 41%|████ | 13/32 [00:17<00:07, 2.41it/s] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 3.75 MiB/4.28 GiB object_store_memory: 44%|████▍ | 14/32 [00:17<00:20, 1.14s/it] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 3.75 MiB/4.28 GiB object_store_memory: 47%|████▋ | 15/32 [00:17<00:15, 1.11it/s] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 4.0 MiB/4.28 GiB object_store_memory: 47%|████▋ | 15/32 [00:17<00:15, 1.11it/s] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 4.0 MiB/4.28 GiB object_store_memory: 50%|█████ | 16/32 [00:18<00:13, 1.17it/s] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 3.75 MiB/4.28 GiB object_store_memory: 50%|█████ | 16/32 [00:18<00:13, 1.17it/s] | |
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 4.0 MiB/4.28 GiB object_store_memory: 50%|█████ | 16/32 [00:18<00:13, 1.17it/s] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 3.75 MiB/4.28 GiB object_store_memory: 50%|█████ | 16/32 [00:24<00:13, 1.17it/s] | |
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 3.75 MiB/4.28 GiB object_store_memory: 53%|█████▎ | 17/32 [00:24<00:35, 2.34s/it] | |
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 3.5 MiB/4.28 GiB object_store_memory: 53%|█████▎ | 17/32 [00:25<00:35, 2.34s/it] | |
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 3.5 MiB/4.28 GiB object_store_memory: 56%|█████▋ | 18/32 [00:25<00:24, 1.73s/it] | |
Running: 13.0/16.0 CPU, 0.0/0.0 GPU, 3.25 MiB/4.28 GiB object_store_memory: 56%|█████▋ | 18/32 [00:25<00:24, 1.73s/it] | |
Running: 13.0/16.0 CPU, 0.0/0.0 GPU, 3.25 MiB/4.28 GiB object_store_memory: 59%|█████▉ | 19/32 [00:25<00:16, 1.27s/it] | |
Running: 11.0/16.0 CPU, 0.0/0.0 GPU, 2.75 MiB/4.28 GiB object_store_memory: 59%|█████▉ | 19/32 [00:25<00:16, 1.27s/it] | |
Running: 11.0/16.0 CPU, 0.0/0.0 GPU, 2.75 MiB/4.28 GiB object_store_memory: 62%|██████▎ | 20/32 [00:25<00:11, 1.07it/s] | |
Running: 8.0/16.0 CPU, 0.0/0.0 GPU, 2.0 MiB/4.28 GiB object_store_memory: 66%|██████▌ | 21/32 [00:25<00:10, 1.07it/s] | |
Running: 8.0/16.0 CPU, 0.0/0.0 GPU, 2.0 MiB/4.28 GiB object_store_memory: 69%|██████▉ | 22/32 [00:25<00:06, 1.58it/s] | |
Running: 7.0/16.0 CPU, 0.0/0.0 GPU, 1.75 MiB/4.28 GiB object_store_memory: 75%|███████▌ | 24/32 [00:25<00:05, 1.58it/s] | |
Running: 7.0/16.0 CPU, 0.0/0.0 GPU, 1.75 MiB/4.28 GiB object_store_memory: 78%|███████▊ | 25/32 [00:25<00:02, 2.96it/s] | |
Running: 4.0/16.0 CPU, 0.0/0.0 GPU, 1.0 MiB/4.28 GiB object_store_memory: 78%|███████▊ | 25/32 [00:25<00:02, 2.96it/s] | |
Running: 4.0/16.0 CPU, 0.0/0.0 GPU, 1.0 MiB/4.28 GiB object_store_memory: 84%|████████▍ | 27/32 [00:25<00:01, 4.05it/s] | |
Running: 3.0/16.0 CPU, 0.0/0.0 GPU, 0.75 MiB/4.28 GiB object_store_memory: 88%|████████▊ | 28/32 [00:26<00:00, 4.05it/s] | |
Running: 3.0/16.0 CPU, 0.0/0.0 GPU, 0.75 MiB/4.28 GiB object_store_memory: 91%|█████████ | 29/32 [00:26<00:00, 4.97it/s] | |
Running: 2.0/16.0 CPU, 0.0/0.0 GPU, 0.5 MiB/4.28 GiB object_store_memory: 94%|█████████▍| 30/32 [00:27<00:00, 4.97it/s] | |
Running: 1.0/16.0 CPU, 0.0/0.0 GPU, 0.25 MiB/4.28 GiB object_store_memory: 94%|█████████▍| 30/32 [00:27<00:00, 4.97it/s] | |
Running: 1.0/16.0 CPU, 0.0/0.0 GPU, 0.25 MiB/4.28 GiB object_store_memory: 97%|█████████▋| 31/32 [00:27<00:00, 2.37it/s] | |
Running: 0.0/16.0 CPU, 0.0/0.0 GPU, 0.0 MiB/4.28 GiB object_store_memory: 97%|█████████▋| 31/32 [00:28<00:00, 2.37it/s] | |
Running: 0.0/16.0 CPU, 0.0/0.0 GPU, 0.0 MiB/4.28 GiB object_store_memory: 100%|██████████| 32/32 [00:28<00:00, 2.65it/s] | |
Running case: tfrecords-images-100-256 | |
Read progress 0: 0%| | 0/32 [00:00<?, ?it/s] | |
Result of case tfrecords-images-100-256: {'time': 0.11246879999998782} | |
Running case: tfrecords-images-100-2048 | |
Read progress 0: 0%| | 0/100 [00:00<?, ?it/s] | |
Read progress 0: 15%|█▌ | 15/100 [00:00<00:00, 106.12it/s] | |
Read progress 0: 26%|██▌ | 26/100 [00:00<00:01, 64.49it/s] | |
Read progress 0: 37%|███▋ | 37/100 [00:00<00:00, 71.89it/s] | |
Read progress 0: 49%|████▉ | 49/100 [00:00<00:00, 82.14it/s] | |
Read progress 0: 61%|██████ | 61/100 [00:00<00:00, 85.51it/s] | |
Read progress 0: 74%|███████▍ | 74/100 [00:00<00:00, 85.95it/s] | |
Read progress 0: 86%|████████▌ | 86/100 [00:01<00:00, 94.34it/s] | |
[2m[36m(_execute_read_task_split pid=10303)[0m 2023-04-18 22:38:35.837302: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 AVX512F AVX512_VNNI FMA[32m [repeated 3x across cluster][0m | |
Read progress 0: 97%|█████████▋| 97/100 [00:01<00:00, 94.34it/s] | |
[2m[36m(_execute_read_task_split pid=10303)[0m To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.[32m [repeated 3x across cluster][0m | |
Read progress 0: 97%|█████████▋| 97/100 [00:01<00:00, 94.34it/s] | |
[2m[36m(ReadRange->MapBatches(generate_features)->Write pid=9740)[0m 2023-04-18 22:38:15.385531: I tensorflow/core/util/port.cc:104] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.[32m [repeated 3x across cluster][0m | |
Read progress 0: 97%|█████████▋| 97/100 [00:01<00:00, 94.34it/s] | |
[2m[36m(ReadRange->MapBatches(generate_features)->Write pid=9740)[0m 2023-04-18 22:38:16.881614: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/nvidia/lib:/usr/local/nvidia/lib64[32m [repeated 6x across cluster][0m | |
Read progress 0: 97%|█████████▋| 97/100 [00:01<00:00, 94.34it/s] | |
[2m[36m(ReadRange->MapBatches(generate_features)->Write pid=9740)[0m 2023-04-18 22:38:16.881630: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.[32m [repeated 3x across cluster][0m | |
Read progress 0: 97%|█████████▋| 97/100 [00:01<00:00, 94.34it/s] | |
Read progress 0: 98%|█████████▊| 98/100 [00:03<00:00, 14.29it/s] | |
Result of case tfrecords-images-100-2048: {'time': 3.969540747999986} | |
Running case: tfrecords-images-1000-mix | |
Read progress 0: 0%| | 0/32 [00:00<?, ?it/s] | |
Result of case tfrecords-images-1000-mix: {'time': 0.0678202810000812} | |
Running case: tfrecords-random-int-1g | |
Read progress 0: 0%| | 0/80 [00:00<?, ?it/s] | |
[2m[36m(_execute_read_task_split pid=11572)[0m 2023-04-18 22:38:39.985242: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 AVX512F AVX512_VNNI FMA[32m [repeated 5x across cluster][0m | |
Read progress 0: 0%| | 0/80 [00:03<?, ?it/s] | |
[2m[36m(_execute_read_task_split pid=11572)[0m To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.[32m [repeated 5x across cluster][0m | |
Read progress 0: 0%| | 0/80 [00:03<?, ?it/s] | |
[2m[36m(_execute_read_task_split pid=11572)[0m 2023-04-18 22:38:40.166711: I tensorflow/core/util/port.cc:104] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.[32m [repeated 6x across cluster][0m | |
Read progress 0: 0%| | 0/80 [00:03<?, ?it/s] | |
[2m[36m(_execute_read_task_split pid=11570)[0m 2023-04-18 22:38:41.374321: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/nvidia/lib:/usr/local/nvidia/lib64[32m [repeated 8x across cluster][0m | |
Read progress 0: 0%| | 0/80 [00:03<?, ?it/s] | |
[2m[36m(_execute_read_task_split pid=11570)[0m 2023-04-18 22:38:41.374332: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.[32m [repeated 4x across cluster][0m | |
Read progress 0: 0%| | 0/80 [00:03<?, ?it/s] | |
Read progress 0: 1%|▏ | 1/80 [04:02<5:19:39, 242.78s/it] | |
Read progress 0: 2%|▎ | 2/80 [04:04<2:11:01, 100.79s/it] | |
Read progress 0: 6%|▋ | 5/80 [04:04<35:58, 28.78s/it] | |
Read progress 0: 8%|▊ | 6/80 [04:04<26:31, 21.50s/it] | |
Read progress 0: 9%|▉ | 7/80 [04:05<19:14, 15.81s/it] | |
Read progress 0: 10%|█ | 8/80 [04:05<13:58, 11.65s/it] | |
Read progress 0: 11%|█▏ | 9/80 [04:05<09:56, 8.40s/it] | |
Read progress 0: 12%|█▎ | 10/80 [04:06<07:01, 6.02s/it] | |
Read progress 0: 14%|█▍ | 11/80 [04:06<04:56, 4.30s/it] | |
Read progress 0: 15%|█▌ | 12/80 [04:06<03:34, 3.15s/it] | |
Read progress 0: 18%|█▊ | 14/80 [04:08<02:15, 2.06s/it] | |
Read progress 0: 19%|█▉ | 15/80 [04:09<02:00, 1.86s/it] | |
Read progress 0: 21%|██▏ | 17/80 [08:05<52:18, 49.82s/it] | |
Read progress 0: 22%|██▎ | 18/80 [08:05<39:50, 38.56s/it] | |
Read progress 0: 25%|██▌ | 20/80 [08:07<23:40, 23.68s/it] | |
Read progress 0: 28%|██▊ | 22/80 [08:07<14:42, 15.21s/it] | |
Read progress 0: 29%|██▉ | 23/80 [08:09<11:45, 12.38s/it] | |
Read progress 0: 30%|███ | 24/80 [08:09<09:00, 9.66s/it] | |
Read progress 0: 31%|███▏ | 25/80 [08:09<06:45, 7.38s/it] | |
Read progress 0: 32%|███▎ | 26/80 [08:10<04:57, 5.51s/it] | |
Read progress 0: 34%|███▍ | 27/80 [08:10<03:40, 4.16s/it] | |
Read progress 0: 35%|███▌ | 28/80 [08:11<02:42, 3.12s/it] | |
Read progress 0: 36%|███▋ | 29/80 [08:11<02:02, 2.40s/it] | |
Read progress 0: 38%|███▊ | 30/80 [08:11<01:27, 1.76s/it] | |
Read progress 0: 39%|███▉ | 31/80 [08:12<01:11, 1.46s/it] | |
Read progress 0: 40%|████ | 32/80 [08:12<00:52, 1.09s/it] | |
Read progress 0: 41%|████▏ | 33/80 [12:08<55:21, 70.66s/it] | |
Read progress 0: 42%|████▎ | 34/80 [12:10<38:21, 50.04s/it] | |
Read progress 0: 44%|████▍ | 35/80 [12:11<26:35, 35.46s/it] | |
Read progress 0: 45%|████▌ | 36/80 [12:11<18:19, 25.00s/it] | |
Read progress 0: 46%|████▋ | 37/80 [12:12<12:48, 17.86s/it] | |
Read progress 0: 48%|████▊ | 38/80 [12:13<08:55, 12.74s/it] | |
Read progress 0: 49%|████▉ | 39/80 [12:15<06:24, 9.38s/it] | |
Read progress 0: 50%|█████ | 40/80 [12:15<04:25, 6.64s/it] | |
Read progress 0: 52%|█████▎ | 42/80 [12:15<02:18, 3.65s/it] | |
Read progress 0: 54%|█████▍ | 43/80 [12:16<01:52, 3.05s/it] | |
Read progress 0: 55%|█████▌ | 44/80 [12:17<01:25, 2.37s/it] | |
Read progress 0: 56%|█████▋ | 45/80 [12:18<01:09, 1.98s/it] | |
Read progress 0: 57%|█████▊ | 46/80 [12:18<00:50, 1.49s/it] | |
Read progress 0: 60%|██████ | 48/80 [12:18<00:28, 1.13it/s] | |
Read progress 0: 61%|██████▏ | 49/80 [16:12<29:23, 56.89s/it] | |
Read progress 0: 62%|██████▎ | 50/80 [16:14<21:30, 43.03s/it] | |
Read progress 0: 64%|██████▍ | 51/80 [16:15<15:19, 31.70s/it] | |
Read progress 0: 65%|██████▌ | 52/80 [16:16<10:49, 23.20s/it] | |
Read progress 0: 66%|██████▋ | 53/80 [16:18<07:44, 17.19s/it] | |
Read progress 0: 69%|██████▉ | 55/80 [16:19<04:02, 9.70s/it] | |
Read progress 0: 70%|███████ | 56/80 [16:20<03:03, 7.65s/it] | |
Read progress 0: 71%|███████▏ | 57/80 [16:20<02:12, 5.77s/it] | |
Read progress 0: 72%|███████▎ | 58/80 [16:21<01:35, 4.36s/it] | |
Read progress 0: 74%|███████▍ | 59/80 [16:22<01:10, 3.36s/it] | |
Read progress 0: 75%|███████▌ | 60/80 [16:23<00:54, 2.72s/it] | |
Read progress 0: 78%|███████▊ | 62/80 [16:23<00:28, 1.56s/it] | |
Read progress 0: 79%|███████▉ | 63/80 [16:24<00:24, 1.45s/it] | |
Read progress 0: 80%|████████ | 64/80 [16:25<00:19, 1.19s/it]Subprocess return code: 124 | |
[INFO 2023-04-18 22:57:37,602] anyscale_job_wrapper.py: 191 Process 1447 exited with return code 124. | |
[ERROR 2023-04-18 22:57:37,603] anyscale_job_wrapper.py: 291 Timed out. Time taken: 1800.069069077 | |
[ERROR 2023-04-18 22:57:37,603] anyscale_job_wrapper.py: 68 Couldn't upload to cloud storage: '/tmp/release_test_out.json' does not exist. | |
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts. | |
azure-cli-core 2.40.0 requires packaging<22.0,>=20.9, but you have packaging 23.1 which is incompatible. | |
mlflow 1.30.0 requires packaging<22, but you have packaging 23.1 which is incompatible. | |
mosaicml 0.12.1 requires importlib-metadata<7,>=5.0.0, but you have importlib-metadata 4.13.0 which is incompatible. | |
mosaicml 0.12.1 requires packaging<23,>=21.3.0, but you have packaging 23.1 which is incompatible. | |
mosaicml 0.12.1 requires pyyaml<7,>=6.0, but you have pyyaml 5.4.1 which is incompatible. | |
tensorflow 2.11.0 requires protobuf<3.20,>=3.9.2, but you have protobuf 3.20.3 which is incompatible. | |
Copying file:///tmp/metrics_test_out.json [Content-Type=application/json]... | |
/ [0 files][ 0.0 B/ 374.0 B] | |
/ [1 files][ 374.0 B/ 374.0 B] | |
Operation completed over 1 objects/374.0 B. | |
Read progress 0: 81%|████████▏ | 65/80 [20:17<15:48, 63.25s/it]Copying file:///tmp/ray/session_2023-04-18_22-27-18_471776_158/runtime_resources/working_dir_files/gs_anyscale-oss-dev-bucket_working_dirs_read_tfrecords_benchmark_single_node_gce_yitigazrhh__anyscale_pkg_aaf02dfc5db28595fbcfe4cd80cdeb4b/output.json [Content-Type=application/json]... | |
/ [0 files][ 0.0 B/ 247.0 B] | |
/ [1 files][ 247.0 B/ 247.0 B] | |
Operation completed over 1 objects/247.0 B. | |
[INFO 2023-04-18 22:58:57,590] anyscale_job_wrapper.py: 346 ### Finished ### | |
[INFO 2023-04-18 22:58:57,591] anyscale_job_wrapper.py: 349 ### JSON |{"collected_metrics":true,"last_prepare_time_taken":null,"prepare_return_codes":[],"return_code":124,"total_time_taken":1878.405071267,"uploaded_artifact":false,"uploaded_metrics":true,"uploaded_results":false,"workload_time_taken":1800.069069077}| ### | |
Read progress 0: 82%|████████▎ | 66/80 [20:20<10:51, 46.55s/it] | |
Read progress 0: 84%|████████▍ | 67/80 [20:21<07:19, 33.80s/it] | |
Read progress 0: 86%|████████▋ | 69/80 [20:24<03:30, 19.17s/it] | |
Read progress 0: 88%|████████▊ | 70/80 [20:24<02:25, 14.57s/it] | |
Read progress 0: 89%|████████▉ | 71/80 [20:25<01:39, 11.03s/it] | |
Read progress 0: 90%|█████████ | 72/80 [20:26<01:07, 8.45s/it] | |
Read progress 0: 91%|█████████▏| 73/80 [20:26<00:43, 6.16s/it] | |
Read progress 0: 92%|█████████▎| 74/80 [20:27<00:27, 4.57s/it] | |
Read progress 0: 95%|█████████▌| 76/80 [20:28<00:10, 2.74s/it] | |
Read progress 0: 98%|█████████▊| 78/80 [20:29<00:03, 1.91s/it] | |
Read progress 0: 99%|█████████▉| 79/80 [20:30<00:01, 1.59s/it] | |
Read progress 0: 100%|██████████| 80/80 [20:31<00:00, 1.46s/it] | |
Result of case tfrecords-random-int-1g: {'time': 1231.250032476} | |
Running case: tfrecords-random-float-1g | |
Read progress 0: 0%| | 0/80 [00:00<?, ?it/s] | |
[2m[36m(_execute_read_task_split pid=25946)[0m To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags. | |
Read progress 0: 0%| | 0/80 [00:01<?, ?it/s] | |
[2m[36m(_execute_read_task_split pid=25946)[0m To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags. | |
Read progress 0: 0%| | 0/80 [00:01<?, ?it/s] | |
[2m[36m(_execute_read_task_split pid=11572)[0m 2023-04-18 22:38:41.466254: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/nvidia/lib:/usr/local/nvidia/lib64[32m [repeated 4x across cluster][0m | |
Read progress 0: 0%| | 0/80 [00:01<?, ?it/s] | |
[2m[36m(_execute_read_task_split pid=11572)[0m 2023-04-18 22:38:41.466269: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.[32m [repeated 2x across cluster][0m | |
Read progress 0: 0%| | 0/80 [00:01<?, ?it/s] | |
[2m[36m(_execute_read_task_split pid=25947)[0m 2023-04-18 22:59:11.160589: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 AVX512F AVX512_VNNI FMA | |
Read progress 0: 0%| | 0/80 [00:01<?, ?it/s] | |
[2m[36m(_execute_read_task_split pid=25947)[0m To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags. | |
Read progress 0: 0%| | 0/80 [00:01<?, ?it/s] | |
[2m[36m(_execute_read_task_split pid=25946)[0m 2023-04-18 22:59:11.328129: I tensorflow/core/util/port.cc:104] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`. | |
Read progress 0: 0%| | 0/80 [00:01<?, ?it/s]*** SIGTERM received at time=1681883967 on cpu 15 *** | |
PC: @ 0x7fb71170dd0b (unknown) unlinkat | |
@ 0x7fb71195e420 (unknown) (unknown) | |
[2023-04-18 22:59:27,143 E 1454 1454] logging.cc:361: *** SIGTERM received at time=1681883967 on cpu 15 *** | |
[2023-04-18 22:59:27,143 E 1454 1454] logging.cc:361: PC: @ 0x7fb71170dd0b (unknown) unlinkat | |
[2023-04-18 22:59:27,143 E 1454 1454] logging.cc:361: @ 0x7fb71195e420 (unknown) (unknown) | |
[2m[36m(_execute_read_task_split pid=25946)[0m 2023-04-18 22:59:12.658665: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/nvidia/lib:/usr/local/nvidia/lib64[32m [repeated 4x across cluster][0m | |
Read progress 0: 0%| | 0/80 [00:17<?, ?it/s] | |
[2m[36m(_execute_read_task_split pid=25946)[0m 2023-04-18 22:59:12.658679: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.[32m [repeated 2x across cluster][0m | |
Read progress 0: 0%| | 0/80 [00:17<?, ?it/s] | |
[2m[36m(_execute_read_task_split pid=25947)[0m 2023-04-18 22:59:11.327625: I tensorflow/core/util/port.cc:104] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment