|
$ python tf_cnn_benchmarks.py --num_gpus=1 --batch_size=8 --model=resnet50 --variable_update=parameter_server |
|
2019-06-08 15:16:45.067180: W tensorflow/core/platform/profile_utils/cpu_utils.cc:98] Failed to find bogomips in /proc/cpuinfo; cannot determine CPU frequency |
|
2019-06-08 15:16:45.068214: I tensorflow/compiler/xla/service/service.cc:161] XLA service 0x55af02e3a0 executing computations on platform Host. Devices: |
|
2019-06-08 15:16:45.068302: I tensorflow/compiler/xla/service/service.cc:168] StreamExecutor device (0): <undefined>, <undefined> |
|
2019-06-08 15:16:45.220278: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:965] ARM64 does not support NUMA - returning NUMA node zero |
|
2019-06-08 15:16:45.220560: I tensorflow/compiler/xla/service/service.cc:161] XLA service 0x55adb2bdc0 executing computations on platform CUDA. Devices: |
|
2019-06-08 15:16:45.220642: I tensorflow/compiler/xla/service/service.cc:168] StreamExecutor device (0): NVIDIA Tegra X1, Compute Capability 5.3 |
|
2019-06-08 15:16:45.221057: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1433] Found device 0 with properties: |
|
name: NVIDIA Tegra X1 major: 5 minor: 3 memoryClockRate(GHz): 0.9216 |
|
pciBusID: 0000:00:00.0 |
|
totalMemory: 3.87GiB freeMemory: 2.37GiB |
|
2019-06-08 15:16:45.221150: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 0 |
|
2019-06-08 15:16:50.816566: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix: |
|
2019-06-08 15:16:50.816650: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 0 |
|
2019-06-08 15:16:50.816688: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 0: N |
|
2019-06-08 15:16:50.816885: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 1648 MB memory) -> physical GPU (device: 0, name: NVIDIA Tegra X1, pci bus id: 0000:00:00.0, compute capability: 5.3) |
|
TensorFlow: 1.13 |
|
Model: resnet50 |
|
Dataset: imagenet (synthetic) |
|
Mode: training |
|
SingleSess: False |
|
Batch size: 8 global |
|
8 per device |
|
Num batches: 100 |
|
Num epochs: 0.00 |
|
Devices: ['/gpu:0'] |
|
NUMA bind: False |
|
Data format: NCHW |
|
Optimizer: sgd |
|
Variables: parameter_server |
|
========== |
|
Generating training model |
|
W0608 15:16:50.843825 548390839744 deprecation.py:323] From /home/miku/venv36/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version. |
|
Instructions for updating: |
|
Colocations handled automatically by placer. |
|
W0608 15:16:50.933079 548390839744 deprecation.py:323] From /home/miku/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:129: conv2d (from tensorflow.python.layers.convolutional) is deprecated and will be removed in a future version. |
|
Instructions for updating: |
|
Use keras.layers.conv2d instead. |
|
W0608 15:16:51.148540 548390839744 deprecation.py:323] From /home/miku/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:261: max_pooling2d (from tensorflow.python.layers.pooling) is deprecated and will be removed in a future version. |
|
Instructions for updating: |
|
Use keras.layers.max_pooling2d instead. |
|
W0608 15:17:03.345488 548390839744 deprecation.py:323] From /home/miku/venv36/lib/python3.6/site-packages/tensorflow/python/ops/losses/losses_impl.py:209: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. |
|
Instructions for updating: |
|
Use tf.cast instead. |
|
W0608 15:17:04.021918 548390839744 deprecation.py:323] From /home/miku/venv36/lib/python3.6/site-packages/tensorflow/python/ops/math_ops.py:3066: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. |
|
Instructions for updating: |
|
Use tf.cast instead. |
|
Initializing graph |
|
W0608 15:17:09.478287 548390839744 deprecation.py:323] From /home/miku/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2238: Supervisor.__init__ (from tensorflow.python.training.supervisor) is deprecated and will be removed in a future version. |
|
Instructions for updating: |
|
Please switch to tf.train.MonitoredTrainingSession |
|
2019-06-08 15:17:19.255147: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 0 |
|
2019-06-08 15:17:19.255265: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix: |
|
2019-06-08 15:17:19.255309: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 0 |
|
2019-06-08 15:17:19.255342: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 0: N |
|
2019-06-08 15:17:19.255448: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 1648 MB memory) -> physical GPU (device: 0, name: NVIDIA Tegra X1, pci bus id: 0000:00:00.0, compute capability: 5.3) |
|
I0608 15:17:22.634904 548390839744 session_manager.py:491] Running local_init_op. |
|
I0608 15:17:22.855322 548390839744 session_manager.py:493] Done running local_init_op. |
|
Running warm up |
|
2019-06-08 15:17:28.219080: I tensorflow/stream_executor/dso_loader.cc:153] successfully opened CUDA library libcublas.so.10.0 locally |
|
2019-06-08 15:18:33.621388: W tensorflow/core/common_runtime/bfc_allocator.cc:211] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.17GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. |
|
2019-06-08 15:18:34.520592: W tensorflow/core/common_runtime/bfc_allocator.cc:211] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.17GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. |
|
2019-06-08 15:18:34.811993: W tensorflow/core/common_runtime/bfc_allocator.cc:211] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.10GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. |
|
2019-06-08 15:18:34.865131: W tensorflow/core/common_runtime/bfc_allocator.cc:211] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.26GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. |
|
2019-06-08 15:18:34.988038: W tensorflow/core/common_runtime/bfc_allocator.cc:211] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.29GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. |
|
2019-06-08 15:18:35.386779: W tensorflow/core/common_runtime/bfc_allocator.cc:211] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.29GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. |
|
2019-06-08 15:18:35.493907: W tensorflow/core/common_runtime/bfc_allocator.cc:211] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.25GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. |
|
2019-06-08 15:18:35.705907: W tensorflow/core/common_runtime/bfc_allocator.cc:211] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.10GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. |
|
2019-06-08 15:18:36.003183: W tensorflow/core/common_runtime/bfc_allocator.cc:211] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.26GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. |
|
2019-06-08 15:18:36.109775: W tensorflow/core/common_runtime/bfc_allocator.cc:211] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.25GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. |
|
Done warm up |
|
Step Img/sec total_loss |
|
1 images/sec: 4.4 +/- 0.0 (jitter = 0.0) 8.510 |
|
10 images/sec: 4.4 +/- 0.0 (jitter = 0.0) 7.602 |
|
20 images/sec: 4.4 +/- 0.0 (jitter = 0.0) 8.671 |
|
30 images/sec: 4.4 +/- 0.0 (jitter = 0.0) 8.026 |
|
40 images/sec: 4.4 +/- 0.0 (jitter = 0.0) 7.519 |
|
50 images/sec: 4.4 +/- 0.0 (jitter = 0.0) 7.523 |
|
60 images/sec: 4.4 +/- 0.0 (jitter = 0.0) 8.667 |
|
70 images/sec: 4.4 +/- 0.0 (jitter = 0.0) 8.359 |
|
80 images/sec: 4.4 +/- 0.0 (jitter = 0.0) 8.034 |
|
90 images/sec: 4.4 +/- 0.0 (jitter = 0.0) 8.291 |
|
100 images/sec: 4.4 +/- 0.0 (jitter = 0.0) 7.673 |
|
---------------------------------------------------------------- |
|
total images/sec: 4.35 |
|
---------------------------------------------------------------- |
|
|
|
$ python tf_cnn_benchmarks.py --num_gpus=1 --batch_size=8 --model=inception3 --variable_update=parameter_server |
|
2019-06-08 15:35:36.421819: W tensorflow/core/platform/profile_utils/cpu_utils.cc:98] Failed to find bogomips in /proc/cpuinfo; cannot determine CPU frequency |
|
2019-06-08 15:35:36.424663: I tensorflow/compiler/xla/service/service.cc:161] XLA service 0x5568ada3b0 executing computations on platform Host. Devices: |
|
2019-06-08 15:35:36.425801: I tensorflow/compiler/xla/service/service.cc:168] StreamExecutor device (0): <undefined>, <undefined> |
|
2019-06-08 15:35:36.548136: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:965] ARM64 does not support NUMA - returning NUMA node zero |
|
2019-06-08 15:35:36.548468: I tensorflow/compiler/xla/service/service.cc:161] XLA service 0x55675d7dd0 executing computations on platform CUDA. Devices: |
|
2019-06-08 15:35:36.548526: I tensorflow/compiler/xla/service/service.cc:168] StreamExecutor device (0): NVIDIA Tegra X1, Compute Capability 5.3 |
|
2019-06-08 15:35:36.548864: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1433] Found device 0 with properties: |
|
name: NVIDIA Tegra X1 major: 5 minor: 3 memoryClockRate(GHz): 0.9216 |
|
pciBusID: 0000:00:00.0 |
|
totalMemory: 3.87GiB freeMemory: 2.26GiB |
|
2019-06-08 15:35:36.548931: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 0 |
|
2019-06-08 15:35:40.941625: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix: |
|
2019-06-08 15:35:40.941709: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 0 |
|
2019-06-08 15:35:40.941740: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 0: N |
|
2019-06-08 15:35:40.941938: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 1623 MB memory) -> physical GPU (device: 0, name: NVIDIA Tegra X1, pci bus id: 0000:00:00.0, compute capability: 5.3) |
|
TensorFlow: 1.13 |
|
Model: inception3 |
|
Dataset: imagenet (synthetic) |
|
Mode: training |
|
SingleSess: False |
|
Batch size: 8 global |
|
8 per device |
|
Num batches: 100 |
|
Num epochs: 0.00 |
|
Devices: ['/gpu:0'] |
|
NUMA bind: False |
|
Data format: NCHW |
|
Optimizer: sgd |
|
Variables: parameter_server |
|
========== |
|
Generating training model |
|
W0608 15:35:40.963752 547771041216 deprecation.py:323] From /home/miku/venv36/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version. |
|
Instructions for updating: |
|
Colocations handled automatically by placer. |
|
W0608 15:35:41.045766 547771041216 deprecation.py:323] From /home/miku/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:129: conv2d (from tensorflow.python.layers.convolutional) is deprecated and will be removed in a future version. |
|
Instructions for updating: |
|
Use keras.layers.conv2d instead. |
|
W0608 15:35:41.635923 547771041216 deprecation.py:323] From /home/miku/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:261: max_pooling2d (from tensorflow.python.layers.pooling) is deprecated and will be removed in a future version. |
|
Instructions for updating: |
|
Use keras.layers.max_pooling2d instead. |
|
W0608 15:35:43.426353 547771041216 deprecation.py:323] From /home/miku/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:261: average_pooling2d (from tensorflow.python.layers.pooling) is deprecated and will be removed in a future version. |
|
Instructions for updating: |
|
Use keras.layers.average_pooling2d instead. |
|
W0608 15:36:00.615895 547771041216 deprecation.py:323] From /home/miku/venv36/lib/python3.6/site-packages/tensorflow/python/ops/losses/losses_impl.py:209: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. |
|
Instructions for updating: |
|
Use tf.cast instead. |
|
Initializing graph |
|
W0608 15:36:09.778275 547771041216 deprecation.py:323] From /home/miku/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2238: Supervisor.__init__ (from tensorflow.python.training.supervisor) is deprecated and will be removed in a future version. |
|
Instructions for updating: |
|
Please switch to tf.train.MonitoredTrainingSession |
|
2019-06-08 15:36:25.259073: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 0 |
|
2019-06-08 15:36:25.259192: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix: |
|
2019-06-08 15:36:25.259235: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 0 |
|
2019-06-08 15:36:25.259268: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 0: N |
|
2019-06-08 15:36:25.259382: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 1623 MB memory) -> physical GPU (device: 0, name: NVIDIA Tegra X1, pci bus id: 0000:00:00.0, compute capability: 5.3) |
|
I0608 15:36:29.572513 547771041216 session_manager.py:491] Running local_init_op. |
|
I0608 15:36:29.917439 547771041216 session_manager.py:493] Done running local_init_op. |
|
Running warm up |
|
2019-06-08 15:36:38.498294: I tensorflow/stream_executor/dso_loader.cc:153] successfully opened CUDA library libcublas.so.10.0 locally |
|
2019-06-08 15:37:05.486777: W tensorflow/core/common_runtime/bfc_allocator.cc:211] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.98GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. |
|
2019-06-08 15:37:08.217127: W tensorflow/core/common_runtime/bfc_allocator.cc:211] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.25GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. |
|
2019-06-08 15:37:08.642988: W tensorflow/core/common_runtime/bfc_allocator.cc:211] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.05GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. |
|
2019-06-08 15:37:09.917505: W tensorflow/core/common_runtime/bfc_allocator.cc:211] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.25GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. |
|
2019-06-08 15:37:09.980112: W tensorflow/core/common_runtime/bfc_allocator.cc:211] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.08GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. |
|
2019-06-08 15:37:10.036479: W tensorflow/core/common_runtime/bfc_allocator.cc:211] Allocator (GPU_0_bfc) ran out of memory trying to allocate 924.06MiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. |
|
2019-06-08 15:37:11.261485: W tensorflow/core/common_runtime/bfc_allocator.cc:211] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.01GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. |
|
2019-06-08 15:37:11.357251: W tensorflow/core/common_runtime/bfc_allocator.cc:211] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.73GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. |
|
2019-06-08 15:37:11.439711: W tensorflow/core/common_runtime/bfc_allocator.cc:211] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.44GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. |
|
2019-06-08 15:37:11.504237: W tensorflow/core/common_runtime/bfc_allocator.cc:211] Allocator (GPU_0_bfc) ran out of memory trying to allocate 901.50MiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. |
|
Done warm up |
|
Step Img/sec total_loss |
|
1 images/sec: 2.8 +/- 0.0 (jitter = 0.0) 7.421 |
|
10 images/sec: 2.8 +/- 0.0 (jitter = 0.0) 7.196 |
|
20 images/sec: 2.8 +/- 0.0 (jitter = 0.0) 7.483 |
|
30 images/sec: 2.8 +/- 0.0 (jitter = 0.0) 7.216 |
|
40 images/sec: 2.8 +/- 0.0 (jitter = 0.0) 7.459 |
|
50 images/sec: 2.8 +/- 0.0 (jitter = 0.0) 7.309 |
|
60 images/sec: 2.8 +/- 0.0 (jitter = 0.0) 7.394 |
|
70 images/sec: 2.8 +/- 0.0 (jitter = 0.0) 7.625 |
|
80 images/sec: 2.8 +/- 0.0 (jitter = 0.0) 7.433 |
|
90 images/sec: 2.8 +/- 0.0 (jitter = 0.0) 7.350 |
|
100 images/sec: 2.8 +/- 0.0 (jitter = 0.0) 7.536 |
|
---------------------------------------------------------------- |
|
total images/sec: 2.83 |
|
---------------------------------------------------------------- |
|
$ python tf_cnn_benchmarks.py --num_gpus=1 --batch_size=1 --model=vgg16 --variable_update=parameter_server |
|
2019-06-08 15:59:49.367576: W tensorflow/core/platform/profile_utils/cpu_utils.cc:98] Failed to find bogomips in /proc/cpuinfo; cannot determine CPU frequency |
|
2019-06-08 15:59:49.368167: I tensorflow/compiler/xla/service/service.cc:161] XLA service 0x55afada390 executing computations on platform Host. Devices: |
|
2019-06-08 15:59:49.368227: I tensorflow/compiler/xla/service/service.cc:168] StreamExecutor device (0): <undefined>, <undefined> |
|
2019-06-08 15:59:49.495954: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:965] ARM64 does not support NUMA - returning NUMA node zero |
|
2019-06-08 15:59:49.496232: I tensorflow/compiler/xla/service/service.cc:161] XLA service 0x55ae5d7db0 executing computations on platform CUDA. Devices: |
|
2019-06-08 15:59:49.496316: I tensorflow/compiler/xla/service/service.cc:168] StreamExecutor device (0): NVIDIA Tegra X1, Compute Capability 5.3 |
|
2019-06-08 15:59:49.496770: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1433] Found device 0 with properties: |
|
name: NVIDIA Tegra X1 major: 5 minor: 3 memoryClockRate(GHz): 0.9216 |
|
pciBusID: 0000:00:00.0 |
|
totalMemory: 3.87GiB freeMemory: 2.24GiB |
|
2019-06-08 15:59:49.496867: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 0 |
|
2019-06-08 15:59:53.468034: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix: |
|
2019-06-08 15:59:53.468107: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 0 |
|
2019-06-08 15:59:53.468136: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 0: N |
|
2019-06-08 15:59:53.468348: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 1615 MB memory) -> physical GPU (device: 0, name: NVIDIA Tegra X1, pci bus id: 0000:00:00.0, compute capability: 5.3) |
|
TensorFlow: 1.13 |
|
Model: vgg16 |
|
Dataset: imagenet (synthetic) |
|
Mode: training |
|
SingleSess: False |
|
Batch size: 1 global |
|
1 per device |
|
Num batches: 100 |
|
Num epochs: 0.00 |
|
Devices: ['/gpu:0'] |
|
NUMA bind: False |
|
Data format: NCHW |
|
Optimizer: sgd |
|
Variables: parameter_server |
|
========== |
|
Generating training model |
|
W0608 15:59:53.489730 548159309248 deprecation.py:323] From /home/miku/venv36/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version. |
|
Instructions for updating: |
|
Colocations handled automatically by placer. |
|
W0608 15:59:53.571482 548159309248 deprecation.py:323] From /home/miku/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:129: conv2d (from tensorflow.python.layers.convolutional) is deprecated and will be removed in a future version. |
|
Instructions for updating: |
|
Use keras.layers.conv2d instead. |
|
W0608 15:59:53.762966 548159309248 deprecation.py:323] From /home/miku/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:261: max_pooling2d (from tensorflow.python.layers.pooling) is deprecated and will be removed in a future version. |
|
Instructions for updating: |
|
Use keras.layers.max_pooling2d instead. |
|
W0608 15:59:55.129193 548159309248 deprecation.py:323] From /home/miku/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:403: dropout (from tensorflow.python.layers.core) is deprecated and will be removed in a future version. |
|
Instructions for updating: |
|
Use keras.layers.dropout instead. |
|
W0608 15:59:55.132718 548159309248 deprecation.py:506] From /home/miku/venv36/lib/python3.6/site-packages/tensorflow/python/keras/layers/core.py:143: calling dropout (from tensorflow.python.ops.nn_ops) with keep_prob is deprecated and will be removed in a future version. |
|
Instructions for updating: |
|
Please use `rate` instead of `keep_prob`. Rate should be set to `rate = 1 - keep_prob`. |
|
W0608 15:59:55.390173 548159309248 deprecation.py:323] From /home/miku/venv36/lib/python3.6/site-packages/tensorflow/python/ops/losses/losses_impl.py:209: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. |
|
Instructions for updating: |
|
Use tf.cast instead. |
|
Initializing graph |
|
W0608 15:59:56.978619 548159309248 deprecation.py:323] From /home/miku/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2238: Supervisor.__init__ (from tensorflow.python.training.supervisor) is deprecated and will be removed in a future version. |
|
Instructions for updating: |
|
Please switch to tf.train.MonitoredTrainingSession |
|
2019-06-08 15:59:58.865290: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 0 |
|
2019-06-08 15:59:58.865408: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix: |
|
2019-06-08 15:59:58.865455: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 0 |
|
2019-06-08 15:59:58.865489: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 0: N |
|
2019-06-08 15:59:58.865689: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 1615 MB memory) -> physical GPU (device: 0, name: NVIDIA Tegra X1, pci bus id: 0000:00:00.0, compute capability: 5.3) |
|
I0608 16:00:00.514552 548159309248 session_manager.py:491] Running local_init_op. |
|
I0608 16:00:01.024017 548159309248 session_manager.py:493] Done running local_init_op. |
|
Running warm up |
|
2019-06-08 16:00:02.062699: I tensorflow/stream_executor/dso_loader.cc:153] successfully opened CUDA library libcublas.so.10.0 locally |
|
2019-06-08 16:00:27.234406: W tensorflow/core/common_runtime/bfc_allocator.cc:211] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.05GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. |
|
2019-06-08 16:00:27.662109: W tensorflow/core/common_runtime/bfc_allocator.cc:211] Allocator (GPU_0_bfc) ran out of memory trying to allocate 882.56MiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. |
|
2019-06-08 16:00:27.728000: W tensorflow/core/common_runtime/bfc_allocator.cc:211] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.02GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. |
|
2019-06-08 16:00:28.034288: W tensorflow/core/common_runtime/bfc_allocator.cc:211] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.05GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. |
|
2019-06-08 16:00:28.435415: W tensorflow/core/common_runtime/bfc_allocator.cc:211] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.04GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. |
|
2019-06-08 16:00:28.798298: W tensorflow/core/common_runtime/bfc_allocator.cc:211] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.07GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. |
|
2019-06-08 16:00:29.386786: W tensorflow/core/common_runtime/bfc_allocator.cc:211] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.07GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. |
|
2019-06-08 16:00:29.396037: W tensorflow/core/common_runtime/bfc_allocator.cc:211] Allocator (GPU_0_bfc) ran out of memory trying to allocate 547.21MiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. |
|
2019-06-08 16:00:29.552577: W tensorflow/core/common_runtime/bfc_allocator.cc:211] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.14GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. |
|
2019-06-08 16:00:29.570247: W tensorflow/core/common_runtime/bfc_allocator.cc:211] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.07GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. |
|
Done warm up |
|
Step Img/sec total_loss |
|
1 images/sec: 1.0 +/- 0.0 (jitter = 0.0) nan |
|
10 images/sec: 1.0 +/- 0.0 (jitter = 0.0) nan |
|
20 images/sec: 1.0 +/- 0.0 (jitter = 0.0) nan |
|
30 images/sec: 1.0 +/- 0.0 (jitter = 0.0) nan |
|
40 images/sec: 1.0 +/- 0.0 (jitter = 0.0) nan |
|
50 images/sec: 1.0 +/- 0.0 (jitter = 0.0) nan |
|
60 images/sec: 1.0 +/- 0.0 (jitter = 0.0) nan |
|
70 images/sec: 1.0 +/- 0.0 (jitter = 0.0) nan |
|
80 images/sec: 1.0 +/- 0.0 (jitter = 0.0) nan |
|
90 images/sec: 1.0 +/- 0.0 (jitter = 0.0) nan |
|
100 images/sec: 1.0 +/- 0.0 (jitter = 0.0) nan |
|
---------------------------------------------------------------- |
|
total images/sec: 1.01 |
|
---------------------------------------------------------------- |