DGX Station benchmark

result.txt

Welcome to NVIDIA DGX Station Version 4.0.7 (GNU/Linux 4.15.0-47-generic x86_64)
user@user-DGX-Station:~$ lscpu
Architecture:        x86_64
CPU op-mode(s):      32-bit, 64-bit
Byte Order:          Little Endian
CPU(s):              40
On-line CPU(s) list: 0-39
Thread(s) per core:  2
Core(s) per socket:  20
Socket(s):           1
NUMA node(s):        1
Vendor ID:           GenuineIntel
CPU family:          6
Model:               79
Model name:          Intel(R) Xeon(R) CPU E5-2698 v4 @ 2.20GHz
Stepping:            1
CPU MHz:             1203.348
CPU max MHz:         3600.0000
CPU min MHz:         1200.0000
BogoMIPS:            4397.56
Virtualization:      VT-x
L1d cache:           32K
L1i cache:           32K
L2 cache:            256K
L3 cache:            51200K
NUMA node0 CPU(s):   0-39
Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cdp_l3 invpcid_single pti intel_ppin ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm rdt_a rdseed adx smap intel_pt xsaveopt cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts flush_l1d
user@user-DGX-Station:~$ free -h
              total        used        free      shared  buff/cache   available
Mem:           251G        1.1G        249G        2.4M        1.6G        248G
Swap:            0B          0B          0B
user@user-DGX-Station:~$ nvidia-smi
Tue Sep 10 21:48:37 2019
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 410.129      Driver Version: 410.129      CUDA Version: 10.0     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla V100-DGXS...  On   | 00000000:07:00.0 Off |                    0 |
| N/A   45C    P0    40W / 300W |     27MiB / 32478MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   1  Tesla V100-DGXS...  On   | 00000000:08:00.0 Off |                    0 |
| N/A   46C    P0    40W / 300W |      0MiB / 32478MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   2  Tesla V100-DGXS...  On   | 00000000:0E:00.0 Off |                    0 |
| N/A   46C    P0    40W / 300W |      0MiB / 32478MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   3  Tesla V100-DGXS...  On   | 00000000:0F:00.0 Off |                    0 |
| N/A   48C    P0    40W / 300W |      0MiB / 32478MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0      2774      G   /usr/lib/xorg/Xorg                             9MiB |
|    0      2800      G   /usr/bin/gnome-shell                          15MiB |
+-----------------------------------------------------------------------------+
user@user-DGX-Station:~$ git clone https://github.com/Mysticial/Flops.git
Cloning into 'Flops'...
remote: Enumerating objects: 726, done.
remote: Total 726 (delta 0), reused 0 (delta 0), pack-reused 726
Receiving objects: 100% (726/726), 8.98 MiB | 4.26 MiB/s, done.
Resolving deltas: 100% (464/464), done.
user@user-DGX-Station:~$ cd Flops/version3/binaries-linux/
user@user-DGX-Station:~/Flops/version3/binaries-linux$ chmod +x 2006-Core2
user@user-DGX-Station:~/Flops/version3/binaries-linux$ ./2006-Core2
Running Core 2 tuned binary with 1 thread...

Single-Precision - 128-bit SSE - Add/Sub
    GFlops = 10.72
    Result = 1.35699e+06

Double-Precision - 128-bit SSE2 - Add/Sub
    GFlops = 5.36
    Result = 684236

Single-Precision - 128-bit SSE - Multiply
    GFlops = 21.456
    Result = 2.72048e+06

Double-Precision - 128-bit SSE2 - Multiply
    GFlops = 10.704
    Result = 1.37177e+06

Single-Precision - 128-bit SSE - Multiply + Add
    GFlops = 21.456
    Result = 2.28675e+06

Double-Precision - 128-bit SSE2 - Multiply + Add
    GFlops = 10.728
    Result = 1.14783e+06


Running Core 2 tuned binary with 40 thread(s)...

Single-Precision - 128-bit SSE - Add/Sub
    GFlops = 214.752
    Result = 2.74514e+07

Double-Precision - 128-bit SSE2 - Add/Sub
    GFlops = 108.096
    Result = 1.37612e+07

Single-Precision - 128-bit SSE - Multiply
    GFlops = 432.144
    Result = 5.50881e+07

Double-Precision - 128-bit SSE2 - Multiply
    GFlops = 215.904
    Result = 2.75107e+07

Single-Precision - 128-bit SSE - Multiply + Add
    GFlops = 432.24
    Result = 4.58174e+07

Double-Precision - 128-bit SSE2 - Multiply + Add
    GFlops = 215.976
    Result = 2.29539e+07


user@user-DGX-Station:~/Flops/version3/binaries-linux$ chmod +x 2013-Haswell
user@user-DGX-Station:~/Flops/version3/binaries-linux$ ./2013-Haswell
Running Haswell tuned binary with 1 thread...

Single-Precision - 128-bit AVX - Add/Sub
    GFlops = 10.72
    Result = 1.39171e+06

Double-Precision - 128-bit AVX - Add/Sub
    GFlops = 5.36
    Result = 677634

Single-Precision - 128-bit AVX - Multiply
    GFlops = 21.456
    Result = 2.72694e+06

Double-Precision - 128-bit AVX - Multiply
    GFlops = 10.728
    Result = 1.37212e+06

Single-Precision - 128-bit AVX - Multiply + Add
    GFlops = 21.456
    Result = 2.30148e+06

Double-Precision - 128-bit AVX - Multiply + Add
    GFlops = 10.728
    Result = 1.14076e+06

Single-Precision - 128-bit FMA3 - Fused Multiply Add
    GFlops = 41.376
    Result = 2.62872e+06

Double-Precision - 128-bit FMA3 - Fused Multiply Add
    GFlops = 20.688
    Result = 1.31907e+06

Single-Precision - 256-bit AVX - Add/Sub
    GFlops = 20.672
    Result = 2.62571e+06

Double-Precision - 256-bit AVX - Add/Sub
    GFlops = 10.336
    Result = 1.31198e+06

Single-Precision - 256-bit AVX - Multiply
    GFlops = 41.376
    Result = 5.27805e+06

Double-Precision - 256-bit AVX - Multiply
    GFlops = 20.64
    Result = 2.62802e+06

Single-Precision - 256-bit AVX - Multiply + Add
    GFlops = 41.376
    Result = 4.39652e+06

Double-Precision - 256-bit AVX - Multiply + Add
    GFlops = 20.736
    Result = 2.20224e+06

Single-Precision - 256-bit FMA3 - Fused Multiply Add
    GFlops = 82.56
    Result = 5.24702e+06

Double-Precision - 256-bit FMA3 - Fused Multiply Add
    GFlops = 41.376
    Result = 2.63064e+06


Running Haswell tuned binary with 40 thread(s)...

Single-Precision - 128-bit AVX - Add/Sub
    GFlops = 214.912
    Result = 2.73979e+07

Double-Precision - 128-bit AVX - Add/Sub
    GFlops = 108.176
    Result = 1.37815e+07

Single-Precision - 128-bit AVX - Multiply
    GFlops = 432.048
    Result = 5.50837e+07

Double-Precision - 128-bit AVX - Multiply
    GFlops = 215.976
    Result = 2.75056e+07

Single-Precision - 128-bit AVX - Multiply + Add
    GFlops = 432.288
    Result = 4.58675e+07

Double-Precision - 128-bit AVX - Multiply + Add
    GFlops = 216.24
    Result = 2.29816e+07

Single-Precision - 128-bit FMA3 - Fused Multiply Add
    GFlops = 832.608
    Result = 5.3022e+07

Double-Precision - 128-bit FMA3 - Fused Multiply Add
    GFlops = 416.304
    Result = 2.65274e+07

Single-Precision - 256-bit AVX - Add/Sub
    GFlops = 416.832
    Result = 5.30652e+07

Double-Precision - 256-bit AVX - Add/Sub
    GFlops = 208.32
    Result = 2.66027e+07

Single-Precision - 256-bit AVX - Multiply
    GFlops = 832.608
    Result = 1.06004e+08

Double-Precision - 256-bit AVX - Multiply
    GFlops = 416.448
    Result = 5.30501e+07

Single-Precision - 256-bit AVX - Multiply + Add
    GFlops = 832.608
    Result = 8.82891e+07

Double-Precision - 256-bit AVX - Multiply + Add
    GFlops = 416.208
    Result = 4.42095e+07

Single-Precision - 256-bit FMA3 - Fused Multiply Add
    GFlops = 1664.64
    Result = 1.06053e+08

Double-Precision - 256-bit FMA3 - Fused Multiply Add
    GFlops = 831.84
    Result = 5.29934e+07


user@user-DGX-Station:~/Flops/version3/binaries-linux$ cd ~
user@user-DGX-Station:~$ sudo docker pull nvcr.io/nvidia/tensorflow:19.08-py3
19.08-py3: Pulling from nvidia/tensorflow
Digest: sha256:64e296668d398a106f64bd840772ffb63372148b8c1170b152e7e577013661c9
Status: Image is up to date for nvcr.io/nvidia/tensorflow:19.08-py3
user@user-DGX-Station:~$ sudo nvidia-docker run -it --rm nvcr.io/nvidia/tensorflow:19.08-py3

================
== TensorFlow ==
================

NVIDIA Release 19.08 (build 7791926)
TensorFlow Version 1.14.0

Container image Copyright (c) 2019, NVIDIA CORPORATION.  All rights reserved.
Copyright 2017-2019 The TensorFlow Authors.  All rights reserved.

Various files include modifications (c) NVIDIA CORPORATION.  All rights reserved.
NVIDIA modifications are covered by the license terms that apply to the underlying project or file.

NOTE: Legacy NVIDIA Driver detected.  Compatibility mode ENABLED.

NOTE: MOFED driver for multi-node communication was not detected.
      Multi-node communication performance may be reduced.

root@692ced701c1d:/workspace# git clone --no-checkout https://github.com/IntelPython/ibench.git
Cloning into 'ibench'...
remote: Enumerating objects: 21, done.
remote: Counting objects: 100% (21/21), done.
remote: Compressing objects: 100% (21/21), done.
remote: Total 467 (delta 8), reused 2 (delta 0), pack-reused 446
Receiving objects: 100% (467/467), 81.04 KiB | 365.00 KiB/s, done.
Resolving deltas: 100% (253/253), done.
root@692ced701c1d:/workspace# cd ibench
root@692ced701c1d:/workspace/ibench# git checkout d2a81d04352427437e6e383654cfbd36e99c5ae9
Note: checking out 'd2a81d04352427437e6e383654cfbd36e99c5ae9'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:

  git checkout -b <new-branch-name>

HEAD is now at d2a81d0 Add comments to clarify the steps
root@692ced701c1d:/workspace/ibench# python -m ibench run -b all --size small --runs 3 --file result.json
Cholesky:
Cholesky:   N = 10000
Cholesky:   elapsed 0.990914 gflops 336.389816
Cholesky:   elapsed 0.967183 gflops 344.643459
Cholesky:   elapsed 0.979904 gflops 340.169316
Cholesky:   gflops 340.169316
Det:
Det:   N = 15000
Det:   elapsed 6.110545 gflops 368.215933
Det:   elapsed 5.973589 gflops 376.657959
Det:   elapsed 5.944663 gflops 378.490784
Det:   gflops 376.657959
Dot:
Dot:   N = 5000
Dot:   elapsed 0.573518 gflops 435.906081
Dot:   elapsed 0.573810 gflops 435.684571
Dot:   elapsed 0.571029 gflops 437.805993
Dot:   gflops 435.906081
Fft:
Fft:   N = 520000
Fft:   elapsed 18.262345 gflops 2.703333
Fft:   elapsed 18.233072 gflops 2.707673
Fft:   elapsed 18.234067 gflops 2.707525
Fft:   gflops 2.707525
Inv:
Inv:   N = 10000
Inv:   elapsed 7.173745 gflops 278.794422
Inv:   elapsed 7.098740 gflops 281.740136
Inv:   elapsed 7.067508 gflops 282.985177
Inv:   gflops 281.740136
Lu:
Lu:   N = 20000
Lu:   elapsed 17.991874 gflops 296.430114
Lu:   elapsed 17.741165 gflops 300.619109
Lu:   elapsed 17.668941 gflops 301.847929
Lu:   gflops 300.619109
Qr:
Qr:   N = 5000
Qr:   elapsed 2.123459 gflops 78.488287
Qr:   elapsed 2.098615 gflops 79.417451
Qr:   elapsed 2.090776 gflops 79.715220
Qr:   gflops 79.417451
Svd:
Svd:   N = 5000
Svd:   elapsed 19.762689 gflops 8.433400
Svd:   elapsed 19.579588 gflops 8.512266
Svd:   elapsed 19.605364 gflops 8.501075
Svd:   gflops 8.501075
root@692ced701c1d:/workspace/ibench# cd /workspace
root@692ced701c1d:/workspace# git clone https://github.com/axboe/fio.git -b fio-3.15
Cloning into 'fio'...
remote: Enumerating objects: 11, done.
remote: Counting objects: 100% (11/11), done.
remote: Compressing objects: 100% (8/8), done.
remote: Total 28931 (delta 4), reused 8 (delta 3), pack-reused 28920
Receiving objects: 100% (28931/28931), 18.45 MiB | 811.00 KiB/s, done.
Resolving deltas: 100% (19259/19259), done.
Note: checking out '01bf5128d0581e267383f280c6a1dcd26517240f'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:

  git checkout -b <new-branch-name>

root@692ced701c1d:/workspace# cd fio/
root@692ced701c1d:/workspace/fio# ./configure
Operating system              Linux
CPU                           x86_64
Big endian                    no
Compiler                      gcc
Cross compile                 no

Static build                  no
Wordsize                      64
zlib                          yes
Linux AIO support             no
POSIX AIO support             yes
POSIX AIO support needs -lrt  yes
POSIX AIO fsync               yes
POSIX pshared support         yes
Solaris AIO support           no
__sync_fetch_and_add          yes
__sync_synchronize            yes
__sync_val_compare_and_swap   yes
libverbs                      no
rdmacm                        no
asprintf()                    yes
vasprintf()                   yes
Linux fallocate               yes
POSIX fadvise                 yes
POSIX fallocate               yes
sched_setaffinity(3 arg)      yes
sched_setaffinity(2 arg)      no
clock_gettime                 yes
CLOCK_MONOTONIC               yes
CLOCK_MONOTONIC_RAW           yes
CLOCK_MONOTONIC_PRECISE       no
clockid_t                     yes
gettimeofday                  yes
fdatasync                     yes
sync_file_range               yes
EXT4 move extent              yes
Linux splice(2)               yes
GUASI                         no
libnuma                       yes
libnuma v2                    yes
strsep                        yes
strcasestr                    yes
strlcat                       no
getopt_long_only()            yes
inet_aton                     yes
socklen_t                     yes
__thread                      yes
RUSAGE_THREAD                 yes
SCHED_IDLE                    yes
TCP_NODELAY                   yes
Net engine window_size        yes
TCP_MAXSEG                    yes
RLIMIT_MEMLOCK                yes
pwritev/preadv                yes
pwritev2/preadv2              yes
IPv6 helpers                  yes
http engine                   no
Rados engine                  no
Rados Block Device engine     no
setvbuf                       yes
Gluster API engine            no
s390_z196_facilities          no
HDFS engine                   no
MTD                           yes
libpmem                       no
libpmemblk                    no
PMDK pmemblk engine           no
PMDK dev-dax engine           no
PMDK libpmem engine           no
DDN's Infinite Memory Engine  no
iscsi engine                  no
lex/yacc for arithmetic       no
getmntent                     yes
getmntinfo                    no
Static Assert                 yes
bool                          yes
strndup                       yes
Valgrind headers              no
Zoned block device support    yes
march_armv8_a_crc_crypto      no
cuda                          no
mkdir(a, b)                   yes
Build march=native            yes
CUnit                         no
__kernel_rwf_t                yes
-Wimplicit-fallthrough        yes
MADV_HUGEPAGE                 yes
gettid                        no
TCMalloc support              no
root@692ced701c1d:/workspace/fio# make -j $(nproc)
FIO_VERSION = fio-3.15
    CC crc/crc32.o
    CC crc/crc16.o
    CC crc/crc32c-arm64.o
    CC crc/crc32c-intel.o
    CC crc/crc32c.o
    CC crc/crc7.o
    CC crc/crc64.o
    CC crc/fnv.o
    CC crc/md5.o
    CC crc/sha1.o
    CC crc/murmur3.o
    CC crc/sha256.o
    CC crc/sha512.o
    CC crc/sha3.o
    CC crc/test.o
    CC crc/xxhash.o
    CC lib/axmap.o
    CC lib/bloom.o
    CC lib/flist_sort.o
    CC lib/getrusage.o
    CC lib/gauss.o
    CC lib/lfsr.o
    CC lib/ieee754.o
    CC lib/memcpy.o
    CC lib/memalign.o
    CC lib/hweight.o
    CC lib/mountcheck.o
    CC lib/num2str.o
    CC lib/prio_tree.o
    CC lib/output_buffer.o
    CC lib/pattern.o
    CC lib/rbtree.o
    CC lib/rand.o
    CC lib/strntol.o
    CC gettime.o
    CC init.o
    CC ioengines.o
    CC lib/zipf.o
    CC stat.o
    CC log.o
    CC time.o
    CC filesetup.o
    CC eta.o
    CC verify.o
    CC memory.o
    CC io_u.o
    CC parse.o
    CC pshared.o
    CC fio_sem.o
    CC rwlock.o
    CC options.o
    CC smalloc.o
    CC filehash.o
    CC profile.o
    CC debug.o
    CC engines/cpu.o
    CC engines/mmap.o
    CC engines/sync.o
    CC engines/null.o
    CC engines/net.o
    CC engines/ftruncate.o
    CC engines/filecreate.o
    CC server.o
    CC client.o
    CC iolog.o
    CC backend.o
    CC libfio.o
    CC flow.o
    CC gettime-thread.o
    CC cconv.o
    CC helpers.o
    CC json.o
    CC td_error.o
    CC idletime.o
    CC profiles/tiobench.o
    CC profiles/act.o
    CC io_u_queue.o
    CC filelock.o
    CC workqueue.o
    CC rate-submit.o
    CC optgroup.o
    CC helper_thread.o
    CC steadystate.o
    CC zone-dist.o
    CC engines/posixaio.o
    CC engines/falloc.o
    CC engines/e4defrag.o
    CC engines/splice.o
    CC oslib/asprintf.o
    CC oslib/strlcat.o
    CC engines/mtd.o
    CC oslib/libmtd.o
    CC oslib/libmtd_legacy.o
    CC zbd.o
    CC diskutil.o
    CC fifo.o
    CC blktrace.o
    CC cgroup.o
    CC trim.o
    CC engines/sg.o
    CC oslib/linux-dev-lookup.o
    CC engines/io_uring.o
    CC fio.o
    CC t/log.o
    CC t/genzipf.o
    CC oslib/strcasestr.o
    CC oslib/strndup.o
    CC t/btrace2fio.o
    CC t/dedupe.o
    CC t/debug.o
    CC t/arch.o
    CC t/verify-state.o
    CC t/stest.o
    CC t/ieee754.o
    CC t/axmap.o
    CC t/gen-rand.o
    CC t/lfsr-test.o
  LINK t/ieee754
  LINK t/stest
  LINK t/lfsr-test
  LINK t/fio-verify-state
  LINK t/gen-rand
  LINK t/axmap
  LINK t/fio-genzipf
  LINK t/fio-dedupe
  LINK t/fio-btrace2fio
  LINK fio
root@692ced701c1d:/workspace/fio# ./fio --loops=5 --size=1g --runtime=10 --stonewall --direct=1 --group_reporting \
>     --name=SeqQ32T1read --bs=128k --iodepth=32 --rw=read \
>     --name=SeqQ32T1write --bs=128k --iodepth=32 --rw=write \
>     --name=4kQ8T8read --bs=4k --iodepth=8 --numjobs=8 --rw=randread \
>     --name=4kQ8T8write --bs=4k --iodepth=8 --numjobs=8 --rw=randwrite \
>     --name=4kQ32T1read --bs=4k --iodepth=32 --rw=randread \
>     --name=4kQ32T1write --bs=4k --iodepth=32 --rw=randwrite \
>     --name=4kQ1T1read --bs=4k --iodepth=1 --rw=randread \
>     --name=4kQ1T1write --bs=4k --iodepth=1 --rw=randwrite
SeqQ32T1read: (g=0): rw=read, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=psync, iodepth=32
SeqQ32T1write: (g=1): rw=write, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=psync, iodepth=32
4kQ8T8read: (g=2): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=8
...
4kQ8T8write: (g=3): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=8
...
4kQ32T1read: (g=4): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=32
4kQ32T1write: (g=5): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=32
4kQ1T1read: (g=6): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
4kQ1T1write: (g=7): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
fio-3.15
Starting 22 processes
SeqQ32T1read: Laying out IO file (1 file / 1024MiB)
SeqQ32T1write: Laying out IO file (1 file / 1024MiB)
4kQ8T8read: Laying out IO file (1 file / 1024MiB)
4kQ8T8read: Laying out IO file (1 file / 1024MiB)
4kQ8T8read: Laying out IO file (1 file / 1024MiB)
4kQ8T8read: Laying out IO file (1 file / 1024MiB)
4kQ8T8read: Laying out IO file (1 file / 1024MiB)
4kQ8T8read: Laying out IO file (1 file / 1024MiB)
4kQ8T8read: Laying out IO file (1 file / 1024MiB)
4kQ8T8read: Laying out IO file (1 file / 1024MiB)
4kQ8T8write: Laying out IO file (1 file / 1024MiB)
4kQ8T8write: Laying out IO file (1 file / 1024MiB)
4kQ8T8write: Laying out IO file (1 file / 1024MiB)
4kQ8T8write: Laying out IO file (1 file / 1024MiB)
4kQ8T8write: Laying out IO file (1 file / 1024MiB)
4kQ8T8write: Laying out IO file (1 file / 1024MiB)
4kQ8T8write: Laying out IO file (1 file / 1024MiB)
4kQ8T8write: Laying out IO file (1 file / 1024MiB)
4kQ32T1read: Laying out IO file (1 file / 1024MiB)
4kQ32T1write: Laying out IO file (1 file / 1024MiB)
4kQ1T1read: Laying out IO file (1 file / 1024MiB)
4kQ1T1write: Laying out IO file (1 file / 1024MiB)
Jobs: 1 (f=1): [_(21),w(1)][79.2%][w=256MiB/s][w=65.5k IOPS][eta 00m:22s]
SeqQ32T1read: (groupid=0, jobs=1): err= 0: pid=3012: Tue Sep 10 14:02:50 2019
  read: IOPS=3901, BW=488MiB/s (511MB/s)(4877MiB/10001msec)
    clat (usec): min=14, max=882, avg=248.18, stdev=234.46
     lat (usec): min=14, max=882, avg=248.25, stdev=234.43
    clat percentiles (usec):
     |  1.00th=[   15],  5.00th=[   15], 10.00th=[   15], 20.00th=[   15],
     | 30.00th=[   15], 40.00th=[   15], 50.00th=[   92], 60.00th=[  474],
     | 70.00th=[  478], 80.00th=[  482], 90.00th=[  482], 95.00th=[  486],
     | 99.00th=[  586], 99.50th=[  611], 99.90th=[  742], 99.95th=[  750],
     | 99.99th=[  791]
   bw (  KiB/s): min=401284, max=507795, per=91.62%, avg=457547.74, stdev=30392.39, samples=19
   iops        : min= 3135, max= 3967, avg=3574.00, stdev=237.45, samples=19
  lat (usec)   : 20=49.95%, 50=0.04%, 100=0.02%, 250=0.01%, 500=48.49%
  lat (usec)   : 750=1.44%, 1000=0.06%
  cpu          : usr=0.58%, sys=15.05%, ctx=19533, majf=0, minf=42
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=39019,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
SeqQ32T1write: (groupid=1, jobs=1): err= 0: pid=3013: Tue Sep 10 14:02:50 2019
  write: IOPS=2071, BW=259MiB/s (272MB/s)(2815MiB/10872msec); 0 zone resets
    clat (usec): min=68, max=1369.1k, avg=359.39, stdev=15630.43
     lat (usec): min=69, max=1369.1k, avg=360.84, stdev=15630.43
    clat percentiles (usec):
     |  1.00th=[    72],  5.00th=[    73], 10.00th=[    73], 20.00th=[    74],
     | 30.00th=[    75], 40.00th=[    84], 50.00th=[    87], 60.00th=[    94],
     | 70.00th=[    98], 80.00th=[   113], 90.00th=[   115], 95.00th=[   127],
     | 99.00th=[   167], 99.50th=[   194], 99.90th=[   229], 99.95th=[   725],
     | 99.99th=[943719]
   bw (  KiB/s): min=  655, max=894252, per=100.00%, avg=409956.00, stdev=298909.51, samples=12
   iops        : min=    5, max= 6986, avg=3202.33, stdev=2335.14, samples=12
  lat (usec)   : 100=73.54%, 250=26.40%, 500=0.01%, 750=0.01%
  lat (msec)   : 2=0.01%, 500=0.01%, 1000=0.02%, 2000=0.01%
  cpu          : usr=0.76%, sys=27.38%, ctx=192, majf=0, minf=12
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,22523,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
4kQ8T8read: (groupid=2, jobs=8): err= 0: pid=3014: Tue Sep 10 14:02:50 2019
  read: IOPS=53.0k, BW=211MiB/s (221MB/s)(2109MiB/10001msec)
    clat (usec): min=99, max=3850, avg=146.54, stdev=23.08
     lat (usec): min=99, max=3850, avg=146.72, stdev=23.09
    clat percentiles (usec):
     |  1.00th=[  117],  5.00th=[  123], 10.00th=[  127], 20.00th=[  131],
     | 30.00th=[  135], 40.00th=[  139], 50.00th=[  143], 60.00th=[  147],
     | 70.00th=[  153], 80.00th=[  159], 90.00th=[  172], 95.00th=[  186],
     | 99.00th=[  219], 99.50th=[  231], 99.90th=[  260], 99.95th=[  273],
     | 99.99th=[  314]
   bw (  KiB/s): min=147195, max=214268, per=70.27%, avg=151708.63, stdev=1850.06, samples=152
   iops        : min=36796, max=53563, avg=37924.42, stdev=462.48, samples=152
  lat (usec)   : 100=0.01%, 250=99.83%, 500=0.16%, 750=0.01%
  lat (msec)   : 2=0.01%, 4=0.01%
  cpu          : usr=1.86%, sys=11.29%, ctx=540117, majf=0, minf=75
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=539779,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8
4kQ8T8write: (groupid=3, jobs=8): err= 0: pid=3022: Tue Sep 10 14:02:50 2019
  write: IOPS=31.5k, BW=123MiB/s (129MB/s)(1256MiB/10200msec); 0 zone resets
    clat (usec): min=4, max=1306.4k, avg=252.44, stdev=12776.17
     lat (usec): min=4, max=1306.4k, avg=252.63, stdev=12776.17
    clat percentiles (usec):
     |  1.00th=[     7],  5.00th=[    10], 10.00th=[    11], 20.00th=[    13],
     | 30.00th=[    15], 40.00th=[    17], 50.00th=[    20], 60.00th=[    26],
     | 70.00th=[   194], 80.00th=[   229], 90.00th=[   265], 95.00th=[   293],
     | 99.00th=[   523], 99.50th=[   775], 99.90th=[  1401], 99.95th=[  1729],
     | 99.99th=[759170]
   bw (  KiB/s): min=20367, max=501819, per=100.00%, avg=181662.04, stdev=15622.51, samples=110
   iops        : min= 5087, max=125451, avg=45412.24, stdev=3905.66, samples=110
  lat (usec)   : 10=6.44%, 20=46.47%, 50=8.16%, 100=0.29%, 250=25.34%
  lat (usec)   : 500=12.19%, 750=0.52%, 1000=0.42%
  lat (msec)   : 2=0.14%, 4=0.02%, 50=0.01%, 100=0.01%, 250=0.01%
  lat (msec)   : 500=0.01%, 750=0.01%, 1000=0.01%, 2000=0.01%
  cpu          : usr=0.84%, sys=8.83%, ctx=135933, majf=0, minf=82
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,321484,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8
4kQ32T1read: (groupid=4, jobs=1): err= 0: pid=3030: Tue Sep 10 14:02:50 2019
  read: IOPS=8640, BW=33.8MiB/s (35.4MB/s)(338MiB/10001msec)
    clat (usec): min=95, max=8677, avg=115.05, stdev=30.52
     lat (usec): min=95, max=8677, avg=115.12, stdev=30.52
    clat percentiles (usec):
     |  1.00th=[   99],  5.00th=[  109], 10.00th=[  110], 20.00th=[  110],
     | 30.00th=[  111], 40.00th=[  111], 50.00th=[  112], 60.00th=[  113],
     | 70.00th=[  125], 80.00th=[  126], 90.00th=[  127], 95.00th=[  127],
     | 99.00th=[  128], 99.50th=[  129], 99.90th=[  135], 99.95th=[  137],
     | 99.99th=[  163]
   bw (  KiB/s): min=27901, max=34240, per=83.27%, avg=28779.26, stdev=1330.24, samples=19
   iops        : min= 6975, max= 8560, avg=7194.42, stdev=332.65, samples=19
  lat (usec)   : 100=1.47%, 250=98.53%, 500=0.01%, 750=0.01%
  lat (msec)   : 2=0.01%, 10=0.01%
  cpu          : usr=1.38%, sys=5.77%, ctx=86417, majf=0, minf=11
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=86413,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
4kQ32T1write: (groupid=5, jobs=1): err= 0: pid=3031: Tue Sep 10 14:02:50 2019
  write: IOPS=24.6k, BW=96.2MiB/s (101MB/s)(1024MiB/10650msec); 0 zone resets
    clat (usec): min=3, max=1300.0k, avg=34.66, stdev=4996.97
     lat (usec): min=3, max=1300.0k, avg=34.75, stdev=4996.97
    clat percentiles (usec):
     |  1.00th=[    5],  5.00th=[    5], 10.00th=[    5], 20.00th=[    5],
     | 30.00th=[    5], 40.00th=[    5], 50.00th=[    6], 60.00th=[    7],
     | 70.00th=[    7], 80.00th=[    9], 90.00th=[   58], 95.00th=[   77],
     | 99.00th=[   94], 99.50th=[   96], 99.90th=[  103], 99.95th=[  128],
     | 99.99th=[  159]
   bw (  KiB/s): min= 4843, max=318196, per=100.00%, avg=134503.25, stdev=90968.52, samples=12
   iops        : min= 1210, max=79549, avg=33625.50, stdev=22742.31, samples=12
  lat (usec)   : 4=0.11%, 10=82.44%, 20=1.52%, 50=4.10%, 100=11.61%
  lat (usec)   : 250=0.22%, 500=0.01%
  lat (msec)   : 2000=0.01%
  cpu          : usr=2.80%, sys=27.19%, ctx=60132, majf=0, minf=11
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,262145,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
4kQ1T1read: (groupid=6, jobs=1): err= 0: pid=3032: Tue Sep 10 14:02:50 2019
  read: IOPS=8657, BW=33.8MiB/s (35.5MB/s)(338MiB/10001msec)
    clat (usec): min=95, max=4880, avg=114.85, stdev=18.65
     lat (usec): min=95, max=4881, avg=114.92, stdev=18.65
    clat percentiles (usec):
     |  1.00th=[   99],  5.00th=[  108], 10.00th=[  109], 20.00th=[  110],
     | 30.00th=[  111], 40.00th=[  111], 50.00th=[  112], 60.00th=[  112],
     | 70.00th=[  125], 80.00th=[  126], 90.00th=[  127], 95.00th=[  127],
     | 99.00th=[  129], 99.50th=[  129], 99.90th=[  133], 99.95th=[  135],
     | 99.99th=[  163]
   bw (  KiB/s): min=23045, max=34358, per=69.14%, avg=23942.63, stdev=2523.49, samples=19
   iops        : min= 5761, max= 8589, avg=5985.21, stdev=630.86, samples=19
  lat (usec)   : 100=2.07%, 250=97.92%, 500=0.01%
  lat (msec)   : 2=0.01%, 10=0.01%
  cpu          : usr=0.97%, sys=6.12%, ctx=86584, majf=0, minf=11
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=86581,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
4kQ1T1write: (groupid=7, jobs=1): err= 0: pid=3033: Tue Sep 10 14:02:50 2019
  write: IOPS=38.3k, BW=149MiB/s (157MB/s)(1600MiB/10702msec); 0 zone resets
    clat (usec): min=3, max=1504.4k, avg=22.01, stdev=3430.79
     lat (usec): min=3, max=1504.4k, avg=22.11, stdev=3430.79
    clat percentiles (usec):
     |  1.00th=[    4],  5.00th=[    4], 10.00th=[    5], 20.00th=[    5],
     | 30.00th=[    5], 40.00th=[    5], 50.00th=[    5], 60.00th=[    6],
     | 70.00th=[    6], 80.00th=[    7], 90.00th=[   39], 95.00th=[   70],
     | 99.00th=[   91], 99.50th=[   96], 99.90th=[  103], 99.95th=[  112],
     | 99.99th=[  190]
   bw (  KiB/s): min= 1053, max=472944, per=100.00%, avg=218439.27, stdev=160092.70, samples=15
   iops        : min=  263, max=118236, avg=54609.80, stdev=40023.20, samples=15
  lat (usec)   : 4=6.63%, 10=81.95%, 20=1.20%, 50=2.56%, 100=7.48%
  lat (usec)   : 250=0.17%, 500=0.01%, 750=0.01%
  lat (msec)   : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 250=0.01%
  lat (msec)   : 500=0.01%, 2000=0.01%
  cpu          : usr=4.68%, sys=30.36%, ctx=56291, majf=0, minf=11
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,409575,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=488MiB/s (511MB/s), 488MiB/s-488MiB/s (511MB/s-511MB/s), io=4877MiB (5114MB), run=10001-10001msec

Run status group 1 (all jobs):
  WRITE: bw=259MiB/s (272MB/s), 259MiB/s-259MiB/s (272MB/s-272MB/s), io=2815MiB (2952MB), run=10872-10872msec

Run status group 2 (all jobs):
   READ: bw=211MiB/s (221MB/s), 211MiB/s-211MiB/s (221MB/s-221MB/s), io=2109MiB (2211MB), run=10001-10001msec

Run status group 3 (all jobs):
  WRITE: bw=123MiB/s (129MB/s), 123MiB/s-123MiB/s (129MB/s-129MB/s), io=1256MiB (1317MB), run=10200-10200msec

Run status group 4 (all jobs):
   READ: bw=33.8MiB/s (35.4MB/s), 33.8MiB/s-33.8MiB/s (35.4MB/s-35.4MB/s), io=338MiB (354MB), run=10001-10001msec

Run status group 5 (all jobs):
  WRITE: bw=96.2MiB/s (101MB/s), 96.2MiB/s-96.2MiB/s (101MB/s-101MB/s), io=1024MiB (1074MB), run=10650-10650msec

Run status group 6 (all jobs):
   READ: bw=33.8MiB/s (35.5MB/s), 33.8MiB/s-33.8MiB/s (35.5MB/s-35.5MB/s), io=338MiB (355MB), run=10001-10001msec

Run status group 7 (all jobs):
  WRITE: bw=149MiB/s (157MB/s), 149MiB/s-149MiB/s (157MB/s-157MB/s), io=1600MiB (1678MB), run=10702-10702msec
root@692ced701c1d:/workspace/fio# cd /workspace
root@692ced701c1d:/workspace# git clone https://github.com/tensorflow/benchmarks.git -b cnn_tf_v1.13_compatible
Cloning into 'benchmarks'...
remote: Enumerating objects: 40, done.
remote: Counting objects: 100% (40/40), done.
remote: Compressing objects: 100% (29/29), done.
remote: Total 4174 (delta 21), reused 17 (delta 11), pack-reused 4134
Receiving objects: 100% (4174/4174), 2.17 MiB | 1.63 MiB/s, done.
Resolving deltas: 100% (2793/2793), done.
root@692ced701c1d:/workspace# cd benchmarks/scripts/tf_cnn_benchmarks/
root@692ced701c1d:/workspace/benchmarks/scripts/tf_cnn_benchmarks# vim run.sh
root@692ced701c1d:/workspace/benchmarks/scripts/tf_cnn_benchmarks# cat run.sh
python tf_cnn_benchmarks.py --data_format=NHWC --batch_size=64 --num_batches=100 --model=resnet50 --optimizer=sgd --variable_update=replicated --use_fp16=False --distortions=False --local_parameter_device=gpu --num_gpus=1 --display_every=10
python tf_cnn_benchmarks.py --data_format=NHWC --batch_size=64 --num_batches=100 --model=resnet50 --optimizer=sgd --variable_update=replicated --use_fp16=True --distortions=False --local_parameter_device=gpu --num_gpus=1 --display_every=10
python tf_cnn_benchmarks.py --data_format=NHWC --batch_size=128 --num_batches=100 --model=resnet50 --optimizer=sgd --variable_update=replicated --use_fp16=False --distortions=False --local_parameter_device=gpu --num_gpus=1 --display_every=10
python tf_cnn_benchmarks.py --data_format=NHWC --batch_size=128 --num_batches=100 --model=resnet50 --optimizer=sgd --variable_update=replicated --use_fp16=True --distortions=False --local_parameter_device=gpu --num_gpus=1 --display_every=10
python tf_cnn_benchmarks.py --data_format=NHWC --batch_size=256 --num_batches=100 --model=resnet50 --optimizer=sgd --variable_update=replicated --use_fp16=False --distortions=False --local_parameter_device=gpu --num_gpus=1 --display_every=10
python tf_cnn_benchmarks.py --data_format=NHWC --batch_size=256 --num_batches=100 --model=resnet50 --optimizer=sgd --variable_update=replicated --use_fp16=True --distortions=False --local_parameter_device=gpu --num_gpus=1 --display_every=10
python tf_cnn_benchmarks.py --data_format=NHWC --batch_size=512 --num_batches=100 --model=resnet50 --optimizer=sgd --variable_update=replicated --use_fp16=True --distortions=False --local_parameter_device=gpu --num_gpus=1 --display_every=10

python tf_cnn_benchmarks.py --data_format=NHWC --batch_size=64 --num_batches=100 --model=resnet50 --optimizer=sgd --variable_update=replicated --use_fp16=False --distortions=False --local_parameter_device=gpu --num_gpus=2 --display_every=10
python tf_cnn_benchmarks.py --data_format=NHWC --batch_size=64 --num_batches=100 --model=resnet50 --optimizer=sgd --variable_update=replicated --use_fp16=True --distortions=False --local_parameter_device=gpu --num_gpus=2 --display_every=10
python tf_cnn_benchmarks.py --data_format=NHWC --batch_size=128 --num_batches=100 --model=resnet50 --optimizer=sgd --variable_update=replicated --use_fp16=False --distortions=False --local_parameter_device=gpu --num_gpus=2 --display_every=10
python tf_cnn_benchmarks.py --data_format=NHWC --batch_size=128 --num_batches=100 --model=resnet50 --optimizer=sgd --variable_update=replicated --use_fp16=True --distortions=False --local_parameter_device=gpu --num_gpus=2 --display_every=10
python tf_cnn_benchmarks.py --data_format=NHWC --batch_size=256 --num_batches=100 --model=resnet50 --optimizer=sgd --variable_update=replicated --use_fp16=False --distortions=False --local_parameter_device=gpu --num_gpus=2 --display_every=10
python tf_cnn_benchmarks.py --data_format=NHWC --batch_size=256 --num_batches=100 --model=resnet50 --optimizer=sgd --variable_update=replicated --use_fp16=True --distortions=False --local_parameter_device=gpu --num_gpus=2 --display_every=10
python tf_cnn_benchmarks.py --data_format=NHWC --batch_size=512 --num_batches=100 --model=resnet50 --optimizer=sgd --variable_update=replicated --use_fp16=True --distortions=False --local_parameter_device=gpu --num_gpus=2 --display_every=10

python tf_cnn_benchmarks.py --data_format=NHWC --batch_size=64 --num_batches=100 --model=resnet50 --optimizer=sgd --variable_update=replicated --use_fp16=False --distortions=False --local_parameter_device=gpu --num_gpus=3 --display_every=10
python tf_cnn_benchmarks.py --data_format=NHWC --batch_size=64 --num_batches=100 --model=resnet50 --optimizer=sgd --variable_update=replicated --use_fp16=True --distortions=False --local_parameter_device=gpu --num_gpus=3 --display_every=10
python tf_cnn_benchmarks.py --data_format=NHWC --batch_size=128 --num_batches=100 --model=resnet50 --optimizer=sgd --variable_update=replicated --use_fp16=False --distortions=False --local_parameter_device=gpu --num_gpus=3 --display_every=10
python tf_cnn_benchmarks.py --data_format=NHWC --batch_size=128 --num_batches=100 --model=resnet50 --optimizer=sgd --variable_update=replicated --use_fp16=True --distortions=False --local_parameter_device=gpu --num_gpus=3 --display_every=10
python tf_cnn_benchmarks.py --data_format=NHWC --batch_size=256 --num_batches=100 --model=resnet50 --optimizer=sgd --variable_update=replicated --use_fp16=False --distortions=False --local_parameter_device=gpu --num_gpus=3 --display_every=10
python tf_cnn_benchmarks.py --data_format=NHWC --batch_size=256 --num_batches=100 --model=resnet50 --optimizer=sgd --variable_update=replicated --use_fp16=True --distortions=False --local_parameter_device=gpu --num_gpus=3 --display_every=10
python tf_cnn_benchmarks.py --data_format=NHWC --batch_size=512 --num_batches=100 --model=resnet50 --optimizer=sgd --variable_update=replicated --use_fp16=True --distortions=False --local_parameter_device=gpu --num_gpus=3 --display_every=10

python tf_cnn_benchmarks.py --data_format=NHWC --batch_size=64 --num_batches=100 --model=resnet50 --optimizer=sgd --variable_update=replicated --use_fp16=False --distortions=False --local_parameter_device=gpu --num_gpus=4 --display_every=10
python tf_cnn_benchmarks.py --data_format=NHWC --batch_size=64 --num_batches=100 --model=resnet50 --optimizer=sgd --variable_update=replicated --use_fp16=True --distortions=False --local_parameter_device=gpu --num_gpus=4 --display_every=10
python tf_cnn_benchmarks.py --data_format=NHWC --batch_size=128 --num_batches=100 --model=resnet50 --optimizer=sgd --variable_update=replicated --use_fp16=False --distortions=False --local_parameter_device=gpu --num_gpus=4 --display_every=10
python tf_cnn_benchmarks.py --data_format=NHWC --batch_size=128 --num_batches=100 --model=resnet50 --optimizer=sgd --variable_update=replicated --use_fp16=True --distortions=False --local_parameter_device=gpu --num_gpus=4 --display_every=10
python tf_cnn_benchmarks.py --data_format=NHWC --batch_size=256 --num_batches=100 --model=resnet50 --optimizer=sgd --variable_update=replicated --use_fp16=False --distortions=False --local_parameter_device=gpu --num_gpus=4 --display_every=10
python tf_cnn_benchmarks.py --data_format=NHWC --batch_size=256 --num_batches=100 --model=resnet50 --optimizer=sgd --variable_update=replicated --use_fp16=True --distortions=False --local_parameter_device=gpu --num_gpus=4 --display_every=10
python tf_cnn_benchmarks.py --data_format=NHWC --batch_size=512 --num_batches=100 --model=resnet50 --optimizer=sgd --variable_update=replicated --use_fp16=True --distortions=False --local_parameter_device=gpu --num_gpus=4 --display_every=10

root@692ced701c1d:/workspace/benchmarks/scripts/tf_cnn_benchmarks# sh run.sh
2019-09-10 14:14:28.007945: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
WARNING: Logging before flag parsing goes to stderr.
W0910 14:14:30.023041 140493887063872 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/models/mobilenet.py:388: The name tf.nn.avg_pool is deprecated. Please use tf.nn.avg_pool2d instead.

W0910 14:14:30.027451 140493887063872 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/models/experimental/deepspeech.py:131: The name tf.nn.rnn_cell.RNNCell is deprecated. Please use tf.compat.v1.nn.rnn_cell.RNNCell instead.

W0910 14:14:30.050584 140493887063872 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:722: The name tf.ConfigProto is deprecated. Please use tf.compat.v1.ConfigProto instead.

W0910 14:14:30.050790 140493887063872 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:3503: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead.

2019-09-10 14:14:30.076690: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2198780000 Hz
2019-09-10 14:14:30.079387: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x40b6af0 executing computations on platform Host. Devices:
2019-09-10 14:14:30.079428: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): <undefined>, <undefined>
2019-09-10 14:14:30.083433: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcuda.so.1
2019-09-10 14:14:30.972866: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x40b7000 executing computations on platform CUDA. Devices:
2019-09-10 14:14:30.972907: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): Tesla V100-DGXS-32GB, Compute Capability 7.0
2019-09-10 14:14:30.972921: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (1): Tesla V100-DGXS-32GB, Compute Capability 7.0
2019-09-10 14:14:30.972932: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (2): Tesla V100-DGXS-32GB, Compute Capability 7.0
2019-09-10 14:14:30.972942: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (3): Tesla V100-DGXS-32GB, Compute Capability 7.0
2019-09-10 14:14:30.982199: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:07:00.0
2019-09-10 14:14:30.984747: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 1 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:08:00.0
2019-09-10 14:14:30.986562: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 2 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:0e:00.0
2019-09-10 14:14:30.988391: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 3 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:0f:00.0
2019-09-10 14:14:30.988427: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
2019-09-10 14:14:30.990480: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10
2019-09-10 14:14:30.992303: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcufft.so.10
2019-09-10 14:14:30.992670: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcurand.so.10
2019-09-10 14:14:30.994771: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusolver.so.10
2019-09-10 14:14:30.996007: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusparse.so.10
2019-09-10 14:14:30.999773: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
2019-09-10 14:14:31.011971: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0, 1, 2, 3
2019-09-10 14:14:31.012005: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
2019-09-10 14:14:32.783594: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1181] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-09-10 14:14:32.783634: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1187]      0 1 2 3
2019-09-10 14:14:32.783647: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 0:   N Y Y Y
2019-09-10 14:14:32.783654: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 1:   Y N Y Y
2019-09-10 14:14:32.783661: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 2:   Y Y N Y
2019-09-10 14:14:32.783667: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 3:   Y Y Y N
2019-09-10 14:14:32.791691: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 30438 MB memory) -> physical GPU (device: 0, name: Tesla V100-DGXS-32GB, pci bus id: 0000:07:00.0, compute capability: 7.0)
2019-09-10 14:14:32.793914: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 30463 MB memory) -> physical GPU (device: 1, name: Tesla V100-DGXS-32GB, pci bus id: 0000:08:00.0, compute capability: 7.0)
2019-09-10 14:14:32.796062: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:2 with 30463 MB memory) -> physical GPU (device: 2, name: Tesla V100-DGXS-32GB, pci bus id: 0000:0e:00.0, compute capability: 7.0)
2019-09-10 14:14:32.798258: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:3 with 30463 MB memory) -> physical GPU (device: 3, name: Tesla V100-DGXS-32GB, pci bus id: 0000:0f:00.0, compute capability: 7.0)
TensorFlow:  1.14
Model:       resnet50
Dataset:     imagenet (synthetic)
Mode:        training
SingleSess:  False
Batch size:  64 global
             64 per device
Num batches: 100
Num epochs:  0.00
Devices:     ['/gpu:0']
NUMA bind:   False
Data format: NHWC
Optimizer:   sgd
Variables:   replicated
AllReduce:   None
==========
W0910 14:14:32.804810 140493887063872 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2762: The name tf.set_random_seed is deprecated. Please use tf.compat.v1.set_random_seed instead.

Generating training model
W0910 14:14:32.805390 140493887063872 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2782: The name tf.train.get_or_create_global_step is deprecated. Please use tf.compat.v1.train.get_or_create_global_step instead.

W0910 14:14:32.814951 140493887063872 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/variable_mgr.py:316: The name tf.variable_scope is deprecated. Please use tf.compat.v1.variable_scope instead.

W0910 14:14:32.815586 140493887063872 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/models/model.py:245: The name tf.truncated_normal is deprecated. Please use tf.random.truncated_normal instead.

W0910 14:14:32.832383 140493887063872 deprecation.py:323] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:129: conv2d (from tensorflow.python.layers.convolutional) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.keras.layers.Conv2D` instead.
W0910 14:14:33.195570 140493887063872 deprecation.py:323] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:261: max_pooling2d (from tensorflow.python.layers.pooling) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.MaxPooling2D instead.
W0910 14:14:35.705537 140493887063872 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:334: The name tf.nn.xw_plus_b is deprecated. Please use tf.compat.v1.nn.xw_plus_b instead.

W0910 14:14:35.707565 140493887063872 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/models/model.py:319: The name tf.losses.sparse_softmax_cross_entropy is deprecated. Please use tf.compat.v1.losses.sparse_softmax_cross_entropy instead.

W0910 14:14:35.718987 140493887063872 deprecation.py:323] From /usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/losses/losses_impl.py:121: add_dispatch_support.<locals>.wrapper (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.where in 2.0, which has the same broadcast rule as np.where
W0910 14:14:36.514074 140493887063872 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/models/resnet_model.py:333: The name tf.train.piecewise_constant is deprecated. Please use tf.compat.v1.train.piecewise_constant instead.

W0910 14:14:36.600245 140493887063872 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:1209: The name tf.train.GradientDescentOptimizer is deprecated. Please use tf.compat.v1.train.GradientDescentOptimizer instead.

W0910 14:14:36.759890 140493887063872 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2090: The name tf.train.get_global_step is deprecated. Please use tf.compat.v1.train.get_global_step instead.

W0910 14:14:36.764339 140493887063872 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2126: The name tf.summary.merge_all is deprecated. Please use tf.compat.v1.summary.merge_all instead.

Initializing graph
W0910 14:14:36.764563 140493887063872 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2183: The name tf.train.Saver is deprecated. Please use tf.compat.v1.train.Saver instead.

W0910 14:14:37.020146 140493887063872 deprecation.py:323] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2238: Supervisor.__init__ (from tensorflow.python.training.supervisor) is deprecated and will be removed in a future version.
Instructions for updating:
Please switch to tf.train.MonitoredTrainingSession
2019-09-10 14:14:37.468795: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:07:00.0
2019-09-10 14:14:37.470325: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 1 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:08:00.0
2019-09-10 14:14:37.471836: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 2 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:0e:00.0
2019-09-10 14:14:37.473342: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 3 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:0f:00.0
2019-09-10 14:14:37.473374: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
2019-09-10 14:14:37.473416: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10
2019-09-10 14:14:37.473437: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcufft.so.10
2019-09-10 14:14:37.473456: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcurand.so.10
2019-09-10 14:14:37.473473: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusolver.so.10
2019-09-10 14:14:37.473492: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusparse.so.10
2019-09-10 14:14:37.473511: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
2019-09-10 14:14:37.485387: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0, 1, 2, 3
2019-09-10 14:14:37.485446: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1181] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-09-10 14:14:37.485456: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1187]      0 1 2 3
2019-09-10 14:14:37.485464: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 0:   N Y Y Y
2019-09-10 14:14:37.485471: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 1:   Y N Y Y
2019-09-10 14:14:37.485478: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 2:   Y Y N Y
2019-09-10 14:14:37.485485: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 3:   Y Y Y N
2019-09-10 14:14:37.493479: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 30438 MB memory) -> physical GPU (device: 0, name: Tesla V100-DGXS-32GB, pci bus id: 0000:07:00.0, compute capability: 7.0)
2019-09-10 14:14:37.495080: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 30463 MB memory) -> physical GPU (device: 1, name: Tesla V100-DGXS-32GB, pci bus id: 0000:08:00.0, compute capability: 7.0)
2019-09-10 14:14:37.496672: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:2 with 30463 MB memory) -> physical GPU (device: 2, name: Tesla V100-DGXS-32GB, pci bus id: 0000:0e:00.0, compute capability: 7.0)
2019-09-10 14:14:37.498259: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:3 with 30463 MB memory) -> physical GPU (device: 3, name: Tesla V100-DGXS-32GB, pci bus id: 0000:0f:00.0, compute capability: 7.0)
2019-09-10 14:14:38.017979: W tensorflow/compiler/jit/mark_for_compilation_pass.cc:1412] (One-time warning): Not using XLA:CPU for cluster because envvar TF_XLA_FLAGS=--tf_xla_cpu_global_jit was not set.  If you want XLA:CPU, either set that envvar, or use experimental_jit_scope to enable XLA:CPU.  To confirm that XLA is active, pass --vmodule=xla_compilation_cache=1 (as a proper command-line flag, not via TF_XLA_FLAGS) or set the envvar XLA_FLAGS=--xla_hlo_profile.
I0910 14:14:38.513500 140493887063872 session_manager.py:500] Running local_init_op.
I0910 14:14:38.560857 140493887063872 session_manager.py:502] Done running local_init_op.
Running warm up
2019-09-10 14:14:40.057473: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10
2019-09-10 14:14:40.379174: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
Done warm up
Step    Img/sec total_loss
1       images/sec: 375.1 +/- 0.0 (jitter = 0.0)        7.695
10      images/sec: 375.4 +/- 0.1 (jitter = 0.4)        8.122
20      images/sec: 375.3 +/- 0.1 (jitter = 0.5)        8.039
30      images/sec: 375.3 +/- 0.1 (jitter = 0.5)        7.974
40      images/sec: 375.3 +/- 0.1 (jitter = 0.5)        7.807
50      images/sec: 375.3 +/- 0.1 (jitter = 0.4)        7.639
60      images/sec: 375.3 +/- 0.0 (jitter = 0.4)        7.610
70      images/sec: 375.3 +/- 0.0 (jitter = 0.4)        8.210
80      images/sec: 375.4 +/- 0.0 (jitter = 0.3)        7.699
90      images/sec: 375.4 +/- 0.0 (jitter = 0.4)        7.823
100     images/sec: 375.3 +/- 0.0 (jitter = 0.4)        7.743
----------------------------------------------------------------
total images/sec: 375.17
----------------------------------------------------------------
2019-09-10 14:15:04.238046: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
WARNING: Logging before flag parsing goes to stderr.
W0910 14:15:06.232517 139697050122048 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/models/mobilenet.py:388: The name tf.nn.avg_pool is deprecated. Please use tf.nn.avg_pool2d instead.

W0910 14:15:06.236979 139697050122048 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/models/experimental/deepspeech.py:131: The name tf.nn.rnn_cell.RNNCell is deprecated. Please use tf.compat.v1.nn.rnn_cell.RNNCell instead.

W0910 14:15:06.260249 139697050122048 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:722: The name tf.ConfigProto is deprecated. Please use tf.compat.v1.ConfigProto instead.

W0910 14:15:06.260469 139697050122048 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:3503: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead.

2019-09-10 14:15:06.284705: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2198780000 Hz
2019-09-10 14:15:06.287244: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x45b0c30 executing computations on platform Host. Devices:
2019-09-10 14:15:06.287278: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): <undefined>, <undefined>
2019-09-10 14:15:06.290724: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcuda.so.1
2019-09-10 14:15:07.130993: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x45b1140 executing computations on platform CUDA. Devices:
2019-09-10 14:15:07.131040: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): Tesla V100-DGXS-32GB, Compute Capability 7.0
2019-09-10 14:15:07.131061: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (1): Tesla V100-DGXS-32GB, Compute Capability 7.0
2019-09-10 14:15:07.131077: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (2): Tesla V100-DGXS-32GB, Compute Capability 7.0
2019-09-10 14:15:07.131092: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (3): Tesla V100-DGXS-32GB, Compute Capability 7.0
2019-09-10 14:15:07.134313: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:07:00.0
2019-09-10 14:15:07.136989: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 1 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:08:00.0
2019-09-10 14:15:07.139629: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 2 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:0e:00.0
2019-09-10 14:15:07.142273: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 3 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:0f:00.0
2019-09-10 14:15:07.142322: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
2019-09-10 14:15:07.145260: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10
2019-09-10 14:15:07.147050: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcufft.so.10
2019-09-10 14:15:07.147396: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcurand.so.10
2019-09-10 14:15:07.149438: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusolver.so.10
2019-09-10 14:15:07.150627: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusparse.so.10
2019-09-10 14:15:07.154879: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
2019-09-10 14:15:07.168760: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0, 1, 2, 3
2019-09-10 14:15:07.168805: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
2019-09-10 14:15:09.016489: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1181] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-09-10 14:15:09.016539: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1187]      0 1 2 3
2019-09-10 14:15:09.016549: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 0:   N Y Y Y
2019-09-10 14:15:09.016556: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 1:   Y N Y Y
2019-09-10 14:15:09.016563: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 2:   Y Y N Y
2019-09-10 14:15:09.016569: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 3:   Y Y Y N
2019-09-10 14:15:09.024561: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 30438 MB memory) -> physical GPU (device: 0, name: Tesla V100-DGXS-32GB, pci bus id: 0000:07:00.0, compute capability: 7.0)
2019-09-10 14:15:09.026792: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 30463 MB memory) -> physical GPU (device: 1, name: Tesla V100-DGXS-32GB, pci bus id: 0000:08:00.0, compute capability: 7.0)
2019-09-10 14:15:09.029027: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:2 with 30463 MB memory) -> physical GPU (device: 2, name: Tesla V100-DGXS-32GB, pci bus id: 0000:0e:00.0, compute capability: 7.0)
2019-09-10 14:15:09.031196: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:3 with 30463 MB memory) -> physical GPU (device: 3, name: Tesla V100-DGXS-32GB, pci bus id: 0000:0f:00.0, compute capability: 7.0)
TensorFlow:  1.14
Model:       resnet50
Dataset:     imagenet (synthetic)
Mode:        training
SingleSess:  False
Batch size:  64 global
             64 per device
Num batches: 100
Num epochs:  0.00
Devices:     ['/gpu:0']
NUMA bind:   False
Data format: NHWC
Optimizer:   sgd
Variables:   replicated
AllReduce:   None
==========
W0910 14:15:09.037749 139697050122048 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2762: The name tf.set_random_seed is deprecated. Please use tf.compat.v1.set_random_seed instead.

Generating training model
W0910 14:15:09.038326 139697050122048 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2782: The name tf.train.get_or_create_global_step is deprecated. Please use tf.compat.v1.train.get_or_create_global_step instead.

W0910 14:15:09.046648 139697050122048 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2739: The name tf.get_variable is deprecated. Please use tf.compat.v1.get_variable instead.

W0910 14:15:09.051440 139697050122048 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/variable_mgr.py:316: The name tf.variable_scope is deprecated. Please use tf.compat.v1.variable_scope instead.

W0910 14:15:09.068489 139697050122048 deprecation.py:323] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:129: conv2d (from tensorflow.python.layers.convolutional) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.keras.layers.Conv2D` instead.
W0910 14:15:09.437392 139697050122048 deprecation.py:323] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:261: max_pooling2d (from tensorflow.python.layers.pooling) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.MaxPooling2D instead.
W0910 14:15:12.105584 139697050122048 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:334: The name tf.nn.xw_plus_b is deprecated. Please use tf.compat.v1.nn.xw_plus_b instead.

W0910 14:15:12.108491 139697050122048 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/models/model.py:319: The name tf.losses.sparse_softmax_cross_entropy is deprecated. Please use tf.compat.v1.losses.sparse_softmax_cross_entropy instead.

W0910 14:15:12.120260 139697050122048 deprecation.py:323] From /usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/losses/losses_impl.py:121: add_dispatch_support.<locals>.wrapper (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.where in 2.0, which has the same broadcast rule as np.where
W0910 14:15:13.469039 139697050122048 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/models/resnet_model.py:333: The name tf.train.piecewise_constant is deprecated. Please use tf.compat.v1.train.piecewise_constant instead.

W0910 14:15:13.556470 139697050122048 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:1209: The name tf.train.GradientDescentOptimizer is deprecated. Please use tf.compat.v1.train.GradientDescentOptimizer instead.

W0910 14:15:13.720479 139697050122048 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2090: The name tf.train.get_global_step is deprecated. Please use tf.compat.v1.train.get_global_step instead.

W0910 14:15:13.725033 139697050122048 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2126: The name tf.summary.merge_all is deprecated. Please use tf.compat.v1.summary.merge_all instead.

Initializing graph
W0910 14:15:13.725260 139697050122048 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2183: The name tf.train.Saver is deprecated. Please use tf.compat.v1.train.Saver instead.

W0910 14:15:14.075250 139697050122048 deprecation.py:323] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2238: Supervisor.__init__ (from tensorflow.python.training.supervisor) is deprecated and will be removed in a future version.
Instructions for updating:
Please switch to tf.train.MonitoredTrainingSession
2019-09-10 14:15:14.568947: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:07:00.0
2019-09-10 14:15:14.570451: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 1 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:08:00.0
2019-09-10 14:15:14.571934: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 2 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:0e:00.0
2019-09-10 14:15:14.573426: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 3 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:0f:00.0
2019-09-10 14:15:14.573456: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
2019-09-10 14:15:14.573491: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10
2019-09-10 14:15:14.573509: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcufft.so.10
2019-09-10 14:15:14.573527: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcurand.so.10
2019-09-10 14:15:14.573544: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusolver.so.10
2019-09-10 14:15:14.573561: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusparse.so.10
2019-09-10 14:15:14.573578: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
2019-09-10 14:15:14.585324: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0, 1, 2, 3
2019-09-10 14:15:14.585382: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1181] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-09-10 14:15:14.585394: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1187]      0 1 2 3
2019-09-10 14:15:14.585402: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 0:   N Y Y Y
2019-09-10 14:15:14.585409: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 1:   Y N Y Y
2019-09-10 14:15:14.585416: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 2:   Y Y N Y
2019-09-10 14:15:14.585423: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 3:   Y Y Y N
2019-09-10 14:15:14.593141: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 30438 MB memory) -> physical GPU (device: 0, name: Tesla V100-DGXS-32GB, pci bus id: 0000:07:00.0, compute capability: 7.0)
2019-09-10 14:15:14.594725: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 30463 MB memory) -> physical GPU (device: 1, name: Tesla V100-DGXS-32GB, pci bus id: 0000:08:00.0, compute capability: 7.0)
2019-09-10 14:15:14.596310: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:2 with 30463 MB memory) -> physical GPU (device: 2, name: Tesla V100-DGXS-32GB, pci bus id: 0000:0e:00.0, compute capability: 7.0)
2019-09-10 14:15:14.597906: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:3 with 30463 MB memory) -> physical GPU (device: 3, name: Tesla V100-DGXS-32GB, pci bus id: 0000:0f:00.0, compute capability: 7.0)
2019-09-10 14:15:15.153951: W tensorflow/compiler/jit/mark_for_compilation_pass.cc:1412] (One-time warning): Not using XLA:CPU for cluster because envvar TF_XLA_FLAGS=--tf_xla_cpu_global_jit was not set.  If you want XLA:CPU, either set that envvar, or use experimental_jit_scope to enable XLA:CPU.  To confirm that XLA is active, pass --vmodule=xla_compilation_cache=1 (as a proper command-line flag, not via TF_XLA_FLAGS) or set the envvar XLA_FLAGS=--xla_hlo_profile.
I0910 14:15:15.656506 139697050122048 session_manager.py:500] Running local_init_op.
I0910 14:15:15.711758 139697050122048 session_manager.py:502] Done running local_init_op.
Running warm up
2019-09-10 14:15:17.575268: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10
2019-09-10 14:15:17.911464: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
Done warm up
Step    Img/sec total_loss
1       images/sec: 689.8 +/- 0.0 (jitter = 0.0)        7.913
10      images/sec: 692.6 +/- 0.6 (jitter = 1.2)        7.814
20      images/sec: 692.3 +/- 0.5 (jitter = 1.6)        7.953
30      images/sec: 692.7 +/- 0.4 (jitter = 1.6)        7.762
40      images/sec: 692.7 +/- 0.3 (jitter = 1.6)        7.956
50      images/sec: 692.4 +/- 0.3 (jitter = 1.4)        7.889
60      images/sec: 692.3 +/- 0.3 (jitter = 1.3)        8.009
70      images/sec: 692.2 +/- 0.2 (jitter = 1.2)        7.811
80      images/sec: 692.2 +/- 0.2 (jitter = 1.2)        7.829
90      images/sec: 692.2 +/- 0.2 (jitter = 1.2)        7.842
100     images/sec: 692.2 +/- 0.2 (jitter = 1.1)        7.834
----------------------------------------------------------------
total images/sec: 691.58
----------------------------------------------------------------
2019-09-10 14:15:33.287054: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
WARNING: Logging before flag parsing goes to stderr.
W0910 14:15:35.317740 140310324315968 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/models/mobilenet.py:388: The name tf.nn.avg_pool is deprecated. Please use tf.nn.avg_pool2d instead.

W0910 14:15:35.322293 140310324315968 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/models/experimental/deepspeech.py:131: The name tf.nn.rnn_cell.RNNCell is deprecated. Please use tf.compat.v1.nn.rnn_cell.RNNCell instead.

W0910 14:15:35.345879 140310324315968 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:722: The name tf.ConfigProto is deprecated. Please use tf.compat.v1.ConfigProto instead.

W0910 14:15:35.346103 140310324315968 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:3503: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead.

2019-09-10 14:15:35.368733: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2198780000 Hz
2019-09-10 14:15:35.371342: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x2bc9ad0 executing computations on platform Host. Devices:
2019-09-10 14:15:35.371380: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): <undefined>, <undefined>
2019-09-10 14:15:35.375354: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcuda.so.1
2019-09-10 14:15:36.248835: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x2bc2fa0 executing computations on platform CUDA. Devices:
2019-09-10 14:15:36.248875: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): Tesla V100-DGXS-32GB, Compute Capability 7.0
2019-09-10 14:15:36.248893: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (1): Tesla V100-DGXS-32GB, Compute Capability 7.0
2019-09-10 14:15:36.248909: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (2): Tesla V100-DGXS-32GB, Compute Capability 7.0
2019-09-10 14:15:36.248926: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (3): Tesla V100-DGXS-32GB, Compute Capability 7.0
2019-09-10 14:15:36.251779: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:07:00.0
2019-09-10 14:15:36.254163: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 1 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:08:00.0
2019-09-10 14:15:36.256521: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 2 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:0e:00.0
2019-09-10 14:15:36.258875: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 3 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:0f:00.0
2019-09-10 14:15:36.258921: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
2019-09-10 14:15:36.261616: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10
2019-09-10 14:15:36.263962: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcufft.so.10
2019-09-10 14:15:36.264416: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcurand.so.10
2019-09-10 14:15:36.266248: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusolver.so.10
2019-09-10 14:15:36.267235: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusparse.so.10
2019-09-10 14:15:36.270798: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
2019-09-10 14:15:36.282704: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0, 1, 2, 3
2019-09-10 14:15:36.282741: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
2019-09-10 14:15:38.061528: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1181] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-09-10 14:15:38.061571: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1187]      0 1 2 3
2019-09-10 14:15:38.061583: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 0:   N Y Y Y
2019-09-10 14:15:38.061593: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 1:   Y N Y Y
2019-09-10 14:15:38.061603: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 2:   Y Y N Y
2019-09-10 14:15:38.061612: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 3:   Y Y Y N
2019-09-10 14:15:38.069655: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 30438 MB memory) -> physical GPU (device: 0, name: Tesla V100-DGXS-32GB, pci bus id: 0000:07:00.0, compute capability: 7.0)
2019-09-10 14:15:38.071865: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 30463 MB memory) -> physical GPU (device: 1, name: Tesla V100-DGXS-32GB, pci bus id: 0000:08:00.0, compute capability: 7.0)
2019-09-10 14:15:38.074037: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:2 with 30463 MB memory) -> physical GPU (device: 2, name: Tesla V100-DGXS-32GB, pci bus id: 0000:0e:00.0, compute capability: 7.0)
2019-09-10 14:15:38.076207: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:3 with 30463 MB memory) -> physical GPU (device: 3, name: Tesla V100-DGXS-32GB, pci bus id: 0000:0f:00.0, compute capability: 7.0)
TensorFlow:  1.14
Model:       resnet50
Dataset:     imagenet (synthetic)
Mode:        training
SingleSess:  False
Batch size:  128 global
             128 per device
Num batches: 100
Num epochs:  0.01
Devices:     ['/gpu:0']
NUMA bind:   False
Data format: NHWC
Optimizer:   sgd
Variables:   replicated
AllReduce:   None
==========
W0910 14:15:38.082777 140310324315968 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2762: The name tf.set_random_seed is deprecated. Please use tf.compat.v1.set_random_seed instead.

Generating training model
W0910 14:15:38.083382 140310324315968 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2782: The name tf.train.get_or_create_global_step is deprecated. Please use tf.compat.v1.train.get_or_create_global_step instead.

W0910 14:15:38.091226 140310324315968 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/variable_mgr.py:316: The name tf.variable_scope is deprecated. Please use tf.compat.v1.variable_scope instead.

W0910 14:15:38.091602 140310324315968 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/models/model.py:245: The name tf.truncated_normal is deprecated. Please use tf.random.truncated_normal instead.

W0910 14:15:38.107810 140310324315968 deprecation.py:323] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:129: conv2d (from tensorflow.python.layers.convolutional) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.keras.layers.Conv2D` instead.
W0910 14:15:38.466178 140310324315968 deprecation.py:323] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:261: max_pooling2d (from tensorflow.python.layers.pooling) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.MaxPooling2D instead.
W0910 14:15:40.952657 140310324315968 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:334: The name tf.nn.xw_plus_b is deprecated. Please use tf.compat.v1.nn.xw_plus_b instead.

W0910 14:15:40.954668 140310324315968 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/models/model.py:319: The name tf.losses.sparse_softmax_cross_entropy is deprecated. Please use tf.compat.v1.losses.sparse_softmax_cross_entropy instead.

W0910 14:15:40.965989 140310324315968 deprecation.py:323] From /usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/losses/losses_impl.py:121: add_dispatch_support.<locals>.wrapper (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.where in 2.0, which has the same broadcast rule as np.where
W0910 14:15:41.754404 140310324315968 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/models/resnet_model.py:333: The name tf.train.piecewise_constant is deprecated. Please use tf.compat.v1.train.piecewise_constant instead.

W0910 14:15:41.839752 140310324315968 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:1209: The name tf.train.GradientDescentOptimizer is deprecated. Please use tf.compat.v1.train.GradientDescentOptimizer instead.

W0910 14:15:41.997255 140310324315968 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2090: The name tf.train.get_global_step is deprecated. Please use tf.compat.v1.train.get_global_step instead.

W0910 14:15:42.001689 140310324315968 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2126: The name tf.summary.merge_all is deprecated. Please use tf.compat.v1.summary.merge_all instead.

Initializing graph
W0910 14:15:42.001904 140310324315968 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2183: The name tf.train.Saver is deprecated. Please use tf.compat.v1.train.Saver instead.

W0910 14:15:42.255005 140310324315968 deprecation.py:323] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2238: Supervisor.__init__ (from tensorflow.python.training.supervisor) is deprecated and will be removed in a future version.
Instructions for updating:
Please switch to tf.train.MonitoredTrainingSession
2019-09-10 14:15:42.695529: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:07:00.0
2019-09-10 14:15:42.697053: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 1 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:08:00.0
2019-09-10 14:15:42.698558: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 2 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:0e:00.0
2019-09-10 14:15:42.700100: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 3 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:0f:00.0
2019-09-10 14:15:42.700129: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
2019-09-10 14:15:42.700162: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10
2019-09-10 14:15:42.700181: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcufft.so.10
2019-09-10 14:15:42.700198: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcurand.so.10
2019-09-10 14:15:42.700215: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusolver.so.10
2019-09-10 14:15:42.700231: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusparse.so.10
2019-09-10 14:15:42.700249: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
2019-09-10 14:15:42.711978: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0, 1, 2, 3
2019-09-10 14:15:42.712039: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1181] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-09-10 14:15:42.712049: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1187]      0 1 2 3
2019-09-10 14:15:42.712058: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 0:   N Y Y Y
2019-09-10 14:15:42.712065: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 1:   Y N Y Y
2019-09-10 14:15:42.712072: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 2:   Y Y N Y
2019-09-10 14:15:42.712079: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 3:   Y Y Y N
2019-09-10 14:15:42.719888: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 30438 MB memory) -> physical GPU (device: 0, name: Tesla V100-DGXS-32GB, pci bus id: 0000:07:00.0, compute capability: 7.0)
2019-09-10 14:15:42.721498: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 30463 MB memory) -> physical GPU (device: 1, name: Tesla V100-DGXS-32GB, pci bus id: 0000:08:00.0, compute capability: 7.0)
2019-09-10 14:15:42.723096: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:2 with 30463 MB memory) -> physical GPU (device: 2, name: Tesla V100-DGXS-32GB, pci bus id: 0000:0e:00.0, compute capability: 7.0)
2019-09-10 14:15:42.724719: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:3 with 30463 MB memory) -> physical GPU (device: 3, name: Tesla V100-DGXS-32GB, pci bus id: 0000:0f:00.0, compute capability: 7.0)
2019-09-10 14:15:43.240712: W tensorflow/compiler/jit/mark_for_compilation_pass.cc:1412] (One-time warning): Not using XLA:CPU for cluster because envvar TF_XLA_FLAGS=--tf_xla_cpu_global_jit was not set.  If you want XLA:CPU, either set that envvar, or use experimental_jit_scope to enable XLA:CPU.  To confirm that XLA is active, pass --vmodule=xla_compilation_cache=1 (as a proper command-line flag, not via TF_XLA_FLAGS) or set the envvar XLA_FLAGS=--xla_hlo_profile.
I0910 14:15:43.737771 140310324315968 session_manager.py:500] Running local_init_op.
I0910 14:15:43.783949 140310324315968 session_manager.py:502] Done running local_init_op.
Running warm up
2019-09-10 14:15:45.269380: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10
2019-09-10 14:15:45.590576: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
Done warm up
Step    Img/sec total_loss
1       images/sec: 397.2 +/- 0.0 (jitter = 0.0)        7.973
10      images/sec: 396.4 +/- 0.2 (jitter = 0.8)        7.882
20      images/sec: 396.4 +/- 0.2 (jitter = 0.8)        7.909
30      images/sec: 396.1 +/- 0.2 (jitter = 1.1)        7.805
40      images/sec: 395.8 +/- 0.1 (jitter = 1.1)        7.989
50      images/sec: 395.6 +/- 0.1 (jitter = 1.2)        7.868
60      images/sec: 395.5 +/- 0.1 (jitter = 1.1)        7.944
70      images/sec: 395.4 +/- 0.1 (jitter = 1.0)        7.757
80      images/sec: 395.3 +/- 0.1 (jitter = 1.0)        7.826
90      images/sec: 395.3 +/- 0.1 (jitter = 0.8)        7.947
100     images/sec: 395.2 +/- 0.1 (jitter = 0.8)        7.789
----------------------------------------------------------------
total images/sec: 395.08
----------------------------------------------------------------
2019-09-10 14:16:28.465683: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
WARNING: Logging before flag parsing goes to stderr.
W0910 14:16:30.496468 140580007774016 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/models/mobilenet.py:388: The name tf.nn.avg_pool is deprecated. Please use tf.nn.avg_pool2d instead.

W0910 14:16:30.501045 140580007774016 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/models/experimental/deepspeech.py:131: The name tf.nn.rnn_cell.RNNCell is deprecated. Please use tf.compat.v1.nn.rnn_cell.RNNCell instead.

W0910 14:16:30.524539 140580007774016 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:722: The name tf.ConfigProto is deprecated. Please use tf.compat.v1.ConfigProto instead.

W0910 14:16:30.524753 140580007774016 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:3503: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead.

2019-09-10 14:16:30.548702: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2198780000 Hz
2019-09-10 14:16:30.551402: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x394c060 executing computations on platform Host. Devices:
2019-09-10 14:16:30.551439: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): <undefined>, <undefined>
2019-09-10 14:16:30.555321: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcuda.so.1
2019-09-10 14:16:31.492745: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x3945530 executing computations on platform CUDA. Devices:
2019-09-10 14:16:31.492779: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): Tesla V100-DGXS-32GB, Compute Capability 7.0
2019-09-10 14:16:31.492791: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (1): Tesla V100-DGXS-32GB, Compute Capability 7.0
2019-09-10 14:16:31.492803: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (2): Tesla V100-DGXS-32GB, Compute Capability 7.0
2019-09-10 14:16:31.492812: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (3): Tesla V100-DGXS-32GB, Compute Capability 7.0
2019-09-10 14:16:31.495267: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:07:00.0
2019-09-10 14:16:31.497403: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 1 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:08:00.0
2019-09-10 14:16:31.499819: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 2 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:0e:00.0
2019-09-10 14:16:31.502224: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 3 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:0f:00.0
2019-09-10 14:16:31.502270: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
2019-09-10 14:16:31.505020: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10
2019-09-10 14:16:31.507436: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcufft.so.10
2019-09-10 14:16:31.507923: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcurand.so.10
2019-09-10 14:16:31.510082: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusolver.so.10
2019-09-10 14:16:31.511097: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusparse.so.10
2019-09-10 14:16:31.514765: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
2019-09-10 14:16:31.527010: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0, 1, 2, 3
2019-09-10 14:16:31.527046: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
2019-09-10 14:16:33.313778: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1181] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-09-10 14:16:33.313821: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1187]      0 1 2 3
2019-09-10 14:16:33.313832: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 0:   N Y Y Y
2019-09-10 14:16:33.313839: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 1:   Y N Y Y
2019-09-10 14:16:33.313845: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 2:   Y Y N Y
2019-09-10 14:16:33.313851: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 3:   Y Y Y N
2019-09-10 14:16:33.322112: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 30438 MB memory) -> physical GPU (device: 0, name: Tesla V100-DGXS-32GB, pci bus id: 0000:07:00.0, compute capability: 7.0)
2019-09-10 14:16:33.324370: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 30463 MB memory) -> physical GPU (device: 1, name: Tesla V100-DGXS-32GB, pci bus id: 0000:08:00.0, compute capability: 7.0)
2019-09-10 14:16:33.326570: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:2 with 30463 MB memory) -> physical GPU (device: 2, name: Tesla V100-DGXS-32GB, pci bus id: 0000:0e:00.0, compute capability: 7.0)
2019-09-10 14:16:33.328799: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:3 with 30463 MB memory) -> physical GPU (device: 3, name: Tesla V100-DGXS-32GB, pci bus id: 0000:0f:00.0, compute capability: 7.0)
TensorFlow:  1.14
Model:       resnet50
Dataset:     imagenet (synthetic)
Mode:        training
SingleSess:  False
Batch size:  128 global
             128 per device
Num batches: 100
Num epochs:  0.01
Devices:     ['/gpu:0']
NUMA bind:   False
Data format: NHWC
Optimizer:   sgd
Variables:   replicated
AllReduce:   None
==========
W0910 14:16:33.335533 140580007774016 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2762: The name tf.set_random_seed is deprecated. Please use tf.compat.v1.set_random_seed instead.

Generating training model
W0910 14:16:33.336146 140580007774016 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2782: The name tf.train.get_or_create_global_step is deprecated. Please use tf.compat.v1.train.get_or_create_global_step instead.

W0910 14:16:33.343653 140580007774016 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2739: The name tf.get_variable is deprecated. Please use tf.compat.v1.get_variable instead.

W0910 14:16:33.348349 140580007774016 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/variable_mgr.py:316: The name tf.variable_scope is deprecated. Please use tf.compat.v1.variable_scope instead.

W0910 14:16:33.364998 140580007774016 deprecation.py:323] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:129: conv2d (from tensorflow.python.layers.convolutional) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.keras.layers.Conv2D` instead.
W0910 14:16:33.723358 140580007774016 deprecation.py:323] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:261: max_pooling2d (from tensorflow.python.layers.pooling) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.MaxPooling2D instead.
W0910 14:16:36.319929 140580007774016 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:334: The name tf.nn.xw_plus_b is deprecated. Please use tf.compat.v1.nn.xw_plus_b instead.

W0910 14:16:36.322793 140580007774016 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/models/model.py:319: The name tf.losses.sparse_softmax_cross_entropy is deprecated. Please use tf.compat.v1.losses.sparse_softmax_cross_entropy instead.

W0910 14:16:36.334097 140580007774016 deprecation.py:323] From /usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/losses/losses_impl.py:121: add_dispatch_support.<locals>.wrapper (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.where in 2.0, which has the same broadcast rule as np.where
W0910 14:16:37.643736 140580007774016 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/models/resnet_model.py:333: The name tf.train.piecewise_constant is deprecated. Please use tf.compat.v1.train.piecewise_constant instead.

W0910 14:16:37.728603 140580007774016 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:1209: The name tf.train.GradientDescentOptimizer is deprecated. Please use tf.compat.v1.train.GradientDescentOptimizer instead.

W0910 14:16:37.888648 140580007774016 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2090: The name tf.train.get_global_step is deprecated. Please use tf.compat.v1.train.get_global_step instead.

W0910 14:16:37.893090 140580007774016 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2126: The name tf.summary.merge_all is deprecated. Please use tf.compat.v1.summary.merge_all instead.

Initializing graph
W0910 14:16:37.893323 140580007774016 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2183: The name tf.train.Saver is deprecated. Please use tf.compat.v1.train.Saver instead.

W0910 14:16:38.235394 140580007774016 deprecation.py:323] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2238: Supervisor.__init__ (from tensorflow.python.training.supervisor) is deprecated and will be removed in a future version.
Instructions for updating:
Please switch to tf.train.MonitoredTrainingSession
2019-09-10 14:16:38.715452: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:07:00.0
2019-09-10 14:16:38.717052: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 1 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:08:00.0
2019-09-10 14:16:38.718579: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 2 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:0e:00.0
2019-09-10 14:16:38.720079: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 3 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:0f:00.0
2019-09-10 14:16:38.720109: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
2019-09-10 14:16:38.720143: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10
2019-09-10 14:16:38.720161: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcufft.so.10
2019-09-10 14:16:38.720180: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcurand.so.10
2019-09-10 14:16:38.720206: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusolver.so.10
2019-09-10 14:16:38.720225: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusparse.so.10
2019-09-10 14:16:38.720248: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
2019-09-10 14:16:38.732009: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0, 1, 2, 3
2019-09-10 14:16:38.732068: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1181] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-09-10 14:16:38.732079: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1187]      0 1 2 3
2019-09-10 14:16:38.732087: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 0:   N Y Y Y
2019-09-10 14:16:38.732094: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 1:   Y N Y Y
2019-09-10 14:16:38.732100: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 2:   Y Y N Y
2019-09-10 14:16:38.732108: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 3:   Y Y Y N
2019-09-10 14:16:38.739951: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 30438 MB memory) -> physical GPU (device: 0, name: Tesla V100-DGXS-32GB, pci bus id: 0000:07:00.0, compute capability: 7.0)
2019-09-10 14:16:38.741566: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 30463 MB memory) -> physical GPU (device: 1, name: Tesla V100-DGXS-32GB, pci bus id: 0000:08:00.0, compute capability: 7.0)
2019-09-10 14:16:38.743171: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:2 with 30463 MB memory) -> physical GPU (device: 2, name: Tesla V100-DGXS-32GB, pci bus id: 0000:0e:00.0, compute capability: 7.0)
2019-09-10 14:16:38.744793: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:3 with 30463 MB memory) -> physical GPU (device: 3, name: Tesla V100-DGXS-32GB, pci bus id: 0000:0f:00.0, compute capability: 7.0)
2019-09-10 14:16:39.295176: W tensorflow/compiler/jit/mark_for_compilation_pass.cc:1412] (One-time warning): Not using XLA:CPU for cluster because envvar TF_XLA_FLAGS=--tf_xla_cpu_global_jit was not set.  If you want XLA:CPU, either set that envvar, or use experimental_jit_scope to enable XLA:CPU.  To confirm that XLA is active, pass --vmodule=xla_compilation_cache=1 (as a proper command-line flag, not via TF_XLA_FLAGS) or set the envvar XLA_FLAGS=--xla_hlo_profile.
I0910 14:16:39.799729 140580007774016 session_manager.py:500] Running local_init_op.
I0910 14:16:39.862023 140580007774016 session_manager.py:502] Done running local_init_op.
Running warm up
2019-09-10 14:16:41.703017: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10
2019-09-10 14:16:42.036113: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
Done warm up
Step    Img/sec total_loss
1       images/sec: 778.1 +/- 0.0 (jitter = 0.0)        7.830
10      images/sec: 778.3 +/- 0.3 (jitter = 0.9)        7.923
20      images/sec: 778.3 +/- 0.3 (jitter = 0.9)        7.879
30      images/sec: 778.2 +/- 0.2 (jitter = 0.7)        7.797
40      images/sec: 778.2 +/- 0.2 (jitter = 0.7)        7.767
50      images/sec: 778.2 +/- 0.2 (jitter = 0.7)        7.844
60      images/sec: 778.4 +/- 0.2 (jitter = 0.7)        7.794
70      images/sec: 778.4 +/- 0.2 (jitter = 0.7)        7.776
80      images/sec: 778.4 +/- 0.1 (jitter = 0.7)        7.743
90      images/sec: 778.4 +/- 0.1 (jitter = 0.8)        7.890
100     images/sec: 778.4 +/- 0.1 (jitter = 0.7)        7.918
----------------------------------------------------------------
total images/sec: 778.05
----------------------------------------------------------------
2019-09-10 14:17:06.530284: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
WARNING: Logging before flag parsing goes to stderr.
W0910 14:17:08.567694 139704933136192 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/models/mobilenet.py:388: The name tf.nn.avg_pool is deprecated. Please use tf.nn.avg_pool2d instead.

W0910 14:17:08.572294 139704933136192 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/models/experimental/deepspeech.py:131: The name tf.nn.rnn_cell.RNNCell is deprecated. Please use tf.compat.v1.nn.rnn_cell.RNNCell instead.

W0910 14:17:08.595902 139704933136192 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:722: The name tf.ConfigProto is deprecated. Please use tf.compat.v1.ConfigProto instead.

W0910 14:17:08.596122 139704933136192 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:3503: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead.

2019-09-10 14:17:08.620706: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2198780000 Hz
2019-09-10 14:17:08.623456: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x4599350 executing computations on platform Host. Devices:
2019-09-10 14:17:08.623493: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): <undefined>, <undefined>
2019-09-10 14:17:08.627239: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcuda.so.1
2019-09-10 14:17:09.508583: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x45928b0 executing computations on platform CUDA. Devices:
2019-09-10 14:17:09.508613: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): Tesla V100-DGXS-32GB, Compute Capability 7.0
2019-09-10 14:17:09.508622: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (1): Tesla V100-DGXS-32GB, Compute Capability 7.0
2019-09-10 14:17:09.508630: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (2): Tesla V100-DGXS-32GB, Compute Capability 7.0
2019-09-10 14:17:09.508636: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (3): Tesla V100-DGXS-32GB, Compute Capability 7.0
2019-09-10 14:17:09.510456: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:07:00.0
2019-09-10 14:17:09.512033: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 1 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:08:00.0
2019-09-10 14:17:09.513840: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 2 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:0e:00.0
2019-09-10 14:17:09.515865: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 3 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:0f:00.0
2019-09-10 14:17:09.515899: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
2019-09-10 14:17:09.518216: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10
2019-09-10 14:17:09.520253: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcufft.so.10
2019-09-10 14:17:09.520647: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcurand.so.10
2019-09-10 14:17:09.523039: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusolver.so.10
2019-09-10 14:17:09.524517: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusparse.so.10
2019-09-10 14:17:09.528215: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
2019-09-10 14:17:09.540129: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0, 1, 2, 3
2019-09-10 14:17:09.540163: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
2019-09-10 14:17:11.324749: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1181] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-09-10 14:17:11.324792: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1187]      0 1 2 3
2019-09-10 14:17:11.324802: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 0:   N Y Y Y
2019-09-10 14:17:11.324808: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 1:   Y N Y Y
2019-09-10 14:17:11.324814: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 2:   Y Y N Y
2019-09-10 14:17:11.324820: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 3:   Y Y Y N
2019-09-10 14:17:11.332875: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 30438 MB memory) -> physical GPU (device: 0, name: Tesla V100-DGXS-32GB, pci bus id: 0000:07:00.0, compute capability: 7.0)
2019-09-10 14:17:11.335144: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 30463 MB memory) -> physical GPU (device: 1, name: Tesla V100-DGXS-32GB, pci bus id: 0000:08:00.0, compute capability: 7.0)
2019-09-10 14:17:11.337331: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:2 with 30463 MB memory) -> physical GPU (device: 2, name: Tesla V100-DGXS-32GB, pci bus id: 0000:0e:00.0, compute capability: 7.0)
2019-09-10 14:17:11.339496: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:3 with 30463 MB memory) -> physical GPU (device: 3, name: Tesla V100-DGXS-32GB, pci bus id: 0000:0f:00.0, compute capability: 7.0)
TensorFlow:  1.14
Model:       resnet50
Dataset:     imagenet (synthetic)
Mode:        training
SingleSess:  False
Batch size:  256 global
             256 per device
Num batches: 100
Num epochs:  0.02
Devices:     ['/gpu:0']
NUMA bind:   False
Data format: NHWC
Optimizer:   sgd
Variables:   replicated
AllReduce:   None
==========
W0910 14:17:11.345422 139704933136192 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2762: The name tf.set_random_seed is deprecated. Please use tf.compat.v1.set_random_seed instead.

Generating training model
W0910 14:17:11.345918 139704933136192 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2782: The name tf.train.get_or_create_global_step is deprecated. Please use tf.compat.v1.train.get_or_create_global_step instead.

W0910 14:17:11.353884 139704933136192 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/variable_mgr.py:316: The name tf.variable_scope is deprecated. Please use tf.compat.v1.variable_scope instead.

W0910 14:17:11.354410 139704933136192 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/models/model.py:245: The name tf.truncated_normal is deprecated. Please use tf.random.truncated_normal instead.

W0910 14:17:11.371625 139704933136192 deprecation.py:323] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:129: conv2d (from tensorflow.python.layers.convolutional) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.keras.layers.Conv2D` instead.
W0910 14:17:11.745485 139704933136192 deprecation.py:323] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:261: max_pooling2d (from tensorflow.python.layers.pooling) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.MaxPooling2D instead.
W0910 14:17:14.301219 139704933136192 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:334: The name tf.nn.xw_plus_b is deprecated. Please use tf.compat.v1.nn.xw_plus_b instead.

W0910 14:17:14.303290 139704933136192 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/models/model.py:319: The name tf.losses.sparse_softmax_cross_entropy is deprecated. Please use tf.compat.v1.losses.sparse_softmax_cross_entropy instead.

W0910 14:17:14.314931 139704933136192 deprecation.py:323] From /usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/losses/losses_impl.py:121: add_dispatch_support.<locals>.wrapper (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.where in 2.0, which has the same broadcast rule as np.where
W0910 14:17:15.124845 139704933136192 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/models/resnet_model.py:333: The name tf.train.piecewise_constant is deprecated. Please use tf.compat.v1.train.piecewise_constant instead.

W0910 14:17:15.212653 139704933136192 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:1209: The name tf.train.GradientDescentOptimizer is deprecated. Please use tf.compat.v1.train.GradientDescentOptimizer instead.

W0910 14:17:15.374406 139704933136192 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2090: The name tf.train.get_global_step is deprecated. Please use tf.compat.v1.train.get_global_step instead.

W0910 14:17:15.378963 139704933136192 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2126: The name tf.summary.merge_all is deprecated. Please use tf.compat.v1.summary.merge_all instead.

Initializing graph
W0910 14:17:15.379185 139704933136192 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2183: The name tf.train.Saver is deprecated. Please use tf.compat.v1.train.Saver instead.

W0910 14:17:15.639576 139704933136192 deprecation.py:323] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2238: Supervisor.__init__ (from tensorflow.python.training.supervisor) is deprecated and will be removed in a future version.
Instructions for updating:
Please switch to tf.train.MonitoredTrainingSession
2019-09-10 14:17:16.089637: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:07:00.0
2019-09-10 14:17:16.091145: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 1 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:08:00.0
2019-09-10 14:17:16.092636: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 2 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:0e:00.0
2019-09-10 14:17:16.094119: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 3 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:0f:00.0
2019-09-10 14:17:16.094148: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
2019-09-10 14:17:16.094181: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10
2019-09-10 14:17:16.094200: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcufft.so.10
2019-09-10 14:17:16.094217: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcurand.so.10
2019-09-10 14:17:16.094235: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusolver.so.10
2019-09-10 14:17:16.094252: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusparse.so.10
2019-09-10 14:17:16.094269: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
2019-09-10 14:17:16.106013: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0, 1, 2, 3
2019-09-10 14:17:16.106071: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1181] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-09-10 14:17:16.106083: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1187]      0 1 2 3
2019-09-10 14:17:16.106091: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 0:   N Y Y Y
2019-09-10 14:17:16.106098: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 1:   Y N Y Y
2019-09-10 14:17:16.106105: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 2:   Y Y N Y
2019-09-10 14:17:16.106111: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 3:   Y Y Y N
2019-09-10 14:17:16.113818: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 30438 MB memory) -> physical GPU (device: 0, name: Tesla V100-DGXS-32GB, pci bus id: 0000:07:00.0, compute capability: 7.0)
2019-09-10 14:17:16.115412: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 30463 MB memory) -> physical GPU (device: 1, name: Tesla V100-DGXS-32GB, pci bus id: 0000:08:00.0, compute capability: 7.0)
2019-09-10 14:17:16.117008: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:2 with 30463 MB memory) -> physical GPU (device: 2, name: Tesla V100-DGXS-32GB, pci bus id: 0000:0e:00.0, compute capability: 7.0)
2019-09-10 14:17:16.118599: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:3 with 30463 MB memory) -> physical GPU (device: 3, name: Tesla V100-DGXS-32GB, pci bus id: 0000:0f:00.0, compute capability: 7.0)
2019-09-10 14:17:16.651760: W tensorflow/compiler/jit/mark_for_compilation_pass.cc:1412] (One-time warning): Not using XLA:CPU for cluster because envvar TF_XLA_FLAGS=--tf_xla_cpu_global_jit was not set.  If you want XLA:CPU, either set that envvar, or use experimental_jit_scope to enable XLA:CPU.  To confirm that XLA is active, pass --vmodule=xla_compilation_cache=1 (as a proper command-line flag, not via TF_XLA_FLAGS) or set the envvar XLA_FLAGS=--xla_hlo_profile.
I0910 14:17:17.153269 139704933136192 session_manager.py:500] Running local_init_op.
I0910 14:17:17.201341 139704933136192 session_manager.py:502] Done running local_init_op.
Running warm up
2019-09-10 14:17:18.732226: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10
2019-09-10 14:17:19.054275: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
Done warm up
Step    Img/sec total_loss
1       images/sec: 406.8 +/- 0.0 (jitter = 0.0)        8.000
10      images/sec: 406.8 +/- 0.1 (jitter = 0.5)        7.877
20      images/sec: 406.7 +/- 0.1 (jitter = 0.4)        7.968
30      images/sec: 406.6 +/- 0.1 (jitter = 0.5)        7.934
40      images/sec: 406.4 +/- 0.1 (jitter = 0.8)        7.734
50      images/sec: 406.3 +/- 0.1 (jitter = 0.7)        7.799
60      images/sec: 406.3 +/- 0.1 (jitter = 0.6)        7.819
70      images/sec: 406.3 +/- 0.1 (jitter = 0.6)        7.740
80      images/sec: 406.3 +/- 0.1 (jitter = 0.6)        7.907
90      images/sec: 406.2 +/- 0.1 (jitter = 0.6)        7.853
100     images/sec: 406.2 +/- 0.1 (jitter = 0.6)        7.827
----------------------------------------------------------------
total images/sec: 406.17
----------------------------------------------------------------
2019-09-10 14:18:39.248265: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
WARNING: Logging before flag parsing goes to stderr.
W0910 14:18:41.235126 139667033356096 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/models/mobilenet.py:388: The name tf.nn.avg_pool is deprecated. Please use tf.nn.avg_pool2d instead.

W0910 14:18:41.239565 139667033356096 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/models/experimental/deepspeech.py:131: The name tf.nn.rnn_cell.RNNCell is deprecated. Please use tf.compat.v1.nn.rnn_cell.RNNCell instead.

W0910 14:18:41.262663 139667033356096 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:722: The name tf.ConfigProto is deprecated. Please use tf.compat.v1.ConfigProto instead.

W0910 14:18:41.262866 139667033356096 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:3503: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead.

2019-09-10 14:18:41.288731: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2198780000 Hz
2019-09-10 14:18:41.291306: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x3327a70 executing computations on platform Host. Devices:
2019-09-10 14:18:41.291342: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): <undefined>, <undefined>
2019-09-10 14:18:41.295140: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcuda.so.1
2019-09-10 14:18:42.117245: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x3327dd0 executing computations on platform CUDA. Devices:
2019-09-10 14:18:42.117278: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): Tesla V100-DGXS-32GB, Compute Capability 7.0
2019-09-10 14:18:42.117289: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (1): Tesla V100-DGXS-32GB, Compute Capability 7.0
2019-09-10 14:18:42.117297: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (2): Tesla V100-DGXS-32GB, Compute Capability 7.0
2019-09-10 14:18:42.117305: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (3): Tesla V100-DGXS-32GB, Compute Capability 7.0
2019-09-10 14:18:42.119612: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:07:00.0
2019-09-10 14:18:42.121799: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 1 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:08:00.0
2019-09-10 14:18:42.124077: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 2 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:0e:00.0
2019-09-10 14:18:42.126372: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 3 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:0f:00.0
2019-09-10 14:18:42.126414: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
2019-09-10 14:18:42.128998: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10
2019-09-10 14:18:42.131276: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcufft.so.10
2019-09-10 14:18:42.131738: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcurand.so.10
2019-09-10 14:18:42.133810: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusolver.so.10
2019-09-10 14:18:42.134823: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusparse.so.10
2019-09-10 14:18:42.138445: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
2019-09-10 14:18:42.150376: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0, 1, 2, 3
2019-09-10 14:18:42.150412: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
2019-09-10 14:18:43.956251: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1181] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-09-10 14:18:43.956292: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1187]      0 1 2 3
2019-09-10 14:18:43.956303: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 0:   N Y Y Y
2019-09-10 14:18:43.956310: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 1:   Y N Y Y
2019-09-10 14:18:43.956316: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 2:   Y Y N Y
2019-09-10 14:18:43.956323: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 3:   Y Y Y N
2019-09-10 14:18:43.964345: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 30438 MB memory) -> physical GPU (device: 0, name: Tesla V100-DGXS-32GB, pci bus id: 0000:07:00.0, compute capability: 7.0)
2019-09-10 14:18:43.966609: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 30463 MB memory) -> physical GPU (device: 1, name: Tesla V100-DGXS-32GB, pci bus id: 0000:08:00.0, compute capability: 7.0)
2019-09-10 14:18:43.968846: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:2 with 30463 MB memory) -> physical GPU (device: 2, name: Tesla V100-DGXS-32GB, pci bus id: 0000:0e:00.0, compute capability: 7.0)
2019-09-10 14:18:43.971003: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:3 with 30463 MB memory) -> physical GPU (device: 3, name: Tesla V100-DGXS-32GB, pci bus id: 0000:0f:00.0, compute capability: 7.0)
TensorFlow:  1.14
Model:       resnet50
Dataset:     imagenet (synthetic)
Mode:        training
SingleSess:  False
Batch size:  256 global
             256 per device
Num batches: 100
Num epochs:  0.02
Devices:     ['/gpu:0']
NUMA bind:   False
Data format: NHWC
Optimizer:   sgd
Variables:   replicated
AllReduce:   None
==========
W0910 14:18:43.977334 139667033356096 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2762: The name tf.set_random_seed is deprecated. Please use tf.compat.v1.set_random_seed instead.

Generating training model
W0910 14:18:43.977944 139667033356096 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2782: The name tf.train.get_or_create_global_step is deprecated. Please use tf.compat.v1.train.get_or_create_global_step instead.

W0910 14:18:43.986305 139667033356096 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2739: The name tf.get_variable is deprecated. Please use tf.compat.v1.get_variable instead.

W0910 14:18:43.991040 139667033356096 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/variable_mgr.py:316: The name tf.variable_scope is deprecated. Please use tf.compat.v1.variable_scope instead.

W0910 14:18:44.007900 139667033356096 deprecation.py:323] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:129: conv2d (from tensorflow.python.layers.convolutional) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.keras.layers.Conv2D` instead.
W0910 14:18:44.368425 139667033356096 deprecation.py:323] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:261: max_pooling2d (from tensorflow.python.layers.pooling) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.MaxPooling2D instead.
W0910 14:18:47.003370 139667033356096 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:334: The name tf.nn.xw_plus_b is deprecated. Please use tf.compat.v1.nn.xw_plus_b instead.

W0910 14:18:47.006294 139667033356096 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/models/model.py:319: The name tf.losses.sparse_softmax_cross_entropy is deprecated. Please use tf.compat.v1.losses.sparse_softmax_cross_entropy instead.

W0910 14:18:47.017845 139667033356096 deprecation.py:323] From /usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/losses/losses_impl.py:121: add_dispatch_support.<locals>.wrapper (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.where in 2.0, which has the same broadcast rule as np.where
W0910 14:18:48.346340 139667033356096 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/models/resnet_model.py:333: The name tf.train.piecewise_constant is deprecated. Please use tf.compat.v1.train.piecewise_constant instead.

W0910 14:18:48.432383 139667033356096 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:1209: The name tf.train.GradientDescentOptimizer is deprecated. Please use tf.compat.v1.train.GradientDescentOptimizer instead.

W0910 14:18:48.594961 139667033356096 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2090: The name tf.train.get_global_step is deprecated. Please use tf.compat.v1.train.get_global_step instead.

W0910 14:18:48.599460 139667033356096 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2126: The name tf.summary.merge_all is deprecated. Please use tf.compat.v1.summary.merge_all instead.

Initializing graph
W0910 14:18:48.599704 139667033356096 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2183: The name tf.train.Saver is deprecated. Please use tf.compat.v1.train.Saver instead.

W0910 14:18:48.946296 139667033356096 deprecation.py:323] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2238: Supervisor.__init__ (from tensorflow.python.training.supervisor) is deprecated and will be removed in a future version.
Instructions for updating:
Please switch to tf.train.MonitoredTrainingSession
2019-09-10 14:18:49.432361: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:07:00.0
2019-09-10 14:18:49.433871: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 1 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:08:00.0
2019-09-10 14:18:49.435362: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 2 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:0e:00.0
2019-09-10 14:18:49.436882: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 3 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:0f:00.0
2019-09-10 14:18:49.436911: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
2019-09-10 14:18:49.436945: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10
2019-09-10 14:18:49.436963: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcufft.so.10
2019-09-10 14:18:49.436980: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcurand.so.10
2019-09-10 14:18:49.436997: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusolver.so.10
2019-09-10 14:18:49.437015: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusparse.so.10
2019-09-10 14:18:49.437032: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
2019-09-10 14:18:49.448728: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0, 1, 2, 3
2019-09-10 14:18:49.448786: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1181] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-09-10 14:18:49.448796: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1187]      0 1 2 3
2019-09-10 14:18:49.448804: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 0:   N Y Y Y
2019-09-10 14:18:49.448810: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 1:   Y N Y Y
2019-09-10 14:18:49.448817: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 2:   Y Y N Y
2019-09-10 14:18:49.448823: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 3:   Y Y Y N
2019-09-10 14:18:49.456572: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 30438 MB memory) -> physical GPU (device: 0, name: Tesla V100-DGXS-32GB, pci bus id: 0000:07:00.0, compute capability: 7.0)
2019-09-10 14:18:49.458238: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 30463 MB memory) -> physical GPU (device: 1, name: Tesla V100-DGXS-32GB, pci bus id: 0000:08:00.0, compute capability: 7.0)
2019-09-10 14:18:49.459823: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:2 with 30463 MB memory) -> physical GPU (device: 2, name: Tesla V100-DGXS-32GB, pci bus id: 0000:0e:00.0, compute capability: 7.0)
2019-09-10 14:18:49.461400: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:3 with 30463 MB memory) -> physical GPU (device: 3, name: Tesla V100-DGXS-32GB, pci bus id: 0000:0f:00.0, compute capability: 7.0)
2019-09-10 14:18:50.017381: W tensorflow/compiler/jit/mark_for_compilation_pass.cc:1412] (One-time warning): Not using XLA:CPU for cluster because envvar TF_XLA_FLAGS=--tf_xla_cpu_global_jit was not set.  If you want XLA:CPU, either set that envvar, or use experimental_jit_scope to enable XLA:CPU.  To confirm that XLA is active, pass --vmodule=xla_compilation_cache=1 (as a proper command-line flag, not via TF_XLA_FLAGS) or set the envvar XLA_FLAGS=--xla_hlo_profile.
I0910 14:18:50.524705 139667033356096 session_manager.py:500] Running local_init_op.
I0910 14:18:50.586862 139667033356096 session_manager.py:502] Done running local_init_op.
Running warm up
2019-09-10 14:18:52.465906: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10
2019-09-10 14:18:52.800992: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
Done warm up
Step    Img/sec total_loss
1       images/sec: 838.5 +/- 0.0 (jitter = 0.0)        7.928
10      images/sec: 838.3 +/- 0.4 (jitter = 1.0)        7.880
20      images/sec: 838.0 +/- 0.3 (jitter = 1.5)        7.827
30      images/sec: 838.0 +/- 0.2 (jitter = 1.2)        7.852
40      images/sec: 837.9 +/- 0.2 (jitter = 1.0)        7.931
50      images/sec: 838.0 +/- 0.2 (jitter = 0.9)        7.947
60      images/sec: 838.0 +/- 0.2 (jitter = 0.9)        7.867
70      images/sec: 837.9 +/- 0.2 (jitter = 1.0)        7.820
80      images/sec: 837.9 +/- 0.1 (jitter = 1.0)        7.837
90      images/sec: 837.8 +/- 0.1 (jitter = 1.1)        7.868
100     images/sec: 837.8 +/- 0.1 (jitter = 1.0)        7.831
----------------------------------------------------------------
total images/sec: 837.62
----------------------------------------------------------------
2019-09-10 14:19:37.101038: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
WARNING: Logging before flag parsing goes to stderr.
W0910 14:19:39.116133 140478665234240 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/models/mobilenet.py:388: The name tf.nn.avg_pool is deprecated. Please use tf.nn.avg_pool2d instead.

W0910 14:19:39.120661 140478665234240 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/models/experimental/deepspeech.py:131: The name tf.nn.rnn_cell.RNNCell is deprecated. Please use tf.compat.v1.nn.rnn_cell.RNNCell instead.

W0910 14:19:39.144004 140478665234240 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:722: The name tf.ConfigProto is deprecated. Please use tf.compat.v1.ConfigProto instead.

W0910 14:19:39.144218 140478665234240 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:3503: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead.

2019-09-10 14:19:39.168721: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2198780000 Hz
2019-09-10 14:19:39.171222: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x30dbdf0 executing computations on platform Host. Devices:
2019-09-10 14:19:39.171254: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): <undefined>, <undefined>
2019-09-10 14:19:39.174375: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcuda.so.1
2019-09-10 14:19:40.174912: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x30dc300 executing computations on platform CUDA. Devices:
2019-09-10 14:19:40.174959: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): Tesla V100-DGXS-32GB, Compute Capability 7.0
2019-09-10 14:19:40.174976: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (1): Tesla V100-DGXS-32GB, Compute Capability 7.0
2019-09-10 14:19:40.174990: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (2): Tesla V100-DGXS-32GB, Compute Capability 7.0
2019-09-10 14:19:40.175003: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (3): Tesla V100-DGXS-32GB, Compute Capability 7.0
2019-09-10 14:19:40.178405: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:07:00.0
2019-09-10 14:19:40.181227: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 1 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:08:00.0
2019-09-10 14:19:40.184028: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 2 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:0e:00.0
2019-09-10 14:19:40.186842: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 3 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:0f:00.0
2019-09-10 14:19:40.186887: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
2019-09-10 14:19:40.189674: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10
2019-09-10 14:19:40.191415: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcufft.so.10
2019-09-10 14:19:40.191738: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcurand.so.10
2019-09-10 14:19:40.193734: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusolver.so.10
2019-09-10 14:19:40.194897: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusparse.so.10
2019-09-10 14:19:40.199100: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
2019-09-10 14:19:40.212789: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0, 1, 2, 3
2019-09-10 14:19:40.212826: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
2019-09-10 14:19:42.021289: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1181] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-09-10 14:19:42.021330: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1187]      0 1 2 3
2019-09-10 14:19:42.021341: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 0:   N Y Y Y
2019-09-10 14:19:42.021348: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 1:   Y N Y Y
2019-09-10 14:19:42.021354: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 2:   Y Y N Y
2019-09-10 14:19:42.021361: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 3:   Y Y Y N
2019-09-10 14:19:42.029272: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 30438 MB memory) -> physical GPU (device: 0, name: Tesla V100-DGXS-32GB, pci bus id: 0000:07:00.0, compute capability: 7.0)
2019-09-10 14:19:42.031518: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 30463 MB memory) -> physical GPU (device: 1, name: Tesla V100-DGXS-32GB, pci bus id: 0000:08:00.0, compute capability: 7.0)
2019-09-10 14:19:42.033666: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:2 with 30463 MB memory) -> physical GPU (device: 2, name: Tesla V100-DGXS-32GB, pci bus id: 0000:0e:00.0, compute capability: 7.0)
2019-09-10 14:19:42.035762: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:3 with 30463 MB memory) -> physical GPU (device: 3, name: Tesla V100-DGXS-32GB, pci bus id: 0000:0f:00.0, compute capability: 7.0)
TensorFlow:  1.14
Model:       resnet50
Dataset:     imagenet (synthetic)
Mode:        training
SingleSess:  False
Batch size:  512 global
             512 per device
Num batches: 100
Num epochs:  0.04
Devices:     ['/gpu:0']
NUMA bind:   False
Data format: NHWC
Optimizer:   sgd
Variables:   replicated
AllReduce:   None
==========
W0910 14:19:42.042193 140478665234240 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2762: The name tf.set_random_seed is deprecated. Please use tf.compat.v1.set_random_seed instead.

Generating training model
W0910 14:19:42.042770 140478665234240 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2782: The name tf.train.get_or_create_global_step is deprecated. Please use tf.compat.v1.train.get_or_create_global_step instead.

W0910 14:19:42.050614 140478665234240 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2739: The name tf.get_variable is deprecated. Please use tf.compat.v1.get_variable instead.

W0910 14:19:42.055263 140478665234240 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/variable_mgr.py:316: The name tf.variable_scope is deprecated. Please use tf.compat.v1.variable_scope instead.

W0910 14:19:42.071847 140478665234240 deprecation.py:323] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:129: conv2d (from tensorflow.python.layers.convolutional) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.keras.layers.Conv2D` instead.
W0910 14:19:42.436394 140478665234240 deprecation.py:323] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:261: max_pooling2d (from tensorflow.python.layers.pooling) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.MaxPooling2D instead.
W0910 14:19:45.031168 140478665234240 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:334: The name tf.nn.xw_plus_b is deprecated. Please use tf.compat.v1.nn.xw_plus_b instead.

W0910 14:19:45.034001 140478665234240 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/models/model.py:319: The name tf.losses.sparse_softmax_cross_entropy is deprecated. Please use tf.compat.v1.losses.sparse_softmax_cross_entropy instead.

W0910 14:19:45.045301 140478665234240 deprecation.py:323] From /usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/losses/losses_impl.py:121: add_dispatch_support.<locals>.wrapper (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.where in 2.0, which has the same broadcast rule as np.where
W0910 14:19:46.350355 140478665234240 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/models/resnet_model.py:333: The name tf.train.piecewise_constant is deprecated. Please use tf.compat.v1.train.piecewise_constant instead.

W0910 14:19:46.434978 140478665234240 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:1209: The name tf.train.GradientDescentOptimizer is deprecated. Please use tf.compat.v1.train.GradientDescentOptimizer instead.

W0910 14:19:46.594228 140478665234240 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2090: The name tf.train.get_global_step is deprecated. Please use tf.compat.v1.train.get_global_step instead.

W0910 14:19:46.598637 140478665234240 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2126: The name tf.summary.merge_all is deprecated. Please use tf.compat.v1.summary.merge_all instead.

Initializing graph
W0910 14:19:46.598854 140478665234240 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2183: The name tf.train.Saver is deprecated. Please use tf.compat.v1.train.Saver instead.

W0910 14:19:46.938732 140478665234240 deprecation.py:323] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2238: Supervisor.__init__ (from tensorflow.python.training.supervisor) is deprecated and will be removed in a future version.
Instructions for updating:
Please switch to tf.train.MonitoredTrainingSession
2019-09-10 14:19:47.418791: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:07:00.0
2019-09-10 14:19:47.420354: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 1 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:08:00.0
2019-09-10 14:19:47.421862: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 2 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:0e:00.0
2019-09-10 14:19:47.423338: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 3 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:0f:00.0
2019-09-10 14:19:47.423366: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
2019-09-10 14:19:47.423401: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10
2019-09-10 14:19:47.423419: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcufft.so.10
2019-09-10 14:19:47.423436: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcurand.so.10
2019-09-10 14:19:47.423452: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusolver.so.10
2019-09-10 14:19:47.423470: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusparse.so.10
2019-09-10 14:19:47.423486: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
2019-09-10 14:19:47.435119: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0, 1, 2, 3
2019-09-10 14:19:47.435177: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1181] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-09-10 14:19:47.435188: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1187]      0 1 2 3
2019-09-10 14:19:47.435196: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 0:   N Y Y Y
2019-09-10 14:19:47.435203: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 1:   Y N Y Y
2019-09-10 14:19:47.435210: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 2:   Y Y N Y
2019-09-10 14:19:47.435217: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 3:   Y Y Y N
2019-09-10 14:19:47.443014: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 30438 MB memory) -> physical GPU (device: 0, name: Tesla V100-DGXS-32GB, pci bus id: 0000:07:00.0, compute capability: 7.0)
2019-09-10 14:19:47.444598: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 30463 MB memory) -> physical GPU (device: 1, name: Tesla V100-DGXS-32GB, pci bus id: 0000:08:00.0, compute capability: 7.0)
2019-09-10 14:19:47.446170: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:2 with 30463 MB memory) -> physical GPU (device: 2, name: Tesla V100-DGXS-32GB, pci bus id: 0000:0e:00.0, compute capability: 7.0)
2019-09-10 14:19:47.447750: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:3 with 30463 MB memory) -> physical GPU (device: 3, name: Tesla V100-DGXS-32GB, pci bus id: 0000:0f:00.0, compute capability: 7.0)
2019-09-10 14:19:47.989868: W tensorflow/compiler/jit/mark_for_compilation_pass.cc:1412] (One-time warning): Not using XLA:CPU for cluster because envvar TF_XLA_FLAGS=--tf_xla_cpu_global_jit was not set.  If you want XLA:CPU, either set that envvar, or use experimental_jit_scope to enable XLA:CPU.  To confirm that XLA is active, pass --vmodule=xla_compilation_cache=1 (as a proper command-line flag, not via TF_XLA_FLAGS) or set the envvar XLA_FLAGS=--xla_hlo_profile.
I0910 14:19:48.486477 140478665234240 session_manager.py:500] Running local_init_op.
I0910 14:19:48.541121 140478665234240 session_manager.py:502] Done running local_init_op.
Running warm up
2019-09-10 14:19:50.368899: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10
2019-09-10 14:19:50.702323: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
Done warm up
Step    Img/sec total_loss
1       images/sec: 848.7 +/- 0.0 (jitter = 0.0)        7.887
10      images/sec: 848.4 +/- 0.2 (jitter = 0.4)        7.908
20      images/sec: 848.4 +/- 0.2 (jitter = 0.8)        7.870
30      images/sec: 848.4 +/- 0.1 (jitter = 0.8)        7.917
40      images/sec: 848.4 +/- 0.1 (jitter = 0.8)        7.860
50      images/sec: 848.4 +/- 0.1 (jitter = 0.8)        7.763
60      images/sec: 848.4 +/- 0.1 (jitter = 0.8)        7.756
70      images/sec: 848.4 +/- 0.1 (jitter = 0.8)        7.749
80      images/sec: 848.3 +/- 0.1 (jitter = 0.8)        7.755
90      images/sec: 848.3 +/- 0.1 (jitter = 0.8)        7.722
100     images/sec: 848.3 +/- 0.1 (jitter = 0.8)        7.713
----------------------------------------------------------------
total images/sec: 848.17
----------------------------------------------------------------
2019-09-10 14:21:13.007125: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
WARNING: Logging before flag parsing goes to stderr.
W0910 14:21:15.035794 140578430060352 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/models/mobilenet.py:388: The name tf.nn.avg_pool is deprecated. Please use tf.nn.avg_pool2d instead.

W0910 14:21:15.040489 140578430060352 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/models/experimental/deepspeech.py:131: The name tf.nn.rnn_cell.RNNCell is deprecated. Please use tf.compat.v1.nn.rnn_cell.RNNCell instead.

W0910 14:21:15.064074 140578430060352 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:722: The name tf.ConfigProto is deprecated. Please use tf.compat.v1.ConfigProto instead.

W0910 14:21:15.064295 140578430060352 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:3503: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead.

2019-09-10 14:21:15.088711: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2198780000 Hz
2019-09-10 14:21:15.091283: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x2faa950 executing computations on platform Host. Devices:
2019-09-10 14:21:15.091317: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): <undefined>, <undefined>
2019-09-10 14:21:15.094614: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcuda.so.1
2019-09-10 14:21:16.102762: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x2faae60 executing computations on platform CUDA. Devices:
2019-09-10 14:21:16.102801: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): Tesla V100-DGXS-32GB, Compute Capability 7.0
2019-09-10 14:21:16.102816: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (1): Tesla V100-DGXS-32GB, Compute Capability 7.0
2019-09-10 14:21:16.102826: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (2): Tesla V100-DGXS-32GB, Compute Capability 7.0
2019-09-10 14:21:16.102836: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (3): Tesla V100-DGXS-32GB, Compute Capability 7.0
2019-09-10 14:21:16.105699: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:07:00.0
2019-09-10 14:21:16.108100: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 1 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:08:00.0
2019-09-10 14:21:16.110464: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 2 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:0e:00.0
2019-09-10 14:21:16.112826: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 3 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:0f:00.0
2019-09-10 14:21:16.112869: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
2019-09-10 14:21:16.115577: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10
2019-09-10 14:21:16.117557: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcufft.so.10
2019-09-10 14:21:16.117852: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcurand.so.10
2019-09-10 14:21:16.119583: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusolver.so.10
2019-09-10 14:21:16.120583: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusparse.so.10
2019-09-10 14:21:16.124216: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
2019-09-10 14:21:16.136173: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0, 1, 2, 3
2019-09-10 14:21:16.136209: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
2019-09-10 14:21:17.951546: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1181] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-09-10 14:21:17.951583: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1187]      0 1 2 3
2019-09-10 14:21:17.951594: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 0:   N Y Y Y
2019-09-10 14:21:17.951600: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 1:   Y N Y Y
2019-09-10 14:21:17.951607: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 2:   Y Y N Y
2019-09-10 14:21:17.951613: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 3:   Y Y Y N
2019-09-10 14:21:17.959628: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 30438 MB memory) -> physical GPU (device: 0, name: Tesla V100-DGXS-32GB, pci bus id: 0000:07:00.0, compute capability: 7.0)
2019-09-10 14:21:17.961828: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 30463 MB memory) -> physical GPU (device: 1, name: Tesla V100-DGXS-32GB, pci bus id: 0000:08:00.0, compute capability: 7.0)
2019-09-10 14:21:17.964004: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:2 with 30463 MB memory) -> physical GPU (device: 2, name: Tesla V100-DGXS-32GB, pci bus id: 0000:0e:00.0, compute capability: 7.0)
2019-09-10 14:21:17.966159: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:3 with 30463 MB memory) -> physical GPU (device: 3, name: Tesla V100-DGXS-32GB, pci bus id: 0000:0f:00.0, compute capability: 7.0)
TensorFlow:  1.14
Model:       resnet50
Dataset:     imagenet (synthetic)
Mode:        training
SingleSess:  False
Batch size:  128 global
             64 per device
Num batches: 100
Num epochs:  0.01
Devices:     ['/gpu:0', '/gpu:1']
NUMA bind:   False
Data format: NHWC
Optimizer:   sgd
Variables:   replicated
AllReduce:   None
==========
W0910 14:21:17.972665 140578430060352 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2762: The name tf.set_random_seed is deprecated. Please use tf.compat.v1.set_random_seed instead.

Generating training model
W0910 14:21:17.973247 140578430060352 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2782: The name tf.train.get_or_create_global_step is deprecated. Please use tf.compat.v1.train.get_or_create_global_step instead.

W0910 14:21:17.981930 140578430060352 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/variable_mgr.py:316: The name tf.variable_scope is deprecated. Please use tf.compat.v1.variable_scope instead.

W0910 14:21:17.982311 140578430060352 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/models/model.py:245: The name tf.truncated_normal is deprecated. Please use tf.random.truncated_normal instead.

W0910 14:21:17.998647 140578430060352 deprecation.py:323] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:129: conv2d (from tensorflow.python.layers.convolutional) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.keras.layers.Conv2D` instead.
W0910 14:21:18.360758 140578430060352 deprecation.py:323] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:261: max_pooling2d (from tensorflow.python.layers.pooling) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.MaxPooling2D instead.
W0910 14:21:20.863417 140578430060352 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:334: The name tf.nn.xw_plus_b is deprecated. Please use tf.compat.v1.nn.xw_plus_b instead.

W0910 14:21:20.865456 140578430060352 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/models/model.py:319: The name tf.losses.sparse_softmax_cross_entropy is deprecated. Please use tf.compat.v1.losses.sparse_softmax_cross_entropy instead.

W0910 14:21:20.876849 140578430060352 deprecation.py:323] From /usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/losses/losses_impl.py:121: add_dispatch_support.<locals>.wrapper (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.where in 2.0, which has the same broadcast rule as np.where
W0910 14:21:24.969537 140578430060352 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/models/resnet_model.py:333: The name tf.train.piecewise_constant is deprecated. Please use tf.compat.v1.train.piecewise_constant instead.

W0910 14:21:25.055859 140578430060352 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:1209: The name tf.train.GradientDescentOptimizer is deprecated. Please use tf.compat.v1.train.GradientDescentOptimizer instead.

W0910 14:21:25.462713 140578430060352 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2090: The name tf.train.get_global_step is deprecated. Please use tf.compat.v1.train.get_global_step instead.

W0910 14:21:25.838522 140578430060352 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2126: The name tf.summary.merge_all is deprecated. Please use tf.compat.v1.summary.merge_all instead.

Initializing graph
W0910 14:21:25.838754 140578430060352 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2183: The name tf.train.Saver is deprecated. Please use tf.compat.v1.train.Saver instead.

W0910 14:21:26.094898 140578430060352 deprecation.py:323] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2238: Supervisor.__init__ (from tensorflow.python.training.supervisor) is deprecated and will be removed in a future version.
Instructions for updating:
Please switch to tf.train.MonitoredTrainingSession
2019-09-10 14:21:26.974114: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:07:00.0
2019-09-10 14:21:26.975641: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 1 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:08:00.0
2019-09-10 14:21:26.977147: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 2 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:0e:00.0
2019-09-10 14:21:26.978629: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 3 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:0f:00.0
2019-09-10 14:21:26.978658: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
2019-09-10 14:21:26.978692: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10
2019-09-10 14:21:26.978709: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcufft.so.10
2019-09-10 14:21:26.978726: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcurand.so.10
2019-09-10 14:21:26.978742: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusolver.so.10
2019-09-10 14:21:26.978759: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusparse.so.10
2019-09-10 14:21:26.978776: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
2019-09-10 14:21:26.990482: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0, 1, 2, 3
2019-09-10 14:21:26.990539: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1181] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-09-10 14:21:26.990549: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1187]      0 1 2 3
2019-09-10 14:21:26.990557: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 0:   N Y Y Y
2019-09-10 14:21:26.990563: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 1:   Y N Y Y
2019-09-10 14:21:26.990569: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 2:   Y Y N Y
2019-09-10 14:21:26.990576: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 3:   Y Y Y N
2019-09-10 14:21:26.998265: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 30438 MB memory) -> physical GPU (device: 0, name: Tesla V100-DGXS-32GB, pci bus id: 0000:07:00.0, compute capability: 7.0)
2019-09-10 14:21:26.999836: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 30463 MB memory) -> physical GPU (device: 1, name: Tesla V100-DGXS-32GB, pci bus id: 0000:08:00.0, compute capability: 7.0)
2019-09-10 14:21:27.001408: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:2 with 30463 MB memory) -> physical GPU (device: 2, name: Tesla V100-DGXS-32GB, pci bus id: 0000:0e:00.0, compute capability: 7.0)
2019-09-10 14:21:27.002992: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:3 with 30463 MB memory) -> physical GPU (device: 3, name: Tesla V100-DGXS-32GB, pci bus id: 0000:0f:00.0, compute capability: 7.0)
2019-09-10 14:21:28.072702: W tensorflow/compiler/jit/mark_for_compilation_pass.cc:1412] (One-time warning): Not using XLA:CPU for cluster because envvar TF_XLA_FLAGS=--tf_xla_cpu_global_jit was not set.  If you want XLA:CPU, either set that envvar, or use experimental_jit_scope to enable XLA:CPU.  To confirm that XLA is active, pass --vmodule=xla_compilation_cache=1 (as a proper command-line flag, not via TF_XLA_FLAGS) or set the envvar XLA_FLAGS=--xla_hlo_profile.
I0910 14:21:30.050031 140578430060352 session_manager.py:500] Running local_init_op.
I0910 14:21:30.640582 140578430060352 session_manager.py:502] Done running local_init_op.
Running warm up
2019-09-10 14:21:33.312138: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10
2019-09-10 14:21:33.985353: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
Done warm up
Step    Img/sec total_loss
1       images/sec: 730.6 +/- 0.0 (jitter = 0.0)        7.687
10      images/sec: 731.9 +/- 0.4 (jitter = 1.6)        8.002
20      images/sec: 732.0 +/- 0.2 (jitter = 1.3)        7.834
30      images/sec: 731.8 +/- 0.2 (jitter = 1.6)        7.958
40      images/sec: 731.6 +/- 0.2 (jitter = 1.7)        7.925
50      images/sec: 731.9 +/- 0.2 (jitter = 1.8)        7.717
60      images/sec: 732.2 +/- 0.2 (jitter = 1.5)        7.856
70      images/sec: 732.5 +/- 0.2 (jitter = 1.7)        8.079
80      images/sec: 732.7 +/- 0.2 (jitter = 1.7)        7.821
90      images/sec: 732.6 +/- 0.2 (jitter = 1.7)        7.871
100     images/sec: 732.6 +/- 0.2 (jitter = 1.6)        7.881
----------------------------------------------------------------
total images/sec: 732.26
----------------------------------------------------------------
2019-09-10 14:21:59.850233: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
WARNING: Logging before flag parsing goes to stderr.
W0910 14:22:01.857193 139913248937792 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/models/mobilenet.py:388: The name tf.nn.avg_pool is deprecated. Please use tf.nn.avg_pool2d instead.

W0910 14:22:01.861687 139913248937792 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/models/experimental/deepspeech.py:131: The name tf.nn.rnn_cell.RNNCell is deprecated. Please use tf.compat.v1.nn.rnn_cell.RNNCell instead.

W0910 14:22:01.885066 139913248937792 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:722: The name tf.ConfigProto is deprecated. Please use tf.compat.v1.ConfigProto instead.

W0910 14:22:01.885278 139913248937792 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:3503: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead.

2019-09-10 14:22:01.908719: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2198780000 Hz
2019-09-10 14:22:01.911279: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x2fe9c20 executing computations on platform Host. Devices:
2019-09-10 14:22:01.911312: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): <undefined>, <undefined>
2019-09-10 14:22:01.914664: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcuda.so.1
2019-09-10 14:22:02.791802: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x4860330 executing computations on platform CUDA. Devices:
2019-09-10 14:22:02.791838: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): Tesla V100-DGXS-32GB, Compute Capability 7.0
2019-09-10 14:22:02.791849: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (1): Tesla V100-DGXS-32GB, Compute Capability 7.0
2019-09-10 14:22:02.791857: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (2): Tesla V100-DGXS-32GB, Compute Capability 7.0
2019-09-10 14:22:02.791866: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (3): Tesla V100-DGXS-32GB, Compute Capability 7.0
2019-09-10 14:22:02.794607: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:07:00.0
2019-09-10 14:22:02.796978: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 1 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:08:00.0
2019-09-10 14:22:02.799343: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 2 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:0e:00.0
2019-09-10 14:22:02.801707: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 3 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:0f:00.0
2019-09-10 14:22:02.801749: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
2019-09-10 14:22:02.804423: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10
2019-09-10 14:22:02.805996: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcufft.so.10
2019-09-10 14:22:02.806274: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcurand.so.10
2019-09-10 14:22:02.807968: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusolver.so.10
2019-09-10 14:22:02.808961: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusparse.so.10
2019-09-10 14:22:02.812512: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
2019-09-10 14:22:02.824369: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0, 1, 2, 3
2019-09-10 14:22:02.824403: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
2019-09-10 14:22:04.600829: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1181] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-09-10 14:22:04.600869: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1187]      0 1 2 3
2019-09-10 14:22:04.600880: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 0:   N Y Y Y
2019-09-10 14:22:04.600887: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 1:   Y N Y Y
2019-09-10 14:22:04.600893: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 2:   Y Y N Y
2019-09-10 14:22:04.600899: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 3:   Y Y Y N
2019-09-10 14:22:04.608958: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 30438 MB memory) -> physical GPU (device: 0, name: Tesla V100-DGXS-32GB, pci bus id: 0000:07:00.0, compute capability: 7.0)
2019-09-10 14:22:04.611161: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 30463 MB memory) -> physical GPU (device: 1, name: Tesla V100-DGXS-32GB, pci bus id: 0000:08:00.0, compute capability: 7.0)
2019-09-10 14:22:04.613269: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:2 with 30463 MB memory) -> physical GPU (device: 2, name: Tesla V100-DGXS-32GB, pci bus id: 0000:0e:00.0, compute capability: 7.0)
2019-09-10 14:22:04.615392: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:3 with 30463 MB memory) -> physical GPU (device: 3, name: Tesla V100-DGXS-32GB, pci bus id: 0000:0f:00.0, compute capability: 7.0)
TensorFlow:  1.14
Model:       resnet50
Dataset:     imagenet (synthetic)
Mode:        training
SingleSess:  False
Batch size:  128 global
             64 per device
Num batches: 100
Num epochs:  0.01
Devices:     ['/gpu:0', '/gpu:1']
NUMA bind:   False
Data format: NHWC
Optimizer:   sgd
Variables:   replicated
AllReduce:   None
==========
W0910 14:22:04.621623 139913248937792 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2762: The name tf.set_random_seed is deprecated. Please use tf.compat.v1.set_random_seed instead.

Generating training model
W0910 14:22:04.622202 139913248937792 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2782: The name tf.train.get_or_create_global_step is deprecated. Please use tf.compat.v1.train.get_or_create_global_step instead.

W0910 14:22:04.630498 139913248937792 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2739: The name tf.get_variable is deprecated. Please use tf.compat.v1.get_variable instead.

W0910 14:22:04.635244 139913248937792 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/variable_mgr.py:316: The name tf.variable_scope is deprecated. Please use tf.compat.v1.variable_scope instead.

W0910 14:22:04.652157 139913248937792 deprecation.py:323] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:129: conv2d (from tensorflow.python.layers.convolutional) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.keras.layers.Conv2D` instead.
W0910 14:22:05.017952 139913248937792 deprecation.py:323] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:261: max_pooling2d (from tensorflow.python.layers.pooling) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.MaxPooling2D instead.
W0910 14:22:07.683459 139913248937792 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:334: The name tf.nn.xw_plus_b is deprecated. Please use tf.compat.v1.nn.xw_plus_b instead.

W0910 14:22:07.686371 139913248937792 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/models/model.py:319: The name tf.losses.sparse_softmax_cross_entropy is deprecated. Please use tf.compat.v1.losses.sparse_softmax_cross_entropy instead.

W0910 14:22:07.698026 139913248937792 deprecation.py:323] From /usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/losses/losses_impl.py:121: add_dispatch_support.<locals>.wrapper (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.where in 2.0, which has the same broadcast rule as np.where
W0910 14:22:13.078341 139913248937792 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/models/resnet_model.py:333: The name tf.train.piecewise_constant is deprecated. Please use tf.compat.v1.train.piecewise_constant instead.

W0910 14:22:13.165047 139913248937792 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:1209: The name tf.train.GradientDescentOptimizer is deprecated. Please use tf.compat.v1.train.GradientDescentOptimizer instead.

W0910 14:22:13.582752 139913248937792 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2090: The name tf.train.get_global_step is deprecated. Please use tf.compat.v1.train.get_global_step instead.

W0910 14:22:13.967311 139913248937792 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2126: The name tf.summary.merge_all is deprecated. Please use tf.compat.v1.summary.merge_all instead.

Initializing graph
W0910 14:22:13.967549 139913248937792 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2183: The name tf.train.Saver is deprecated. Please use tf.compat.v1.train.Saver instead.

W0910 14:22:14.230364 139913248937792 deprecation.py:323] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2238: Supervisor.__init__ (from tensorflow.python.training.supervisor) is deprecated and will be removed in a future version.
Instructions for updating:
Please switch to tf.train.MonitoredTrainingSession
2019-09-10 14:22:15.338605: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:07:00.0
2019-09-10 14:22:15.340129: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 1 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:08:00.0
2019-09-10 14:22:15.341660: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 2 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:0e:00.0
2019-09-10 14:22:15.343177: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 3 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:0f:00.0
2019-09-10 14:22:15.343207: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
2019-09-10 14:22:15.343242: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10
2019-09-10 14:22:15.343260: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcufft.so.10
2019-09-10 14:22:15.343277: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcurand.so.10
2019-09-10 14:22:15.343293: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusolver.so.10
2019-09-10 14:22:15.343311: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusparse.so.10
2019-09-10 14:22:15.343330: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
2019-09-10 14:22:15.355167: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0, 1, 2, 3
2019-09-10 14:22:15.355228: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1181] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-09-10 14:22:15.355239: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1187]      0 1 2 3
2019-09-10 14:22:15.355248: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 0:   N Y Y Y
2019-09-10 14:22:15.355255: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 1:   Y N Y Y
2019-09-10 14:22:15.355263: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 2:   Y Y N Y
2019-09-10 14:22:15.355269: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 3:   Y Y Y N
2019-09-10 14:22:15.363077: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 30438 MB memory) -> physical GPU (device: 0, name: Tesla V100-DGXS-32GB, pci bus id: 0000:07:00.0, compute capability: 7.0)
2019-09-10 14:22:15.364692: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 30463 MB memory) -> physical GPU (device: 1, name: Tesla V100-DGXS-32GB, pci bus id: 0000:08:00.0, compute capability: 7.0)
2019-09-10 14:22:15.366287: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:2 with 30463 MB memory) -> physical GPU (device: 2, name: Tesla V100-DGXS-32GB, pci bus id: 0000:0e:00.0, compute capability: 7.0)
2019-09-10 14:22:15.367929: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:3 with 30463 MB memory) -> physical GPU (device: 3, name: Tesla V100-DGXS-32GB, pci bus id: 0000:0f:00.0, compute capability: 7.0)
2019-09-10 14:22:16.525356: W tensorflow/compiler/jit/mark_for_compilation_pass.cc:1412] (One-time warning): Not using XLA:CPU for cluster because envvar TF_XLA_FLAGS=--tf_xla_cpu_global_jit was not set.  If you want XLA:CPU, either set that envvar, or use experimental_jit_scope to enable XLA:CPU.  To confirm that XLA is active, pass --vmodule=xla_compilation_cache=1 (as a proper command-line flag, not via TF_XLA_FLAGS) or set the envvar XLA_FLAGS=--xla_hlo_profile.
I0910 14:22:18.515134 139913248937792 session_manager.py:500] Running local_init_op.
I0910 14:22:19.141375 139913248937792 session_manager.py:502] Done running local_init_op.
Running warm up
2019-09-10 14:22:22.505595: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10
2019-09-10 14:22:23.232971: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
Done warm up
Step    Img/sec total_loss
1       images/sec: 1296.0 +/- 0.0 (jitter = 0.0)       8.091
10      images/sec: 1303.0 +/- 2.5 (jitter = 10.1)      7.844
20      images/sec: 1292.5 +/- 2.8 (jitter = 14.6)      7.962
30      images/sec: 1290.0 +/- 2.1 (jitter = 11.9)      7.945
40      images/sec: 1288.5 +/- 1.7 (jitter = 9.4)       7.991
50      images/sec: 1287.1 +/- 1.5 (jitter = 9.0)       7.895
60      images/sec: 1286.7 +/- 1.3 (jitter = 8.1)       7.971
70      images/sec: 1286.3 +/- 1.1 (jitter = 7.1)       7.906
80      images/sec: 1286.1 +/- 1.0 (jitter = 7.0)       7.802
90      images/sec: 1286.3 +/- 0.9 (jitter = 6.8)       7.878
100     images/sec: 1286.5 +/- 0.8 (jitter = 7.8)       7.815
----------------------------------------------------------------
total images/sec: 1285.55
----------------------------------------------------------------
2019-09-10 14:22:40.788280: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
WARNING: Logging before flag parsing goes to stderr.
W0910 14:22:42.793020 140266530510656 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/models/mobilenet.py:388: The name tf.nn.avg_pool is deprecated. Please use tf.nn.avg_pool2d instead.

W0910 14:22:42.797582 140266530510656 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/models/experimental/deepspeech.py:131: The name tf.nn.rnn_cell.RNNCell is deprecated. Please use tf.compat.v1.nn.rnn_cell.RNNCell instead.

W0910 14:22:42.821213 140266530510656 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:722: The name tf.ConfigProto is deprecated. Please use tf.compat.v1.ConfigProto instead.

W0910 14:22:42.821424 140266530510656 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:3503: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead.

2019-09-10 14:22:42.844707: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2198780000 Hz
2019-09-10 14:22:42.847382: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x409e030 executing computations on platform Host. Devices:
2019-09-10 14:22:42.847423: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): <undefined>, <undefined>
2019-09-10 14:22:42.851444: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcuda.so.1
2019-09-10 14:22:43.789975: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x4097500 executing computations on platform CUDA. Devices:
2019-09-10 14:22:43.790014: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): Tesla V100-DGXS-32GB, Compute Capability 7.0
2019-09-10 14:22:43.790027: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (1): Tesla V100-DGXS-32GB, Compute Capability 7.0
2019-09-10 14:22:43.790038: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (2): Tesla V100-DGXS-32GB, Compute Capability 7.0
2019-09-10 14:22:43.790048: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (3): Tesla V100-DGXS-32GB, Compute Capability 7.0
2019-09-10 14:22:43.792929: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:07:00.0
2019-09-10 14:22:43.795356: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 1 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:08:00.0
2019-09-10 14:22:43.797544: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 2 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:0e:00.0
2019-09-10 14:22:43.799187: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 3 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:0f:00.0
2019-09-10 14:22:43.799217: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
2019-09-10 14:22:43.801062: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10
2019-09-10 14:22:43.802738: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcufft.so.10
2019-09-10 14:22:43.803064: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcurand.so.10
2019-09-10 14:22:43.804972: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusolver.so.10
2019-09-10 14:22:43.806084: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusparse.so.10
2019-09-10 14:22:43.809961: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
2019-09-10 14:22:43.821910: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0, 1, 2, 3
2019-09-10 14:22:43.821942: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
2019-09-10 14:22:45.589580: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1181] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-09-10 14:22:45.589622: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1187]      0 1 2 3
2019-09-10 14:22:45.589633: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 0:   N Y Y Y
2019-09-10 14:22:45.589640: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 1:   Y N Y Y
2019-09-10 14:22:45.589646: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 2:   Y Y N Y
2019-09-10 14:22:45.589652: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 3:   Y Y Y N
2019-09-10 14:22:45.597620: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 30438 MB memory) -> physical GPU (device: 0, name: Tesla V100-DGXS-32GB, pci bus id: 0000:07:00.0, compute capability: 7.0)
2019-09-10 14:22:45.599830: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 30463 MB memory) -> physical GPU (device: 1, name: Tesla V100-DGXS-32GB, pci bus id: 0000:08:00.0, compute capability: 7.0)
2019-09-10 14:22:45.601972: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:2 with 30463 MB memory) -> physical GPU (device: 2, name: Tesla V100-DGXS-32GB, pci bus id: 0000:0e:00.0, compute capability: 7.0)
2019-09-10 14:22:45.604103: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:3 with 30463 MB memory) -> physical GPU (device: 3, name: Tesla V100-DGXS-32GB, pci bus id: 0000:0f:00.0, compute capability: 7.0)
TensorFlow:  1.14
Model:       resnet50
Dataset:     imagenet (synthetic)
Mode:        training
SingleSess:  False
Batch size:  256 global
             128 per device
Num batches: 100
Num epochs:  0.02
Devices:     ['/gpu:0', '/gpu:1']
NUMA bind:   False
Data format: NHWC
Optimizer:   sgd
Variables:   replicated
AllReduce:   None
==========
W0910 14:22:45.610832 140266530510656 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2762: The name tf.set_random_seed is deprecated. Please use tf.compat.v1.set_random_seed instead.

Generating training model
W0910 14:22:45.611408 140266530510656 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2782: The name tf.train.get_or_create_global_step is deprecated. Please use tf.compat.v1.train.get_or_create_global_step instead.

W0910 14:22:45.619366 140266530510656 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/variable_mgr.py:316: The name tf.variable_scope is deprecated. Please use tf.compat.v1.variable_scope instead.

W0910 14:22:45.619744 140266530510656 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/models/model.py:245: The name tf.truncated_normal is deprecated. Please use tf.random.truncated_normal instead.

W0910 14:22:45.635990 140266530510656 deprecation.py:323] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:129: conv2d (from tensorflow.python.layers.convolutional) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.keras.layers.Conv2D` instead.
W0910 14:22:45.999307 140266530510656 deprecation.py:323] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:261: max_pooling2d (from tensorflow.python.layers.pooling) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.MaxPooling2D instead.
W0910 14:22:48.550598 140266530510656 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:334: The name tf.nn.xw_plus_b is deprecated. Please use tf.compat.v1.nn.xw_plus_b instead.

W0910 14:22:48.552660 140266530510656 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/models/model.py:319: The name tf.losses.sparse_softmax_cross_entropy is deprecated. Please use tf.compat.v1.losses.sparse_softmax_cross_entropy instead.

W0910 14:22:48.564251 140266530510656 deprecation.py:323] From /usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/losses/losses_impl.py:121: add_dispatch_support.<locals>.wrapper (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.where in 2.0, which has the same broadcast rule as np.where
W0910 14:22:52.737199 140266530510656 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/models/resnet_model.py:333: The name tf.train.piecewise_constant is deprecated. Please use tf.compat.v1.train.piecewise_constant instead.

W0910 14:22:52.825487 140266530510656 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:1209: The name tf.train.GradientDescentOptimizer is deprecated. Please use tf.compat.v1.train.GradientDescentOptimizer instead.

W0910 14:22:53.239958 140266530510656 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2090: The name tf.train.get_global_step is deprecated. Please use tf.compat.v1.train.get_global_step instead.

W0910 14:22:53.623955 140266530510656 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2126: The name tf.summary.merge_all is deprecated. Please use tf.compat.v1.summary.merge_all instead.

Initializing graph
W0910 14:22:53.624194 140266530510656 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2183: The name tf.train.Saver is deprecated. Please use tf.compat.v1.train.Saver instead.

W0910 14:22:53.885126 140266530510656 deprecation.py:323] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2238: Supervisor.__init__ (from tensorflow.python.training.supervisor) is deprecated and will be removed in a future version.
Instructions for updating:
Please switch to tf.train.MonitoredTrainingSession
2019-09-10 14:22:54.782751: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:07:00.0
2019-09-10 14:22:54.784293: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 1 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:08:00.0
2019-09-10 14:22:54.785785: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 2 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:0e:00.0
2019-09-10 14:22:54.787266: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 3 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:0f:00.0
2019-09-10 14:22:54.787296: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
2019-09-10 14:22:54.787331: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10
2019-09-10 14:22:54.787348: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcufft.so.10
2019-09-10 14:22:54.787366: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcurand.so.10
2019-09-10 14:22:54.787382: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusolver.so.10
2019-09-10 14:22:54.787397: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusparse.so.10
2019-09-10 14:22:54.787415: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
2019-09-10 14:22:54.799057: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0, 1, 2, 3
2019-09-10 14:22:54.799117: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1181] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-09-10 14:22:54.799127: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1187]      0 1 2 3
2019-09-10 14:22:54.799135: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 0:   N Y Y Y
2019-09-10 14:22:54.799141: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 1:   Y N Y Y
2019-09-10 14:22:54.799148: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 2:   Y Y N Y
2019-09-10 14:22:54.799154: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 3:   Y Y Y N
2019-09-10 14:22:54.806917: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 30438 MB memory) -> physical GPU (device: 0, name: Tesla V100-DGXS-32GB, pci bus id: 0000:07:00.0, compute capability: 7.0)
2019-09-10 14:22:54.808502: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 30463 MB memory) -> physical GPU (device: 1, name: Tesla V100-DGXS-32GB, pci bus id: 0000:08:00.0, compute capability: 7.0)
2019-09-10 14:22:54.810070: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:2 with 30463 MB memory) -> physical GPU (device: 2, name: Tesla V100-DGXS-32GB, pci bus id: 0000:0e:00.0, compute capability: 7.0)
2019-09-10 14:22:54.811651: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:3 with 30463 MB memory) -> physical GPU (device: 3, name: Tesla V100-DGXS-32GB, pci bus id: 0000:0f:00.0, compute capability: 7.0)
2019-09-10 14:22:55.907285: W tensorflow/compiler/jit/mark_for_compilation_pass.cc:1412] (One-time warning): Not using XLA:CPU for cluster because envvar TF_XLA_FLAGS=--tf_xla_cpu_global_jit was not set.  If you want XLA:CPU, either set that envvar, or use experimental_jit_scope to enable XLA:CPU.  To confirm that XLA is active, pass --vmodule=xla_compilation_cache=1 (as a proper command-line flag, not via TF_XLA_FLAGS) or set the envvar XLA_FLAGS=--xla_hlo_profile.
I0910 14:22:57.894693 140266530510656 session_manager.py:500] Running local_init_op.
I0910 14:22:58.504334 140266530510656 session_manager.py:502] Done running local_init_op.
Running warm up
2019-09-10 14:23:01.230805: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10
2019-09-10 14:23:01.905064: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
Done warm up
Step    Img/sec total_loss
1       images/sec: 784.8 +/- 0.0 (jitter = 0.0)        7.943
10      images/sec: 783.9 +/- 0.4 (jitter = 1.0)        7.891
20      images/sec: 783.4 +/- 0.3 (jitter = 1.5)        7.917
30      images/sec: 783.2 +/- 0.2 (jitter = 1.4)        7.786
40      images/sec: 782.9 +/- 0.2 (jitter = 1.4)        7.899
50      images/sec: 782.9 +/- 0.2 (jitter = 1.7)        7.873
60      images/sec: 782.7 +/- 0.2 (jitter = 1.5)        7.820
70      images/sec: 782.6 +/- 0.2 (jitter = 1.5)        7.785
80      images/sec: 782.5 +/- 0.2 (jitter = 1.5)        7.772
90      images/sec: 782.5 +/- 0.1 (jitter = 1.6)        7.847
100     images/sec: 782.4 +/- 0.1 (jitter = 1.6)        7.804
----------------------------------------------------------------
total images/sec: 782.23
----------------------------------------------------------------
2019-09-10 14:23:45.600726: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
WARNING: Logging before flag parsing goes to stderr.
W0910 14:23:47.584838 140504485586752 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/models/mobilenet.py:388: The name tf.nn.avg_pool is deprecated. Please use tf.nn.avg_pool2d instead.

W0910 14:23:47.589249 140504485586752 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/models/experimental/deepspeech.py:131: The name tf.nn.rnn_cell.RNNCell is deprecated. Please use tf.compat.v1.nn.rnn_cell.RNNCell instead.

W0910 14:23:47.612278 140504485586752 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:722: The name tf.ConfigProto is deprecated. Please use tf.compat.v1.ConfigProto instead.

W0910 14:23:47.612483 140504485586752 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:3503: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead.

2019-09-10 14:23:47.636707: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2198780000 Hz
2019-09-10 14:23:47.639307: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x3f0b020 executing computations on platform Host. Devices:
2019-09-10 14:23:47.639348: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): <undefined>, <undefined>
2019-09-10 14:23:47.643307: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcuda.so.1
2019-09-10 14:23:48.524085: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x3f044f0 executing computations on platform CUDA. Devices:
2019-09-10 14:23:48.524123: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): Tesla V100-DGXS-32GB, Compute Capability 7.0
2019-09-10 14:23:48.524135: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (1): Tesla V100-DGXS-32GB, Compute Capability 7.0
2019-09-10 14:23:48.524145: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (2): Tesla V100-DGXS-32GB, Compute Capability 7.0
2019-09-10 14:23:48.524154: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (3): Tesla V100-DGXS-32GB, Compute Capability 7.0
2019-09-10 14:23:48.526746: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:07:00.0
2019-09-10 14:23:48.528936: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 1 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:08:00.0
2019-09-10 14:23:48.531461: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 2 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:0e:00.0
2019-09-10 14:23:48.533992: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 3 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:0f:00.0
2019-09-10 14:23:48.534034: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
2019-09-10 14:23:48.536900: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10
2019-09-10 14:23:48.539422: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcufft.so.10
2019-09-10 14:23:48.539907: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcurand.so.10
2019-09-10 14:23:48.542071: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusolver.so.10
2019-09-10 14:23:48.543073: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusparse.so.10
2019-09-10 14:23:48.546688: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
2019-09-10 14:23:48.558710: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0, 1, 2, 3
2019-09-10 14:23:48.558744: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
2019-09-10 14:23:50.362620: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1181] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-09-10 14:23:50.362663: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1187]      0 1 2 3
2019-09-10 14:23:50.362673: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 0:   N Y Y Y
2019-09-10 14:23:50.362680: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 1:   Y N Y Y
2019-09-10 14:23:50.362686: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 2:   Y Y N Y
2019-09-10 14:23:50.362692: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 3:   Y Y Y N
2019-09-10 14:23:50.370660: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 30438 MB memory) -> physical GPU (device: 0, name: Tesla V100-DGXS-32GB, pci bus id: 0000:07:00.0, compute capability: 7.0)
2019-09-10 14:23:50.372905: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 30463 MB memory) -> physical GPU (device: 1, name: Tesla V100-DGXS-32GB, pci bus id: 0000:08:00.0, compute capability: 7.0)
2019-09-10 14:23:50.375087: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:2 with 30463 MB memory) -> physical GPU (device: 2, name: Tesla V100-DGXS-32GB, pci bus id: 0000:0e:00.0, compute capability: 7.0)
2019-09-10 14:23:50.377276: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:3 with 30463 MB memory) -> physical GPU (device: 3, name: Tesla V100-DGXS-32GB, pci bus id: 0000:0f:00.0, compute capability: 7.0)
TensorFlow:  1.14
Model:       resnet50
Dataset:     imagenet (synthetic)
Mode:        training
SingleSess:  False
Batch size:  256 global
             128 per device
Num batches: 100
Num epochs:  0.02
Devices:     ['/gpu:0', '/gpu:1']
NUMA bind:   False
Data format: NHWC
Optimizer:   sgd
Variables:   replicated
AllReduce:   None
==========
W0910 14:23:50.383536 140504485586752 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2762: The name tf.set_random_seed is deprecated. Please use tf.compat.v1.set_random_seed instead.

Generating training model
W0910 14:23:50.384116 140504485586752 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2782: The name tf.train.get_or_create_global_step is deprecated. Please use tf.compat.v1.train.get_or_create_global_step instead.

W0910 14:23:50.393350 140504485586752 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2739: The name tf.get_variable is deprecated. Please use tf.compat.v1.get_variable instead.

W0910 14:23:50.398173 140504485586752 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/variable_mgr.py:316: The name tf.variable_scope is deprecated. Please use tf.compat.v1.variable_scope instead.

W0910 14:23:50.415276 140504485586752 deprecation.py:323] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:129: conv2d (from tensorflow.python.layers.convolutional) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.keras.layers.Conv2D` instead.
W0910 14:23:50.782157 140504485586752 deprecation.py:323] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:261: max_pooling2d (from tensorflow.python.layers.pooling) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.MaxPooling2D instead.
W0910 14:23:53.464663 140504485586752 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:334: The name tf.nn.xw_plus_b is deprecated. Please use tf.compat.v1.nn.xw_plus_b instead.

W0910 14:23:53.467579 140504485586752 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/models/model.py:319: The name tf.losses.sparse_softmax_cross_entropy is deprecated. Please use tf.compat.v1.losses.sparse_softmax_cross_entropy instead.

W0910 14:23:53.479324 140504485586752 deprecation.py:323] From /usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/losses/losses_impl.py:121: add_dispatch_support.<locals>.wrapper (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.where in 2.0, which has the same broadcast rule as np.where
W0910 14:23:58.887962 140504485586752 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/models/resnet_model.py:333: The name tf.train.piecewise_constant is deprecated. Please use tf.compat.v1.train.piecewise_constant instead.

W0910 14:23:58.975307 140504485586752 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:1209: The name tf.train.GradientDescentOptimizer is deprecated. Please use tf.compat.v1.train.GradientDescentOptimizer instead.

W0910 14:23:59.395919 140504485586752 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2090: The name tf.train.get_global_step is deprecated. Please use tf.compat.v1.train.get_global_step instead.

W0910 14:23:59.782965 140504485586752 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2126: The name tf.summary.merge_all is deprecated. Please use tf.compat.v1.summary.merge_all instead.

Initializing graph
W0910 14:23:59.783206 140504485586752 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2183: The name tf.train.Saver is deprecated. Please use tf.compat.v1.train.Saver instead.

W0910 14:24:00.047021 140504485586752 deprecation.py:323] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2238: Supervisor.__init__ (from tensorflow.python.training.supervisor) is deprecated and will be removed in a future version.
Instructions for updating:
Please switch to tf.train.MonitoredTrainingSession
2019-09-10 14:24:01.156133: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:07:00.0
2019-09-10 14:24:01.157660: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 1 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:08:00.0
2019-09-10 14:24:01.159161: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 2 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:0e:00.0
2019-09-10 14:24:01.160681: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 3 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:0f:00.0
2019-09-10 14:24:01.160712: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
2019-09-10 14:24:01.160746: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10
2019-09-10 14:24:01.160765: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcufft.so.10
2019-09-10 14:24:01.160783: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcurand.so.10
2019-09-10 14:24:01.160802: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusolver.so.10
2019-09-10 14:24:01.160821: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusparse.so.10
2019-09-10 14:24:01.160840: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
2019-09-10 14:24:01.172536: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0, 1, 2, 3
2019-09-10 14:24:01.172601: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1181] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-09-10 14:24:01.172613: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1187]      0 1 2 3
2019-09-10 14:24:01.172622: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 0:   N Y Y Y
2019-09-10 14:24:01.172629: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 1:   Y N Y Y
2019-09-10 14:24:01.172636: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 2:   Y Y N Y
2019-09-10 14:24:01.172644: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 3:   Y Y Y N
2019-09-10 14:24:01.180462: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 30438 MB memory) -> physical GPU (device: 0, name: Tesla V100-DGXS-32GB, pci bus id: 0000:07:00.0, compute capability: 7.0)
2019-09-10 14:24:01.182089: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 30463 MB memory) -> physical GPU (device: 1, name: Tesla V100-DGXS-32GB, pci bus id: 0000:08:00.0, compute capability: 7.0)
2019-09-10 14:24:01.183680: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:2 with 30463 MB memory) -> physical GPU (device: 2, name: Tesla V100-DGXS-32GB, pci bus id: 0000:0e:00.0, compute capability: 7.0)
2019-09-10 14:24:01.185277: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:3 with 30463 MB memory) -> physical GPU (device: 3, name: Tesla V100-DGXS-32GB, pci bus id: 0000:0f:00.0, compute capability: 7.0)
2019-09-10 14:24:02.361322: W tensorflow/compiler/jit/mark_for_compilation_pass.cc:1412] (One-time warning): Not using XLA:CPU for cluster because envvar TF_XLA_FLAGS=--tf_xla_cpu_global_jit was not set.  If you want XLA:CPU, either set that envvar, or use experimental_jit_scope to enable XLA:CPU.  To confirm that XLA is active, pass --vmodule=xla_compilation_cache=1 (as a proper command-line flag, not via TF_XLA_FLAGS) or set the envvar XLA_FLAGS=--xla_hlo_profile.
I0910 14:24:04.354744 140504485586752 session_manager.py:500] Running local_init_op.
I0910 14:24:05.001707 140504485586752 session_manager.py:502] Done running local_init_op.
Running warm up
2019-09-10 14:24:08.421011: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10
2019-09-10 14:24:09.150870: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
Done warm up
Step    Img/sec total_loss
1       images/sec: 1509.3 +/- 0.0 (jitter = 0.0)       7.795
10      images/sec: 1503.9 +/- 1.4 (jitter = 5.6)       7.907
20      images/sec: 1500.6 +/- 1.1 (jitter = 4.7)       7.837
30      images/sec: 1499.8 +/- 0.9 (jitter = 4.9)       7.865
40      images/sec: 1499.1 +/- 0.8 (jitter = 4.7)       7.863
50      images/sec: 1499.1 +/- 0.7 (jitter = 4.1)       7.809
60      images/sec: 1499.1 +/- 0.6 (jitter = 4.1)       7.763
70      images/sec: 1499.5 +/- 0.6 (jitter = 4.6)       7.798
80      images/sec: 1499.2 +/- 0.5 (jitter = 4.6)       7.779
90      images/sec: 1499.0 +/- 0.5 (jitter = 4.2)       7.834
100     images/sec: 1498.7 +/- 0.5 (jitter = 4.5)       7.834
----------------------------------------------------------------
total images/sec: 1497.98
----------------------------------------------------------------
2019-09-10 14:24:35.550971: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
WARNING: Logging before flag parsing goes to stderr.
W0910 14:24:37.575564 140465520486208 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/models/mobilenet.py:388: The name tf.nn.avg_pool is deprecated. Please use tf.nn.avg_pool2d instead.

W0910 14:24:37.580106 140465520486208 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/models/experimental/deepspeech.py:131: The name tf.nn.rnn_cell.RNNCell is deprecated. Please use tf.compat.v1.nn.rnn_cell.RNNCell instead.

W0910 14:24:37.603518 140465520486208 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:722: The name tf.ConfigProto is deprecated. Please use tf.compat.v1.ConfigProto instead.

W0910 14:24:37.603733 140465520486208 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:3503: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead.

2019-09-10 14:24:37.628730: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2198780000 Hz
2019-09-10 14:24:37.631293: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x2ccbe40 executing computations on platform Host. Devices:
2019-09-10 14:24:37.631330: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): <undefined>, <undefined>
2019-09-10 14:24:37.634484: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcuda.so.1
2019-09-10 14:24:38.429684: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x2cc54c0 executing computations on platform CUDA. Devices:
2019-09-10 14:24:38.429719: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): Tesla V100-DGXS-32GB, Compute Capability 7.0
2019-09-10 14:24:38.429731: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (1): Tesla V100-DGXS-32GB, Compute Capability 7.0
2019-09-10 14:24:38.429740: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (2): Tesla V100-DGXS-32GB, Compute Capability 7.0
2019-09-10 14:24:38.429748: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (3): Tesla V100-DGXS-32GB, Compute Capability 7.0
2019-09-10 14:24:38.432086: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:07:00.0
2019-09-10 14:24:38.434270: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 1 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:08:00.0
2019-09-10 14:24:38.436478: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 2 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:0e:00.0
2019-09-10 14:24:38.438702: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 3 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:0f:00.0
2019-09-10 14:24:38.438739: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
2019-09-10 14:24:38.441257: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10
2019-09-10 14:24:38.443465: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcufft.so.10
2019-09-10 14:24:38.443889: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcurand.so.10
2019-09-10 14:24:38.445934: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusolver.so.10
2019-09-10 14:24:38.446905: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusparse.so.10
2019-09-10 14:24:38.450450: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
2019-09-10 14:24:38.462229: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0, 1, 2, 3
2019-09-10 14:24:38.462262: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
2019-09-10 14:24:40.243775: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1181] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-09-10 14:24:40.243817: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1187]      0 1 2 3
2019-09-10 14:24:40.243830: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 0:   N Y Y Y
2019-09-10 14:24:40.243837: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 1:   Y N Y Y
2019-09-10 14:24:40.243842: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 2:   Y Y N Y
2019-09-10 14:24:40.243848: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 3:   Y Y Y N
2019-09-10 14:24:40.251920: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 30438 MB memory) -> physical GPU (device: 0, name: Tesla V100-DGXS-32GB, pci bus id: 0000:07:00.0, compute capability: 7.0)
2019-09-10 14:24:40.254154: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 30463 MB memory) -> physical GPU (device: 1, name: Tesla V100-DGXS-32GB, pci bus id: 0000:08:00.0, compute capability: 7.0)
2019-09-10 14:24:40.256281: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:2 with 30463 MB memory) -> physical GPU (device: 2, name: Tesla V100-DGXS-32GB, pci bus id: 0000:0e:00.0, compute capability: 7.0)
2019-09-10 14:24:40.258410: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:3 with 30463 MB memory) -> physical GPU (device: 3, name: Tesla V100-DGXS-32GB, pci bus id: 0000:0f:00.0, compute capability: 7.0)
TensorFlow:  1.14
Model:       resnet50
Dataset:     imagenet (synthetic)
Mode:        training
SingleSess:  False
Batch size:  512 global
             256 per device
Num batches: 100
Num epochs:  0.04
Devices:     ['/gpu:0', '/gpu:1']
NUMA bind:   False
Data format: NHWC
Optimizer:   sgd
Variables:   replicated
AllReduce:   None
==========
W0910 14:24:40.264776 140465520486208 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2762: The name tf.set_random_seed is deprecated. Please use tf.compat.v1.set_random_seed instead.

Generating training model
W0910 14:24:40.265367 140465520486208 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2782: The name tf.train.get_or_create_global_step is deprecated. Please use tf.compat.v1.train.get_or_create_global_step instead.

W0910 14:24:40.274071 140465520486208 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/variable_mgr.py:316: The name tf.variable_scope is deprecated. Please use tf.compat.v1.variable_scope instead.

W0910 14:24:40.274465 140465520486208 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/models/model.py:245: The name tf.truncated_normal is deprecated. Please use tf.random.truncated_normal instead.

W0910 14:24:40.291061 140465520486208 deprecation.py:323] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:129: conv2d (from tensorflow.python.layers.convolutional) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.keras.layers.Conv2D` instead.
W0910 14:24:40.660473 140465520486208 deprecation.py:323] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:261: max_pooling2d (from tensorflow.python.layers.pooling) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.MaxPooling2D instead.
W0910 14:24:43.193747 140465520486208 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:334: The name tf.nn.xw_plus_b is deprecated. Please use tf.compat.v1.nn.xw_plus_b instead.

W0910 14:24:43.195797 140465520486208 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/models/model.py:319: The name tf.losses.sparse_softmax_cross_entropy is deprecated. Please use tf.compat.v1.losses.sparse_softmax_cross_entropy instead.

W0910 14:24:43.207361 140465520486208 deprecation.py:323] From /usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/losses/losses_impl.py:121: add_dispatch_support.<locals>.wrapper (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.where in 2.0, which has the same broadcast rule as np.where
W0910 14:24:47.347999 140465520486208 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/models/resnet_model.py:333: The name tf.train.piecewise_constant is deprecated. Please use tf.compat.v1.train.piecewise_constant instead.

W0910 14:24:47.435391 140465520486208 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:1209: The name tf.train.GradientDescentOptimizer is deprecated. Please use tf.compat.v1.train.GradientDescentOptimizer instead.

W0910 14:24:47.846177 140465520486208 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2090: The name tf.train.get_global_step is deprecated. Please use tf.compat.v1.train.get_global_step instead.

W0910 14:24:48.224692 140465520486208 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2126: The name tf.summary.merge_all is deprecated. Please use tf.compat.v1.summary.merge_all instead.

Initializing graph
W0910 14:24:48.224929 140465520486208 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2183: The name tf.train.Saver is deprecated. Please use tf.compat.v1.train.Saver instead.

W0910 14:24:48.483238 140465520486208 deprecation.py:323] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2238: Supervisor.__init__ (from tensorflow.python.training.supervisor) is deprecated and will be removed in a future version.
Instructions for updating:
Please switch to tf.train.MonitoredTrainingSession
2019-09-10 14:24:49.371149: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:07:00.0
2019-09-10 14:24:49.372665: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 1 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:08:00.0
2019-09-10 14:24:49.374149: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 2 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:0e:00.0
2019-09-10 14:24:49.375634: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 3 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:0f:00.0
2019-09-10 14:24:49.375666: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
2019-09-10 14:24:49.375707: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10
2019-09-10 14:24:49.375728: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcufft.so.10
2019-09-10 14:24:49.375748: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcurand.so.10
2019-09-10 14:24:49.375768: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusolver.so.10
2019-09-10 14:24:49.375787: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusparse.so.10
2019-09-10 14:24:49.375807: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
2019-09-10 14:24:49.387674: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0, 1, 2, 3
2019-09-10 14:24:49.387733: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1181] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-09-10 14:24:49.387744: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1187]      0 1 2 3
2019-09-10 14:24:49.387753: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 0:   N Y Y Y
2019-09-10 14:24:49.387760: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 1:   Y N Y Y
2019-09-10 14:24:49.387767: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 2:   Y Y N Y
2019-09-10 14:24:49.387774: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 3:   Y Y Y N
2019-09-10 14:24:49.395656: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 30438 MB memory) -> physical GPU (device: 0, name: Tesla V100-DGXS-32GB, pci bus id: 0000:07:00.0, compute capability: 7.0)
2019-09-10 14:24:49.397259: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 30463 MB memory) -> physical GPU (device: 1, name: Tesla V100-DGXS-32GB, pci bus id: 0000:08:00.0, compute capability: 7.0)
2019-09-10 14:24:49.398853: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:2 with 30463 MB memory) -> physical GPU (device: 2, name: Tesla V100-DGXS-32GB, pci bus id: 0000:0e:00.0, compute capability: 7.0)
2019-09-10 14:24:49.400477: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:3 with 30463 MB memory) -> physical GPU (device: 3, name: Tesla V100-DGXS-32GB, pci bus id: 0000:0f:00.0, compute capability: 7.0)
2019-09-10 14:24:50.491537: W tensorflow/compiler/jit/mark_for_compilation_pass.cc:1412] (One-time warning): Not using XLA:CPU for cluster because envvar TF_XLA_FLAGS=--tf_xla_cpu_global_jit was not set.  If you want XLA:CPU, either set that envvar, or use experimental_jit_scope to enable XLA:CPU.  To confirm that XLA is active, pass --vmodule=xla_compilation_cache=1 (as a proper command-line flag, not via TF_XLA_FLAGS) or set the envvar XLA_FLAGS=--xla_hlo_profile.
I0910 14:24:52.478892 140465520486208 session_manager.py:500] Running local_init_op.
I0910 14:24:53.103343 140465520486208 session_manager.py:502] Done running local_init_op.
Running warm up
2019-09-10 14:24:55.816936: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10
2019-09-10 14:24:56.494341: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
Done warm up
Step    Img/sec total_loss
1       images/sec: 805.3 +/- 0.0 (jitter = 0.0)        7.936
10      images/sec: 805.0 +/- 0.5 (jitter = 0.6)        7.861
20      images/sec: 804.3 +/- 0.4 (jitter = 1.2)        7.871
30      images/sec: 803.9 +/- 0.3 (jitter = 1.5)        7.901
40      images/sec: 803.9 +/- 0.2 (jitter = 1.4)        7.808
50      images/sec: 803.9 +/- 0.2 (jitter = 1.4)        7.802
60      images/sec: 803.6 +/- 0.2 (jitter = 1.6)        7.736
70      images/sec: 803.5 +/- 0.2 (jitter = 1.6)        7.754
80      images/sec: 803.4 +/- 0.2 (jitter = 1.5)        7.776
90      images/sec: 803.4 +/- 0.2 (jitter = 1.3)        7.701
100     images/sec: 803.3 +/- 0.1 (jitter = 1.5)        7.707
----------------------------------------------------------------
total images/sec: 803.21
----------------------------------------------------------------
2019-09-10 14:26:17.101463: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
WARNING: Logging before flag parsing goes to stderr.
W0910 14:26:19.133256 140219604993856 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/models/mobilenet.py:388: The name tf.nn.avg_pool is deprecated. Please use tf.nn.avg_pool2d instead.

W0910 14:26:19.137797 140219604993856 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/models/experimental/deepspeech.py:131: The name tf.nn.rnn_cell.RNNCell is deprecated. Please use tf.compat.v1.nn.rnn_cell.RNNCell instead.

W0910 14:26:19.161279 140219604993856 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:722: The name tf.ConfigProto is deprecated. Please use tf.compat.v1.ConfigProto instead.

W0910 14:26:19.161490 140219604993856 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:3503: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead.

2019-09-10 14:26:19.184711: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2198780000 Hz
2019-09-10 14:26:19.187359: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x30bfa80 executing computations on platform Host. Devices:
2019-09-10 14:26:19.187390: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): <undefined>, <undefined>
2019-09-10 14:26:19.190534: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcuda.so.1
2019-09-10 14:26:20.017661: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x30b9190 executing computations on platform CUDA. Devices:
2019-09-10 14:26:20.017707: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): Tesla V100-DGXS-32GB, Compute Capability 7.0
2019-09-10 14:26:20.017723: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (1): Tesla V100-DGXS-32GB, Compute Capability 7.0
2019-09-10 14:26:20.017736: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (2): Tesla V100-DGXS-32GB, Compute Capability 7.0
2019-09-10 14:26:20.017748: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (3): Tesla V100-DGXS-32GB, Compute Capability 7.0
2019-09-10 14:26:20.021181: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:07:00.0
2019-09-10 14:26:20.024024: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 1 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:08:00.0
2019-09-10 14:26:20.026878: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 2 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:0e:00.0
2019-09-10 14:26:20.029698: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 3 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:0f:00.0
2019-09-10 14:26:20.029745: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
2019-09-10 14:26:20.032875: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10
2019-09-10 14:26:20.034383: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcufft.so.10
2019-09-10 14:26:20.034665: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcurand.so.10
2019-09-10 14:26:20.036394: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusolver.so.10
2019-09-10 14:26:20.037393: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusparse.so.10
2019-09-10 14:26:20.041017: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
2019-09-10 14:26:20.052905: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0, 1, 2, 3
2019-09-10 14:26:20.052940: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
2019-09-10 14:26:21.860352: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1181] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-09-10 14:26:21.860392: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1187]      0 1 2 3
2019-09-10 14:26:21.860403: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 0:   N Y Y Y
2019-09-10 14:26:21.860410: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 1:   Y N Y Y
2019-09-10 14:26:21.860416: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 2:   Y Y N Y
2019-09-10 14:26:21.860422: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 3:   Y Y Y N
2019-09-10 14:26:21.868529: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 30438 MB memory) -> physical GPU (device: 0, name: Tesla V100-DGXS-32GB, pci bus id: 0000:07:00.0, compute capability: 7.0)
2019-09-10 14:26:21.870714: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 30463 MB memory) -> physical GPU (device: 1, name: Tesla V100-DGXS-32GB, pci bus id: 0000:08:00.0, compute capability: 7.0)
2019-09-10 14:26:21.872869: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:2 with 30463 MB memory) -> physical GPU (device: 2, name: Tesla V100-DGXS-32GB, pci bus id: 0000:0e:00.0, compute capability: 7.0)
2019-09-10 14:26:21.875005: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:3 with 30463 MB memory) -> physical GPU (device: 3, name: Tesla V100-DGXS-32GB, pci bus id: 0000:0f:00.0, compute capability: 7.0)
TensorFlow:  1.14
Model:       resnet50
Dataset:     imagenet (synthetic)
Mode:        training
SingleSess:  False
Batch size:  512 global
             256 per device
Num batches: 100
Num epochs:  0.04
Devices:     ['/gpu:0', '/gpu:1']
NUMA bind:   False
Data format: NHWC
Optimizer:   sgd
Variables:   replicated
AllReduce:   None
==========
W0910 14:26:21.881631 140219604993856 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2762: The name tf.set_random_seed is deprecated. Please use tf.compat.v1.set_random_seed instead.

Generating training model
W0910 14:26:21.882249 140219604993856 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2782: The name tf.train.get_or_create_global_step is deprecated. Please use tf.compat.v1.train.get_or_create_global_step instead.

W0910 14:26:21.889526 140219604993856 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2739: The name tf.get_variable is deprecated. Please use tf.compat.v1.get_variable instead.

W0910 14:26:21.895006 140219604993856 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/variable_mgr.py:316: The name tf.variable_scope is deprecated. Please use tf.compat.v1.variable_scope instead.

W0910 14:26:21.912440 140219604993856 deprecation.py:323] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:129: conv2d (from tensorflow.python.layers.convolutional) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.keras.layers.Conv2D` instead.
W0910 14:26:22.286693 140219604993856 deprecation.py:323] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:261: max_pooling2d (from tensorflow.python.layers.pooling) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.MaxPooling2D instead.
W0910 14:26:24.972420 140219604993856 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:334: The name tf.nn.xw_plus_b is deprecated. Please use tf.compat.v1.nn.xw_plus_b instead.

W0910 14:26:24.975386 140219604993856 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/models/model.py:319: The name tf.losses.sparse_softmax_cross_entropy is deprecated. Please use tf.compat.v1.losses.sparse_softmax_cross_entropy instead.

W0910 14:26:24.987217 140219604993856 deprecation.py:323] From /usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/losses/losses_impl.py:121: add_dispatch_support.<locals>.wrapper (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.where in 2.0, which has the same broadcast rule as np.where
W0910 14:26:30.412933 140219604993856 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/models/resnet_model.py:333: The name tf.train.piecewise_constant is deprecated. Please use tf.compat.v1.train.piecewise_constant instead.

W0910 14:26:30.501047 140219604993856 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:1209: The name tf.train.GradientDescentOptimizer is deprecated. Please use tf.compat.v1.train.GradientDescentOptimizer instead.

W0910 14:26:30.924280 140219604993856 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2090: The name tf.train.get_global_step is deprecated. Please use tf.compat.v1.train.get_global_step instead.

W0910 14:26:31.314418 140219604993856 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2126: The name tf.summary.merge_all is deprecated. Please use tf.compat.v1.summary.merge_all instead.

Initializing graph
W0910 14:26:31.314657 140219604993856 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2183: The name tf.train.Saver is deprecated. Please use tf.compat.v1.train.Saver instead.

W0910 14:26:31.579755 140219604993856 deprecation.py:323] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2238: Supervisor.__init__ (from tensorflow.python.training.supervisor) is deprecated and will be removed in a future version.
Instructions for updating:
Please switch to tf.train.MonitoredTrainingSession
2019-09-10 14:26:32.687091: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:07:00.0
2019-09-10 14:26:32.688625: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 1 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:08:00.0
2019-09-10 14:26:32.690125: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 2 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:0e:00.0
2019-09-10 14:26:32.691622: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 3 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:0f:00.0
2019-09-10 14:26:32.691653: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
2019-09-10 14:26:32.691687: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10
2019-09-10 14:26:32.691705: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcufft.so.10
2019-09-10 14:26:32.691723: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcurand.so.10
2019-09-10 14:26:32.691740: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusolver.so.10
2019-09-10 14:26:32.691755: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusparse.so.10
2019-09-10 14:26:32.691773: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
2019-09-10 14:26:32.703621: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0, 1, 2, 3
2019-09-10 14:26:32.703680: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1181] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-09-10 14:26:32.703692: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1187]      0 1 2 3
2019-09-10 14:26:32.703700: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 0:   N Y Y Y
2019-09-10 14:26:32.703706: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 1:   Y N Y Y
2019-09-10 14:26:32.703713: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 2:   Y Y N Y
2019-09-10 14:26:32.703720: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 3:   Y Y Y N
2019-09-10 14:26:32.711478: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 30438 MB memory) -> physical GPU (device: 0, name: Tesla V100-DGXS-32GB, pci bus id: 0000:07:00.0, compute capability: 7.0)
2019-09-10 14:26:32.713088: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 30463 MB memory) -> physical GPU (device: 1, name: Tesla V100-DGXS-32GB, pci bus id: 0000:08:00.0, compute capability: 7.0)
2019-09-10 14:26:32.714673: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:2 with 30463 MB memory) -> physical GPU (device: 2, name: Tesla V100-DGXS-32GB, pci bus id: 0000:0e:00.0, compute capability: 7.0)
2019-09-10 14:26:32.716263: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:3 with 30463 MB memory) -> physical GPU (device: 3, name: Tesla V100-DGXS-32GB, pci bus id: 0000:0f:00.0, compute capability: 7.0)
2019-09-10 14:26:33.863633: W tensorflow/compiler/jit/mark_for_compilation_pass.cc:1412] (One-time warning): Not using XLA:CPU for cluster because envvar TF_XLA_FLAGS=--tf_xla_cpu_global_jit was not set.  If you want XLA:CPU, either set that envvar, or use experimental_jit_scope to enable XLA:CPU.  To confirm that XLA is active, pass --vmodule=xla_compilation_cache=1 (as a proper command-line flag, not via TF_XLA_FLAGS) or set the envvar XLA_FLAGS=--xla_hlo_profile.
I0910 14:26:35.855477 140219604993856 session_manager.py:500] Running local_init_op.
I0910 14:26:36.503135 140219604993856 session_manager.py:502] Done running local_init_op.
Running warm up
2019-09-10 14:26:39.864129: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10
2019-09-10 14:26:40.591895: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
Done warm up
Step    Img/sec total_loss
1       images/sec: 1637.3 +/- 0.0 (jitter = 0.0)       7.941
10      images/sec: 1640.3 +/- 0.7 (jitter = 1.8)       7.941
20      images/sec: 1639.6 +/- 0.6 (jitter = 2.7)       7.881
30      images/sec: 1638.7 +/- 0.5 (jitter = 3.7)       7.858
40      images/sec: 1638.1 +/- 0.5 (jitter = 3.2)       7.845
50      images/sec: 1638.1 +/- 0.4 (jitter = 3.2)       7.878
60      images/sec: 1638.3 +/- 0.4 (jitter = 2.9)       7.795
70      images/sec: 1638.2 +/- 0.3 (jitter = 2.9)       7.733
80      images/sec: 1638.4 +/- 0.3 (jitter = 2.7)       7.745
90      images/sec: 1638.4 +/- 0.3 (jitter = 2.7)       7.707
100     images/sec: 1638.5 +/- 0.3 (jitter = 2.6)       7.678
----------------------------------------------------------------
total images/sec: 1638.03
----------------------------------------------------------------
2019-09-10 14:27:24.666012: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
WARNING: Logging before flag parsing goes to stderr.
W0910 14:27:26.685798 140315749975872 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/models/mobilenet.py:388: The name tf.nn.avg_pool is deprecated. Please use tf.nn.avg_pool2d instead.

W0910 14:27:26.690323 140315749975872 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/models/experimental/deepspeech.py:131: The name tf.nn.rnn_cell.RNNCell is deprecated. Please use tf.compat.v1.nn.rnn_cell.RNNCell instead.

W0910 14:27:26.713760 140315749975872 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:722: The name tf.ConfigProto is deprecated. Please use tf.compat.v1.ConfigProto instead.

W0910 14:27:26.713972 140315749975872 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:3503: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead.

2019-09-10 14:27:26.736723: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2198780000 Hz
2019-09-10 14:27:26.739181: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x3faec20 executing computations on platform Host. Devices:
2019-09-10 14:27:26.739216: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): <undefined>, <undefined>
2019-09-10 14:27:26.742711: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcuda.so.1
2019-09-10 14:27:27.539075: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x3faef80 executing computations on platform CUDA. Devices:
2019-09-10 14:27:27.539109: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): Tesla V100-DGXS-32GB, Compute Capability 7.0
2019-09-10 14:27:27.539120: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (1): Tesla V100-DGXS-32GB, Compute Capability 7.0
2019-09-10 14:27:27.539130: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (2): Tesla V100-DGXS-32GB, Compute Capability 7.0
2019-09-10 14:27:27.539138: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (3): Tesla V100-DGXS-32GB, Compute Capability 7.0
2019-09-10 14:27:27.541656: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:07:00.0
2019-09-10 14:27:27.543800: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 1 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:08:00.0
2019-09-10 14:27:27.545926: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 2 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:0e:00.0
2019-09-10 14:27:27.548027: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 3 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:0f:00.0
2019-09-10 14:27:27.548062: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
2019-09-10 14:27:27.550465: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10
2019-09-10 14:27:27.552595: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcufft.so.10
2019-09-10 14:27:27.552982: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcurand.so.10
2019-09-10 14:27:27.554680: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusolver.so.10
2019-09-10 14:27:27.555666: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusparse.so.10
2019-09-10 14:27:27.559227: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
2019-09-10 14:27:27.571105: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0, 1, 2, 3
2019-09-10 14:27:27.571138: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
2019-09-10 14:27:29.338866: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1181] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-09-10 14:27:29.338908: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1187]      0 1 2 3
2019-09-10 14:27:29.338919: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 0:   N Y Y Y
2019-09-10 14:27:29.338925: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 1:   Y N Y Y
2019-09-10 14:27:29.338931: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 2:   Y Y N Y
2019-09-10 14:27:29.338937: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 3:   Y Y Y N
2019-09-10 14:27:29.347004: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 30438 MB memory) -> physical GPU (device: 0, name: Tesla V100-DGXS-32GB, pci bus id: 0000:07:00.0, compute capability: 7.0)
2019-09-10 14:27:29.349219: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 30463 MB memory) -> physical GPU (device: 1, name: Tesla V100-DGXS-32GB, pci bus id: 0000:08:00.0, compute capability: 7.0)
2019-09-10 14:27:29.351382: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:2 with 30463 MB memory) -> physical GPU (device: 2, name: Tesla V100-DGXS-32GB, pci bus id: 0000:0e:00.0, compute capability: 7.0)
2019-09-10 14:27:29.353560: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:3 with 30463 MB memory) -> physical GPU (device: 3, name: Tesla V100-DGXS-32GB, pci bus id: 0000:0f:00.0, compute capability: 7.0)
TensorFlow:  1.14
Model:       resnet50
Dataset:     imagenet (synthetic)
Mode:        training
SingleSess:  False
Batch size:  1024 global
             512 per device
Num batches: 100
Num epochs:  0.08
Devices:     ['/gpu:0', '/gpu:1']
NUMA bind:   False
Data format: NHWC
Optimizer:   sgd
Variables:   replicated
AllReduce:   None
==========
W0910 14:27:29.359734 140315749975872 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2762: The name tf.set_random_seed is deprecated. Please use tf.compat.v1.set_random_seed instead.

Generating training model
W0910 14:27:29.360307 140315749975872 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2782: The name tf.train.get_or_create_global_step is deprecated. Please use tf.compat.v1.train.get_or_create_global_step instead.

W0910 14:27:29.369190 140315749975872 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2739: The name tf.get_variable is deprecated. Please use tf.compat.v1.get_variable instead.

W0910 14:27:29.373850 140315749975872 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/variable_mgr.py:316: The name tf.variable_scope is deprecated. Please use tf.compat.v1.variable_scope instead.

W0910 14:27:29.390364 140315749975872 deprecation.py:323] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:129: conv2d (from tensorflow.python.layers.convolutional) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.keras.layers.Conv2D` instead.
W0910 14:27:29.751940 140315749975872 deprecation.py:323] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:261: max_pooling2d (from tensorflow.python.layers.pooling) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.MaxPooling2D instead.
W0910 14:27:32.344232 140315749975872 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:334: The name tf.nn.xw_plus_b is deprecated. Please use tf.compat.v1.nn.xw_plus_b instead.

W0910 14:27:32.347057 140315749975872 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/models/model.py:319: The name tf.losses.sparse_softmax_cross_entropy is deprecated. Please use tf.compat.v1.losses.sparse_softmax_cross_entropy instead.

W0910 14:27:32.358335 140315749975872 deprecation.py:323] From /usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/losses/losses_impl.py:121: add_dispatch_support.<locals>.wrapper (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.where in 2.0, which has the same broadcast rule as np.where
W0910 14:27:37.570375 140315749975872 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/models/resnet_model.py:333: The name tf.train.piecewise_constant is deprecated. Please use tf.compat.v1.train.piecewise_constant instead.

W0910 14:27:37.654525 140315749975872 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:1209: The name tf.train.GradientDescentOptimizer is deprecated. Please use tf.compat.v1.train.GradientDescentOptimizer instead.

W0910 14:27:38.058170 140315749975872 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2090: The name tf.train.get_global_step is deprecated. Please use tf.compat.v1.train.get_global_step instead.

W0910 14:27:38.429493 140315749975872 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2126: The name tf.summary.merge_all is deprecated. Please use tf.compat.v1.summary.merge_all instead.

Initializing graph
W0910 14:27:38.429723 140315749975872 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2183: The name tf.train.Saver is deprecated. Please use tf.compat.v1.train.Saver instead.

W0910 14:27:38.683707 140315749975872 deprecation.py:323] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2238: Supervisor.__init__ (from tensorflow.python.training.supervisor) is deprecated and will be removed in a future version.
Instructions for updating:
Please switch to tf.train.MonitoredTrainingSession
2019-09-10 14:27:39.764123: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:07:00.0
2019-09-10 14:27:39.765637: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 1 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:08:00.0
2019-09-10 14:27:39.767119: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 2 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:0e:00.0
2019-09-10 14:27:39.768604: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 3 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:0f:00.0
2019-09-10 14:27:39.768633: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
2019-09-10 14:27:39.768667: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10
2019-09-10 14:27:39.768685: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcufft.so.10
2019-09-10 14:27:39.768701: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcurand.so.10
2019-09-10 14:27:39.768717: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusolver.so.10
2019-09-10 14:27:39.768733: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusparse.so.10
2019-09-10 14:27:39.768749: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
2019-09-10 14:27:39.780334: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0, 1, 2, 3
2019-09-10 14:27:39.780392: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1181] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-09-10 14:27:39.780402: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1187]      0 1 2 3
2019-09-10 14:27:39.780410: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 0:   N Y Y Y
2019-09-10 14:27:39.780417: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 1:   Y N Y Y
2019-09-10 14:27:39.780423: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 2:   Y Y N Y
2019-09-10 14:27:39.780431: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 3:   Y Y Y N
2019-09-10 14:27:39.788214: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 30438 MB memory) -> physical GPU (device: 0, name: Tesla V100-DGXS-32GB, pci bus id: 0000:07:00.0, compute capability: 7.0)
2019-09-10 14:27:39.789794: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 30463 MB memory) -> physical GPU (device: 1, name: Tesla V100-DGXS-32GB, pci bus id: 0000:08:00.0, compute capability: 7.0)
2019-09-10 14:27:39.791359: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:2 with 30463 MB memory) -> physical GPU (device: 2, name: Tesla V100-DGXS-32GB, pci bus id: 0000:0e:00.0, compute capability: 7.0)
2019-09-10 14:27:39.792926: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:3 with 30463 MB memory) -> physical GPU (device: 3, name: Tesla V100-DGXS-32GB, pci bus id: 0000:0f:00.0, compute capability: 7.0)
2019-09-10 14:27:40.918063: W tensorflow/compiler/jit/mark_for_compilation_pass.cc:1412] (One-time warning): Not using XLA:CPU for cluster because envvar TF_XLA_FLAGS=--tf_xla_cpu_global_jit was not set.  If you want XLA:CPU, either set that envvar, or use experimental_jit_scope to enable XLA:CPU.  To confirm that XLA is active, pass --vmodule=xla_compilation_cache=1 (as a proper command-line flag, not via TF_XLA_FLAGS) or set the envvar XLA_FLAGS=--xla_hlo_profile.
I0910 14:27:42.901646 140315749975872 session_manager.py:500] Running local_init_op.
I0910 14:27:43.553680 140315749975872 session_manager.py:502] Done running local_init_op.
Running warm up
2019-09-10 14:27:46.930607: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10
2019-09-10 14:27:47.657608: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
Done warm up
Step    Img/sec total_loss
1       images/sec: 1677.6 +/- 0.0 (jitter = 0.0)       7.881
10      images/sec: 1674.6 +/- 1.0 (jitter = 2.7)       7.880
20      images/sec: 1674.5 +/- 0.6 (jitter = 2.9)       7.836
30      images/sec: 1674.7 +/- 0.5 (jitter = 2.7)       7.768
40      images/sec: 1674.3 +/- 0.5 (jitter = 2.7)       7.737
50      images/sec: 1674.4 +/- 0.4 (jitter = 2.5)       7.663
60      images/sec: 1674.4 +/- 0.4 (jitter = 2.9)       7.622
70      images/sec: 1674.2 +/- 0.3 (jitter = 2.8)       7.608
80      images/sec: 1674.5 +/- 0.3 (jitter = 3.2)       7.590
90      images/sec: 1674.6 +/- 0.3 (jitter = 3.3)       7.569
100     images/sec: 1674.5 +/- 0.3 (jitter = 3.1)       7.527
----------------------------------------------------------------
total images/sec: 1674.28
----------------------------------------------------------------
2019-09-10 14:29:08.669428: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
WARNING: Logging before flag parsing goes to stderr.
W0910 14:29:10.656620 140108273727296 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/models/mobilenet.py:388: The name tf.nn.avg_pool is deprecated. Please use tf.nn.avg_pool2d instead.

W0910 14:29:10.661067 140108273727296 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/models/experimental/deepspeech.py:131: The name tf.nn.rnn_cell.RNNCell is deprecated. Please use tf.compat.v1.nn.rnn_cell.RNNCell instead.

W0910 14:29:10.684174 140108273727296 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:722: The name tf.ConfigProto is deprecated. Please use tf.compat.v1.ConfigProto instead.

W0910 14:29:10.684391 140108273727296 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:3503: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead.

2019-09-10 14:29:10.708721: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2198780000 Hz
2019-09-10 14:29:10.711331: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x34e5040 executing computations on platform Host. Devices:
2019-09-10 14:29:10.711372: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): <undefined>, <undefined>
2019-09-10 14:29:10.714902: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcuda.so.1
2019-09-10 14:29:11.547614: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x34de510 executing computations on platform CUDA. Devices:
2019-09-10 14:29:11.547655: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): Tesla V100-DGXS-32GB, Compute Capability 7.0
2019-09-10 14:29:11.547669: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (1): Tesla V100-DGXS-32GB, Compute Capability 7.0
2019-09-10 14:29:11.547679: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (2): Tesla V100-DGXS-32GB, Compute Capability 7.0
2019-09-10 14:29:11.547689: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (3): Tesla V100-DGXS-32GB, Compute Capability 7.0
2019-09-10 14:29:11.550588: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:07:00.0
2019-09-10 14:29:11.553011: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 1 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:08:00.0
2019-09-10 14:29:11.555391: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 2 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:0e:00.0
2019-09-10 14:29:11.557807: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 3 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:0f:00.0
2019-09-10 14:29:11.557848: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
2019-09-10 14:29:11.560576: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10
2019-09-10 14:29:11.562285: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcufft.so.10
2019-09-10 14:29:11.562579: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcurand.so.10
2019-09-10 14:29:11.564325: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusolver.so.10
2019-09-10 14:29:11.565341: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusparse.so.10
2019-09-10 14:29:11.568983: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
2019-09-10 14:29:11.580898: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0, 1, 2, 3
2019-09-10 14:29:11.580931: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
2019-09-10 14:29:13.377180: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1181] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-09-10 14:29:13.377221: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1187]      0 1 2 3
2019-09-10 14:29:13.377232: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 0:   N Y Y Y
2019-09-10 14:29:13.377239: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 1:   Y N Y Y
2019-09-10 14:29:13.377245: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 2:   Y Y N Y
2019-09-10 14:29:13.377251: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 3:   Y Y Y N
2019-09-10 14:29:13.385337: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 30438 MB memory) -> physical GPU (device: 0, name: Tesla V100-DGXS-32GB, pci bus id: 0000:07:00.0, compute capability: 7.0)
2019-09-10 14:29:13.387552: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 30463 MB memory) -> physical GPU (device: 1, name: Tesla V100-DGXS-32GB, pci bus id: 0000:08:00.0, compute capability: 7.0)
2019-09-10 14:29:13.389660: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:2 with 30463 MB memory) -> physical GPU (device: 2, name: Tesla V100-DGXS-32GB, pci bus id: 0000:0e:00.0, compute capability: 7.0)
2019-09-10 14:29:13.391819: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:3 with 30463 MB memory) -> physical GPU (device: 3, name: Tesla V100-DGXS-32GB, pci bus id: 0000:0f:00.0, compute capability: 7.0)
TensorFlow:  1.14
Model:       resnet50
Dataset:     imagenet (synthetic)
Mode:        training
SingleSess:  False
Batch size:  192 global
             64 per device
Num batches: 100
Num epochs:  0.01
Devices:     ['/gpu:0', '/gpu:1', '/gpu:2']
NUMA bind:   False
Data format: NHWC
Optimizer:   sgd
Variables:   replicated
AllReduce:   None
==========
W0910 14:29:13.398070 140108273727296 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2762: The name tf.set_random_seed is deprecated. Please use tf.compat.v1.set_random_seed instead.

Generating training model
W0910 14:29:13.398653 140108273727296 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2782: The name tf.train.get_or_create_global_step is deprecated. Please use tf.compat.v1.train.get_or_create_global_step instead.

W0910 14:29:13.407375 140108273727296 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/variable_mgr.py:316: The name tf.variable_scope is deprecated. Please use tf.compat.v1.variable_scope instead.

W0910 14:29:13.407935 140108273727296 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/models/model.py:245: The name tf.truncated_normal is deprecated. Please use tf.random.truncated_normal instead.

W0910 14:29:13.426264 140108273727296 deprecation.py:323] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:129: conv2d (from tensorflow.python.layers.convolutional) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.keras.layers.Conv2D` instead.
W0910 14:29:13.800179 140108273727296 deprecation.py:323] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:261: max_pooling2d (from tensorflow.python.layers.pooling) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.MaxPooling2D instead.
W0910 14:29:16.361115 140108273727296 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:334: The name tf.nn.xw_plus_b is deprecated. Please use tf.compat.v1.nn.xw_plus_b instead.

W0910 14:29:16.363334 140108273727296 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/models/model.py:319: The name tf.losses.sparse_softmax_cross_entropy is deprecated. Please use tf.compat.v1.losses.sparse_softmax_cross_entropy instead.

W0910 14:29:16.375196 140108273727296 deprecation.py:323] From /usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/losses/losses_impl.py:121: add_dispatch_support.<locals>.wrapper (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.where in 2.0, which has the same broadcast rule as np.where
W0910 14:29:23.781066 140108273727296 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/models/resnet_model.py:333: The name tf.train.piecewise_constant is deprecated. Please use tf.compat.v1.train.piecewise_constant instead.

W0910 14:29:23.869140 140108273727296 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:1209: The name tf.train.GradientDescentOptimizer is deprecated. Please use tf.compat.v1.train.GradientDescentOptimizer instead.

W0910 14:29:24.535114 140108273727296 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2090: The name tf.train.get_global_step is deprecated. Please use tf.compat.v1.train.get_global_step instead.

W0910 14:29:25.302121 140108273727296 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2126: The name tf.summary.merge_all is deprecated. Please use tf.compat.v1.summary.merge_all instead.

Initializing graph
W0910 14:29:25.302373 140108273727296 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2183: The name tf.train.Saver is deprecated. Please use tf.compat.v1.train.Saver instead.

W0910 14:29:25.565145 140108273727296 deprecation.py:323] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2238: Supervisor.__init__ (from tensorflow.python.training.supervisor) is deprecated and will be removed in a future version.
Instructions for updating:
Please switch to tf.train.MonitoredTrainingSession
2019-09-10 14:29:27.057529: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:07:00.0
2019-09-10 14:29:27.059127: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 1 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:08:00.0
2019-09-10 14:29:27.060884: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 2 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:0e:00.0
2019-09-10 14:29:27.062640: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 3 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:0f:00.0
2019-09-10 14:29:27.062672: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
2019-09-10 14:29:27.062708: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10
2019-09-10 14:29:27.062727: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcufft.so.10
2019-09-10 14:29:27.062744: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcurand.so.10
2019-09-10 14:29:27.062762: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusolver.so.10
2019-09-10 14:29:27.062781: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusparse.so.10
2019-09-10 14:29:27.062799: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
2019-09-10 14:29:27.074690: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0, 1, 2, 3
2019-09-10 14:29:27.074750: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1181] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-09-10 14:29:27.074760: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1187]      0 1 2 3
2019-09-10 14:29:27.074768: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 0:   N Y Y Y
2019-09-10 14:29:27.074775: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 1:   Y N Y Y
2019-09-10 14:29:27.074781: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 2:   Y Y N Y
2019-09-10 14:29:27.074789: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 3:   Y Y Y N
2019-09-10 14:29:27.082613: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 30438 MB memory) -> physical GPU (device: 0, name: Tesla V100-DGXS-32GB, pci bus id: 0000:07:00.0, compute capability: 7.0)
2019-09-10 14:29:27.084232: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 30463 MB memory) -> physical GPU (device: 1, name: Tesla V100-DGXS-32GB, pci bus id: 0000:08:00.0, compute capability: 7.0)
2019-09-10 14:29:27.085840: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:2 with 30463 MB memory) -> physical GPU (device: 2, name: Tesla V100-DGXS-32GB, pci bus id: 0000:0e:00.0, compute capability: 7.0)
2019-09-10 14:29:27.087426: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:3 with 30463 MB memory) -> physical GPU (device: 3, name: Tesla V100-DGXS-32GB, pci bus id: 0000:0f:00.0, compute capability: 7.0)
2019-09-10 14:29:28.765411: W tensorflow/compiler/jit/mark_for_compilation_pass.cc:1412] (One-time warning): Not using XLA:CPU for cluster because envvar TF_XLA_FLAGS=--tf_xla_cpu_global_jit was not set.  If you want XLA:CPU, either set that envvar, or use experimental_jit_scope to enable XLA:CPU.  To confirm that XLA is active, pass --vmodule=xla_compilation_cache=1 (as a proper command-line flag, not via TF_XLA_FLAGS) or set the envvar XLA_FLAGS=--xla_hlo_profile.
I0910 14:29:31.292917 140108273727296 session_manager.py:500] Running local_init_op.
I0910 14:29:32.368149 140108273727296 session_manager.py:502] Done running local_init_op.
Running warm up
2019-09-10 14:29:36.315383: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10
2019-09-10 14:29:37.345462: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
Done warm up
Step    Img/sec total_loss
1       images/sec: 1059.7 +/- 0.0 (jitter = 0.0)       7.680
10      images/sec: 1059.5 +/- 0.4 (jitter = 0.8)       7.907
20      images/sec: 1059.6 +/- 0.3 (jitter = 1.5)       7.866
30      images/sec: 1059.0 +/- 0.3 (jitter = 2.0)       7.926
40      images/sec: 1058.2 +/- 0.3 (jitter = 2.2)       7.935
50      images/sec: 1057.8 +/- 0.4 (jitter = 2.3)       7.797
60      images/sec: 1057.8 +/- 0.4 (jitter = 2.2)       7.832
70      images/sec: 1057.8 +/- 0.3 (jitter = 2.2)       7.922
80      images/sec: 1057.7 +/- 0.3 (jitter = 1.9)       7.809
90      images/sec: 1057.2 +/- 0.3 (jitter = 2.3)       7.848
100     images/sec: 1056.8 +/- 0.3 (jitter = 2.9)       7.803
----------------------------------------------------------------
total images/sec: 1056.40
----------------------------------------------------------------
2019-09-10 14:30:06.546204: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
WARNING: Logging before flag parsing goes to stderr.
W0910 14:30:08.574933 139862905734976 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/models/mobilenet.py:388: The name tf.nn.avg_pool is deprecated. Please use tf.nn.avg_pool2d instead.

W0910 14:30:08.579488 139862905734976 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/models/experimental/deepspeech.py:131: The name tf.nn.rnn_cell.RNNCell is deprecated. Please use tf.compat.v1.nn.rnn_cell.RNNCell instead.

W0910 14:30:08.603000 139862905734976 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:722: The name tf.ConfigProto is deprecated. Please use tf.compat.v1.ConfigProto instead.

W0910 14:30:08.603210 139862905734976 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:3503: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead.

2019-09-10 14:30:08.628722: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2198780000 Hz
2019-09-10 14:30:08.631197: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x39b6c70 executing computations on platform Host. Devices:
2019-09-10 14:30:08.631223: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): <undefined>, <undefined>
2019-09-10 14:30:08.633866: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcuda.so.1
2019-09-10 14:30:09.536844: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x39b0380 executing computations on platform CUDA. Devices:
2019-09-10 14:30:09.536878: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): Tesla V100-DGXS-32GB, Compute Capability 7.0
2019-09-10 14:30:09.536889: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (1): Tesla V100-DGXS-32GB, Compute Capability 7.0
2019-09-10 14:30:09.536897: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (2): Tesla V100-DGXS-32GB, Compute Capability 7.0
2019-09-10 14:30:09.536906: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (3): Tesla V100-DGXS-32GB, Compute Capability 7.0
2019-09-10 14:30:09.539260: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:07:00.0
2019-09-10 14:30:09.541237: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 1 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:08:00.0
2019-09-10 14:30:09.546732: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 2 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:0e:00.0
2019-09-10 14:30:09.548709: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 3 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:0f:00.0
2019-09-10 14:30:09.548744: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
2019-09-10 14:30:09.550947: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10
2019-09-10 14:30:09.552897: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcufft.so.10
2019-09-10 14:30:09.553179: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcurand.so.10
2019-09-10 14:30:09.554908: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusolver.so.10
2019-09-10 14:30:09.555908: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusparse.so.10
2019-09-10 14:30:09.559524: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
2019-09-10 14:30:09.571359: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0, 1, 2, 3
2019-09-10 14:30:09.571392: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
2019-09-10 14:30:11.359701: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1181] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-09-10 14:30:11.359738: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1187]      0 1 2 3
2019-09-10 14:30:11.359748: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 0:   N Y Y Y
2019-09-10 14:30:11.359754: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 1:   Y N Y Y
2019-09-10 14:30:11.359761: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 2:   Y Y N Y
2019-09-10 14:30:11.359767: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 3:   Y Y Y N
2019-09-10 14:30:11.367853: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 30438 MB memory) -> physical GPU (device: 0, name: Tesla V100-DGXS-32GB, pci bus id: 0000:07:00.0, compute capability: 7.0)
2019-09-10 14:30:11.370095: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 30463 MB memory) -> physical GPU (device: 1, name: Tesla V100-DGXS-32GB, pci bus id: 0000:08:00.0, compute capability: 7.0)
2019-09-10 14:30:11.372362: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:2 with 30463 MB memory) -> physical GPU (device: 2, name: Tesla V100-DGXS-32GB, pci bus id: 0000:0e:00.0, compute capability: 7.0)
2019-09-10 14:30:11.374537: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:3 with 30463 MB memory) -> physical GPU (device: 3, name: Tesla V100-DGXS-32GB, pci bus id: 0000:0f:00.0, compute capability: 7.0)
TensorFlow:  1.14
Model:       resnet50
Dataset:     imagenet (synthetic)
Mode:        training
SingleSess:  False
Batch size:  192 global
             64 per device
Num batches: 100
Num epochs:  0.01
Devices:     ['/gpu:0', '/gpu:1', '/gpu:2']
NUMA bind:   False
Data format: NHWC
Optimizer:   sgd
Variables:   replicated
AllReduce:   None
==========
W0910 14:30:11.380581 139862905734976 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2762: The name tf.set_random_seed is deprecated. Please use tf.compat.v1.set_random_seed instead.

Generating training model
W0910 14:30:11.381137 139862905734976 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2782: The name tf.train.get_or_create_global_step is deprecated. Please use tf.compat.v1.train.get_or_create_global_step instead.

W0910 14:30:11.389535 139862905734976 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2739: The name tf.get_variable is deprecated. Please use tf.compat.v1.get_variable instead.

W0910 14:30:11.394240 139862905734976 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/variable_mgr.py:316: The name tf.variable_scope is deprecated. Please use tf.compat.v1.variable_scope instead.

W0910 14:30:11.410990 139862905734976 deprecation.py:323] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:129: conv2d (from tensorflow.python.layers.convolutional) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.keras.layers.Conv2D` instead.
W0910 14:30:11.774429 139862905734976 deprecation.py:323] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:261: max_pooling2d (from tensorflow.python.layers.pooling) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.MaxPooling2D instead.
W0910 14:30:14.393703 139862905734976 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:334: The name tf.nn.xw_plus_b is deprecated. Please use tf.compat.v1.nn.xw_plus_b instead.

W0910 14:30:14.396571 139862905734976 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/models/model.py:319: The name tf.losses.sparse_softmax_cross_entropy is deprecated. Please use tf.compat.v1.losses.sparse_softmax_cross_entropy instead.

W0910 14:30:14.408024 139862905734976 deprecation.py:323] From /usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/losses/losses_impl.py:121: add_dispatch_support.<locals>.wrapper (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.where in 2.0, which has the same broadcast rule as np.where
W0910 14:30:23.494632 139862905734976 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/models/resnet_model.py:333: The name tf.train.piecewise_constant is deprecated. Please use tf.compat.v1.train.piecewise_constant instead.

W0910 14:30:23.580518 139862905734976 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:1209: The name tf.train.GradientDescentOptimizer is deprecated. Please use tf.compat.v1.train.GradientDescentOptimizer instead.

W0910 14:30:24.393837 139862905734976 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2090: The name tf.train.get_global_step is deprecated. Please use tf.compat.v1.train.get_global_step instead.

W0910 14:30:25.147120 139862905734976 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2126: The name tf.summary.merge_all is deprecated. Please use tf.compat.v1.summary.merge_all instead.

Initializing graph
W0910 14:30:25.147358 139862905734976 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2183: The name tf.train.Saver is deprecated. Please use tf.compat.v1.train.Saver instead.

W0910 14:30:25.405670 139862905734976 deprecation.py:323] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2238: Supervisor.__init__ (from tensorflow.python.training.supervisor) is deprecated and will be removed in a future version.
Instructions for updating:
Please switch to tf.train.MonitoredTrainingSession
2019-09-10 14:30:26.863619: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:07:00.0
2019-09-10 14:30:26.865215: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 1 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:08:00.0
2019-09-10 14:30:26.866987: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 2 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:0e:00.0
2019-09-10 14:30:26.868881: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 3 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:0f:00.0
2019-09-10 14:30:26.868912: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
2019-09-10 14:30:26.868947: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10
2019-09-10 14:30:26.868966: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcufft.so.10
2019-09-10 14:30:26.868982: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcurand.so.10
2019-09-10 14:30:26.868999: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusolver.so.10
2019-09-10 14:30:26.869016: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusparse.so.10
2019-09-10 14:30:26.869034: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
2019-09-10 14:30:26.880888: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0, 1, 2, 3
2019-09-10 14:30:26.880945: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1181] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-09-10 14:30:26.880955: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1187]      0 1 2 3
2019-09-10 14:30:26.880962: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 0:   N Y Y Y
2019-09-10 14:30:26.880969: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 1:   Y N Y Y
2019-09-10 14:30:26.880975: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 2:   Y Y N Y
2019-09-10 14:30:26.880982: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 3:   Y Y Y N
2019-09-10 14:30:26.888736: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 30438 MB memory) -> physical GPU (device: 0, name: Tesla V100-DGXS-32GB, pci bus id: 0000:07:00.0, compute capability: 7.0)
2019-09-10 14:30:26.890344: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 30463 MB memory) -> physical GPU (device: 1, name: Tesla V100-DGXS-32GB, pci bus id: 0000:08:00.0, compute capability: 7.0)
2019-09-10 14:30:26.891927: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:2 with 30463 MB memory) -> physical GPU (device: 2, name: Tesla V100-DGXS-32GB, pci bus id: 0000:0e:00.0, compute capability: 7.0)
2019-09-10 14:30:26.893526: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:3 with 30463 MB memory) -> physical GPU (device: 3, name: Tesla V100-DGXS-32GB, pci bus id: 0000:0f:00.0, compute capability: 7.0)
2019-09-10 14:30:28.642114: W tensorflow/compiler/jit/mark_for_compilation_pass.cc:1412] (One-time warning): Not using XLA:CPU for cluster because envvar TF_XLA_FLAGS=--tf_xla_cpu_global_jit was not set.  If you want XLA:CPU, either set that envvar, or use experimental_jit_scope to enable XLA:CPU.  To confirm that XLA is active, pass --vmodule=xla_compilation_cache=1 (as a proper command-line flag, not via TF_XLA_FLAGS) or set the envvar XLA_FLAGS=--xla_hlo_profile.
I0910 14:30:31.150029 139862905734976 session_manager.py:500] Running local_init_op.
I0910 14:30:32.218698 139862905734976 session_manager.py:502] Done running local_init_op.
Running warm up
2019-09-10 14:30:37.024126: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10
2019-09-10 14:30:38.113790: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
Done warm up
Step    Img/sec total_loss
1       images/sec: 1860.4 +/- 0.0 (jitter = 0.0)       8.040
10      images/sec: 1852.7 +/- 6.0 (jitter = 16.8)      7.810
20      images/sec: 1848.8 +/- 3.8 (jitter = 16.5)      7.898
30      images/sec: 1846.2 +/- 2.9 (jitter = 15.0)      7.924
40      images/sec: 1845.3 +/- 2.4 (jitter = 14.1)      7.868
50      images/sec: 1844.1 +/- 2.1 (jitter = 14.3)      7.889
60      images/sec: 1843.9 +/- 1.9 (jitter = 16.0)      7.920
70      images/sec: 1844.2 +/- 1.7 (jitter = 14.6)      7.861
80      images/sec: 1844.2 +/- 1.6 (jitter = 14.6)      7.786
90      images/sec: 1844.2 +/- 1.5 (jitter = 15.7)      7.890
100     images/sec: 1844.1 +/- 1.4 (jitter = 14.5)      7.715
----------------------------------------------------------------
total images/sec: 1842.96
----------------------------------------------------------------
2019-09-10 14:30:58.862575: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
WARNING: Logging before flag parsing goes to stderr.
W0910 14:31:00.887480 139713216689984 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/models/mobilenet.py:388: The name tf.nn.avg_pool is deprecated. Please use tf.nn.avg_pool2d instead.

W0910 14:31:00.892082 139713216689984 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/models/experimental/deepspeech.py:131: The name tf.nn.rnn_cell.RNNCell is deprecated. Please use tf.compat.v1.nn.rnn_cell.RNNCell instead.

W0910 14:31:00.915890 139713216689984 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:722: The name tf.ConfigProto is deprecated. Please use tf.compat.v1.ConfigProto instead.

W0910 14:31:00.916103 139713216689984 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:3503: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead.

2019-09-10 14:31:00.940724: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2198780000 Hz
2019-09-10 14:31:00.943524: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x2d80910 executing computations on platform Host. Devices:
2019-09-10 14:31:00.943560: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): <undefined>, <undefined>
2019-09-10 14:31:00.947318: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcuda.so.1
2019-09-10 14:31:01.788284: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x2d80e20 executing computations on platform CUDA. Devices:
2019-09-10 14:31:01.788333: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): Tesla V100-DGXS-32GB, Compute Capability 7.0
2019-09-10 14:31:01.788349: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (1): Tesla V100-DGXS-32GB, Compute Capability 7.0
2019-09-10 14:31:01.788363: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (2): Tesla V100-DGXS-32GB, Compute Capability 7.0
2019-09-10 14:31:01.788376: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (3): Tesla V100-DGXS-32GB, Compute Capability 7.0
2019-09-10 14:31:01.791845: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:07:00.0
2019-09-10 14:31:01.794753: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 1 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:08:00.0
2019-09-10 14:31:01.797617: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 2 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:0e:00.0
2019-09-10 14:31:01.800465: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 3 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:0f:00.0
2019-09-10 14:31:01.800526: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
2019-09-10 14:31:01.803872: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10
2019-09-10 14:31:01.806009: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcufft.so.10
2019-09-10 14:31:01.806307: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcurand.so.10
2019-09-10 14:31:01.808057: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusolver.so.10
2019-09-10 14:31:01.809089: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusparse.so.10
2019-09-10 14:31:01.812747: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
2019-09-10 14:31:01.824786: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0, 1, 2, 3
2019-09-10 14:31:01.824822: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
2019-09-10 14:31:03.616053: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1181] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-09-10 14:31:03.616097: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1187]      0 1 2 3
2019-09-10 14:31:03.616107: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 0:   N Y Y Y
2019-09-10 14:31:03.616114: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 1:   Y N Y Y
2019-09-10 14:31:03.616121: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 2:   Y Y N Y
2019-09-10 14:31:03.616127: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 3:   Y Y Y N
2019-09-10 14:31:03.624240: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 30438 MB memory) -> physical GPU (device: 0, name: Tesla V100-DGXS-32GB, pci bus id: 0000:07:00.0, compute capability: 7.0)
2019-09-10 14:31:03.626439: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 30463 MB memory) -> physical GPU (device: 1, name: Tesla V100-DGXS-32GB, pci bus id: 0000:08:00.0, compute capability: 7.0)
2019-09-10 14:31:03.628592: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:2 with 30463 MB memory) -> physical GPU (device: 2, name: Tesla V100-DGXS-32GB, pci bus id: 0000:0e:00.0, compute capability: 7.0)
2019-09-10 14:31:03.630723: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:3 with 30463 MB memory) -> physical GPU (device: 3, name: Tesla V100-DGXS-32GB, pci bus id: 0000:0f:00.0, compute capability: 7.0)
TensorFlow:  1.14
Model:       resnet50
Dataset:     imagenet (synthetic)
Mode:        training
SingleSess:  False
Batch size:  384 global
             128 per device
Num batches: 100
Num epochs:  0.03
Devices:     ['/gpu:0', '/gpu:1', '/gpu:2']
NUMA bind:   False
Data format: NHWC
Optimizer:   sgd
Variables:   replicated
AllReduce:   None
==========
W0910 14:31:03.637035 139713216689984 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2762: The name tf.set_random_seed is deprecated. Please use tf.compat.v1.set_random_seed instead.

Generating training model
W0910 14:31:03.637629 139713216689984 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2782: The name tf.train.get_or_create_global_step is deprecated. Please use tf.compat.v1.train.get_or_create_global_step instead.

W0910 14:31:03.647360 139713216689984 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/variable_mgr.py:316: The name tf.variable_scope is deprecated. Please use tf.compat.v1.variable_scope instead.

W0910 14:31:03.648018 139713216689984 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/models/model.py:245: The name tf.truncated_normal is deprecated. Please use tf.random.truncated_normal instead.

W0910 14:31:03.665177 139713216689984 deprecation.py:323] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:129: conv2d (from tensorflow.python.layers.convolutional) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.keras.layers.Conv2D` instead.
W0910 14:31:04.041753 139713216689984 deprecation.py:323] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:261: max_pooling2d (from tensorflow.python.layers.pooling) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.MaxPooling2D instead.
W0910 14:31:06.613382 139713216689984 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:334: The name tf.nn.xw_plus_b is deprecated. Please use tf.compat.v1.nn.xw_plus_b instead.

W0910 14:31:06.615478 139713216689984 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/models/model.py:319: The name tf.losses.sparse_softmax_cross_entropy is deprecated. Please use tf.compat.v1.losses.sparse_softmax_cross_entropy instead.

W0910 14:31:06.627191 139713216689984 deprecation.py:323] From /usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/losses/losses_impl.py:121: add_dispatch_support.<locals>.wrapper (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.where in 2.0, which has the same broadcast rule as np.where
W0910 14:31:14.082817 139713216689984 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/models/resnet_model.py:333: The name tf.train.piecewise_constant is deprecated. Please use tf.compat.v1.train.piecewise_constant instead.

W0910 14:31:14.171029 139713216689984 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:1209: The name tf.train.GradientDescentOptimizer is deprecated. Please use tf.compat.v1.train.GradientDescentOptimizer instead.

W0910 14:31:14.845149 139713216689984 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2090: The name tf.train.get_global_step is deprecated. Please use tf.compat.v1.train.get_global_step instead.

W0910 14:31:15.620382 139713216689984 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2126: The name tf.summary.merge_all is deprecated. Please use tf.compat.v1.summary.merge_all instead.

Initializing graph
W0910 14:31:15.620633 139713216689984 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2183: The name tf.train.Saver is deprecated. Please use tf.compat.v1.train.Saver instead.

W0910 14:31:15.885851 139713216689984 deprecation.py:323] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2238: Supervisor.__init__ (from tensorflow.python.training.supervisor) is deprecated and will be removed in a future version.
Instructions for updating:
Please switch to tf.train.MonitoredTrainingSession
2019-09-10 14:31:17.393387: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:07:00.0
2019-09-10 14:31:17.395242: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 1 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:08:00.0
2019-09-10 14:31:17.396816: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 2 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:0e:00.0
2019-09-10 14:31:17.398514: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 3 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:0f:00.0
2019-09-10 14:31:17.398545: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
2019-09-10 14:31:17.398582: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10
2019-09-10 14:31:17.398601: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcufft.so.10
2019-09-10 14:31:17.398619: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcurand.so.10
2019-09-10 14:31:17.398636: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusolver.so.10
2019-09-10 14:31:17.398653: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusparse.so.10
2019-09-10 14:31:17.398671: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
2019-09-10 14:31:17.410674: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0, 1, 2, 3
2019-09-10 14:31:17.410736: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1181] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-09-10 14:31:17.410747: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1187]      0 1 2 3
2019-09-10 14:31:17.410755: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 0:   N Y Y Y
2019-09-10 14:31:17.410762: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 1:   Y N Y Y
2019-09-10 14:31:17.410771: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 2:   Y Y N Y
2019-09-10 14:31:17.410778: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 3:   Y Y Y N
2019-09-10 14:31:17.418533: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 30438 MB memory) -> physical GPU (device: 0, name: Tesla V100-DGXS-32GB, pci bus id: 0000:07:00.0, compute capability: 7.0)
2019-09-10 14:31:17.420180: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 30463 MB memory) -> physical GPU (device: 1, name: Tesla V100-DGXS-32GB, pci bus id: 0000:08:00.0, compute capability: 7.0)
2019-09-10 14:31:17.421786: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:2 with 30463 MB memory) -> physical GPU (device: 2, name: Tesla V100-DGXS-32GB, pci bus id: 0000:0e:00.0, compute capability: 7.0)
2019-09-10 14:31:17.423369: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:3 with 30463 MB memory) -> physical GPU (device: 3, name: Tesla V100-DGXS-32GB, pci bus id: 0000:0f:00.0, compute capability: 7.0)
2019-09-10 14:31:19.098435: W tensorflow/compiler/jit/mark_for_compilation_pass.cc:1412] (One-time warning): Not using XLA:CPU for cluster because envvar TF_XLA_FLAGS=--tf_xla_cpu_global_jit was not set.  If you want XLA:CPU, either set that envvar, or use experimental_jit_scope to enable XLA:CPU.  To confirm that XLA is active, pass --vmodule=xla_compilation_cache=1 (as a proper command-line flag, not via TF_XLA_FLAGS) or set the envvar XLA_FLAGS=--xla_hlo_profile.
I0910 14:31:21.618732 139713216689984 session_manager.py:500] Running local_init_op.
I0910 14:31:22.686068 139713216689984 session_manager.py:502] Done running local_init_op.
Running warm up
2019-09-10 14:31:26.611475: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10
2019-09-10 14:31:27.638775: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
Done warm up
Step    Img/sec total_loss
1       images/sec: 1149.6 +/- 0.0 (jitter = 0.0)       7.894
10      images/sec: 1148.6 +/- 0.6 (jitter = 1.5)       7.914
20      images/sec: 1148.6 +/- 0.3 (jitter = 1.0)       7.929
30      images/sec: 1148.3 +/- 0.2 (jitter = 1.1)       7.821
40      images/sec: 1148.1 +/- 0.2 (jitter = 1.2)       7.887
50      images/sec: 1148.1 +/- 0.2 (jitter = 1.3)       7.840
60      images/sec: 1148.1 +/- 0.2 (jitter = 1.4)       7.807
70      images/sec: 1147.9 +/- 0.2 (jitter = 1.5)       7.780
80      images/sec: 1147.6 +/- 0.2 (jitter = 1.8)       7.796
90      images/sec: 1147.5 +/- 0.2 (jitter = 1.8)       7.732
100     images/sec: 1147.3 +/- 0.2 (jitter = 1.8)       7.750
----------------------------------------------------------------
total images/sec: 1147.10
----------------------------------------------------------------
2019-09-10 14:32:14.609965: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
WARNING: Logging before flag parsing goes to stderr.
W0910 14:32:16.587633 140460509329216 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/models/mobilenet.py:388: The name tf.nn.avg_pool is deprecated. Please use tf.nn.avg_pool2d instead.

W0910 14:32:16.592045 140460509329216 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/models/experimental/deepspeech.py:131: The name tf.nn.rnn_cell.RNNCell is deprecated. Please use tf.compat.v1.nn.rnn_cell.RNNCell instead.

W0910 14:32:16.615108 140460509329216 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:722: The name tf.ConfigProto is deprecated. Please use tf.compat.v1.ConfigProto instead.

W0910 14:32:16.615323 140460509329216 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:3503: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead.

2019-09-10 14:32:16.640722: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2198780000 Hz
2019-09-10 14:32:16.643587: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x2bdce90 executing computations on platform Host. Devices:
2019-09-10 14:32:16.643626: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): <undefined>, <undefined>
2019-09-10 14:32:16.647652: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcuda.so.1
2019-09-10 14:32:17.607048: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x2bdd1f0 executing computations on platform CUDA. Devices:
2019-09-10 14:32:17.607097: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): Tesla V100-DGXS-32GB, Compute Capability 7.0
2019-09-10 14:32:17.607114: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (1): Tesla V100-DGXS-32GB, Compute Capability 7.0
2019-09-10 14:32:17.607128: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (2): Tesla V100-DGXS-32GB, Compute Capability 7.0
2019-09-10 14:32:17.607140: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (3): Tesla V100-DGXS-32GB, Compute Capability 7.0
2019-09-10 14:32:17.610448: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:07:00.0
2019-09-10 14:32:17.613143: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 1 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:08:00.0
2019-09-10 14:32:17.615800: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 2 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:0e:00.0
2019-09-10 14:32:17.618468: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 3 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:0f:00.0
2019-09-10 14:32:17.618512: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
2019-09-10 14:32:17.621288: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10
2019-09-10 14:32:17.623048: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcufft.so.10
2019-09-10 14:32:17.623389: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcurand.so.10
2019-09-10 14:32:17.625416: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusolver.so.10
2019-09-10 14:32:17.626590: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusparse.so.10
2019-09-10 14:32:17.630782: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
2019-09-10 14:32:17.644455: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0, 1, 2, 3
2019-09-10 14:32:17.644504: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
2019-09-10 14:32:19.444678: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1181] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-09-10 14:32:19.444720: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1187]      0 1 2 3
2019-09-10 14:32:19.444731: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 0:   N Y Y Y
2019-09-10 14:32:19.444739: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 1:   Y N Y Y
2019-09-10 14:32:19.444745: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 2:   Y Y N Y
2019-09-10 14:32:19.444751: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 3:   Y Y Y N
2019-09-10 14:32:19.453029: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 30438 MB memory) -> physical GPU (device: 0, name: Tesla V100-DGXS-32GB, pci bus id: 0000:07:00.0, compute capability: 7.0)
2019-09-10 14:32:19.455246: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 30463 MB memory) -> physical GPU (device: 1, name: Tesla V100-DGXS-32GB, pci bus id: 0000:08:00.0, compute capability: 7.0)
2019-09-10 14:32:19.457397: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:2 with 30463 MB memory) -> physical GPU (device: 2, name: Tesla V100-DGXS-32GB, pci bus id: 0000:0e:00.0, compute capability: 7.0)
2019-09-10 14:32:19.459538: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:3 with 30463 MB memory) -> physical GPU (device: 3, name: Tesla V100-DGXS-32GB, pci bus id: 0000:0f:00.0, compute capability: 7.0)
TensorFlow:  1.14
Model:       resnet50
Dataset:     imagenet (synthetic)
Mode:        training
SingleSess:  False
Batch size:  384 global
             128 per device
Num batches: 100
Num epochs:  0.03
Devices:     ['/gpu:0', '/gpu:1', '/gpu:2']
NUMA bind:   False
Data format: NHWC
Optimizer:   sgd
Variables:   replicated
AllReduce:   None
==========
W0910 14:32:19.465443 140460509329216 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2762: The name tf.set_random_seed is deprecated. Please use tf.compat.v1.set_random_seed instead.

Generating training model
W0910 14:32:19.466017 140460509329216 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2782: The name tf.train.get_or_create_global_step is deprecated. Please use tf.compat.v1.train.get_or_create_global_step instead.

W0910 14:32:19.475411 140460509329216 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2739: The name tf.get_variable is deprecated. Please use tf.compat.v1.get_variable instead.

W0910 14:32:19.480674 140460509329216 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/variable_mgr.py:316: The name tf.variable_scope is deprecated. Please use tf.compat.v1.variable_scope instead.

W0910 14:32:19.497605 140460509329216 deprecation.py:323] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:129: conv2d (from tensorflow.python.layers.convolutional) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.keras.layers.Conv2D` instead.
W0910 14:32:19.866362 140460509329216 deprecation.py:323] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:261: max_pooling2d (from tensorflow.python.layers.pooling) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.MaxPooling2D instead.
W0910 14:32:22.526313 140460509329216 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:334: The name tf.nn.xw_plus_b is deprecated. Please use tf.compat.v1.nn.xw_plus_b instead.

W0910 14:32:22.529229 140460509329216 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/models/model.py:319: The name tf.losses.sparse_softmax_cross_entropy is deprecated. Please use tf.compat.v1.losses.sparse_softmax_cross_entropy instead.

W0910 14:32:22.540902 140460509329216 deprecation.py:323] From /usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/losses/losses_impl.py:121: add_dispatch_support.<locals>.wrapper (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.where in 2.0, which has the same broadcast rule as np.where
W0910 14:32:31.768654 140460509329216 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/models/resnet_model.py:333: The name tf.train.piecewise_constant is deprecated. Please use tf.compat.v1.train.piecewise_constant instead.

W0910 14:32:31.855767 140460509329216 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:1209: The name tf.train.GradientDescentOptimizer is deprecated. Please use tf.compat.v1.train.GradientDescentOptimizer instead.

W0910 14:32:32.681496 140460509329216 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2090: The name tf.train.get_global_step is deprecated. Please use tf.compat.v1.train.get_global_step instead.

W0910 14:32:33.447739 140460509329216 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2126: The name tf.summary.merge_all is deprecated. Please use tf.compat.v1.summary.merge_all instead.

Initializing graph
W0910 14:32:33.447987 140460509329216 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2183: The name tf.train.Saver is deprecated. Please use tf.compat.v1.train.Saver instead.

W0910 14:32:33.711128 140460509329216 deprecation.py:323] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2238: Supervisor.__init__ (from tensorflow.python.training.supervisor) is deprecated and will be removed in a future version.
Instructions for updating:
Please switch to tf.train.MonitoredTrainingSession
2019-09-10 14:32:35.168059: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:07:00.0
2019-09-10 14:32:35.169649: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 1 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:08:00.0
2019-09-10 14:32:35.171173: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 2 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:0e:00.0
2019-09-10 14:32:35.172755: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 3 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:0f:00.0
2019-09-10 14:32:35.172786: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
2019-09-10 14:32:35.172821: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10
2019-09-10 14:32:35.172840: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcufft.so.10
2019-09-10 14:32:35.172857: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcurand.so.10
2019-09-10 14:32:35.172874: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusolver.so.10
2019-09-10 14:32:35.172890: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusparse.so.10
2019-09-10 14:32:35.172910: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
2019-09-10 14:32:35.185408: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0, 1, 2, 3
2019-09-10 14:32:35.185469: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1181] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-09-10 14:32:35.185480: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1187]      0 1 2 3
2019-09-10 14:32:35.185488: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 0:   N Y Y Y
2019-09-10 14:32:35.185494: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 1:   Y N Y Y
2019-09-10 14:32:35.185501: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 2:   Y Y N Y
2019-09-10 14:32:35.185508: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 3:   Y Y Y N
2019-09-10 14:32:35.193297: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 30438 MB memory) -> physical GPU (device: 0, name: Tesla V100-DGXS-32GB, pci bus id: 0000:07:00.0, compute capability: 7.0)
2019-09-10 14:32:35.194894: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 30463 MB memory) -> physical GPU (device: 1, name: Tesla V100-DGXS-32GB, pci bus id: 0000:08:00.0, compute capability: 7.0)
2019-09-10 14:32:35.196481: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:2 with 30463 MB memory) -> physical GPU (device: 2, name: Tesla V100-DGXS-32GB, pci bus id: 0000:0e:00.0, compute capability: 7.0)
2019-09-10 14:32:35.198087: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:3 with 30463 MB memory) -> physical GPU (device: 3, name: Tesla V100-DGXS-32GB, pci bus id: 0000:0f:00.0, compute capability: 7.0)
2019-09-10 14:32:36.967002: W tensorflow/compiler/jit/mark_for_compilation_pass.cc:1412] (One-time warning): Not using XLA:CPU for cluster because envvar TF_XLA_FLAGS=--tf_xla_cpu_global_jit was not set.  If you want XLA:CPU, either set that envvar, or use experimental_jit_scope to enable XLA:CPU.  To confirm that XLA is active, pass --vmodule=xla_compilation_cache=1 (as a proper command-line flag, not via TF_XLA_FLAGS) or set the envvar XLA_FLAGS=--xla_hlo_profile.
I0910 14:32:39.493895 140460509329216 session_manager.py:500] Running local_init_op.
I0910 14:32:40.576540 140460509329216 session_manager.py:502] Done running local_init_op.
Running warm up
2019-09-10 14:32:45.436112: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10
2019-09-10 14:32:46.536391: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
Done warm up
Step    Img/sec total_loss
1       images/sec: 2175.0 +/- 0.0 (jitter = 0.0)       7.807
10      images/sec: 2175.4 +/- 2.6 (jitter = 9.7)       7.885
20      images/sec: 2178.1 +/- 1.6 (jitter = 7.2)       7.872
30      images/sec: 2178.3 +/- 1.3 (jitter = 6.4)       7.828
40      images/sec: 2178.7 +/- 1.1 (jitter = 6.3)       7.835
50      images/sec: 2177.7 +/- 1.1 (jitter = 6.8)       7.777
60      images/sec: 2177.2 +/- 1.0 (jitter = 7.8)       7.759
70      images/sec: 2177.1 +/- 1.0 (jitter = 7.7)       7.777
80      images/sec: 2177.1 +/- 0.9 (jitter = 7.9)       7.775
90      images/sec: 2176.4 +/- 0.8 (jitter = 8.4)       7.766
100     images/sec: 2176.3 +/- 0.8 (jitter = 7.9)       7.755
----------------------------------------------------------------
total images/sec: 2175.48
----------------------------------------------------------------
2019-09-10 14:33:15.866450: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
WARNING: Logging before flag parsing goes to stderr.
W0910 14:33:17.905010 140656632866624 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/models/mobilenet.py:388: The name tf.nn.avg_pool is deprecated. Please use tf.nn.avg_pool2d instead.

W0910 14:33:17.909595 140656632866624 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/models/experimental/deepspeech.py:131: The name tf.nn.rnn_cell.RNNCell is deprecated. Please use tf.compat.v1.nn.rnn_cell.RNNCell instead.

W0910 14:33:17.933259 140656632866624 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:722: The name tf.ConfigProto is deprecated. Please use tf.compat.v1.ConfigProto instead.

W0910 14:33:17.933474 140656632866624 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:3503: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead.

2019-09-10 14:33:17.956725: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2198780000 Hz
2019-09-10 14:33:17.959067: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x3ab88a0 executing computations on platform Host. Devices:
2019-09-10 14:33:17.959095: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): <undefined>, <undefined>
2019-09-10 14:33:17.961790: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcuda.so.1
2019-09-10 14:33:18.850970: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x3ab8c00 executing computations on platform CUDA. Devices:
2019-09-10 14:33:18.851004: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): Tesla V100-DGXS-32GB, Compute Capability 7.0
2019-09-10 14:33:18.851017: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (1): Tesla V100-DGXS-32GB, Compute Capability 7.0
2019-09-10 14:33:18.851025: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (2): Tesla V100-DGXS-32GB, Compute Capability 7.0
2019-09-10 14:33:18.851034: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (3): Tesla V100-DGXS-32GB, Compute Capability 7.0
2019-09-10 14:33:18.853408: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:07:00.0
2019-09-10 14:33:18.855362: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 1 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:08:00.0
2019-09-10 14:33:18.857314: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 2 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:0e:00.0
2019-09-10 14:33:18.859290: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 3 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:0f:00.0
2019-09-10 14:33:18.859325: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
2019-09-10 14:33:18.861556: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10
2019-09-10 14:33:18.863520: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcufft.so.10
2019-09-10 14:33:18.863891: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcurand.so.10
2019-09-10 14:33:18.865769: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusolver.so.10
2019-09-10 14:33:18.866755: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusparse.so.10
2019-09-10 14:33:18.870352: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
2019-09-10 14:33:18.882180: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0, 1, 2, 3
2019-09-10 14:33:18.882213: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
2019-09-10 14:33:20.650176: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1181] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-09-10 14:33:20.650219: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1187]      0 1 2 3
2019-09-10 14:33:20.650230: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 0:   N Y Y Y
2019-09-10 14:33:20.650237: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 1:   Y N Y Y
2019-09-10 14:33:20.650243: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 2:   Y Y N Y
2019-09-10 14:33:20.650249: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 3:   Y Y Y N
2019-09-10 14:33:20.658206: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 30438 MB memory) -> physical GPU (device: 0, name: Tesla V100-DGXS-32GB, pci bus id: 0000:07:00.0, compute capability: 7.0)
2019-09-10 14:33:20.660421: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 30463 MB memory) -> physical GPU (device: 1, name: Tesla V100-DGXS-32GB, pci bus id: 0000:08:00.0, compute capability: 7.0)
2019-09-10 14:33:20.662605: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:2 with 30463 MB memory) -> physical GPU (device: 2, name: Tesla V100-DGXS-32GB, pci bus id: 0000:0e:00.0, compute capability: 7.0)
2019-09-10 14:33:20.664798: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:3 with 30463 MB memory) -> physical GPU (device: 3, name: Tesla V100-DGXS-32GB, pci bus id: 0000:0f:00.0, compute capability: 7.0)
TensorFlow:  1.14
Model:       resnet50
Dataset:     imagenet (synthetic)
Mode:        training
SingleSess:  False
Batch size:  768 global
             256 per device
Num batches: 100
Num epochs:  0.06
Devices:     ['/gpu:0', '/gpu:1', '/gpu:2']
NUMA bind:   False
Data format: NHWC
Optimizer:   sgd
Variables:   replicated
AllReduce:   None
==========
W0910 14:33:20.669903 140656632866624 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2762: The name tf.set_random_seed is deprecated. Please use tf.compat.v1.set_random_seed instead.

Generating training model
W0910 14:33:20.670258 140656632866624 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2782: The name tf.train.get_or_create_global_step is deprecated. Please use tf.compat.v1.train.get_or_create_global_step instead.

W0910 14:33:20.675268 140656632866624 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/variable_mgr.py:316: The name tf.variable_scope is deprecated. Please use tf.compat.v1.variable_scope instead.

W0910 14:33:20.675594 140656632866624 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/models/model.py:245: The name tf.truncated_normal is deprecated. Please use tf.random.truncated_normal instead.

W0910 14:33:20.689463 140656632866624 deprecation.py:323] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:129: conv2d (from tensorflow.python.layers.convolutional) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.keras.layers.Conv2D` instead.
W0910 14:33:21.041664 140656632866624 deprecation.py:323] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:261: max_pooling2d (from tensorflow.python.layers.pooling) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.MaxPooling2D instead.
W0910 14:33:23.559040 140656632866624 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:334: The name tf.nn.xw_plus_b is deprecated. Please use tf.compat.v1.nn.xw_plus_b instead.

W0910 14:33:23.561100 140656632866624 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/models/model.py:319: The name tf.losses.sparse_softmax_cross_entropy is deprecated. Please use tf.compat.v1.losses.sparse_softmax_cross_entropy instead.

W0910 14:33:23.572565 140656632866624 deprecation.py:323] From /usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/losses/losses_impl.py:121: add_dispatch_support.<locals>.wrapper (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.where in 2.0, which has the same broadcast rule as np.where
W0910 14:33:30.864940 140656632866624 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/models/resnet_model.py:333: The name tf.train.piecewise_constant is deprecated. Please use tf.compat.v1.train.piecewise_constant instead.

W0910 14:33:30.951225 140656632866624 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:1209: The name tf.train.GradientDescentOptimizer is deprecated. Please use tf.compat.v1.train.GradientDescentOptimizer instead.

W0910 14:33:31.605484 140656632866624 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2090: The name tf.train.get_global_step is deprecated. Please use tf.compat.v1.train.get_global_step instead.

W0910 14:33:32.362595 140656632866624 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2126: The name tf.summary.merge_all is deprecated. Please use tf.compat.v1.summary.merge_all instead.

Initializing graph
W0910 14:33:32.362834 140656632866624 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2183: The name tf.train.Saver is deprecated. Please use tf.compat.v1.train.Saver instead.

W0910 14:33:32.621598 140656632866624 deprecation.py:323] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2238: Supervisor.__init__ (from tensorflow.python.training.supervisor) is deprecated and will be removed in a future version.
Instructions for updating:
Please switch to tf.train.MonitoredTrainingSession
2019-09-10 14:33:34.100666: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:07:00.0
2019-09-10 14:33:34.102496: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 1 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:08:00.0
2019-09-10 14:33:34.104035: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 2 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:0e:00.0
2019-09-10 14:33:34.105685: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 3 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:0f:00.0
2019-09-10 14:33:34.105717: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
2019-09-10 14:33:34.105751: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10
2019-09-10 14:33:34.105769: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcufft.so.10
2019-09-10 14:33:34.105786: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcurand.so.10
2019-09-10 14:33:34.105802: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusolver.so.10
2019-09-10 14:33:34.105818: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusparse.so.10
2019-09-10 14:33:34.105833: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
2019-09-10 14:33:34.117745: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0, 1, 2, 3
2019-09-10 14:33:34.117806: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1181] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-09-10 14:33:34.117816: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1187]      0 1 2 3
2019-09-10 14:33:34.117824: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 0:   N Y Y Y
2019-09-10 14:33:34.117831: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 1:   Y N Y Y
2019-09-10 14:33:34.117838: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 2:   Y Y N Y
2019-09-10 14:33:34.117845: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 3:   Y Y Y N
2019-09-10 14:33:34.125567: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 30438 MB memory) -> physical GPU (device: 0, name: Tesla V100-DGXS-32GB, pci bus id: 0000:07:00.0, compute capability: 7.0)
2019-09-10 14:33:34.127183: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 30463 MB memory) -> physical GPU (device: 1, name: Tesla V100-DGXS-32GB, pci bus id: 0000:08:00.0, compute capability: 7.0)
2019-09-10 14:33:34.128771: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:2 with 30463 MB memory) -> physical GPU (device: 2, name: Tesla V100-DGXS-32GB, pci bus id: 0000:0e:00.0, compute capability: 7.0)
2019-09-10 14:33:34.130360: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:3 with 30463 MB memory) -> physical GPU (device: 3, name: Tesla V100-DGXS-32GB, pci bus id: 0000:0f:00.0, compute capability: 7.0)
2019-09-10 14:33:35.773982: W tensorflow/compiler/jit/mark_for_compilation_pass.cc:1412] (One-time warning): Not using XLA:CPU for cluster because envvar TF_XLA_FLAGS=--tf_xla_cpu_global_jit was not set.  If you want XLA:CPU, either set that envvar, or use experimental_jit_scope to enable XLA:CPU.  To confirm that XLA is active, pass --vmodule=xla_compilation_cache=1 (as a proper command-line flag, not via TF_XLA_FLAGS) or set the envvar XLA_FLAGS=--xla_hlo_profile.
I0910 14:33:38.281344 140656632866624 session_manager.py:500] Running local_init_op.
I0910 14:33:39.331176 140656632866624 session_manager.py:502] Done running local_init_op.
Running warm up
2019-09-10 14:33:43.195456: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10
2019-09-10 14:33:44.219971: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
Done warm up
Step    Img/sec total_loss
1       images/sec: 1192.5 +/- 0.0 (jitter = 0.0)       7.916
10      images/sec: 1191.6 +/- 0.5 (jitter = 1.6)       7.829
20      images/sec: 1192.3 +/- 0.4 (jitter = 1.7)       7.843
30      images/sec: 1192.0 +/- 0.3 (jitter = 1.8)       7.825
40      images/sec: 1191.8 +/- 0.3 (jitter = 1.9)       7.767
50      images/sec: 1191.7 +/- 0.2 (jitter = 1.8)       7.742
60      images/sec: 1191.4 +/- 0.2 (jitter = 2.1)       7.701
70      images/sec: 1191.2 +/- 0.2 (jitter = 2.1)       7.699
80      images/sec: 1191.0 +/- 0.2 (jitter = 2.2)       7.680
90      images/sec: 1191.0 +/- 0.2 (jitter = 2.1)       7.621
100     images/sec: 1190.9 +/- 0.2 (jitter = 2.1)       7.635
----------------------------------------------------------------
total images/sec: 1190.78
----------------------------------------------------------------
2019-09-10 14:35:07.601698: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
WARNING: Logging before flag parsing goes to stderr.
W0910 14:35:09.625108 140291211429696 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/models/mobilenet.py:388: The name tf.nn.avg_pool is deprecated. Please use tf.nn.avg_pool2d instead.

W0910 14:35:09.629648 140291211429696 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/models/experimental/deepspeech.py:131: The name tf.nn.rnn_cell.RNNCell is deprecated. Please use tf.compat.v1.nn.rnn_cell.RNNCell instead.

W0910 14:35:09.653266 140291211429696 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:722: The name tf.ConfigProto is deprecated. Please use tf.compat.v1.ConfigProto instead.

W0910 14:35:09.653477 140291211429696 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:3503: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead.

2019-09-10 14:35:09.676691: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2198780000 Hz
2019-09-10 14:35:09.679294: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x33f44d0 executing computations on platform Host. Devices:
2019-09-10 14:35:09.679327: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): <undefined>, <undefined>
2019-09-10 14:35:09.682479: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcuda.so.1
2019-09-10 14:35:10.529886: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x33eda30 executing computations on platform CUDA. Devices:
2019-09-10 14:35:10.529926: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): Tesla V100-DGXS-32GB, Compute Capability 7.0
2019-09-10 14:35:10.529939: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (1): Tesla V100-DGXS-32GB, Compute Capability 7.0
2019-09-10 14:35:10.529950: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (2): Tesla V100-DGXS-32GB, Compute Capability 7.0
2019-09-10 14:35:10.529960: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (3): Tesla V100-DGXS-32GB, Compute Capability 7.0
2019-09-10 14:35:10.532725: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:07:00.0
2019-09-10 14:35:10.534964: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 1 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:08:00.0
2019-09-10 14:35:10.537237: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 2 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:0e:00.0
2019-09-10 14:35:10.539485: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 3 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:0f:00.0
2019-09-10 14:35:10.539528: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
2019-09-10 14:35:10.542103: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10
2019-09-10 14:35:10.544351: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcufft.so.10
2019-09-10 14:35:10.544780: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcurand.so.10
2019-09-10 14:35:10.546496: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusolver.so.10
2019-09-10 14:35:10.547494: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusparse.so.10
2019-09-10 14:35:10.551107: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
2019-09-10 14:35:10.563043: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0, 1, 2, 3
2019-09-10 14:35:10.563082: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
2019-09-10 14:35:12.345276: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1181] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-09-10 14:35:12.345312: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1187]      0 1 2 3
2019-09-10 14:35:12.345323: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 0:   N Y Y Y
2019-09-10 14:35:12.345330: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 1:   Y N Y Y
2019-09-10 14:35:12.345337: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 2:   Y Y N Y
2019-09-10 14:35:12.345344: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 3:   Y Y Y N
2019-09-10 14:35:12.353353: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 30438 MB memory) -> physical GPU (device: 0, name: Tesla V100-DGXS-32GB, pci bus id: 0000:07:00.0, compute capability: 7.0)
2019-09-10 14:35:12.355553: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 30463 MB memory) -> physical GPU (device: 1, name: Tesla V100-DGXS-32GB, pci bus id: 0000:08:00.0, compute capability: 7.0)
2019-09-10 14:35:12.357723: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:2 with 30463 MB memory) -> physical GPU (device: 2, name: Tesla V100-DGXS-32GB, pci bus id: 0000:0e:00.0, compute capability: 7.0)
2019-09-10 14:35:12.359861: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:3 with 30463 MB memory) -> physical GPU (device: 3, name: Tesla V100-DGXS-32GB, pci bus id: 0000:0f:00.0, compute capability: 7.0)
TensorFlow:  1.14
Model:       resnet50
Dataset:     imagenet (synthetic)
Mode:        training
SingleSess:  False
Batch size:  768 global
             256 per device
Num batches: 100
Num epochs:  0.06
Devices:     ['/gpu:0', '/gpu:1', '/gpu:2']
NUMA bind:   False
Data format: NHWC
Optimizer:   sgd
Variables:   replicated
AllReduce:   None
==========
W0910 14:35:12.366173 140291211429696 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2762: The name tf.set_random_seed is deprecated. Please use tf.compat.v1.set_random_seed instead.

Generating training model
W0910 14:35:12.366757 140291211429696 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2782: The name tf.train.get_or_create_global_step is deprecated. Please use tf.compat.v1.train.get_or_create_global_step instead.

W0910 14:35:12.374750 140291211429696 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2739: The name tf.get_variable is deprecated. Please use tf.compat.v1.get_variable instead.

W0910 14:35:12.379552 140291211429696 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/variable_mgr.py:316: The name tf.variable_scope is deprecated. Please use tf.compat.v1.variable_scope instead.

W0910 14:35:12.396618 140291211429696 deprecation.py:323] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:129: conv2d (from tensorflow.python.layers.convolutional) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.keras.layers.Conv2D` instead.
W0910 14:35:12.762197 140291211429696 deprecation.py:323] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:261: max_pooling2d (from tensorflow.python.layers.pooling) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.MaxPooling2D instead.
W0910 14:35:15.441936 140291211429696 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:334: The name tf.nn.xw_plus_b is deprecated. Please use tf.compat.v1.nn.xw_plus_b instead.

W0910 14:35:15.444860 140291211429696 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/models/model.py:319: The name tf.losses.sparse_softmax_cross_entropy is deprecated. Please use tf.compat.v1.losses.sparse_softmax_cross_entropy instead.

W0910 14:35:15.456628 140291211429696 deprecation.py:323] From /usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/losses/losses_impl.py:121: add_dispatch_support.<locals>.wrapper (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.where in 2.0, which has the same broadcast rule as np.where
W0910 14:35:24.748151 140291211429696 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/models/resnet_model.py:333: The name tf.train.piecewise_constant is deprecated. Please use tf.compat.v1.train.piecewise_constant instead.

W0910 14:35:24.835855 140291211429696 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:1209: The name tf.train.GradientDescentOptimizer is deprecated. Please use tf.compat.v1.train.GradientDescentOptimizer instead.

W0910 14:35:25.665865 140291211429696 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2090: The name tf.train.get_global_step is deprecated. Please use tf.compat.v1.train.get_global_step instead.

W0910 14:35:26.437985 140291211429696 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2126: The name tf.summary.merge_all is deprecated. Please use tf.compat.v1.summary.merge_all instead.

Initializing graph
W0910 14:35:26.438231 140291211429696 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2183: The name tf.train.Saver is deprecated. Please use tf.compat.v1.train.Saver instead.

W0910 14:35:26.702493 140291211429696 deprecation.py:323] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2238: Supervisor.__init__ (from tensorflow.python.training.supervisor) is deprecated and will be removed in a future version.
Instructions for updating:
Please switch to tf.train.MonitoredTrainingSession
2019-09-10 14:35:28.176218: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:07:00.0
2019-09-10 14:35:28.177812: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 1 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:08:00.0
2019-09-10 14:35:28.179423: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 2 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:0e:00.0
2019-09-10 14:35:28.181154: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 3 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:0f:00.0
2019-09-10 14:35:28.181188: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
2019-09-10 14:35:28.181234: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10
2019-09-10 14:35:28.181257: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcufft.so.10
2019-09-10 14:35:28.181277: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcurand.so.10
2019-09-10 14:35:28.181297: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusolver.so.10
2019-09-10 14:35:28.181317: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusparse.so.10
2019-09-10 14:35:28.181337: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
2019-09-10 14:35:28.193578: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0, 1, 2, 3
2019-09-10 14:35:28.193642: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1181] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-09-10 14:35:28.193653: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1187]      0 1 2 3
2019-09-10 14:35:28.193662: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 0:   N Y Y Y
2019-09-10 14:35:28.193669: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 1:   Y N Y Y
2019-09-10 14:35:28.193676: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 2:   Y Y N Y
2019-09-10 14:35:28.193682: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 3:   Y Y Y N
2019-09-10 14:35:28.201526: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 30438 MB memory) -> physical GPU (device: 0, name: Tesla V100-DGXS-32GB, pci bus id: 0000:07:00.0, compute capability: 7.0)
2019-09-10 14:35:28.203125: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 30463 MB memory) -> physical GPU (device: 1, name: Tesla V100-DGXS-32GB, pci bus id: 0000:08:00.0, compute capability: 7.0)
2019-09-10 14:35:28.204720: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:2 with 30463 MB memory) -> physical GPU (device: 2, name: Tesla V100-DGXS-32GB, pci bus id: 0000:0e:00.0, compute capability: 7.0)
2019-09-10 14:35:28.206300: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:3 with 30463 MB memory) -> physical GPU (device: 3, name: Tesla V100-DGXS-32GB, pci bus id: 0000:0f:00.0, compute capability: 7.0)
2019-09-10 14:35:30.003689: W tensorflow/compiler/jit/mark_for_compilation_pass.cc:1412] (One-time warning): Not using XLA:CPU for cluster because envvar TF_XLA_FLAGS=--tf_xla_cpu_global_jit was not set.  If you want XLA:CPU, either set that envvar, or use experimental_jit_scope to enable XLA:CPU.  To confirm that XLA is active, pass --vmodule=xla_compilation_cache=1 (as a proper command-line flag, not via TF_XLA_FLAGS) or set the envvar XLA_FLAGS=--xla_hlo_profile.
I0910 14:35:32.526953 140291211429696 session_manager.py:500] Running local_init_op.
I0910 14:35:33.634510 140291211429696 session_manager.py:502] Done running local_init_op.
Running warm up
2019-09-10 14:35:38.550253: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10
2019-09-10 14:35:39.657877: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
Done warm up
Step    Img/sec total_loss
1       images/sec: 2401.6 +/- 0.0 (jitter = 0.0)       7.908
10      images/sec: 2412.1 +/- 2.3 (jitter = 8.9)       7.929
20      images/sec: 2412.3 +/- 1.7 (jitter = 9.4)       7.830
30      images/sec: 2409.2 +/- 1.6 (jitter = 8.4)       7.826
40      images/sec: 2408.9 +/- 1.2 (jitter = 7.5)       7.757
50      images/sec: 2409.0 +/- 1.1 (jitter = 8.4)       7.784
60      images/sec: 2408.6 +/- 1.0 (jitter = 7.5)       7.744
70      images/sec: 2408.0 +/- 0.9 (jitter = 6.8)       7.679
80      images/sec: 2407.6 +/- 0.8 (jitter = 6.8)       7.649
90      images/sec: 2407.7 +/- 0.8 (jitter = 6.6)       7.618
100     images/sec: 2407.4 +/- 0.7 (jitter = 6.6)       7.575
----------------------------------------------------------------
total images/sec: 2406.89
----------------------------------------------------------------
2019-09-10 14:36:26.756683: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
WARNING: Logging before flag parsing goes to stderr.
W0910 14:36:28.777641 140153510606656 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/models/mobilenet.py:388: The name tf.nn.avg_pool is deprecated. Please use tf.nn.avg_pool2d instead.

W0910 14:36:28.782037 140153510606656 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/models/experimental/deepspeech.py:131: The name tf.nn.rnn_cell.RNNCell is deprecated. Please use tf.compat.v1.nn.rnn_cell.RNNCell instead.

W0910 14:36:28.805100 140153510606656 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:722: The name tf.ConfigProto is deprecated. Please use tf.compat.v1.ConfigProto instead.

W0910 14:36:28.805310 140153510606656 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:3503: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead.

2019-09-10 14:36:28.828723: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2198780000 Hz
2019-09-10 14:36:28.831522: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x462cb10 executing computations on platform Host. Devices:
2019-09-10 14:36:28.831563: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): <undefined>, <undefined>
2019-09-10 14:36:28.835566: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcuda.so.1
2019-09-10 14:36:29.683237: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x5ea1e90 executing computations on platform CUDA. Devices:
2019-09-10 14:36:29.683275: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): Tesla V100-DGXS-32GB, Compute Capability 7.0
2019-09-10 14:36:29.683287: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (1): Tesla V100-DGXS-32GB, Compute Capability 7.0
2019-09-10 14:36:29.683297: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (2): Tesla V100-DGXS-32GB, Compute Capability 7.0
2019-09-10 14:36:29.683306: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (3): Tesla V100-DGXS-32GB, Compute Capability 7.0
2019-09-10 14:36:29.686131: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:07:00.0
2019-09-10 14:36:29.688632: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 1 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:08:00.0
2019-09-10 14:36:29.691109: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 2 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:0e:00.0
2019-09-10 14:36:29.693613: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 3 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:0f:00.0
2019-09-10 14:36:29.693654: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
2019-09-10 14:36:29.696517: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10
2019-09-10 14:36:29.698348: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcufft.so.10
2019-09-10 14:36:29.698682: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcurand.so.10
2019-09-10 14:36:29.700715: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusolver.so.10
2019-09-10 14:36:29.701908: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusparse.so.10
2019-09-10 14:36:29.706148: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
2019-09-10 14:36:29.719880: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0, 1, 2, 3
2019-09-10 14:36:29.719917: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
2019-09-10 14:36:31.536367: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1181] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-09-10 14:36:31.536408: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1187]      0 1 2 3
2019-09-10 14:36:31.536420: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 0:   N Y Y Y
2019-09-10 14:36:31.536429: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 1:   Y N Y Y
2019-09-10 14:36:31.536436: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 2:   Y Y N Y
2019-09-10 14:36:31.536442: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 3:   Y Y Y N
2019-09-10 14:36:31.544454: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 30438 MB memory) -> physical GPU (device: 0, name: Tesla V100-DGXS-32GB, pci bus id: 0000:07:00.0, compute capability: 7.0)
2019-09-10 14:36:31.546716: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 30463 MB memory) -> physical GPU (device: 1, name: Tesla V100-DGXS-32GB, pci bus id: 0000:08:00.0, compute capability: 7.0)
2019-09-10 14:36:31.548895: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:2 with 30463 MB memory) -> physical GPU (device: 2, name: Tesla V100-DGXS-32GB, pci bus id: 0000:0e:00.0, compute capability: 7.0)
2019-09-10 14:36:31.551042: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:3 with 30463 MB memory) -> physical GPU (device: 3, name: Tesla V100-DGXS-32GB, pci bus id: 0000:0f:00.0, compute capability: 7.0)
TensorFlow:  1.14
Model:       resnet50
Dataset:     imagenet (synthetic)
Mode:        training
SingleSess:  False
Batch size:  1536 global
             512 per device
Num batches: 100
Num epochs:  0.12
Devices:     ['/gpu:0', '/gpu:1', '/gpu:2']
NUMA bind:   False
Data format: NHWC
Optimizer:   sgd
Variables:   replicated
AllReduce:   None
==========
W0910 14:36:31.557293 140153510606656 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2762: The name tf.set_random_seed is deprecated. Please use tf.compat.v1.set_random_seed instead.

Generating training model
W0910 14:36:31.557902 140153510606656 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2782: The name tf.train.get_or_create_global_step is deprecated. Please use tf.compat.v1.train.get_or_create_global_step instead.

W0910 14:36:31.566257 140153510606656 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2739: The name tf.get_variable is deprecated. Please use tf.compat.v1.get_variable instead.

W0910 14:36:31.571086 140153510606656 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/variable_mgr.py:316: The name tf.variable_scope is deprecated. Please use tf.compat.v1.variable_scope instead.

W0910 14:36:31.588083 140153510606656 deprecation.py:323] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:129: conv2d (from tensorflow.python.layers.convolutional) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.keras.layers.Conv2D` instead.
W0910 14:36:31.949458 140153510606656 deprecation.py:323] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:261: max_pooling2d (from tensorflow.python.layers.pooling) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.MaxPooling2D instead.
W0910 14:36:34.601019 140153510606656 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:334: The name tf.nn.xw_plus_b is deprecated. Please use tf.compat.v1.nn.xw_plus_b instead.

W0910 14:36:34.603948 140153510606656 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/models/model.py:319: The name tf.losses.sparse_softmax_cross_entropy is deprecated. Please use tf.compat.v1.losses.sparse_softmax_cross_entropy instead.

W0910 14:36:34.615601 140153510606656 deprecation.py:323] From /usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/losses/losses_impl.py:121: add_dispatch_support.<locals>.wrapper (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.where in 2.0, which has the same broadcast rule as np.where
W0910 14:36:43.831400 140153510606656 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/models/resnet_model.py:333: The name tf.train.piecewise_constant is deprecated. Please use tf.compat.v1.train.piecewise_constant instead.

W0910 14:36:43.918720 140153510606656 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:1209: The name tf.train.GradientDescentOptimizer is deprecated. Please use tf.compat.v1.train.GradientDescentOptimizer instead.

W0910 14:36:44.743673 140153510606656 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2090: The name tf.train.get_global_step is deprecated. Please use tf.compat.v1.train.get_global_step instead.

W0910 14:36:45.510879 140153510606656 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2126: The name tf.summary.merge_all is deprecated. Please use tf.compat.v1.summary.merge_all instead.

Initializing graph
W0910 14:36:45.511124 140153510606656 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2183: The name tf.train.Saver is deprecated. Please use tf.compat.v1.train.Saver instead.

W0910 14:36:45.774051 140153510606656 deprecation.py:323] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2238: Supervisor.__init__ (from tensorflow.python.training.supervisor) is deprecated and will be removed in a future version.
Instructions for updating:
Please switch to tf.train.MonitoredTrainingSession
2019-09-10 14:36:47.260350: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:07:00.0
2019-09-10 14:36:47.262202: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 1 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:08:00.0
2019-09-10 14:36:47.263754: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 2 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:0e:00.0
2019-09-10 14:36:47.265319: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 3 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:0f:00.0
2019-09-10 14:36:47.265352: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
2019-09-10 14:36:47.265397: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10
2019-09-10 14:36:47.265419: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcufft.so.10
2019-09-10 14:36:47.265438: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcurand.so.10
2019-09-10 14:36:47.265458: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusolver.so.10
2019-09-10 14:36:47.265478: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusparse.so.10
2019-09-10 14:36:47.265498: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
2019-09-10 14:36:47.277504: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0, 1, 2, 3
2019-09-10 14:36:47.277563: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1181] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-09-10 14:36:47.277574: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1187]      0 1 2 3
2019-09-10 14:36:47.277582: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 0:   N Y Y Y
2019-09-10 14:36:47.277589: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 1:   Y N Y Y
2019-09-10 14:36:47.277596: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 2:   Y Y N Y
2019-09-10 14:36:47.277603: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 3:   Y Y Y N
2019-09-10 14:36:47.285324: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 30438 MB memory) -> physical GPU (device: 0, name: Tesla V100-DGXS-32GB, pci bus id: 0000:07:00.0, compute capability: 7.0)
2019-09-10 14:36:47.286911: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 30463 MB memory) -> physical GPU (device: 1, name: Tesla V100-DGXS-32GB, pci bus id: 0000:08:00.0, compute capability: 7.0)
2019-09-10 14:36:47.288468: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:2 with 30463 MB memory) -> physical GPU (device: 2, name: Tesla V100-DGXS-32GB, pci bus id: 0000:0e:00.0, compute capability: 7.0)
2019-09-10 14:36:47.290053: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:3 with 30463 MB memory) -> physical GPU (device: 3, name: Tesla V100-DGXS-32GB, pci bus id: 0000:0f:00.0, compute capability: 7.0)
2019-09-10 14:36:49.026768: W tensorflow/compiler/jit/mark_for_compilation_pass.cc:1412] (One-time warning): Not using XLA:CPU for cluster because envvar TF_XLA_FLAGS=--tf_xla_cpu_global_jit was not set.  If you want XLA:CPU, either set that envvar, or use experimental_jit_scope to enable XLA:CPU.  To confirm that XLA is active, pass --vmodule=xla_compilation_cache=1 (as a proper command-line flag, not via TF_XLA_FLAGS) or set the envvar XLA_FLAGS=--xla_hlo_profile.
I0910 14:36:51.533483 140153510606656 session_manager.py:500] Running local_init_op.
I0910 14:36:52.590555 140153510606656 session_manager.py:502] Done running local_init_op.
Running warm up
2019-09-10 14:36:57.377398: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10
2019-09-10 14:36:58.474053: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
Done warm up
Step    Img/sec total_loss
1       images/sec: 2485.6 +/- 0.0 (jitter = 0.0)       7.865
10      images/sec: 2488.5 +/- 1.4 (jitter = 3.8)       7.840
20      images/sec: 2488.0 +/- 0.9 (jitter = 4.1)       7.782
30      images/sec: 2487.2 +/- 0.8 (jitter = 3.8)       7.706
40      images/sec: 2485.5 +/- 0.9 (jitter = 5.0)       7.615
50      images/sec: 2485.5 +/- 0.8 (jitter = 5.3)       7.584
60      images/sec: 2485.3 +/- 0.7 (jitter = 5.4)       7.542
70      images/sec: 2485.3 +/- 0.7 (jitter = 5.4)       7.521
80      images/sec: 2485.6 +/- 0.6 (jitter = 5.4)       7.500
90      images/sec: 2484.9 +/- 0.6 (jitter = 5.7)       7.490
100     images/sec: 2484.9 +/- 0.5 (jitter = 5.8)       7.474
----------------------------------------------------------------
total images/sec: 2484.56
----------------------------------------------------------------
2019-09-10 14:38:21.365960: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
WARNING: Logging before flag parsing goes to stderr.
W0910 14:38:23.393384 140479090112320 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/models/mobilenet.py:388: The name tf.nn.avg_pool is deprecated. Please use tf.nn.avg_pool2d instead.

W0910 14:38:23.397919 140479090112320 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/models/experimental/deepspeech.py:131: The name tf.nn.rnn_cell.RNNCell is deprecated. Please use tf.compat.v1.nn.rnn_cell.RNNCell instead.

W0910 14:38:23.421399 140479090112320 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:722: The name tf.ConfigProto is deprecated. Please use tf.compat.v1.ConfigProto instead.

W0910 14:38:23.421613 140479090112320 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:3503: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead.

2019-09-10 14:38:23.444707: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2198780000 Hz
2019-09-10 14:38:23.447263: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x3251db0 executing computations on platform Host. Devices:
2019-09-10 14:38:23.447298: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): <undefined>, <undefined>
2019-09-10 14:38:23.450949: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcuda.so.1
2019-09-10 14:38:24.233403: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x4ac82a0 executing computations on platform CUDA. Devices:
2019-09-10 14:38:24.233436: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): Tesla V100-DGXS-32GB, Compute Capability 7.0
2019-09-10 14:38:24.233448: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (1): Tesla V100-DGXS-32GB, Compute Capability 7.0
2019-09-10 14:38:24.233456: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (2): Tesla V100-DGXS-32GB, Compute Capability 7.0
2019-09-10 14:38:24.233465: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (3): Tesla V100-DGXS-32GB, Compute Capability 7.0
2019-09-10 14:38:24.235791: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:07:00.0
2019-09-10 14:38:24.237931: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 1 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:08:00.0
2019-09-10 14:38:24.240262: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 2 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:0e:00.0
2019-09-10 14:38:24.242604: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 3 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:0f:00.0
2019-09-10 14:38:24.242644: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
2019-09-10 14:38:24.245324: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10
2019-09-10 14:38:24.247671: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcufft.so.10
2019-09-10 14:38:24.248111: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcurand.so.10
2019-09-10 14:38:24.250044: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusolver.so.10
2019-09-10 14:38:24.251025: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusparse.so.10
2019-09-10 14:38:24.254593: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
2019-09-10 14:38:24.266346: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0, 1, 2, 3
2019-09-10 14:38:24.266379: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
2019-09-10 14:38:26.032171: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1181] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-09-10 14:38:26.032212: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1187]      0 1 2 3
2019-09-10 14:38:26.032223: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 0:   N Y Y Y
2019-09-10 14:38:26.032230: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 1:   Y N Y Y
2019-09-10 14:38:26.032237: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 2:   Y Y N Y
2019-09-10 14:38:26.032243: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 3:   Y Y Y N
2019-09-10 14:38:26.040285: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 30438 MB memory) -> physical GPU (device: 0, name: Tesla V100-DGXS-32GB, pci bus id: 0000:07:00.0, compute capability: 7.0)
2019-09-10 14:38:26.042474: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 30463 MB memory) -> physical GPU (device: 1, name: Tesla V100-DGXS-32GB, pci bus id: 0000:08:00.0, compute capability: 7.0)
2019-09-10 14:38:26.044607: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:2 with 30463 MB memory) -> physical GPU (device: 2, name: Tesla V100-DGXS-32GB, pci bus id: 0000:0e:00.0, compute capability: 7.0)
2019-09-10 14:38:26.046722: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:3 with 30463 MB memory) -> physical GPU (device: 3, name: Tesla V100-DGXS-32GB, pci bus id: 0000:0f:00.0, compute capability: 7.0)
TensorFlow:  1.14
Model:       resnet50
Dataset:     imagenet (synthetic)
Mode:        training
SingleSess:  False
Batch size:  256 global
             64 per device
Num batches: 100
Num epochs:  0.02
Devices:     ['/gpu:0', '/gpu:1', '/gpu:2', '/gpu:3']
NUMA bind:   False
Data format: NHWC
Optimizer:   sgd
Variables:   replicated
AllReduce:   None
==========
W0910 14:38:26.052796 140479090112320 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2762: The name tf.set_random_seed is deprecated. Please use tf.compat.v1.set_random_seed instead.

Generating training model
W0910 14:38:26.053402 140479090112320 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2782: The name tf.train.get_or_create_global_step is deprecated. Please use tf.compat.v1.train.get_or_create_global_step instead.

W0910 14:38:26.062278 140479090112320 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/variable_mgr.py:316: The name tf.variable_scope is deprecated. Please use tf.compat.v1.variable_scope instead.

W0910 14:38:26.062677 140479090112320 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/models/model.py:245: The name tf.truncated_normal is deprecated. Please use tf.random.truncated_normal instead.

W0910 14:38:26.079476 140479090112320 deprecation.py:323] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:129: conv2d (from tensorflow.python.layers.convolutional) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.keras.layers.Conv2D` instead.
W0910 14:38:26.445255 140479090112320 deprecation.py:323] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:261: max_pooling2d (from tensorflow.python.layers.pooling) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.MaxPooling2D instead.
W0910 14:38:29.036207 140479090112320 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:334: The name tf.nn.xw_plus_b is deprecated. Please use tf.compat.v1.nn.xw_plus_b instead.

W0910 14:38:29.038326 140479090112320 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/models/model.py:319: The name tf.losses.sparse_softmax_cross_entropy is deprecated. Please use tf.compat.v1.losses.sparse_softmax_cross_entropy instead.

W0910 14:38:29.050102 140479090112320 deprecation.py:323] From /usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/losses/losses_impl.py:121: add_dispatch_support.<locals>.wrapper (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.where in 2.0, which has the same broadcast rule as np.where
W0910 14:38:39.846082 140479090112320 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/models/resnet_model.py:333: The name tf.train.piecewise_constant is deprecated. Please use tf.compat.v1.train.piecewise_constant instead.

W0910 14:38:39.934732 140479090112320 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:1209: The name tf.train.GradientDescentOptimizer is deprecated. Please use tf.compat.v1.train.GradientDescentOptimizer instead.

W0910 14:38:40.865384 140479090112320 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2090: The name tf.train.get_global_step is deprecated. Please use tf.compat.v1.train.get_global_step instead.

W0910 14:38:42.026576 140479090112320 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2126: The name tf.summary.merge_all is deprecated. Please use tf.compat.v1.summary.merge_all instead.

Initializing graph
W0910 14:38:42.026824 140479090112320 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2183: The name tf.train.Saver is deprecated. Please use tf.compat.v1.train.Saver instead.

W0910 14:38:42.293349 140479090112320 deprecation.py:323] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2238: Supervisor.__init__ (from tensorflow.python.training.supervisor) is deprecated and will be removed in a future version.
Instructions for updating:
Please switch to tf.train.MonitoredTrainingSession
2019-09-10 14:38:44.319723: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:07:00.0
2019-09-10 14:38:44.321581: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 1 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:08:00.0
2019-09-10 14:38:44.323313: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 2 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:0e:00.0
2019-09-10 14:38:44.325092: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 3 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:0f:00.0
2019-09-10 14:38:44.325126: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
2019-09-10 14:38:44.325164: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10
2019-09-10 14:38:44.325184: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcufft.so.10
2019-09-10 14:38:44.325203: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcurand.so.10
2019-09-10 14:38:44.325221: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusolver.so.10
2019-09-10 14:38:44.325239: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusparse.so.10
2019-09-10 14:38:44.325259: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
2019-09-10 14:38:44.337884: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0, 1, 2, 3
2019-09-10 14:38:44.337946: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1181] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-09-10 14:38:44.337957: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1187]      0 1 2 3
2019-09-10 14:38:44.337966: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 0:   N Y Y Y
2019-09-10 14:38:44.337972: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 1:   Y N Y Y
2019-09-10 14:38:44.337980: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 2:   Y Y N Y
2019-09-10 14:38:44.337987: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 3:   Y Y Y N
2019-09-10 14:38:44.345757: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 30438 MB memory) -> physical GPU (device: 0, name: Tesla V100-DGXS-32GB, pci bus id: 0000:07:00.0, compute capability: 7.0)
2019-09-10 14:38:44.347368: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 30463 MB memory) -> physical GPU (device: 1, name: Tesla V100-DGXS-32GB, pci bus id: 0000:08:00.0, compute capability: 7.0)
2019-09-10 14:38:44.348958: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:2 with 30463 MB memory) -> physical GPU (device: 2, name: Tesla V100-DGXS-32GB, pci bus id: 0000:0e:00.0, compute capability: 7.0)
2019-09-10 14:38:44.350572: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:3 with 30463 MB memory) -> physical GPU (device: 3, name: Tesla V100-DGXS-32GB, pci bus id: 0000:0f:00.0, compute capability: 7.0)
2019-09-10 14:38:46.679864: W tensorflow/compiler/jit/mark_for_compilation_pass.cc:1412] (One-time warning): Not using XLA:CPU for cluster because envvar TF_XLA_FLAGS=--tf_xla_cpu_global_jit was not set.  If you want XLA:CPU, either set that envvar, or use experimental_jit_scope to enable XLA:CPU.  To confirm that XLA is active, pass --vmodule=xla_compilation_cache=1 (as a proper command-line flag, not via TF_XLA_FLAGS) or set the envvar XLA_FLAGS=--xla_hlo_profile.
I0910 14:38:49.708881 140479090112320 session_manager.py:500] Running local_init_op.
I0910 14:38:51.242393 140479090112320 session_manager.py:502] Done running local_init_op.
Running warm up
2019-09-10 14:38:56.471976: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10
2019-09-10 14:38:57.857615: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
Done warm up
Step    Img/sec total_loss
1       images/sec: 1435.8 +/- 0.0 (jitter = 0.0)       7.719
10      images/sec: 1437.9 +/- 1.7 (jitter = 6.7)       7.944
20      images/sec: 1439.0 +/- 1.0 (jitter = 4.1)       7.885
30      images/sec: 1438.5 +/- 0.8 (jitter = 3.8)       7.907
40      images/sec: 1438.4 +/- 0.7 (jitter = 3.6)       7.931
50      images/sec: 1437.7 +/- 0.6 (jitter = 4.5)       7.796
60      images/sec: 1437.3 +/- 0.6 (jitter = 4.9)       7.828
70      images/sec: 1436.6 +/- 0.5 (jitter = 5.3)       7.883
80      images/sec: 1435.6 +/- 0.7 (jitter = 5.2)       7.760
90      images/sec: 1434.2 +/- 0.8 (jitter = 5.8)       7.825
100     images/sec: 1434.3 +/- 0.8 (jitter = 5.2)       7.815
----------------------------------------------------------------
total images/sec: 1433.72
----------------------------------------------------------------
2019-09-10 14:39:29.193982: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
WARNING: Logging before flag parsing goes to stderr.
W0910 14:39:31.194494 140373536360256 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/models/mobilenet.py:388: The name tf.nn.avg_pool is deprecated. Please use tf.nn.avg_pool2d instead.

W0910 14:39:31.198968 140373536360256 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/models/experimental/deepspeech.py:131: The name tf.nn.rnn_cell.RNNCell is deprecated. Please use tf.compat.v1.nn.rnn_cell.RNNCell instead.

W0910 14:39:31.222224 140373536360256 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:722: The name tf.ConfigProto is deprecated. Please use tf.compat.v1.ConfigProto instead.

W0910 14:39:31.222431 140373536360256 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:3503: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead.

2019-09-10 14:39:31.248721: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2198780000 Hz
2019-09-10 14:39:31.251280: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x31917d0 executing computations on platform Host. Devices:
2019-09-10 14:39:31.251314: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): <undefined>, <undefined>
2019-09-10 14:39:31.254771: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcuda.so.1
2019-09-10 14:39:32.064302: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x3191b30 executing computations on platform CUDA. Devices:
2019-09-10 14:39:32.064351: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): Tesla V100-DGXS-32GB, Compute Capability 7.0
2019-09-10 14:39:32.064367: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (1): Tesla V100-DGXS-32GB, Compute Capability 7.0
2019-09-10 14:39:32.064379: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (2): Tesla V100-DGXS-32GB, Compute Capability 7.0
2019-09-10 14:39:32.064391: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (3): Tesla V100-DGXS-32GB, Compute Capability 7.0
2019-09-10 14:39:32.067818: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:07:00.0
2019-09-10 14:39:32.070653: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 1 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:08:00.0
2019-09-10 14:39:32.073478: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 2 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:0e:00.0
2019-09-10 14:39:32.076270: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 3 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:0f:00.0
2019-09-10 14:39:32.076316: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
2019-09-10 14:39:32.079543: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10
2019-09-10 14:39:32.081668: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcufft.so.10
2019-09-10 14:39:32.081945: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcurand.so.10
2019-09-10 14:39:32.083638: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusolver.so.10
2019-09-10 14:39:32.084632: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusparse.so.10
2019-09-10 14:39:32.088180: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
2019-09-10 14:39:32.099957: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0, 1, 2, 3
2019-09-10 14:39:32.099989: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
2019-09-10 14:39:33.860195: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1181] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-09-10 14:39:33.860235: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1187]      0 1 2 3
2019-09-10 14:39:33.860248: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 0:   N Y Y Y
2019-09-10 14:39:33.860255: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 1:   Y N Y Y
2019-09-10 14:39:33.860262: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 2:   Y Y N Y
2019-09-10 14:39:33.860268: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 3:   Y Y Y N
2019-09-10 14:39:33.868318: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 30438 MB memory) -> physical GPU (device: 0, name: Tesla V100-DGXS-32GB, pci bus id: 0000:07:00.0, compute capability: 7.0)
2019-09-10 14:39:33.870509: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 30463 MB memory) -> physical GPU (device: 1, name: Tesla V100-DGXS-32GB, pci bus id: 0000:08:00.0, compute capability: 7.0)
2019-09-10 14:39:33.872661: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:2 with 30463 MB memory) -> physical GPU (device: 2, name: Tesla V100-DGXS-32GB, pci bus id: 0000:0e:00.0, compute capability: 7.0)
2019-09-10 14:39:33.874773: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:3 with 30463 MB memory) -> physical GPU (device: 3, name: Tesla V100-DGXS-32GB, pci bus id: 0000:0f:00.0, compute capability: 7.0)
TensorFlow:  1.14
Model:       resnet50
Dataset:     imagenet (synthetic)
Mode:        training
SingleSess:  False
Batch size:  256 global
             64 per device
Num batches: 100
Num epochs:  0.02
Devices:     ['/gpu:0', '/gpu:1', '/gpu:2', '/gpu:3']
NUMA bind:   False
Data format: NHWC
Optimizer:   sgd
Variables:   replicated
AllReduce:   None
==========
W0910 14:39:33.880883 140373536360256 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2762: The name tf.set_random_seed is deprecated. Please use tf.compat.v1.set_random_seed instead.

Generating training model
W0910 14:39:33.881470 140373536360256 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2782: The name tf.train.get_or_create_global_step is deprecated. Please use tf.compat.v1.train.get_or_create_global_step instead.

W0910 14:39:33.890151 140373536360256 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2739: The name tf.get_variable is deprecated. Please use tf.compat.v1.get_variable instead.

W0910 14:39:33.895046 140373536360256 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/variable_mgr.py:316: The name tf.variable_scope is deprecated. Please use tf.compat.v1.variable_scope instead.

W0910 14:39:33.912401 140373536360256 deprecation.py:323] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:129: conv2d (from tensorflow.python.layers.convolutional) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.keras.layers.Conv2D` instead.
W0910 14:39:34.278283 140373536360256 deprecation.py:323] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:261: max_pooling2d (from tensorflow.python.layers.pooling) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.MaxPooling2D instead.
W0910 14:39:36.983918 140373536360256 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:334: The name tf.nn.xw_plus_b is deprecated. Please use tf.compat.v1.nn.xw_plus_b instead.

W0910 14:39:36.986926 140373536360256 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/models/model.py:319: The name tf.losses.sparse_softmax_cross_entropy is deprecated. Please use tf.compat.v1.losses.sparse_softmax_cross_entropy instead.

W0910 14:39:36.998892 140373536360256 deprecation.py:323] From /usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/losses/losses_impl.py:121: add_dispatch_support.<locals>.wrapper (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.where in 2.0, which has the same broadcast rule as np.where
W0910 14:39:50.398809 140373536360256 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/models/resnet_model.py:333: The name tf.train.piecewise_constant is deprecated. Please use tf.compat.v1.train.piecewise_constant instead.

W0910 14:39:50.487930 140373536360256 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:1209: The name tf.train.GradientDescentOptimizer is deprecated. Please use tf.compat.v1.train.GradientDescentOptimizer instead.

W0910 14:39:51.618725 140373536360256 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2090: The name tf.train.get_global_step is deprecated. Please use tf.compat.v1.train.get_global_step instead.

W0910 14:39:52.795925 140373536360256 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2126: The name tf.summary.merge_all is deprecated. Please use tf.compat.v1.summary.merge_all instead.

Initializing graph
W0910 14:39:52.796192 140373536360256 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2183: The name tf.train.Saver is deprecated. Please use tf.compat.v1.train.Saver instead.

W0910 14:39:53.066552 140373536360256 deprecation.py:323] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2238: Supervisor.__init__ (from tensorflow.python.training.supervisor) is deprecated and will be removed in a future version.
Instructions for updating:
Please switch to tf.train.MonitoredTrainingSession
2019-09-10 14:39:55.149767: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:07:00.0
2019-09-10 14:39:55.151607: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 1 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:08:00.0
2019-09-10 14:39:55.153463: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 2 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:0e:00.0
2019-09-10 14:39:55.155187: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 3 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:0f:00.0
2019-09-10 14:39:55.155220: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
2019-09-10 14:39:55.155254: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10
2019-09-10 14:39:55.155274: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcufft.so.10
2019-09-10 14:39:55.155291: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcurand.so.10
2019-09-10 14:39:55.155308: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusolver.so.10
2019-09-10 14:39:55.155325: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusparse.so.10
2019-09-10 14:39:55.155343: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
2019-09-10 14:39:55.168207: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0, 1, 2, 3
2019-09-10 14:39:55.168269: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1181] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-09-10 14:39:55.168281: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1187]      0 1 2 3
2019-09-10 14:39:55.168289: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 0:   N Y Y Y
2019-09-10 14:39:55.168296: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 1:   Y N Y Y
2019-09-10 14:39:55.168302: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 2:   Y Y N Y
2019-09-10 14:39:55.168308: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 3:   Y Y Y N
2019-09-10 14:39:55.176192: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 30438 MB memory) -> physical GPU (device: 0, name: Tesla V100-DGXS-32GB, pci bus id: 0000:07:00.0, compute capability: 7.0)
2019-09-10 14:39:55.177821: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 30463 MB memory) -> physical GPU (device: 1, name: Tesla V100-DGXS-32GB, pci bus id: 0000:08:00.0, compute capability: 7.0)
2019-09-10 14:39:55.179408: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:2 with 30463 MB memory) -> physical GPU (device: 2, name: Tesla V100-DGXS-32GB, pci bus id: 0000:0e:00.0, compute capability: 7.0)
2019-09-10 14:39:55.180997: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:3 with 30463 MB memory) -> physical GPU (device: 3, name: Tesla V100-DGXS-32GB, pci bus id: 0000:0f:00.0, compute capability: 7.0)
2019-09-10 14:39:57.639631: W tensorflow/compiler/jit/mark_for_compilation_pass.cc:1412] (One-time warning): Not using XLA:CPU for cluster because envvar TF_XLA_FLAGS=--tf_xla_cpu_global_jit was not set.  If you want XLA:CPU, either set that envvar, or use experimental_jit_scope to enable XLA:CPU.  To confirm that XLA is active, pass --vmodule=xla_compilation_cache=1 (as a proper command-line flag, not via TF_XLA_FLAGS) or set the envvar XLA_FLAGS=--xla_hlo_profile.
I0910 14:40:00.674548 140373536360256 session_manager.py:500] Running local_init_op.
I0910 14:40:02.234329 140373536360256 session_manager.py:502] Done running local_init_op.
Running warm up
2019-09-10 14:40:08.754754: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10
2019-09-10 14:40:10.229719: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
Done warm up
Step    Img/sec total_loss
1       images/sec: 2373.4 +/- 0.0 (jitter = 0.0)       8.015
10      images/sec: 2374.5 +/- 3.6 (jitter = 8.9)       7.900
20      images/sec: 2370.0 +/- 2.4 (jitter = 10.2)      7.928
30      images/sec: 2369.8 +/- 1.9 (jitter = 10.1)      7.899
40      images/sec: 2370.9 +/- 1.6 (jitter = 9.3)       7.836
50      images/sec: 2371.3 +/- 1.5 (jitter = 12.1)      7.880
60      images/sec: 2370.8 +/- 1.4 (jitter = 11.6)      7.853
70      images/sec: 2371.2 +/- 1.3 (jitter = 10.6)      7.800
80      images/sec: 2371.6 +/- 1.2 (jitter = 10.8)      7.787
90      images/sec: 2372.2 +/- 1.2 (jitter = 12.9)      7.886
100     images/sec: 2372.4 +/- 1.1 (jitter = 12.7)      7.696
----------------------------------------------------------------
total images/sec: 2370.85
----------------------------------------------------------------
2019-09-10 14:40:33.355935: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
WARNING: Logging before flag parsing goes to stderr.
W0910 14:40:35.351562 140068409890624 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/models/mobilenet.py:388: The name tf.nn.avg_pool is deprecated. Please use tf.nn.avg_pool2d instead.

W0910 14:40:35.356027 140068409890624 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/models/experimental/deepspeech.py:131: The name tf.nn.rnn_cell.RNNCell is deprecated. Please use tf.compat.v1.nn.rnn_cell.RNNCell instead.

W0910 14:40:35.379163 140068409890624 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:722: The name tf.ConfigProto is deprecated. Please use tf.compat.v1.ConfigProto instead.

W0910 14:40:35.379368 140068409890624 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:3503: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead.

2019-09-10 14:40:35.404723: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2198780000 Hz
2019-09-10 14:40:35.407009: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x3f6ada0 executing computations on platform Host. Devices:
2019-09-10 14:40:35.407030: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): <undefined>, <undefined>
2019-09-10 14:40:35.409093: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcuda.so.1
2019-09-10 14:40:36.177913: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x57cccd0 executing computations on platform CUDA. Devices:
2019-09-10 14:40:36.177952: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): Tesla V100-DGXS-32GB, Compute Capability 7.0
2019-09-10 14:40:36.177965: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (1): Tesla V100-DGXS-32GB, Compute Capability 7.0
2019-09-10 14:40:36.177975: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (2): Tesla V100-DGXS-32GB, Compute Capability 7.0
2019-09-10 14:40:36.177985: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (3): Tesla V100-DGXS-32GB, Compute Capability 7.0
2019-09-10 14:40:36.180824: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:07:00.0
2019-09-10 14:40:36.183185: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 1 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:08:00.0
2019-09-10 14:40:36.185524: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 2 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:0e:00.0
2019-09-10 14:40:36.187890: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 3 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:0f:00.0
2019-09-10 14:40:36.187930: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
2019-09-10 14:40:36.190617: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10
2019-09-10 14:40:36.192957: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcufft.so.10
2019-09-10 14:40:36.193266: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcurand.so.10
2019-09-10 14:40:36.194965: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusolver.so.10
2019-09-10 14:40:36.195946: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusparse.so.10
2019-09-10 14:40:36.199504: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
2019-09-10 14:40:36.211310: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0, 1, 2, 3
2019-09-10 14:40:36.211343: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
2019-09-10 14:40:37.981401: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1181] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-09-10 14:40:37.981441: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1187]      0 1 2 3
2019-09-10 14:40:37.981451: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 0:   N Y Y Y
2019-09-10 14:40:37.981457: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 1:   Y N Y Y
2019-09-10 14:40:37.981463: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 2:   Y Y N Y
2019-09-10 14:40:37.981469: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 3:   Y Y Y N
2019-09-10 14:40:37.989418: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 30438 MB memory) -> physical GPU (device: 0, name: Tesla V100-DGXS-32GB, pci bus id: 0000:07:00.0, compute capability: 7.0)
2019-09-10 14:40:37.991696: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 30463 MB memory) -> physical GPU (device: 1, name: Tesla V100-DGXS-32GB, pci bus id: 0000:08:00.0, compute capability: 7.0)
2019-09-10 14:40:37.993831: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:2 with 30463 MB memory) -> physical GPU (device: 2, name: Tesla V100-DGXS-32GB, pci bus id: 0000:0e:00.0, compute capability: 7.0)
2019-09-10 14:40:37.995936: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:3 with 30463 MB memory) -> physical GPU (device: 3, name: Tesla V100-DGXS-32GB, pci bus id: 0000:0f:00.0, compute capability: 7.0)
TensorFlow:  1.14
Model:       resnet50
Dataset:     imagenet (synthetic)
Mode:        training
SingleSess:  False
Batch size:  512 global
             128 per device
Num batches: 100
Num epochs:  0.04
Devices:     ['/gpu:0', '/gpu:1', '/gpu:2', '/gpu:3']
NUMA bind:   False
Data format: NHWC
Optimizer:   sgd
Variables:   replicated
AllReduce:   None
==========
W0910 14:40:38.001996 140068409890624 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2762: The name tf.set_random_seed is deprecated. Please use tf.compat.v1.set_random_seed instead.

Generating training model
W0910 14:40:38.002607 140068409890624 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2782: The name tf.train.get_or_create_global_step is deprecated. Please use tf.compat.v1.train.get_or_create_global_step instead.

W0910 14:40:38.010828 140068409890624 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/variable_mgr.py:316: The name tf.variable_scope is deprecated. Please use tf.compat.v1.variable_scope instead.

W0910 14:40:38.011236 140068409890624 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/models/model.py:245: The name tf.truncated_normal is deprecated. Please use tf.random.truncated_normal instead.

W0910 14:40:38.027825 140068409890624 deprecation.py:323] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:129: conv2d (from tensorflow.python.layers.convolutional) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.keras.layers.Conv2D` instead.
W0910 14:40:38.394341 140068409890624 deprecation.py:323] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:261: max_pooling2d (from tensorflow.python.layers.pooling) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.MaxPooling2D instead.
W0910 14:40:40.938336 140068409890624 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:334: The name tf.nn.xw_plus_b is deprecated. Please use tf.compat.v1.nn.xw_plus_b instead.

W0910 14:40:40.940391 140068409890624 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/models/model.py:319: The name tf.losses.sparse_softmax_cross_entropy is deprecated. Please use tf.compat.v1.losses.sparse_softmax_cross_entropy instead.

W0910 14:40:40.951967 140068409890624 deprecation.py:323] From /usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/losses/losses_impl.py:121: add_dispatch_support.<locals>.wrapper (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.where in 2.0, which has the same broadcast rule as np.where
W0910 14:40:51.559518 140068409890624 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/models/resnet_model.py:333: The name tf.train.piecewise_constant is deprecated. Please use tf.compat.v1.train.piecewise_constant instead.

W0910 14:40:51.646619 140068409890624 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:1209: The name tf.train.GradientDescentOptimizer is deprecated. Please use tf.compat.v1.train.GradientDescentOptimizer instead.

W0910 14:40:52.556450 140068409890624 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2090: The name tf.train.get_global_step is deprecated. Please use tf.compat.v1.train.get_global_step instead.

W0910 14:40:53.695517 140068409890624 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2126: The name tf.summary.merge_all is deprecated. Please use tf.compat.v1.summary.merge_all instead.

Initializing graph
W0910 14:40:53.695766 140068409890624 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2183: The name tf.train.Saver is deprecated. Please use tf.compat.v1.train.Saver instead.

W0910 14:40:53.956712 140068409890624 deprecation.py:323] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2238: Supervisor.__init__ (from tensorflow.python.training.supervisor) is deprecated and will be removed in a future version.
Instructions for updating:
Please switch to tf.train.MonitoredTrainingSession
2019-09-10 14:40:55.917164: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:07:00.0
2019-09-10 14:40:55.918885: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 1 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:08:00.0
2019-09-10 14:40:55.920731: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 2 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:0e:00.0
2019-09-10 14:40:55.922285: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 3 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:0f:00.0
2019-09-10 14:40:55.922319: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
2019-09-10 14:40:55.922354: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10
2019-09-10 14:40:55.922372: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcufft.so.10
2019-09-10 14:40:55.922389: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcurand.so.10
2019-09-10 14:40:55.922406: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusolver.so.10
2019-09-10 14:40:55.922431: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusparse.so.10
2019-09-10 14:40:55.922461: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
2019-09-10 14:40:55.934869: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0, 1, 2, 3
2019-09-10 14:40:55.934930: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1181] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-09-10 14:40:55.934942: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1187]      0 1 2 3
2019-09-10 14:40:55.934950: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 0:   N Y Y Y
2019-09-10 14:40:55.934956: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 1:   Y N Y Y
2019-09-10 14:40:55.934962: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 2:   Y Y N Y
2019-09-10 14:40:55.934969: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 3:   Y Y Y N
2019-09-10 14:40:55.942813: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 30438 MB memory) -> physical GPU (device: 0, name: Tesla V100-DGXS-32GB, pci bus id: 0000:07:00.0, compute capability: 7.0)
2019-09-10 14:40:55.944464: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 30463 MB memory) -> physical GPU (device: 1, name: Tesla V100-DGXS-32GB, pci bus id: 0000:08:00.0, compute capability: 7.0)
2019-09-10 14:40:55.946085: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:2 with 30463 MB memory) -> physical GPU (device: 2, name: Tesla V100-DGXS-32GB, pci bus id: 0000:0e:00.0, compute capability: 7.0)
2019-09-10 14:40:55.947695: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:3 with 30463 MB memory) -> physical GPU (device: 3, name: Tesla V100-DGXS-32GB, pci bus id: 0000:0f:00.0, compute capability: 7.0)
2019-09-10 14:40:58.241728: W tensorflow/compiler/jit/mark_for_compilation_pass.cc:1412] (One-time warning): Not using XLA:CPU for cluster because envvar TF_XLA_FLAGS=--tf_xla_cpu_global_jit was not set.  If you want XLA:CPU, either set that envvar, or use experimental_jit_scope to enable XLA:CPU.  To confirm that XLA is active, pass --vmodule=xla_compilation_cache=1 (as a proper command-line flag, not via TF_XLA_FLAGS) or set the envvar XLA_FLAGS=--xla_hlo_profile.
I0910 14:41:01.260072 140068409890624 session_manager.py:500] Running local_init_op.
I0910 14:41:02.757647 140068409890624 session_manager.py:502] Done running local_init_op.
Running warm up
2019-09-10 14:41:07.928875: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10
2019-09-10 14:41:09.316597: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
Done warm up
Step    Img/sec total_loss
1       images/sec: 1548.1 +/- 0.0 (jitter = 0.0)       7.911
10      images/sec: 1545.3 +/- 1.1 (jitter = 4.1)       7.909
20      images/sec: 1544.6 +/- 0.8 (jitter = 3.4)       7.886
30      images/sec: 1544.6 +/- 0.6 (jitter = 3.5)       7.778
40      images/sec: 1544.4 +/- 0.5 (jitter = 3.3)       7.868
50      images/sec: 1544.2 +/- 0.4 (jitter = 3.4)       7.778
60      images/sec: 1544.3 +/- 0.4 (jitter = 3.5)       7.791
70      images/sec: 1544.3 +/- 0.4 (jitter = 3.5)       7.748
80      images/sec: 1544.6 +/- 0.4 (jitter = 3.5)       7.754
90      images/sec: 1544.6 +/- 0.3 (jitter = 3.8)       7.691
100     images/sec: 1544.7 +/- 0.3 (jitter = 3.5)       7.702
----------------------------------------------------------------
total images/sec: 1544.37
----------------------------------------------------------------
2019-09-10 14:41:58.338581: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
WARNING: Logging before flag parsing goes to stderr.
W0910 14:42:00.345152 140084530083648 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/models/mobilenet.py:388: The name tf.nn.avg_pool is deprecated. Please use tf.nn.avg_pool2d instead.

W0910 14:42:00.349649 140084530083648 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/models/experimental/deepspeech.py:131: The name tf.nn.rnn_cell.RNNCell is deprecated. Please use tf.compat.v1.nn.rnn_cell.RNNCell instead.

W0910 14:42:00.372978 140084530083648 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:722: The name tf.ConfigProto is deprecated. Please use tf.compat.v1.ConfigProto instead.

W0910 14:42:00.373190 140084530083648 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:3503: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead.

2019-09-10 14:42:00.396688: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2198780000 Hz
2019-09-10 14:42:00.399358: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x4a9bb50 executing computations on platform Host. Devices:
2019-09-10 14:42:00.399392: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): <undefined>, <undefined>
2019-09-10 14:42:00.402927: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcuda.so.1
2019-09-10 14:42:01.175071: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x4a9c060 executing computations on platform CUDA. Devices:
2019-09-10 14:42:01.175103: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): Tesla V100-DGXS-32GB, Compute Capability 7.0
2019-09-10 14:42:01.175113: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (1): Tesla V100-DGXS-32GB, Compute Capability 7.0
2019-09-10 14:42:01.175121: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (2): Tesla V100-DGXS-32GB, Compute Capability 7.0
2019-09-10 14:42:01.175128: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (3): Tesla V100-DGXS-32GB, Compute Capability 7.0
2019-09-10 14:42:01.177331: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:07:00.0
2019-09-10 14:42:01.179466: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 1 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:08:00.0
2019-09-10 14:42:01.181590: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 2 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:0e:00.0
2019-09-10 14:42:01.183731: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 3 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:0f:00.0
2019-09-10 14:42:01.183769: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
2019-09-10 14:42:01.186188: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10
2019-09-10 14:42:01.188303: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcufft.so.10
2019-09-10 14:42:01.188701: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcurand.so.10
2019-09-10 14:42:01.190412: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusolver.so.10
2019-09-10 14:42:01.191398: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusparse.so.10
2019-09-10 14:42:01.194987: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
2019-09-10 14:42:01.206841: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0, 1, 2, 3
2019-09-10 14:42:01.206877: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
2019-09-10 14:42:02.971131: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1181] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-09-10 14:42:02.971175: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1187]      0 1 2 3
2019-09-10 14:42:02.971185: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 0:   N Y Y Y
2019-09-10 14:42:02.971192: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 1:   Y N Y Y
2019-09-10 14:42:02.971198: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 2:   Y Y N Y
2019-09-10 14:42:02.971204: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 3:   Y Y Y N
2019-09-10 14:42:02.979115: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 30438 MB memory) -> physical GPU (device: 0, name: Tesla V100-DGXS-32GB, pci bus id: 0000:07:00.0, compute capability: 7.0)
2019-09-10 14:42:02.981260: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 30463 MB memory) -> physical GPU (device: 1, name: Tesla V100-DGXS-32GB, pci bus id: 0000:08:00.0, compute capability: 7.0)
2019-09-10 14:42:02.983389: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:2 with 30463 MB memory) -> physical GPU (device: 2, name: Tesla V100-DGXS-32GB, pci bus id: 0000:0e:00.0, compute capability: 7.0)
2019-09-10 14:42:02.985780: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:3 with 30463 MB memory) -> physical GPU (device: 3, name: Tesla V100-DGXS-32GB, pci bus id: 0000:0f:00.0, compute capability: 7.0)
TensorFlow:  1.14
Model:       resnet50
Dataset:     imagenet (synthetic)
Mode:        training
SingleSess:  False
Batch size:  512 global
             128 per device
Num batches: 100
Num epochs:  0.04
Devices:     ['/gpu:0', '/gpu:1', '/gpu:2', '/gpu:3']
NUMA bind:   False
Data format: NHWC
Optimizer:   sgd
Variables:   replicated
AllReduce:   None
==========
W0910 14:42:02.991589 140084530083648 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2762: The name tf.set_random_seed is deprecated. Please use tf.compat.v1.set_random_seed instead.

Generating training model
W0910 14:42:02.992118 140084530083648 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2782: The name tf.train.get_or_create_global_step is deprecated. Please use tf.compat.v1.train.get_or_create_global_step instead.

W0910 14:42:03.000892 140084530083648 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2739: The name tf.get_variable is deprecated. Please use tf.compat.v1.get_variable instead.

W0910 14:42:03.007102 140084530083648 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/variable_mgr.py:316: The name tf.variable_scope is deprecated. Please use tf.compat.v1.variable_scope instead.

W0910 14:42:03.023902 140084530083648 deprecation.py:323] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:129: conv2d (from tensorflow.python.layers.convolutional) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.keras.layers.Conv2D` instead.
W0910 14:42:03.396012 140084530083648 deprecation.py:323] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:261: max_pooling2d (from tensorflow.python.layers.pooling) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.MaxPooling2D instead.
W0910 14:42:06.038249 140084530083648 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:334: The name tf.nn.xw_plus_b is deprecated. Please use tf.compat.v1.nn.xw_plus_b instead.

W0910 14:42:06.041155 140084530083648 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/models/model.py:319: The name tf.losses.sparse_softmax_cross_entropy is deprecated. Please use tf.compat.v1.losses.sparse_softmax_cross_entropy instead.

W0910 14:42:06.052740 140084530083648 deprecation.py:323] From /usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/losses/losses_impl.py:121: add_dispatch_support.<locals>.wrapper (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.where in 2.0, which has the same broadcast rule as np.where
W0910 14:42:19.066690 140084530083648 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/models/resnet_model.py:333: The name tf.train.piecewise_constant is deprecated. Please use tf.compat.v1.train.piecewise_constant instead.

W0910 14:42:19.153356 140084530083648 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:1209: The name tf.train.GradientDescentOptimizer is deprecated. Please use tf.compat.v1.train.GradientDescentOptimizer instead.

W0910 14:42:20.247669 140084530083648 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2090: The name tf.train.get_global_step is deprecated. Please use tf.compat.v1.train.get_global_step instead.

W0910 14:42:21.380501 140084530083648 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2126: The name tf.summary.merge_all is deprecated. Please use tf.compat.v1.summary.merge_all instead.

Initializing graph
W0910 14:42:21.380757 140084530083648 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2183: The name tf.train.Saver is deprecated. Please use tf.compat.v1.train.Saver instead.

W0910 14:42:21.642261 140084530083648 deprecation.py:323] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2238: Supervisor.__init__ (from tensorflow.python.training.supervisor) is deprecated and will be removed in a future version.
Instructions for updating:
Please switch to tf.train.MonitoredTrainingSession
2019-09-10 14:42:23.576749: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:07:00.0
2019-09-10 14:42:23.578613: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 1 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:08:00.0
2019-09-10 14:42:23.580386: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 2 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:0e:00.0
2019-09-10 14:42:23.581974: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 3 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:0f:00.0
2019-09-10 14:42:23.582006: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
2019-09-10 14:42:23.582044: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10
2019-09-10 14:42:23.582066: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcufft.so.10
2019-09-10 14:42:23.582083: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcurand.so.10
2019-09-10 14:42:23.582099: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusolver.so.10
2019-09-10 14:42:23.582120: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusparse.so.10
2019-09-10 14:42:23.582140: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
2019-09-10 14:42:23.594867: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0, 1, 2, 3
2019-09-10 14:42:23.594928: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1181] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-09-10 14:42:23.594942: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1187]      0 1 2 3
2019-09-10 14:42:23.594952: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 0:   N Y Y Y
2019-09-10 14:42:23.594958: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 1:   Y N Y Y
2019-09-10 14:42:23.594965: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 2:   Y Y N Y
2019-09-10 14:42:23.594972: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 3:   Y Y Y N
2019-09-10 14:42:23.602803: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 30438 MB memory) -> physical GPU (device: 0, name: Tesla V100-DGXS-32GB, pci bus id: 0000:07:00.0, compute capability: 7.0)
2019-09-10 14:42:23.604400: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 30463 MB memory) -> physical GPU (device: 1, name: Tesla V100-DGXS-32GB, pci bus id: 0000:08:00.0, compute capability: 7.0)
2019-09-10 14:42:23.605992: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:2 with 30463 MB memory) -> physical GPU (device: 2, name: Tesla V100-DGXS-32GB, pci bus id: 0000:0e:00.0, compute capability: 7.0)
2019-09-10 14:42:23.607597: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:3 with 30463 MB memory) -> physical GPU (device: 3, name: Tesla V100-DGXS-32GB, pci bus id: 0000:0f:00.0, compute capability: 7.0)
2019-09-10 14:42:25.980176: W tensorflow/compiler/jit/mark_for_compilation_pass.cc:1412] (One-time warning): Not using XLA:CPU for cluster because envvar TF_XLA_FLAGS=--tf_xla_cpu_global_jit was not set.  If you want XLA:CPU, either set that envvar, or use experimental_jit_scope to enable XLA:CPU.  To confirm that XLA is active, pass --vmodule=xla_compilation_cache=1 (as a proper command-line flag, not via TF_XLA_FLAGS) or set the envvar XLA_FLAGS=--xla_hlo_profile.
I0910 14:42:29.000877 140084530083648 session_manager.py:500] Running local_init_op.
I0910 14:42:30.535269 140084530083648 session_manager.py:502] Done running local_init_op.
Running warm up
2019-09-10 14:42:37.047870: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10
2019-09-10 14:42:38.519216: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
Done warm up
Step    Img/sec total_loss
1       images/sec: 2836.0 +/- 0.0 (jitter = 0.0)       7.836
10      images/sec: 2847.8 +/- 3.1 (jitter = 15.4)      7.857
20      images/sec: 2843.7 +/- 2.7 (jitter = 13.8)      7.873
30      images/sec: 2838.9 +/- 2.8 (jitter = 14.3)      7.816
40      images/sec: 2838.3 +/- 2.4 (jitter = 18.0)      7.818
50      images/sec: 2837.6 +/- 2.1 (jitter = 12.7)      7.773
60      images/sec: 2837.9 +/- 1.8 (jitter = 10.9)      7.723
70      images/sec: 2836.2 +/- 1.7 (jitter = 11.3)      7.727
80      images/sec: 2836.4 +/- 1.6 (jitter = 10.5)      7.670
90      images/sec: 2835.1 +/- 1.5 (jitter = 10.6)      7.736
100     images/sec: 2833.2 +/- 1.5 (jitter = 13.3)      7.713
----------------------------------------------------------------
total images/sec: 2832.00
----------------------------------------------------------------
2019-09-10 14:43:10.852005: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
WARNING: Logging before flag parsing goes to stderr.
W0910 14:43:12.849007 139844950951744 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/models/mobilenet.py:388: The name tf.nn.avg_pool is deprecated. Please use tf.nn.avg_pool2d instead.

W0910 14:43:12.853490 139844950951744 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/models/experimental/deepspeech.py:131: The name tf.nn.rnn_cell.RNNCell is deprecated. Please use tf.compat.v1.nn.rnn_cell.RNNCell instead.

W0910 14:43:12.876675 139844950951744 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:722: The name tf.ConfigProto is deprecated. Please use tf.compat.v1.ConfigProto instead.

W0910 14:43:12.876881 139844950951744 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:3503: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead.

2019-09-10 14:43:12.900728: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2198780000 Hz
2019-09-10 14:43:12.903535: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x3d9a760 executing computations on platform Host. Devices:
2019-09-10 14:43:12.903580: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): <undefined>, <undefined>
2019-09-10 14:43:12.907585: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcuda.so.1
2019-09-10 14:43:13.729113: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x3d9ac70 executing computations on platform CUDA. Devices:
2019-09-10 14:43:13.729150: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): Tesla V100-DGXS-32GB, Compute Capability 7.0
2019-09-10 14:43:13.729164: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (1): Tesla V100-DGXS-32GB, Compute Capability 7.0
2019-09-10 14:43:13.729174: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (2): Tesla V100-DGXS-32GB, Compute Capability 7.0
2019-09-10 14:43:13.729183: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (3): Tesla V100-DGXS-32GB, Compute Capability 7.0
2019-09-10 14:43:13.731730: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:07:00.0
2019-09-10 14:43:13.734102: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 1 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:08:00.0
2019-09-10 14:43:13.736581: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 2 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:0e:00.0
2019-09-10 14:43:13.739039: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 3 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:0f:00.0
2019-09-10 14:43:13.739080: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
2019-09-10 14:43:13.741924: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10
2019-09-10 14:43:13.744380: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcufft.so.10
2019-09-10 14:43:13.744844: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcurand.so.10
2019-09-10 14:43:13.746831: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusolver.so.10
2019-09-10 14:43:13.747991: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusparse.so.10
2019-09-10 14:43:13.752180: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
2019-09-10 14:43:13.765892: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0, 1, 2, 3
2019-09-10 14:43:13.765929: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
2019-09-10 14:43:15.551812: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1181] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-09-10 14:43:15.551853: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1187]      0 1 2 3
2019-09-10 14:43:15.551863: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 0:   N Y Y Y
2019-09-10 14:43:15.551870: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 1:   Y N Y Y
2019-09-10 14:43:15.551876: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 2:   Y Y N Y
2019-09-10 14:43:15.551883: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 3:   Y Y Y N
2019-09-10 14:43:15.559899: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 30438 MB memory) -> physical GPU (device: 0, name: Tesla V100-DGXS-32GB, pci bus id: 0000:07:00.0, compute capability: 7.0)
2019-09-10 14:43:15.562079: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 30463 MB memory) -> physical GPU (device: 1, name: Tesla V100-DGXS-32GB, pci bus id: 0000:08:00.0, compute capability: 7.0)
2019-09-10 14:43:15.564197: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:2 with 30463 MB memory) -> physical GPU (device: 2, name: Tesla V100-DGXS-32GB, pci bus id: 0000:0e:00.0, compute capability: 7.0)
2019-09-10 14:43:15.566344: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:3 with 30463 MB memory) -> physical GPU (device: 3, name: Tesla V100-DGXS-32GB, pci bus id: 0000:0f:00.0, compute capability: 7.0)
TensorFlow:  1.14
Model:       resnet50
Dataset:     imagenet (synthetic)
Mode:        training
SingleSess:  False
Batch size:  1024 global
             256 per device
Num batches: 100
Num epochs:  0.08
Devices:     ['/gpu:0', '/gpu:1', '/gpu:2', '/gpu:3']
NUMA bind:   False
Data format: NHWC
Optimizer:   sgd
Variables:   replicated
AllReduce:   None
==========
W0910 14:43:15.572416 139844950951744 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2762: The name tf.set_random_seed is deprecated. Please use tf.compat.v1.set_random_seed instead.

Generating training model
W0910 14:43:15.573017 139844950951744 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2782: The name tf.train.get_or_create_global_step is deprecated. Please use tf.compat.v1.train.get_or_create_global_step instead.

W0910 14:43:15.581869 139844950951744 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/variable_mgr.py:316: The name tf.variable_scope is deprecated. Please use tf.compat.v1.variable_scope instead.

W0910 14:43:15.582245 139844950951744 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/models/model.py:245: The name tf.truncated_normal is deprecated. Please use tf.random.truncated_normal instead.

W0910 14:43:15.598568 139844950951744 deprecation.py:323] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:129: conv2d (from tensorflow.python.layers.convolutional) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.keras.layers.Conv2D` instead.
W0910 14:43:15.965062 139844950951744 deprecation.py:323] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:261: max_pooling2d (from tensorflow.python.layers.pooling) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.MaxPooling2D instead.
W0910 14:43:18.525661 139844950951744 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:334: The name tf.nn.xw_plus_b is deprecated. Please use tf.compat.v1.nn.xw_plus_b instead.

W0910 14:43:18.527738 139844950951744 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/models/model.py:319: The name tf.losses.sparse_softmax_cross_entropy is deprecated. Please use tf.compat.v1.losses.sparse_softmax_cross_entropy instead.

W0910 14:43:18.539392 139844950951744 deprecation.py:323] From /usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/losses/losses_impl.py:121: add_dispatch_support.<locals>.wrapper (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.where in 2.0, which has the same broadcast rule as np.where
W0910 14:43:29.232611 139844950951744 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/models/resnet_model.py:333: The name tf.train.piecewise_constant is deprecated. Please use tf.compat.v1.train.piecewise_constant instead.

W0910 14:43:29.320429 139844950951744 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:1209: The name tf.train.GradientDescentOptimizer is deprecated. Please use tf.compat.v1.train.GradientDescentOptimizer instead.

W0910 14:43:30.241079 139844950951744 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2090: The name tf.train.get_global_step is deprecated. Please use tf.compat.v1.train.get_global_step instead.

W0910 14:43:31.388153 139844950951744 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2126: The name tf.summary.merge_all is deprecated. Please use tf.compat.v1.summary.merge_all instead.

Initializing graph
W0910 14:43:31.388401 139844950951744 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2183: The name tf.train.Saver is deprecated. Please use tf.compat.v1.train.Saver instead.

W0910 14:43:31.651107 139844950951744 deprecation.py:323] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2238: Supervisor.__init__ (from tensorflow.python.training.supervisor) is deprecated and will be removed in a future version.
Instructions for updating:
Please switch to tf.train.MonitoredTrainingSession
2019-09-10 14:43:33.634589: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:07:00.0
2019-09-10 14:43:33.636350: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 1 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:08:00.0
2019-09-10 14:43:33.637971: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 2 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:0e:00.0
2019-09-10 14:43:33.639558: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 3 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:0f:00.0
2019-09-10 14:43:33.639590: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
2019-09-10 14:43:33.639624: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10
2019-09-10 14:43:33.639643: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcufft.so.10
2019-09-10 14:43:33.639661: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcurand.so.10
2019-09-10 14:43:33.639678: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusolver.so.10
2019-09-10 14:43:33.639694: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusparse.so.10
2019-09-10 14:43:33.639711: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
2019-09-10 14:43:33.652171: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0, 1, 2, 3
2019-09-10 14:43:33.652232: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1181] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-09-10 14:43:33.652243: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1187]      0 1 2 3
2019-09-10 14:43:33.652251: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 0:   N Y Y Y
2019-09-10 14:43:33.652258: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 1:   Y N Y Y
2019-09-10 14:43:33.652264: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 2:   Y Y N Y
2019-09-10 14:43:33.652271: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 3:   Y Y Y N
2019-09-10 14:43:33.660022: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 30438 MB memory) -> physical GPU (device: 0, name: Tesla V100-DGXS-32GB, pci bus id: 0000:07:00.0, compute capability: 7.0)
2019-09-10 14:43:33.661638: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 30463 MB memory) -> physical GPU (device: 1, name: Tesla V100-DGXS-32GB, pci bus id: 0000:08:00.0, compute capability: 7.0)
2019-09-10 14:43:33.663217: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:2 with 30463 MB memory) -> physical GPU (device: 2, name: Tesla V100-DGXS-32GB, pci bus id: 0000:0e:00.0, compute capability: 7.0)
2019-09-10 14:43:33.664818: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:3 with 30463 MB memory) -> physical GPU (device: 3, name: Tesla V100-DGXS-32GB, pci bus id: 0000:0f:00.0, compute capability: 7.0)
2019-09-10 14:43:35.981527: W tensorflow/compiler/jit/mark_for_compilation_pass.cc:1412] (One-time warning): Not using XLA:CPU for cluster because envvar TF_XLA_FLAGS=--tf_xla_cpu_global_jit was not set.  If you want XLA:CPU, either set that envvar, or use experimental_jit_scope to enable XLA:CPU.  To confirm that XLA is active, pass --vmodule=xla_compilation_cache=1 (as a proper command-line flag, not via TF_XLA_FLAGS) or set the envvar XLA_FLAGS=--xla_hlo_profile.
I0910 14:43:39.000194 139844950951744 session_manager.py:500] Running local_init_op.
I0910 14:43:40.515112 139844950951744 session_manager.py:502] Done running local_init_op.
Running warm up
2019-09-10 14:43:45.714711: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10
2019-09-10 14:43:47.097411: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
Done warm up
Step    Img/sec total_loss
1       images/sec: 1597.2 +/- 0.0 (jitter = 0.0)       7.912
10      images/sec: 1597.7 +/- 0.5 (jitter = 1.9)       7.832
20      images/sec: 1597.7 +/- 0.5 (jitter = 2.4)       7.820
30      images/sec: 1597.3 +/- 0.5 (jitter = 2.0)       7.779
40      images/sec: 1597.0 +/- 0.4 (jitter = 2.0)       7.698
50      images/sec: 1596.4 +/- 0.4 (jitter = 2.4)       7.668
60      images/sec: 1596.2 +/- 0.4 (jitter = 2.5)       7.652
70      images/sec: 1595.6 +/- 0.4 (jitter = 2.5)       7.602
80      images/sec: 1595.4 +/- 0.4 (jitter = 2.7)       7.606
90      images/sec: 1595.1 +/- 0.4 (jitter = 2.9)       7.549
100     images/sec: 1594.7 +/- 0.4 (jitter = 3.6)       7.540
----------------------------------------------------------------
total images/sec: 1594.55
----------------------------------------------------------------
2019-09-10 14:45:12.106464: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
WARNING: Logging before flag parsing goes to stderr.
W0910 14:45:14.095043 140505083311936 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/models/mobilenet.py:388: The name tf.nn.avg_pool is deprecated. Please use tf.nn.avg_pool2d instead.

W0910 14:45:14.099504 140505083311936 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/models/experimental/deepspeech.py:131: The name tf.nn.rnn_cell.RNNCell is deprecated. Please use tf.compat.v1.nn.rnn_cell.RNNCell instead.

W0910 14:45:14.122840 140505083311936 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:722: The name tf.ConfigProto is deprecated. Please use tf.compat.v1.ConfigProto instead.

W0910 14:45:14.123056 140505083311936 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:3503: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead.

2019-09-10 14:45:14.148721: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2198780000 Hz
2019-09-10 14:45:14.151158: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x338eaa0 executing computations on platform Host. Devices:
2019-09-10 14:45:14.151197: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): <undefined>, <undefined>
2019-09-10 14:45:14.155209: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcuda.so.1
2019-09-10 14:45:14.938336: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x338ee00 executing computations on platform CUDA. Devices:
2019-09-10 14:45:14.938371: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): Tesla V100-DGXS-32GB, Compute Capability 7.0
2019-09-10 14:45:14.938382: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (1): Tesla V100-DGXS-32GB, Compute Capability 7.0
2019-09-10 14:45:14.938391: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (2): Tesla V100-DGXS-32GB, Compute Capability 7.0
2019-09-10 14:45:14.938400: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (3): Tesla V100-DGXS-32GB, Compute Capability 7.0
2019-09-10 14:45:14.940939: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:07:00.0
2019-09-10 14:45:14.943287: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 1 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:08:00.0
2019-09-10 14:45:14.945629: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 2 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:0e:00.0
2019-09-10 14:45:14.947958: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 3 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:0f:00.0
2019-09-10 14:45:14.948002: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
2019-09-10 14:45:14.950709: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10
2019-09-10 14:45:14.952981: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcufft.so.10
2019-09-10 14:45:14.953268: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcurand.so.10
2019-09-10 14:45:14.954995: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusolver.so.10
2019-09-10 14:45:14.955993: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusparse.so.10
2019-09-10 14:45:14.959616: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
2019-09-10 14:45:14.971484: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0, 1, 2, 3
2019-09-10 14:45:14.971521: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
2019-09-10 14:45:16.766139: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1181] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-09-10 14:45:16.766181: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1187]      0 1 2 3
2019-09-10 14:45:16.766192: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 0:   N Y Y Y
2019-09-10 14:45:16.766199: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 1:   Y N Y Y
2019-09-10 14:45:16.766205: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 2:   Y Y N Y
2019-09-10 14:45:16.766211: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 3:   Y Y Y N
2019-09-10 14:45:16.774212: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 30438 MB memory) -> physical GPU (device: 0, name: Tesla V100-DGXS-32GB, pci bus id: 0000:07:00.0, compute capability: 7.0)
2019-09-10 14:45:16.776464: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 30463 MB memory) -> physical GPU (device: 1, name: Tesla V100-DGXS-32GB, pci bus id: 0000:08:00.0, compute capability: 7.0)
2019-09-10 14:45:16.778609: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:2 with 30463 MB memory) -> physical GPU (device: 2, name: Tesla V100-DGXS-32GB, pci bus id: 0000:0e:00.0, compute capability: 7.0)
2019-09-10 14:45:16.780744: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:3 with 30463 MB memory) -> physical GPU (device: 3, name: Tesla V100-DGXS-32GB, pci bus id: 0000:0f:00.0, compute capability: 7.0)
TensorFlow:  1.14
Model:       resnet50
Dataset:     imagenet (synthetic)
Mode:        training
SingleSess:  False
Batch size:  1024 global
             256 per device
Num batches: 100
Num epochs:  0.08
Devices:     ['/gpu:0', '/gpu:1', '/gpu:2', '/gpu:3']
NUMA bind:   False
Data format: NHWC
Optimizer:   sgd
Variables:   replicated
AllReduce:   None
==========
W0910 14:45:16.786987 140505083311936 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2762: The name tf.set_random_seed is deprecated. Please use tf.compat.v1.set_random_seed instead.

Generating training model
W0910 14:45:16.787610 140505083311936 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2782: The name tf.train.get_or_create_global_step is deprecated. Please use tf.compat.v1.train.get_or_create_global_step instead.

W0910 14:45:16.795541 140505083311936 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2739: The name tf.get_variable is deprecated. Please use tf.compat.v1.get_variable instead.

W0910 14:45:16.800475 140505083311936 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/variable_mgr.py:316: The name tf.variable_scope is deprecated. Please use tf.compat.v1.variable_scope instead.

W0910 14:45:16.818038 140505083311936 deprecation.py:323] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:129: conv2d (from tensorflow.python.layers.convolutional) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.keras.layers.Conv2D` instead.
W0910 14:45:17.184011 140505083311936 deprecation.py:323] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:261: max_pooling2d (from tensorflow.python.layers.pooling) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.MaxPooling2D instead.
W0910 14:45:19.924096 140505083311936 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:334: The name tf.nn.xw_plus_b is deprecated. Please use tf.compat.v1.nn.xw_plus_b instead.

W0910 14:45:19.927115 140505083311936 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/models/model.py:319: The name tf.losses.sparse_softmax_cross_entropy is deprecated. Please use tf.compat.v1.losses.sparse_softmax_cross_entropy instead.

W0910 14:45:19.939219 140505083311936 deprecation.py:323] From /usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/losses/losses_impl.py:121: add_dispatch_support.<locals>.wrapper (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.where in 2.0, which has the same broadcast rule as np.where
W0910 14:45:33.494938 140505083311936 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/models/resnet_model.py:333: The name tf.train.piecewise_constant is deprecated. Please use tf.compat.v1.train.piecewise_constant instead.

W0910 14:45:33.585247 140505083311936 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:1209: The name tf.train.GradientDescentOptimizer is deprecated. Please use tf.compat.v1.train.GradientDescentOptimizer instead.

W0910 14:45:34.727916 140505083311936 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2090: The name tf.train.get_global_step is deprecated. Please use tf.compat.v1.train.get_global_step instead.

W0910 14:45:35.915678 140505083311936 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2126: The name tf.summary.merge_all is deprecated. Please use tf.compat.v1.summary.merge_all instead.

Initializing graph
W0910 14:45:35.915928 140505083311936 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2183: The name tf.train.Saver is deprecated. Please use tf.compat.v1.train.Saver instead.

W0910 14:45:36.188548 140505083311936 deprecation.py:323] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2238: Supervisor.__init__ (from tensorflow.python.training.supervisor) is deprecated and will be removed in a future version.
Instructions for updating:
Please switch to tf.train.MonitoredTrainingSession
2019-09-10 14:45:38.263424: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:07:00.0
2019-09-10 14:45:38.265288: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 1 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:08:00.0
2019-09-10 14:45:38.267183: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 2 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:0e:00.0
2019-09-10 14:45:38.269051: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 3 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:0f:00.0
2019-09-10 14:45:38.269084: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
2019-09-10 14:45:38.269120: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10
2019-09-10 14:45:38.269140: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcufft.so.10
2019-09-10 14:45:38.269158: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcurand.so.10
2019-09-10 14:45:38.269175: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusolver.so.10
2019-09-10 14:45:38.269193: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusparse.so.10
2019-09-10 14:45:38.269212: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
2019-09-10 14:45:38.281934: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0, 1, 2, 3
2019-09-10 14:45:38.281996: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1181] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-09-10 14:45:38.282006: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1187]      0 1 2 3
2019-09-10 14:45:38.282015: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 0:   N Y Y Y
2019-09-10 14:45:38.282021: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 1:   Y N Y Y
2019-09-10 14:45:38.282028: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 2:   Y Y N Y
2019-09-10 14:45:38.282034: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 3:   Y Y Y N
2019-09-10 14:45:38.289954: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 30438 MB memory) -> physical GPU (device: 0, name: Tesla V100-DGXS-32GB, pci bus id: 0000:07:00.0, compute capability: 7.0)
2019-09-10 14:45:38.291599: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 30463 MB memory) -> physical GPU (device: 1, name: Tesla V100-DGXS-32GB, pci bus id: 0000:08:00.0, compute capability: 7.0)
2019-09-10 14:45:38.293206: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:2 with 30463 MB memory) -> physical GPU (device: 2, name: Tesla V100-DGXS-32GB, pci bus id: 0000:0e:00.0, compute capability: 7.0)
2019-09-10 14:45:38.294782: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:3 with 30463 MB memory) -> physical GPU (device: 3, name: Tesla V100-DGXS-32GB, pci bus id: 0000:0f:00.0, compute capability: 7.0)
2019-09-10 14:45:40.777442: W tensorflow/compiler/jit/mark_for_compilation_pass.cc:1412] (One-time warning): Not using XLA:CPU for cluster because envvar TF_XLA_FLAGS=--tf_xla_cpu_global_jit was not set.  If you want XLA:CPU, either set that envvar, or use experimental_jit_scope to enable XLA:CPU.  To confirm that XLA is active, pass --vmodule=xla_compilation_cache=1 (as a proper command-line flag, not via TF_XLA_FLAGS) or set the envvar XLA_FLAGS=--xla_hlo_profile.
I0910 14:45:43.813993 140505083311936 session_manager.py:500] Running local_init_op.
I0910 14:45:45.382833 140505083311936 session_manager.py:502] Done running local_init_op.
Running warm up
2019-09-10 14:45:51.948691: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10
2019-09-10 14:45:53.433423: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
Done warm up
Step    Img/sec total_loss
1       images/sec: 3165.2 +/- 0.0 (jitter = 0.0)       7.944
10      images/sec: 3165.8 +/- 1.1 (jitter = 1.9)       7.908
20      images/sec: 3168.8 +/- 1.5 (jitter = 3.9)       7.823
30      images/sec: 3168.9 +/- 1.1 (jitter = 7.2)       7.786
40      images/sec: 3167.5 +/- 1.1 (jitter = 7.4)       7.710
50      images/sec: 3166.9 +/- 0.9 (jitter = 7.2)       7.712
60      images/sec: 3168.0 +/- 1.0 (jitter = 8.1)       7.654
70      images/sec: 3168.1 +/- 0.9 (jitter = 8.3)       7.621
80      images/sec: 3168.1 +/- 0.9 (jitter = 8.6)       7.562
90      images/sec: 3168.0 +/- 0.8 (jitter = 8.4)       7.541
100     images/sec: 3168.4 +/- 0.7 (jitter = 8.3)       7.511
----------------------------------------------------------------
total images/sec: 3167.72
----------------------------------------------------------------
2019-09-10 14:46:43.058012: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
WARNING: Logging before flag parsing goes to stderr.
W0910 14:46:45.087880 139836202166080 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/models/mobilenet.py:388: The name tf.nn.avg_pool is deprecated. Please use tf.nn.avg_pool2d instead.

W0910 14:46:45.092403 139836202166080 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/models/experimental/deepspeech.py:131: The name tf.nn.rnn_cell.RNNCell is deprecated. Please use tf.compat.v1.nn.rnn_cell.RNNCell instead.

W0910 14:46:45.115877 139836202166080 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:722: The name tf.ConfigProto is deprecated. Please use tf.compat.v1.ConfigProto instead.

W0910 14:46:45.116084 139836202166080 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:3503: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead.

2019-09-10 14:46:45.140710: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2198780000 Hz
2019-09-10 14:46:45.143276: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x3062300 executing computations on platform Host. Devices:
2019-09-10 14:46:45.143309: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): <undefined>, <undefined>
2019-09-10 14:46:45.146618: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcuda.so.1
2019-09-10 14:46:45.926347: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x48e7650 executing computations on platform CUDA. Devices:
2019-09-10 14:46:45.926394: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): Tesla V100-DGXS-32GB, Compute Capability 7.0
2019-09-10 14:46:45.926412: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (1): Tesla V100-DGXS-32GB, Compute Capability 7.0
2019-09-10 14:46:45.926426: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (2): Tesla V100-DGXS-32GB, Compute Capability 7.0
2019-09-10 14:46:45.926439: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (3): Tesla V100-DGXS-32GB, Compute Capability 7.0
2019-09-10 14:46:45.929875: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:07:00.0
2019-09-10 14:46:45.932745: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 1 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:08:00.0
2019-09-10 14:46:45.935583: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 2 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:0e:00.0
2019-09-10 14:46:45.938430: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 3 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:0f:00.0
2019-09-10 14:46:45.938477: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
2019-09-10 14:46:45.941454: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10
2019-09-10 14:46:45.943216: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcufft.so.10
2019-09-10 14:46:45.943555: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcurand.so.10
2019-09-10 14:46:45.945592: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusolver.so.10
2019-09-10 14:46:45.946771: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusparse.so.10
2019-09-10 14:46:45.950975: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
2019-09-10 14:46:45.964752: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0, 1, 2, 3
2019-09-10 14:46:45.964790: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
2019-09-10 14:46:47.763412: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1181] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-09-10 14:46:47.763451: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1187]      0 1 2 3
2019-09-10 14:46:47.763461: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 0:   N Y Y Y
2019-09-10 14:46:47.763468: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 1:   Y N Y Y
2019-09-10 14:46:47.763475: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 2:   Y Y N Y
2019-09-10 14:46:47.763481: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 3:   Y Y Y N
2019-09-10 14:46:47.771501: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 30438 MB memory) -> physical GPU (device: 0, name: Tesla V100-DGXS-32GB, pci bus id: 0000:07:00.0, compute capability: 7.0)
2019-09-10 14:46:47.773692: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 30463 MB memory) -> physical GPU (device: 1, name: Tesla V100-DGXS-32GB, pci bus id: 0000:08:00.0, compute capability: 7.0)
2019-09-10 14:46:47.775869: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:2 with 30463 MB memory) -> physical GPU (device: 2, name: Tesla V100-DGXS-32GB, pci bus id: 0000:0e:00.0, compute capability: 7.0)
2019-09-10 14:46:47.778028: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:3 with 30463 MB memory) -> physical GPU (device: 3, name: Tesla V100-DGXS-32GB, pci bus id: 0000:0f:00.0, compute capability: 7.0)
TensorFlow:  1.14
Model:       resnet50
Dataset:     imagenet (synthetic)
Mode:        training
SingleSess:  False
Batch size:  2048 global
             512 per device
Num batches: 100
Num epochs:  0.16
Devices:     ['/gpu:0', '/gpu:1', '/gpu:2', '/gpu:3']
NUMA bind:   False
Data format: NHWC
Optimizer:   sgd
Variables:   replicated
AllReduce:   None
==========
W0910 14:46:47.783979 139836202166080 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2762: The name tf.set_random_seed is deprecated. Please use tf.compat.v1.set_random_seed instead.

Generating training model
W0910 14:46:47.784570 139836202166080 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2782: The name tf.train.get_or_create_global_step is deprecated. Please use tf.compat.v1.train.get_or_create_global_step instead.

W0910 14:46:47.793416 139836202166080 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2739: The name tf.get_variable is deprecated. Please use tf.compat.v1.get_variable instead.

W0910 14:46:47.798059 139836202166080 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/variable_mgr.py:316: The name tf.variable_scope is deprecated. Please use tf.compat.v1.variable_scope instead.

W0910 14:46:47.814631 139836202166080 deprecation.py:323] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:129: conv2d (from tensorflow.python.layers.convolutional) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.keras.layers.Conv2D` instead.
W0910 14:46:48.174123 139836202166080 deprecation.py:323] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:261: max_pooling2d (from tensorflow.python.layers.pooling) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.MaxPooling2D instead.
W0910 14:46:50.758634 139836202166080 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:334: The name tf.nn.xw_plus_b is deprecated. Please use tf.compat.v1.nn.xw_plus_b instead.

W0910 14:46:50.761462 139836202166080 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/models/model.py:319: The name tf.losses.sparse_softmax_cross_entropy is deprecated. Please use tf.compat.v1.losses.sparse_softmax_cross_entropy instead.

W0910 14:46:50.772780 139836202166080 deprecation.py:323] From /usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/losses/losses_impl.py:121: add_dispatch_support.<locals>.wrapper (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.where in 2.0, which has the same broadcast rule as np.where
W0910 14:47:03.522151 139836202166080 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/models/resnet_model.py:333: The name tf.train.piecewise_constant is deprecated. Please use tf.compat.v1.train.piecewise_constant instead.

W0910 14:47:03.607043 139836202166080 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:1209: The name tf.train.GradientDescentOptimizer is deprecated. Please use tf.compat.v1.train.GradientDescentOptimizer instead.

W0910 14:47:04.679104 139836202166080 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2090: The name tf.train.get_global_step is deprecated. Please use tf.compat.v1.train.get_global_step instead.

W0910 14:47:05.784687 139836202166080 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2126: The name tf.summary.merge_all is deprecated. Please use tf.compat.v1.summary.merge_all instead.

Initializing graph
W0910 14:47:05.784940 139836202166080 deprecation_wrapper.py:119] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2183: The name tf.train.Saver is deprecated. Please use tf.compat.v1.train.Saver instead.

W0910 14:47:06.040027 139836202166080 deprecation.py:323] From /workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2238: Supervisor.__init__ (from tensorflow.python.training.supervisor) is deprecated and will be removed in a future version.
Instructions for updating:
Please switch to tf.train.MonitoredTrainingSession
2019-09-10 14:47:07.945425: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:07:00.0
2019-09-10 14:47:07.947246: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 1 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:08:00.0
2019-09-10 14:47:07.949090: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 2 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:0e:00.0
2019-09-10 14:47:07.950916: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 3 with properties:
name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:0f:00.0
2019-09-10 14:47:07.950948: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
2019-09-10 14:47:07.950982: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10
2019-09-10 14:47:07.951000: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcufft.so.10
2019-09-10 14:47:07.951017: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcurand.so.10
2019-09-10 14:47:07.951033: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusolver.so.10
2019-09-10 14:47:07.951049: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusparse.so.10
2019-09-10 14:47:07.951066: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
2019-09-10 14:47:07.963704: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0, 1, 2, 3
2019-09-10 14:47:07.963762: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1181] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-09-10 14:47:07.963772: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1187]      0 1 2 3
2019-09-10 14:47:07.963780: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 0:   N Y Y Y
2019-09-10 14:47:07.963786: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 1:   Y N Y Y
2019-09-10 14:47:07.963793: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 2:   Y Y N Y
2019-09-10 14:47:07.963799: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 3:   Y Y Y N
2019-09-10 14:47:07.971680: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 30438 MB memory) -> physical GPU (device: 0, name: Tesla V100-DGXS-32GB, pci bus id: 0000:07:00.0, compute capability: 7.0)
2019-09-10 14:47:07.973274: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 30463 MB memory) -> physical GPU (device: 1, name: Tesla V100-DGXS-32GB, pci bus id: 0000:08:00.0, compute capability: 7.0)
2019-09-10 14:47:07.974844: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:2 with 30463 MB memory) -> physical GPU (device: 2, name: Tesla V100-DGXS-32GB, pci bus id: 0000:0e:00.0, compute capability: 7.0)
2019-09-10 14:47:07.976407: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:3 with 30463 MB memory) -> physical GPU (device: 3, name: Tesla V100-DGXS-32GB, pci bus id: 0000:0f:00.0, compute capability: 7.0)
2019-09-10 14:47:10.332216: W tensorflow/compiler/jit/mark_for_compilation_pass.cc:1412] (One-time warning): Not using XLA:CPU for cluster because envvar TF_XLA_FLAGS=--tf_xla_cpu_global_jit was not set.  If you want XLA:CPU, either set that envvar, or use experimental_jit_scope to enable XLA:CPU.  To confirm that XLA is active, pass --vmodule=xla_compilation_cache=1 (as a proper command-line flag, not via TF_XLA_FLAGS) or set the envvar XLA_FLAGS=--xla_hlo_profile.
I0910 14:47:13.328989 139836202166080 session_manager.py:500] Running local_init_op.
I0910 14:47:14.842230 139836202166080 session_manager.py:502] Done running local_init_op.
Running warm up
2019-09-10 14:47:21.165005: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10
2019-09-10 14:47:22.630395: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
Done warm up
Step    Img/sec total_loss
1       images/sec: 3289.7 +/- 0.0 (jitter = 0.0)       7.853
10      images/sec: 3288.5 +/- 1.6 (jitter = 5.1)       7.798
20      images/sec: 3288.1 +/- 0.9 (jitter = 3.4)       7.711
30      images/sec: 3289.6 +/- 0.8 (jitter = 3.3)       7.608
40      images/sec: 3289.9 +/- 0.6 (jitter = 4.1)       7.547
50      images/sec: 3289.9 +/- 0.6 (jitter = 4.1)       7.516
60      images/sec: 3290.2 +/- 0.5 (jitter = 4.1)       7.492
70      images/sec: 3290.7 +/- 0.5 (jitter = 4.1)       7.480
80      images/sec: 3290.9 +/- 0.5 (jitter = 4.3)       7.476
90      images/sec: 3291.1 +/- 0.4 (jitter = 4.2)       7.477
100     images/sec: 3291.0 +/- 0.4 (jitter = 3.9)       7.468
----------------------------------------------------------------
total images/sec: 3290.58
----------------------------------------------------------------
root@692ced701c1d:/workspace/benchmarks/scripts/tf_cnn_benchmarks#