runtrace2.txt

(gpuc) 14:34:49 vladf@---:/data/vladf$ python tf_random_speed.py 
2017-12-23 14:35:45.782703: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1030] Found device 0 with properties: 
name: Tesla P100-PCIE-16GB major: 6 minor: 0 memoryClockRate(GHz): 1.3285
pciBusID: 0000:0d:00.0
totalMemory: 15.89GiB freeMemory: 14.60GiB
2017-12-23 14:35:45.782759: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1120] Creating TensorFlow device (/device:GPU:0) -> (device: 0, name: Tesla P100-PCIE-16GB, pci bus id: 0000:0d:00.0, compute capability: 6.0)
****************************************
feed_dict (randomize jit) 17.10606360435486 sec
loss 4542.5 -> 4.09381
****************************************
2017-12-23 14:36:05.769577: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1120] Creating TensorFlow device (/device:GPU:0) -> (device: 0, name: Tesla P100-PCIE-16GB, pci bus id: 0000:0d:00.0, compute capability: 6.0)
****************************************
feed_dict (randomize up-front) 17.26680040359497 sec
loss 4542.5 -> 4.06041
****************************************
2017-12-23 14:36:24.530787: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1120] Creating TensorFlow device (/device:GPU:0) -> (device: 0, name: Tesla P100-PCIE-16GB, pci bus id: 0000:0d:00.0, compute capability: 6.0)
****************************************
foldl 8.42160439491272 sec
loss 4539.77 -> 4.03162
****************************************
2017-12-23 14:36:34.383681: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1120] Creating TensorFlow device (/device:GPU:0) -> (device: 0, name: Tesla P100-PCIE-16GB, pci bus id: 0000:0d:00.0, compute capability: 6.0)
****************************************
while 7.964935779571533 sec
loss 4562.97 -> 4.39892
****************************************
2017-12-23 14:36:44.017151: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1120] Creating TensorFlow device (/device:GPU:0) -> (device: 0, name: Tesla P100-PCIE-16GB, pci bus id: 0000:0d:00.0, compute capability: 6.0)
****************************************
tf dataset (bufsize 1000 ) 12.421366214752197 sec
loss 4550.22 -> 4.2837
****************************************
2017-12-23 14:36:58.115517: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1120] Creating TensorFlow device (/device:GPU:0) -> (device: 0, name: Tesla P100-PCIE-16GB, pci bus id: 0000:0d:00.0, compute capability: 6.0)
****************************************
tf dataset (bufsize 10000 ) 12.547032833099365 sec
loss 4550.22 -> 4.10689
****************************************
2017-12-23 14:37:12.260579: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1120] Creating TensorFlow device (/device:GPU:0) -> (device: 0, name: Tesla P100-PCIE-16GB, pci bus id: 0000:0d:00.0, compute capability: 6.0)
****************************************
tf dataset (bufsize 100000 ) 13.499586343765259 sec
loss 4550.22 -> 4.62112
****************************************
2017-12-23 14:37:27.245548: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1120] Creating TensorFlow device (/device:GPU:0) -> (device: 0, name: Tesla P100-PCIE-16GB, pci bus id: 0000:0d:00.0, compute capability: 6.0)
****************************************
tf dataset (bufsize 1000000 ) 16.641777515411377 sec
loss 4550.22 -> 3.9002