fp16_bsz4_1gpu_error.txt

[0308 18:43:36 @base.py:274] Start Epoch 1 ...
  0%|                                                                                                                                                                                |0/15000[00:00<?,?it/s]2019-03-08 18:43:57.515207: E tensorflow/stream_executor/cuda/cuda_blas.cc:694] failed to run cuBLAS routine cublasSgemmEx: CUBLAS_STATUS_EXECUTION_FAILED
2019-03-08 18:43:57.522957: F tensorflow/stream_executor/gpu/gpu_timer.cc:65] Check failed: start_event_ != nullptr && stop_event_ != nullptr
[ip-172-31-14-112:17513] *** Process received signal ***
[ip-172-31-14-112:17513] Signal: Aborted (6)
[ip-172-31-14-112:17513] Signal code:  (-6)
[ip-172-31-14-112:17513] [ 0] /lib/x86_64-linux-gnu/libpthread.so.0(+0x11390)[0x7fc9d8af2390]
[ip-172-31-14-112:17513] [ 1] /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x38)[0x7fc9d874c428]
[ip-172-31-14-112:17513] [ 2] /lib/x86_64-linux-gnu/libc.so.6(abort+0x16a)[0x7fc9d874e02a]
[ip-172-31-14-112:17513] [ 3] /home/ubuntu/anaconda3/envs/tensorflow_p36_13rc1/lib/python3.6/site-packages/tensorflow/python/_pywrap_tensorflow_internal.so(+0x6c66fb4)[0x7fc9bac65fb4]
[ip-172-31-14-112:17513] [ 4] /home/ubuntu/anaconda3/envs/tensorflow_p36_13rc1/lib/python3.6/site-packages/tensorflow/python/../libtensorflow_framework.so(_ZNK15stream_executor3gpu8GpuTimer22GetElapsedMillisecondsEv+0x97)[0x7fc9b3a6f507]
[ip-172-31-14-112:17513] [ 5] /home/ubuntu/anaconda3/envs/tensorflow_p36_13rc1/lib/python3.6/site-packages/tensorflow/python/../libtensorflow_framework.so(_ZNK15stream_executor3gpu8GpuTimer12MicrosecondsEv+0x9)[0x7fc9b39d8959]
[ip-172-31-14-112:17513] [ 6] /home/ubuntu/anaconda3/envs/tensorflow_p36_13rc1/lib/python3.6/site-packages/tensorflow/python/_pywrap_tensorflow_internal.so(_ZN10tensorflow10BiasGradOpIN5Eigen9GpuDeviceENS1_4halfEE7ComputeEPNS_15OpKernelContextE+0x33a)[0x7fc9b7b752da]
[ip-172-31-14-112:17513] [ 7] /home/ubuntu/anaconda3/envs/tensorflow_p36_13rc1/lib/python3.6/site-packages/tensorflow/python/../libtensorflow_framework.so(_ZN10tensorflow13BaseGPUDevice13ComputeHelperEPNS_8OpKernelEPNS_15OpKernelContextE+0x48a)[0x7fc9b354aa6a]
[ip-172-31-14-112:17513] [ 8] /home/ubuntu/anaconda3/envs/tensorflow_p36_13rc1/lib/python3.6/site-packages/tensorflow/python/../libtensorflow_framework.so(_ZN10tensorflow13BaseGPUDevice7ComputeEPNS_8OpKernelEPNS_15OpKernelContextE+0x2a)[0x7fc9b354b78a]
[ip-172-31-14-112:17513] [ 9] /home/ubuntu/anaconda3/envs/tensorflow_p36_13rc1/lib/python3.6/site-packages/tensorflow/python/../libtensorflow_framework.so(+0x77dbe0)[0x7fc9b35a1be0]
[ip-172-31-14-112:17513] [10] /home/ubuntu/anaconda3/envs/tensorflow_p36_13rc1/lib/python3.6/site-packages/tensorflow/python/../libtensorflow_framework.so(+0x77dc6f)[0x7fc9b35a1c6f]
[ip-172-31-14-112:17513] [11] /home/ubuntu/anaconda3/envs/tensorflow_p36_13rc1/lib/python3.6/site-packages/tensorflow/python/../libtensorflow_framework.so(_ZN5Eigen15ThreadPoolTemplIN10tensorflow6thread16EigenEnvironmentEE10WorkerLoopEi+0x2e2)[0x7fc9b3630c72]
[ip-172-31-14-112:17513] [12] /home/ubuntu/anaconda3/envs/tensorflow_p36_13rc1/lib/python3.6/site-packages/tensorflow/python/../libtensorflow_framework.so(_ZNSt17_Function_handlerIFvvEZN10tensorflow6thread16EigenEnvironment12CreateThreadESt8functionIS0_EEUlvE_E9_M_invokeERKSt9_Any_data+0x48)[0x7fc9b362de68]
[ip-172-31-14-112:17513] [13] /home/ubuntu/anaconda3/envs/tensorflow_p36_13rc1/bin/../lib/libstdc++.so.6(+0xafc5c)[0x7fc9c682ec5c]
[ip-172-31-14-112:17513] [14] /lib/x86_64-linux-gnu/libpthread.so.0(+0x76ba)[0x7fc9d8ae86ba]
[ip-172-31-14-112:17513] [15] /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7fc9d881e41d]
[ip-172-31-14-112:17513] *** End of error message ***