my TF gan training crash log

log.txt

global_step = 243830
global_step = 243840
global_step = 243850
2018-11-20 23:42:59.740578: W tensorflow/core/common_runtime/bfc_allocator.cc:219] Allocator (GPU_0_bfc) ran out of memory trying to allocate 301.50MiB. The caller indicates that this is not a failure, bu
t may mean that there could be performance gains if more memory were available.
2018-11-20 23:42:59.741211: W tensorflow/core/common_runtime/bfc_allocator.cc:219] Allocator (GPU_0_bfc) ran out of memory trying to allocate 301.50MiB. The caller indicates that this is not a failure, bu
t may mean that there could be performance gains if more memory were available.
2018-11-20 23:42:59.741553: E tensorflow/stream_executor/cuda/cuda_dnn.cc:81] 
Traceback (most recent call last):
  File "/home/yusuke/.pyenv/versions/tf1.10/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1278, in _do_call
    return fn(*args)
  File "/home/yusuke/.pyenv/versions/tf1.10/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1263, in _run_fn
    options, feed_dict, fetch_list, target_list, run_metadata)
  File "/home/yusuke/.pyenv/versions/tf1.10/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1350, in _call_tf_sessionrun
    run_metadata)
tensorflow.python.framework.errors_impl.InternalError: cuDNN Backward Filter function launch failure : input shape([16,1024,6,4]) filter shape([3,3,1024,2048])
         [[Node: gradients/gradients_2/model/discriminator_2/conv2d_5/Conv2D_grad/Conv2DBackpropFilter = Conv2DBackpropFilter[T=DT_FLOAT, data_format="NCHW", dilations=[1, 1, 1, 1], padding="SAME", stride
s=[1, 1, 1, 1], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](model/discriminator_2/max_pooling2d_4/MaxPool, ConstantFolding/gradients/gradients_2/model/discriminator_2/co
nv2d_5/Conv2D_grad/ShapeN-matshapes-1, gradients/gradients_2/AddN_1, ^gradients/gradients_2/model/discriminator_2/conv2d_5/BiasAdd_grad/BiasAddGrad)]]
         [[Node: ConstantFoldingCtrl/train/case/cond/cond/Switch_0/_1846 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/
task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_8464_ConstantFoldingCtrl/train/case/cond/cond/Switch_0", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]
]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "memo.py", line 436, in <module>
    main(args)
  File "memo.py", line 427, in main
    _ = sess.run([train_op])
  File "/home/yusuke/.pyenv/versions/tf1.10/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 877, in run
    run_metadata_ptr)
  File "/home/yusuke/.pyenv/versions/tf1.10/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1100, in _run
    feed_dict_tensor, options, run_metadata)
  File "/home/yusuke/.pyenv/versions/tf1.10/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1272, in _do_run
    run_metadata)
  File "/home/yusuke/.pyenv/versions/tf1.10/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1291, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InternalError: cuDNN Backward Filter function launch failure : input shape([16,1024,6,4]) filter shape([3,3,1024,2048])
         [[Node: gradients/gradients_2/model/discriminator_2/conv2d_5/Conv2D_grad/Conv2DBackpropFilter = Conv2DBackpropFilter[T=DT_FLOAT, data_format="NCHW", dilations=[1, 1, 1, 1], padding="SAME", stride
s=[1, 1, 1, 1], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](model/discriminator_2/max_pooling2d_4/MaxPool, ConstantFolding/gradients/gradients_2/model/discriminator_2/co
nv2d_5/Conv2D_grad/ShapeN-matshapes-1, gradients/gradients_2/AddN_1, ^gradients/gradients_2/model/discriminator_2/conv2d_5/BiasAdd_grad/BiasAddGrad)]]
         [[Node: ConstantFoldingCtrl/train/case/cond/cond/Switch_0/_1846 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/
task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_8464_ConstantFoldingCtrl/train/case/cond/cond/Switch_0", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]
]