cuda_racecheck.txt

(lasagne) lulu@lulus ➜  theano git:(master) ✗  THEANO_FLAGS='device=gpu0' cuda-memcheck --tool racecheck python digits.py 
========= CUDA-MEMCHECK
Using gpu device 0: GeForce GTX 980 Ti (CNMeM is disabled, CuDNN 4007)
========= Race reported between Write access at 0x00000370 in kernel_reduce_ccontig_node_meb404c8cd39208f6884dd773b584b7d7_0(unsigned int, float const *, float*)
=========     and Read access at 0x000003a8 in kernel_reduce_ccontig_node_meb404c8cd39208f6884dd773b584b7d7_0(unsigned int, float const *, float*) [64 hazards]
=========
========= Race reported between Read access at 0x00000458 in kernel_reduce_ccontig_node_meb404c8cd39208f6884dd773b584b7d7_0(unsigned int, float const *, float*)
=========     and Write access at 0x00000450 in kernel_reduce_ccontig_node_meb404c8cd39208f6884dd773b584b7d7_0(unsigned int, float const *, float*) [56 hazards]
=========     and Write access at 0x00000488 in kernel_reduce_ccontig_node_meb404c8cd39208f6884dd773b584b7d7_0(unsigned int, float const *, float*) [56 hazards]
=========
========= Race reported between Read access at 0x00000490 in kernel_reduce_ccontig_node_meb404c8cd39208f6884dd773b584b7d7_0(unsigned int, float const *, float*)
=========     and Write access at 0x000004b8 in kernel_reduce_ccontig_node_meb404c8cd39208f6884dd773b584b7d7_0(unsigned int, float const *, float*) [60 hazards]
=========     and Write access at 0x00000488 in kernel_reduce_ccontig_node_meb404c8cd39208f6884dd773b584b7d7_0(unsigned int, float const *, float*) [60 hazards]
=========
========= Race reported between Write access at 0x00000450 in kernel_reduce_ccontig_node_meb404c8cd39208f6884dd773b584b7d7_0(unsigned int, float const *, float*)
=========     and Read access at 0x00000418 in kernel_reduce_ccontig_node_meb404c8cd39208f6884dd773b584b7d7_0(unsigned int, float const *, float*) [48 hazards]
=========     and Read access at 0x00000458 in kernel_reduce_ccontig_node_meb404c8cd39208f6884dd773b584b7d7_0(unsigned int, float const *, float*) [56 hazards]
=========
========= Race reported between Read access at 0x00000418 in kernel_reduce_ccontig_node_meb404c8cd39208f6884dd773b584b7d7_0(unsigned int, float const *, float*)
=========     and Write access at 0x00000450 in kernel_reduce_ccontig_node_meb404c8cd39208f6884dd773b584b7d7_0(unsigned int, float const *, float*) [48 hazards]
=========     and Write access at 0x00000410 in kernel_reduce_ccontig_node_meb404c8cd39208f6884dd773b584b7d7_0(unsigned int, float const *, float*) [48 hazards]
=========
========= Race reported between Write access at 0x00000410 in kernel_reduce_ccontig_node_meb404c8cd39208f6884dd773b584b7d7_0(unsigned int, float const *, float*)
=========     and Read access at 0x00000418 in kernel_reduce_ccontig_node_meb404c8cd39208f6884dd773b584b7d7_0(unsigned int, float const *, float*) [48 hazards]
=========     and Read access at 0x000003d8 in kernel_reduce_ccontig_node_meb404c8cd39208f6884dd773b584b7d7_0(unsigned int, float const *, float*) [32 hazards]
=========
========= Race reported between Read access at 0x000003d8 in kernel_reduce_ccontig_node_meb404c8cd39208f6884dd773b584b7d7_0(unsigned int, float const *, float*)
=========     and Write access at 0x000003d0 in kernel_reduce_ccontig_node_meb404c8cd39208f6884dd773b584b7d7_0(unsigned int, float const *, float*) [32 hazards]
=========     and Write access at 0x00000410 in kernel_reduce_ccontig_node_meb404c8cd39208f6884dd773b584b7d7_0(unsigned int, float const *, float*) [32 hazards]
=========
/home/lulu/envs/lasagne/lib/python3.5/site-packages/theano/tensor/signal/downsample.py:5: UserWarning: downsample module has been moved to the pool module.
  warnings.warn("downsample module has been moved to the pool module.")
# Neural Network with 33344 learnable parameters

## Layer information

name            size      total    cap.Y    cap.X    cov.Y    cov.X
--------------  ------  -------  -------  -------  -------  -------
InputLayer      1x8x8        64   100.00   100.00   100.00   100.00
Conv2DLayer     64x6x6     2304   100.00   100.00    37.50    37.50
MaxPool2DLayer  64x2x2      256   100.00   100.00    37.50    37.50
DenseLayer      128         128   100.00   100.00   100.00   100.00
InverseLayer    64x2x2      256   100.00   100.00   100.00   100.00
InverseLayer    64x6x6     2304   100.00   100.00   100.00   100.00
InverseLayer    1x8x8        64   100.00   100.00   100.00   100.00
FlattenLayer    64           64   100.00   100.00   100.00   100.00

Explanation
    X, Y:    image dimensions
    cap.:    learning capacity
    cov.:    coverage of image
    magenta: capacity too low (<1/6)
    cyan:    image coverage too high (>100%)
    red:     capacity too low and coverage too high


Error when trying to find the memory information on the GPU: an illegal memory access was encountered
Error allocating 65536 bytes of device memory (an illegal memory access was encountered). Driver report 0 bytes free and 0 bytes total 
Traceback (most recent call last):
  File "/home/lulu/envs/lasagne/lib/python3.5/site-packages/theano/compile/function_module.py", line 859, in __call__
    outputs = self.fn()
MemoryError: Error allocating 65536 bytes of device memory (an illegal memory access was encountered).

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "digits.py", line 31, in <module>
    net.fit(X, X_flat)
  File "/home/lulu/envs/lasagne/lib/python3.5/site-packages/nolearn/lasagne/base.py", line 521, in fit
    self.train_loop(X, y, epochs=epochs)
  File "/home/lulu/envs/lasagne/lib/python3.5/site-packages/nolearn/lasagne/base.py", line 578, in train_loop
    self.train_iter_, Xb, yb)
  File "/home/lulu/envs/lasagne/lib/python3.5/site-packages/nolearn/lasagne/base.py", line 638, in apply_batch_func
    return func(Xb) if yb is None else func(Xb, yb)
  File "/home/lulu/envs/lasagne/lib/python3.5/site-packages/theano/compile/function_module.py", line 871, in __call__
    storage_map=getattr(self.fn, 'storage_map', None))
  File "/home/lulu/envs/lasagne/lib/python3.5/site-packages/theano/gof/link.py", line 314, in raise_with_op
    reraise(exc_type, exc_value, exc_trace)
  File "/usr/lib/python3.5/site-packages/six.py", line 685, in reraise
    raise value.with_traceback(tb)
  File "/home/lulu/envs/lasagne/lib/python3.5/site-packages/theano/compile/function_module.py", line 859, in __call__
    outputs = self.fn()
MemoryError: Error allocating 65536 bytes of device memory (an illegal memory access was encountered).
Apply node that caused the error: GpuDot22(GpuReshape{2}.0, W)
Toposort index: 58
Inputs types: [CudaNdarrayType(float32, matrix), CudaNdarrayType(float32, matrix)]
Inputs shapes: [(128, 256), (256, 128)]
Inputs strides: [(256, 1), (128, 1)]
Inputs values: ['not shown', 'not shown']
Outputs clients: [[GpuDot22(GpuDot22.0, GpuDimShuffle{1,0}.0), GpuGemm{inplace}(GpuDot22.0, TensorConstant{0.009999999776482582}, GpuDimShuffle{1,0}.0, GpuDot22.0, TensorConstant{0.009999999776482582})]]

HINT: Re-running with most Theano optimization disabled could give you a back-trace of when this node was created. This can be done with by setting the Theano flag 'optimizer=fast_compile'. If that does not work, Theano optimizations can be disabled with 'optimizer=None'.
HINT: Use the Theano flag 'exception_verbosity=high' for a debugprint and storage map footprint of this apply node.
========= RACECHECK SUMMARY: 7 hazards displayed (7 errors, 0 warnings)