Created
August 27, 2014 15:13
-
-
Save benanne/4128e5998122295f992b to your computer and use it in GitHub Desktop.
cublasSgemm bug, CUDA 6.0
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| * GeForce GTX 780Ti "Superclocked" | |
| * drivers 340.24 | |
| * CUDA 6.0 | |
| sander@sander-precision:~/tmp/schluter/Theano$ for x in full valid subsample grads; do cuda-memcheck nosetests theano/sandbox/cuda/tests/test_conv_cuda_ndarray.py:test_gemm_$x; done | |
| ========= CUDA-MEMCHECK | |
| Using gpu device 0: GeForce GTX 780 Ti | |
| ========= Invalid __global__ read of size 4 | |
| ========= at 0x000000e0 in sgemm_sm_heavy_nt_ldg | |
| ========= by thread (255,0,0) in block (0,0,0) | |
| ========= Address 0xb0484807c is out of bounds | |
| ========= Saved host backtrace up to driver entry point at kernel launch time | |
| ========= Host Frame:/usr/lib/libcuda.so (cuLaunchKernel + 0x2c5) [0x14ad25] | |
| ========= Host Frame:/usr/local/cuda-6.0/lib64/libcublas.so.6.0 [0x210808] | |
| ========= Host Frame:/usr/local/cuda-6.0/lib64/libcublas.so.6.0 [0x231143] | |
| ========= Host Frame:/usr/local/cuda-6.0/lib64/libcublas.so.6.0 [0x1ea762] | |
| ========= Host Frame:/usr/local/cuda-6.0/lib64/libcublas.so.6.0 [0x1de029] | |
| ========= Host Frame:/usr/local/cuda-6.0/lib64/libcublas.so.6.0 [0x1decce] | |
| ========= Host Frame:/usr/local/cuda-6.0/lib64/libcublas.so.6.0 [0x1df1e3] | |
| ========= Host Frame:/usr/local/cuda-6.0/lib64/libcublas.so.6.0 [0xebf24] | |
| ========= Host Frame:/home/sander/.theano/compiledir_Linux-3.13.0-30-generic-x86_64-with-Ubuntu-14.04-trusty-x86_64-2.7.6-64/tmpWl5d5K/fa8118cce2bd717ca59c4bf8d7e8bfde.so (_Z6corrMMP11CudaNdarrayS0_S0_iiiii + 0x71f) [0x2cff] | |
| ========= Host Frame:/home/sander/.theano/compiledir_Linux-3.13.0-30-generic-x86_64-with-Ubuntu-14.04-trusty-x86_64-2.7.6-64/tmpWl5d5K/fa8118cce2bd717ca59c4bf8d7e8bfde.so (_ZN62_GLOBAL__N__38_tmpxft_000015ac_00000000_8_mod_cpp1_ii_9233ac7753__struct_compiled_op_fa8118cce2bd717ca59c4bf8d7e8bfde3runEv + 0x4ec) [0x344c] | |
| ========= Host Frame:/home/sander/.theano/compiledir_Linux-3.13.0-30-generic-x86_64-with-Ubuntu-14.04-trusty-x86_64-2.7.6-64/lazylinker_ext/lazylinker_ext.so [0x35cc] | |
| ========= Host Frame:/home/sander/.theano/compiledir_Linux-3.13.0-30-generic-x86_64-with-Ubuntu-14.04-trusty-x86_64-2.7.6-64/lazylinker_ext/lazylinker_ext.so [0x4009] | |
| ========= Host Frame:/home/sander/.theano/compiledir_Linux-3.13.0-30-generic-x86_64-with-Ubuntu-14.04-trusty-x86_64-2.7.6-64/lazylinker_ext/lazylinker_ext.so [0x4698] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x970) [0x12cc20] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python [0xd9854] | |
| ========= Host Frame:/usr/bin/python [0xd51c2] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x970) [0x12cc20] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalCodeEx + 0x2a4) [0x15c594] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x7dd) [0x12ca8d] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalCodeEx + 0x2a4) [0x15c594] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x7dd) [0x12ca8d] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalCodeEx + 0x2a4) [0x15c594] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x7dd) [0x12ca8d] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x1f36) [0x12e1e6] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0xc82) [0x12cf32] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x1f36) [0x12e1e6] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python [0xd9854] | |
| ========= Host Frame:/usr/bin/python [0xd51c2] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x970) [0x12cc20] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0xc82) [0x12cf32] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x1f36) [0x12e1e6] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python [0xd9854] | |
| ========= Host Frame:/usr/bin/python [0xd51c2] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x970) [0x12cc20] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x1f36) [0x12e1e6] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python [0xd9854] | |
| ========= Host Frame:/usr/bin/python [0xd51c2] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x970) [0x12cc20] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x1f36) [0x12e1e6] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python [0xd9854] | |
| ========= Host Frame:/usr/bin/python [0xd51c2] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x970) [0x12cc20] | |
| ========= | |
| ========= Invalid __global__ read of size 4 | |
| ========= at 0x000000e0 in sgemm_sm_heavy_nt_ldg | |
| ========= by thread (254,0,0) in block (0,0,0) | |
| ========= Address 0xb04848078 is out of bounds | |
| ========= Saved host backtrace up to driver entry point at kernel launch time | |
| ========= Host Frame:/usr/lib/libcuda.so (cuLaunchKernel + 0x2c5) [0x14ad25] | |
| ========= Host Frame:/usr/local/cuda-6.0/lib64/libcublas.so.6.0 [0x210808] | |
| ========= Host Frame:/usr/local/cuda-6.0/lib64/libcublas.so.6.0 [0x231143] | |
| ========= Host Frame:/usr/local/cuda-6.0/lib64/libcublas.so.6.0 [0x1ea762] | |
| ========= Host Frame:/usr/local/cuda-6.0/lib64/libcublas.so.6.0 [0x1de029] | |
| ========= Host Frame:/usr/local/cuda-6.0/lib64/libcublas.so.6.0 [0x1decce] | |
| ========= Host Frame:/usr/local/cuda-6.0/lib64/libcublas.so.6.0 [0x1df1e3] | |
| ========= Host Frame:/usr/local/cuda-6.0/lib64/libcublas.so.6.0 [0xebf24] | |
| ========= Host Frame:/home/sander/.theano/compiledir_Linux-3.13.0-30-generic-x86_64-with-Ubuntu-14.04-trusty-x86_64-2.7.6-64/tmpWl5d5K/fa8118cce2bd717ca59c4bf8d7e8bfde.so (_Z6corrMMP11CudaNdarrayS0_S0_iiiii + 0x71f) [0x2cff] | |
| ========= Host Frame:/home/sander/.theano/compiledir_Linux-3.13.0-30-generic-x86_64-with-Ubuntu-14.04-trusty-x86_64-2.7.6-64/tmpWl5d5K/fa8118cce2bd717ca59c4bf8d7e8bfde.so (_ZN62_GLOBAL__N__38_tmpxft_000015ac_00000000_8_mod_cpp1_ii_9233ac7753__struct_compiled_op_fa8118cce2bd717ca59c4bf8d7e8bfde3runEv + 0x4ec) [0x344c] | |
| ========= Host Frame:/home/sander/.theano/compiledir_Linux-3.13.0-30-generic-x86_64-with-Ubuntu-14.04-trusty-x86_64-2.7.6-64/lazylinker_ext/lazylinker_ext.so [0x35cc] | |
| ========= Host Frame:/home/sander/.theano/compiledir_Linux-3.13.0-30-generic-x86_64-with-Ubuntu-14.04-trusty-x86_64-2.7.6-64/lazylinker_ext/lazylinker_ext.so [0x4009] | |
| ========= Host Frame:/home/sander/.theano/compiledir_Linux-3.13.0-30-generic-x86_64-with-Ubuntu-14.04-trusty-x86_64-2.7.6-64/lazylinker_ext/lazylinker_ext.so [0x4698] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x970) [0x12cc20] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python [0xd9854] | |
| ========= Host Frame:/usr/bin/python [0xd51c2] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x970) [0x12cc20] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalCodeEx + 0x2a4) [0x15c594] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x7dd) [0x12ca8d] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalCodeEx + 0x2a4) [0x15c594] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x7dd) [0x12ca8d] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalCodeEx + 0x2a4) [0x15c594] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x7dd) [0x12ca8d] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x1f36) [0x12e1e6] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0xc82) [0x12cf32] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x1f36) [0x12e1e6] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python [0xd9854] | |
| ========= Host Frame:/usr/bin/python [0xd51c2] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x970) [0x12cc20] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0xc82) [0x12cf32] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x1f36) [0x12e1e6] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python [0xd9854] | |
| ========= Host Frame:/usr/bin/python [0xd51c2] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x970) [0x12cc20] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x1f36) [0x12e1e6] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python [0xd9854] | |
| ========= Host Frame:/usr/bin/python [0xd51c2] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x970) [0x12cc20] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x1f36) [0x12e1e6] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python [0xd9854] | |
| ========= Host Frame:/usr/bin/python [0xd51c2] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x970) [0x12cc20] | |
| ========= | |
| ========= Invalid __global__ read of size 4 | |
| ========= at 0x000000e0 in sgemm_sm_heavy_nt_ldg | |
| ========= by thread (253,0,0) in block (0,0,0) | |
| ========= Address 0xb04848074 is out of bounds | |
| ========= Saved host backtrace up to driver entry point at kernel launch time | |
| ========= Host Frame:/usr/lib/libcuda.so (cuLaunchKernel + 0x2c5) [0x14ad25] | |
| ========= Host Frame:/usr/local/cuda-6.0/lib64/libcublas.so.6.0 [0x210808] | |
| ========= Host Frame:/usr/local/cuda-6.0/lib64/libcublas.so.6.0 [0x231143] | |
| ========= Host Frame:/usr/local/cuda-6.0/lib64/libcublas.so.6.0 [0x1ea762] | |
| ========= Host Frame:/usr/local/cuda-6.0/lib64/libcublas.so.6.0 [0x1de029] | |
| ========= Host Frame:/usr/local/cuda-6.0/lib64/libcublas.so.6.0 [0x1decce] | |
| ========= Host Frame:/usr/local/cuda-6.0/lib64/libcublas.so.6.0 [0x1df1e3] | |
| ========= Host Frame:/usr/local/cuda-6.0/lib64/libcublas.so.6.0 [0xebf24] | |
| ========= Host Frame:/home/sander/.theano/compiledir_Linux-3.13.0-30-generic-x86_64-with-Ubuntu-14.04-trusty-x86_64-2.7.6-64/tmpWl5d5K/fa8118cce2bd717ca59c4bf8d7e8bfde.so (_Z6corrMMP11CudaNdarrayS0_S0_iiiii + 0x71f) [0x2cff] | |
| ========= Host Frame:/home/sander/.theano/compiledir_Linux-3.13.0-30-generic-x86_64-with-Ubuntu-14.04-trusty-x86_64-2.7.6-64/tmpWl5d5K/fa8118cce2bd717ca59c4bf8d7e8bfde.so (_ZN62_GLOBAL__N__38_tmpxft_000015ac_00000000_8_mod_cpp1_ii_9233ac7753__struct_compiled_op_fa8118cce2bd717ca59c4bf8d7e8bfde3runEv + 0x4ec) [0x344c] | |
| ========= Host Frame:/home/sander/.theano/compiledir_Linux-3.13.0-30-generic-x86_64-with-Ubuntu-14.04-trusty-x86_64-2.7.6-64/lazylinker_ext/lazylinker_ext.so [0x35cc] | |
| ========= Host Frame:/home/sander/.theano/compiledir_Linux-3.13.0-30-generic-x86_64-with-Ubuntu-14.04-trusty-x86_64-2.7.6-64/lazylinker_ext/lazylinker_ext.so [0x4009] | |
| ========= Host Frame:/home/sander/.theano/compiledir_Linux-3.13.0-30-generic-x86_64-with-Ubuntu-14.04-trusty-x86_64-2.7.6-64/lazylinker_ext/lazylinker_ext.so [0x4698] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x970) [0x12cc20] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python [0xd9854] | |
| ========= Host Frame:/usr/bin/python [0xd51c2] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x970) [0x12cc20] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalCodeEx + 0x2a4) [0x15c594] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x7dd) [0x12ca8d] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalCodeEx + 0x2a4) [0x15c594] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x7dd) [0x12ca8d] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalCodeEx + 0x2a4) [0x15c594] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x7dd) [0x12ca8d] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x1f36) [0x12e1e6] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0xc82) [0x12cf32] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x1f36) [0x12e1e6] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python [0xd9854] | |
| ========= Host Frame:/usr/bin/python [0xd51c2] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x970) [0x12cc20] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0xc82) [0x12cf32] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x1f36) [0x12e1e6] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python [0xd9854] | |
| ========= Host Frame:/usr/bin/python [0xd51c2] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x970) [0x12cc20] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x1f36) [0x12e1e6] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python [0xd9854] | |
| ========= Host Frame:/usr/bin/python [0xd51c2] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x970) [0x12cc20] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x1f36) [0x12e1e6] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python [0xd9854] | |
| ========= Host Frame:/usr/bin/python [0xd51c2] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x970) [0x12cc20] | |
| ========= | |
| ========= Invalid __global__ read of size 4 | |
| ========= at 0x000000e0 in sgemm_sm_heavy_nt_ldg | |
| ========= by thread (252,0,0) in block (0,0,0) | |
| ========= Address 0xb04848070 is out of bounds | |
| ========= Saved host backtrace up to driver entry point at kernel launch time | |
| ========= Host Frame:/usr/lib/libcuda.so (cuLaunchKernel + 0x2c5) [0x14ad25] | |
| ========= Host Frame:/usr/local/cuda-6.0/lib64/libcublas.so.6.0 [0x210808] | |
| ========= Host Frame:/usr/local/cuda-6.0/lib64/libcublas.so.6.0 [0x231143] | |
| ========= Host Frame:/usr/local/cuda-6.0/lib64/libcublas.so.6.0 [0x1ea762] | |
| ========= Host Frame:/usr/local/cuda-6.0/lib64/libcublas.so.6.0 [0x1de029] | |
| ========= Host Frame:/usr/local/cuda-6.0/lib64/libcublas.so.6.0 [0x1decce] | |
| ========= Host Frame:/usr/local/cuda-6.0/lib64/libcublas.so.6.0 [0x1df1e3] | |
| ========= Host Frame:/usr/local/cuda-6.0/lib64/libcublas.so.6.0 [0xebf24] | |
| ========= Host Frame:/home/sander/.theano/compiledir_Linux-3.13.0-30-generic-x86_64-with-Ubuntu-14.04-trusty-x86_64-2.7.6-64/tmpWl5d5K/fa8118cce2bd717ca59c4bf8d7e8bfde.so (_Z6corrMMP11CudaNdarrayS0_S0_iiiii + 0x71f) [0x2cff] | |
| ========= Host Frame:/home/sander/.theano/compiledir_Linux-3.13.0-30-generic-x86_64-with-Ubuntu-14.04-trusty-x86_64-2.7.6-64/tmpWl5d5K/fa8118cce2bd717ca59c4bf8d7e8bfde.so (_ZN62_GLOBAL__N__38_tmpxft_000015ac_00000000_8_mod_cpp1_ii_9233ac7753__struct_compiled_op_fa8118cce2bd717ca59c4bf8d7e8bfde3runEv + 0x4ec) [0x344c] | |
| ========= Host Frame:/home/sander/.theano/compiledir_Linux-3.13.0-30-generic-x86_64-with-Ubuntu-14.04-trusty-x86_64-2.7.6-64/lazylinker_ext/lazylinker_ext.so [0x35cc] | |
| ========= Host Frame:/home/sander/.theano/compiledir_Linux-3.13.0-30-generic-x86_64-with-Ubuntu-14.04-trusty-x86_64-2.7.6-64/lazylinker_ext/lazylinker_ext.so [0x4009] | |
| ========= Host Frame:/home/sander/.theano/compiledir_Linux-3.13.0-30-generic-x86_64-with-Ubuntu-14.04-trusty-x86_64-2.7.6-64/lazylinker_ext/lazylinker_ext.so [0x4698] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x970) [0x12cc20] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python [0xd9854] | |
| ========= Host Frame:/usr/bin/python [0xd51c2] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x970) [0x12cc20] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalCodeEx + 0x2a4) [0x15c594] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x7dd) [0x12ca8d] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalCodeEx + 0x2a4) [0x15c594] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x7dd) [0x12ca8d] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalCodeEx + 0x2a4) [0x15c594] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x7dd) [0x12ca8d] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x1f36) [0x12e1e6] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0xc82) [0x12cf32] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x1f36) [0x12e1e6] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python [0xd9854] | |
| ========= Host Frame:/usr/bin/python [0xd51c2] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x970) [0x12cc20] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0xc82) [0x12cf32] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x1f36) [0x12e1e6] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python [0xd9854] | |
| ========= Host Frame:/usr/bin/python [0xd51c2] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x970) [0x12cc20] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x1f36) [0x12e1e6] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python [0xd9854] | |
| ========= Host Frame:/usr/bin/python [0xd51c2] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x970) [0x12cc20] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x1f36) [0x12e1e6] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python [0xd9854] | |
| ========= Host Frame:/usr/bin/python [0xd51c2] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x970) [0x12cc20] | |
| ========= | |
| ========= Invalid __global__ read of size 4 | |
| ========= at 0x000000e0 in sgemm_sm_heavy_nt_ldg | |
| ========= by thread (251,0,0) in block (0,0,0) | |
| ========= Address 0xb0484806c is out of bounds | |
| ========= Saved host backtrace up to driver entry point at kernel launch time | |
| ========= Host Frame:/usr/lib/libcuda.so (cuLaunchKernel + 0x2c5) [0x14ad25] | |
| ========= Host Frame:/usr/local/cuda-6.0/lib64/libcublas.so.6.0 [0x210808] | |
| ========= Host Frame:/usr/local/cuda-6.0/lib64/libcublas.so.6.0 [0x231143] | |
| ========= Host Frame:/usr/local/cuda-6.0/lib64/libcublas.so.6.0 [0x1ea762] | |
| ========= Host Frame:/usr/local/cuda-6.0/lib64/libcublas.so.6.0 [0x1de029] | |
| ========= Host Frame:/usr/local/cuda-6.0/lib64/libcublas.so.6.0 [0x1decce] | |
| ========= Host Frame:/usr/local/cuda-6.0/lib64/libcublas.so.6.0 [0x1df1e3] | |
| ========= Host Frame:/usr/local/cuda-6.0/lib64/libcublas.so.6.0 [0xebf24] | |
| ========= Host Frame:/home/sander/.theano/compiledir_Linux-3.13.0-30-generic-x86_64-with-Ubuntu-14.04-trusty-x86_64-2.7.6-64/tmpWl5d5K/fa8118cce2bd717ca59c4bf8d7e8bfde.so (_Z6corrMMP11CudaNdarrayS0_S0_iiiii + 0x71f) [0x2cff] | |
| ========= Host Frame:/home/sander/.theano/compiledir_Linux-3.13.0-30-generic-x86_64-with-Ubuntu-14.04-trusty-x86_64-2.7.6-64/tmpWl5d5K/fa8118cce2bd717ca59c4bf8d7e8bfde.so (_ZN62_GLOBAL__N__38_tmpxft_000015ac_00000000_8_mod_cpp1_ii_9233ac7753__struct_compiled_op_fa8118cce2bd717ca59c4bf8d7e8bfde3runEv + 0x4ec) [0x344c] | |
| ========= Host Frame:/home/sander/.theano/compiledir_Linux-3.13.0-30-generic-x86_64-with-Ubuntu-14.04-trusty-x86_64-2.7.6-64/lazylinker_ext/lazylinker_ext.so [0x35cc] | |
| ========= Host Frame:/home/sander/.theano/compiledir_Linux-3.13.0-30-generic-x86_64-with-Ubuntu-14.04-trusty-x86_64-2.7.6-64/lazylinker_ext/lazylinker_ext.so [0x4009] | |
| ========= Host Frame:/home/sander/.theano/compiledir_Linux-3.13.0-30-generic-x86_64-with-Ubuntu-14.04-trusty-x86_64-2.7.6-64/lazylinker_ext/lazylinker_ext.so [0x4698] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x970) [0x12cc20] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python [0xd9854] | |
| ========= Host Frame:/usr/bin/python [0xd51c2] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x970) [0x12cc20] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalCodeEx + 0x2a4) [0x15c594] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x7dd) [0x12ca8d] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalCodeEx + 0x2a4) [0x15c594] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x7dd) [0x12ca8d] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalCodeEx + 0x2a4) [0x15c594] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x7dd) [0x12ca8d] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x1f36) [0x12e1e6] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0xc82) [0x12cf32] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x1f36) [0x12e1e6] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python [0xd9854] | |
| ========= Host Frame:/usr/bin/python [0xd51c2] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x970) [0x12cc20] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0xc82) [0x12cf32] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x1f36) [0x12e1e6] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python [0xd9854] | |
| ========= Host Frame:/usr/bin/python [0xd51c2] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x970) [0x12cc20] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x1f36) [0x12e1e6] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python [0xd9854] | |
| ========= Host Frame:/usr/bin/python [0xd51c2] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x970) [0x12cc20] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x1f36) [0x12e1e6] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python [0xd9854] | |
| ========= Host Frame:/usr/bin/python [0xd51c2] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x970) [0x12cc20] | |
| ========= | |
| ========= Invalid __global__ read of size 4 | |
| ========= at 0x000000e0 in sgemm_sm_heavy_nt_ldg | |
| ========= by thread (250,0,0) in block (0,0,0) | |
| ========= Address 0xb04848068 is out of bounds | |
| ========= Saved host backtrace up to driver entry point at kernel launch time | |
| ========= Host Frame:/usr/lib/libcuda.so (cuLaunchKernel + 0x2c5) [0x14ad25] | |
| ========= Host Frame:/usr/local/cuda-6.0/lib64/libcublas.so.6.0 [0x210808] | |
| ========= Host Frame:/usr/local/cuda-6.0/lib64/libcublas.so.6.0 [0x231143] | |
| ========= Host Frame:/usr/local/cuda-6.0/lib64/libcublas.so.6.0 [0x1ea762] | |
| ========= Host Frame:/usr/local/cuda-6.0/lib64/libcublas.so.6.0 [0x1de029] | |
| ========= Host Frame:/usr/local/cuda-6.0/lib64/libcublas.so.6.0 [0x1decce] | |
| ========= Host Frame:/usr/local/cuda-6.0/lib64/libcublas.so.6.0 [0x1df1e3] | |
| ========= Host Frame:/usr/local/cuda-6.0/lib64/libcublas.so.6.0 [0xebf24] | |
| ========= Host Frame:/home/sander/.theano/compiledir_Linux-3.13.0-30-generic-x86_64-with-Ubuntu-14.04-trusty-x86_64-2.7.6-64/tmpWl5d5K/fa8118cce2bd717ca59c4bf8d7e8bfde.so (_Z6corrMMP11CudaNdarrayS0_S0_iiiii + 0x71f) [0x2cff] | |
| ========= Host Frame:/home/sander/.theano/compiledir_Linux-3.13.0-30-generic-x86_64-with-Ubuntu-14.04-trusty-x86_64-2.7.6-64/tmpWl5d5K/fa8118cce2bd717ca59c4bf8d7e8bfde.so (_ZN62_GLOBAL__N__38_tmpxft_000015ac_00000000_8_mod_cpp1_ii_9233ac7753__struct_compiled_op_fa8118cce2bd717ca59c4bf8d7e8bfde3runEv + 0x4ec) [0x344c] | |
| ========= Host Frame:/home/sander/.theano/compiledir_Linux-3.13.0-30-generic-x86_64-with-Ubuntu-14.04-trusty-x86_64-2.7.6-64/lazylinker_ext/lazylinker_ext.so [0x35cc] | |
| ========= Host Frame:/home/sander/.theano/compiledir_Linux-3.13.0-30-generic-x86_64-with-Ubuntu-14.04-trusty-x86_64-2.7.6-64/lazylinker_ext/lazylinker_ext.so [0x4009] | |
| ========= Host Frame:/home/sander/.theano/compiledir_Linux-3.13.0-30-generic-x86_64-with-Ubuntu-14.04-trusty-x86_64-2.7.6-64/lazylinker_ext/lazylinker_ext.so [0x4698] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x970) [0x12cc20] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python [0xd9854] | |
| ========= Host Frame:/usr/bin/python [0xd51c2] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x970) [0x12cc20] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalCodeEx + 0x2a4) [0x15c594] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x7dd) [0x12ca8d] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalCodeEx + 0x2a4) [0x15c594] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x7dd) [0x12ca8d] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalCodeEx + 0x2a4) [0x15c594] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x7dd) [0x12ca8d] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x1f36) [0x12e1e6] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0xc82) [0x12cf32] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x1f36) [0x12e1e6] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python [0xd9854] | |
| ========= Host Frame:/usr/bin/python [0xd51c2] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x970) [0x12cc20] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0xc82) [0x12cf32] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x1f36) [0x12e1e6] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python [0xd9854] | |
| ========= Host Frame:/usr/bin/python [0xd51c2] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x970) [0x12cc20] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x1f36) [0x12e1e6] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python [0xd9854] | |
| ========= Host Frame:/usr/bin/python [0xd51c2] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x970) [0x12cc20] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x1f36) [0x12e1e6] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python [0xd9854] | |
| ========= Host Frame:/usr/bin/python [0xd51c2] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x970) [0x12cc20] | |
| ========= | |
| ========= Invalid __global__ read of size 4 | |
| ========= at 0x000000e0 in sgemm_sm_heavy_nt_ldg | |
| ========= by thread (249,0,0) in block (0,0,0) | |
| ========= Address 0xb04848064 is out of bounds | |
| ========= Saved host backtrace up to driver entry point at kernel launch time | |
| ========= Host Frame:/usr/lib/libcuda.so (cuLaunchKernel + 0x2c5) [0x14ad25] | |
| ========= Host Frame:/usr/local/cuda-6.0/lib64/libcublas.so.6.0 [0x210808] | |
| ========= Host Frame:/usr/local/cuda-6.0/lib64/libcublas.so.6.0 [0x231143] | |
| ========= Host Frame:/usr/local/cuda-6.0/lib64/libcublas.so.6.0 [0x1ea762] | |
| ========= Host Frame:/usr/local/cuda-6.0/lib64/libcublas.so.6.0 [0x1de029] | |
| ========= Host Frame:/usr/local/cuda-6.0/lib64/libcublas.so.6.0 [0x1decce] | |
| ========= Host Frame:/usr/local/cuda-6.0/lib64/libcublas.so.6.0 [0x1df1e3] | |
| ========= Host Frame:/usr/local/cuda-6.0/lib64/libcublas.so.6.0 [0xebf24] | |
| ========= Host Frame:/home/sander/.theano/compiledir_Linux-3.13.0-30-generic-x86_64-with-Ubuntu-14.04-trusty-x86_64-2.7.6-64/tmpWl5d5K/fa8118cce2bd717ca59c4bf8d7e8bfde.so (_Z6corrMMP11CudaNdarrayS0_S0_iiiii + 0x71f) [0x2cff] | |
| ========= Host Frame:/home/sander/.theano/compiledir_Linux-3.13.0-30-generic-x86_64-with-Ubuntu-14.04-trusty-x86_64-2.7.6-64/tmpWl5d5K/fa8118cce2bd717ca59c4bf8d7e8bfde.so (_ZN62_GLOBAL__N__38_tmpxft_000015ac_00000000_8_mod_cpp1_ii_9233ac7753__struct_compiled_op_fa8118cce2bd717ca59c4bf8d7e8bfde3runEv + 0x4ec) [0x344c] | |
| ========= Host Frame:/home/sander/.theano/compiledir_Linux-3.13.0-30-generic-x86_64-with-Ubuntu-14.04-trusty-x86_64-2.7.6-64/lazylinker_ext/lazylinker_ext.so [0x35cc] | |
| ========= Host Frame:/home/sander/.theano/compiledir_Linux-3.13.0-30-generic-x86_64-with-Ubuntu-14.04-trusty-x86_64-2.7.6-64/lazylinker_ext/lazylinker_ext.so [0x4009] | |
| ========= Host Frame:/home/sander/.theano/compiledir_Linux-3.13.0-30-generic-x86_64-with-Ubuntu-14.04-trusty-x86_64-2.7.6-64/lazylinker_ext/lazylinker_ext.so [0x4698] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x970) [0x12cc20] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python [0xd9854] | |
| ========= Host Frame:/usr/bin/python [0xd51c2] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x970) [0x12cc20] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalCodeEx + 0x2a4) [0x15c594] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x7dd) [0x12ca8d] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalCodeEx + 0x2a4) [0x15c594] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x7dd) [0x12ca8d] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalCodeEx + 0x2a4) [0x15c594] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x7dd) [0x12ca8d] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x1f36) [0x12e1e6] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0xc82) [0x12cf32] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x1f36) [0x12e1e6] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python [0xd9854] | |
| ========= Host Frame:/usr/bin/python [0xd51c2] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x970) [0x12cc20] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0xc82) [0x12cf32] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x1f36) [0x12e1e6] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python [0xd9854] | |
| ========= Host Frame:/usr/bin/python [0xd51c2] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x970) [0x12cc20] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x1f36) [0x12e1e6] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python [0xd9854] | |
| ========= Host Frame:/usr/bin/python [0xd51c2] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x970) [0x12cc20] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x1f36) [0x12e1e6] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python [0xd9854] | |
| ========= Host Frame:/usr/bin/python [0xd51c2] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x970) [0x12cc20] | |
| ========= | |
| ========= Invalid __global__ read of size 4 | |
| ========= at 0x000000e0 in sgemm_sm_heavy_nt_ldg | |
| ========= by thread (248,0,0) in block (0,0,0) | |
| ========= Address 0xb04848060 is out of bounds | |
| ========= Saved host backtrace up to driver entry point at kernel launch time | |
| ========= Host Frame:/usr/lib/libcuda.so (cuLaunchKernel + 0x2c5) [0x14ad25] | |
| ========= Host Frame:/usr/local/cuda-6.0/lib64/libcublas.so.6.0 [0x210808] | |
| ========= Host Frame:/usr/local/cuda-6.0/lib64/libcublas.so.6.0 [0x231143] | |
| ========= Host Frame:/usr/local/cuda-6.0/lib64/libcublas.so.6.0 [0x1ea762] | |
| ========= Host Frame:/usr/local/cuda-6.0/lib64/libcublas.so.6.0 [0x1de029] | |
| ========= Host Frame:/usr/local/cuda-6.0/lib64/libcublas.so.6.0 [0x1decce] | |
| ========= Host Frame:/usr/local/cuda-6.0/lib64/libcublas.so.6.0 [0x1df1e3] | |
| ========= Host Frame:/usr/local/cuda-6.0/lib64/libcublas.so.6.0 [0xebf24] | |
| ========= Host Frame:/home/sander/.theano/compiledir_Linux-3.13.0-30-generic-x86_64-with-Ubuntu-14.04-trusty-x86_64-2.7.6-64/tmpWl5d5K/fa8118cce2bd717ca59c4bf8d7e8bfde.so (_Z6corrMMP11CudaNdarrayS0_S0_iiiii + 0x71f) [0x2cff] | |
| ========= Host Frame:/home/sander/.theano/compiledir_Linux-3.13.0-30-generic-x86_64-with-Ubuntu-14.04-trusty-x86_64-2.7.6-64/tmpWl5d5K/fa8118cce2bd717ca59c4bf8d7e8bfde.so (_ZN62_GLOBAL__N__38_tmpxft_000015ac_00000000_8_mod_cpp1_ii_9233ac7753__struct_compiled_op_fa8118cce2bd717ca59c4bf8d7e8bfde3runEv + 0x4ec) [0x344c] | |
| ========= Host Frame:/home/sander/.theano/compiledir_Linux-3.13.0-30-generic-x86_64-with-Ubuntu-14.04-trusty-x86_64-2.7.6-64/lazylinker_ext/lazylinker_ext.so [0x35cc] | |
| ========= Host Frame:/home/sander/.theano/compiledir_Linux-3.13.0-30-generic-x86_64-with-Ubuntu-14.04-trusty-x86_64-2.7.6-64/lazylinker_ext/lazylinker_ext.so [0x4009] | |
| ========= Host Frame:/home/sander/.theano/compiledir_Linux-3.13.0-30-generic-x86_64-with-Ubuntu-14.04-trusty-x86_64-2.7.6-64/lazylinker_ext/lazylinker_ext.so [0x4698] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x970) [0x12cc20] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python [0xd9854] | |
| ========= Host Frame:/usr/bin/python [0xd51c2] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x970) [0x12cc20] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalCodeEx + 0x2a4) [0x15c594] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x7dd) [0x12ca8d] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalCodeEx + 0x2a4) [0x15c594] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x7dd) [0x12ca8d] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalCodeEx + 0x2a4) [0x15c594] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x7dd) [0x12ca8d] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x1f36) [0x12e1e6] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0xc82) [0x12cf32] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x1f36) [0x12e1e6] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python [0xd9854] | |
| ========= Host Frame:/usr/bin/python [0xd51c2] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x970) [0x12cc20] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0xc82) [0x12cf32] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x1f36) [0x12e1e6] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python [0xd9854] | |
| ========= Host Frame:/usr/bin/python [0xd51c2] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x970) [0x12cc20] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x1f36) [0x12e1e6] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python [0xd9854] | |
| ========= Host Frame:/usr/bin/python [0xd51c2] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x970) [0x12cc20] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x1f36) [0x12e1e6] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python [0xd9854] | |
| ========= Host Frame:/usr/bin/python [0xd51c2] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x970) [0x12cc20] | |
| ========= | |
| ========= Invalid __global__ read of size 4 | |
| ========= at 0x000000e0 in sgemm_sm_heavy_nt_ldg | |
| ========= by thread (247,0,0) in block (0,0,0) | |
| ========= Address 0xb0484805c is out of bounds | |
| ========= Saved host backtrace up to driver entry point at kernel launch time | |
| ========= Host Frame:/usr/lib/libcuda.so (cuLaunchKernel + 0x2c5) [0x14ad25] | |
| ========= Host Frame:/usr/local/cuda-6.0/lib64/libcublas.so.6.0 [0x210808] | |
| ========= Host Frame:/usr/local/cuda-6.0/lib64/libcublas.so.6.0 [0x231143] | |
| ========= Host Frame:/usr/local/cuda-6.0/lib64/libcublas.so.6.0 [0x1ea762] | |
| ========= Host Frame:/usr/local/cuda-6.0/lib64/libcublas.so.6.0 [0x1de029] | |
| ========= Host Frame:/usr/local/cuda-6.0/lib64/libcublas.so.6.0 [0x1decce] | |
| ========= Host Frame:/usr/local/cuda-6.0/lib64/libcublas.so.6.0 [0x1df1e3] | |
| ========= Host Frame:/usr/local/cuda-6.0/lib64/libcublas.so.6.0 [0xebf24] | |
| ========= Host Frame:/home/sander/.theano/compiledir_Linux-3.13.0-30-generic-x86_64-with-Ubuntu-14.04-trusty-x86_64-2.7.6-64/tmpWl5d5K/fa8118cce2bd717ca59c4bf8d7e8bfde.so (_Z6corrMMP11CudaNdarrayS0_S0_iiiii + 0x71f) [0x2cff] | |
| ========= Host Frame:/home/sander/.theano/compiledir_Linux-3.13.0-30-generic-x86_64-with-Ubuntu-14.04-trusty-x86_64-2.7.6-64/tmpWl5d5K/fa8118cce2bd717ca59c4bf8d7e8bfde.so (_ZN62_GLOBAL__N__38_tmpxft_000015ac_00000000_8_mod_cpp1_ii_9233ac7753__struct_compiled_op_fa8118cce2bd717ca59c4bf8d7e8bfde3runEv + 0x4ec) [0x344c] | |
| ========= Host Frame:/home/sander/.theano/compiledir_Linux-3.13.0-30-generic-x86_64-with-Ubuntu-14.04-trusty-x86_64-2.7.6-64/lazylinker_ext/lazylinker_ext.so [0x35cc] | |
| ========= Host Frame:/home/sander/.theano/compiledir_Linux-3.13.0-30-generic-x86_64-with-Ubuntu-14.04-trusty-x86_64-2.7.6-64/lazylinker_ext/lazylinker_ext.so [0x4009] | |
| ========= Host Frame:/home/sander/.theano/compiledir_Linux-3.13.0-30-generic-x86_64-with-Ubuntu-14.04-trusty-x86_64-2.7.6-64/lazylinker_ext/lazylinker_ext.so [0x4698] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x970) [0x12cc20] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python [0xd9854] | |
| ========= Host Frame:/usr/bin/python [0xd51c2] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x970) [0x12cc20] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalCodeEx + 0x2a4) [0x15c594] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x7dd) [0x12ca8d] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalCodeEx + 0x2a4) [0x15c594] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x7dd) [0x12ca8d] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalCodeEx + 0x2a4) [0x15c594] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x7dd) [0x12ca8d] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x1f36) [0x12e1e6] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0xc82) [0x12cf32] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x1f36) [0x12e1e6] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python [0xd9854] | |
| ========= Host Frame:/usr/bin/python [0xd51c2] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x970) [0x12cc20] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0xc82) [0x12cf32] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x1f36) [0x12e1e6] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python [0xd9854] | |
| ========= Host Frame:/usr/bin/python [0xd51c2] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x970) [0x12cc20] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x1f36) [0x12e1e6] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python [0xd9854] | |
| ========= Host Frame:/usr/bin/python [0xd51c2] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x970) [0x12cc20] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x1f36) [0x12e1e6] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python [0xd9854] | |
| ========= Host Frame:/usr/bin/python [0xd51c2] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x970) [0x12cc20] | |
| ========= | |
| ========= Invalid __global__ read of size 4 | |
| ========= at 0x000000e0 in sgemm_sm_heavy_nt_ldg | |
| ========= by thread (246,0,0) in block (0,0,0) | |
| ========= Address 0xb04848058 is out of bounds | |
| ========= Saved host backtrace up to driver entry point at kernel launch time | |
| ========= Host Frame:/usr/lib/libcuda.so (cuLaunchKernel + 0x2c5) [0x14ad25] | |
| ========= Host Frame:/usr/local/cuda-6.0/lib64/libcublas.so.6.0 [0x210808] | |
| ========= Host Frame:/usr/local/cuda-6.0/lib64/libcublas.so.6.0 [0x231143] | |
| ========= Host Frame:/usr/local/cuda-6.0/lib64/libcublas.so.6.0 [0x1ea762] | |
| ========= Host Frame:/usr/local/cuda-6.0/lib64/libcublas.so.6.0 [0x1de029] | |
| ========= Host Frame:/usr/local/cuda-6.0/lib64/libcublas.so.6.0 [0x1decce] | |
| ========= Host Frame:/usr/local/cuda-6.0/lib64/libcublas.so.6.0 [0x1df1e3] | |
| ========= Host Frame:/usr/local/cuda-6.0/lib64/libcublas.so.6.0 [0xebf24] | |
| ========= Host Frame:/home/sander/.theano/compiledir_Linux-3.13.0-30-generic-x86_64-with-Ubuntu-14.04-trusty-x86_64-2.7.6-64/tmpWl5d5K/fa8118cce2bd717ca59c4bf8d7e8bfde.so (_Z6corrMMP11CudaNdarrayS0_S0_iiiii + 0x71f) [0x2cff] | |
| ========= Host Frame:/home/sander/.theano/compiledir_Linux-3.13.0-30-generic-x86_64-with-Ubuntu-14.04-trusty-x86_64-2.7.6-64/tmpWl5d5K/fa8118cce2bd717ca59c4bf8d7e8bfde.so (_ZN62_GLOBAL__N__38_tmpxft_000015ac_00000000_8_mod_cpp1_ii_9233ac7753__struct_compiled_op_fa8118cce2bd717ca59c4bf8d7e8bfde3runEv + 0x4ec) [0x344c] | |
| ========= Host Frame:/home/sander/.theano/compiledir_Linux-3.13.0-30-generic-x86_64-with-Ubuntu-14.04-trusty-x86_64-2.7.6-64/lazylinker_ext/lazylinker_ext.so [0x35cc] | |
| ========= Host Frame:/home/sander/.theano/compiledir_Linux-3.13.0-30-generic-x86_64-with-Ubuntu-14.04-trusty-x86_64-2.7.6-64/lazylinker_ext/lazylinker_ext.so [0x4009] | |
| ========= Host Frame:/home/sander/.theano/compiledir_Linux-3.13.0-30-generic-x86_64-with-Ubuntu-14.04-trusty-x86_64-2.7.6-64/lazylinker_ext/lazylinker_ext.so [0x4698] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x970) [0x12cc20] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python [0xd9854] | |
| ========= Host Frame:/usr/bin/python [0xd51c2] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x970) [0x12cc20] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalCodeEx + 0x2a4) [0x15c594] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x7dd) [0x12ca8d] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalCodeEx + 0x2a4) [0x15c594] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x7dd) [0x12ca8d] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalCodeEx + 0x2a4) [0x15c594] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x7dd) [0x12ca8d] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x1f36) [0x12e1e6] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0xc82) [0x12cf32] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x1f36) [0x12e1e6] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python [0xd9854] | |
| ========= Host Frame:/usr/bin/python [0xd51c2] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x970) [0x12cc20] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0xc82) [0x12cf32] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x1f36) [0x12e1e6] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python [0xd9854] | |
| ========= Host Frame:/usr/bin/python [0xd51c2] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x970) [0x12cc20] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x1f36) [0x12e1e6] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python [0xd9854] | |
| ========= Host Frame:/usr/bin/python [0xd51c2] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x970) [0x12cc20] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x1f36) [0x12e1e6] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python [0xd9854] | |
| ========= Host Frame:/usr/bin/python [0xd51c2] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x970) [0x12cc20] | |
| ========= | |
| ========= Invalid __global__ read of size 4 | |
| ========= at 0x000000e0 in sgemm_sm_heavy_nt_ldg | |
| ========= by thread (245,0,0) in block (0,0,0) | |
| ========= Address 0xb04848054 is out of bounds | |
| ========= Saved host backtrace up to driver entry point at kernel launch time | |
| ========= Host Frame:/usr/lib/libcuda.so (cuLaunchKernel + 0x2c5) [0x14ad25] | |
| ========= Host Frame:/usr/local/cuda-6.0/lib64/libcublas.so.6.0 [0x210808] | |
| ========= Host Frame:/usr/local/cuda-6.0/lib64/libcublas.so.6.0 [0x231143] | |
| ========= Host Frame:/usr/local/cuda-6.0/lib64/libcublas.so.6.0 [0x1ea762] | |
| ========= Host Frame:/usr/local/cuda-6.0/lib64/libcublas.so.6.0 [0x1de029] | |
| ========= Host Frame:/usr/local/cuda-6.0/lib64/libcublas.so.6.0 [0x1decce] | |
| ========= Host Frame:/usr/local/cuda-6.0/lib64/libcublas.so.6.0 [0x1df1e3] | |
| ========= Host Frame:/usr/local/cuda-6.0/lib64/libcublas.so.6.0 [0xebf24] | |
| ========= Host Frame:/home/sander/.theano/compiledir_Linux-3.13.0-30-generic-x86_64-with-Ubuntu-14.04-trusty-x86_64-2.7.6-64/tmpWl5d5K/fa8118cce2bd717ca59c4bf8d7e8bfde.so (_Z6corrMMP11CudaNdarrayS0_S0_iiiii + 0x71f) [0x2cff] | |
| ========= Host Frame:/home/sander/.theano/compiledir_Linux-3.13.0-30-generic-x86_64-with-Ubuntu-14.04-trusty-x86_64-2.7.6-64/tmpWl5d5K/fa8118cce2bd717ca59c4bf8d7e8bfde.so (_ZN62_GLOBAL__N__38_tmpxft_000015ac_00000000_8_mod_cpp1_ii_9233ac7753__struct_compiled_op_fa8118cce2bd717ca59c4bf8d7e8bfde3runEv + 0x4ec) [0x344c] | |
| ========= Host Frame:/home/sander/.theano/compiledir_Linux-3.13.0-30-generic-x86_64-with-Ubuntu-14.04-trusty-x86_64-2.7.6-64/lazylinker_ext/lazylinker_ext.so [0x35cc] | |
| ========= Host Frame:/home/sander/.theano/compiledir_Linux-3.13.0-30-generic-x86_64-with-Ubuntu-14.04-trusty-x86_64-2.7.6-64/lazylinker_ext/lazylinker_ext.so [0x4009] | |
| ========= Host Frame:/home/sander/.theano/compiledir_Linux-3.13.0-30-generic-x86_64-with-Ubuntu-14.04-trusty-x86_64-2.7.6-64/lazylinker_ext/lazylinker_ext.so [0x4698] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x970) [0x12cc20] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python [0xd9854] | |
| ========= Host Frame:/usr/bin/python [0xd51c2] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x970) [0x12cc20] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalCodeEx + 0x2a4) [0x15c594] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x7dd) [0x12ca8d] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalCodeEx + 0x2a4) [0x15c594] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x7dd) [0x12ca8d] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalCodeEx + 0x2a4) [0x15c594] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x7dd) [0x12ca8d] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x1f36) [0x12e1e6] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0xc82) [0x12cf32] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x1f36) [0x12e1e6] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python [0xd9854] | |
| ========= Host Frame:/usr/bin/python [0xd51c2] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x970) [0x12cc20] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0xc82) [0x12cf32] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x1f36) [0x12e1e6] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python [0xd9854] | |
| ========= Host Frame:/usr/bin/python [0xd51c2] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x970) [0x12cc20] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x1f36) [0x12e1e6] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python [0xd9854] | |
| ========= Host Frame:/usr/bin/python [0xd51c2] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x970) [0x12cc20] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x1f36) [0x12e1e6] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python [0xd9854] | |
| ========= Host Frame:/usr/bin/python [0xd51c2] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x970) [0x12cc20] | |
| ========= | |
| ========= Invalid __global__ read of size 4 | |
| ========= at 0x000000e0 in sgemm_sm_heavy_nt_ldg | |
| ========= by thread (244,0,0) in block (0,0,0) | |
| ========= Address 0xb04848050 is out of bounds | |
| ========= Saved host backtrace up to driver entry point at kernel launch time | |
| ========= Host Frame:/usr/lib/libcuda.so (cuLaunchKernel + 0x2c5) [0x14ad25] | |
| ========= Host Frame:/usr/local/cuda-6.0/lib64/libcublas.so.6.0 [0x210808] | |
| ========= Host Frame:/usr/local/cuda-6.0/lib64/libcublas.so.6.0 [0x231143] | |
| ========= Host Frame:/usr/local/cuda-6.0/lib64/libcublas.so.6.0 [0x1ea762] | |
| ========= Host Frame:/usr/local/cuda-6.0/lib64/libcublas.so.6.0 [0x1de029] | |
| ========= Host Frame:/usr/local/cuda-6.0/lib64/libcublas.so.6.0 [0x1decce] | |
| ========= Host Frame:/usr/local/cuda-6.0/lib64/libcublas.so.6.0 [0x1df1e3] | |
| ========= Host Frame:/usr/local/cuda-6.0/lib64/libcublas.so.6.0 [0xebf24] | |
| ========= Host Frame:/home/sander/.theano/compiledir_Linux-3.13.0-30-generic-x86_64-with-Ubuntu-14.04-trusty-x86_64-2.7.6-64/tmpWl5d5K/fa8118cce2bd717ca59c4bf8d7e8bfde.so (_Z6corrMMP11CudaNdarrayS0_S0_iiiii + 0x71f) [0x2cff] | |
| ========= Host Frame:/home/sander/.theano/compiledir_Linux-3.13.0-30-generic-x86_64-with-Ubuntu-14.04-trusty-x86_64-2.7.6-64/tmpWl5d5K/fa8118cce2bd717ca59c4bf8d7e8bfde.so (_ZN62_GLOBAL__N__38_tmpxft_000015ac_00000000_8_mod_cpp1_ii_9233ac7753__struct_compiled_op_fa8118cce2bd717ca59c4bf8d7e8bfde3runEv + 0x4ec) [0x344c] | |
| ========= Host Frame:/home/sander/.theano/compiledir_Linux-3.13.0-30-generic-x86_64-with-Ubuntu-14.04-trusty-x86_64-2.7.6-64/lazylinker_ext/lazylinker_ext.so [0x35cc] | |
| ========= Host Frame:/home/sander/.theano/compiledir_Linux-3.13.0-30-generic-x86_64-with-Ubuntu-14.04-trusty-x86_64-2.7.6-64/lazylinker_ext/lazylinker_ext.so [0x4009] | |
| ========= Host Frame:/home/sander/.theano/compiledir_Linux-3.13.0-30-generic-x86_64-with-Ubuntu-14.04-trusty-x86_64-2.7.6-64/lazylinker_ext/lazylinker_ext.so [0x4698] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x970) [0x12cc20] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python [0xd9854] | |
| ========= Host Frame:/usr/bin/python [0xd51c2] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x970) [0x12cc20] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalCodeEx + 0x2a4) [0x15c594] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x7dd) [0x12ca8d] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalCodeEx + 0x2a4) [0x15c594] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x7dd) [0x12ca8d] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalCodeEx + 0x2a4) [0x15c594] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x7dd) [0x12ca8d] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x1f36) [0x12e1e6] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0xc82) [0x12cf32] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x1f36) [0x12e1e6] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python [0xd9854] | |
| ========= Host Frame:/usr/bin/python [0xd51c2] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x970) [0x12cc20] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0xc82) [0x12cf32] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x1f36) [0x12e1e6] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python [0xd9854] | |
| ========= Host Frame:/usr/bin/python [0xd51c2] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x970) [0x12cc20] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x1f36) [0x12e1e6] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python [0xd9854] | |
| ========= Host Frame:/usr/bin/python [0xd51c2] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x970) [0x12cc20] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x1f36) [0x12e1e6] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python [0xd9854] | |
| ========= Host Frame:/usr/bin/python [0xd51c2] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x970) [0x12cc20] | |
| ========= | |
| ========= Invalid __global__ read of size 4 | |
| ========= at 0x000000e0 in sgemm_sm_heavy_nt_ldg | |
| ========= by thread (243,0,0) in block (0,0,0) | |
| ========= Address 0xb0484804c is out of bounds | |
| ========= Saved host backtrace up to driver entry point at kernel launch time | |
| ========= Host Frame:/usr/lib/libcuda.so (cuLaunchKernel + 0x2c5) [0x14ad25] | |
| ========= Host Frame:/usr/local/cuda-6.0/lib64/libcublas.so.6.0 [0x210808] | |
| ========= Host Frame:/usr/local/cuda-6.0/lib64/libcublas.so.6.0 [0x231143] | |
| ========= Host Frame:/usr/local/cuda-6.0/lib64/libcublas.so.6.0 [0x1ea762] | |
| ========= Host Frame:/usr/local/cuda-6.0/lib64/libcublas.so.6.0 [0x1de029] | |
| ========= Host Frame:/usr/local/cuda-6.0/lib64/libcublas.so.6.0 [0x1decce] | |
| ========= Host Frame:/usr/local/cuda-6.0/lib64/libcublas.so.6.0 [0x1df1e3] | |
| ========= Host Frame:/usr/local/cuda-6.0/lib64/libcublas.so.6.0 [0xebf24] | |
| ========= Host Frame:/home/sander/.theano/compiledir_Linux-3.13.0-30-generic-x86_64-with-Ubuntu-14.04-trusty-x86_64-2.7.6-64/tmpWl5d5K/fa8118cce2bd717ca59c4bf8d7e8bfde.so (_Z6corrMMP11CudaNdarrayS0_S0_iiiii + 0x71f) [0x2cff] | |
| ========= Host Frame:/home/sander/.theano/compiledir_Linux-3.13.0-30-generic-x86_64-with-Ubuntu-14.04-trusty-x86_64-2.7.6-64/tmpWl5d5K/fa8118cce2bd717ca59c4bf8d7e8bfde.so (_ZN62_GLOBAL__N__38_tmpxft_000015ac_00000000_8_mod_cpp1_ii_9233ac7753__struct_compiled_op_fa8118cce2bd717ca59c4bf8d7e8bfde3runEv + 0x4ec) [0x344c] | |
| ========= Host Frame:/home/sander/.theano/compiledir_Linux-3.13.0-30-generic-x86_64-with-Ubuntu-14.04-trusty-x86_64-2.7.6-64/lazylinker_ext/lazylinker_ext.so [0x35cc] | |
| ========= Host Frame:/home/sander/.theano/compiledir_Linux-3.13.0-30-generic-x86_64-with-Ubuntu-14.04-trusty-x86_64-2.7.6-64/lazylinker_ext/lazylinker_ext.so [0x4009] | |
| ========= Host Frame:/home/sander/.theano/compiledir_Linux-3.13.0-30-generic-x86_64-with-Ubuntu-14.04-trusty-x86_64-2.7.6-64/lazylinker_ext/lazylinker_ext.so [0x4698] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x970) [0x12cc20] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python [0xd9854] | |
| ========= Host Frame:/usr/bin/python [0xd51c2] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x970) [0x12cc20] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalCodeEx + 0x2a4) [0x15c594] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x7dd) [0x12ca8d] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalCodeEx + 0x2a4) [0x15c594] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x7dd) [0x12ca8d] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalCodeEx + 0x2a4) [0x15c594] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x7dd) [0x12ca8d] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x1f36) [0x12e1e6] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0xc82) [0x12cf32] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x1f36) [0x12e1e6] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python [0xd9854] | |
| ========= Host Frame:/usr/bin/python [0xd51c2] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x970) [0x12cc20] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0xc82) [0x12cf32] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x1f36) [0x12e1e6] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python [0xd9854] | |
| ========= Host Frame:/usr/bin/python [0xd51c2] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x970) [0x12cc20] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x1f36) [0x12e1e6] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python [0xd9854] | |
| ========= Host Frame:/usr/bin/python [0xd51c2] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x970) [0x12cc20] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x1f36) [0x12e1e6] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python [0xd9854] | |
| ========= Host Frame:/usr/bin/python [0xd51c2] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x970) [0x12cc20] | |
| ========= | |
| ========= Invalid __global__ read of size 4 | |
| ========= at 0x000000e0 in sgemm_sm_heavy_nt_ldg | |
| ========= by thread (242,0,0) in block (0,0,0) | |
| ========= Address 0xb04848048 is out of bounds | |
| ========= Saved host backtrace up to driver entry point at kernel launch time | |
| ========= Host Frame:/usr/lib/libcuda.so (cuLaunchKernel + 0x2c5) [0x14ad25] | |
| ========= Host Frame:/usr/local/cuda-6.0/lib64/libcublas.so.6.0 [0x210808] | |
| ========= Host Frame:/usr/local/cuda-6.0/lib64/libcublas.so.6.0 [0x231143] | |
| ========= Host Frame:/usr/local/cuda-6.0/lib64/libcublas.so.6.0 [0x1ea762] | |
| ========= Host Frame:/usr/local/cuda-6.0/lib64/libcublas.so.6.0 [0x1de029] | |
| ========= Host Frame:/usr/local/cuda-6.0/lib64/libcublas.so.6.0 [0x1decce] | |
| ========= Host Frame:/usr/local/cuda-6.0/lib64/libcublas.so.6.0 [0x1df1e3] | |
| ========= Host Frame:/usr/local/cuda-6.0/lib64/libcublas.so.6.0 [0xebf24] | |
| ========= Host Frame:/home/sander/.theano/compiledir_Linux-3.13.0-30-generic-x86_64-with-Ubuntu-14.04-trusty-x86_64-2.7.6-64/tmpWl5d5K/fa8118cce2bd717ca59c4bf8d7e8bfde.so (_Z6corrMMP11CudaNdarrayS0_S0_iiiii + 0x71f) [0x2cff] | |
| ========= Host Frame:/home/sander/.theano/compiledir_Linux-3.13.0-30-generic-x86_64-with-Ubuntu-14.04-trusty-x86_64-2.7.6-64/tmpWl5d5K/fa8118cce2bd717ca59c4bf8d7e8bfde.so (_ZN62_GLOBAL__N__38_tmpxft_000015ac_00000000_8_mod_cpp1_ii_9233ac7753__struct_compiled_op_fa8118cce2bd717ca59c4bf8d7e8bfde3runEv + 0x4ec) [0x344c] | |
| ========= Host Frame:/home/sander/.theano/compiledir_Linux-3.13.0-30-generic-x86_64-with-Ubuntu-14.04-trusty-x86_64-2.7.6-64/lazylinker_ext/lazylinker_ext.so [0x35cc] | |
| ========= Host Frame:/home/sander/.theano/compiledir_Linux-3.13.0-30-generic-x86_64-with-Ubuntu-14.04-trusty-x86_64-2.7.6-64/lazylinker_ext/lazylinker_ext.so [0x4009] | |
| ========= Host Frame:/home/sander/.theano/compiledir_Linux-3.13.0-30-generic-x86_64-with-Ubuntu-14.04-trusty-x86_64-2.7.6-64/lazylinker_ext/lazylinker_ext.so [0x4698] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x970) [0x12cc20] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python [0xd9854] | |
| ========= Host Frame:/usr/bin/python [0xd51c2] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x970) [0x12cc20] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalCodeEx + 0x2a4) [0x15c594] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x7dd) [0x12ca8d] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalCodeEx + 0x2a4) [0x15c594] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x7dd) [0x12ca8d] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalCodeEx + 0x2a4) [0x15c594] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x7dd) [0x12ca8d] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x1f36) [0x12e1e6] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0xc82) [0x12cf32] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x1f36) [0x12e1e6] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python [0xd9854] | |
| ========= Host Frame:/usr/bin/python [0xd51c2] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x970) [0x12cc20] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0xc82) [0x12cf32] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x1f36) [0x12e1e6] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python [0xd9854] | |
| ========= Host Frame:/usr/bin/python [0xd51c2] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x970) [0x12cc20] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x1f36) [0x12e1e6] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python [0xd9854] | |
| ========= Host Frame:/usr/bin/python [0xd51c2] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x970) [0x12cc20] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x1f36) [0x12e1e6] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python [0xd9854] | |
| ========= Host Frame:/usr/bin/python [0xd51c2] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x970) [0x12cc20] | |
| ========= | |
| ========= Invalid __global__ read of size 4 | |
| ========= at 0x000000e0 in sgemm_sm_heavy_nt_ldg | |
| ========= by thread (241,0,0) in block (0,0,0) | |
| ========= Address 0xb04848044 is out of bounds | |
| ========= Saved host backtrace up to driver entry point at kernel launch time | |
| ========= Host Frame:/usr/lib/libcuda.so (cuLaunchKernel + 0x2c5) [0x14ad25] | |
| ========= Host Frame:/usr/local/cuda-6.0/lib64/libcublas.so.6.0 [0x210808] | |
| ========= Host Frame:/usr/local/cuda-6.0/lib64/libcublas.so.6.0 [0x231143] | |
| ========= Host Frame:/usr/local/cuda-6.0/lib64/libcublas.so.6.0 [0x1ea762] | |
| ========= Host Frame:/usr/local/cuda-6.0/lib64/libcublas.so.6.0 [0x1de029] | |
| ========= Host Frame:/usr/local/cuda-6.0/lib64/libcublas.so.6.0 [0x1decce] | |
| ========= Host Frame:/usr/local/cuda-6.0/lib64/libcublas.so.6.0 [0x1df1e3] | |
| ========= Host Frame:/usr/local/cuda-6.0/lib64/libcublas.so.6.0 [0xebf24] | |
| ========= Host Frame:/home/sander/.theano/compiledir_Linux-3.13.0-30-generic-x86_64-with-Ubuntu-14.04-trusty-x86_64-2.7.6-64/tmpWl5d5K/fa8118cce2bd717ca59c4bf8d7e8bfde.so (_Z6corrMMP11CudaNdarrayS0_S0_iiiii + 0x71f) [0x2cff] | |
| ========= Host Frame:/home/sander/.theano/compiledir_Linux-3.13.0-30-generic-x86_64-with-Ubuntu-14.04-trusty-x86_64-2.7.6-64/tmpWl5d5K/fa8118cce2bd717ca59c4bf8d7e8bfde.so (_ZN62_GLOBAL__N__38_tmpxft_000015ac_00000000_8_mod_cpp1_ii_9233ac7753__struct_compiled_op_fa8118cce2bd717ca59c4bf8d7e8bfde3runEv + 0x4ec) [0x344c] | |
| ========= Host Frame:/home/sander/.theano/compiledir_Linux-3.13.0-30-generic-x86_64-with-Ubuntu-14.04-trusty-x86_64-2.7.6-64/lazylinker_ext/lazylinker_ext.so [0x35cc] | |
| ========= Host Frame:/home/sander/.theano/compiledir_Linux-3.13.0-30-generic-x86_64-with-Ubuntu-14.04-trusty-x86_64-2.7.6-64/lazylinker_ext/lazylinker_ext.so [0x4009] | |
| ========= Host Frame:/home/sander/.theano/compiledir_Linux-3.13.0-30-generic-x86_64-with-Ubuntu-14.04-trusty-x86_64-2.7.6-64/lazylinker_ext/lazylinker_ext.so [0x4698] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x970) [0x12cc20] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python [0xd9854] | |
| ========= Host Frame:/usr/bin/python [0xd51c2] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x970) [0x12cc20] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalCodeEx + 0x2a4) [0x15c594] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x7dd) [0x12ca8d] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalCodeEx + 0x2a4) [0x15c594] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x7dd) [0x12ca8d] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalCodeEx + 0x2a4) [0x15c594] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x7dd) [0x12ca8d] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x1f36) [0x12e1e6] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0xc82) [0x12cf32] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x1f36) [0x12e1e6] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python [0xd9854] | |
| ========= Host Frame:/usr/bin/python [0xd51c2] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x970) [0x12cc20] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0xc82) [0x12cf32] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x1f36) [0x12e1e6] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python [0xd9854] | |
| ========= Host Frame:/usr/bin/python [0xd51c2] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x970) [0x12cc20] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x1f36) [0x12e1e6] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python [0xd9854] | |
| ========= Host Frame:/usr/bin/python [0xd51c2] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x970) [0x12cc20] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x1f36) [0x12e1e6] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python [0xd9854] | |
| ========= Host Frame:/usr/bin/python [0xd51c2] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x970) [0x12cc20] | |
| ========= | |
| ========= Invalid __global__ read of size 4 | |
| ========= at 0x000000e0 in sgemm_sm_heavy_nt_ldg | |
| ========= by thread (240,0,0) in block (0,0,0) | |
| ========= Address 0xb04848040 is out of bounds | |
| ========= Saved host backtrace up to driver entry point at kernel launch time | |
| ========= Host Frame:/usr/lib/libcuda.so (cuLaunchKernel + 0x2c5) [0x14ad25] | |
| ========= Host Frame:/usr/local/cuda-6.0/lib64/libcublas.so.6.0 [0x210808] | |
| ========= Host Frame:/usr/local/cuda-6.0/lib64/libcublas.so.6.0 [0x231143] | |
| ========= Host Frame:/usr/local/cuda-6.0/lib64/libcublas.so.6.0 [0x1ea762] | |
| ========= Host Frame:/usr/local/cuda-6.0/lib64/libcublas.so.6.0 [0x1de029] | |
| ========= Host Frame:/usr/local/cuda-6.0/lib64/libcublas.so.6.0 [0x1decce] | |
| ========= Host Frame:/usr/local/cuda-6.0/lib64/libcublas.so.6.0 [0x1df1e3] | |
| ========= Host Frame:/usr/local/cuda-6.0/lib64/libcublas.so.6.0 [0xebf24] | |
| ========= Host Frame:/home/sander/.theano/compiledir_Linux-3.13.0-30-generic-x86_64-with-Ubuntu-14.04-trusty-x86_64-2.7.6-64/tmpWl5d5K/fa8118cce2bd717ca59c4bf8d7e8bfde.so (_Z6corrMMP11CudaNdarrayS0_S0_iiiii + 0x71f) [0x2cff] | |
| ========= Host Frame:/home/sander/.theano/compiledir_Linux-3.13.0-30-generic-x86_64-with-Ubuntu-14.04-trusty-x86_64-2.7.6-64/tmpWl5d5K/fa8118cce2bd717ca59c4bf8d7e8bfde.so (_ZN62_GLOBAL__N__38_tmpxft_000015ac_00000000_8_mod_cpp1_ii_9233ac7753__struct_compiled_op_fa8118cce2bd717ca59c4bf8d7e8bfde3runEv + 0x4ec) [0x344c] | |
| ========= Host Frame:/home/sander/.theano/compiledir_Linux-3.13.0-30-generic-x86_64-with-Ubuntu-14.04-trusty-x86_64-2.7.6-64/lazylinker_ext/lazylinker_ext.so [0x35cc] | |
| ========= Host Frame:/home/sander/.theano/compiledir_Linux-3.13.0-30-generic-x86_64-with-Ubuntu-14.04-trusty-x86_64-2.7.6-64/lazylinker_ext/lazylinker_ext.so [0x4009] | |
| ========= Host Frame:/home/sander/.theano/compiledir_Linux-3.13.0-30-generic-x86_64-with-Ubuntu-14.04-trusty-x86_64-2.7.6-64/lazylinker_ext/lazylinker_ext.so [0x4698] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x970) [0x12cc20] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python [0xd9854] | |
| ========= Host Frame:/usr/bin/python [0xd51c2] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x970) [0x12cc20] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalCodeEx + 0x2a4) [0x15c594] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x7dd) [0x12ca8d] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalCodeEx + 0x2a4) [0x15c594] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x7dd) [0x12ca8d] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalCodeEx + 0x2a4) [0x15c594] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x7dd) [0x12ca8d] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x1f36) [0x12e1e6] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0xc82) [0x12cf32] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x1f36) [0x12e1e6] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python [0xd9854] | |
| ========= Host Frame:/usr/bin/python [0xd51c2] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x970) [0x12cc20] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0xc82) [0x12cf32] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x1f36) [0x12e1e6] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python [0xd9854] | |
| ========= Host Frame:/usr/bin/python [0xd51c2] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x970) [0x12cc20] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x1f36) [0x12e1e6] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python [0xd9854] | |
| ========= Host Frame:/usr/bin/python [0xd51c2] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x970) [0x12cc20] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x1f36) [0x12e1e6] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python [0xd9854] | |
| ========= Host Frame:/usr/bin/python [0xd51c2] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x970) [0x12cc20] | |
| ========= | |
| ========= Invalid __global__ read of size 4 | |
| ========= at 0x000000e0 in sgemm_sm_heavy_nt_ldg | |
| ========= by thread (239,0,0) in block (0,0,0) | |
| ========= Address 0xb0484803c is out of bounds | |
| ========= Saved host backtrace up to driver entry point at kernel launch time | |
| ========= Host Frame:/usr/lib/libcuda.so (cuLaunchKernel + 0x2c5) [0x14ad25] | |
| ========= Host Frame:/usr/local/cuda-6.0/lib64/libcublas.so.6.0 [0x210808] | |
| ========= Host Frame:/usr/local/cuda-6.0/lib64/libcublas.so.6.0 [0x231143] | |
| ========= Host Frame:/usr/local/cuda-6.0/lib64/libcublas.so.6.0 [0x1ea762] | |
| ========= Host Frame:/usr/local/cuda-6.0/lib64/libcublas.so.6.0 [0x1de029] | |
| ========= Host Frame:/usr/local/cuda-6.0/lib64/libcublas.so.6.0 [0x1decce] | |
| ========= Host Frame:/usr/local/cuda-6.0/lib64/libcublas.so.6.0 [0x1df1e3] | |
| ========= Host Frame:/usr/local/cuda-6.0/lib64/libcublas.so.6.0 [0xebf24] | |
| ========= Host Frame:/home/sander/.theano/compiledir_Linux-3.13.0-30-generic-x86_64-with-Ubuntu-14.04-trusty-x86_64-2.7.6-64/tmpWl5d5K/fa8118cce2bd717ca59c4bf8d7e8bfde.so (_Z6corrMMP11CudaNdarrayS0_S0_iiiii + 0x71f) [0x2cff] | |
| ========= Host Frame:/home/sander/.theano/compiledir_Linux-3.13.0-30-generic-x86_64-with-Ubuntu-14.04-trusty-x86_64-2.7.6-64/tmpWl5d5K/fa8118cce2bd717ca59c4bf8d7e8bfde.so (_ZN62_GLOBAL__N__38_tmpxft_000015ac_00000000_8_mod_cpp1_ii_9233ac7753__struct_compiled_op_fa8118cce2bd717ca59c4bf8d7e8bfde3runEv + 0x4ec) [0x344c] | |
| ========= Host Frame:/home/sander/.theano/compiledir_Linux-3.13.0-30-generic-x86_64-with-Ubuntu-14.04-trusty-x86_64-2.7.6-64/lazylinker_ext/lazylinker_ext.so [0x35cc] | |
| ========= Host Frame:/home/sander/.theano/compiledir_Linux-3.13.0-30-generic-x86_64-with-Ubuntu-14.04-trusty-x86_64-2.7.6-64/lazylinker_ext/lazylinker_ext.so [0x4009] | |
| ========= Host Frame:/home/sander/.theano/compiledir_Linux-3.13.0-30-generic-x86_64-with-Ubuntu-14.04-trusty-x86_64-2.7.6-64/lazylinker_ext/lazylinker_ext.so [0x4698] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x970) [0x12cc20] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python [0xd9854] | |
| ========= Host Frame:/usr/bin/python [0xd51c2] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x970) [0x12cc20] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalCodeEx + 0x2a4) [0x15c594] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x7dd) [0x12ca8d] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalCodeEx + 0x2a4) [0x15c594] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x7dd) [0x12ca8d] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalCodeEx + 0x2a4) [0x15c594] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x7dd) [0x12ca8d] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x1f36) [0x12e1e6] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0xc82) [0x12cf32] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x1f36) [0x12e1e6] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python [0xd9854] | |
| ========= Host Frame:/usr/bin/python [0xd51c2] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x970) [0x12cc20] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0xc82) [0x12cf32] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x1f36) [0x12e1e6] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python [0xd9854] | |
| ========= Host Frame:/usr/bin/python [0xd51c2] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x970) [0x12cc20] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x1f36) [0x12e1e6] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python [0xd9854] | |
| ========= Host Frame:/usr/bin/python [0xd51c2] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x970) [0x12cc20] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x1f36) [0x12e1e6] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python [0xd9854] | |
| ========= Host Frame:/usr/bin/python [0xd51c2] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x970) [0x12cc20] | |
| ========= | |
| ========= Invalid __global__ read of size 4 | |
| ========= at 0x000000e0 in sgemm_sm_heavy_nt_ldg | |
| ========= by thread (238,0,0) in block (0,0,0) | |
| ========= Address 0xb04848038 is out of bounds | |
| ========= Saved host backtrace up to driver entry point at kernel launch time | |
| ========= Host Frame:/usr/lib/libcuda.so (cuLaunchKernel + 0x2c5) [0x14ad25] | |
| ========= Host Frame:/usr/local/cuda-6.0/lib64/libcublas.so.6.0 [0x210808] | |
| ========= Host Frame:/usr/local/cuda-6.0/lib64/libcublas.so.6.0 [0x231143] | |
| ========= Host Frame:/usr/local/cuda-6.0/lib64/libcublas.so.6.0 [0x1ea762] | |
| ========= Host Frame:/usr/local/cuda-6.0/lib64/libcublas.so.6.0 [0x1de029] | |
| ========= Host Frame:/usr/local/cuda-6.0/lib64/libcublas.so.6.0 [0x1decce] | |
| ========= Host Frame:/usr/local/cuda-6.0/lib64/libcublas.so.6.0 [0x1df1e3] | |
| ========= Host Frame:/usr/local/cuda-6.0/lib64/libcublas.so.6.0 [0xebf24] | |
| ========= Host Frame:/home/sander/.theano/compiledir_Linux-3.13.0-30-generic-x86_64-with-Ubuntu-14.04-trusty-x86_64-2.7.6-64/tmpWl5d5K/fa8118cce2bd717ca59c4bf8d7e8bfde.so (_Z6corrMMP11CudaNdarrayS0_S0_iiiii + 0x71f) [0x2cff] | |
| ========= Host Frame:/home/sander/.theano/compiledir_Linux-3.13.0-30-generic-x86_64-with-Ubuntu-14.04-trusty-x86_64-2.7.6-64/tmpWl5d5K/fa8118cce2bd717ca59c4bf8d7e8bfde.so (_ZN62_GLOBAL__N__38_tmpxft_000015ac_00000000_8_mod_cpp1_ii_9233ac7753__struct_compiled_op_fa8118cce2bd717ca59c4bf8d7e8bfde3runEv + 0x4ec) [0x344c] | |
| ========= Host Frame:/home/sander/.theano/compiledir_Linux-3.13.0-30-generic-x86_64-with-Ubuntu-14.04-trusty-x86_64-2.7.6-64/lazylinker_ext/lazylinker_ext.so [0x35cc] | |
| ========= Host Frame:/home/sander/.theano/compiledir_Linux-3.13.0-30-generic-x86_64-with-Ubuntu-14.04-trusty-x86_64-2.7.6-64/lazylinker_ext/lazylinker_ext.so [0x4009] | |
| ========= Host Frame:/home/sander/.theano/compiledir_Linux-3.13.0-30-generic-x86_64-with-Ubuntu-14.04-trusty-x86_64-2.7.6-64/lazylinker_ext/lazylinker_ext.so [0x4698] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x970) [0x12cc20] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python [0xd9854] | |
| ========= Host Frame:/usr/bin/python [0xd51c2] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x970) [0x12cc20] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalCodeEx + 0x2a4) [0x15c594] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x7dd) [0x12ca8d] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalCodeEx + 0x2a4) [0x15c594] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x7dd) [0x12ca8d] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalCodeEx + 0x2a4) [0x15c594] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x7dd) [0x12ca8d] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x1f36) [0x12e1e6] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0xc82) [0x12cf32] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x1f36) [0x12e1e6] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python [0xd9854] | |
| ========= Host Frame:/usr/bin/python [0xd51c2] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x970) [0x12cc20] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0xc82) [0x12cf32] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x1f36) [0x12e1e6] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python [0xd9854] | |
| ========= Host Frame:/usr/bin/python [0xd51c2] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x970) [0x12cc20] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x1f36) [0x12e1e6] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python [0xd9854] | |
| ========= Host Frame:/usr/bin/python [0xd51c2] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x970) [0x12cc20] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x1f36) [0x12e1e6] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python [0xd9854] | |
| ========= Host Frame:/usr/bin/python [0xd51c2] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x970) [0x12cc20] | |
| ========= | |
| ========= Invalid __global__ read of size 4 | |
| ========= at 0x000000e0 in sgemm_sm_heavy_nt_ldg | |
| ========= by thread (237,0,0) in block (0,0,0) | |
| ========= Address 0xb04848034 is out of bounds | |
| ========= Saved host backtrace up to driver entry point at kernel launch time | |
| ========= Host Frame:/usr/lib/libcuda.so (cuLaunchKernel + 0x2c5) [0x14ad25] | |
| ========= Host Frame:/usr/local/cuda-6.0/lib64/libcublas.so.6.0 [0x210808] | |
| ========= Host Frame:/usr/local/cuda-6.0/lib64/libcublas.so.6.0 [0x231143] | |
| ========= Host Frame:/usr/local/cuda-6.0/lib64/libcublas.so.6.0 [0x1ea762] | |
| ========= Host Frame:/usr/local/cuda-6.0/lib64/libcublas.so.6.0 [0x1de029] | |
| ========= Host Frame:/usr/local/cuda-6.0/lib64/libcublas.so.6.0 [0x1decce] | |
| ========= Host Frame:/usr/local/cuda-6.0/lib64/libcublas.so.6.0 [0x1df1e3] | |
| ========= Host Frame:/usr/local/cuda-6.0/lib64/libcublas.so.6.0 [0xebf24] | |
| ========= Host Frame:/home/sander/.theano/compiledir_Linux-3.13.0-30-generic-x86_64-with-Ubuntu-14.04-trusty-x86_64-2.7.6-64/tmpWl5d5K/fa8118cce2bd717ca59c4bf8d7e8bfde.so (_Z6corrMMP11CudaNdarrayS0_S0_iiiii + 0x71f) [0x2cff] | |
| ========= Host Frame:/home/sander/.theano/compiledir_Linux-3.13.0-30-generic-x86_64-with-Ubuntu-14.04-trusty-x86_64-2.7.6-64/tmpWl5d5K/fa8118cce2bd717ca59c4bf8d7e8bfde.so (_ZN62_GLOBAL__N__38_tmpxft_000015ac_00000000_8_mod_cpp1_ii_9233ac7753__struct_compiled_op_fa8118cce2bd717ca59c4bf8d7e8bfde3runEv + 0x4ec) [0x344c] | |
| ========= Host Frame:/home/sander/.theano/compiledir_Linux-3.13.0-30-generic-x86_64-with-Ubuntu-14.04-trusty-x86_64-2.7.6-64/lazylinker_ext/lazylinker_ext.so [0x35cc] | |
| ========= Host Frame:/home/sander/.theano/compiledir_Linux-3.13.0-30-generic-x86_64-with-Ubuntu-14.04-trusty-x86_64-2.7.6-64/lazylinker_ext/lazylinker_ext.so [0x4009] | |
| ========= Host Frame:/home/sander/.theano/compiledir_Linux-3.13.0-30-generic-x86_64-with-Ubuntu-14.04-trusty-x86_64-2.7.6-64/lazylinker_ext/lazylinker_ext.so [0x4698] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x970) [0x12cc20] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python [0xd9854] | |
| ========= Host Frame:/usr/bin/python [0xd51c2] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x970) [0x12cc20] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalCodeEx + 0x2a4) [0x15c594] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x7dd) [0x12ca8d] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalCodeEx + 0x2a4) [0x15c594] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x7dd) [0x12ca8d] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalCodeEx + 0x2a4) [0x15c594] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x7dd) [0x12ca8d] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x1f36) [0x12e1e6] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0xc82) [0x12cf32] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x1f36) [0x12e1e6] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python [0xd9854] | |
| ========= Host Frame:/usr/bin/python [0xd51c2] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x970) [0x12cc20] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0xc82) [0x12cf32] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x1f36) [0x12e1e6] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python [0xd9854] | |
| ========= Host Frame:/usr/bin/python [0xd51c2] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x970) [0x12cc20] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x1f36) [0x12e1e6] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python [0xd9854] | |
| ========= Host Frame:/usr/bin/python [0xd51c2] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x970) [0x12cc20] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x1f36) [0x12e1e6] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python [0xd9854] | |
| ========= Host Frame:/usr/bin/python [0xd51c2] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x970) [0x12cc20] | |
| ========= | |
| ========= Invalid __global__ read of size 4 | |
| ========= at 0x000000e0 in sgemm_sm_heavy_nt_ldg | |
| ========= by thread (236,0,0) in block (0,0,0) | |
| ========= Address 0xb04848030 is out of bounds | |
| ========= Saved host backtrace up to driver entry point at kernel launch time | |
| ========= Host Frame:/usr/lib/libcuda.so (cuLaunchKernel + 0x2c5) [0x14ad25] | |
| ========= Host Frame:/usr/local/cuda-6.0/lib64/libcublas.so.6.0 [0x210808] | |
| ========= Host Frame:/usr/local/cuda-6.0/lib64/libcublas.so.6.0 [0x231143] | |
| ========= Host Frame:/usr/local/cuda-6.0/lib64/libcublas.so.6.0 [0x1ea762] | |
| ========= Host Frame:/usr/local/cuda-6.0/lib64/libcublas.so.6.0 [0x1de029] | |
| ========= Host Frame:/usr/local/cuda-6.0/lib64/libcublas.so.6.0 [0x1decce] | |
| ========= Host Frame:/usr/local/cuda-6.0/lib64/libcublas.so.6.0 [0x1df1e3] | |
| ========= Host Frame:/usr/local/cuda-6.0/lib64/libcublas.so.6.0 [0xebf24] | |
| ========= Host Frame:/home/sander/.theano/compiledir_Linux-3.13.0-30-generic-x86_64-with-Ubuntu-14.04-trusty-x86_64-2.7.6-64/tmpWl5d5K/fa8118cce2bd717ca59c4bf8d7e8bfde.so (_Z6corrMMP11CudaNdarrayS0_S0_iiiii + 0x71f) [0x2cff] | |
| ========= Host Frame:/home/sander/.theano/compiledir_Linux-3.13.0-30-generic-x86_64-with-Ubuntu-14.04-trusty-x86_64-2.7.6-64/tmpWl5d5K/fa8118cce2bd717ca59c4bf8d7e8bfde.so (_ZN62_GLOBAL__N__38_tmpxft_000015ac_00000000_8_mod_cpp1_ii_9233ac7753__struct_compiled_op_fa8118cce2bd717ca59c4bf8d7e8bfde3runEv + 0x4ec) [0x344c] | |
| ========= Host Frame:/home/sander/.theano/compiledir_Linux-3.13.0-30-generic-x86_64-with-Ubuntu-14.04-trusty-x86_64-2.7.6-64/lazylinker_ext/lazylinker_ext.so [0x35cc] | |
| ========= Host Frame:/home/sander/.theano/compiledir_Linux-3.13.0-30-generic-x86_64-with-Ubuntu-14.04-trusty-x86_64-2.7.6-64/lazylinker_ext/lazylinker_ext.so [0x4009] | |
| ========= Host Frame:/home/sander/.theano/compiledir_Linux-3.13.0-30-generic-x86_64-with-Ubuntu-14.04-trusty-x86_64-2.7.6-64/lazylinker_ext/lazylinker_ext.so [0x4698] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x970) [0x12cc20] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python [0xd9854] | |
| ========= Host Frame:/usr/bin/python [0xd51c2] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x970) [0x12cc20] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalCodeEx + 0x2a4) [0x15c594] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x7dd) [0x12ca8d] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalCodeEx + 0x2a4) [0x15c594] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x7dd) [0x12ca8d] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalCodeEx + 0x2a4) [0x15c594] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x7dd) [0x12ca8d] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x1f36) [0x12e1e6] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0xc82) [0x12cf32] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x1f36) [0x12e1e6] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python [0xd9854] | |
| ========= Host Frame:/usr/bin/python [0xd51c2] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x970) [0x12cc20] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0xc82) [0x12cf32] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x1f36) [0x12e1e6] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python [0xd9854] | |
| ========= Host Frame:/usr/bin/python [0xd51c2] | |
| Error when tring to find the memory information on the GPU: the launch timed out and was terminated | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x970) [0x12cc20] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| Error freeing device pointer 0xb04a80000 (the launch timed out and was terminated). Driver report 0 bytes free and 0 bytes total | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x1f36) [0x12e1e6] | |
| device_free: cudaFree() returned an error, but there is already an Python error set. This happen during the clean up when there is a first error and the CUDA driver is in a so bad state that it don't work anymore. We keep the previous error set to help debugging it.========= Host Frame:/usr/bin/python [0x16d0aa] | |
| CudaNdarray_uninit: error freeing self->devdata. (self=0x7f6ae028c8f0, self->devata=0xb04a80000) | |
| ========= Host Frame:/usr/bin/python [0xd9854] | |
| ========= Host Frame:/usr/bin/python [0xd51c2] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x970) [0x12cc20] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x1f36) [0x12e1e6] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python [0xd9854] | |
| ========= Host Frame:/usr/bin/python [0xd51c2] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x970) [0x12cc20] | |
| ========= | |
| ========= Invalid __global__ read of size 4 | |
| ========= at 0x000000e0 in sgemm_sm_heavy_nt_ldg | |
| ========= by thread (235,0,0) in block (0,0,0) | |
| ========= Address 0xb0484802c is out of bounds | |
| ========= Saved host backtrace up to driver entry point at kernel launch time | |
| ========= Host Frame:/usr/lib/libcuda.so (cuLaunchKernel + 0x2c5) [0x14ad25] | |
| ========= Host Frame:/usr/local/cuda-6.0/lib64/libcublas.so.6.0 [0x210808] | |
| ========= Host Frame:/usr/local/cuda-6.0/lib64/libcublas.so.6.0 [0x231143] | |
| ========= Host Frame:/usr/local/cuda-6.0/lib64/libcublas.so.6.0 [0x1ea762] | |
| ========= Host Frame:/usr/local/cuda-6.0/lib64/libcublas.so.6.0 [0x1de029] | |
| ========= Host Frame:/usr/local/cuda-6.0/lib64/libcublas.so.6.0 [0x1decce] | |
| ========= Host Frame:/usr/local/cuda-6.0/lib64/libcublas.so.6.0 [0x1df1e3] | |
| ========= Host Frame:/usr/local/cuda-6.0/lib64/libcublas.so.6.0 [0xebf24] | |
| ========= Host Frame:/home/sander/.theano/compiledir_Linux-3.13.0-30-generic-x86_64-with-Ubuntu-14.04-trusty-x86_64-2.7.6-64/tmpWl5d5K/fa8118cce2bd717ca59c4bf8d7e8bfde.so (_Z6corrMMP11CudaNdarrayS0_S0_iiiii + 0x71f) [0x2cff] | |
| ========= Host Frame:/home/sander/.theano/compiledir_Linux-3.13.0-30-generic-x86_64-with-Ubuntu-14.04-trusty-x86_64-2.7.6-64/tmpWl5d5K/fa8118cce2bd717ca59c4bf8d7e8bfde.so (_ZN62_GLOBAL__N__38_tmpxft_000015ac_00000000_8_mod_cpp1_ii_9233ac7753__struct_compiled_op_fa8118cce2bd717ca59c4bf8d7e8bfde3runEv + 0x4ec) [0x344c] | |
| ========= Host Frame:/home/sander/.theano/compiledir_Linux-3.13.0-30-generic-x86_64-with-Ubuntu-14.04-trusty-x86_64-2.7.6-64/lazylinker_ext/lazylinker_ext.so [0x35cc] | |
| ========= Host Frame:/home/sander/.theano/compiledir_Linux-3.13.0-30-generic-x86_64-with-Ubuntu-14.04-trusty-x86_64-2.7.6-64/lazylinker_ext/lazylinker_ext.so [0x4009] | |
| ========= Host Frame:/home/sander/.theano/compiledir_Linux-3.13.0-30-generic-x86_64-with-Ubuntu-14.04-trusty-x86_64-2.7.6-64/lazylinker_ext/lazylinker_ext.so [0x4698] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x970) [0x12cc20] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python [0xd9854] | |
| ========= Host Frame:/usr/bin/python [0xd51c2] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x970) [0x12cc20] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalCodeEx + 0x2a4) [0x15c594] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x7dd) [0x12ca8d] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalCodeEx + 0x2a4) [0x15c594] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x7dd) [0x12ca8d] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalCodeEx + 0x2a4) [0x15c594] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x7dd) [0x12ca8d] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x1f36) [0x12e1e6] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0xc82) [0x12cf32] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x1f36) [0x12e1e6] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python [0xd9854] | |
| ========= Host Frame:/usr/bin/python [0xd51c2] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x970) [0x12cc20] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0xc82) [0x12cf32] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x1f36) [0x12e1e6] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python [0xd9854] | |
| ========= Host Frame:/usr/bin/python [0xd51c2] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x970) [0x12cc20] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x1f36) [0x12e1e6] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python [0xd9854] | |
| ========= Host Frame:/usr/bin/python [0xd51c2] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x970) [0x12cc20] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x1f36) [0x12e1e6] | |
| ========= Host Frame:/usr/bin/python [0x16d0aa] | |
| ========= Host Frame:/usr/bin/python [0xd9854] | |
| ========= Host Frame:/usr/bin/python [0xd51c2] | |
| ========= Host Frame:/usr/bin/python (PyEval_EvalFrameEx + 0x970) [0x12cc20] | |
| ========= | |
| Error when trying to find the memory information on the GPU: the launch timed out and was terminated | |
| Error allocating 50176 bytes of device memory (the launch timed out and was terminated). Driver report 0 bytes free and 0 bytes total | |
| Error when tring to find the memory information on the GPU: the launch timed out and was terminated | |
| Error freeing device pointer 0xb04844800 (the launch timed out and was terminated). Driver report 0 bytes free and 0 bytes total | |
| CudaNdarray_uninit: error freeing self->devdata. (self=0x7f6ae04b9930, self->devata=0xb04844800) | |
| E | |
| ====================================================================== | |
| ERROR: theano.sandbox.cuda.tests.test_conv_cuda_ndarray.test_gemm_full | |
| ---------------------------------------------------------------------- | |
| Traceback (most recent call last): | |
| File "/usr/lib/python2.7/dist-packages/nose/case.py", line 197, in runTest | |
| self.test(*self.arg) | |
| File "/home/sander/tmp/schluter/Theano/theano/sandbox/cuda/tests/test_conv_cuda_ndarray.py", line 726, in test_gemm_full | |
| test_full(conv_gemm=True) | |
| File "/home/sander/tmp/schluter/Theano/theano/sandbox/cuda/tests/test_conv_cuda_ndarray.py", line 722, in test_full | |
| theano_mode=mode, cls=cls) | |
| File "/home/sander/tmp/schluter/Theano/theano/sandbox/cuda/tests/test_conv_cuda_ndarray.py", line 286, in exec_conv | |
| print ver, id, (ishape, kshape, subshape, istride, kstride) | |
| MemoryError: error freeing device pointer 0xb04844800 (the launch timed out and was terminated) | |
| -------------------- >> begin captured stdout << --------------------- | |
| -1 355 ((16, 5, 64, 64), (8, 5, 8, 8), (1, 1), (1, 1), (1, 1)) | |
| Exception <type 'exceptions.RuntimeError'> GpuCorrMM encountered a CUBLAS error: an internal operation failed | |
| Apply node that caused the error: GpuCorrMM_gradInputs{valid, (1, 1), pad=(0, 0)}(GpuContiguous.0, GpuContiguous.0) | |
| Inputs types: [CudaNdarrayType(float32, 4D), CudaNdarrayType(float32, 4D)] | |
| Inputs shapes: [(5, 8, 8, 8), (16, 5, 64, 64)] | |
| Inputs strides: [(512, 64, 8, 1), (20480, 4096, 64, 1)] | |
| Inputs scalar values: ['not scalar', 'not scalar'] | |
| HINT: Re-running with most Theano optimization disabled could give you a back-traces when this node was created. This can be done with by setting the Theano flags optimizer=fast_compile | |
| HINT: Use the Theano flag 'exception_verbosity=high' for a debugprint of this apply node. | |
| -1 | |
| --------------------- >> end captured stdout << ---------------------- | |
| ---------------------------------------------------------------------- | |
| Ran 1 test in 56.608s | |
| FAILED (errors=1) | |
| ========= CUDA-MEMCHECK | |
| Using gpu device 0: GeForce GTX 780 Ti | |
| . | |
| ---------------------------------------------------------------------- | |
| Ran 1 test in 38.507s | |
| OK | |
| ========= ERROR SUMMARY: 0 errors | |
| ========= CUDA-MEMCHECK | |
| Using gpu device 0: GeForce GTX 780 Ti | |
| . | |
| ---------------------------------------------------------------------- | |
| Ran 1 test in 46.769s | |
| OK | |
| ========= ERROR SUMMARY: 0 errors | |
| ========= CUDA-MEMCHECK | |
| Using gpu device 0: GeForce GTX 780 Ti | |
| . | |
| ---------------------------------------------------------------------- | |
| Ran 1 test in 352.461s | |
| OK | |
| ========= ERROR SUMMARY: 0 errors |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment