Created
February 10, 2020 15:55
-
-
Save thomasaarholt/5a2395307dec047208dc6c9004f993b6 to your computer and use it in GitHub Desktop.
Error from calling matmul on tensorflow 2.1. TensorRT errors are not the problem.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| (tensor)/shared/users/thomasaar/2020Feb: python | |
| Python 3.7.6 | packaged by conda-forge | (default, Jan 7 2020, 22:33:48) | |
| [GCC 7.3.0] on linux | |
| Type "help", "copyright", "credits" or "license" for more information. | |
| >>> import tensorflow as tf | |
| 2020-02-10 16:54:19.147934: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libnvinfer.so.6'; dlerror: libnvinfer.so.6: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/cuda-10.1/lib64:/usr/local/cuda-10.1/lib64:/shared/users/thomasaar/py37/lib:/shared/users/thomasaar/compiled/fftw/lib:/shared/users/thomasaar/compiled/prismatic/lib: | |
| 2020-02-10 16:54:19.149366: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libnvinfer_plugin.so.6'; dlerror: libnvinfer_plugin.so.6: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/cuda-10.1/lib64:/usr/local/cuda-10.1/lib64:/shared/users/thomasaar/py37/lib:/shared/users/thomasaar/compiled/fftw/lib:/shared/users/thomasaar/compiled/prismatic/lib: | |
| 2020-02-10 16:54:19.149409: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:30] Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly. | |
| >>> print(tf.matmul([[1., 2.],[3., 4.]], [[1., 2.],[3., 4.]])) | |
| 2020-02-10 16:54:24.258113: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 | |
| 2020-02-10 16:54:24.292352: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1555] Found device 0 with properties: | |
| pciBusID: 0000:18:00.0 name: GeForce RTX 2080 Ti computeCapability: 7.5 | |
| coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.73GiB deviceMemoryBandwidth: 573.69GiB/s | |
| 2020-02-10 16:54:24.294096: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1555] Found device 1 with properties: | |
| pciBusID: 0000:3b:00.0 name: GeForce RTX 2080 Ti computeCapability: 7.5 | |
| coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.73GiB deviceMemoryBandwidth: 573.69GiB/s | |
| 2020-02-10 16:54:24.295844: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1555] Found device 2 with properties: | |
| pciBusID: 0000:86:00.0 name: GeForce RTX 2080 Ti computeCapability: 7.5 | |
| coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.73GiB deviceMemoryBandwidth: 573.69GiB/s | |
| 2020-02-10 16:54:24.297417: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1555] Found device 3 with properties: | |
| pciBusID: 0000:af:00.0 name: GeForce RTX 2080 Ti computeCapability: 7.5 | |
| coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.73GiB deviceMemoryBandwidth: 573.69GiB/s | |
| 2020-02-10 16:54:24.298093: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 | |
| 2020-02-10 16:54:24.351926: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 | |
| 2020-02-10 16:54:24.380620: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 | |
| 2020-02-10 16:54:24.381554: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 | |
| 2020-02-10 16:54:24.437279: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 | |
| 2020-02-10 16:54:24.439037: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 | |
| 2020-02-10 16:54:24.671890: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 | |
| 2020-02-10 16:54:24.682857: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1697] Adding visible gpu devices: 0, 1, 2, 3 | |
| 2020-02-10 16:54:24.683363: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 AVX512F FMA | |
| 2020-02-10 16:54:24.694905: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2200005000 Hz | |
| 2020-02-10 16:54:24.697381: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x558d5288a320 initialized for platform Host (this does not guarantee that XLA will be used). Devices: | |
| 2020-02-10 16:54:24.697403: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version | |
| 2020-02-10 16:54:25.262258: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x558d520552c0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: | |
| 2020-02-10 16:54:25.262292: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): GeForce RTX 2080 Ti, Compute Capability 7.5 | |
| 2020-02-10 16:54:25.262306: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (1): GeForce RTX 2080 Ti, Compute Capability 7.5 | |
| 2020-02-10 16:54:25.262315: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (2): GeForce RTX 2080 Ti, Compute Capability 7.5 | |
| 2020-02-10 16:54:25.262324: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (3): GeForce RTX 2080 Ti, Compute Capability 7.5 | |
| 2020-02-10 16:54:25.264129: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1555] Found device 0 with properties: | |
| pciBusID: 0000:18:00.0 name: GeForce RTX 2080 Ti computeCapability: 7.5 | |
| coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.73GiB deviceMemoryBandwidth: 573.69GiB/s | |
| 2020-02-10 16:54:25.265402: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1555] Found device 1 with properties: | |
| pciBusID: 0000:3b:00.0 name: GeForce RTX 2080 Ti computeCapability: 7.5 | |
| coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.73GiB deviceMemoryBandwidth: 573.69GiB/s | |
| 2020-02-10 16:54:25.266689: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1555] Found device 2 with properties: | |
| pciBusID: 0000:86:00.0 name: GeForce RTX 2080 Ti computeCapability: 7.5 | |
| coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.73GiB deviceMemoryBandwidth: 573.69GiB/s | |
| 2020-02-10 16:54:25.267940: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1555] Found device 3 with properties: | |
| pciBusID: 0000:af:00.0 name: GeForce RTX 2080 Ti computeCapability: 7.5 | |
| coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.73GiB deviceMemoryBandwidth: 573.69GiB/s | |
| 2020-02-10 16:54:25.267989: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 | |
| 2020-02-10 16:54:25.268004: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 | |
| 2020-02-10 16:54:25.268017: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 | |
| 2020-02-10 16:54:25.268029: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 | |
| 2020-02-10 16:54:25.268041: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 | |
| 2020-02-10 16:54:25.268052: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 | |
| 2020-02-10 16:54:25.268074: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 | |
| 2020-02-10 16:54:25.277573: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1697] Adding visible gpu devices: 0, 1, 2, 3 | |
| 2020-02-10 16:54:25.277620: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 | |
| 2020-02-10 16:54:25.283414: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1096] Device interconnect StreamExecutor with strength 1 edge matrix: | |
| 2020-02-10 16:54:25.283431: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] 0 1 2 3 | |
| 2020-02-10 16:54:25.283454: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] 0: N N N N | |
| 2020-02-10 16:54:25.283460: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] 1: N N N N | |
| 2020-02-10 16:54:25.283471: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] 2: N N N N | |
| 2020-02-10 16:54:25.283481: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] 3: N N N N | |
| 2020-02-10 16:54:25.289711: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1241] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 1061 MB memory) -> physical GPU (device: 0, name: GeForce RTX 2080 Ti, pci bus id: 0000:18:00.0, compute capability: 7.5) | |
| 2020-02-10 16:54:25.291627: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1241] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 10284 MB memory) -> physical GPU (device: 1, name: GeForce RTX 2080 Ti, pci bus id: 0000:3b:00.0, compute capability: 7.5) | |
| 2020-02-10 16:54:25.293531: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1241] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:2 with 10284 MB memory) -> physical GPU (device: 2, name: GeForce RTX 2080 Ti, pci bus id: 0000:86:00.0, compute capability: 7.5) | |
| 2020-02-10 16:54:25.299741: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1241] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:3 with 9141 MB memory) -> physical GPU (device: 3, name: GeForce RTX 2080 Ti, pci bus id: 0000:af:00.0, compute capability: 7.5) | |
| 2020-02-10 16:54:25.305533: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 | |
| 2020-02-10 16:54:25.306084: E tensorflow/stream_executor/cuda/cuda_blas.cc:238] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED | |
| 2020-02-10 16:54:25.381095: E tensorflow/stream_executor/cuda/cuda_blas.cc:238] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED | |
| 2020-02-10 16:54:25.381141: W tensorflow/stream_executor/stream.cc:2041] attempting to perform BLAS operation using StreamExecutor without BLAS support | |
| Traceback (most recent call last): | |
| File "<stdin>", line 1, in <module> | |
| File "//shared/users/thomasaar/tensor/lib/python3.7/site-packages/tensorflow_core/python/util/dispatch.py", line 180, in wrapper | |
| return target(*args, **kwargs) | |
| File "//shared/users/thomasaar/tensor/lib/python3.7/site-packages/tensorflow_core/python/ops/math_ops.py", line 2798, in matmul | |
| a, b, transpose_a=transpose_a, transpose_b=transpose_b, name=name) | |
| File "//shared/users/thomasaar/tensor/lib/python3.7/site-packages/tensorflow_core/python/ops/gen_math_ops.py", line 5616, in mat_mul | |
| _ops.raise_from_not_ok_status(e, name) | |
| File "//shared/users/thomasaar/tensor/lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py", line 6606, in raise_from_not_ok_status | |
| six.raise_from(core._status_to_exception(e.code, message), None) | |
| File "<string>", line 3, in raise_from | |
| tensorflow.python.framework.errors_impl.InternalError: Blas GEMM launch failed : a.shape=(2, 2), b.shape=(2, 2), m=2, n=2, k=2 [Op:MatMul] name: MatMul/ |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment