If you want to be able to debug CUDA kernels, this isn't for you
There are tons and tons of tutorials teaching you how to debug the TensorFlow Python APIs, which is good. But there is nearly zero converage anywhere on how to debug the internals of TensorFlow. By internals I mean everything under the Python layer, i.e. the C and C++ runtime library.
Here I'll show you how to setup the environment to debug, and how to use gdb to see under the hood of TensorFlow. But before we start, I have to mention this open source book. It covered so many internal workings of TensorFlow and is definitely worth reading. But sadly it doesn't have an official English version. You can go to the issue page and there's a thread mentioning a Google translated version. I still find it useful, despite kinda hard to read.
Let's get down to it.