Max H. Gerlach maxhgerlach
maxhgerlach
/ thread-apply-all-bt.txt
Created
October 24, 2018 12:30
(gdb) thread apply all backtrace
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| (gdb) thread apply all backtrace | |
| Thread 88 (Thread 0x7f2ae5afa700 (LWP 22174)): | |
| #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 | |
| #1 0x00007f2c14f1691c in std::condition_variable::wait(std::unique_lock<std::mutex>&) () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6 | |
| #2 0x00007f2c1e2e7fb7 in Eigen::NonBlockingThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tensorflow::thread::EigenEnvironment::Task*) () from /usr/local/lib/python2.7/dist-packages/tensorflow/python/../libtensorflow_framework.so | |
| #3 0x00007f2c1e2e8a24 in Eigen::NonBlockingThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WorkerLoop(int) () from /usr/local/lib/python2.7/dist-packages/tensorflow/python/../libtensorflow_framework.so | |
| #4 0x00007f2c1e2e7752 in std::_Function_handler<void (), tensorflow::thread::EigenEnvironment::CreateThread(std::function<void ()>)::{lambda()#1}>::_M_invoke(std::_Any_data const&) () from /usr/local/lib/python2.7/dist |
maxhgerlach
/ gdb_thread_apply_all_backtrace.txt
Created
November 9, 2018 10:50
Backtrace for Segmentation Fault (11) after about 20 hours of training, no. 2
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Thread 88 (Thread 0x7f92af7fe700 (LWP 14136)): | |
| #0 0x00007f94ff2d8593 in select () at ../sysdeps/unix/syscall-template.S:84 | |
| #1 0x0000000000597569 in ?? () | |
| #2 0x00000000004c45fa in PyEval_EvalFrameEx () | |
| #3 0x00000000004c2705 in PyEval_EvalCodeEx () | |
| #4 0x00000000004de858 in ?? () | |
| #5 0x00000000004b0c93 in PyObject_Call () | |
| #6 0x00000000004c6ef6 in PyEval_EvalFrameEx () | |
| #7 0x00000000004c9d7f in PyEval_EvalFrameEx () |
maxhgerlach
/ gist:2c7030198b86da5093c56272ede114b6
Created
May 28, 2019 14:27
Log of hanging Horovod shutdown
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Tue May 28 15:56:20 2019[1,6]<stdout>:[2019-05-28 15:56:20.361660: I horovod/common/operations.cc:1106] [6]: MaxG Shutdown A | |
| Tue May 28 15:56:20 2019[1,4]<stdout>:[2019-05-28 15:56:20.361653: I horovod/common/operations.cc:1106] [4]: MaxG Shutdown A | |
| Tue May 28 15:56:20 2019[1,7]<stdout>:[2019-05-28 15:56:20.361644: I horovod/common/operations.cc:1106] [7]: MaxG Shutdown A | |
| Tue May 28 15:56:20 2019[1,3]<stdout>:[2019-05-28 15:56:20.361649: I horovod/common/operations.cc:1106] [3]: MaxG Shutdown A | |
| Tue May 28 15:56:20 2019[1,1]<stdout>:[2019-05-28 15:56:20.361646: I horovod/common/operations.cc:1106] [1]: MaxG Shutdown A | |
| Tue May 28 15:56:20 2019[1,2]<stdout>:[2019-05-28 15:56:20.361652: I horovod/common/operations.cc:1106] [2]: MaxG Shutdown A | |
| Tue May 28 15:56:20 2019[1,0]<stdout>:[2019-05-28 15:56:20.361660: I horovod/common/operations.cc:1106] [0]: MaxG Shutdown A | |
| Tue May 28 15:56:20 2019[1,5]<stdout>:[2019-05-28 15:56:20.361700: I horovod/common/operations.cc:1106] [5]: MaxG Shutdown A | |
| Tue May 28 15:56:20 2019 |
maxhgerlach
/ gdb_thread_apply_all_backtrace-231633.txt
Created
May 29, 2019 14:52
hanging in MPI_Finalize, 75 threads
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Thread 75 (Thread 0x7f9f7103a700 (LWP 232884)): | |
| #0 0x00007fa4d186d827 in futex_abstimed_wait_cancelable (private=0, abstime=0x0, expected=0, futex_word=0x10f69ba8) at ../sysdeps/unix/sysv/linux/futex-internal.h:205 | |
| #1 do_futex_wait (sem=sem@entry=0x10f69ba8, abstime=0x0) at sem_waitcommon.c:111 | |
| #2 0x00007fa4d186d8d4 in __new_sem_wait_slow (sem=0x10f69ba8, abstime=0x0) at sem_waitcommon.c:181 | |
| #3 0x00007fa4d186d97a in __new_sem_wait (sem=<optimized out>) at sem_wait.c:29 | |
| #4 0x00007fa45e538957 in ?? () from /usr/lib/x86_64-linux-gnu/libcuda.so.1 | |
| #5 0x00007fa45e523881 in ?? () from /usr/lib/x86_64-linux-gnu/libcuda.so.1 | |
| #6 0x00007fa45e53a988 in ?? () from /usr/lib/x86_64-linux-gnu/libcuda.so.1 | |
| #7 0x00007fa4d18656ba in start_thread (arg=0x7f9f7103a700) at pthread_create.c:333 |
maxhgerlach
/ gdb_thread_apply_all_backtrace-232676.txt
Created
May 29, 2019 14:54
hanging in MPI_Finalize, 2 threads
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Thread 2 (Thread 0x7fa2c5ffb700 (LWP 232683)): | |
| #0 0x00007fa4d186e4bd in write () at ../sysdeps/unix/syscall-template.S:84 | |
| #1 0x00007fa4ccad9ede in ?? () from /usr/lib/python2.7/lib-dynload/_multiprocessing.x86_64-linux-gnu.so | |
| #2 0x00007fa4ccadb481 in ?? () from /usr/lib/python2.7/lib-dynload/_multiprocessing.x86_64-linux-gnu.so | |
| #3 0x00000000004c182d in PyEval_EvalFrameEx () | |
| #4 0x00000000004b9b66 in PyEval_EvalCodeEx () | |
| #5 0x00000000004d57a3 in ?? () | |
| #6 0x00000000004a587e in PyObject_Call () | |
| #7 0x00000000004be51e in PyEval_EvalFrameEx () |
maxhgerlach
/ gdb_thread_apply_all_backtrace_hanging_at_init.txt
Created
September 5, 2019 09:17
hvd.init() hangs with error message help-opal-shmem-mmap.txt
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| (gdb) set pagination off | |
| (gdb) thread apply all bt | |
| Thread 4 (Thread 0x7efdfdfab700 (LWP 330906)): | |
| #0 0x00007efe181faa13 in epoll_wait () at ../sysdeps/unix/syscall-template.S:84 | |
| #1 0x00007efd73f3b138 in epoll_dispatch (base=0x7efd280b78d0, tv=<optimized out>) at epoll.c:407 | |
| #2 0x00007efd73f3e4ff in opal_libevent2022_event_base_loop (base=0x7efd280b78d0, flags=1) at event.c:1630 | |
| #3 0x00007efd2dbdee9e in progress_engine () from /opt/openmpi/lib/openmpi/mca_pmix_pmix3x.so | |
| #4 0x00007efe184c46ba in start_thread (arg=0x7efdfdfab700) at pthread_create.c:333 | |
| #5 0x00007efe181fa41d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109 |