Exception while running odeint-v2 MPI Phase Chain example. MPI code can be found here - https://github.com/headmyshoulder/odeint-v2/blob/master/examples/mpi/phase_chain.cpp

error_16ps_run1_XeonE5.log

[test-X9DAi:13934] *** Process received signal ***
[test-X9DAi:13934] Signal: Segmentation fault (11)
[test-X9DAi:13934] Signal code: Address not mapped (1)
[test-X9DAi:13934] Failing at address: 0xd8dc3be9
terminate called after throwing an instance of 'boost::exception_detail::clone_impl<boost::exception_detail::error_info_injector<boost::mpi::exception> >'
  what():  MPI_Send: MPI_ERR_COUNT: invalid count argument
[test-X9DAi:13933] *** Process received signal ***
[test-X9DAi:13933] Signal: Aborted (6)
[test-X9DAi:13933] Signal code:  (-6)
[test-X9DAi:13934] [ 0] /lib/x86_64-linux-gnu/libc.so.6(+0x36d40) [0x7fab82bb3d40]
[test-X9DAi:13934] [ 1] /usr/lib/x86_64-linux-gnu/libboost_mpi.so.1.54.0(_ZN5boost3mpi6detail19packed_archive_recvEP19ompi_communicator_tiiRNS0_15packed_iarchiveER20ompi_status_public_t+0x49) [0x7fab84393cc9]
[test-X9DAi:13934] [ 2] ./phase_chain() [0x417d38]
[test-X9DAi:13934] [ 3] ./phase_chain() [0x4168fd]
[test-X9DAi:13934] [ 4] ./phase_chain() [0x4158cb]
[test-X9DAi:13934] [ 5] ./phase_chain() [0x413ead]
[test-X9DAi:13934] [ 6] ./phase_chain() [0x412d78]
[test-X9DAi:13934] [ 7] ./phase_chain() [0x40bfe8]
[test-X9DAi:13934] [ 8] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5) [0x7fab82b9eec5]
[test-X9DAi:13934] [ 9] ./phase_chain() [0x40bd49]
[test-X9DAi:13934] *** End of error message ***
[test-X9DAi:13933] [ 0] /lib/x86_64-linux-gnu/libc.so.6(+0x36d40) [0x7f1237c40d40]
[test-X9DAi:13933] [ 1] /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x39) [0x7f1237c40cc9]
[test-X9DAi:13933] [ 2] /lib/x86_64-linux-gnu/libc.so.6(abort+0x148) [0x7f1237c440d8]
[test-X9DAi:13933] [ 3] /usr/lib/x86_64-linux-gnu/libstdc++.so.6(_ZN9__gnu_cxx27__verbose_terminate_handlerEv+0x155) [0x7f123854c6b5]
[test-X9DAi:13933] [ 4] /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0x5e836) [0x7f123854a836]
[test-X9DAi:13933] [ 5] /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0x5e863) [0x7f123854a863]
[test-X9DAi:13933] [ 6] /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0x5eaa2) [0x7f123854aaa2]
[test-X9DAi:13933] [ 7] /usr/lib/x86_64-linux-gnu/libboost_mpi.so.1.54.0(_ZN5boost15throw_exceptionINS_3mpi9exceptionEEEvRKT_+0x1f5) [0x7f123941a685]
[test-X9DAi:13933] [ 8] /usr/lib/x86_64-linux-gnu/libboost_mpi.so.1.54.0(_ZN5boost3mpi6detail19packed_archive_sendEP19ompi_communicator_tiiRKNS0_15packed_oarchiveE+0xc5) [0x7f1239420a95]
[test-X9DAi:13933] [ 9] ./phase_chain() [0x417c5b]
[test-X9DAi:13933] [10] ./phase_chain() [0x4168de]
[test-X9DAi:13933] [11] ./phase_chain() [0x4158b1]
[test-X9DAi:13933] [12] ./phase_chain() [0x413ead]
[test-X9DAi:13933] [13] ./phase_chain() [0x412d78]
[test-X9DAi:13933] [14] ./phase_chain() [0x40bfe8]
[test-X9DAi:13933] [15] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5) [0x7f1237c2bec5]
[test-X9DAi:13933] [16] ./phase_chain() [0x40bd49]
[test-X9DAi:13933] *** End of error message ***
--------------------------------------------------------------------------
mpirun noticed that process rank 0 with PID 13933 on node test-X9DAi exited on signal 6 (Aborted).
--------------------------------------------------------------------------

error_2ps_run1.log

[pranavacer:24007] *** Process received signal ***
terminate called after throwing an instance of 'boost::exception_detail::clone_impl<boost::exception_detail::error_info_injector<boost::mpi::exception> >'
  what():  [pranavacer:24007] Signal: Segmentation fault (11)
[pranavacer:24007] Signal code: Address not mapped (1)
[pranavacer:24007] Failing at address: 0xd8dc3be9
MPI_Send: MPI_ERR_COUNT: invalid count argument
[pranavacer:24006] *** Process received signal ***
[pranavacer:24006] Signal: Aborted (6)
[pranavacer:24006] Signal code:  (-6)
[pranavacer:24007] [ 0] /lib/x86_64-linux-gnu/libc.so.6(+0x36d40) [0x7f9d77781d40]
[pranavacer:24007] [ 1] /usr/lib/x86_64-linux-gnu/libboost_mpi.so.1.54.0(_ZN5boost3mpi6detail19packed_archive_recvEP19ompi_communicator_tiiRNS0_15packed_iarchiveER20ompi_status_public_t+0x49) [0x7f9d78f61cc9]
[pranavacer:24007] [ 2] ./odeint_mpi() [0x4180f2]
[pranavacer:24007] [ 3] ./odeint_mpi() [0x416bef]
[pranavacer:24007] [ 4] ./odeint_mpi() [0x415b0d]
[pranavacer:24007] [ 5] ./odeint_mpi() [0x414057]
[pranavacer:24007] [ 6] ./odeint_mpi() [0x412e36]
[pranavacer:24007] [ 7] ./odeint_mpi() [0x40c049]
[pranavacer:24007] [ 8] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5) [0x7f9d7776cec5]
[pranavacer:24007] [ 9] ./odeint_mpi() [0x40bd89]
[pranavacer:24006] [ 0] /lib/x86_64-linux-gnu/libc.so.6(+0x36d40) [0x7f92c0589d40]
[pranavacer:24006] [ 1] /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x39) [0x7f92c0589cc9]
[pranavacer:24006] [ 2] /lib/x86_64-linux-gnu/libc.so.6(abort+0x148) [0x7f92c058d0d8]
[pranavacer:24006] [ 3] /usr/lib/x86_64-linux-gnu/libstdc++.so.6(_ZN9__gnu_cxx27__verbose_terminate_handlerEv+0x155) [0x7f92c0e956b5]
[pranavacer:24006] [ 4] /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0x5e836) [0x7f92c0e93836]
[pranavacer:24006] [ 5] /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0x5e863) [0x7f92c0e93863]
[pranavacer:24006] [ 6] /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0x5eaa2) [0x7f92c0e93aa2]
[pranavacer:24006] [ 7] /usr/lib/x86_64-linux-gnu/libboost_mpi.so.1.54.0(_ZN5boost15throw_exceptionINS_3mpi9exceptionEEEvRKT_+0x1f5) [0x7f92c1d63685]
[pranavacer:24006] [ 8] /usr/lib/x86_64-linux-gnu/libboost_mpi.so.1.54.0(_ZN5boost3mpi6detail19packed_archive_sendEP19ompi_communicator_tiiRKNS0_15packed_oarchiveE+0xc5) [0x7f92c1d69a95]
[pranavacer:24006] [ 9] ./odeint_mpi() [0x418015]
[pranavacer:24006] [10] ./odeint_mpi() [0x416bd0]
[pranavacer:24006] [11] ./odeint_mpi() [0x415af3]
[pranavacer:24006] [12] ./odeint_mpi() [0x414057]
[pranavacer:24006] [13] ./odeint_mpi() [0x412e36]
[pranavacer:24006] [14] ./odeint_mpi() [0x40c049]
[pranavacer:24006] [15] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5) [0x7f92c0574ec5]
[pranavacer:24006] [16] ./odeint_mpi() [0x40bd89]
[pranavacer:24006] *** End of error message ***
[pranavacer:24007] *** End of error message ***
--------------------------------------------------------------------------
mpirun noticed that process rank 1 with PID 24007 on node pranavacer exited on signal 11 (Segmentation fault).
--------------------------------------------------------------------------

error_2ps_run2.log

[pranavacer:24126] *** Process received signal ***
[pranavacer:24126] Signal: Segmentation fault (11)
[pranavacer:24126] Signal code: Address not mapped (1)
[pranavacer:24126] Failing at address: 0xd8dc3be9
[pranavacer:24126] [ 0] /lib/x86_64-linux-gnu/libc.so.6(+0x36d40) [0x7f929dc85d40]
[pranavacer:24126] [ 1] /usr/lib/x86_64-linux-gnu/libboost_mpi.so.1.54.0(_ZN5boost3mpi6detail19packed_archive_recvEP19ompi_communicator_tiiRNS0_15packed_iarchiveER20ompi_status_public_t+0x49) [0x7f929f465cc9]
[pranavacer:24126] [ 2] ./odeint_mpi() [0x4180f2]
[pranavacer:24126] [ 3] ./odeint_mpi() [0x416bef]
[pranavacer:24126] [ 4] ./odeint_mpi() [0x415b0d]
[pranavacer:24126] [ 5] ./odeint_mpi() [0x414057]
[pranavacer:24126] [ 6] ./odeint_mpi() [0x412e36]
[pranavacer:24126] [ 7] ./odeint_mpi() [0x40c049]
[pranavacer:24126] [ 8] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5) [0x7f929dc70ec5]
[pranavacer:24126] [ 9] ./odeint_mpi() [0x40bd89]
[pranavacer:24126] *** End of error message ***
terminate called after throwing an instance of 'boost::exception_detail::clone_impl<boost::exception_detail::error_info_injector<boost::mpi::exception> >'
  what():  MPI_Send: MPI_ERR_COUNT: invalid count argument
[pranavacer:24125] *** Process received signal ***
[pranavacer:24125] Signal: Aborted (6)
[pranavacer:24125] Signal code:  (-6)
[pranavacer:24125] [ 0] /lib/x86_64-linux-gnu/libc.so.6(+0x36d40) [0x7fc78d36dd40]
[pranavacer:24125] [ 1] /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x39) [0x7fc78d36dcc9]
[pranavacer:24125] [ 2] /lib/x86_64-linux-gnu/libc.so.6(abort+0x148) [0x7fc78d3710d8]
[pranavacer:24125] [ 3] /usr/lib/x86_64-linux-gnu/libstdc++.so.6(_ZN9__gnu_cxx27__verbose_terminate_handlerEv+0x155) [0x7fc78dc796b5]
[pranavacer:24125] [ 4] /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0x5e836) [0x7fc78dc77836]
[pranavacer:24125] [ 5] /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0x5e863) [0x7fc78dc77863]
[pranavacer:24125] [ 6] /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0x5eaa2) [0x7fc78dc77aa2]
[pranavacer:24125] [ 7] /usr/lib/x86_64-linux-gnu/libboost_mpi.so.1.54.0(_ZN5boost15throw_exceptionINS_3mpi9exceptionEEEvRKT_+0x1f5) [0x7fc78eb47685]
[pranavacer:24125] [ 8] /usr/lib/x86_64-linux-gnu/libboost_mpi.so.1.54.0(_ZN5boost3mpi6detail19packed_archive_sendEP19ompi_communicator_tiiRKNS0_15packed_oarchiveE+0xc5) [0x7fc78eb4da95]
[pranavacer:24125] [ 9] ./odeint_mpi() [0x418015]
[pranavacer:24125] [10] ./odeint_mpi() [0x416bd0]
[pranavacer:24125] [11] ./odeint_mpi() [0x415af3]
[pranavacer:24125] [12] ./odeint_mpi() [0x414057]
[pranavacer:24125] [13] ./odeint_mpi() [0x412e36]
[pranavacer:24125] [14] ./odeint_mpi() [0x40c049]
[pranavacer:24125] [15] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5) [0x7fc78d358ec5]
[pranavacer:24125] [16] ./odeint_mpi() [0x40bd89]
[pranavacer:24125] *** End of error message ***
--------------------------------------------------------------------------
mpirun noticed that process rank 1 with PID 24126 on node pranavacer exited on signal 11 (Segmentation fault).
--------------------------------------------------------------------------

error_2ps_run3_XeonE5.log

[test-X9DAi:13756] *** Process received signal ***
[test-X9DAi:13756] Signal: Segmentation fault (11)
[test-X9DAi:13756] Signal code: Address not mapped (1)
[test-X9DAi:13756] Failing at address: 0xd8dc3be9
terminate called after throwing an instance of 'boost::exception_detail::clone_impl<boost::exception_detail::error_info_injector<boost::mpi::exception> >'
  what():  MPI_Send: MPI_ERR_COUNT: invalid count argument
[test-X9DAi:13755] *** Process received signal ***
[test-X9DAi:13755] Signal: Aborted (6)
[test-X9DAi:13755] Signal code:  (-6)
[test-X9DAi:13756] [ 0] /lib/x86_64-linux-gnu/libc.so.6(+0x36d40) [0x7f405e3b6d40]
[test-X9DAi:13756] [ 1] /usr/lib/x86_64-linux-gnu/libboost_mpi.so.1.54.0(_ZN5boost3mpi6detail19packed_archive_recvEP19ompi_communicator_tiiRNS0_15packed_iarchiveER20ompi_status_public_t+0x49) [0x7f405fb96cc9]
[test-X9DAi:13756] [ 2] ./phase_chain() [0x417d38]
[test-X9DAi:13756] [ 3] ./phase_chain() [0x4168fd]
[test-X9DAi:13756] [ 4] ./phase_chain() [0x4158cb]
[test-X9DAi:13756] [ 5] ./phase_chain() [0x413ead]
[test-X9DAi:13756] [ 6] ./phase_chain() [0x412d78]
[test-X9DAi:13756] [ 7] ./phase_chain() [0x40bfe8]
[test-X9DAi:13756] [ 8] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5) [0x7f405e3a1ec5]
[test-X9DAi:13756] [ 9] ./phase_chain() [0x40bd49]
[test-X9DAi:13756] *** End of error message ***
[test-X9DAi:13755] [ 0] /lib/x86_64-linux-gnu/libc.so.6(+0x36d40) [0x7f269a422d40]
[test-X9DAi:13755] [ 1] /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x39) [0x7f269a422cc9]
[test-X9DAi:13755] [ 2] /lib/x86_64-linux-gnu/libc.so.6(abort+0x148) [0x7f269a4260d8]
[test-X9DAi:13755] [ 3] /usr/lib/x86_64-linux-gnu/libstdc++.so.6(_ZN9__gnu_cxx27__verbose_terminate_handlerEv+0x155) [0x7f269ad2e6b5]
[test-X9DAi:13755] [ 4] /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0x5e836) [0x7f269ad2c836]
[test-X9DAi:13755] [ 5] /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0x5e863) [0x7f269ad2c863]
[test-X9DAi:13755] [ 6] /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0x5eaa2) [0x7f269ad2caa2]
[test-X9DAi:13755] [ 7] /usr/lib/x86_64-linux-gnu/libboost_mpi.so.1.54.0(_ZN5boost15throw_exceptionINS_3mpi9exceptionEEEvRKT_+0x1f5) [0x7f269bbfc685]
[test-X9DAi:13755] [ 8] /usr/lib/x86_64-linux-gnu/libboost_mpi.so.1.54.0(_ZN5boost3mpi6detail19packed_archive_sendEP19ompi_communicator_tiiRKNS0_15packed_oarchiveE+0xc5) [0x7f269bc02a95]
[test-X9DAi:13755] [ 9] ./phase_chain() [0x417c5b]
[test-X9DAi:13755] [10] ./phase_chain() [0x4168de]
[test-X9DAi:13755] [11] ./phase_chain() [0x4158b1]
[test-X9DAi:13755] [12] ./phase_chain() [0x413ead]
[test-X9DAi:13755] [13] ./phase_chain() [0x412d78]
[test-X9DAi:13755] [14] ./phase_chain() [0x40bfe8]
[test-X9DAi:13755] [15] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5) [0x7f269a40dec5]
[test-X9DAi:13755] [16] ./phase_chain() [0x40bd49]
[test-X9DAi:13755] *** End of error message ***
--------------------------------------------------------------------------
mpirun noticed that process rank 0 with PID 13755 on node test-X9DAi exited on signal 6 (Aborted).
--------------------------------------------------------------------------

error_4ps_run1.log

[pranavacer:24246] *** Process received signal ***
terminate called after throwing an instance of 'boost::exception_detail::clone_impl<boost::exception_detail::error_info_injector<boost::mpi::exception> >'
  what():  MPI_Send: MPI_ERR_COUNT: invalid count argument
[pranavacer:24245] *** Process received signal ***
[pranavacer:24245] Signal: Aborted (6)
[pranavacer:24245] Signal code:  (-6)
[pranavacer:24246] Signal: Segmentation fault (11)
[pranavacer:24246] Signal code: Address not mapped (1)
[pranavacer:24246] Failing at address: 0xd8dc3be9
[pranavacer:24246] [ 0] /lib/x86_64-linux-gnu/libc.so.6(+0x36d40) [0x7f28a04ded40]
[pranavacer:24246] [ 1] /usr/lib/x86_64-linux-gnu/libboost_mpi.so.1.54.0(_ZN5boost3mpi6detail19packed_archive_recvEP19ompi_communicator_tiiRNS0_15packed_iarchiveER20ompi_status_public_t+0x49) [0x7f28a1cbecc9]
[pranavacer:24245] [ 0] /lib/x86_64-linux-gnu/libc.so.6(+0x36d40) [0x7f4fa95a8d40]
[pranavacer:24245] [ 1] /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x39) [0x7f4fa95a8cc9]
[pranavacer:24245] [ 2] /lib/x86_64-linux-gnu/libc.so.6(abort+0x148) [0x7f4fa95ac0d8]
[pranavacer:24245] [ 3] /usr/lib/x86_64-linux-gnu/libstdc++.so.6(_ZN9__gnu_cxx27__verbose_terminate_handlerEv+0x155) [0x7f4fa9eb46b5]
[pranavacer:24245] [ 4] /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0x5e836) [0x7f4fa9eb2836]
[pranavacer:24245] [ 5] /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0x5e863) [0x7f4fa9eb2863]
[pranavacer:24245] [ 6] /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0x5eaa2) [0x7f4fa9eb2aa2]
[pranavacer:24245] [ 7] /usr/lib/x86_64-linux-gnu/libboost_mpi.so.1.54.0(_ZN5boost15throw_exceptionINS_3mpi9exceptionEEEvRKT_+0x1f5) [0x7f4faad82685]
[pranavacer:24245] [ 8] /usr/lib/x86_64-linux-gnu/libboost_mpi.so.1.54.0(_ZN5boost3mpi6detail19packed_archive_sendEP19ompi_communicator_tiiRKNS0_15packed_oarchiveE+0xc5) [0x7f4faad88a95]
[pranavacer:24245] [ 9] ./odeint_mpi() [0x418015]
[pranavacer:24245] [10] ./odeint_mpi() [0x416bd0]
[pranavacer:24245] [11] ./odeint_mpi() [0x415af3]
[pranavacer:24245] [12] ./odeint_mpi() [0x414057]
[pranavacer:24245] [13] ./odeint_mpi() [0x412e36]
[pranavacer:24245] [14] ./odeint_mpi() [0x40c049]
[pranavacer:24245] [15] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5) [0x7f4fa9593ec5]
[pranavacer:24245] [16] ./odeint_mpi() [0x40bd89]
[pranavacer:24245] *** End of error message ***
[pranavacer:24246] [ 2] ./odeint_mpi() [0x4180f2]
[pranavacer:24246] [ 3] ./odeint_mpi() [0x416bef]
[pranavacer:24246] [ 4] ./odeint_mpi() [0x415b0d]
[pranavacer:24246] [ 5] ./odeint_mpi() [0x414057]
[pranavacer:24246] [ 6] ./odeint_mpi() [0x412e36]
[pranavacer:24246] [ 7] ./odeint_mpi() [0x40c049]
[pranavacer:24246] [ 8] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5) [0x7f28a04c9ec5]
[pranavacer:24246] [ 9] ./odeint_mpi() [0x40bd89]
[pranavacer:24246] *** End of error message ***
--------------------------------------------------------------------------
mpirun noticed that process rank 0 with PID 24245 on node pranavacer exited on signal 6 (Aborted).
--------------------------------------------------------------------------

error_4ps_run2_XeonE5.log

[test-X9DAi:13747] *** Process received signal ***
[test-X9DAi:13747] Signal: Segmentation fault (11)
[test-X9DAi:13747] Signal code: Address not mapped (1)
[test-X9DAi:13747] Failing at address: 0xd8dc3be9
[test-X9DAi:13747] [ 0] /lib/x86_64-linux-gnu/libc.so.6(+0x36d40) [0x7fec4d007d40]
[test-X9DAi:13747] [ 1] /usr/lib/x86_64-linux-gnu/libboost_mpi.so.1.54.0(_ZN5boost3mpi6detail19packed_archive_recvEP19ompi_communicator_tiiRNS0_15packed_iarchiveER20ompi_status_public_t+0x49) [0x7fec4e7e7cc9]
[test-X9DAi:13747] [ 2] ./phase_chain() [0x417d38]
[test-X9DAi:13747] [ 3] ./phase_chain() [0x4168fd]
[test-X9DAi:13747] [ 4] ./phase_chain() [0x4158cb]
[test-X9DAi:13747] [ 5] ./phase_chain() [0x413ead]
[test-X9DAi:13747] [ 6] ./phase_chain() [0x412d78]
[test-X9DAi:13747] [ 7] ./phase_chain() [0x40bfe8]
[test-X9DAi:13747] [ 8] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5) [0x7fec4cff2ec5]
[test-X9DAi:13747] [ 9] ./phase_chain() [0x40bd49]
[test-X9DAi:13747] *** End of error message ***
terminate called after throwing an instance of 'boost::exception_detail::clone_impl<boost::exception_detail::error_info_injector<boost::mpi::exception> >'
  what():  MPI_Send: MPI_ERR_COUNT: invalid count argument
[test-X9DAi:13746] *** Process received signal ***
[test-X9DAi:13746] Signal: Aborted (6)
[test-X9DAi:13746] Signal code:  (-6)
[test-X9DAi:13746] [ 0] /lib/x86_64-linux-gnu/libc.so.6(+0x36d40) [0x7f48b512ad40]
[test-X9DAi:13746] [ 1] /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x39) [0x7f48b512acc9]
[test-X9DAi:13746] [ 2] /lib/x86_64-linux-gnu/libc.so.6(abort+0x148) [0x7f48b512e0d8]
[test-X9DAi:13746] [ 3] /usr/lib/x86_64-linux-gnu/libstdc++.so.6(_ZN9__gnu_cxx27__verbose_terminate_handlerEv+0x155) [0x7f48b5a366b5]
[test-X9DAi:13746] [ 4] /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0x5e836) [0x7f48b5a34836]
[test-X9DAi:13746] [ 5] /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0x5e863) [0x7f48b5a34863]
[test-X9DAi:13746] [ 6] /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0x5eaa2) [0x7f48b5a34aa2]
[test-X9DAi:13746] [ 7] /usr/lib/x86_64-linux-gnu/libboost_mpi.so.1.54.0(_ZN5boost15throw_exceptionINS_3mpi9exceptionEEEvRKT_+0x1f5) [0x7f48b6904685]
[test-X9DAi:13746] [ 8] /usr/lib/x86_64-linux-gnu/libboost_mpi.so.1.54.0(_ZN5boost3mpi6detail19packed_archive_sendEP19ompi_communicator_tiiRKNS0_15packed_oarchiveE+0xc5) [0x7f48b690aa95]
[test-X9DAi:13746] [ 9] ./phase_chain() [0x417c5b]
[test-X9DAi:13746] [10] ./phase_chain() [0x4168de]
[test-X9DAi:13746] [11] ./phase_chain() [0x4158b1]
[test-X9DAi:13746] [12] ./phase_chain() [0x413ead]
[test-X9DAi:13746] [13] ./phase_chain() [0x412d78]
[test-X9DAi:13746] [14] ./phase_chain() [0x40bfe8]
[test-X9DAi:13746] [15] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5) [0x7f48b5115ec5]
[test-X9DAi:13746] [16] ./phase_chain() [0x40bd49]
[test-X9DAi:13746] *** End of error message ***
--------------------------------------------------------------------------
mpirun noticed that process rank 1 with PID 13747 on node test-X9DAi exited on signal 11 (Segmentation fault).
--------------------------------------------------------------------------

error_8ps_run1_XeonE5.log

[test-X9DAi:13803] *** Process received signal ***
[test-X9DAi:13803] Signal: Segmentation fault (11)
[test-X9DAi:13803] Signal code:  (128)
[test-X9DAi:13803] Failing at address: (nil)
[test-X9DAi:13804] *** Process received signal ***
[test-X9DAi:13804] Signal: Segmentation fault (11)
[test-X9DAi:13804] Signal code: Address not mapped (1)
[test-X9DAi:13804] Failing at address: 0xd8dc3be9
[test-X9DAi:13804] [ 0] /lib/x86_64-linux-gnu/libc.so.6(+0x36d40) [0x7f4191485d40]
[test-X9DAi:13804] [ 1] /usr/lib/x86_64-linux-gnu/libboost_mpi.so.1.54.0(_ZN5boost3mpi6detail19packed_archive_recvEP19ompi_communicator_tiiRNS0_15packed_iarchiveER20ompi_status_public_t+0x49) [0x7f4192c65cc9]
[test-X9DAi:13804] [ 2] ./phase_chain() [0x417d38]
[test-X9DAi:13804] [ 3] ./phase_chain() [0x4168fd]
[test-X9DAi:13804] [ 4] ./phase_chain() [0x4158cb]
[test-X9DAi:13804] [ 5] ./phase_chain() [0x413ead]
[test-X9DAi:13804] [ 6] ./phase_chain() [0x412d78]
[test-X9DAi:13804] [ 7] ./phase_chain() [0x40bfe8]
[test-X9DAi:13804] [ 8] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5) [0x7f4191470ec5]
[test-X9DAi:13804] [ 9] ./phase_chain() [0x40bd49]
[test-X9DAi:13804] *** End of error message ***
[test-X9DAi:13803] [ 0] /lib/x86_64-linux-gnu/libc.so.6(+0x36d40) [0x7fb5ab890d40]
[test-X9DAi:13803] [ 1] /lib/x86_64-linux-gnu/libc.so.6(+0x98b80) [0x7fb5ab8f2b80]
[test-X9DAi:13803] [ 2] /usr/lib/libmpi.so.1(opal_convertor_pack+0x129) [0x7fb5ac53f779]
[test-X9DAi:13803] [ 3] /usr/lib/openmpi/lib/openmpi/mca_btl_sm.so(mca_btl_sm_prepare_src+0x1fd) [0x7fb5a54c885d]
[test-X9DAi:13803] [ 4] /usr/lib/openmpi/lib/openmpi/mca_pml_ob1.so(mca_pml_ob1_send_request_start_rndv+0x1dc) [0x7fb5a6331c9c]
[test-X9DAi:13803] [ 5] /usr/lib/openmpi/lib/openmpi/mca_pml_ob1.so(mca_pml_ob1_send+0xd0c) [0x7fb5a632528c]
[test-X9DAi:13803] [ 6] /usr/lib/libmpi.so.1(MPI_Send+0x11c) [0x7fb5ac4b6ebc]
[test-X9DAi:13803] [ 7] /usr/lib/x86_64-linux-gnu/libboost_mpi.so.1.54.0(_ZN5boost3mpi6detail19packed_archive_sendEP19ompi_communicator_tiiRKNS0_15packed_oarchiveE+0x6b) [0x7fb5ad070a3b]
[test-X9DAi:13803] [ 8] ./phase_chain() [0x417c5b]
[test-X9DAi:13803] [ 9] ./phase_chain() [0x4168de]
[test-X9DAi:13803] [10] ./phase_chain() [0x4158b1]
[test-X9DAi:13803] [11] ./phase_chain() [0x413ead]
[test-X9DAi:13803] [12] ./phase_chain() [0x412d78]
[test-X9DAi:13803] [13] ./phase_chain() [0x40bfe8]
[test-X9DAi:13803] [14] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5) [0x7fb5ab87bec5]
[test-X9DAi:13803] [15] ./phase_chain() [0x40bd49]
[test-X9DAi:13803] *** End of error message ***
--------------------------------------------------------------------------
mpirun noticed that process rank 0 with PID 13803 on node test-X9DAi exited on signal 11 (Segmentation fault).
--------------------------------------------------------------------------

error_clang_16ps_run1_XeonE5.log

$ mpirun -n 16 ./phase_chain
[test-X9DAi:14308] *** Process received signal ***
[test-X9DAi:14308] Signal: Segmentation fault (11)
[test-X9DAi:14308] Signal code: Address not mapped (1)
[test-X9DAi:14308] Failing at address: 0x8
terminate called after throwing an instance of 'boost::exception_detail::clone_impl<boost::exception_detail::error_info_injector<boost::mpi::exception> >'
  what():  MPI_Send: MPI_ERR_COUNT: invalid count argument
[test-X9DAi:14307] *** Process received signal ***
[test-X9DAi:14307] Signal: Aborted (6)
[test-X9DAi:14307] Signal code:  (-6)
[test-X9DAi:14308] [ 0] /lib/x86_64-linux-gnu/libpthread.so.0(+0x10340) [0x7f60e2917340]
[test-X9DAi:14308] [ 1] /usr/lib/x86_64-linux-gnu/libboost_mpi.so.1.54.0(_ZN5boost3mpi6detail19packed_archive_recvEP19ompi_communicator_tiiRNS0_15packed_iarchiveER20ompi_status_public_t+0x49) [0x7f60e43b9cc9]
[test-X9DAi:14308] [ 2] ./phase_chain() [0x425187]
[test-X9DAi:14308] [ 3] ./phase_chain() [0x424ebd]
[test-X9DAi:14308] [ 4] ./phase_chain() [0x424e59]
[test-X9DAi:14308] [ 5] ./phase_chain() [0x424c9c]
[test-X9DAi:14308] [ 6] ./phase_chain() [0x40cc7d]
[test-X9DAi:14308] [ 7] ./phase_chain() [0x40bda0]
[test-X9DAi:14308] [ 8] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5) [0x7f60e2562ec5]
[test-X9DAi:14308] [ 9] ./phase_chain() [0x40ba8c]
[test-X9DAi:14308] *** End of error message ***
[test-X9DAi:14307] [ 0] /lib/x86_64-linux-gnu/libpthread.so.0(+0x10340) [0x7f0973134340]
[test-X9DAi:14307] [ 1] /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x39) [0x7f0972d94cc9]
[test-X9DAi:14307] [ 2] /lib/x86_64-linux-gnu/libc.so.6(abort+0x148) [0x7f0972d980d8]
[test-X9DAi:14307] [ 3] /usr/lib/x86_64-linux-gnu/libstdc++.so.6(_ZN9__gnu_cxx27__verbose_terminate_handlerEv+0x155) [0x7f097429d6b5]
[test-X9DAi:14307] [ 4] /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0x5e836) [0x7f097429b836]
[test-X9DAi:14307] [ 5] /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0x5e863) [0x7f097429b863]
[test-X9DAi:14307] [ 6] /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0x5eaa2) [0x7f097429baa2]
[test-X9DAi:14307] [ 7] /usr/lib/x86_64-linux-gnu/libboost_mpi.so.1.54.0(_ZN5boost15throw_exceptionINS_3mpi9exceptionEEEvRKT_+0x1f5) [0x7f0974bd0685]
[test-X9DAi:14307] [ 8] /usr/lib/x86_64-linux-gnu/libboost_mpi.so.1.54.0(_ZN5boost3mpi6detail19packed_archive_sendEP19ompi_communicator_tiiRKNS0_15packed_oarchiveE+0xc5) [0x7f0974bd6a95]
[test-X9DAi:14307] [ 9] ./phase_chain() [0x425092]
[test-X9DAi:14307] [10] ./phase_chain() [0x424ea3]
[test-X9DAi:14307] [11] ./phase_chain() [0x424e3a]
[test-X9DAi:14307] [12] ./phase_chain() [0x424c9c]
[test-X9DAi:14307] [13] ./phase_chain() [0x40cc7d]
[test-X9DAi:14307] [14] ./phase_chain() [0x40bda0]
[test-X9DAi:14307] [15] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5) [0x7f0972d7fec5]
[test-X9DAi:14307] [16] ./phase_chain() [0x40ba8c]
[test-X9DAi:14307] *** End of error message ***
--------------------------------------------------------------------------
mpirun noticed that process rank 0 with PID 14307 on node test-X9DAi exited on signal 6 (Aborted).
--------------------------------------------------------------------------

error_clang_2ps_run1_XeonE5.log

$ mpirun -n 2 ./phase_chain
[test-X9DAi:14357] *** Process received signal ***
[test-X9DAi:14357] Signal: Segmentation fault (11)
[test-X9DAi:14357] Signal code: Address not mapped (1)
[test-X9DAi:14357] Failing at address: 0x8
terminate called after throwing an instance of 'boost::exception_detail::clone_impl<boost::exception_detail::error_info_injector<boost::mpi::exception> >'
  what():  MPI_Send: MPI_ERR_COUNT: invalid count argument
[test-X9DAi:14356] *** Process received signal ***
[test-X9DAi:14356] Signal: Aborted (6)
[test-X9DAi:14356] Signal code:  (-6)
[test-X9DAi:14357] [ 0] /lib/x86_64-linux-gnu/libpthread.so.0(+0x10340) [0x7f6d7208f340]
[test-X9DAi:14357] [ 1] /usr/lib/x86_64-linux-gnu/libboost_mpi.so.1.54.0(_ZN5boost3mpi6detail19packed_archive_recvEP19ompi_communicator_tiiRNS0_15packed_iarchiveER20ompi_status_public_t+0x49) [0x7f6d73b31cc9]
[test-X9DAi:14357] [ 2] ./phase_chain() [0x425187]
[test-X9DAi:14357] [ 3] ./phase_chain() [0x424ebd]
[test-X9DAi:14357] [ 4] ./phase_chain() [0x424e59]
[test-X9DAi:14357] [ 5] ./phase_chain() [0x424c9c]
[test-X9DAi:14357] [ 6] ./phase_chain() [0x40cc7d]
[test-X9DAi:14357] [ 7] ./phase_chain() [0x40bda0]
[test-X9DAi:14357] [ 8] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5) [0x7f6d71cdaec5]
[test-X9DAi:14357] [ 9] ./phase_chain() [0x40ba8c]
[test-X9DAi:14357] *** End of error message ***
[test-X9DAi:14356] [ 0] /lib/x86_64-linux-gnu/libpthread.so.0(+0x10340) [0x7f7ed8652340]
[test-X9DAi:14356] [ 1] /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x39) [0x7f7ed82b2cc9]
[test-X9DAi:14356] [ 2] /lib/x86_64-linux-gnu/libc.so.6(abort+0x148) [0x7f7ed82b60d8]
[test-X9DAi:14356] [ 3] /usr/lib/x86_64-linux-gnu/libstdc++.so.6(_ZN9__gnu_cxx27__verbose_terminate_handlerEv+0x155) [0x7f7ed97bb6b5]
[test-X9DAi:14356] [ 4] /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0x5e836) [0x7f7ed97b9836]
[test-X9DAi:14356] [ 5] /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0x5e863) [0x7f7ed97b9863]
[test-X9DAi:14356] [ 6] /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0x5eaa2) [0x7f7ed97b9aa2]
[test-X9DAi:14356] [ 7] /usr/lib/x86_64-linux-gnu/libboost_mpi.so.1.54.0(_ZN5boost15throw_exceptionINS_3mpi9exceptionEEEvRKT_+0x1f5) [0x7f7eda0ee685]
[test-X9DAi:14356] [ 8] /usr/lib/x86_64-linux-gnu/libboost_mpi.so.1.54.0(_ZN5boost3mpi6detail19packed_archive_sendEP19ompi_communicator_tiiRKNS0_15packed_oarchiveE+0xc5) [0x7f7eda0f4a95]
[test-X9DAi:14356] [ 9] ./phase_chain() [0x425092]
[test-X9DAi:14356] [10] ./phase_chain() [0x424ea3]
[test-X9DAi:14356] [11] ./phase_chain() [0x424e3a]
[test-X9DAi:14356] [12] ./phase_chain() [0x424c9c]
[test-X9DAi:14356] [13] ./phase_chain() [0x40cc7d]
[test-X9DAi:14356] [14] ./phase_chain() [0x40bda0]
[test-X9DAi:14356] [15] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5) [0x7f7ed829dec5]
[test-X9DAi:14356] [16] ./phase_chain() [0x40ba8c]
[test-X9DAi:14356] *** End of error message ***
--------------------------------------------------------------------------
mpirun noticed that process rank 0 with PID 14356 on node test-X9DAi exited on signal 6 (Aborted).
--------------------------------------------------------------------------

error_clang_4ps_run1_XeonE5.log

 $ mpirun -n 4 ./phase_chain
[test-X9DAi:14347] *** Process received signal ***
[test-X9DAi:14347] Signal: Segmentation fault (11)
[test-X9DAi:14347] Signal code: Address not mapped (1)
[test-X9DAi:14347] Failing at address: 0x8
terminate called after throwing an instance of 'boost::exception_detail::clone_impl<boost::exception_detail::error_info_injector<boost::mpi::exception> >'
  what():  MPI_Send: MPI_ERR_COUNT: invalid count argument
[test-X9DAi:14346] *** Process received signal ***
[test-X9DAi:14346] Signal: Aborted (6)
[test-X9DAi:14346] Signal code:  (-6)
[test-X9DAi:14347] [ 0] /lib/x86_64-linux-gnu/libpthread.so.0(+0x10340) [0x7fa41d605340]
[test-X9DAi:14347] [ 1] /usr/lib/x86_64-linux-gnu/libboost_mpi.so.1.54.0(_ZN5boost3mpi6detail19packed_archive_recvEP19ompi_communicator_tiiRNS0_15packed_iarchiveER20ompi_status_public_t+0x49) [0x7fa41f0a7cc9]
[test-X9DAi:14347] [ 2] ./phase_chain() [0x425187]
[test-X9DAi:14347] [ 3] ./phase_chain() [0x424ebd]
[test-X9DAi:14347] [ 4] ./phase_chain() [0x424e59]
[test-X9DAi:14347] [ 5] ./phase_chain() [0x424c9c]
[test-X9DAi:14347] [ 6] ./phase_chain() [0x40cc7d]
[test-X9DAi:14347] [ 7] ./phase_chain() [0x40bda0]
[test-X9DAi:14347] [ 8] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5) [0x7fa41d250ec5]
[test-X9DAi:14347] [ 9] ./phase_chain() [0x40ba8c]
[test-X9DAi:14347] *** End of error message ***
[test-X9DAi:14346] [ 0] /lib/x86_64-linux-gnu/libpthread.so.0(+0x10340) [0x7f38497b3340]
[test-X9DAi:14346] [ 1] /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x39) [0x7f3849413cc9]
[test-X9DAi:14346] [ 2] /lib/x86_64-linux-gnu/libc.so.6(abort+0x148) [0x7f38494170d8]
[test-X9DAi:14346] [ 3] /usr/lib/x86_64-linux-gnu/libstdc++.so.6(_ZN9__gnu_cxx27__verbose_terminate_handlerEv+0x155) [0x7f384a91c6b5]
[test-X9DAi:14346] [ 4] /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0x5e836) [0x7f384a91a836]
[test-X9DAi:14346] [ 5] /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0x5e863) [0x7f384a91a863]
[test-X9DAi:14346] [ 6] /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0x5eaa2) [0x7f384a91aaa2]
[test-X9DAi:14346] [ 7] /usr/lib/x86_64-linux-gnu/libboost_mpi.so.1.54.0(_ZN5boost15throw_exceptionINS_3mpi9exceptionEEEvRKT_+0x1f5) [0x7f384b24f685]
[test-X9DAi:14346] [ 8] /usr/lib/x86_64-linux-gnu/libboost_mpi.so.1.54.0(_ZN5boost3mpi6detail19packed_archive_sendEP19ompi_communicator_tiiRKNS0_15packed_oarchiveE+0xc5) [0x7f384b255a95]
[test-X9DAi:14346] [ 9] ./phase_chain() [0x425092]
[test-X9DAi:14346] [10] ./phase_chain() [0x424ea3]
[test-X9DAi:14346] [11] ./phase_chain() [0x424e3a]
[test-X9DAi:14346] [12] ./phase_chain() [0x424c9c]
[test-X9DAi:14346] [13] ./phase_chain() [0x40cc7d]
[test-X9DAi:14346] [14] ./phase_chain() [0x40bda0]
[test-X9DAi:14346] [15] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5) [0x7f38493feec5]
[test-X9DAi:14346] [16] ./phase_chain() [0x40ba8c]
[test-X9DAi:14346] *** End of error message ***
--------------------------------------------------------------------------
mpirun noticed that process rank 0 with PID 14346 on node test-X9DAi exited on signal 6 (Aborted).
--------------------------------------------------------------------------

error_clang_8ps_run1_XeonE5.log

$ mpirun -n 8 ./phase_chain
[test-X9DAi:14331] *** Process received signal ***
[test-X9DAi:14331] Signal: Segmentation fault (11)
[test-X9DAi:14331] Signal code:  (128)
[test-X9DAi:14331] Failing at address: (nil)
[test-X9DAi:14332] *** Process received signal ***
[test-X9DAi:14332] Signal: Segmentation fault (11)
[test-X9DAi:14332] Signal code: Address not mapped (1)
[test-X9DAi:14332] Failing at address: 0x8
[test-X9DAi:14332] [ 0] /lib/x86_64-linux-gnu/libpthread.so.0(+0x10340) [0x7fa8d65c6340]
[test-X9DAi:14332] [ 1] /usr/lib/x86_64-linux-gnu/libboost_mpi.so.1.54.0(_ZN5boost3mpi6detail19packed_archive_recvEP19ompi_communicator_tiiRNS0_15packed_iarchiveER20ompi_status_public_t+0x49) [0x7fa8d8068cc9]
[test-X9DAi:14332] [ 2] ./phase_chain() [0x425187]
[test-X9DAi:14332] [ 3] ./phase_chain() [0x424ebd]
[test-X9DAi:14332] [ 4] ./phase_chain() [0x424e59]
[test-X9DAi:14332] [ 5] ./phase_chain() [0x424c9c]
[test-X9DAi:14332] [ 6] ./phase_chain() [0x40cc7d]
[test-X9DAi:14332] [ 7] ./phase_chain() [0x40bda0]
[test-X9DAi:14332] [ 8] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5) [0x7fa8d6211ec5]
[test-X9DAi:14332] [ 9] ./phase_chain() [0x40ba8c]
[test-X9DAi:14332] *** End of error message ***
[test-X9DAi:14331] [ 0] /lib/x86_64-linux-gnu/libpthread.so.0(+0x10340) [0x7f2aece6a340]
[test-X9DAi:14331] [ 1] /lib/x86_64-linux-gnu/libc.so.6(+0x98b80) [0x7f2aecb2cb80]
[test-X9DAi:14331] [ 2] /usr/lib/libmpi.so.1(opal_convertor_pack+0x129) [0x7f2aed7d1779]
[test-X9DAi:14331] [ 3] /usr/lib/openmpi/lib/openmpi/mca_btl_sm.so(mca_btl_sm_prepare_src+0x1fd) [0x7f2ae6d6485d]
[test-X9DAi:14331] [ 4] /usr/lib/openmpi/lib/openmpi/mca_pml_ob1.so(mca_pml_ob1_send_request_start_rndv+0x1dc) [0x7f2ae7bcdc9c]
[test-X9DAi:14331] [ 5] /usr/lib/openmpi/lib/openmpi/mca_pml_ob1.so(mca_pml_ob1_send+0xd0c) [0x7f2ae7bc128c]
[test-X9DAi:14331] [ 6] /usr/lib/libmpi.so.1(MPI_Send+0x11c) [0x7f2aed748ebc]
[test-X9DAi:14331] [ 7] /usr/lib/x86_64-linux-gnu/libboost_mpi.so.1.54.0(_ZN5boost3mpi6detail19packed_archive_sendEP19ompi_communicator_tiiRKNS0_15packed_oarchiveE+0x6b) [0x7f2aee90ca3b]
[test-X9DAi:14331] [ 8] ./phase_chain() [0x425092]
[test-X9DAi:14331] [ 9] ./phase_chain() [0x424ea3]
[test-X9DAi:14331] [10] ./phase_chain() [0x424e3a]
[test-X9DAi:14331] [11] ./phase_chain() [0x424c9c]
[test-X9DAi:14331] [12] ./phase_chain() [0x40cc7d]
[test-X9DAi:14331] [13] ./phase_chain() [0x40bda0]
[test-X9DAi:14331] [14] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5) [0x7f2aecab5ec5]
[test-X9DAi:14331] [15] ./phase_chain() [0x40ba8c]
[test-X9DAi:14331] *** End of error message ***
--------------------------------------------------------------------------
mpirun noticed that process rank 1 with PID 14332 on node test-X9DAi exited on signal 11 (Segmentation fault).
--------------------------------------------------------------------------

1st and 2nd output is for 2 processes and 3rd one if of 4 processes.

Files with XeonE5 suffix are error logs from Intel Xeon E5 box.

Files with clang substring were compiled with clang++ as a compiler for mpic++. Others are all g++.