Skip to content

Instantly share code, notes, and snippets.

@relyt0925

relyt0925/gist:b7ce2a25adf83d3887c7fd81e9ac9736

Created July 27, 2024 18:45
Show Gist options
  • Save relyt0925/b7ce2a25adf83d3887c7fd81e9ac9736 to your computer and use it in GitHub Desktop.
Save relyt0925/b7ce2a25adf83d3887c7fd81e9ac9736 to your computer and use it in GitHub Desktop.
Download ZIP
mt_bench eval
Raw
gistfile1.txt
[root@tyler-rhel-newimage instructlab]# /root/ilab model evaluate --benchmark mt_bench --model /var/instructlabbigdisk/instructlab/skillscheckpoints/hf_format/samples_1056/ --judge-model /var/instructlabbigdisk/instructlab/models/prometheus-eval/prometheus-8x7b-v2.0/ --base-model /var/instructlabbigdisk/instructlab/models/ibm-granite/granite-7b-base/ --output-dir /var/instructlabbigdisk/instructlab/evaltracker/skillscheckpoints/samples_1056/ --gpus 8 --backend vllm --enable-serving-output
INFO 2024-07-27 18:36:02,004 numexpr.utils:145: Note: detected 80 virtual cores but NumExpr set to maximum of 64, check "NUMEXPR_MAX_THREADS" environment variable.
INFO 2024-07-27 18:36:02,004 numexpr.utils:148: Note: NumExpr detected 80 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 16.
INFO 2024-07-27 18:36:02,005 numexpr.utils:161: NumExpr defaulting to 16 threads.
Generating answers...
WARNING 2024-07-27 18:36:02,158 instructlab.model.evaluate:288: Based on your hardware configuration, when using vLLM, we recommend setting max-workers between 53.0 and 160.0 for optimal performance
INFO 2024-07-27 18:36:02,670 instructlab.model.backends.backends:315: Trying to connect to model server at http://127.0.0.1:8000/v1
INFO 2024-07-27 18:36:05,181 instructlab.model.backends.vllm:196: vLLM starting up on pid 84 at http://127.0.0.1:60583/v1
INFO 2024-07-27 18:36:05,181 instructlab.model.backends.backends:328: Starting a temporary vLLM server at http://127.0.0.1:60583/v1
INFO 2024-07-27 18:36:05,181 instructlab.model.backends.backends:336: Waiting for the vLLM server to start at http://127.0.0.1:60583/v1, this might take a moment... Attempt: 1/80
INFO 2024-07-27 18:36:09,807 instructlab.model.backends.backends:336: Waiting for the vLLM server to start at http://127.0.0.1:60583/v1, this might take a moment... Attempt: 2/80
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
INFO 2024-07-27 18:36:14,540 instructlab.model.backends.backends:336: Waiting for the vLLM server to start at http://127.0.0.1:60583/v1, this might take a moment... Attempt: 3/80
INFO 2024-07-27 18:36:19,235 instructlab.model.backends.backends:336: Waiting for the vLLM server to start at http://127.0.0.1:60583/v1, this might take a moment... Attempt: 4/80
tyler-rhel-newimage:84:84 [0] NCCL INFO Bootstrap : Using enp8s0:192.168.48.11<0>
tyler-rhel-newimage:84:84 [0] NCCL INFO NET/Plugin : dlerror=libnccl-net.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net.so), using internal implementation
tyler-rhel-newimage:84:84 [0] NCCL INFO cudaDriverVersion 12040
NCCL version 2.20.5+cuda12.4
tyler-rhel-newimage:113:113 [7] NCCL INFO cudaDriverVersion 12040
tyler-rhel-newimage:113:113 [7] NCCL INFO Bootstrap : Using enp8s0:192.168.48.11<0>
tyler-rhel-newimage:113:113 [7] NCCL INFO NET/Plugin : dlerror=libnccl-net.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net.so), using internal implementation
tyler-rhel-newimage:107:107 [1] NCCL INFO cudaDriverVersion 12040
tyler-rhel-newimage:107:107 [1] NCCL INFO Bootstrap : Using enp8s0:192.168.48.11<0>
tyler-rhel-newimage:107:107 [1] NCCL INFO NET/Plugin : dlerror=libnccl-net.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net.so), using internal implementation
tyler-rhel-newimage:112:112 [6] NCCL INFO cudaDriverVersion 12040
tyler-rhel-newimage:112:112 [6] NCCL INFO Bootstrap : Using enp8s0:192.168.48.11<0>
tyler-rhel-newimage:112:112 [6] NCCL INFO NET/Plugin : dlerror=libnccl-net.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net.so), using internal implementation
tyler-rhel-newimage:111:111 [5] NCCL INFO cudaDriverVersion 12040
tyler-rhel-newimage:111:111 [5] NCCL INFO Bootstrap : Using enp8s0:192.168.48.11<0>
tyler-rhel-newimage:111:111 [5] NCCL INFO NET/Plugin : dlerror=libnccl-net.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net.so), using internal implementation
tyler-rhel-newimage:110:110 [4] NCCL INFO cudaDriverVersion 12040
tyler-rhel-newimage:110:110 [4] NCCL INFO Bootstrap : Using enp8s0:192.168.48.11<0>
tyler-rhel-newimage:108:108 [2] NCCL INFO cudaDriverVersion 12040
tyler-rhel-newimage:110:110 [4] NCCL INFO NET/Plugin : dlerror=libnccl-net.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net.so), using internal implementation
tyler-rhel-newimage:108:108 [2] NCCL INFO Bootstrap : Using enp8s0:192.168.48.11<0>
tyler-rhel-newimage:109:109 [3] NCCL INFO cudaDriverVersion 12040
tyler-rhel-newimage:109:109 [3] NCCL INFO Bootstrap : Using enp8s0:192.168.48.11<0>
tyler-rhel-newimage:108:108 [2] NCCL INFO NET/Plugin : dlerror=libnccl-net.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net.so), using internal implementation
tyler-rhel-newimage:109:109 [3] NCCL INFO NET/Plugin : dlerror=libnccl-net.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net.so), using internal implementation
tyler-rhel-newimage:113:113 [7] NCCL INFO NET/IB : No device found.
tyler-rhel-newimage:113:113 [7] NCCL INFO NET/Socket : Using [0]enp8s0:192.168.48.11<0>
tyler-rhel-newimage:113:113 [7] NCCL INFO Using non-device net plugin version 0
tyler-rhel-newimage:113:113 [7] NCCL INFO Using network Socket
tyler-rhel-newimage:111:111 [5] NCCL INFO NET/IB : No device found.
tyler-rhel-newimage:111:111 [5] NCCL INFO NET/Socket : Using [0]enp8s0:192.168.48.11<0>
tyler-rhel-newimage:111:111 [5] NCCL INFO Using non-device net plugin version 0
tyler-rhel-newimage:111:111 [5] NCCL INFO Using network Socket
tyler-rhel-newimage:112:112 [6] NCCL INFO NET/IB : No device found.
tyler-rhel-newimage:112:112 [6] NCCL INFO NET/Socket : Using [0]enp8s0:192.168.48.11<0>
tyler-rhel-newimage:112:112 [6] NCCL INFO Using non-device net plugin version 0
tyler-rhel-newimage:112:112 [6] NCCL INFO Using network Socket
tyler-rhel-newimage:109:109 [3] NCCL INFO NET/IB : No device found.
tyler-rhel-newimage:84:84 [0] NCCL INFO NET/IB : No device found.
tyler-rhel-newimage:108:108 [2] NCCL INFO NET/IB : No device found.
tyler-rhel-newimage:109:109 [3] NCCL INFO NET/Socket : Using [0]enp8s0:192.168.48.11<0>
tyler-rhel-newimage:84:84 [0] NCCL INFO NET/Socket : Using [0]enp8s0:192.168.48.11<0>
tyler-rhel-newimage:108:108 [2] NCCL INFO NET/Socket : Using [0]enp8s0:192.168.48.11<0>
tyler-rhel-newimage:108:108 [2] NCCL INFO Using non-device net plugin version 0
tyler-rhel-newimage:108:108 [2] NCCL INFO Using network Socket
tyler-rhel-newimage:84:84 [0] NCCL INFO Using non-device net plugin version 0
tyler-rhel-newimage:84:84 [0] NCCL INFO Using network Socket
tyler-rhel-newimage:109:109 [3] NCCL INFO Using non-device net plugin version 0
tyler-rhel-newimage:109:109 [3] NCCL INFO Using network Socket
tyler-rhel-newimage:110:110 [4] NCCL INFO NET/IB : No device found.
tyler-rhel-newimage:110:110 [4] NCCL INFO NET/Socket : Using [0]enp8s0:192.168.48.11<0>
tyler-rhel-newimage:110:110 [4] NCCL INFO Using non-device net plugin version 0
tyler-rhel-newimage:110:110 [4] NCCL INFO Using network Socket
tyler-rhel-newimage:107:107 [1] NCCL INFO NET/IB : No device found.
tyler-rhel-newimage:107:107 [1] NCCL INFO NET/Socket : Using [0]enp8s0:192.168.48.11<0>
tyler-rhel-newimage:107:107 [1] NCCL INFO Using non-device net plugin version 0
tyler-rhel-newimage:107:107 [1] NCCL INFO Using network Socket
tyler-rhel-newimage:84:84 [0] NCCL INFO comm 0x56063284b3e0 rank 0 nranks 8 cudaDev 0 nvmlDev 0 busId 8010 commId 0xef46f6c69be18ced - Init START
tyler-rhel-newimage:108:108 [2] NCCL INFO comm 0x560632842260 rank 2 nranks 8 cudaDev 2 nvmlDev 2 busId a030 commId 0xef46f6c69be18ced - Init START
tyler-rhel-newimage:112:112 [6] NCCL INFO comm 0x560632844000 rank 6 nranks 8 cudaDev 6 nvmlDev 6 busId e070 commId 0xef46f6c69be18ced - Init START
tyler-rhel-newimage:107:107 [1] NCCL INFO comm 0x560632840d50 rank 1 nranks 8 cudaDev 1 nvmlDev 1 busId 8020 commId 0xef46f6c69be18ced - Init START
tyler-rhel-newimage:111:111 [5] NCCL INFO comm 0x560632845230 rank 5 nranks 8 cudaDev 5 nvmlDev 5 busId c060 commId 0xef46f6c69be18ced - Init START
tyler-rhel-newimage:110:110 [4] NCCL INFO comm 0x560632843570 rank 4 nranks 8 cudaDev 4 nvmlDev 4 busId c050 commId 0xef46f6c69be18ced - Init START
tyler-rhel-newimage:109:109 [3] NCCL INFO comm 0x560632843410 rank 3 nranks 8 cudaDev 3 nvmlDev 3 busId a040 commId 0xef46f6c69be18ced - Init START
tyler-rhel-newimage:113:113 [7] NCCL INFO comm 0x560632844260 rank 7 nranks 8 cudaDev 7 nvmlDev 7 busId e080 commId 0xef46f6c69be18ced - Init START
INFO 2024-07-27 18:36:23,942 instructlab.model.backends.backends:336: Waiting for the vLLM server to start at http://127.0.0.1:60583/v1, this might take a moment... Attempt: 5/80
tyler-rhel-newimage:113:113 [7] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0.
tyler-rhel-newimage:107:107 [1] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0.
tyler-rhel-newimage:84:84 [0] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0.
tyler-rhel-newimage:111:111 [5] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0.
tyler-rhel-newimage:109:109 [3] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0.
tyler-rhel-newimage:110:110 [4] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0.
tyler-rhel-newimage:108:108 [2] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0.
tyler-rhel-newimage:84:84 [0] NCCL INFO Setting affinity for GPU 0 to ff,ffffffff
tyler-rhel-newimage:113:113 [7] NCCL INFO Setting affinity for GPU 7 to ffff,ffffff00,00000000
tyler-rhel-newimage:113:113 [7] NCCL INFO NVLS multicast support is not available on dev 7
tyler-rhel-newimage:84:84 [0] NCCL INFO NVLS multicast support is not available on dev 0
tyler-rhel-newimage:112:112 [6] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0.
tyler-rhel-newimage:107:107 [1] NCCL INFO Setting affinity for GPU 1 to ff,ffffffff
tyler-rhel-newimage:107:107 [1] NCCL INFO NVLS multicast support is not available on dev 1
tyler-rhel-newimage:109:109 [3] NCCL INFO Setting affinity for GPU 3 to ff,ffffffff
tyler-rhel-newimage:110:110 [4] NCCL INFO Setting affinity for GPU 4 to ffff,ffffff00,00000000
tyler-rhel-newimage:110:110 [4] NCCL INFO NVLS multicast support is not available on dev 4
tyler-rhel-newimage:109:109 [3] NCCL INFO NVLS multicast support is not available on dev 3
tyler-rhel-newimage:111:111 [5] NCCL INFO Setting affinity for GPU 5 to ffff,ffffff00,00000000
tyler-rhel-newimage:111:111 [5] NCCL INFO NVLS multicast support is not available on dev 5
tyler-rhel-newimage:108:108 [2] NCCL INFO Setting affinity for GPU 2 to ff,ffffffff
tyler-rhel-newimage:108:108 [2] NCCL INFO NVLS multicast support is not available on dev 2
tyler-rhel-newimage:112:112 [6] NCCL INFO Setting affinity for GPU 6 to ffff,ffffff00,00000000
tyler-rhel-newimage:112:112 [6] NCCL INFO NVLS multicast support is not available on dev 6
tyler-rhel-newimage:112:112 [6] NCCL INFO comm 0x560632844000 rank 6 nRanks 8 nNodes 1 localRanks 8 localRank 6 MNNVL 0
tyler-rhel-newimage:113:113 [7] NCCL INFO comm 0x560632844260 rank 7 nRanks 8 nNodes 1 localRanks 8 localRank 7 MNNVL 0
tyler-rhel-newimage:110:110 [4] NCCL INFO comm 0x560632843570 rank 4 nRanks 8 nNodes 1 localRanks 8 localRank 4 MNNVL 0
tyler-rhel-newimage:111:111 [5] NCCL INFO comm 0x560632845230 rank 5 nRanks 8 nNodes 1 localRanks 8 localRank 5 MNNVL 0
tyler-rhel-newimage:109:109 [3] NCCL INFO comm 0x560632843410 rank 3 nRanks 8 nNodes 1 localRanks 8 localRank 3 MNNVL 0
tyler-rhel-newimage:108:108 [2] NCCL INFO comm 0x560632842260 rank 2 nRanks 8 nNodes 1 localRanks 8 localRank 2 MNNVL 0
tyler-rhel-newimage:107:107 [1] NCCL INFO comm 0x560632840d50 rank 1 nRanks 8 nNodes 1 localRanks 8 localRank 1 MNNVL 0
tyler-rhel-newimage:84:84 [0] NCCL INFO comm 0x56063284b3e0 rank 0 nRanks 8 nNodes 1 localRanks 8 localRank 0 MNNVL 0
tyler-rhel-newimage:112:112 [6] NCCL INFO Trees [0] 7/-1/-1->6->5 [1] 7/-1/-1->6->5 [2] 7/-1/-1->6->5 [3] 7/-1/-1->6->5 [4] 7/-1/-1->6->5 [5] 7/-1/-1->6->5 [6] 7/-1/-1->6->5 [7] 7/-1/-1->6->5 [8] 7/-1/-1->6->5 [9] 7/-1/-1->6->5 [10] 7/-1/-1->6->5 [11] 7/-1/-1->6->5 [12] 7/-1/-1->6->5 [13] 7/-1/-1->6->5 [14] 7/-1/-1->6->5 [15] 7/-1/-1->6->5 [16] 7/-1/-1->6->5 [17] 7/-1/-1->6->5 [18] 7/-1/-1->6->5 [19] 7/-1/-1->6->5 [20] 7/-1/-1->6->5 [21] 7/-1/-1->6->5 [22] 7/-1/-1->6->5 [23] 7/-1/-1->6->5
tyler-rhel-newimage:110:110 [4] NCCL INFO Trees [0] 5/-1/-1->4->3 [1] 5/-1/-1->4->3 [2] 5/-1/-1->4->3 [3] 5/-1/-1->4->3 [4] 5/-1/-1->4->3 [5] 5/-1/-1->4->3 [6] 5/-1/-1->4->3 [7] 5/-1/-1->4->3 [8] 5/-1/-1->4->3 [9] 5/-1/-1->4->3 [10] 5/-1/-1->4->3 [11] 5/-1/-1->4->3 [12] 5/-1/-1->4->3 [13] 5/-1/-1->4->3 [14] 5/-1/-1->4->3 [15] 5/-1/-1->4->3 [16] 5/-1/-1->4->3 [17] 5/-1/-1->4->3 [18] 5/-1/-1->4->3 [19] 5/-1/-1->4->3 [20] 5/-1/-1->4->3 [21] 5/-1/-1->4->3 [22] 5/-1/-1->4->3 [23] 5/-1/-1->4->3
tyler-rhel-newimage:111:111 [5] NCCL INFO Trees [0] 6/-1/-1->5->4 [1] 6/-1/-1->5->4 [2] 6/-1/-1->5->4 [3] 6/-1/-1->5->4 [4] 6/-1/-1->5->4 [5] 6/-1/-1->5->4 [6] 6/-1/-1->5->4 [7] 6/-1/-1->5->4 [8] 6/-1/-1->5->4 [9] 6/-1/-1->5->4 [10] 6/-1/-1->5->4 [11] 6/-1/-1->5->4 [12] 6/-1/-1->5->4 [13] 6/-1/-1->5->4 [14] 6/-1/-1->5->4 [15] 6/-1/-1->5->4 [16] 6/-1/-1->5->4 [17] 6/-1/-1->5->4 [18] 6/-1/-1->5->4 [19] 6/-1/-1->5->4 [20] 6/-1/-1->5->4 [21] 6/-1/-1->5->4 [22] 6/-1/-1->5->4 [23] 6/-1/-1->5->4
tyler-rhel-newimage:84:84 [0] NCCL INFO Channel 00/24 : 0 1 2 3 4 5 6 7
tyler-rhel-newimage:110:110 [4] NCCL INFO P2P Chunksize set to 524288
tyler-rhel-newimage:112:112 [6] NCCL INFO P2P Chunksize set to 524288
tyler-rhel-newimage:113:113 [7] NCCL INFO Trees [0] -1/-1/-1->7->6 [1] -1/-1/-1->7->6 [2] -1/-1/-1->7->6 [3] -1/-1/-1->7->6 [4] -1/-1/-1->7->6 [5] -1/-1/-1->7->6 [6] -1/-1/-1->7->6 [7] -1/-1/-1->7->6 [8] -1/-1/-1->7->6 [9] -1/-1/-1->7->6 [10] -1/-1/-1->7->6 [11] -1/-1/-1->7->6 [12] -1/-1/-1->7->6 [13] -1/-1/-1->7->6 [14] -1/-1/-1->7->6 [15] -1/-1/-1->7->6 [16] -1/-1/-1->7->6 [17] -1/-1/-1->7->6 [18] -1/-1/-1->7->6 [19] -1/-1/-1->7->6 [20] -1/-1/-1->7->6 [21] -1/-1/-1->7->6 [22] -1/-1/-1->7->6 [23] -1/-1/-1->7->6
tyler-rhel-newimage:111:111 [5] NCCL INFO P2P Chunksize set to 524288
tyler-rhel-newimage:109:109 [3] NCCL INFO Trees [0] 4/-1/-1->3->2 [1] 4/-1/-1->3->2 [2] 4/-1/-1->3->2 [3] 4/-1/-1->3->2 [4] 4/-1/-1->3->2 [5] 4/-1/-1->3->2 [6] 4/-1/-1->3->2 [7] 4/-1/-1->3->2 [8] 4/-1/-1->3->2 [9] 4/-1/-1->3->2 [10] 4/-1/-1->3->2 [11] 4/-1/-1->3->2 [12] 4/-1/-1->3->2 [13] 4/-1/-1->3->2 [14] 4/-1/-1->3->2 [15] 4/-1/-1->3->2 [16] 4/-1/-1->3->2 [17] 4/-1/-1->3->2 [18] 4/-1/-1->3->2 [19] 4/-1/-1->3->2 [20] 4/-1/-1->3->2 [21] 4/-1/-1->3->2 [22] 4/-1/-1->3->2 [23] 4/-1/-1->3->2
tyler-rhel-newimage:108:108 [2] NCCL INFO Trees [0] 3/-1/-1->2->1 [1] 3/-1/-1->2->1 [2] 3/-1/-1->2->1 [3] 3/-1/-1->2->1 [4] 3/-1/-1->2->1 [5] 3/-1/-1->2->1 [6] 3/-1/-1->2->1 [7] 3/-1/-1->2->1 [8] 3/-1/-1->2->1 [9] 3/-1/-1->2->1 [10] 3/-1/-1->2->1 [11] 3/-1/-1->2->1 [12] 3/-1/-1->2->1 [13] 3/-1/-1->2->1 [14] 3/-1/-1->2->1 [15] 3/-1/-1->2->1 [16] 3/-1/-1->2->1 [17] 3/-1/-1->2->1 [18] 3/-1/-1->2->1 [19] 3/-1/-1->2->1 [20] 3/-1/-1->2->1 [21] 3/-1/-1->2->1 [22] 3/-1/-1->2->1 [23] 3/-1/-1->2->1
tyler-rhel-newimage:84:84 [0] NCCL INFO Channel 01/24 : 0 1 2 3 4 5 6 7
tyler-rhel-newimage:107:107 [1] NCCL INFO Trees [0] 2/-1/-1->1->0 [1] 2/-1/-1->1->0 [2] 2/-1/-1->1->0 [3] 2/-1/-1->1->0 [4] 2/-1/-1->1->0 [5] 2/-1/-1->1->0 [6] 2/-1/-1->1->0 [7] 2/-1/-1->1->0 [8] 2/-1/-1->1->0 [9] 2/-1/-1->1->0 [10] 2/-1/-1->1->0 [11] 2/-1/-1->1->0 [12] 2/-1/-1->1->0 [13] 2/-1/-1->1->0 [14] 2/-1/-1->1->0 [15] 2/-1/-1->1->0 [16] 2/-1/-1->1->0 [17] 2/-1/-1->1->0 [18] 2/-1/-1->1->0 [19] 2/-1/-1->1->0 [20] 2/-1/-1->1->0 [21] 2/-1/-1->1->0 [22] 2/-1/-1->1->0 [23] 2/-1/-1->1->0
tyler-rhel-newimage:113:113 [7] NCCL INFO P2P Chunksize set to 524288
tyler-rhel-newimage:109:109 [3] NCCL INFO P2P Chunksize set to 524288
tyler-rhel-newimage:108:108 [2] NCCL INFO P2P Chunksize set to 524288
tyler-rhel-newimage:84:84 [0] NCCL INFO Channel 02/24 : 0 1 2 3 4 5 6 7
tyler-rhel-newimage:107:107 [1] NCCL INFO P2P Chunksize set to 524288
tyler-rhel-newimage:84:84 [0] NCCL INFO Channel 03/24 : 0 1 2 3 4 5 6 7
tyler-rhel-newimage:84:84 [0] NCCL INFO Channel 04/24 : 0 1 2 3 4 5 6 7
tyler-rhel-newimage:84:84 [0] NCCL INFO Channel 05/24 : 0 1 2 3 4 5 6 7
tyler-rhel-newimage:84:84 [0] NCCL INFO Channel 06/24 : 0 1 2 3 4 5 6 7
tyler-rhel-newimage:84:84 [0] NCCL INFO Channel 07/24 : 0 1 2 3 4 5 6 7
tyler-rhel-newimage:84:84 [0] NCCL INFO Channel 08/24 : 0 1 2 3 4 5 6 7
tyler-rhel-newimage:84:84 [0] NCCL INFO Channel 09/24 : 0 1 2 3 4 5 6 7
tyler-rhel-newimage:84:84 [0] NCCL INFO Channel 10/24 : 0 1 2 3 4 5 6 7
tyler-rhel-newimage:84:84 [0] NCCL INFO Channel 11/24 : 0 1 2 3 4 5 6 7
tyler-rhel-newimage:84:84 [0] NCCL INFO Channel 12/24 : 0 1 2 3 4 5 6 7
tyler-rhel-newimage:84:84 [0] NCCL INFO Channel 13/24 : 0 1 2 3 4 5 6 7
tyler-rhel-newimage:84:84 [0] NCCL INFO Channel 14/24 : 0 1 2 3 4 5 6 7
tyler-rhel-newimage:84:84 [0] NCCL INFO Channel 15/24 : 0 1 2 3 4 5 6 7
tyler-rhel-newimage:84:84 [0] NCCL INFO Channel 16/24 : 0 1 2 3 4 5 6 7
tyler-rhel-newimage:84:84 [0] NCCL INFO Channel 17/24 : 0 1 2 3 4 5 6 7
tyler-rhel-newimage:84:84 [0] NCCL INFO Channel 18/24 : 0 1 2 3 4 5 6 7
tyler-rhel-newimage:84:84 [0] NCCL INFO Channel 19/24 : 0 1 2 3 4 5 6 7
tyler-rhel-newimage:84:84 [0] NCCL INFO Channel 20/24 : 0 1 2 3 4 5 6 7
tyler-rhel-newimage:84:84 [0] NCCL INFO Channel 21/24 : 0 1 2 3 4 5 6 7
tyler-rhel-newimage:84:84 [0] NCCL INFO Channel 22/24 : 0 1 2 3 4 5 6 7
tyler-rhel-newimage:84:84 [0] NCCL INFO Channel 23/24 : 0 1 2 3 4 5 6 7
tyler-rhel-newimage:84:84 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1 [2] 1/-1/-1->0->-1 [3] 1/-1/-1->0->-1 [4] 1/-1/-1->0->-1 [5] 1/-1/-1->0->-1 [6] 1/-1/-1->0->-1 [7] 1/-1/-1->0->-1 [8] 1/-1/-1->0->-1 [9] 1/-1/-1->0->-1 [10] 1/-1/-1->0->-1 [11] 1/-1/-1->0->-1 [12] 1/-1/-1->0->-1 [13] 1/-1/-1->0->-1 [14] 1/-1/-1->0->-1 [15] 1/-1/-1->0->-1 [16] 1/-1/-1->0->-1 [17] 1/-1/-1->0->-1 [18] 1/-1/-1->0->-1 [19] 1/-1/-1->0->-1 [20] 1/-1/-1->0->-1 [21] 1/-1/-1->0->-1 [22] 1/-1/-1->0->-1 [23] 1/-1/-1->0->-1
tyler-rhel-newimage:84:84 [0] NCCL INFO P2P Chunksize set to 524288
tyler-rhel-newimage:113:113 [7] NCCL INFO Channel 00/0 : 7[7] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:112:112 [6] NCCL INFO Channel 00/0 : 6[6] -> 7[7] via P2P/IPC/read
tyler-rhel-newimage:110:110 [4] NCCL INFO Channel 00/0 : 4[4] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:111:111 [5] NCCL INFO Channel 00/0 : 5[5] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:109:109 [3] NCCL INFO Channel 00/0 : 3[3] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:107:107 [1] NCCL INFO Channel 00/0 : 1[1] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:113:113 [7] NCCL INFO Channel 01/0 : 7[7] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:112:112 [6] NCCL INFO Channel 01/0 : 6[6] -> 7[7] via P2P/IPC/read
tyler-rhel-newimage:110:110 [4] NCCL INFO Channel 01/0 : 4[4] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:111:111 [5] NCCL INFO Channel 01/0 : 5[5] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:108:108 [2] NCCL INFO Channel 00/0 : 2[2] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:84:84 [0] NCCL INFO Channel 00/0 : 0[0] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:109:109 [3] NCCL INFO Channel 01/0 : 3[3] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:107:107 [1] NCCL INFO Channel 01/0 : 1[1] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:113:113 [7] NCCL INFO Channel 02/0 : 7[7] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:112:112 [6] NCCL INFO Channel 02/0 : 6[6] -> 7[7] via P2P/IPC/read
tyler-rhel-newimage:110:110 [4] NCCL INFO Channel 02/0 : 4[4] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:111:111 [5] NCCL INFO Channel 02/0 : 5[5] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:108:108 [2] NCCL INFO Channel 01/0 : 2[2] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:84:84 [0] NCCL INFO Channel 01/0 : 0[0] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:109:109 [3] NCCL INFO Channel 02/0 : 3[3] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:107:107 [1] NCCL INFO Channel 02/0 : 1[1] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:113:113 [7] NCCL INFO Channel 03/0 : 7[7] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:112:112 [6] NCCL INFO Channel 03/0 : 6[6] -> 7[7] via P2P/IPC/read
tyler-rhel-newimage:110:110 [4] NCCL INFO Channel 03/0 : 4[4] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:111:111 [5] NCCL INFO Channel 03/0 : 5[5] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:108:108 [2] NCCL INFO Channel 02/0 : 2[2] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:107:107 [1] NCCL INFO Channel 03/0 : 1[1] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:109:109 [3] NCCL INFO Channel 03/0 : 3[3] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:84:84 [0] NCCL INFO Channel 02/0 : 0[0] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:113:113 [7] NCCL INFO Channel 04/0 : 7[7] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:112:112 [6] NCCL INFO Channel 04/0 : 6[6] -> 7[7] via P2P/IPC/read
tyler-rhel-newimage:110:110 [4] NCCL INFO Channel 04/0 : 4[4] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:111:111 [5] NCCL INFO Channel 04/0 : 5[5] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:108:108 [2] NCCL INFO Channel 03/0 : 2[2] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:109:109 [3] NCCL INFO Channel 04/0 : 3[3] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:107:107 [1] NCCL INFO Channel 04/0 : 1[1] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:84:84 [0] NCCL INFO Channel 03/0 : 0[0] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:113:113 [7] NCCL INFO Channel 05/0 : 7[7] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:112:112 [6] NCCL INFO Channel 05/0 : 6[6] -> 7[7] via P2P/IPC/read
tyler-rhel-newimage:110:110 [4] NCCL INFO Channel 05/0 : 4[4] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:111:111 [5] NCCL INFO Channel 05/0 : 5[5] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:109:109 [3] NCCL INFO Channel 05/0 : 3[3] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:108:108 [2] NCCL INFO Channel 04/0 : 2[2] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:107:107 [1] NCCL INFO Channel 05/0 : 1[1] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:84:84 [0] NCCL INFO Channel 04/0 : 0[0] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:113:113 [7] NCCL INFO Channel 06/0 : 7[7] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:112:112 [6] NCCL INFO Channel 06/0 : 6[6] -> 7[7] via P2P/IPC/read
tyler-rhel-newimage:110:110 [4] NCCL INFO Channel 06/0 : 4[4] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:111:111 [5] NCCL INFO Channel 06/0 : 5[5] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:109:109 [3] NCCL INFO Channel 06/0 : 3[3] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:107:107 [1] NCCL INFO Channel 06/0 : 1[1] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:108:108 [2] NCCL INFO Channel 05/0 : 2[2] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:84:84 [0] NCCL INFO Channel 05/0 : 0[0] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:113:113 [7] NCCL INFO Channel 07/0 : 7[7] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:112:112 [6] NCCL INFO Channel 07/0 : 6[6] -> 7[7] via P2P/IPC/read
tyler-rhel-newimage:110:110 [4] NCCL INFO Channel 07/0 : 4[4] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:111:111 [5] NCCL INFO Channel 07/0 : 5[5] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:109:109 [3] NCCL INFO Channel 07/0 : 3[3] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:108:108 [2] NCCL INFO Channel 06/0 : 2[2] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:107:107 [1] NCCL INFO Channel 07/0 : 1[1] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:84:84 [0] NCCL INFO Channel 06/0 : 0[0] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:113:113 [7] NCCL INFO Channel 08/0 : 7[7] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:112:112 [6] NCCL INFO Channel 08/0 : 6[6] -> 7[7] via P2P/IPC/read
tyler-rhel-newimage:110:110 [4] NCCL INFO Channel 08/0 : 4[4] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:109:109 [3] NCCL INFO Channel 08/0 : 3[3] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:111:111 [5] NCCL INFO Channel 08/0 : 5[5] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:108:108 [2] NCCL INFO Channel 07/0 : 2[2] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:107:107 [1] NCCL INFO Channel 08/0 : 1[1] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:84:84 [0] NCCL INFO Channel 07/0 : 0[0] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:113:113 [7] NCCL INFO Channel 09/0 : 7[7] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:112:112 [6] NCCL INFO Channel 09/0 : 6[6] -> 7[7] via P2P/IPC/read
tyler-rhel-newimage:110:110 [4] NCCL INFO Channel 09/0 : 4[4] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:109:109 [3] NCCL INFO Channel 09/0 : 3[3] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:111:111 [5] NCCL INFO Channel 09/0 : 5[5] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:108:108 [2] NCCL INFO Channel 08/0 : 2[2] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:107:107 [1] NCCL INFO Channel 09/0 : 1[1] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:84:84 [0] NCCL INFO Channel 08/0 : 0[0] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:113:113 [7] NCCL INFO Channel 10/0 : 7[7] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:110:110 [4] NCCL INFO Channel 10/0 : 4[4] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:112:112 [6] NCCL INFO Channel 10/0 : 6[6] -> 7[7] via P2P/IPC/read
tyler-rhel-newimage:109:109 [3] NCCL INFO Channel 10/0 : 3[3] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:111:111 [5] NCCL INFO Channel 10/0 : 5[5] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:108:108 [2] NCCL INFO Channel 09/0 : 2[2] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:107:107 [1] NCCL INFO Channel 10/0 : 1[1] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:84:84 [0] NCCL INFO Channel 09/0 : 0[0] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:113:113 [7] NCCL INFO Channel 11/0 : 7[7] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:110:110 [4] NCCL INFO Channel 11/0 : 4[4] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:112:112 [6] NCCL INFO Channel 11/0 : 6[6] -> 7[7] via P2P/IPC/read
tyler-rhel-newimage:109:109 [3] NCCL INFO Channel 11/0 : 3[3] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:111:111 [5] NCCL INFO Channel 11/0 : 5[5] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:108:108 [2] NCCL INFO Channel 10/0 : 2[2] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:107:107 [1] NCCL INFO Channel 11/0 : 1[1] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:84:84 [0] NCCL INFO Channel 10/0 : 0[0] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:113:113 [7] NCCL INFO Channel 12/0 : 7[7] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:110:110 [4] NCCL INFO Channel 12/0 : 4[4] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:112:112 [6] NCCL INFO Channel 12/0 : 6[6] -> 7[7] via P2P/IPC/read
tyler-rhel-newimage:109:109 [3] NCCL INFO Channel 12/0 : 3[3] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:111:111 [5] NCCL INFO Channel 12/0 : 5[5] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:108:108 [2] NCCL INFO Channel 11/0 : 2[2] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:107:107 [1] NCCL INFO Channel 12/0 : 1[1] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:84:84 [0] NCCL INFO Channel 11/0 : 0[0] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:113:113 [7] NCCL INFO Channel 13/0 : 7[7] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:110:110 [4] NCCL INFO Channel 13/0 : 4[4] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:112:112 [6] NCCL INFO Channel 13/0 : 6[6] -> 7[7] via P2P/IPC/read
tyler-rhel-newimage:109:109 [3] NCCL INFO Channel 13/0 : 3[3] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:111:111 [5] NCCL INFO Channel 13/0 : 5[5] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:108:108 [2] NCCL INFO Channel 12/0 : 2[2] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:84:84 [0] NCCL INFO Channel 12/0 : 0[0] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:107:107 [1] NCCL INFO Channel 13/0 : 1[1] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:113:113 [7] NCCL INFO Channel 14/0 : 7[7] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:110:110 [4] NCCL INFO Channel 14/0 : 4[4] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:112:112 [6] NCCL INFO Channel 14/0 : 6[6] -> 7[7] via P2P/IPC/read
tyler-rhel-newimage:109:109 [3] NCCL INFO Channel 14/0 : 3[3] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:111:111 [5] NCCL INFO Channel 14/0 : 5[5] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:108:108 [2] NCCL INFO Channel 13/0 : 2[2] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:84:84 [0] NCCL INFO Channel 13/0 : 0[0] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:107:107 [1] NCCL INFO Channel 14/0 : 1[1] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:113:113 [7] NCCL INFO Channel 15/0 : 7[7] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:110:110 [4] NCCL INFO Channel 15/0 : 4[4] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:112:112 [6] NCCL INFO Channel 15/0 : 6[6] -> 7[7] via P2P/IPC/read
tyler-rhel-newimage:109:109 [3] NCCL INFO Channel 15/0 : 3[3] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:111:111 [5] NCCL INFO Channel 15/0 : 5[5] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:108:108 [2] NCCL INFO Channel 14/0 : 2[2] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:84:84 [0] NCCL INFO Channel 14/0 : 0[0] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:107:107 [1] NCCL INFO Channel 15/0 : 1[1] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:113:113 [7] NCCL INFO Channel 16/0 : 7[7] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:110:110 [4] NCCL INFO Channel 16/0 : 4[4] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:112:112 [6] NCCL INFO Channel 16/0 : 6[6] -> 7[7] via P2P/IPC/read
tyler-rhel-newimage:109:109 [3] NCCL INFO Channel 16/0 : 3[3] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:111:111 [5] NCCL INFO Channel 16/0 : 5[5] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:108:108 [2] NCCL INFO Channel 15/0 : 2[2] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:84:84 [0] NCCL INFO Channel 15/0 : 0[0] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:107:107 [1] NCCL INFO Channel 16/0 : 1[1] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:113:113 [7] NCCL INFO Channel 17/0 : 7[7] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:110:110 [4] NCCL INFO Channel 17/0 : 4[4] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:112:112 [6] NCCL INFO Channel 17/0 : 6[6] -> 7[7] via P2P/IPC/read
tyler-rhel-newimage:109:109 [3] NCCL INFO Channel 17/0 : 3[3] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:111:111 [5] NCCL INFO Channel 17/0 : 5[5] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:108:108 [2] NCCL INFO Channel 16/0 : 2[2] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:84:84 [0] NCCL INFO Channel 16/0 : 0[0] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:107:107 [1] NCCL INFO Channel 17/0 : 1[1] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:113:113 [7] NCCL INFO Channel 18/0 : 7[7] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:110:110 [4] NCCL INFO Channel 18/0 : 4[4] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:112:112 [6] NCCL INFO Channel 18/0 : 6[6] -> 7[7] via P2P/IPC/read
tyler-rhel-newimage:109:109 [3] NCCL INFO Channel 18/0 : 3[3] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:111:111 [5] NCCL INFO Channel 18/0 : 5[5] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:108:108 [2] NCCL INFO Channel 17/0 : 2[2] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:84:84 [0] NCCL INFO Channel 17/0 : 0[0] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:107:107 [1] NCCL INFO Channel 18/0 : 1[1] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:113:113 [7] NCCL INFO Channel 19/0 : 7[7] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:110:110 [4] NCCL INFO Channel 19/0 : 4[4] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:112:112 [6] NCCL INFO Channel 19/0 : 6[6] -> 7[7] via P2P/IPC/read
tyler-rhel-newimage:109:109 [3] NCCL INFO Channel 19/0 : 3[3] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:111:111 [5] NCCL INFO Channel 19/0 : 5[5] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:108:108 [2] NCCL INFO Channel 18/0 : 2[2] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:84:84 [0] NCCL INFO Channel 18/0 : 0[0] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:107:107 [1] NCCL INFO Channel 19/0 : 1[1] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:113:113 [7] NCCL INFO Channel 20/0 : 7[7] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:110:110 [4] NCCL INFO Channel 20/0 : 4[4] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:109:109 [3] NCCL INFO Channel 20/0 : 3[3] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:112:112 [6] NCCL INFO Channel 20/0 : 6[6] -> 7[7] via P2P/IPC/read
tyler-rhel-newimage:111:111 [5] NCCL INFO Channel 20/0 : 5[5] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:108:108 [2] NCCL INFO Channel 19/0 : 2[2] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:84:84 [0] NCCL INFO Channel 19/0 : 0[0] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:107:107 [1] NCCL INFO Channel 20/0 : 1[1] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:113:113 [7] NCCL INFO Channel 21/0 : 7[7] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:110:110 [4] NCCL INFO Channel 21/0 : 4[4] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:109:109 [3] NCCL INFO Channel 21/0 : 3[3] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:112:112 [6] NCCL INFO Channel 21/0 : 6[6] -> 7[7] via P2P/IPC/read
tyler-rhel-newimage:111:111 [5] NCCL INFO Channel 21/0 : 5[5] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:108:108 [2] NCCL INFO Channel 20/0 : 2[2] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:84:84 [0] NCCL INFO Channel 20/0 : 0[0] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:107:107 [1] NCCL INFO Channel 21/0 : 1[1] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:113:113 [7] NCCL INFO Channel 22/0 : 7[7] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:110:110 [4] NCCL INFO Channel 22/0 : 4[4] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:109:109 [3] NCCL INFO Channel 22/0 : 3[3] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:112:112 [6] NCCL INFO Channel 22/0 : 6[6] -> 7[7] via P2P/IPC/read
tyler-rhel-newimage:111:111 [5] NCCL INFO Channel 22/0 : 5[5] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:108:108 [2] NCCL INFO Channel 21/0 : 2[2] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:84:84 [0] NCCL INFO Channel 21/0 : 0[0] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:107:107 [1] NCCL INFO Channel 22/0 : 1[1] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:109:109 [3] NCCL INFO Channel 23/0 : 3[3] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:113:113 [7] NCCL INFO Channel 23/0 : 7[7] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:110:110 [4] NCCL INFO Channel 23/0 : 4[4] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:112:112 [6] NCCL INFO Channel 23/0 : 6[6] -> 7[7] via P2P/IPC/read
tyler-rhel-newimage:111:111 [5] NCCL INFO Channel 23/0 : 5[5] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:108:108 [2] NCCL INFO Channel 22/0 : 2[2] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:84:84 [0] NCCL INFO Channel 22/0 : 0[0] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:107:107 [1] NCCL INFO Channel 23/0 : 1[1] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:108:108 [2] NCCL INFO Channel 23/0 : 2[2] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:84:84 [0] NCCL INFO Channel 23/0 : 0[0] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:112:112 [6] NCCL INFO Connected all rings
tyler-rhel-newimage:110:110 [4] NCCL INFO Connected all rings
tyler-rhel-newimage:111:111 [5] NCCL INFO Connected all rings
tyler-rhel-newimage:109:109 [3] NCCL INFO Connected all rings
tyler-rhel-newimage:107:107 [1] NCCL INFO Connected all rings
tyler-rhel-newimage:108:108 [2] NCCL INFO Connected all rings
tyler-rhel-newimage:113:113 [7] NCCL INFO Connected all rings
tyler-rhel-newimage:84:84 [0] NCCL INFO Connected all rings
tyler-rhel-newimage:113:113 [7] NCCL INFO Channel 00/0 : 7[7] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:113:113 [7] NCCL INFO Channel 01/0 : 7[7] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:113:113 [7] NCCL INFO Channel 02/0 : 7[7] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:113:113 [7] NCCL INFO Channel 03/0 : 7[7] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:113:113 [7] NCCL INFO Channel 04/0 : 7[7] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:113:113 [7] NCCL INFO Channel 05/0 : 7[7] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:113:113 [7] NCCL INFO Channel 06/0 : 7[7] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:113:113 [7] NCCL INFO Channel 07/0 : 7[7] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:113:113 [7] NCCL INFO Channel 08/0 : 7[7] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:113:113 [7] NCCL INFO Channel 09/0 : 7[7] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:113:113 [7] NCCL INFO Channel 10/0 : 7[7] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:113:113 [7] NCCL INFO Channel 11/0 : 7[7] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:113:113 [7] NCCL INFO Channel 12/0 : 7[7] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:113:113 [7] NCCL INFO Channel 13/0 : 7[7] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:113:113 [7] NCCL INFO Channel 14/0 : 7[7] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:113:113 [7] NCCL INFO Channel 15/0 : 7[7] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:113:113 [7] NCCL INFO Channel 16/0 : 7[7] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:113:113 [7] NCCL INFO Channel 17/0 : 7[7] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:113:113 [7] NCCL INFO Channel 18/0 : 7[7] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:113:113 [7] NCCL INFO Channel 19/0 : 7[7] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:113:113 [7] NCCL INFO Channel 20/0 : 7[7] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:113:113 [7] NCCL INFO Channel 21/0 : 7[7] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:113:113 [7] NCCL INFO Channel 22/0 : 7[7] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:113:113 [7] NCCL INFO Channel 23/0 : 7[7] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:111:111 [5] NCCL INFO Channel 00/0 : 5[5] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:110:110 [4] NCCL INFO Channel 00/0 : 4[4] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:107:107 [1] NCCL INFO Channel 00/0 : 1[1] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:109:109 [3] NCCL INFO Channel 00/0 : 3[3] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:112:112 [6] NCCL INFO Channel 00/0 : 6[6] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:108:108 [2] NCCL INFO Channel 00/0 : 2[2] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:111:111 [5] NCCL INFO Channel 01/0 : 5[5] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:110:110 [4] NCCL INFO Channel 01/0 : 4[4] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:107:107 [1] NCCL INFO Channel 01/0 : 1[1] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:109:109 [3] NCCL INFO Channel 01/0 : 3[3] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:112:112 [6] NCCL INFO Channel 01/0 : 6[6] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:108:108 [2] NCCL INFO Channel 01/0 : 2[2] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:110:110 [4] NCCL INFO Channel 02/0 : 4[4] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:111:111 [5] NCCL INFO Channel 02/0 : 5[5] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:107:107 [1] NCCL INFO Channel 02/0 : 1[1] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:109:109 [3] NCCL INFO Channel 02/0 : 3[3] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:108:108 [2] NCCL INFO Channel 02/0 : 2[2] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:112:112 [6] NCCL INFO Channel 02/0 : 6[6] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:110:110 [4] NCCL INFO Channel 03/0 : 4[4] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:111:111 [5] NCCL INFO Channel 03/0 : 5[5] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:107:107 [1] NCCL INFO Channel 03/0 : 1[1] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:109:109 [3] NCCL INFO Channel 03/0 : 3[3] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:108:108 [2] NCCL INFO Channel 03/0 : 2[2] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:112:112 [6] NCCL INFO Channel 03/0 : 6[6] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:111:111 [5] NCCL INFO Channel 04/0 : 5[5] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:110:110 [4] NCCL INFO Channel 04/0 : 4[4] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:107:107 [1] NCCL INFO Channel 04/0 : 1[1] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:109:109 [3] NCCL INFO Channel 04/0 : 3[3] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:108:108 [2] NCCL INFO Channel 04/0 : 2[2] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:112:112 [6] NCCL INFO Channel 04/0 : 6[6] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:110:110 [4] NCCL INFO Channel 05/0 : 4[4] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:111:111 [5] NCCL INFO Channel 05/0 : 5[5] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:107:107 [1] NCCL INFO Channel 05/0 : 1[1] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:109:109 [3] NCCL INFO Channel 05/0 : 3[3] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:108:108 [2] NCCL INFO Channel 05/0 : 2[2] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:112:112 [6] NCCL INFO Channel 05/0 : 6[6] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:110:110 [4] NCCL INFO Channel 06/0 : 4[4] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:111:111 [5] NCCL INFO Channel 06/0 : 5[5] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:107:107 [1] NCCL INFO Channel 06/0 : 1[1] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:109:109 [3] NCCL INFO Channel 06/0 : 3[3] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:108:108 [2] NCCL INFO Channel 06/0 : 2[2] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:112:112 [6] NCCL INFO Channel 06/0 : 6[6] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:110:110 [4] NCCL INFO Channel 07/0 : 4[4] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:111:111 [5] NCCL INFO Channel 07/0 : 5[5] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:107:107 [1] NCCL INFO Channel 07/0 : 1[1] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:109:109 [3] NCCL INFO Channel 07/0 : 3[3] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:108:108 [2] NCCL INFO Channel 07/0 : 2[2] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:112:112 [6] NCCL INFO Channel 07/0 : 6[6] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:110:110 [4] NCCL INFO Channel 08/0 : 4[4] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:111:111 [5] NCCL INFO Channel 08/0 : 5[5] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:107:107 [1] NCCL INFO Channel 08/0 : 1[1] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:109:109 [3] NCCL INFO Channel 08/0 : 3[3] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:108:108 [2] NCCL INFO Channel 08/0 : 2[2] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:112:112 [6] NCCL INFO Channel 08/0 : 6[6] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:110:110 [4] NCCL INFO Channel 09/0 : 4[4] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:111:111 [5] NCCL INFO Channel 09/0 : 5[5] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:107:107 [1] NCCL INFO Channel 09/0 : 1[1] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:109:109 [3] NCCL INFO Channel 09/0 : 3[3] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:108:108 [2] NCCL INFO Channel 09/0 : 2[2] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:112:112 [6] NCCL INFO Channel 09/0 : 6[6] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:110:110 [4] NCCL INFO Channel 10/0 : 4[4] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:111:111 [5] NCCL INFO Channel 10/0 : 5[5] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:107:107 [1] NCCL INFO Channel 10/0 : 1[1] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:109:109 [3] NCCL INFO Channel 10/0 : 3[3] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:108:108 [2] NCCL INFO Channel 10/0 : 2[2] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:112:112 [6] NCCL INFO Channel 10/0 : 6[6] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:111:111 [5] NCCL INFO Channel 11/0 : 5[5] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:110:110 [4] NCCL INFO Channel 11/0 : 4[4] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:107:107 [1] NCCL INFO Channel 11/0 : 1[1] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:109:109 [3] NCCL INFO Channel 11/0 : 3[3] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:108:108 [2] NCCL INFO Channel 11/0 : 2[2] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:112:112 [6] NCCL INFO Channel 11/0 : 6[6] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:111:111 [5] NCCL INFO Channel 12/0 : 5[5] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:110:110 [4] NCCL INFO Channel 12/0 : 4[4] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:107:107 [1] NCCL INFO Channel 12/0 : 1[1] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:109:109 [3] NCCL INFO Channel 12/0 : 3[3] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:108:108 [2] NCCL INFO Channel 12/0 : 2[2] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:112:112 [6] NCCL INFO Channel 12/0 : 6[6] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:111:111 [5] NCCL INFO Channel 13/0 : 5[5] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:110:110 [4] NCCL INFO Channel 13/0 : 4[4] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:107:107 [1] NCCL INFO Channel 13/0 : 1[1] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:108:108 [2] NCCL INFO Channel 13/0 : 2[2] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:112:112 [6] NCCL INFO Channel 13/0 : 6[6] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:111:111 [5] NCCL INFO Channel 14/0 : 5[5] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:110:110 [4] NCCL INFO Channel 14/0 : 4[4] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:107:107 [1] NCCL INFO Channel 14/0 : 1[1] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:108:108 [2] NCCL INFO Channel 14/0 : 2[2] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:112:112 [6] NCCL INFO Channel 14/0 : 6[6] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:111:111 [5] NCCL INFO Channel 15/0 : 5[5] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:110:110 [4] NCCL INFO Channel 15/0 : 4[4] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:107:107 [1] NCCL INFO Channel 15/0 : 1[1] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:108:108 [2] NCCL INFO Channel 15/0 : 2[2] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:112:112 [6] NCCL INFO Channel 15/0 : 6[6] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:111:111 [5] NCCL INFO Channel 16/0 : 5[5] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:110:110 [4] NCCL INFO Channel 16/0 : 4[4] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:107:107 [1] NCCL INFO Channel 16/0 : 1[1] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:108:108 [2] NCCL INFO Channel 16/0 : 2[2] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:109:109 [3] NCCL INFO Channel 13/0 : 3[3] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:112:112 [6] NCCL INFO Channel 16/0 : 6[6] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:111:111 [5] NCCL INFO Channel 17/0 : 5[5] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:107:107 [1] NCCL INFO Channel 17/0 : 1[1] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:110:110 [4] NCCL INFO Channel 17/0 : 4[4] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:108:108 [2] NCCL INFO Channel 17/0 : 2[2] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:109:109 [3] NCCL INFO Channel 14/0 : 3[3] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:112:112 [6] NCCL INFO Channel 17/0 : 6[6] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:111:111 [5] NCCL INFO Channel 18/0 : 5[5] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:107:107 [1] NCCL INFO Channel 18/0 : 1[1] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:110:110 [4] NCCL INFO Channel 18/0 : 4[4] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:108:108 [2] NCCL INFO Channel 18/0 : 2[2] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:109:109 [3] NCCL INFO Channel 15/0 : 3[3] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:112:112 [6] NCCL INFO Channel 18/0 : 6[6] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:111:111 [5] NCCL INFO Channel 19/0 : 5[5] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:107:107 [1] NCCL INFO Channel 19/0 : 1[1] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:110:110 [4] NCCL INFO Channel 19/0 : 4[4] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:108:108 [2] NCCL INFO Channel 19/0 : 2[2] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:109:109 [3] NCCL INFO Channel 16/0 : 3[3] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:112:112 [6] NCCL INFO Channel 19/0 : 6[6] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:111:111 [5] NCCL INFO Channel 20/0 : 5[5] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:107:107 [1] NCCL INFO Channel 20/0 : 1[1] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:110:110 [4] NCCL INFO Channel 20/0 : 4[4] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:108:108 [2] NCCL INFO Channel 20/0 : 2[2] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:109:109 [3] NCCL INFO Channel 17/0 : 3[3] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:112:112 [6] NCCL INFO Channel 20/0 : 6[6] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:111:111 [5] NCCL INFO Channel 21/0 : 5[5] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:107:107 [1] NCCL INFO Channel 21/0 : 1[1] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:110:110 [4] NCCL INFO Channel 21/0 : 4[4] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:108:108 [2] NCCL INFO Channel 21/0 : 2[2] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:109:109 [3] NCCL INFO Channel 18/0 : 3[3] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:112:112 [6] NCCL INFO Channel 21/0 : 6[6] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:111:111 [5] NCCL INFO Channel 22/0 : 5[5] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:107:107 [1] NCCL INFO Channel 22/0 : 1[1] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:108:108 [2] NCCL INFO Channel 22/0 : 2[2] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:110:110 [4] NCCL INFO Channel 22/0 : 4[4] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:109:109 [3] NCCL INFO Channel 19/0 : 3[3] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:112:112 [6] NCCL INFO Channel 22/0 : 6[6] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:111:111 [5] NCCL INFO Channel 23/0 : 5[5] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:107:107 [1] NCCL INFO Channel 23/0 : 1[1] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:108:108 [2] NCCL INFO Channel 23/0 : 2[2] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:110:110 [4] NCCL INFO Channel 23/0 : 4[4] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:109:109 [3] NCCL INFO Channel 20/0 : 3[3] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:112:112 [6] NCCL INFO Channel 23/0 : 6[6] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:109:109 [3] NCCL INFO Channel 21/0 : 3[3] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:109:109 [3] NCCL INFO Channel 22/0 : 3[3] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:109:109 [3] NCCL INFO Channel 23/0 : 3[3] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:84:84 [0] NCCL INFO Connected all trees
tyler-rhel-newimage:84:84 [0] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512
tyler-rhel-newimage:84:84 [0] NCCL INFO 24 coll channels, 0 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer
tyler-rhel-newimage:113:113 [7] NCCL INFO Connected all trees
tyler-rhel-newimage:113:113 [7] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512
tyler-rhel-newimage:113:113 [7] NCCL INFO 24 coll channels, 0 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer
tyler-rhel-newimage:107:107 [1] NCCL INFO Connected all trees
tyler-rhel-newimage:107:107 [1] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512
tyler-rhel-newimage:107:107 [1] NCCL INFO 24 coll channels, 0 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer
tyler-rhel-newimage:112:112 [6] NCCL INFO Connected all trees
tyler-rhel-newimage:112:112 [6] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512
tyler-rhel-newimage:112:112 [6] NCCL INFO 24 coll channels, 0 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer
tyler-rhel-newimage:108:108 [2] NCCL INFO Connected all trees
tyler-rhel-newimage:108:108 [2] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512
tyler-rhel-newimage:108:108 [2] NCCL INFO 24 coll channels, 0 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer
tyler-rhel-newimage:109:109 [3] NCCL INFO Connected all trees
tyler-rhel-newimage:109:109 [3] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512
tyler-rhel-newimage:109:109 [3] NCCL INFO 24 coll channels, 0 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer
tyler-rhel-newimage:111:111 [5] NCCL INFO Connected all trees
tyler-rhel-newimage:111:111 [5] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512
tyler-rhel-newimage:111:111 [5] NCCL INFO 24 coll channels, 0 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer
tyler-rhel-newimage:110:110 [4] NCCL INFO Connected all trees
tyler-rhel-newimage:110:110 [4] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512
tyler-rhel-newimage:110:110 [4] NCCL INFO 24 coll channels, 0 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer
tyler-rhel-newimage:84:84 [0] NCCL INFO NCCL_WORK_FIFO_DEPTH set by environment to 4194304.
tyler-rhel-newimage:113:113 [7] NCCL INFO NCCL_WORK_FIFO_DEPTH set by environment to 4194304.
tyler-rhel-newimage:107:107 [1] NCCL INFO NCCL_WORK_FIFO_DEPTH set by environment to 4194304.
tyler-rhel-newimage:108:108 [2] NCCL INFO NCCL_WORK_FIFO_DEPTH set by environment to 4194304.
tyler-rhel-newimage:112:112 [6] NCCL INFO NCCL_WORK_FIFO_DEPTH set by environment to 4194304.
tyler-rhel-newimage:109:109 [3] NCCL INFO NCCL_WORK_FIFO_DEPTH set by environment to 4194304.
tyler-rhel-newimage:110:110 [4] NCCL INFO NCCL_WORK_FIFO_DEPTH set by environment to 4194304.
tyler-rhel-newimage:111:111 [5] NCCL INFO NCCL_WORK_FIFO_DEPTH set by environment to 4194304.
tyler-rhel-newimage:112:112 [6] NCCL INFO comm 0x560632844000 rank 6 nranks 8 cudaDev 6 nvmlDev 6 busId e070 commId 0xef46f6c69be18ced - Init COMPLETE
tyler-rhel-newimage:110:110 [4] NCCL INFO comm 0x560632843570 rank 4 nranks 8 cudaDev 4 nvmlDev 4 busId c050 commId 0xef46f6c69be18ced - Init COMPLETE
tyler-rhel-newimage:111:111 [5] NCCL INFO comm 0x560632845230 rank 5 nranks 8 cudaDev 5 nvmlDev 5 busId c060 commId 0xef46f6c69be18ced - Init COMPLETE
tyler-rhel-newimage:113:113 [7] NCCL INFO comm 0x560632844260 rank 7 nranks 8 cudaDev 7 nvmlDev 7 busId e080 commId 0xef46f6c69be18ced - Init COMPLETE
tyler-rhel-newimage:107:107 [1] NCCL INFO comm 0x560632840d50 rank 1 nranks 8 cudaDev 1 nvmlDev 1 busId 8020 commId 0xef46f6c69be18ced - Init COMPLETE
tyler-rhel-newimage:108:108 [2] NCCL INFO comm 0x560632842260 rank 2 nranks 8 cudaDev 2 nvmlDev 2 busId a030 commId 0xef46f6c69be18ced - Init COMPLETE
tyler-rhel-newimage:109:109 [3] NCCL INFO comm 0x560632843410 rank 3 nranks 8 cudaDev 3 nvmlDev 3 busId a040 commId 0xef46f6c69be18ced - Init COMPLETE
tyler-rhel-newimage:84:84 [0] NCCL INFO comm 0x56063284b3e0 rank 0 nranks 8 cudaDev 0 nvmlDev 0 busId 8010 commId 0xef46f6c69be18ced - Init COMPLETE
Loading pt checkpoint shards: 0% Completed | 0/1 [00:00<?, ?it/s]
INFO 2024-07-27 18:36:28,309 instructlab.model.backends.backends:336: Waiting for the vLLM server to start at http://127.0.0.1:60583/v1, this might take a moment... Attempt: 6/80
INFO 2024-07-27 18:36:32,867 instructlab.model.backends.backends:336: Waiting for the vLLM server to start at http://127.0.0.1:60583/v1, this might take a moment... Attempt: 7/80
Loading pt checkpoint shards: 100% Completed | 1/1 [00:06<00:00, 6.27s/it]
Loading pt checkpoint shards: 100% Completed | 1/1 [00:06<00:00, 6.27s/it]
tyler-rhel-newimage:84:646 [0] NCCL INFO Using non-device net plugin version 0
tyler-rhel-newimage:84:646 [0] NCCL INFO Using network Socket
tyler-rhel-newimage:107:650 [1] NCCL INFO Using non-device net plugin version 0
tyler-rhel-newimage:107:650 [1] NCCL INFO Using network Socket
tyler-rhel-newimage:108:651 [2] NCCL INFO Using non-device net plugin version 0
tyler-rhel-newimage:108:651 [2] NCCL INFO Using network Socket
tyler-rhel-newimage:110:649 [4] NCCL INFO Using non-device net plugin version 0
tyler-rhel-newimage:110:649 [4] NCCL INFO Using network Socket
tyler-rhel-newimage:113:648 [7] NCCL INFO Using non-device net plugin version 0
tyler-rhel-newimage:113:648 [7] NCCL INFO Using network Socket
tyler-rhel-newimage:111:647 [5] NCCL INFO Using non-device net plugin version 0
tyler-rhel-newimage:111:647 [5] NCCL INFO Using network Socket
tyler-rhel-newimage:109:652 [3] NCCL INFO Using non-device net plugin version 0
tyler-rhel-newimage:109:652 [3] NCCL INFO Using network Socket
tyler-rhel-newimage:112:653 [6] NCCL INFO Using non-device net plugin version 0
tyler-rhel-newimage:112:653 [6] NCCL INFO Using network Socket
tyler-rhel-newimage:108:651 [2] NCCL INFO comm 0x5606496c1f20 rank 2 nranks 8 cudaDev 2 nvmlDev 2 busId a030 commId 0xb3f11bd99f7ae044 - Init START
tyler-rhel-newimage:107:650 [1] NCCL INFO comm 0x5606496c0b00 rank 1 nranks 8 cudaDev 1 nvmlDev 1 busId 8020 commId 0xb3f11bd99f7ae044 - Init START
tyler-rhel-newimage:113:648 [7] NCCL INFO comm 0x5606496ad9a0 rank 7 nranks 8 cudaDev 7 nvmlDev 7 busId e080 commId 0xb3f11bd99f7ae044 - Init START
tyler-rhel-newimage:111:647 [5] NCCL INFO comm 0x5606496c43b0 rank 5 nranks 8 cudaDev 5 nvmlDev 5 busId c060 commId 0xb3f11bd99f7ae044 - Init START
tyler-rhel-newimage:112:653 [6] NCCL INFO comm 0x5606496c38f0 rank 6 nranks 8 cudaDev 6 nvmlDev 6 busId e070 commId 0xb3f11bd99f7ae044 - Init START
tyler-rhel-newimage:110:649 [4] NCCL INFO comm 0x5606496c3d80 rank 4 nranks 8 cudaDev 4 nvmlDev 4 busId c050 commId 0xb3f11bd99f7ae044 - Init START
tyler-rhel-newimage:84:646 [0] NCCL INFO comm 0x5606496a0560 rank 0 nranks 8 cudaDev 0 nvmlDev 0 busId 8010 commId 0xb3f11bd99f7ae044 - Init START
tyler-rhel-newimage:109:652 [3] NCCL INFO comm 0x5606496c31a0 rank 3 nranks 8 cudaDev 3 nvmlDev 3 busId a040 commId 0xb3f11bd99f7ae044 - Init START
tyler-rhel-newimage:113:648 [7] NCCL INFO Setting affinity for GPU 7 to ffff,ffffff00,00000000
tyler-rhel-newimage:113:648 [7] NCCL INFO NVLS multicast support is not available on dev 7
tyler-rhel-newimage:84:646 [0] NCCL INFO Setting affinity for GPU 0 to ff,ffffffff
tyler-rhel-newimage:84:646 [0] NCCL INFO NVLS multicast support is not available on dev 0
tyler-rhel-newimage:112:653 [6] NCCL INFO Setting affinity for GPU 6 to ffff,ffffff00,00000000
tyler-rhel-newimage:112:653 [6] NCCL INFO NVLS multicast support is not available on dev 6
tyler-rhel-newimage:111:647 [5] NCCL INFO Setting affinity for GPU 5 to ffff,ffffff00,00000000
tyler-rhel-newimage:111:647 [5] NCCL INFO NVLS multicast support is not available on dev 5
tyler-rhel-newimage:109:652 [3] NCCL INFO Setting affinity for GPU 3 to ff,ffffffff
tyler-rhel-newimage:109:652 [3] NCCL INFO NVLS multicast support is not available on dev 3
tyler-rhel-newimage:108:651 [2] NCCL INFO Setting affinity for GPU 2 to ff,ffffffff
tyler-rhel-newimage:108:651 [2] NCCL INFO NVLS multicast support is not available on dev 2
tyler-rhel-newimage:110:649 [4] NCCL INFO Setting affinity for GPU 4 to ffff,ffffff00,00000000
tyler-rhel-newimage:110:649 [4] NCCL INFO NVLS multicast support is not available on dev 4
tyler-rhel-newimage:107:650 [1] NCCL INFO Setting affinity for GPU 1 to ff,ffffffff
tyler-rhel-newimage:107:650 [1] NCCL INFO NVLS multicast support is not available on dev 1
tyler-rhel-newimage:111:647 [5] NCCL INFO comm 0x5606496c43b0 rank 5 nRanks 8 nNodes 1 localRanks 8 localRank 5 MNNVL 0
tyler-rhel-newimage:111:647 [5] NCCL INFO Trees [0] 6/-1/-1->5->4 [1] 6/-1/-1->5->4 [2] 6/-1/-1->5->4 [3] 6/-1/-1->5->4 [4] 6/-1/-1->5->4 [5] 6/-1/-1->5->4 [6] 6/-1/-1->5->4 [7] 6/-1/-1->5->4 [8] 6/-1/-1->5->4 [9] 6/-1/-1->5->4 [10] 6/-1/-1->5->4 [11] 6/-1/-1->5->4 [12] 6/-1/-1->5->4 [13] 6/-1/-1->5->4 [14] 6/-1/-1->5->4 [15] 6/-1/-1->5->4 [16] 6/-1/-1->5->4 [17] 6/-1/-1->5->4 [18] 6/-1/-1->5->4 [19] 6/-1/-1->5->4 [20] 6/-1/-1->5->4 [21] 6/-1/-1->5->4 [22] 6/-1/-1->5->4 [23] 6/-1/-1->5->4
tyler-rhel-newimage:111:647 [5] NCCL INFO P2P Chunksize set to 524288
tyler-rhel-newimage:109:652 [3] NCCL INFO comm 0x5606496c31a0 rank 3 nRanks 8 nNodes 1 localRanks 8 localRank 3 MNNVL 0
tyler-rhel-newimage:108:651 [2] NCCL INFO comm 0x5606496c1f20 rank 2 nRanks 8 nNodes 1 localRanks 8 localRank 2 MNNVL 0
tyler-rhel-newimage:110:649 [4] NCCL INFO comm 0x5606496c3d80 rank 4 nRanks 8 nNodes 1 localRanks 8 localRank 4 MNNVL 0
tyler-rhel-newimage:113:648 [7] NCCL INFO comm 0x5606496ad9a0 rank 7 nRanks 8 nNodes 1 localRanks 8 localRank 7 MNNVL 0
tyler-rhel-newimage:107:650 [1] NCCL INFO comm 0x5606496c0b00 rank 1 nRanks 8 nNodes 1 localRanks 8 localRank 1 MNNVL 0
tyler-rhel-newimage:84:646 [0] NCCL INFO comm 0x5606496a0560 rank 0 nRanks 8 nNodes 1 localRanks 8 localRank 0 MNNVL 0
tyler-rhel-newimage:109:652 [3] NCCL INFO Trees [0] 4/-1/-1->3->2 [1] 4/-1/-1->3->2 [2] 4/-1/-1->3->2 [3] 4/-1/-1->3->2 [4] 4/-1/-1->3->2 [5] 4/-1/-1->3->2 [6] 4/-1/-1->3->2 [7] 4/-1/-1->3->2 [8] 4/-1/-1->3->2 [9] 4/-1/-1->3->2 [10] 4/-1/-1->3->2 [11] 4/-1/-1->3->2 [12] 4/-1/-1->3->2 [13] 4/-1/-1->3->2 [14] 4/-1/-1->3->2 [15] 4/-1/-1->3->2 [16] 4/-1/-1->3->2 [17] 4/-1/-1->3->2 [18] 4/-1/-1->3->2 [19] 4/-1/-1->3->2 [20] 4/-1/-1->3->2 [21] 4/-1/-1->3->2 [22] 4/-1/-1->3->2 [23] 4/-1/-1->3->2
tyler-rhel-newimage:108:651 [2] NCCL INFO Trees [0] 3/-1/-1->2->1 [1] 3/-1/-1->2->1 [2] 3/-1/-1->2->1 [3] 3/-1/-1->2->1 [4] 3/-1/-1->2->1 [5] 3/-1/-1->2->1 [6] 3/-1/-1->2->1 [7] 3/-1/-1->2->1 [8] 3/-1/-1->2->1 [9] 3/-1/-1->2->1 [10] 3/-1/-1->2->1 [11] 3/-1/-1->2->1 [12] 3/-1/-1->2->1 [13] 3/-1/-1->2->1 [14] 3/-1/-1->2->1 [15] 3/-1/-1->2->1 [16] 3/-1/-1->2->1 [17] 3/-1/-1->2->1 [18] 3/-1/-1->2->1 [19] 3/-1/-1->2->1 [20] 3/-1/-1->2->1 [21] 3/-1/-1->2->1 [22] 3/-1/-1->2->1 [23] 3/-1/-1->2->1
tyler-rhel-newimage:110:649 [4] NCCL INFO Trees [0] 5/-1/-1->4->3 [1] 5/-1/-1->4->3 [2] 5/-1/-1->4->3 [3] 5/-1/-1->4->3 [4] 5/-1/-1->4->3 [5] 5/-1/-1->4->3 [6] 5/-1/-1->4->3 [7] 5/-1/-1->4->3 [8] 5/-1/-1->4->3 [9] 5/-1/-1->4->3 [10] 5/-1/-1->4->3 [11] 5/-1/-1->4->3 [12] 5/-1/-1->4->3 [13] 5/-1/-1->4->3 [14] 5/-1/-1->4->3 [15] 5/-1/-1->4->3 [16] 5/-1/-1->4->3 [17] 5/-1/-1->4->3 [18] 5/-1/-1->4->3 [19] 5/-1/-1->4->3 [20] 5/-1/-1->4->3 [21] 5/-1/-1->4->3 [22] 5/-1/-1->4->3 [23] 5/-1/-1->4->3
tyler-rhel-newimage:109:652 [3] NCCL INFO P2P Chunksize set to 524288
tyler-rhel-newimage:113:648 [7] NCCL INFO Trees [0] -1/-1/-1->7->6 [1] -1/-1/-1->7->6 [2] -1/-1/-1->7->6 [3] -1/-1/-1->7->6 [4] -1/-1/-1->7->6 [5] -1/-1/-1->7->6 [6] -1/-1/-1->7->6 [7] -1/-1/-1->7->6 [8] -1/-1/-1->7->6 [9] -1/-1/-1->7->6 [10] -1/-1/-1->7->6 [11] -1/-1/-1->7->6 [12] -1/-1/-1->7->6 [13] -1/-1/-1->7->6 [14] -1/-1/-1->7->6 [15] -1/-1/-1->7->6 [16] -1/-1/-1->7->6 [17] -1/-1/-1->7->6 [18] -1/-1/-1->7->6 [19] -1/-1/-1->7->6 [20] -1/-1/-1->7->6 [21] -1/-1/-1->7->6 [22] -1/-1/-1->7->6 [23] -1/-1/-1->7->6
tyler-rhel-newimage:108:651 [2] NCCL INFO P2P Chunksize set to 524288
tyler-rhel-newimage:107:650 [1] NCCL INFO Trees [0] 2/-1/-1->1->0 [1] 2/-1/-1->1->0 [2] 2/-1/-1->1->0 [3] 2/-1/-1->1->0 [4] 2/-1/-1->1->0 [5] 2/-1/-1->1->0 [6] 2/-1/-1->1->0 [7] 2/-1/-1->1->0 [8] 2/-1/-1->1->0 [9] 2/-1/-1->1->0 [10] 2/-1/-1->1->0 [11] 2/-1/-1->1->0 [12] 2/-1/-1->1->0 [13] 2/-1/-1->1->0 [14] 2/-1/-1->1->0 [15] 2/-1/-1->1->0 [16] 2/-1/-1->1->0 [17] 2/-1/-1->1->0 [18] 2/-1/-1->1->0 [19] 2/-1/-1->1->0 [20] 2/-1/-1->1->0 [21] 2/-1/-1->1->0 [22] 2/-1/-1->1->0 [23] 2/-1/-1->1->0
tyler-rhel-newimage:110:649 [4] NCCL INFO P2P Chunksize set to 524288
tyler-rhel-newimage:107:650 [1] NCCL INFO P2P Chunksize set to 524288
tyler-rhel-newimage:113:648 [7] NCCL INFO P2P Chunksize set to 524288
tyler-rhel-newimage:84:646 [0] NCCL INFO Channel 00/24 : 0 1 2 3 4 5 6 7
tyler-rhel-newimage:112:653 [6] NCCL INFO comm 0x5606496c38f0 rank 6 nRanks 8 nNodes 1 localRanks 8 localRank 6 MNNVL 0
tyler-rhel-newimage:84:646 [0] NCCL INFO Channel 01/24 : 0 1 2 3 4 5 6 7
tyler-rhel-newimage:84:646 [0] NCCL INFO Channel 02/24 : 0 1 2 3 4 5 6 7
tyler-rhel-newimage:84:646 [0] NCCL INFO Channel 03/24 : 0 1 2 3 4 5 6 7
tyler-rhel-newimage:84:646 [0] NCCL INFO Channel 04/24 : 0 1 2 3 4 5 6 7
tyler-rhel-newimage:84:646 [0] NCCL INFO Channel 05/24 : 0 1 2 3 4 5 6 7
tyler-rhel-newimage:84:646 [0] NCCL INFO Channel 06/24 : 0 1 2 3 4 5 6 7
tyler-rhel-newimage:84:646 [0] NCCL INFO Channel 07/24 : 0 1 2 3 4 5 6 7
tyler-rhel-newimage:84:646 [0] NCCL INFO Channel 08/24 : 0 1 2 3 4 5 6 7
tyler-rhel-newimage:84:646 [0] NCCL INFO Channel 09/24 : 0 1 2 3 4 5 6 7
tyler-rhel-newimage:84:646 [0] NCCL INFO Channel 10/24 : 0 1 2 3 4 5 6 7
tyler-rhel-newimage:84:646 [0] NCCL INFO Channel 11/24 : 0 1 2 3 4 5 6 7
tyler-rhel-newimage:84:646 [0] NCCL INFO Channel 12/24 : 0 1 2 3 4 5 6 7
tyler-rhel-newimage:84:646 [0] NCCL INFO Channel 13/24 : 0 1 2 3 4 5 6 7
tyler-rhel-newimage:112:653 [6] NCCL INFO Trees [0] 7/-1/-1->6->5 [1] 7/-1/-1->6->5 [2] 7/-1/-1->6->5 [3] 7/-1/-1->6->5 [4] 7/-1/-1->6->5 [5] 7/-1/-1->6->5 [6] 7/-1/-1->6->5 [7] 7/-1/-1->6->5 [8] 7/-1/-1->6->5 [9] 7/-1/-1->6->5 [10] 7/-1/-1->6->5 [11] 7/-1/-1->6->5 [12] 7/-1/-1->6->5 [13] 7/-1/-1->6->5 [14] 7/-1/-1->6->5 [15] 7/-1/-1->6->5 [16] 7/-1/-1->6->5 [17] 7/-1/-1->6->5 [18] 7/-1/-1->6->5 [19] 7/-1/-1->6->5 [20] 7/-1/-1->6->5 [21] 7/-1/-1->6->5 [22] 7/-1/-1->6->5 [23] 7/-1/-1->6->5
tyler-rhel-newimage:84:646 [0] NCCL INFO Channel 14/24 : 0 1 2 3 4 5 6 7
tyler-rhel-newimage:84:646 [0] NCCL INFO Channel 15/24 : 0 1 2 3 4 5 6 7
tyler-rhel-newimage:112:653 [6] NCCL INFO P2P Chunksize set to 524288
tyler-rhel-newimage:84:646 [0] NCCL INFO Channel 16/24 : 0 1 2 3 4 5 6 7
tyler-rhel-newimage:84:646 [0] NCCL INFO Channel 17/24 : 0 1 2 3 4 5 6 7
tyler-rhel-newimage:84:646 [0] NCCL INFO Channel 18/24 : 0 1 2 3 4 5 6 7
tyler-rhel-newimage:84:646 [0] NCCL INFO Channel 19/24 : 0 1 2 3 4 5 6 7
tyler-rhel-newimage:84:646 [0] NCCL INFO Channel 20/24 : 0 1 2 3 4 5 6 7
tyler-rhel-newimage:84:646 [0] NCCL INFO Channel 21/24 : 0 1 2 3 4 5 6 7
tyler-rhel-newimage:84:646 [0] NCCL INFO Channel 22/24 : 0 1 2 3 4 5 6 7
tyler-rhel-newimage:84:646 [0] NCCL INFO Channel 23/24 : 0 1 2 3 4 5 6 7
tyler-rhel-newimage:84:646 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1 [2] 1/-1/-1->0->-1 [3] 1/-1/-1->0->-1 [4] 1/-1/-1->0->-1 [5] 1/-1/-1->0->-1 [6] 1/-1/-1->0->-1 [7] 1/-1/-1->0->-1 [8] 1/-1/-1->0->-1 [9] 1/-1/-1->0->-1 [10] 1/-1/-1->0->-1 [11] 1/-1/-1->0->-1 [12] 1/-1/-1->0->-1 [13] 1/-1/-1->0->-1 [14] 1/-1/-1->0->-1 [15] 1/-1/-1->0->-1 [16] 1/-1/-1->0->-1 [17] 1/-1/-1->0->-1 [18] 1/-1/-1->0->-1 [19] 1/-1/-1->0->-1 [20] 1/-1/-1->0->-1 [21] 1/-1/-1->0->-1 [22] 1/-1/-1->0->-1 [23] 1/-1/-1->0->-1
tyler-rhel-newimage:84:646 [0] NCCL INFO P2P Chunksize set to 524288
tyler-rhel-newimage:111:647 [5] NCCL INFO Channel 00/0 : 5[5] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:111:647 [5] NCCL INFO Channel 01/0 : 5[5] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:111:647 [5] NCCL INFO Channel 02/0 : 5[5] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:111:647 [5] NCCL INFO Channel 03/0 : 5[5] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:111:647 [5] NCCL INFO Channel 04/0 : 5[5] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:111:647 [5] NCCL INFO Channel 05/0 : 5[5] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:111:647 [5] NCCL INFO Channel 06/0 : 5[5] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:111:647 [5] NCCL INFO Channel 07/0 : 5[5] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:111:647 [5] NCCL INFO Channel 08/0 : 5[5] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:111:647 [5] NCCL INFO Channel 09/0 : 5[5] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:108:651 [2] NCCL INFO Channel 00/0 : 2[2] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:113:648 [7] NCCL INFO Channel 00/0 : 7[7] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:110:649 [4] NCCL INFO Channel 00/0 : 4[4] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:111:647 [5] NCCL INFO Channel 10/0 : 5[5] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:109:652 [3] NCCL INFO Channel 00/0 : 3[3] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:108:651 [2] NCCL INFO Channel 01/0 : 2[2] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:113:648 [7] NCCL INFO Channel 01/0 : 7[7] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:110:649 [4] NCCL INFO Channel 01/0 : 4[4] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:111:647 [5] NCCL INFO Channel 11/0 : 5[5] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:109:652 [3] NCCL INFO Channel 01/0 : 3[3] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:108:651 [2] NCCL INFO Channel 02/0 : 2[2] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:113:648 [7] NCCL INFO Channel 02/0 : 7[7] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:110:649 [4] NCCL INFO Channel 02/0 : 4[4] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:111:647 [5] NCCL INFO Channel 12/0 : 5[5] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:109:652 [3] NCCL INFO Channel 02/0 : 3[3] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:108:651 [2] NCCL INFO Channel 03/0 : 2[2] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:113:648 [7] NCCL INFO Channel 03/0 : 7[7] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:110:649 [4] NCCL INFO Channel 03/0 : 4[4] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:111:647 [5] NCCL INFO Channel 13/0 : 5[5] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:109:652 [3] NCCL INFO Channel 03/0 : 3[3] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:108:651 [2] NCCL INFO Channel 04/0 : 2[2] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:113:648 [7] NCCL INFO Channel 04/0 : 7[7] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:110:649 [4] NCCL INFO Channel 04/0 : 4[4] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:111:647 [5] NCCL INFO Channel 14/0 : 5[5] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:109:652 [3] NCCL INFO Channel 04/0 : 3[3] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:108:651 [2] NCCL INFO Channel 05/0 : 2[2] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:113:648 [7] NCCL INFO Channel 05/0 : 7[7] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:110:649 [4] NCCL INFO Channel 05/0 : 4[4] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:111:647 [5] NCCL INFO Channel 15/0 : 5[5] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:109:652 [3] NCCL INFO Channel 05/0 : 3[3] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:108:651 [2] NCCL INFO Channel 06/0 : 2[2] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:113:648 [7] NCCL INFO Channel 06/0 : 7[7] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:111:647 [5] NCCL INFO Channel 16/0 : 5[5] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:109:652 [3] NCCL INFO Channel 06/0 : 3[3] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:108:651 [2] NCCL INFO Channel 07/0 : 2[2] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:113:648 [7] NCCL INFO Channel 07/0 : 7[7] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:107:650 [1] NCCL INFO Channel 00/0 : 1[1] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:111:647 [5] NCCL INFO Channel 17/0 : 5[5] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:109:652 [3] NCCL INFO Channel 07/0 : 3[3] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:108:651 [2] NCCL INFO Channel 08/0 : 2[2] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:113:648 [7] NCCL INFO Channel 08/0 : 7[7] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:107:650 [1] NCCL INFO Channel 01/0 : 1[1] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:111:647 [5] NCCL INFO Channel 18/0 : 5[5] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:109:652 [3] NCCL INFO Channel 08/0 : 3[3] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:108:651 [2] NCCL INFO Channel 09/0 : 2[2] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:113:648 [7] NCCL INFO Channel 09/0 : 7[7] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:107:650 [1] NCCL INFO Channel 02/0 : 1[1] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:111:647 [5] NCCL INFO Channel 19/0 : 5[5] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:109:652 [3] NCCL INFO Channel 09/0 : 3[3] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:108:651 [2] NCCL INFO Channel 10/0 : 2[2] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:113:648 [7] NCCL INFO Channel 10/0 : 7[7] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:107:650 [1] NCCL INFO Channel 03/0 : 1[1] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:111:647 [5] NCCL INFO Channel 20/0 : 5[5] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:109:652 [3] NCCL INFO Channel 10/0 : 3[3] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:108:651 [2] NCCL INFO Channel 11/0 : 2[2] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:113:648 [7] NCCL INFO Channel 11/0 : 7[7] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:107:650 [1] NCCL INFO Channel 04/0 : 1[1] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:111:647 [5] NCCL INFO Channel 21/0 : 5[5] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:109:652 [3] NCCL INFO Channel 11/0 : 3[3] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:108:651 [2] NCCL INFO Channel 12/0 : 2[2] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:113:648 [7] NCCL INFO Channel 12/0 : 7[7] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:107:650 [1] NCCL INFO Channel 05/0 : 1[1] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:111:647 [5] NCCL INFO Channel 22/0 : 5[5] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:109:652 [3] NCCL INFO Channel 12/0 : 3[3] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:108:651 [2] NCCL INFO Channel 13/0 : 2[2] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:113:648 [7] NCCL INFO Channel 13/0 : 7[7] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:107:650 [1] NCCL INFO Channel 06/0 : 1[1] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:111:647 [5] NCCL INFO Channel 23/0 : 5[5] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:109:652 [3] NCCL INFO Channel 13/0 : 3[3] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:108:651 [2] NCCL INFO Channel 14/0 : 2[2] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:113:648 [7] NCCL INFO Channel 14/0 : 7[7] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:107:650 [1] NCCL INFO Channel 07/0 : 1[1] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:109:652 [3] NCCL INFO Channel 14/0 : 3[3] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:108:651 [2] NCCL INFO Channel 15/0 : 2[2] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:113:648 [7] NCCL INFO Channel 15/0 : 7[7] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:107:650 [1] NCCL INFO Channel 08/0 : 1[1] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:84:646 [0] NCCL INFO Channel 00/0 : 0[0] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:110:649 [4] NCCL INFO Channel 06/0 : 4[4] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:109:652 [3] NCCL INFO Channel 15/0 : 3[3] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:108:651 [2] NCCL INFO Channel 16/0 : 2[2] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:113:648 [7] NCCL INFO Channel 16/0 : 7[7] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:107:650 [1] NCCL INFO Channel 09/0 : 1[1] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:84:646 [0] NCCL INFO Channel 01/0 : 0[0] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:110:649 [4] NCCL INFO Channel 07/0 : 4[4] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:109:652 [3] NCCL INFO Channel 16/0 : 3[3] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:108:651 [2] NCCL INFO Channel 17/0 : 2[2] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:113:648 [7] NCCL INFO Channel 17/0 : 7[7] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:107:650 [1] NCCL INFO Channel 10/0 : 1[1] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:84:646 [0] NCCL INFO Channel 02/0 : 0[0] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:110:649 [4] NCCL INFO Channel 08/0 : 4[4] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:109:652 [3] NCCL INFO Channel 17/0 : 3[3] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:108:651 [2] NCCL INFO Channel 18/0 : 2[2] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:113:648 [7] NCCL INFO Channel 18/0 : 7[7] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:107:650 [1] NCCL INFO Channel 11/0 : 1[1] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:84:646 [0] NCCL INFO Channel 03/0 : 0[0] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:110:649 [4] NCCL INFO Channel 09/0 : 4[4] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:109:652 [3] NCCL INFO Channel 18/0 : 3[3] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:108:651 [2] NCCL INFO Channel 19/0 : 2[2] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:113:648 [7] NCCL INFO Channel 19/0 : 7[7] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:110:649 [4] NCCL INFO Channel 10/0 : 4[4] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:107:650 [1] NCCL INFO Channel 12/0 : 1[1] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:84:646 [0] NCCL INFO Channel 04/0 : 0[0] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:109:652 [3] NCCL INFO Channel 19/0 : 3[3] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:108:651 [2] NCCL INFO Channel 20/0 : 2[2] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:113:648 [7] NCCL INFO Channel 20/0 : 7[7] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:110:649 [4] NCCL INFO Channel 11/0 : 4[4] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:107:650 [1] NCCL INFO Channel 13/0 : 1[1] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:84:646 [0] NCCL INFO Channel 05/0 : 0[0] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:109:652 [3] NCCL INFO Channel 20/0 : 3[3] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:108:651 [2] NCCL INFO Channel 21/0 : 2[2] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:113:648 [7] NCCL INFO Channel 21/0 : 7[7] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:110:649 [4] NCCL INFO Channel 12/0 : 4[4] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:84:646 [0] NCCL INFO Channel 06/0 : 0[0] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:107:650 [1] NCCL INFO Channel 14/0 : 1[1] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:108:651 [2] NCCL INFO Channel 22/0 : 2[2] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:109:652 [3] NCCL INFO Channel 21/0 : 3[3] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:113:648 [7] NCCL INFO Channel 22/0 : 7[7] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:112:653 [6] NCCL INFO Channel 00/0 : 6[6] -> 7[7] via P2P/IPC/read
tyler-rhel-newimage:110:649 [4] NCCL INFO Channel 13/0 : 4[4] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:84:646 [0] NCCL INFO Channel 07/0 : 0[0] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:109:652 [3] NCCL INFO Channel 22/0 : 3[3] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:107:650 [1] NCCL INFO Channel 15/0 : 1[1] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:108:651 [2] NCCL INFO Channel 23/0 : 2[2] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:113:648 [7] NCCL INFO Channel 23/0 : 7[7] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:112:653 [6] NCCL INFO Channel 01/0 : 6[6] -> 7[7] via P2P/IPC/read
tyler-rhel-newimage:110:649 [4] NCCL INFO Channel 14/0 : 4[4] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:84:646 [0] NCCL INFO Channel 08/0 : 0[0] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:109:652 [3] NCCL INFO Channel 23/0 : 3[3] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:107:650 [1] NCCL INFO Channel 16/0 : 1[1] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:112:653 [6] NCCL INFO Channel 02/0 : 6[6] -> 7[7] via P2P/IPC/read
tyler-rhel-newimage:110:649 [4] NCCL INFO Channel 15/0 : 4[4] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:84:646 [0] NCCL INFO Channel 09/0 : 0[0] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:107:650 [1] NCCL INFO Channel 17/0 : 1[1] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:112:653 [6] NCCL INFO Channel 03/0 : 6[6] -> 7[7] via P2P/IPC/read
tyler-rhel-newimage:110:649 [4] NCCL INFO Channel 16/0 : 4[4] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:84:646 [0] NCCL INFO Channel 10/0 : 0[0] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:107:650 [1] NCCL INFO Channel 18/0 : 1[1] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:112:653 [6] NCCL INFO Channel 04/0 : 6[6] -> 7[7] via P2P/IPC/read
tyler-rhel-newimage:110:649 [4] NCCL INFO Channel 17/0 : 4[4] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:84:646 [0] NCCL INFO Channel 11/0 : 0[0] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:107:650 [1] NCCL INFO Channel 19/0 : 1[1] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:112:653 [6] NCCL INFO Channel 05/0 : 6[6] -> 7[7] via P2P/IPC/read
tyler-rhel-newimage:110:649 [4] NCCL INFO Channel 18/0 : 4[4] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:84:646 [0] NCCL INFO Channel 12/0 : 0[0] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:107:650 [1] NCCL INFO Channel 20/0 : 1[1] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:112:653 [6] NCCL INFO Channel 06/0 : 6[6] -> 7[7] via P2P/IPC/read
tyler-rhel-newimage:110:649 [4] NCCL INFO Channel 19/0 : 4[4] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:107:650 [1] NCCL INFO Channel 21/0 : 1[1] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:84:646 [0] NCCL INFO Channel 13/0 : 0[0] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:112:653 [6] NCCL INFO Channel 07/0 : 6[6] -> 7[7] via P2P/IPC/read
tyler-rhel-newimage:110:649 [4] NCCL INFO Channel 20/0 : 4[4] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:107:650 [1] NCCL INFO Channel 22/0 : 1[1] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:84:646 [0] NCCL INFO Channel 14/0 : 0[0] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:112:653 [6] NCCL INFO Channel 08/0 : 6[6] -> 7[7] via P2P/IPC/read
tyler-rhel-newimage:110:649 [4] NCCL INFO Channel 21/0 : 4[4] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:84:646 [0] NCCL INFO Channel 15/0 : 0[0] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:107:650 [1] NCCL INFO Channel 23/0 : 1[1] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:112:653 [6] NCCL INFO Channel 09/0 : 6[6] -> 7[7] via P2P/IPC/read
tyler-rhel-newimage:110:649 [4] NCCL INFO Channel 22/0 : 4[4] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:84:646 [0] NCCL INFO Channel 16/0 : 0[0] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:112:653 [6] NCCL INFO Channel 10/0 : 6[6] -> 7[7] via P2P/IPC/read
tyler-rhel-newimage:110:649 [4] NCCL INFO Channel 23/0 : 4[4] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:84:646 [0] NCCL INFO Channel 17/0 : 0[0] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:112:653 [6] NCCL INFO Channel 11/0 : 6[6] -> 7[7] via P2P/IPC/read
tyler-rhel-newimage:84:646 [0] NCCL INFO Channel 18/0 : 0[0] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:112:653 [6] NCCL INFO Channel 12/0 : 6[6] -> 7[7] via P2P/IPC/read
tyler-rhel-newimage:84:646 [0] NCCL INFO Channel 19/0 : 0[0] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:112:653 [6] NCCL INFO Channel 13/0 : 6[6] -> 7[7] via P2P/IPC/read
tyler-rhel-newimage:84:646 [0] NCCL INFO Channel 20/0 : 0[0] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:84:646 [0] NCCL INFO Channel 21/0 : 0[0] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:112:653 [6] NCCL INFO Channel 14/0 : 6[6] -> 7[7] via P2P/IPC/read
tyler-rhel-newimage:84:646 [0] NCCL INFO Channel 22/0 : 0[0] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:112:653 [6] NCCL INFO Channel 15/0 : 6[6] -> 7[7] via P2P/IPC/read
tyler-rhel-newimage:84:646 [0] NCCL INFO Channel 23/0 : 0[0] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:112:653 [6] NCCL INFO Channel 16/0 : 6[6] -> 7[7] via P2P/IPC/read
tyler-rhel-newimage:112:653 [6] NCCL INFO Channel 17/0 : 6[6] -> 7[7] via P2P/IPC/read
tyler-rhel-newimage:112:653 [6] NCCL INFO Channel 18/0 : 6[6] -> 7[7] via P2P/IPC/read
tyler-rhel-newimage:112:653 [6] NCCL INFO Channel 19/0 : 6[6] -> 7[7] via P2P/IPC/read
tyler-rhel-newimage:112:653 [6] NCCL INFO Channel 20/0 : 6[6] -> 7[7] via P2P/IPC/read
tyler-rhel-newimage:112:653 [6] NCCL INFO Channel 21/0 : 6[6] -> 7[7] via P2P/IPC/read
tyler-rhel-newimage:112:653 [6] NCCL INFO Channel 22/0 : 6[6] -> 7[7] via P2P/IPC/read
tyler-rhel-newimage:112:653 [6] NCCL INFO Channel 23/0 : 6[6] -> 7[7] via P2P/IPC/read
tyler-rhel-newimage:109:652 [3] NCCL INFO Connected all rings
tyler-rhel-newimage:108:651 [2] NCCL INFO Connected all rings
tyler-rhel-newimage:107:650 [1] NCCL INFO Connected all rings
tyler-rhel-newimage:111:647 [5] NCCL INFO Connected all rings
tyler-rhel-newimage:110:649 [4] NCCL INFO Connected all rings
tyler-rhel-newimage:84:646 [0] NCCL INFO Connected all rings
tyler-rhel-newimage:113:648 [7] NCCL INFO Connected all rings
tyler-rhel-newimage:112:653 [6] NCCL INFO Connected all rings
tyler-rhel-newimage:113:648 [7] NCCL INFO Channel 00/0 : 7[7] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:113:648 [7] NCCL INFO Channel 01/0 : 7[7] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:113:648 [7] NCCL INFO Channel 02/0 : 7[7] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:113:648 [7] NCCL INFO Channel 03/0 : 7[7] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:113:648 [7] NCCL INFO Channel 04/0 : 7[7] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:113:648 [7] NCCL INFO Channel 05/0 : 7[7] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:113:648 [7] NCCL INFO Channel 06/0 : 7[7] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:113:648 [7] NCCL INFO Channel 07/0 : 7[7] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:113:648 [7] NCCL INFO Channel 08/0 : 7[7] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:113:648 [7] NCCL INFO Channel 09/0 : 7[7] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:108:651 [2] NCCL INFO Channel 00/0 : 2[2] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:113:648 [7] NCCL INFO Channel 10/0 : 7[7] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:108:651 [2] NCCL INFO Channel 01/0 : 2[2] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:107:650 [1] NCCL INFO Channel 00/0 : 1[1] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:109:652 [3] NCCL INFO Channel 00/0 : 3[3] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:113:648 [7] NCCL INFO Channel 11/0 : 7[7] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:108:651 [2] NCCL INFO Channel 02/0 : 2[2] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:107:650 [1] NCCL INFO Channel 01/0 : 1[1] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:109:652 [3] NCCL INFO Channel 01/0 : 3[3] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:113:648 [7] NCCL INFO Channel 12/0 : 7[7] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:108:651 [2] NCCL INFO Channel 03/0 : 2[2] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:107:650 [1] NCCL INFO Channel 02/0 : 1[1] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:109:652 [3] NCCL INFO Channel 02/0 : 3[3] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:113:648 [7] NCCL INFO Channel 13/0 : 7[7] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:108:651 [2] NCCL INFO Channel 04/0 : 2[2] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:107:650 [1] NCCL INFO Channel 03/0 : 1[1] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:109:652 [3] NCCL INFO Channel 03/0 : 3[3] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:113:648 [7] NCCL INFO Channel 14/0 : 7[7] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:107:650 [1] NCCL INFO Channel 04/0 : 1[1] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:108:651 [2] NCCL INFO Channel 05/0 : 2[2] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:109:652 [3] NCCL INFO Channel 04/0 : 3[3] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:113:648 [7] NCCL INFO Channel 15/0 : 7[7] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:107:650 [1] NCCL INFO Channel 05/0 : 1[1] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:108:651 [2] NCCL INFO Channel 06/0 : 2[2] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:109:652 [3] NCCL INFO Channel 05/0 : 3[3] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:113:648 [7] NCCL INFO Channel 16/0 : 7[7] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:107:650 [1] NCCL INFO Channel 06/0 : 1[1] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:108:651 [2] NCCL INFO Channel 07/0 : 2[2] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:109:652 [3] NCCL INFO Channel 06/0 : 3[3] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:113:648 [7] NCCL INFO Channel 17/0 : 7[7] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:107:650 [1] NCCL INFO Channel 07/0 : 1[1] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:108:651 [2] NCCL INFO Channel 08/0 : 2[2] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:109:652 [3] NCCL INFO Channel 07/0 : 3[3] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:113:648 [7] NCCL INFO Channel 18/0 : 7[7] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:111:647 [5] NCCL INFO Channel 00/0 : 5[5] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:107:650 [1] NCCL INFO Channel 08/0 : 1[1] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:108:651 [2] NCCL INFO Channel 09/0 : 2[2] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:109:652 [3] NCCL INFO Channel 08/0 : 3[3] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:113:648 [7] NCCL INFO Channel 19/0 : 7[7] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:111:647 [5] NCCL INFO Channel 01/0 : 5[5] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:107:650 [1] NCCL INFO Channel 09/0 : 1[1] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:108:651 [2] NCCL INFO Channel 10/0 : 2[2] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:109:652 [3] NCCL INFO Channel 09/0 : 3[3] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:113:648 [7] NCCL INFO Channel 20/0 : 7[7] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:111:647 [5] NCCL INFO Channel 02/0 : 5[5] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:107:650 [1] NCCL INFO Channel 10/0 : 1[1] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:108:651 [2] NCCL INFO Channel 11/0 : 2[2] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:109:652 [3] NCCL INFO Channel 10/0 : 3[3] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:113:648 [7] NCCL INFO Channel 21/0 : 7[7] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:110:649 [4] NCCL INFO Channel 00/0 : 4[4] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:111:647 [5] NCCL INFO Channel 03/0 : 5[5] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:107:650 [1] NCCL INFO Channel 11/0 : 1[1] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:108:651 [2] NCCL INFO Channel 12/0 : 2[2] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:109:652 [3] NCCL INFO Channel 11/0 : 3[3] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:113:648 [7] NCCL INFO Channel 22/0 : 7[7] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:110:649 [4] NCCL INFO Channel 01/0 : 4[4] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:111:647 [5] NCCL INFO Channel 04/0 : 5[5] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:107:650 [1] NCCL INFO Channel 12/0 : 1[1] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:108:651 [2] NCCL INFO Channel 13/0 : 2[2] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:109:652 [3] NCCL INFO Channel 12/0 : 3[3] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:113:648 [7] NCCL INFO Channel 23/0 : 7[7] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:110:649 [4] NCCL INFO Channel 02/0 : 4[4] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:111:647 [5] NCCL INFO Channel 05/0 : 5[5] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:107:650 [1] NCCL INFO Channel 13/0 : 1[1] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:108:651 [2] NCCL INFO Channel 14/0 : 2[2] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:109:652 [3] NCCL INFO Channel 13/0 : 3[3] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:110:649 [4] NCCL INFO Channel 03/0 : 4[4] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:111:647 [5] NCCL INFO Channel 06/0 : 5[5] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:107:650 [1] NCCL INFO Channel 14/0 : 1[1] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:108:651 [2] NCCL INFO Channel 15/0 : 2[2] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:109:652 [3] NCCL INFO Channel 14/0 : 3[3] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:112:653 [6] NCCL INFO Channel 00/0 : 6[6] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:110:649 [4] NCCL INFO Channel 04/0 : 4[4] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:111:647 [5] NCCL INFO Channel 07/0 : 5[5] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:107:650 [1] NCCL INFO Channel 15/0 : 1[1] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:108:651 [2] NCCL INFO Channel 16/0 : 2[2] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:109:652 [3] NCCL INFO Channel 15/0 : 3[3] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:112:653 [6] NCCL INFO Channel 01/0 : 6[6] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:110:649 [4] NCCL INFO Channel 05/0 : 4[4] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:111:647 [5] NCCL INFO Channel 08/0 : 5[5] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:107:650 [1] NCCL INFO Channel 16/0 : 1[1] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:108:651 [2] NCCL INFO Channel 17/0 : 2[2] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:109:652 [3] NCCL INFO Channel 16/0 : 3[3] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:112:653 [6] NCCL INFO Channel 02/0 : 6[6] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:110:649 [4] NCCL INFO Channel 06/0 : 4[4] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:111:647 [5] NCCL INFO Channel 09/0 : 5[5] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:107:650 [1] NCCL INFO Channel 17/0 : 1[1] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:109:652 [3] NCCL INFO Channel 17/0 : 3[3] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:108:651 [2] NCCL INFO Channel 18/0 : 2[2] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:112:653 [6] NCCL INFO Channel 03/0 : 6[6] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:110:649 [4] NCCL INFO Channel 07/0 : 4[4] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:111:647 [5] NCCL INFO Channel 10/0 : 5[5] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:107:650 [1] NCCL INFO Channel 18/0 : 1[1] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:109:652 [3] NCCL INFO Channel 18/0 : 3[3] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:108:651 [2] NCCL INFO Channel 19/0 : 2[2] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:112:653 [6] NCCL INFO Channel 04/0 : 6[6] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:110:649 [4] NCCL INFO Channel 08/0 : 4[4] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:111:647 [5] NCCL INFO Channel 11/0 : 5[5] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:107:650 [1] NCCL INFO Channel 19/0 : 1[1] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:109:652 [3] NCCL INFO Channel 19/0 : 3[3] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:108:651 [2] NCCL INFO Channel 20/0 : 2[2] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:112:653 [6] NCCL INFO Channel 05/0 : 6[6] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:110:649 [4] NCCL INFO Channel 09/0 : 4[4] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:111:647 [5] NCCL INFO Channel 12/0 : 5[5] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:107:650 [1] NCCL INFO Channel 20/0 : 1[1] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:109:652 [3] NCCL INFO Channel 20/0 : 3[3] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:112:653 [6] NCCL INFO Channel 06/0 : 6[6] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:108:651 [2] NCCL INFO Channel 21/0 : 2[2] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:110:649 [4] NCCL INFO Channel 10/0 : 4[4] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:111:647 [5] NCCL INFO Channel 13/0 : 5[5] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:107:650 [1] NCCL INFO Channel 21/0 : 1[1] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:109:652 [3] NCCL INFO Channel 21/0 : 3[3] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:112:653 [6] NCCL INFO Channel 07/0 : 6[6] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:108:651 [2] NCCL INFO Channel 22/0 : 2[2] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:110:649 [4] NCCL INFO Channel 11/0 : 4[4] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:111:647 [5] NCCL INFO Channel 14/0 : 5[5] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:107:650 [1] NCCL INFO Channel 22/0 : 1[1] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:109:652 [3] NCCL INFO Channel 22/0 : 3[3] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:112:653 [6] NCCL INFO Channel 08/0 : 6[6] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:108:651 [2] NCCL INFO Channel 23/0 : 2[2] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:110:649 [4] NCCL INFO Channel 12/0 : 4[4] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:111:647 [5] NCCL INFO Channel 15/0 : 5[5] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:107:650 [1] NCCL INFO Channel 23/0 : 1[1] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:109:652 [3] NCCL INFO Channel 23/0 : 3[3] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:112:653 [6] NCCL INFO Channel 09/0 : 6[6] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:110:649 [4] NCCL INFO Channel 13/0 : 4[4] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:111:647 [5] NCCL INFO Channel 16/0 : 5[5] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:112:653 [6] NCCL INFO Channel 10/0 : 6[6] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:110:649 [4] NCCL INFO Channel 14/0 : 4[4] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:111:647 [5] NCCL INFO Channel 17/0 : 5[5] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:110:649 [4] NCCL INFO Channel 15/0 : 4[4] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:112:653 [6] NCCL INFO Channel 11/0 : 6[6] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:111:647 [5] NCCL INFO Channel 18/0 : 5[5] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:110:649 [4] NCCL INFO Channel 16/0 : 4[4] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:112:653 [6] NCCL INFO Channel 12/0 : 6[6] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:111:647 [5] NCCL INFO Channel 19/0 : 5[5] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:110:649 [4] NCCL INFO Channel 17/0 : 4[4] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:112:653 [6] NCCL INFO Channel 13/0 : 6[6] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:111:647 [5] NCCL INFO Channel 20/0 : 5[5] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:110:649 [4] NCCL INFO Channel 18/0 : 4[4] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:112:653 [6] NCCL INFO Channel 14/0 : 6[6] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:111:647 [5] NCCL INFO Channel 21/0 : 5[5] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:110:649 [4] NCCL INFO Channel 19/0 : 4[4] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:112:653 [6] NCCL INFO Channel 15/0 : 6[6] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:111:647 [5] NCCL INFO Channel 22/0 : 5[5] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:110:649 [4] NCCL INFO Channel 20/0 : 4[4] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:112:653 [6] NCCL INFO Channel 16/0 : 6[6] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:111:647 [5] NCCL INFO Channel 23/0 : 5[5] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:110:649 [4] NCCL INFO Channel 21/0 : 4[4] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:112:653 [6] NCCL INFO Channel 17/0 : 6[6] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:110:649 [4] NCCL INFO Channel 22/0 : 4[4] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:112:653 [6] NCCL INFO Channel 18/0 : 6[6] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:110:649 [4] NCCL INFO Channel 23/0 : 4[4] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:112:653 [6] NCCL INFO Channel 19/0 : 6[6] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:112:653 [6] NCCL INFO Channel 20/0 : 6[6] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:112:653 [6] NCCL INFO Channel 21/0 : 6[6] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:112:653 [6] NCCL INFO Channel 22/0 : 6[6] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:112:653 [6] NCCL INFO Channel 23/0 : 6[6] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:84:646 [0] NCCL INFO Connected all trees
tyler-rhel-newimage:84:646 [0] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512
tyler-rhel-newimage:84:646 [0] NCCL INFO 24 coll channels, 0 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer
tyler-rhel-newimage:107:650 [1] NCCL INFO Connected all trees
tyler-rhel-newimage:107:650 [1] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512
tyler-rhel-newimage:107:650 [1] NCCL INFO 24 coll channels, 0 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer
tyler-rhel-newimage:108:651 [2] NCCL INFO Connected all trees
tyler-rhel-newimage:111:647 [5] NCCL INFO Connected all trees
tyler-rhel-newimage:108:651 [2] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512
tyler-rhel-newimage:111:647 [5] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512
tyler-rhel-newimage:108:651 [2] NCCL INFO 24 coll channels, 0 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer
tyler-rhel-newimage:111:647 [5] NCCL INFO 24 coll channels, 0 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer
tyler-rhel-newimage:113:648 [7] NCCL INFO Connected all trees
tyler-rhel-newimage:110:649 [4] NCCL INFO Connected all trees
tyler-rhel-newimage:113:648 [7] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512
tyler-rhel-newimage:113:648 [7] NCCL INFO 24 coll channels, 0 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer
tyler-rhel-newimage:110:649 [4] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512
tyler-rhel-newimage:112:653 [6] NCCL INFO Connected all trees
tyler-rhel-newimage:110:649 [4] NCCL INFO 24 coll channels, 0 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer
tyler-rhel-newimage:109:652 [3] NCCL INFO Connected all trees
tyler-rhel-newimage:112:653 [6] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512
tyler-rhel-newimage:112:653 [6] NCCL INFO 24 coll channels, 0 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer
tyler-rhel-newimage:109:652 [3] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512
tyler-rhel-newimage:109:652 [3] NCCL INFO 24 coll channels, 0 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer
INFO 2024-07-27 18:36:37,726 instructlab.model.backends.backends:336: Waiting for the vLLM server to start at http://127.0.0.1:60583/v1, this might take a moment... Attempt: 8/80
tyler-rhel-newimage:110:649 [4] NCCL INFO comm 0x5606496c3d80 rank 4 nranks 8 cudaDev 4 nvmlDev 4 busId c050 commId 0xb3f11bd99f7ae044 - Init COMPLETE
tyler-rhel-newimage:108:651 [2] NCCL INFO comm 0x5606496c1f20 rank 2 nranks 8 cudaDev 2 nvmlDev 2 busId a030 commId 0xb3f11bd99f7ae044 - Init COMPLETE
tyler-rhel-newimage:84:646 [0] NCCL INFO comm 0x5606496a0560 rank 0 nranks 8 cudaDev 0 nvmlDev 0 busId 8010 commId 0xb3f11bd99f7ae044 - Init COMPLETE
tyler-rhel-newimage:107:650 [1] NCCL INFO comm 0x5606496c0b00 rank 1 nranks 8 cudaDev 1 nvmlDev 1 busId 8020 commId 0xb3f11bd99f7ae044 - Init COMPLETE
tyler-rhel-newimage:113:648 [7] NCCL INFO comm 0x5606496ad9a0 rank 7 nranks 8 cudaDev 7 nvmlDev 7 busId e080 commId 0xb3f11bd99f7ae044 - Init COMPLETE
tyler-rhel-newimage:109:652 [3] NCCL INFO comm 0x5606496c31a0 rank 3 nranks 8 cudaDev 3 nvmlDev 3 busId a040 commId 0xb3f11bd99f7ae044 - Init COMPLETE
tyler-rhel-newimage:111:647 [5] NCCL INFO comm 0x5606496c43b0 rank 5 nranks 8 cudaDev 5 nvmlDev 5 busId c060 commId 0xb3f11bd99f7ae044 - Init COMPLETE
tyler-rhel-newimage:112:653 [6] NCCL INFO comm 0x5606496c38f0 rank 6 nranks 8 cudaDev 6 nvmlDev 6 busId e070 commId 0xb3f11bd99f7ae044 - Init COMPLETE
tyler-rhel-newimage:110:678 [4] NCCL INFO Channel 00/1 : 4[4] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:109:679 [3] NCCL INFO Channel 00/1 : 3[3] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:111:680 [5] NCCL INFO Channel 00/1 : 5[5] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:110:678 [4] NCCL INFO Channel 01/1 : 4[4] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:113:681 [7] NCCL INFO Channel 00/1 : 7[7] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:108:682 [2] NCCL INFO Channel 00/1 : 2[2] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:110:678 [4] NCCL INFO Channel 02/1 : 4[4] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:111:680 [5] NCCL INFO Channel 01/1 : 5[5] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:109:679 [3] NCCL INFO Channel 01/1 : 3[3] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:113:681 [7] NCCL INFO Channel 01/1 : 7[7] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:110:678 [4] NCCL INFO Channel 03/1 : 4[4] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:107:683 [1] NCCL INFO Channel 00/1 : 1[1] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:108:682 [2] NCCL INFO Channel 01/1 : 2[2] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:111:680 [5] NCCL INFO Channel 02/1 : 5[5] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:109:679 [3] NCCL INFO Channel 02/1 : 3[3] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:113:681 [7] NCCL INFO Channel 02/1 : 7[7] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:110:678 [4] NCCL INFO Channel 04/1 : 4[4] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:111:680 [5] NCCL INFO Channel 03/1 : 5[5] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:107:683 [1] NCCL INFO Channel 01/1 : 1[1] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:108:682 [2] NCCL INFO Channel 02/1 : 2[2] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:109:679 [3] NCCL INFO Channel 03/1 : 3[3] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:113:681 [7] NCCL INFO Channel 03/1 : 7[7] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:110:678 [4] NCCL INFO Channel 05/1 : 4[4] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:111:680 [5] NCCL INFO Channel 04/1 : 5[5] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:109:679 [3] NCCL INFO Channel 04/1 : 3[3] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:107:683 [1] NCCL INFO Channel 02/1 : 1[1] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:113:681 [7] NCCL INFO Channel 04/1 : 7[7] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:108:682 [2] NCCL INFO Channel 03/1 : 2[2] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:111:680 [5] NCCL INFO Channel 05/1 : 5[5] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:110:678 [4] NCCL INFO Channel 06/1 : 4[4] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:112:685 [6] NCCL INFO Channel 00/1 : 6[6] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:109:679 [3] NCCL INFO Channel 05/1 : 3[3] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:113:681 [7] NCCL INFO Channel 05/1 : 7[7] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:107:683 [1] NCCL INFO Channel 03/1 : 1[1] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:108:682 [2] NCCL INFO Channel 04/1 : 2[2] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:111:680 [5] NCCL INFO Channel 06/1 : 5[5] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:110:678 [4] NCCL INFO Channel 07/1 : 4[4] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:109:679 [3] NCCL INFO Channel 06/1 : 3[3] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:112:685 [6] NCCL INFO Channel 01/1 : 6[6] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:113:681 [7] NCCL INFO Channel 06/1 : 7[7] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:108:682 [2] NCCL INFO Channel 05/1 : 2[2] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:107:683 [1] NCCL INFO Channel 04/1 : 1[1] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:111:680 [5] NCCL INFO Channel 07/1 : 5[5] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:110:678 [4] NCCL INFO Channel 08/1 : 4[4] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:109:679 [3] NCCL INFO Channel 07/1 : 3[3] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:112:685 [6] NCCL INFO Channel 02/1 : 6[6] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:113:681 [7] NCCL INFO Channel 07/1 : 7[7] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:107:683 [1] NCCL INFO Channel 05/1 : 1[1] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:111:680 [5] NCCL INFO Channel 08/1 : 5[5] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:110:678 [4] NCCL INFO Channel 09/1 : 4[4] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:108:682 [2] NCCL INFO Channel 06/1 : 2[2] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:109:679 [3] NCCL INFO Channel 08/1 : 3[3] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:112:685 [6] NCCL INFO Channel 03/1 : 6[6] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:113:681 [7] NCCL INFO Channel 08/1 : 7[7] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:111:680 [5] NCCL INFO Channel 09/1 : 5[5] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:110:678 [4] NCCL INFO Channel 10/1 : 4[4] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:107:683 [1] NCCL INFO Channel 06/1 : 1[1] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:108:682 [2] NCCL INFO Channel 07/1 : 2[2] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:109:679 [3] NCCL INFO Channel 09/1 : 3[3] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:112:685 [6] NCCL INFO Channel 04/1 : 6[6] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:113:681 [7] NCCL INFO Channel 09/1 : 7[7] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:111:680 [5] NCCL INFO Channel 10/1 : 5[5] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:110:678 [4] NCCL INFO Channel 11/1 : 4[4] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:108:682 [2] NCCL INFO Channel 08/1 : 2[2] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:107:683 [1] NCCL INFO Channel 07/1 : 1[1] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:109:679 [3] NCCL INFO Channel 10/1 : 3[3] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:113:681 [7] NCCL INFO Channel 10/1 : 7[7] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:112:685 [6] NCCL INFO Channel 05/1 : 6[6] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:111:680 [5] NCCL INFO Channel 11/1 : 5[5] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:110:678 [4] NCCL INFO Channel 12/1 : 4[4] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:108:682 [2] NCCL INFO Channel 09/1 : 2[2] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:107:683 [1] NCCL INFO Channel 08/1 : 1[1] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:109:679 [3] NCCL INFO Channel 11/1 : 3[3] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:113:681 [7] NCCL INFO Channel 11/1 : 7[7] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:111:680 [5] NCCL INFO Channel 12/1 : 5[5] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:112:685 [6] NCCL INFO Channel 06/1 : 6[6] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:110:678 [4] NCCL INFO Channel 13/1 : 4[4] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:108:682 [2] NCCL INFO Channel 10/1 : 2[2] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:109:679 [3] NCCL INFO Channel 12/1 : 3[3] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:107:683 [1] NCCL INFO Channel 09/1 : 1[1] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:112:685 [6] NCCL INFO Channel 07/1 : 6[6] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:113:681 [7] NCCL INFO Channel 12/1 : 7[7] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:111:680 [5] NCCL INFO Channel 13/1 : 5[5] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:110:678 [4] NCCL INFO Channel 14/1 : 4[4] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:108:682 [2] NCCL INFO Channel 11/1 : 2[2] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:109:679 [3] NCCL INFO Channel 13/1 : 3[3] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:107:683 [1] NCCL INFO Channel 10/1 : 1[1] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:113:681 [7] NCCL INFO Channel 13/1 : 7[7] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:112:685 [6] NCCL INFO Channel 08/1 : 6[6] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:111:680 [5] NCCL INFO Channel 14/1 : 5[5] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:110:678 [4] NCCL INFO Channel 15/1 : 4[4] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:108:682 [2] NCCL INFO Channel 12/1 : 2[2] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:109:679 [3] NCCL INFO Channel 14/1 : 3[3] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:107:683 [1] NCCL INFO Channel 11/1 : 1[1] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:113:681 [7] NCCL INFO Channel 14/1 : 7[7] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:112:685 [6] NCCL INFO Channel 09/1 : 6[6] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:111:680 [5] NCCL INFO Channel 15/1 : 5[5] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:110:678 [4] NCCL INFO Channel 16/1 : 4[4] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:108:682 [2] NCCL INFO Channel 13/1 : 2[2] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:109:679 [3] NCCL INFO Channel 15/1 : 3[3] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:112:685 [6] NCCL INFO Channel 10/1 : 6[6] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:113:681 [7] NCCL INFO Channel 15/1 : 7[7] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:107:683 [1] NCCL INFO Channel 12/1 : 1[1] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:111:680 [5] NCCL INFO Channel 16/1 : 5[5] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:110:678 [4] NCCL INFO Channel 17/1 : 4[4] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:108:682 [2] NCCL INFO Channel 14/1 : 2[2] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:109:679 [3] NCCL INFO Channel 16/1 : 3[3] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:113:681 [7] NCCL INFO Channel 16/1 : 7[7] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:112:685 [6] NCCL INFO Channel 11/1 : 6[6] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:110:678 [4] NCCL INFO Channel 18/1 : 4[4] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:111:680 [5] NCCL INFO Channel 17/1 : 5[5] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:107:683 [1] NCCL INFO Channel 13/1 : 1[1] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:108:682 [2] NCCL INFO Channel 15/1 : 2[2] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:109:679 [3] NCCL INFO Channel 17/1 : 3[3] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:113:681 [7] NCCL INFO Channel 17/1 : 7[7] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:112:685 [6] NCCL INFO Channel 12/1 : 6[6] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:110:678 [4] NCCL INFO Channel 19/1 : 4[4] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:111:680 [5] NCCL INFO Channel 18/1 : 5[5] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:107:683 [1] NCCL INFO Channel 14/1 : 1[1] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:108:682 [2] NCCL INFO Channel 16/1 : 2[2] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:109:679 [3] NCCL INFO Channel 18/1 : 3[3] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:113:681 [7] NCCL INFO Channel 18/1 : 7[7] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:110:678 [4] NCCL INFO Channel 20/1 : 4[4] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:112:685 [6] NCCL INFO Channel 13/1 : 6[6] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:111:680 [5] NCCL INFO Channel 19/1 : 5[5] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:109:679 [3] NCCL INFO Channel 19/1 : 3[3] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:107:683 [1] NCCL INFO Channel 15/1 : 1[1] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:113:681 [7] NCCL INFO Channel 19/1 : 7[7] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:108:682 [2] NCCL INFO Channel 17/1 : 2[2] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:111:680 [5] NCCL INFO Channel 20/1 : 5[5] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:112:685 [6] NCCL INFO Channel 14/1 : 6[6] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:110:678 [4] NCCL INFO Channel 21/1 : 4[4] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:109:679 [3] NCCL INFO Channel 20/1 : 3[3] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:113:681 [7] NCCL INFO Channel 20/1 : 7[7] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:107:683 [1] NCCL INFO Channel 16/1 : 1[1] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:108:682 [2] NCCL INFO Channel 18/1 : 2[2] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:111:680 [5] NCCL INFO Channel 21/1 : 5[5] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:112:685 [6] NCCL INFO Channel 15/1 : 6[6] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:110:678 [4] NCCL INFO Channel 22/1 : 4[4] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:109:679 [3] NCCL INFO Channel 21/1 : 3[3] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:113:681 [7] NCCL INFO Channel 21/1 : 7[7] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:107:683 [1] NCCL INFO Channel 17/1 : 1[1] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:108:682 [2] NCCL INFO Channel 19/1 : 2[2] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:111:680 [5] NCCL INFO Channel 22/1 : 5[5] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:110:678 [4] NCCL INFO Channel 23/1 : 4[4] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:112:685 [6] NCCL INFO Channel 16/1 : 6[6] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:109:679 [3] NCCL INFO Channel 22/1 : 3[3] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:113:681 [7] NCCL INFO Channel 22/1 : 7[7] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:107:683 [1] NCCL INFO Channel 18/1 : 1[1] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:111:680 [5] NCCL INFO Channel 23/1 : 5[5] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:108:682 [2] NCCL INFO Channel 20/1 : 2[2] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:110:678 [4] NCCL INFO Channel 24/1 : 4[4] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:112:685 [6] NCCL INFO Channel 17/1 : 6[6] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:109:679 [3] NCCL INFO Channel 23/1 : 3[3] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:113:681 [7] NCCL INFO Channel 23/1 : 7[7] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:111:680 [5] NCCL INFO Channel 24/1 : 5[5] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:107:683 [1] NCCL INFO Channel 19/1 : 1[1] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:110:678 [4] NCCL INFO Channel 25/1 : 4[4] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:108:682 [2] NCCL INFO Channel 21/1 : 2[2] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:112:685 [6] NCCL INFO Channel 18/1 : 6[6] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:109:679 [3] NCCL INFO Channel 24/1 : 3[3] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:113:681 [7] NCCL INFO Channel 24/1 : 7[7] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:111:680 [5] NCCL INFO Channel 25/1 : 5[5] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:107:683 [1] NCCL INFO Channel 20/1 : 1[1] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:110:678 [4] NCCL INFO Channel 26/1 : 4[4] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:108:682 [2] NCCL INFO Channel 22/1 : 2[2] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:112:685 [6] NCCL INFO Channel 19/1 : 6[6] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:109:679 [3] NCCL INFO Channel 25/1 : 3[3] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:113:681 [7] NCCL INFO Channel 25/1 : 7[7] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:111:680 [5] NCCL INFO Channel 26/1 : 5[5] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:110:678 [4] NCCL INFO Channel 27/1 : 4[4] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:109:679 [3] NCCL INFO Channel 26/1 : 3[3] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:113:681 [7] NCCL INFO Channel 26/1 : 7[7] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:111:680 [5] NCCL INFO Channel 27/1 : 5[5] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:110:678 [4] NCCL INFO Channel 28/1 : 4[4] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:109:679 [3] NCCL INFO Channel 27/1 : 3[3] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:113:681 [7] NCCL INFO Channel 27/1 : 7[7] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:111:680 [5] NCCL INFO Channel 28/1 : 5[5] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:110:678 [4] NCCL INFO Channel 29/1 : 4[4] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:109:679 [3] NCCL INFO Channel 28/1 : 3[3] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:113:681 [7] NCCL INFO Channel 28/1 : 7[7] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:111:680 [5] NCCL INFO Channel 29/1 : 5[5] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:110:678 [4] NCCL INFO Channel 30/1 : 4[4] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:109:679 [3] NCCL INFO Channel 29/1 : 3[3] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:113:681 [7] NCCL INFO Channel 29/1 : 7[7] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:111:680 [5] NCCL INFO Channel 30/1 : 5[5] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:110:678 [4] NCCL INFO Channel 31/1 : 4[4] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:109:679 [3] NCCL INFO Channel 30/1 : 3[3] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:113:681 [7] NCCL INFO Channel 30/1 : 7[7] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:111:680 [5] NCCL INFO Channel 31/1 : 5[5] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:107:683 [1] NCCL INFO Channel 21/1 : 1[1] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:109:679 [3] NCCL INFO Channel 31/1 : 3[3] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:113:681 [7] NCCL INFO Channel 31/1 : 7[7] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:107:683 [1] NCCL INFO Channel 22/1 : 1[1] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:108:682 [2] NCCL INFO Channel 23/1 : 2[2] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:107:683 [1] NCCL INFO Channel 23/1 : 1[1] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:112:685 [6] NCCL INFO Channel 20/1 : 6[6] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:108:682 [2] NCCL INFO Channel 24/1 : 2[2] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:107:683 [1] NCCL INFO Channel 24/1 : 1[1] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:112:685 [6] NCCL INFO Channel 21/1 : 6[6] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:108:682 [2] NCCL INFO Channel 25/1 : 2[2] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:107:683 [1] NCCL INFO Channel 25/1 : 1[1] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:112:685 [6] NCCL INFO Channel 22/1 : 6[6] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:108:682 [2] NCCL INFO Channel 26/1 : 2[2] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:107:683 [1] NCCL INFO Channel 26/1 : 1[1] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:112:685 [6] NCCL INFO Channel 23/1 : 6[6] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:108:682 [2] NCCL INFO Channel 27/1 : 2[2] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:112:685 [6] NCCL INFO Channel 24/1 : 6[6] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:107:683 [1] NCCL INFO Channel 27/1 : 1[1] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:108:682 [2] NCCL INFO Channel 28/1 : 2[2] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:112:685 [6] NCCL INFO Channel 25/1 : 6[6] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:107:683 [1] NCCL INFO Channel 28/1 : 1[1] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:108:682 [2] NCCL INFO Channel 29/1 : 2[2] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:112:685 [6] NCCL INFO Channel 26/1 : 6[6] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:107:683 [1] NCCL INFO Channel 29/1 : 1[1] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:108:682 [2] NCCL INFO Channel 30/1 : 2[2] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:112:685 [6] NCCL INFO Channel 27/1 : 6[6] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:107:683 [1] NCCL INFO Channel 30/1 : 1[1] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:108:682 [2] NCCL INFO Channel 31/1 : 2[2] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:112:685 [6] NCCL INFO Channel 28/1 : 6[6] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:107:683 [1] NCCL INFO Channel 31/1 : 1[1] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:112:685 [6] NCCL INFO Channel 29/1 : 6[6] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:112:685 [6] NCCL INFO Channel 30/1 : 6[6] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:112:685 [6] NCCL INFO Channel 31/1 : 6[6] -> 0[0] via P2P/IPC/read
INFO 2024-07-27 18:36:42,438 instructlab.model.backends.backends:336: Waiting for the vLLM server to start at http://127.0.0.1:60583/v1, this might take a moment... Attempt: 9/80
INFO 2024-07-27 18:36:47,110 instructlab.model.backends.backends:336: Waiting for the vLLM server to start at http://127.0.0.1:60583/v1, this might take a moment... Attempt: 10/80
INFO 2024-07-27 18:36:51,757 instructlab.model.backends.backends:336: Waiting for the vLLM server to start at http://127.0.0.1:60583/v1, this might take a moment... Attempt: 11/80
INFO 2024-07-27 18:36:56,647 instructlab.model.backends.backends:336: Waiting for the vLLM server to start at http://127.0.0.1:60583/v1, this might take a moment... Attempt: 12/80
embedding_mode is False. Embedding API will not work.
INFO: Started server process [84]
INFO: Waiting for application startup.
INFO: Application startup complete.
INFO: Uvicorn running on http://127.0.0.1:60583 (Press CTRL+C to quit)
INFO 2024-07-27 18:37:01,152 instructlab.model.backends.backends:336: Waiting for the vLLM server to start at http://127.0.0.1:60583/v1, this might take a moment... Attempt: 13/80
INFO: 127.0.0.1:55172 - "GET /v1/models HTTP/1.1" 200 OK
INFO 2024-07-27 18:37:01,164 instructlab.model.backends.backends:343: vLLM engine successfully started at http://127.0.0.1:60583/v1
0%| | 0/80 [00:00<?, ?it/s]INFO: 127.0.0.1:55284 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:55284 - "POST /v1/chat/completions HTTP/1.1" 200 OK
1%|██▌ | 1/80 [00:00<00:11, 7.12it/s]INFO: 127.0.0.1:55206 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:55318 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:55260 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:55192 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:55302 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:55318 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2%|█████ | 2/80 [00:00<00:18, 4.11it/s]INFO: 127.0.0.1:55206 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:55270 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:55284 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:55260 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:55192 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:55224 - "POST /v1/chat/completions HTTP/1.1" 200 OK
5%|██████████ | 4/80 [00:00<00:12, 5.96it/s]INFO: 127.0.0.1:55330 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:55186 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:55244 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:55270 - "POST /v1/chat/completions HTTP/1.1" 200 OK
8%|███████████████▏ | 6/80 [00:00<00:10, 7.07it/s]INFO: 127.0.0.1:55236 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:55260 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:55300 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:55318 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:55248 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:55206 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:55302 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:55224 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:55306 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:55192 - "POST /v1/chat/completions HTTP/1.1" 200 OK
9%|█████████████████▋ | 7/80 [00:01<00:11, 6.10it/s]INFO: 127.0.0.1:55284 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:55212 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:55260 - "POST /v1/chat/completions HTTP/1.1" 200 OK
12%|█████████████████████████▏ | 10/80 [00:01<00:07, 9.80it/s]INFO: 127.0.0.1:55260 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:55302 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:55270 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:55306 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:55186 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:55260 - "POST /v1/chat/completions HTTP/1.1" 200 OK
15%|██████████████████████████████▏ | 12/80 [00:01<00:08, 8.21it/s]INFO: 127.0.0.1:55302 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:55212 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:55270 - "POST /v1/chat/completions HTTP/1.1" 200 OK
19%|█████████████████████████████████████▋ | 15/80 [00:01<00:05, 11.38it/s]INFO: 127.0.0.1:55284 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:55206 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:55306 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:55186 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:55302 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:55318 - "POST /v1/chat/completions HTTP/1.1" 200 OK
24%|███████████████████████████████████████████████▋ | 19/80 [00:01<00:04, 14.91it/s]INFO: 127.0.0.1:55300 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:55248 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:55244 - "POST /v1/chat/completions HTTP/1.1" 200 OK
28%|███████████████████████████████████████████████████████▎ | 22/80 [00:02<00:03, 16.92it/s]INFO: 127.0.0.1:55260 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:55236 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:55302 - "POST /v1/chat/completions HTTP/1.1" 200 OK
30%|████████████████████████████████████████████████████████████▎ | 24/80 [00:02<00:03, 17.46it/s]INFO: 127.0.0.1:55206 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:55244 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:55260 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:55224 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:55302 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:55330 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:55306 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:55300 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:55236 - "POST /v1/chat/completions HTTP/1.1" 200 OK
32%|█████████████████████████████████████████████████████████████████▎ | 26/80 [00:02<00:04, 12.88it/s]INFO: 127.0.0.1:55248 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:55186 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:55192 - "POST /v1/chat/completions HTTP/1.1" 200 OK
35%|██████████████████████████████████████████████████████████████████████▎ | 28/80 [00:02<00:03, 13.26it/s]INFO: 127.0.0.1:55212 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:55302 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:55206 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:55318 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:55244 - "POST /v1/chat/completions HTTP/1.1" 200 OK
39%|█████████████████████████████████████████████████████████████████████████████▉ | 31/80 [00:02<00:03, 14.51it/s]INFO: 127.0.0.1:55236 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:55330 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:55260 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:55300 - "POST /v1/chat/completions HTTP/1.1" 200 OK
41%|██████████████████████████████████████████████████████████████████████████████████▉ | 33/80 [00:02<00:03, 12.59it/s]INFO: 127.0.0.1:55248 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:55192 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:55306 - "POST /v1/chat/completions HTTP/1.1" 200 OK
44%|███████████████████████████████████████████████████████████████████████████████████████▉ | 35/80 [00:03<00:03, 13.67it/s]INFO: 127.0.0.1:55318 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:55192 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:55302 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:55330 - "POST /v1/chat/completions HTTP/1.1" 200 OK
48%|███████████████████████████████████████████████████████████████████████████████████████████████▍ | 38/80 [00:03<00:02, 15.58it/s]INFO: 127.0.0.1:55284 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:55212 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:55236 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:55260 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:55248 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:55186 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:55270 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:55224 - "POST /v1/chat/completions HTTP/1.1" 200 OK
51%|███████████████████████████████████████████████████████████████████████████████████████████████████████ | 41/80 [00:03<00:02, 14.88it/s]INFO: 127.0.0.1:55212 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:55302 - "POST /v1/chat/completions HTTP/1.1" 200 OK
54%|████████████████████████████████████████████████████████████████████████████████████████████████████████████ | 43/80 [00:03<00:02, 14.09it/s]INFO: 127.0.0.1:55330 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:55206 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:55236 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:55244 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:55306 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:55224 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:55302 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:55318 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:55330 - "POST /v1/chat/completions HTTP/1.1" 200 OK
56%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████ | 45/80 [00:03<00:03, 9.43it/s]INFO: 127.0.0.1:55224 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:55186 - "POST /v1/chat/completions HTTP/1.1" 200 OK
59%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████ | 47/80 [00:04<00:03, 10.38it/s]INFO: 127.0.0.1:55206 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:55302 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:55236 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:55248 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:55284 - "POST /v1/chat/completions HTTP/1.1" 200 OK
62%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▋ | 50/80 [00:04<00:02, 10.79it/s]INFO: 127.0.0.1:55212 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:55330 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:55244 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:55306 - "POST /v1/chat/completions HTTP/1.1" 200 OK
65%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▋ | 52/80 [00:04<00:02, 9.75it/s]INFO: 127.0.0.1:55248 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:55300 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:55192 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:55302 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:55236 - "POST /v1/chat/completions HTTP/1.1" 200 OK
68%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▋ | 54/80 [00:04<00:02, 9.03it/s]INFO: 127.0.0.1:55318 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:55284 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:55300 - "POST /v1/chat/completions HTTP/1.1" 200 OK
70%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▋ | 56/80 [00:05<00:02, 9.97it/s]INFO: 127.0.0.1:55248 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:55330 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:55192 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:55212 - "POST /v1/chat/completions HTTP/1.1" 200 OK
74%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▏ | 59/80 [00:05<00:01, 13.11it/s]INFO: 127.0.0.1:55206 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:55244 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:55306 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:55224 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:55284 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:55212 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:55248 - "POST /v1/chat/completions HTTP/1.1" 200 OK
76%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▎ | 61/80 [00:05<00:01, 11.74it/s]INFO: 127.0.0.1:55302 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:55318 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:55192 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:55306 - "POST /v1/chat/completions HTTP/1.1" 200 OK
79%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▎ | 63/80 [00:05<00:01, 11.35it/s]INFO: 127.0.0.1:55330 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:55212 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:55270 - "POST /v1/chat/completions HTTP/1.1" 200 OK
81%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▎ | 65/80 [00:05<00:01, 12.17it/s]INFO: 127.0.0.1:55300 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:55192 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:55306 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:55244 - "POST /v1/chat/completions HTTP/1.1" 200 OK
84%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▎ | 67/80 [00:05<00:01, 11.98it/s]INFO: 127.0.0.1:55318 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:55224 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:55300 - "POST /v1/chat/completions HTTP/1.1" 200 OK
86%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▎ | 69/80 [00:06<00:00, 12.75it/s]INFO: 127.0.0.1:55330 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:55236 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:55302 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:55186 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:55206 - "POST /v1/chat/completions HTTP/1.1" 200 OK
90%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▉ | 72/80 [00:06<00:00, 13.05it/s]INFO: 127.0.0.1:55248 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:55192 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:55236 - "POST /v1/chat/completions HTTP/1.1" 200 OK
92%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▉ | 74/80 [00:06<00:00, 8.86it/s]INFO: 127.0.0.1:55284 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:55212 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:55260 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:55224 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:55212 - "POST /v1/chat/completions HTTP/1.1" 200 OK
95%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▉ | 76/80 [00:07<00:00, 7.59it/s]INFO: 127.0.0.1:55192 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:55236 - "POST /v1/chat/completions HTTP/1.1" 200 OK
98%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▉ | 78/80 [00:07<00:00, 8.83it/s]INFO: 127.0.0.1:55206 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:55186 - "POST /v1/chat/completions HTTP/1.1" 200 OK
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 80/80 [00:12<00:00, 6.19it/s]
INFO: Shutting down
INFO: Waiting for application shutdown.
INFO: Application shutdown complete.
INFO: Finished server process [84]
[rank0]:[W CudaIPCTypes.cpp:16] Producer process has been terminated before all shared CUDA tensors released. See Note [Sharing CUDA tensors]
/usr/lib64/python3.11/multiprocessing/resource_tracker.py:254: UserWarning: resource_tracker: There appear to be 1 leaked shared_memory objects to clean up at shutdown
warnings.warn('resource_tracker: There appear to be %d '
Evaluating answers...
WARNING 2024-07-27 18:37:26,124 instructlab.model.evaluate:288: Based on your hardware configuration, when using vLLM, we recommend setting max-workers between 53.0 and 160.0 for optimal performance
INFO 2024-07-27 18:37:26,189 instructlab.model.backends.backends:315: Trying to connect to model server at http://127.0.0.1:8000/v1
INFO 2024-07-27 18:37:28,923 instructlab.model.backends.vllm:196: vLLM starting up on pid 801 at http://127.0.0.1:59587/v1
INFO 2024-07-27 18:37:28,923 instructlab.model.backends.backends:328: Starting a temporary vLLM server at http://127.0.0.1:59587/v1
INFO 2024-07-27 18:37:28,923 instructlab.model.backends.backends:336: Waiting for the vLLM server to start at http://127.0.0.1:59587/v1, this might take a moment... Attempt: 1/80
INFO 2024-07-27 18:37:33,466 instructlab.model.backends.backends:336: Waiting for the vLLM server to start at http://127.0.0.1:59587/v1, this might take a moment... Attempt: 2/80
INFO 2024-07-27 18:37:38,160 instructlab.model.backends.backends:336: Waiting for the vLLM server to start at http://127.0.0.1:59587/v1, this might take a moment... Attempt: 3/80
INFO 2024-07-27 18:37:42,743 instructlab.model.backends.backends:336: Waiting for the vLLM server to start at http://127.0.0.1:59587/v1, this might take a moment... Attempt: 4/80
tyler-rhel-newimage:801:801 [0] NCCL INFO Bootstrap : Using enp8s0:192.168.48.11<0>
tyler-rhel-newimage:801:801 [0] NCCL INFO NET/Plugin : dlerror=libnccl-net.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net.so), using internal implementation
tyler-rhel-newimage:801:801 [0] NCCL INFO cudaDriverVersion 12040
NCCL version 2.20.5+cuda12.4
tyler-rhel-newimage:827:827 [4] NCCL INFO cudaDriverVersion 12040
tyler-rhel-newimage:827:827 [4] NCCL INFO Bootstrap : Using enp8s0:192.168.48.11<0>
tyler-rhel-newimage:827:827 [4] NCCL INFO NET/Plugin : dlerror=libnccl-net.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net.so), using internal implementation
tyler-rhel-newimage:830:830 [7] NCCL INFO cudaDriverVersion 12040
tyler-rhel-newimage:830:830 [7] NCCL INFO Bootstrap : Using enp8s0:192.168.48.11<0>
tyler-rhel-newimage:830:830 [7] NCCL INFO NET/Plugin : dlerror=libnccl-net.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net.so), using internal implementation
tyler-rhel-newimage:828:828 [5] NCCL INFO cudaDriverVersion 12040
tyler-rhel-newimage:828:828 [5] NCCL INFO Bootstrap : Using enp8s0:192.168.48.11<0>
tyler-rhel-newimage:828:828 [5] NCCL INFO NET/Plugin : dlerror=libnccl-net.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net.so), using internal implementation
tyler-rhel-newimage:825:825 [2] NCCL INFO cudaDriverVersion 12040
tyler-rhel-newimage:825:825 [2] NCCL INFO Bootstrap : Using enp8s0:192.168.48.11<0>
tyler-rhel-newimage:825:825 [2] NCCL INFO NET/Plugin : dlerror=libnccl-net.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net.so), using internal implementation
tyler-rhel-newimage:824:824 [1] NCCL INFO cudaDriverVersion 12040
tyler-rhel-newimage:824:824 [1] NCCL INFO Bootstrap : Using enp8s0:192.168.48.11<0>
tyler-rhel-newimage:824:824 [1] NCCL INFO NET/Plugin : dlerror=libnccl-net.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net.so), using internal implementation
tyler-rhel-newimage:829:829 [6] NCCL INFO cudaDriverVersion 12040
tyler-rhel-newimage:829:829 [6] NCCL INFO Bootstrap : Using enp8s0:192.168.48.11<0>
tyler-rhel-newimage:826:826 [3] NCCL INFO cudaDriverVersion 12040
tyler-rhel-newimage:826:826 [3] NCCL INFO Bootstrap : Using enp8s0:192.168.48.11<0>
tyler-rhel-newimage:829:829 [6] NCCL INFO NET/Plugin : dlerror=libnccl-net.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net.so), using internal implementation
tyler-rhel-newimage:826:826 [3] NCCL INFO NET/Plugin : dlerror=libnccl-net.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net.so), using internal implementation
tyler-rhel-newimage:801:801 [0] NCCL INFO NET/IB : No device found.
tyler-rhel-newimage:801:801 [0] NCCL INFO NET/Socket : Using [0]enp8s0:192.168.48.11<0>
tyler-rhel-newimage:801:801 [0] NCCL INFO Using non-device net plugin version 0
tyler-rhel-newimage:801:801 [0] NCCL INFO Using network Socket
tyler-rhel-newimage:827:827 [4] NCCL INFO NET/IB : No device found.
tyler-rhel-newimage:827:827 [4] NCCL INFO NET/Socket : Using [0]enp8s0:192.168.48.11<0>
tyler-rhel-newimage:827:827 [4] NCCL INFO Using non-device net plugin version 0
tyler-rhel-newimage:827:827 [4] NCCL INFO Using network Socket
tyler-rhel-newimage:830:830 [7] NCCL INFO NET/IB : No device found.
tyler-rhel-newimage:830:830 [7] NCCL INFO NET/Socket : Using [0]enp8s0:192.168.48.11<0>
tyler-rhel-newimage:830:830 [7] NCCL INFO Using non-device net plugin version 0
tyler-rhel-newimage:830:830 [7] NCCL INFO Using network Socket
tyler-rhel-newimage:825:825 [2] NCCL INFO NET/IB : No device found.
tyler-rhel-newimage:825:825 [2] NCCL INFO NET/Socket : Using [0]enp8s0:192.168.48.11<0>
tyler-rhel-newimage:825:825 [2] NCCL INFO Using non-device net plugin version 0
tyler-rhel-newimage:825:825 [2] NCCL INFO Using network Socket
tyler-rhel-newimage:826:826 [3] NCCL INFO NET/IB : No device found.
tyler-rhel-newimage:826:826 [3] NCCL INFO NET/Socket : Using [0]enp8s0:192.168.48.11<0>
tyler-rhel-newimage:826:826 [3] NCCL INFO Using non-device net plugin version 0
tyler-rhel-newimage:826:826 [3] NCCL INFO Using network Socket
tyler-rhel-newimage:824:824 [1] NCCL INFO NET/IB : No device found.
tyler-rhel-newimage:824:824 [1] NCCL INFO NET/Socket : Using [0]enp8s0:192.168.48.11<0>
tyler-rhel-newimage:824:824 [1] NCCL INFO Using non-device net plugin version 0
tyler-rhel-newimage:824:824 [1] NCCL INFO Using network Socket
INFO 2024-07-27 18:37:47,235 instructlab.model.backends.backends:336: Waiting for the vLLM server to start at http://127.0.0.1:59587/v1, this might take a moment... Attempt: 5/80
tyler-rhel-newimage:828:828 [5] NCCL INFO NET/IB : No device found.
tyler-rhel-newimage:828:828 [5] NCCL INFO NET/Socket : Using [0]enp8s0:192.168.48.11<0>
tyler-rhel-newimage:828:828 [5] NCCL INFO Using non-device net plugin version 0
tyler-rhel-newimage:828:828 [5] NCCL INFO Using network Socket
tyler-rhel-newimage:829:829 [6] NCCL INFO NET/IB : No device found.
tyler-rhel-newimage:829:829 [6] NCCL INFO NET/Socket : Using [0]enp8s0:192.168.48.11<0>
tyler-rhel-newimage:829:829 [6] NCCL INFO Using non-device net plugin version 0
tyler-rhel-newimage:829:829 [6] NCCL INFO Using network Socket
tyler-rhel-newimage:828:828 [5] NCCL INFO comm 0x555b94f76f30 rank 5 nranks 8 cudaDev 5 nvmlDev 5 busId c060 commId 0xc1aaaefa85b64c46 - Init START
tyler-rhel-newimage:825:825 [2] NCCL INFO comm 0x555b94f76350 rank 2 nranks 8 cudaDev 2 nvmlDev 2 busId a030 commId 0xc1aaaefa85b64c46 - Init START
tyler-rhel-newimage:829:829 [6] NCCL INFO comm 0x555b94f77fe0 rank 6 nranks 8 cudaDev 6 nvmlDev 6 busId e070 commId 0xc1aaaefa85b64c46 - Init START
tyler-rhel-newimage:801:801 [0] NCCL INFO comm 0x555b94f7de20 rank 0 nranks 8 cudaDev 0 nvmlDev 0 busId 8010 commId 0xc1aaaefa85b64c46 - Init START
tyler-rhel-newimage:824:824 [1] NCCL INFO comm 0x555b94f76110 rank 1 nranks 8 cudaDev 1 nvmlDev 1 busId 8020 commId 0xc1aaaefa85b64c46 - Init START
tyler-rhel-newimage:827:827 [4] NCCL INFO comm 0x555b94f76b60 rank 4 nranks 8 cudaDev 4 nvmlDev 4 busId c050 commId 0xc1aaaefa85b64c46 - Init START
tyler-rhel-newimage:830:830 [7] NCCL INFO comm 0x555b94f79470 rank 7 nranks 8 cudaDev 7 nvmlDev 7 busId e080 commId 0xc1aaaefa85b64c46 - Init START
tyler-rhel-newimage:826:826 [3] NCCL INFO comm 0x555b94f76680 rank 3 nranks 8 cudaDev 3 nvmlDev 3 busId a040 commId 0xc1aaaefa85b64c46 - Init START
tyler-rhel-newimage:828:828 [5] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0.
tyler-rhel-newimage:824:824 [1] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0.
tyler-rhel-newimage:829:829 [6] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0.
tyler-rhel-newimage:825:825 [2] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0.
tyler-rhel-newimage:801:801 [0] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0.
tyler-rhel-newimage:827:827 [4] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0.
tyler-rhel-newimage:830:830 [7] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0.
tyler-rhel-newimage:828:828 [5] NCCL INFO Setting affinity for GPU 5 to ffff,ffffff00,00000000
tyler-rhel-newimage:828:828 [5] NCCL INFO NVLS multicast support is not available on dev 5
tyler-rhel-newimage:829:829 [6] NCCL INFO Setting affinity for GPU 6 to ffff,ffffff00,00000000
tyler-rhel-newimage:829:829 [6] NCCL INFO NVLS multicast support is not available on dev 6
tyler-rhel-newimage:826:826 [3] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0.
tyler-rhel-newimage:801:801 [0] NCCL INFO Setting affinity for GPU 0 to ff,ffffffff
tyler-rhel-newimage:801:801 [0] NCCL INFO NVLS multicast support is not available on dev 0
tyler-rhel-newimage:825:825 [2] NCCL INFO Setting affinity for GPU 2 to ff,ffffffff
tyler-rhel-newimage:824:824 [1] NCCL INFO Setting affinity for GPU 1 to ff,ffffffff
tyler-rhel-newimage:824:824 [1] NCCL INFO NVLS multicast support is not available on dev 1
tyler-rhel-newimage:825:825 [2] NCCL INFO NVLS multicast support is not available on dev 2
tyler-rhel-newimage:827:827 [4] NCCL INFO Setting affinity for GPU 4 to ffff,ffffff00,00000000
tyler-rhel-newimage:827:827 [4] NCCL INFO NVLS multicast support is not available on dev 4
tyler-rhel-newimage:830:830 [7] NCCL INFO Setting affinity for GPU 7 to ffff,ffffff00,00000000
tyler-rhel-newimage:830:830 [7] NCCL INFO NVLS multicast support is not available on dev 7
tyler-rhel-newimage:826:826 [3] NCCL INFO Setting affinity for GPU 3 to ff,ffffffff
tyler-rhel-newimage:826:826 [3] NCCL INFO NVLS multicast support is not available on dev 3
tyler-rhel-newimage:826:826 [3] NCCL INFO comm 0x555b94f76680 rank 3 nRanks 8 nNodes 1 localRanks 8 localRank 3 MNNVL 0
tyler-rhel-newimage:825:825 [2] NCCL INFO comm 0x555b94f76350 rank 2 nRanks 8 nNodes 1 localRanks 8 localRank 2 MNNVL 0
tyler-rhel-newimage:824:824 [1] NCCL INFO comm 0x555b94f76110 rank 1 nRanks 8 nNodes 1 localRanks 8 localRank 1 MNNVL 0
tyler-rhel-newimage:801:801 [0] NCCL INFO comm 0x555b94f7de20 rank 0 nRanks 8 nNodes 1 localRanks 8 localRank 0 MNNVL 0
tyler-rhel-newimage:830:830 [7] NCCL INFO comm 0x555b94f79470 rank 7 nRanks 8 nNodes 1 localRanks 8 localRank 7 MNNVL 0
tyler-rhel-newimage:827:827 [4] NCCL INFO comm 0x555b94f76b60 rank 4 nRanks 8 nNodes 1 localRanks 8 localRank 4 MNNVL 0
tyler-rhel-newimage:829:829 [6] NCCL INFO comm 0x555b94f77fe0 rank 6 nRanks 8 nNodes 1 localRanks 8 localRank 6 MNNVL 0
tyler-rhel-newimage:828:828 [5] NCCL INFO comm 0x555b94f76f30 rank 5 nRanks 8 nNodes 1 localRanks 8 localRank 5 MNNVL 0
tyler-rhel-newimage:801:801 [0] NCCL INFO Channel 00/24 : 0 1 2 3 4 5 6 7
tyler-rhel-newimage:801:801 [0] NCCL INFO Channel 01/24 : 0 1 2 3 4 5 6 7
tyler-rhel-newimage:801:801 [0] NCCL INFO Channel 02/24 : 0 1 2 3 4 5 6 7
tyler-rhel-newimage:825:825 [2] NCCL INFO Trees [0] 3/-1/-1->2->1 [1] 3/-1/-1->2->1 [2] 3/-1/-1->2->1 [3] 3/-1/-1->2->1 [4] 3/-1/-1->2->1 [5] 3/-1/-1->2->1 [6] 3/-1/-1->2->1 [7] 3/-1/-1->2->1 [8] 3/-1/-1->2->1 [9] 3/-1/-1->2->1 [10] 3/-1/-1->2->1 [11] 3/-1/-1->2->1 [12] 3/-1/-1->2->1 [13] 3/-1/-1->2->1 [14] 3/-1/-1->2->1 [15] 3/-1/-1->2->1 [16] 3/-1/-1->2->1 [17] 3/-1/-1->2->1 [18] 3/-1/-1->2->1 [19] 3/-1/-1->2->1 [20] 3/-1/-1->2->1 [21] 3/-1/-1->2->1 [22] 3/-1/-1->2->1 [23] 3/-1/-1->2->1
tyler-rhel-newimage:826:826 [3] NCCL INFO Trees [0] 4/-1/-1->3->2 [1] 4/-1/-1->3->2 [2] 4/-1/-1->3->2 [3] 4/-1/-1->3->2 [4] 4/-1/-1->3->2 [5] 4/-1/-1->3->2 [6] 4/-1/-1->3->2 [7] 4/-1/-1->3->2 [8] 4/-1/-1->3->2 [9] 4/-1/-1->3->2 [10] 4/-1/-1->3->2 [11] 4/-1/-1->3->2 [12] 4/-1/-1->3->2 [13] 4/-1/-1->3->2 [14] 4/-1/-1->3->2 [15] 4/-1/-1->3->2 [16] 4/-1/-1->3->2 [17] 4/-1/-1->3->2 [18] 4/-1/-1->3->2 [19] 4/-1/-1->3->2 [20] 4/-1/-1->3->2 [21] 4/-1/-1->3->2 [22] 4/-1/-1->3->2 [23] 4/-1/-1->3->2
tyler-rhel-newimage:824:824 [1] NCCL INFO Trees [0] 2/-1/-1->1->0 [1] 2/-1/-1->1->0 [2] 2/-1/-1->1->0 [3] 2/-1/-1->1->0 [4] 2/-1/-1->1->0 [5] 2/-1/-1->1->0 [6] 2/-1/-1->1->0 [7] 2/-1/-1->1->0 [8] 2/-1/-1->1->0 [9] 2/-1/-1->1->0 [10] 2/-1/-1->1->0 [11] 2/-1/-1->1->0 [12] 2/-1/-1->1->0 [13] 2/-1/-1->1->0 [14] 2/-1/-1->1->0 [15] 2/-1/-1->1->0 [16] 2/-1/-1->1->0 [17] 2/-1/-1->1->0 [18] 2/-1/-1->1->0 [19] 2/-1/-1->1->0 [20] 2/-1/-1->1->0 [21] 2/-1/-1->1->0 [22] 2/-1/-1->1->0 [23] 2/-1/-1->1->0
tyler-rhel-newimage:801:801 [0] NCCL INFO Channel 03/24 : 0 1 2 3 4 5 6 7
tyler-rhel-newimage:830:830 [7] NCCL INFO Trees [0] -1/-1/-1->7->6 [1] -1/-1/-1->7->6 [2] -1/-1/-1->7->6 [3] -1/-1/-1->7->6 [4] -1/-1/-1->7->6 [5] -1/-1/-1->7->6 [6] -1/-1/-1->7->6 [7] -1/-1/-1->7->6 [8] -1/-1/-1->7->6 [9] -1/-1/-1->7->6 [10] -1/-1/-1->7->6 [11] -1/-1/-1->7->6 [12] -1/-1/-1->7->6 [13] -1/-1/-1->7->6 [14] -1/-1/-1->7->6 [15] -1/-1/-1->7->6 [16] -1/-1/-1->7->6 [17] -1/-1/-1->7->6 [18] -1/-1/-1->7->6 [19] -1/-1/-1->7->6 [20] -1/-1/-1->7->6 [21] -1/-1/-1->7->6 [22] -1/-1/-1->7->6 [23] -1/-1/-1->7->6
tyler-rhel-newimage:825:825 [2] NCCL INFO P2P Chunksize set to 524288
tyler-rhel-newimage:826:826 [3] NCCL INFO P2P Chunksize set to 524288
tyler-rhel-newimage:824:824 [1] NCCL INFO P2P Chunksize set to 524288
tyler-rhel-newimage:801:801 [0] NCCL INFO Channel 04/24 : 0 1 2 3 4 5 6 7
tyler-rhel-newimage:827:827 [4] NCCL INFO Trees [0] 5/-1/-1->4->3 [1] 5/-1/-1->4->3 [2] 5/-1/-1->4->3 [3] 5/-1/-1->4->3 [4] 5/-1/-1->4->3 [5] 5/-1/-1->4->3 [6] 5/-1/-1->4->3 [7] 5/-1/-1->4->3 [8] 5/-1/-1->4->3 [9] 5/-1/-1->4->3 [10] 5/-1/-1->4->3 [11] 5/-1/-1->4->3 [12] 5/-1/-1->4->3 [13] 5/-1/-1->4->3 [14] 5/-1/-1->4->3 [15] 5/-1/-1->4->3 [16] 5/-1/-1->4->3 [17] 5/-1/-1->4->3 [18] 5/-1/-1->4->3 [19] 5/-1/-1->4->3 [20] 5/-1/-1->4->3 [21] 5/-1/-1->4->3 [22] 5/-1/-1->4->3 [23] 5/-1/-1->4->3
tyler-rhel-newimage:829:829 [6] NCCL INFO Trees [0] 7/-1/-1->6->5 [1] 7/-1/-1->6->5 [2] 7/-1/-1->6->5 [3] 7/-1/-1->6->5 [4] 7/-1/-1->6->5 [5] 7/-1/-1->6->5 [6] 7/-1/-1->6->5 [7] 7/-1/-1->6->5 [8] 7/-1/-1->6->5 [9] 7/-1/-1->6->5 [10] 7/-1/-1->6->5 [11] 7/-1/-1->6->5 [12] 7/-1/-1->6->5 [13] 7/-1/-1->6->5 [14] 7/-1/-1->6->5 [15] 7/-1/-1->6->5 [16] 7/-1/-1->6->5 [17] 7/-1/-1->6->5 [18] 7/-1/-1->6->5 [19] 7/-1/-1->6->5 [20] 7/-1/-1->6->5 [21] 7/-1/-1->6->5 [22] 7/-1/-1->6->5 [23] 7/-1/-1->6->5
tyler-rhel-newimage:828:828 [5] NCCL INFO Trees [0] 6/-1/-1->5->4 [1] 6/-1/-1->5->4 [2] 6/-1/-1->5->4 [3] 6/-1/-1->5->4 [4] 6/-1/-1->5->4 [5] 6/-1/-1->5->4 [6] 6/-1/-1->5->4 [7] 6/-1/-1->5->4 [8] 6/-1/-1->5->4 [9] 6/-1/-1->5->4 [10] 6/-1/-1->5->4 [11] 6/-1/-1->5->4 [12] 6/-1/-1->5->4 [13] 6/-1/-1->5->4 [14] 6/-1/-1->5->4 [15] 6/-1/-1->5->4 [16] 6/-1/-1->5->4 [17] 6/-1/-1->5->4 [18] 6/-1/-1->5->4 [19] 6/-1/-1->5->4 [20] 6/-1/-1->5->4 [21] 6/-1/-1->5->4 [22] 6/-1/-1->5->4 [23] 6/-1/-1->5->4
tyler-rhel-newimage:829:829 [6] NCCL INFO P2P Chunksize set to 524288
tyler-rhel-newimage:830:830 [7] NCCL INFO P2P Chunksize set to 524288
tyler-rhel-newimage:801:801 [0] NCCL INFO Channel 05/24 : 0 1 2 3 4 5 6 7
tyler-rhel-newimage:827:827 [4] NCCL INFO P2P Chunksize set to 524288
tyler-rhel-newimage:828:828 [5] NCCL INFO P2P Chunksize set to 524288
tyler-rhel-newimage:801:801 [0] NCCL INFO Channel 06/24 : 0 1 2 3 4 5 6 7
tyler-rhel-newimage:801:801 [0] NCCL INFO Channel 07/24 : 0 1 2 3 4 5 6 7
tyler-rhel-newimage:801:801 [0] NCCL INFO Channel 08/24 : 0 1 2 3 4 5 6 7
tyler-rhel-newimage:801:801 [0] NCCL INFO Channel 09/24 : 0 1 2 3 4 5 6 7
tyler-rhel-newimage:801:801 [0] NCCL INFO Channel 10/24 : 0 1 2 3 4 5 6 7
tyler-rhel-newimage:801:801 [0] NCCL INFO Channel 11/24 : 0 1 2 3 4 5 6 7
tyler-rhel-newimage:801:801 [0] NCCL INFO Channel 12/24 : 0 1 2 3 4 5 6 7
tyler-rhel-newimage:801:801 [0] NCCL INFO Channel 13/24 : 0 1 2 3 4 5 6 7
tyler-rhel-newimage:801:801 [0] NCCL INFO Channel 14/24 : 0 1 2 3 4 5 6 7
tyler-rhel-newimage:801:801 [0] NCCL INFO Channel 15/24 : 0 1 2 3 4 5 6 7
tyler-rhel-newimage:801:801 [0] NCCL INFO Channel 16/24 : 0 1 2 3 4 5 6 7
tyler-rhel-newimage:801:801 [0] NCCL INFO Channel 17/24 : 0 1 2 3 4 5 6 7
tyler-rhel-newimage:801:801 [0] NCCL INFO Channel 18/24 : 0 1 2 3 4 5 6 7
tyler-rhel-newimage:801:801 [0] NCCL INFO Channel 19/24 : 0 1 2 3 4 5 6 7
tyler-rhel-newimage:801:801 [0] NCCL INFO Channel 20/24 : 0 1 2 3 4 5 6 7
tyler-rhel-newimage:801:801 [0] NCCL INFO Channel 21/24 : 0 1 2 3 4 5 6 7
tyler-rhel-newimage:801:801 [0] NCCL INFO Channel 22/24 : 0 1 2 3 4 5 6 7
tyler-rhel-newimage:801:801 [0] NCCL INFO Channel 23/24 : 0 1 2 3 4 5 6 7
tyler-rhel-newimage:801:801 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1 [2] 1/-1/-1->0->-1 [3] 1/-1/-1->0->-1 [4] 1/-1/-1->0->-1 [5] 1/-1/-1->0->-1 [6] 1/-1/-1->0->-1 [7] 1/-1/-1->0->-1 [8] 1/-1/-1->0->-1 [9] 1/-1/-1->0->-1 [10] 1/-1/-1->0->-1 [11] 1/-1/-1->0->-1 [12] 1/-1/-1->0->-1 [13] 1/-1/-1->0->-1 [14] 1/-1/-1->0->-1 [15] 1/-1/-1->0->-1 [16] 1/-1/-1->0->-1 [17] 1/-1/-1->0->-1 [18] 1/-1/-1->0->-1 [19] 1/-1/-1->0->-1 [20] 1/-1/-1->0->-1 [21] 1/-1/-1->0->-1 [22] 1/-1/-1->0->-1 [23] 1/-1/-1->0->-1
tyler-rhel-newimage:801:801 [0] NCCL INFO P2P Chunksize set to 524288
tyler-rhel-newimage:829:829 [6] NCCL INFO Channel 00/0 : 6[6] -> 7[7] via P2P/IPC/read
tyler-rhel-newimage:828:828 [5] NCCL INFO Channel 00/0 : 5[5] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:829:829 [6] NCCL INFO Channel 01/0 : 6[6] -> 7[7] via P2P/IPC/read
tyler-rhel-newimage:824:824 [1] NCCL INFO Channel 00/0 : 1[1] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:801:801 [0] NCCL INFO Channel 00/0 : 0[0] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:828:828 [5] NCCL INFO Channel 01/0 : 5[5] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:829:829 [6] NCCL INFO Channel 02/0 : 6[6] -> 7[7] via P2P/IPC/read
tyler-rhel-newimage:824:824 [1] NCCL INFO Channel 01/0 : 1[1] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:801:801 [0] NCCL INFO Channel 01/0 : 0[0] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:828:828 [5] NCCL INFO Channel 02/0 : 5[5] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:830:830 [7] NCCL INFO Channel 00/0 : 7[7] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:829:829 [6] NCCL INFO Channel 03/0 : 6[6] -> 7[7] via P2P/IPC/read
tyler-rhel-newimage:824:824 [1] NCCL INFO Channel 02/0 : 1[1] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:801:801 [0] NCCL INFO Channel 02/0 : 0[0] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:828:828 [5] NCCL INFO Channel 03/0 : 5[5] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:830:830 [7] NCCL INFO Channel 01/0 : 7[7] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:829:829 [6] NCCL INFO Channel 04/0 : 6[6] -> 7[7] via P2P/IPC/read
tyler-rhel-newimage:824:824 [1] NCCL INFO Channel 03/0 : 1[1] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:801:801 [0] NCCL INFO Channel 03/0 : 0[0] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:828:828 [5] NCCL INFO Channel 04/0 : 5[5] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:830:830 [7] NCCL INFO Channel 02/0 : 7[7] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:829:829 [6] NCCL INFO Channel 05/0 : 6[6] -> 7[7] via P2P/IPC/read
tyler-rhel-newimage:824:824 [1] NCCL INFO Channel 04/0 : 1[1] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:801:801 [0] NCCL INFO Channel 04/0 : 0[0] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:828:828 [5] NCCL INFO Channel 05/0 : 5[5] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:830:830 [7] NCCL INFO Channel 03/0 : 7[7] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:829:829 [6] NCCL INFO Channel 06/0 : 6[6] -> 7[7] via P2P/IPC/read
tyler-rhel-newimage:801:801 [0] NCCL INFO Channel 05/0 : 0[0] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:824:824 [1] NCCL INFO Channel 05/0 : 1[1] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:828:828 [5] NCCL INFO Channel 06/0 : 5[5] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:830:830 [7] NCCL INFO Channel 04/0 : 7[7] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:829:829 [6] NCCL INFO Channel 07/0 : 6[6] -> 7[7] via P2P/IPC/read
tyler-rhel-newimage:801:801 [0] NCCL INFO Channel 06/0 : 0[0] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:824:824 [1] NCCL INFO Channel 06/0 : 1[1] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:828:828 [5] NCCL INFO Channel 07/0 : 5[5] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:827:827 [4] NCCL INFO Channel 00/0 : 4[4] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:830:830 [7] NCCL INFO Channel 05/0 : 7[7] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:829:829 [6] NCCL INFO Channel 08/0 : 6[6] -> 7[7] via P2P/IPC/read
tyler-rhel-newimage:801:801 [0] NCCL INFO Channel 07/0 : 0[0] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:824:824 [1] NCCL INFO Channel 07/0 : 1[1] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:827:827 [4] NCCL INFO Channel 01/0 : 4[4] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:828:828 [5] NCCL INFO Channel 08/0 : 5[5] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:830:830 [7] NCCL INFO Channel 06/0 : 7[7] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:829:829 [6] NCCL INFO Channel 09/0 : 6[6] -> 7[7] via P2P/IPC/read
tyler-rhel-newimage:801:801 [0] NCCL INFO Channel 08/0 : 0[0] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:824:824 [1] NCCL INFO Channel 08/0 : 1[1] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:827:827 [4] NCCL INFO Channel 02/0 : 4[4] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:828:828 [5] NCCL INFO Channel 09/0 : 5[5] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:830:830 [7] NCCL INFO Channel 07/0 : 7[7] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:829:829 [6] NCCL INFO Channel 10/0 : 6[6] -> 7[7] via P2P/IPC/read
tyler-rhel-newimage:801:801 [0] NCCL INFO Channel 09/0 : 0[0] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:824:824 [1] NCCL INFO Channel 09/0 : 1[1] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:825:825 [2] NCCL INFO Channel 00/0 : 2[2] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:828:828 [5] NCCL INFO Channel 10/0 : 5[5] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:826:826 [3] NCCL INFO Channel 00/0 : 3[3] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:827:827 [4] NCCL INFO Channel 03/0 : 4[4] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:830:830 [7] NCCL INFO Channel 08/0 : 7[7] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:829:829 [6] NCCL INFO Channel 11/0 : 6[6] -> 7[7] via P2P/IPC/read
tyler-rhel-newimage:801:801 [0] NCCL INFO Channel 10/0 : 0[0] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:824:824 [1] NCCL INFO Channel 10/0 : 1[1] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:825:825 [2] NCCL INFO Channel 01/0 : 2[2] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:828:828 [5] NCCL INFO Channel 11/0 : 5[5] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:826:826 [3] NCCL INFO Channel 01/0 : 3[3] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:827:827 [4] NCCL INFO Channel 04/0 : 4[4] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:830:830 [7] NCCL INFO Channel 09/0 : 7[7] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:829:829 [6] NCCL INFO Channel 12/0 : 6[6] -> 7[7] via P2P/IPC/read
tyler-rhel-newimage:801:801 [0] NCCL INFO Channel 11/0 : 0[0] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:824:824 [1] NCCL INFO Channel 11/0 : 1[1] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:825:825 [2] NCCL INFO Channel 02/0 : 2[2] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:828:828 [5] NCCL INFO Channel 12/0 : 5[5] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:826:826 [3] NCCL INFO Channel 02/0 : 3[3] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:827:827 [4] NCCL INFO Channel 05/0 : 4[4] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:830:830 [7] NCCL INFO Channel 10/0 : 7[7] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:829:829 [6] NCCL INFO Channel 13/0 : 6[6] -> 7[7] via P2P/IPC/read
tyler-rhel-newimage:801:801 [0] NCCL INFO Channel 12/0 : 0[0] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:824:824 [1] NCCL INFO Channel 12/0 : 1[1] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:825:825 [2] NCCL INFO Channel 03/0 : 2[2] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:828:828 [5] NCCL INFO Channel 13/0 : 5[5] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:826:826 [3] NCCL INFO Channel 03/0 : 3[3] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:827:827 [4] NCCL INFO Channel 06/0 : 4[4] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:801:801 [0] NCCL INFO Channel 13/0 : 0[0] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:830:830 [7] NCCL INFO Channel 11/0 : 7[7] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:829:829 [6] NCCL INFO Channel 14/0 : 6[6] -> 7[7] via P2P/IPC/read
tyler-rhel-newimage:824:824 [1] NCCL INFO Channel 13/0 : 1[1] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:825:825 [2] NCCL INFO Channel 04/0 : 2[2] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:828:828 [5] NCCL INFO Channel 14/0 : 5[5] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:826:826 [3] NCCL INFO Channel 04/0 : 3[3] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:827:827 [4] NCCL INFO Channel 07/0 : 4[4] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:801:801 [0] NCCL INFO Channel 14/0 : 0[0] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:830:830 [7] NCCL INFO Channel 12/0 : 7[7] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:829:829 [6] NCCL INFO Channel 15/0 : 6[6] -> 7[7] via P2P/IPC/read
tyler-rhel-newimage:824:824 [1] NCCL INFO Channel 14/0 : 1[1] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:825:825 [2] NCCL INFO Channel 05/0 : 2[2] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:828:828 [5] NCCL INFO Channel 15/0 : 5[5] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:826:826 [3] NCCL INFO Channel 05/0 : 3[3] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:827:827 [4] NCCL INFO Channel 08/0 : 4[4] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:801:801 [0] NCCL INFO Channel 15/0 : 0[0] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:830:830 [7] NCCL INFO Channel 13/0 : 7[7] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:829:829 [6] NCCL INFO Channel 16/0 : 6[6] -> 7[7] via P2P/IPC/read
tyler-rhel-newimage:824:824 [1] NCCL INFO Channel 15/0 : 1[1] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:825:825 [2] NCCL INFO Channel 06/0 : 2[2] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:828:828 [5] NCCL INFO Channel 16/0 : 5[5] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:826:826 [3] NCCL INFO Channel 06/0 : 3[3] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:827:827 [4] NCCL INFO Channel 09/0 : 4[4] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:829:829 [6] NCCL INFO Channel 17/0 : 6[6] -> 7[7] via P2P/IPC/read
tyler-rhel-newimage:801:801 [0] NCCL INFO Channel 16/0 : 0[0] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:830:830 [7] NCCL INFO Channel 14/0 : 7[7] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:824:824 [1] NCCL INFO Channel 16/0 : 1[1] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:825:825 [2] NCCL INFO Channel 07/0 : 2[2] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:828:828 [5] NCCL INFO Channel 17/0 : 5[5] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:826:826 [3] NCCL INFO Channel 07/0 : 3[3] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:827:827 [4] NCCL INFO Channel 10/0 : 4[4] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:801:801 [0] NCCL INFO Channel 17/0 : 0[0] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:829:829 [6] NCCL INFO Channel 18/0 : 6[6] -> 7[7] via P2P/IPC/read
tyler-rhel-newimage:830:830 [7] NCCL INFO Channel 15/0 : 7[7] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:824:824 [1] NCCL INFO Channel 17/0 : 1[1] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:825:825 [2] NCCL INFO Channel 08/0 : 2[2] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:828:828 [5] NCCL INFO Channel 18/0 : 5[5] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:826:826 [3] NCCL INFO Channel 08/0 : 3[3] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:827:827 [4] NCCL INFO Channel 11/0 : 4[4] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:801:801 [0] NCCL INFO Channel 18/0 : 0[0] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:829:829 [6] NCCL INFO Channel 19/0 : 6[6] -> 7[7] via P2P/IPC/read
tyler-rhel-newimage:830:830 [7] NCCL INFO Channel 16/0 : 7[7] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:824:824 [1] NCCL INFO Channel 18/0 : 1[1] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:825:825 [2] NCCL INFO Channel 09/0 : 2[2] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:826:826 [3] NCCL INFO Channel 09/0 : 3[3] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:828:828 [5] NCCL INFO Channel 19/0 : 5[5] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:827:827 [4] NCCL INFO Channel 12/0 : 4[4] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:801:801 [0] NCCL INFO Channel 19/0 : 0[0] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:829:829 [6] NCCL INFO Channel 20/0 : 6[6] -> 7[7] via P2P/IPC/read
tyler-rhel-newimage:830:830 [7] NCCL INFO Channel 17/0 : 7[7] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:824:824 [1] NCCL INFO Channel 19/0 : 1[1] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:825:825 [2] NCCL INFO Channel 10/0 : 2[2] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:826:826 [3] NCCL INFO Channel 10/0 : 3[3] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:828:828 [5] NCCL INFO Channel 20/0 : 5[5] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:830:830 [7] NCCL INFO Channel 18/0 : 7[7] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:801:801 [0] NCCL INFO Channel 20/0 : 0[0] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:829:829 [6] NCCL INFO Channel 21/0 : 6[6] -> 7[7] via P2P/IPC/read
tyler-rhel-newimage:827:827 [4] NCCL INFO Channel 13/0 : 4[4] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:824:824 [1] NCCL INFO Channel 20/0 : 1[1] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:826:826 [3] NCCL INFO Channel 11/0 : 3[3] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:825:825 [2] NCCL INFO Channel 11/0 : 2[2] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:828:828 [5] NCCL INFO Channel 21/0 : 5[5] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:830:830 [7] NCCL INFO Channel 19/0 : 7[7] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:801:801 [0] NCCL INFO Channel 21/0 : 0[0] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:829:829 [6] NCCL INFO Channel 22/0 : 6[6] -> 7[7] via P2P/IPC/read
tyler-rhel-newimage:827:827 [4] NCCL INFO Channel 14/0 : 4[4] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:824:824 [1] NCCL INFO Channel 21/0 : 1[1] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:826:826 [3] NCCL INFO Channel 12/0 : 3[3] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:825:825 [2] NCCL INFO Channel 12/0 : 2[2] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:828:828 [5] NCCL INFO Channel 22/0 : 5[5] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:830:830 [7] NCCL INFO Channel 20/0 : 7[7] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:801:801 [0] NCCL INFO Channel 22/0 : 0[0] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:829:829 [6] NCCL INFO Channel 23/0 : 6[6] -> 7[7] via P2P/IPC/read
tyler-rhel-newimage:827:827 [4] NCCL INFO Channel 15/0 : 4[4] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:824:824 [1] NCCL INFO Channel 22/0 : 1[1] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:826:826 [3] NCCL INFO Channel 13/0 : 3[3] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:825:825 [2] NCCL INFO Channel 13/0 : 2[2] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:828:828 [5] NCCL INFO Channel 23/0 : 5[5] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:830:830 [7] NCCL INFO Channel 21/0 : 7[7] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:801:801 [0] NCCL INFO Channel 23/0 : 0[0] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:827:827 [4] NCCL INFO Channel 16/0 : 4[4] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:824:824 [1] NCCL INFO Channel 23/0 : 1[1] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:826:826 [3] NCCL INFO Channel 14/0 : 3[3] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:825:825 [2] NCCL INFO Channel 14/0 : 2[2] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:830:830 [7] NCCL INFO Channel 22/0 : 7[7] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:827:827 [4] NCCL INFO Channel 17/0 : 4[4] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:826:826 [3] NCCL INFO Channel 15/0 : 3[3] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:825:825 [2] NCCL INFO Channel 15/0 : 2[2] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:830:830 [7] NCCL INFO Channel 23/0 : 7[7] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:827:827 [4] NCCL INFO Channel 18/0 : 4[4] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:826:826 [3] NCCL INFO Channel 16/0 : 3[3] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:825:825 [2] NCCL INFO Channel 16/0 : 2[2] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:827:827 [4] NCCL INFO Channel 19/0 : 4[4] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:826:826 [3] NCCL INFO Channel 17/0 : 3[3] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:825:825 [2] NCCL INFO Channel 17/0 : 2[2] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:827:827 [4] NCCL INFO Channel 20/0 : 4[4] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:826:826 [3] NCCL INFO Channel 18/0 : 3[3] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:825:825 [2] NCCL INFO Channel 18/0 : 2[2] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:827:827 [4] NCCL INFO Channel 21/0 : 4[4] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:826:826 [3] NCCL INFO Channel 19/0 : 3[3] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:825:825 [2] NCCL INFO Channel 19/0 : 2[2] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:827:827 [4] NCCL INFO Channel 22/0 : 4[4] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:826:826 [3] NCCL INFO Channel 20/0 : 3[3] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:825:825 [2] NCCL INFO Channel 20/0 : 2[2] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:827:827 [4] NCCL INFO Channel 23/0 : 4[4] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:826:826 [3] NCCL INFO Channel 21/0 : 3[3] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:825:825 [2] NCCL INFO Channel 21/0 : 2[2] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:826:826 [3] NCCL INFO Channel 22/0 : 3[3] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:825:825 [2] NCCL INFO Channel 22/0 : 2[2] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:826:826 [3] NCCL INFO Channel 23/0 : 3[3] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:825:825 [2] NCCL INFO Channel 23/0 : 2[2] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:830:830 [7] NCCL INFO Connected all rings
tyler-rhel-newimage:830:830 [7] NCCL INFO Channel 00/0 : 7[7] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:830:830 [7] NCCL INFO Channel 01/0 : 7[7] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:829:829 [6] NCCL INFO Connected all rings
tyler-rhel-newimage:828:828 [5] NCCL INFO Connected all rings
tyler-rhel-newimage:830:830 [7] NCCL INFO Channel 02/0 : 7[7] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:830:830 [7] NCCL INFO Channel 03/0 : 7[7] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:830:830 [7] NCCL INFO Channel 04/0 : 7[7] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:827:827 [4] NCCL INFO Connected all rings
tyler-rhel-newimage:801:801 [0] NCCL INFO Connected all rings
tyler-rhel-newimage:826:826 [3] NCCL INFO Connected all rings
tyler-rhel-newimage:830:830 [7] NCCL INFO Channel 05/0 : 7[7] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:824:824 [1] NCCL INFO Connected all rings
tyler-rhel-newimage:825:825 [2] NCCL INFO Connected all rings
tyler-rhel-newimage:830:830 [7] NCCL INFO Channel 06/0 : 7[7] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:830:830 [7] NCCL INFO Channel 07/0 : 7[7] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:830:830 [7] NCCL INFO Channel 08/0 : 7[7] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:830:830 [7] NCCL INFO Channel 09/0 : 7[7] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:830:830 [7] NCCL INFO Channel 10/0 : 7[7] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:830:830 [7] NCCL INFO Channel 11/0 : 7[7] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:830:830 [7] NCCL INFO Channel 12/0 : 7[7] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:830:830 [7] NCCL INFO Channel 13/0 : 7[7] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:830:830 [7] NCCL INFO Channel 14/0 : 7[7] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:830:830 [7] NCCL INFO Channel 15/0 : 7[7] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:830:830 [7] NCCL INFO Channel 16/0 : 7[7] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:830:830 [7] NCCL INFO Channel 17/0 : 7[7] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:830:830 [7] NCCL INFO Channel 18/0 : 7[7] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:830:830 [7] NCCL INFO Channel 19/0 : 7[7] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:830:830 [7] NCCL INFO Channel 20/0 : 7[7] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:830:830 [7] NCCL INFO Channel 21/0 : 7[7] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:830:830 [7] NCCL INFO Channel 22/0 : 7[7] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:830:830 [7] NCCL INFO Channel 23/0 : 7[7] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:828:828 [5] NCCL INFO Channel 00/0 : 5[5] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:829:829 [6] NCCL INFO Channel 00/0 : 6[6] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:828:828 [5] NCCL INFO Channel 01/0 : 5[5] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:829:829 [6] NCCL INFO Channel 01/0 : 6[6] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:828:828 [5] NCCL INFO Channel 02/0 : 5[5] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:829:829 [6] NCCL INFO Channel 02/0 : 6[6] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:828:828 [5] NCCL INFO Channel 03/0 : 5[5] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:824:824 [1] NCCL INFO Channel 00/0 : 1[1] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:826:826 [3] NCCL INFO Channel 00/0 : 3[3] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:825:825 [2] NCCL INFO Channel 00/0 : 2[2] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:829:829 [6] NCCL INFO Channel 03/0 : 6[6] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:828:828 [5] NCCL INFO Channel 04/0 : 5[5] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:825:825 [2] NCCL INFO Channel 01/0 : 2[2] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:824:824 [1] NCCL INFO Channel 01/0 : 1[1] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:826:826 [3] NCCL INFO Channel 01/0 : 3[3] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:829:829 [6] NCCL INFO Channel 04/0 : 6[6] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:827:827 [4] NCCL INFO Channel 00/0 : 4[4] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:828:828 [5] NCCL INFO Channel 05/0 : 5[5] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:825:825 [2] NCCL INFO Channel 02/0 : 2[2] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:824:824 [1] NCCL INFO Channel 02/0 : 1[1] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:826:826 [3] NCCL INFO Channel 02/0 : 3[3] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:829:829 [6] NCCL INFO Channel 05/0 : 6[6] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:827:827 [4] NCCL INFO Channel 01/0 : 4[4] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:828:828 [5] NCCL INFO Channel 06/0 : 5[5] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:825:825 [2] NCCL INFO Channel 03/0 : 2[2] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:824:824 [1] NCCL INFO Channel 03/0 : 1[1] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:826:826 [3] NCCL INFO Channel 03/0 : 3[3] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:829:829 [6] NCCL INFO Channel 06/0 : 6[6] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:827:827 [4] NCCL INFO Channel 02/0 : 4[4] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:828:828 [5] NCCL INFO Channel 07/0 : 5[5] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:825:825 [2] NCCL INFO Channel 04/0 : 2[2] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:824:824 [1] NCCL INFO Channel 04/0 : 1[1] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:826:826 [3] NCCL INFO Channel 04/0 : 3[3] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:829:829 [6] NCCL INFO Channel 07/0 : 6[6] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:827:827 [4] NCCL INFO Channel 03/0 : 4[4] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:828:828 [5] NCCL INFO Channel 08/0 : 5[5] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:825:825 [2] NCCL INFO Channel 05/0 : 2[2] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:829:829 [6] NCCL INFO Channel 08/0 : 6[6] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:824:824 [1] NCCL INFO Channel 05/0 : 1[1] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:826:826 [3] NCCL INFO Channel 05/0 : 3[3] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:827:827 [4] NCCL INFO Channel 04/0 : 4[4] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:828:828 [5] NCCL INFO Channel 09/0 : 5[5] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:825:825 [2] NCCL INFO Channel 06/0 : 2[2] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:829:829 [6] NCCL INFO Channel 09/0 : 6[6] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:826:826 [3] NCCL INFO Channel 06/0 : 3[3] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:824:824 [1] NCCL INFO Channel 06/0 : 1[1] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:827:827 [4] NCCL INFO Channel 05/0 : 4[4] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:828:828 [5] NCCL INFO Channel 10/0 : 5[5] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:825:825 [2] NCCL INFO Channel 07/0 : 2[2] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:829:829 [6] NCCL INFO Channel 10/0 : 6[6] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:826:826 [3] NCCL INFO Channel 07/0 : 3[3] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:824:824 [1] NCCL INFO Channel 07/0 : 1[1] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:827:827 [4] NCCL INFO Channel 06/0 : 4[4] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:828:828 [5] NCCL INFO Channel 11/0 : 5[5] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:825:825 [2] NCCL INFO Channel 08/0 : 2[2] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:829:829 [6] NCCL INFO Channel 11/0 : 6[6] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:826:826 [3] NCCL INFO Channel 08/0 : 3[3] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:824:824 [1] NCCL INFO Channel 08/0 : 1[1] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:828:828 [5] NCCL INFO Channel 12/0 : 5[5] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:825:825 [2] NCCL INFO Channel 09/0 : 2[2] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:829:829 [6] NCCL INFO Channel 12/0 : 6[6] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:826:826 [3] NCCL INFO Channel 09/0 : 3[3] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:824:824 [1] NCCL INFO Channel 09/0 : 1[1] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:825:825 [2] NCCL INFO Channel 10/0 : 2[2] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:828:828 [5] NCCL INFO Channel 13/0 : 5[5] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:829:829 [6] NCCL INFO Channel 13/0 : 6[6] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:826:826 [3] NCCL INFO Channel 10/0 : 3[3] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:824:824 [1] NCCL INFO Channel 10/0 : 1[1] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:827:827 [4] NCCL INFO Channel 07/0 : 4[4] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:825:825 [2] NCCL INFO Channel 11/0 : 2[2] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:828:828 [5] NCCL INFO Channel 14/0 : 5[5] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:829:829 [6] NCCL INFO Channel 14/0 : 6[6] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:826:826 [3] NCCL INFO Channel 11/0 : 3[3] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:824:824 [1] NCCL INFO Channel 11/0 : 1[1] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:825:825 [2] NCCL INFO Channel 12/0 : 2[2] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:827:827 [4] NCCL INFO Channel 08/0 : 4[4] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:828:828 [5] NCCL INFO Channel 15/0 : 5[5] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:829:829 [6] NCCL INFO Channel 15/0 : 6[6] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:826:826 [3] NCCL INFO Channel 12/0 : 3[3] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:824:824 [1] NCCL INFO Channel 12/0 : 1[1] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:825:825 [2] NCCL INFO Channel 13/0 : 2[2] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:827:827 [4] NCCL INFO Channel 09/0 : 4[4] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:828:828 [5] NCCL INFO Channel 16/0 : 5[5] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:829:829 [6] NCCL INFO Channel 16/0 : 6[6] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:826:826 [3] NCCL INFO Channel 13/0 : 3[3] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:824:824 [1] NCCL INFO Channel 13/0 : 1[1] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:825:825 [2] NCCL INFO Channel 14/0 : 2[2] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:827:827 [4] NCCL INFO Channel 10/0 : 4[4] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:828:828 [5] NCCL INFO Channel 17/0 : 5[5] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:829:829 [6] NCCL INFO Channel 17/0 : 6[6] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:826:826 [3] NCCL INFO Channel 14/0 : 3[3] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:824:824 [1] NCCL INFO Channel 14/0 : 1[1] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:825:825 [2] NCCL INFO Channel 15/0 : 2[2] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:827:827 [4] NCCL INFO Channel 11/0 : 4[4] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:828:828 [5] NCCL INFO Channel 18/0 : 5[5] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:829:829 [6] NCCL INFO Channel 18/0 : 6[6] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:826:826 [3] NCCL INFO Channel 15/0 : 3[3] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:824:824 [1] NCCL INFO Channel 15/0 : 1[1] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:825:825 [2] NCCL INFO Channel 16/0 : 2[2] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:827:827 [4] NCCL INFO Channel 12/0 : 4[4] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:828:828 [5] NCCL INFO Channel 19/0 : 5[5] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:829:829 [6] NCCL INFO Channel 19/0 : 6[6] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:826:826 [3] NCCL INFO Channel 16/0 : 3[3] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:824:824 [1] NCCL INFO Channel 16/0 : 1[1] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:825:825 [2] NCCL INFO Channel 17/0 : 2[2] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:827:827 [4] NCCL INFO Channel 13/0 : 4[4] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:828:828 [5] NCCL INFO Channel 20/0 : 5[5] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:829:829 [6] NCCL INFO Channel 20/0 : 6[6] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:826:826 [3] NCCL INFO Channel 17/0 : 3[3] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:825:825 [2] NCCL INFO Channel 18/0 : 2[2] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:824:824 [1] NCCL INFO Channel 17/0 : 1[1] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:827:827 [4] NCCL INFO Channel 14/0 : 4[4] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:828:828 [5] NCCL INFO Channel 21/0 : 5[5] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:829:829 [6] NCCL INFO Channel 21/0 : 6[6] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:826:826 [3] NCCL INFO Channel 18/0 : 3[3] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:825:825 [2] NCCL INFO Channel 19/0 : 2[2] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:824:824 [1] NCCL INFO Channel 18/0 : 1[1] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:827:827 [4] NCCL INFO Channel 15/0 : 4[4] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:828:828 [5] NCCL INFO Channel 22/0 : 5[5] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:826:826 [3] NCCL INFO Channel 19/0 : 3[3] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:829:829 [6] NCCL INFO Channel 22/0 : 6[6] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:825:825 [2] NCCL INFO Channel 20/0 : 2[2] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:824:824 [1] NCCL INFO Channel 19/0 : 1[1] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:827:827 [4] NCCL INFO Channel 16/0 : 4[4] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:828:828 [5] NCCL INFO Channel 23/0 : 5[5] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:826:826 [3] NCCL INFO Channel 20/0 : 3[3] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:829:829 [6] NCCL INFO Channel 23/0 : 6[6] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:825:825 [2] NCCL INFO Channel 21/0 : 2[2] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:824:824 [1] NCCL INFO Channel 20/0 : 1[1] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:827:827 [4] NCCL INFO Channel 17/0 : 4[4] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:826:826 [3] NCCL INFO Channel 21/0 : 3[3] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:825:825 [2] NCCL INFO Channel 22/0 : 2[2] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:824:824 [1] NCCL INFO Channel 21/0 : 1[1] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:827:827 [4] NCCL INFO Channel 18/0 : 4[4] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:826:826 [3] NCCL INFO Channel 22/0 : 3[3] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:825:825 [2] NCCL INFO Channel 23/0 : 2[2] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:827:827 [4] NCCL INFO Channel 19/0 : 4[4] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:824:824 [1] NCCL INFO Channel 22/0 : 1[1] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:826:826 [3] NCCL INFO Channel 23/0 : 3[3] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:827:827 [4] NCCL INFO Channel 20/0 : 4[4] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:824:824 [1] NCCL INFO Channel 23/0 : 1[1] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:827:827 [4] NCCL INFO Channel 21/0 : 4[4] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:827:827 [4] NCCL INFO Channel 22/0 : 4[4] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:827:827 [4] NCCL INFO Channel 23/0 : 4[4] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:830:830 [7] NCCL INFO Connected all trees
tyler-rhel-newimage:830:830 [7] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512
tyler-rhel-newimage:830:830 [7] NCCL INFO 24 coll channels, 0 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer
tyler-rhel-newimage:829:829 [6] NCCL INFO Connected all trees
tyler-rhel-newimage:829:829 [6] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512
tyler-rhel-newimage:829:829 [6] NCCL INFO 24 coll channels, 0 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer
tyler-rhel-newimage:801:801 [0] NCCL INFO Connected all trees
tyler-rhel-newimage:825:825 [2] NCCL INFO Connected all trees
tyler-rhel-newimage:801:801 [0] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512
tyler-rhel-newimage:824:824 [1] NCCL INFO Connected all trees
tyler-rhel-newimage:801:801 [0] NCCL INFO 24 coll channels, 0 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer
tyler-rhel-newimage:825:825 [2] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512
tyler-rhel-newimage:824:824 [1] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512
tyler-rhel-newimage:825:825 [2] NCCL INFO 24 coll channels, 0 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer
tyler-rhel-newimage:824:824 [1] NCCL INFO 24 coll channels, 0 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer
tyler-rhel-newimage:826:826 [3] NCCL INFO Connected all trees
tyler-rhel-newimage:826:826 [3] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512
tyler-rhel-newimage:826:826 [3] NCCL INFO 24 coll channels, 0 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer
tyler-rhel-newimage:828:828 [5] NCCL INFO Connected all trees
tyler-rhel-newimage:827:827 [4] NCCL INFO Connected all trees
tyler-rhel-newimage:828:828 [5] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512
tyler-rhel-newimage:828:828 [5] NCCL INFO 24 coll channels, 0 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer
tyler-rhel-newimage:827:827 [4] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512
tyler-rhel-newimage:827:827 [4] NCCL INFO 24 coll channels, 0 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer
tyler-rhel-newimage:830:830 [7] NCCL INFO NCCL_WORK_FIFO_DEPTH set by environment to 4194304.
tyler-rhel-newimage:829:829 [6] NCCL INFO NCCL_WORK_FIFO_DEPTH set by environment to 4194304.
tyler-rhel-newimage:801:801 [0] NCCL INFO NCCL_WORK_FIFO_DEPTH set by environment to 4194304.
tyler-rhel-newimage:825:825 [2] NCCL INFO NCCL_WORK_FIFO_DEPTH set by environment to 4194304.
tyler-rhel-newimage:824:824 [1] NCCL INFO NCCL_WORK_FIFO_DEPTH set by environment to 4194304.
tyler-rhel-newimage:826:826 [3] NCCL INFO NCCL_WORK_FIFO_DEPTH set by environment to 4194304.
tyler-rhel-newimage:827:827 [4] NCCL INFO NCCL_WORK_FIFO_DEPTH set by environment to 4194304.
tyler-rhel-newimage:828:828 [5] NCCL INFO NCCL_WORK_FIFO_DEPTH set by environment to 4194304.
tyler-rhel-newimage:829:829 [6] NCCL INFO comm 0x555b94f77fe0 rank 6 nranks 8 cudaDev 6 nvmlDev 6 busId e070 commId 0xc1aaaefa85b64c46 - Init COMPLETE
tyler-rhel-newimage:827:827 [4] NCCL INFO comm 0x555b94f76b60 rank 4 nranks 8 cudaDev 4 nvmlDev 4 busId c050 commId 0xc1aaaefa85b64c46 - Init COMPLETE
tyler-rhel-newimage:830:830 [7] NCCL INFO comm 0x555b94f79470 rank 7 nranks 8 cudaDev 7 nvmlDev 7 busId e080 commId 0xc1aaaefa85b64c46 - Init COMPLETE
tyler-rhel-newimage:824:824 [1] NCCL INFO comm 0x555b94f76110 rank 1 nranks 8 cudaDev 1 nvmlDev 1 busId 8020 commId 0xc1aaaefa85b64c46 - Init COMPLETE
tyler-rhel-newimage:828:828 [5] NCCL INFO comm 0x555b94f76f30 rank 5 nranks 8 cudaDev 5 nvmlDev 5 busId c060 commId 0xc1aaaefa85b64c46 - Init COMPLETE
tyler-rhel-newimage:826:826 [3] NCCL INFO comm 0x555b94f76680 rank 3 nranks 8 cudaDev 3 nvmlDev 3 busId a040 commId 0xc1aaaefa85b64c46 - Init COMPLETE
tyler-rhel-newimage:825:825 [2] NCCL INFO comm 0x555b94f76350 rank 2 nranks 8 cudaDev 2 nvmlDev 2 busId a030 commId 0xc1aaaefa85b64c46 - Init COMPLETE
tyler-rhel-newimage:801:801 [0] NCCL INFO comm 0x555b94f7de20 rank 0 nranks 8 cudaDev 0 nvmlDev 0 busId 8010 commId 0xc1aaaefa85b64c46 - Init COMPLETE
Loading safetensors checkpoint shards: 0% Completed | 0/19 [00:00<?, ?it/s]
Loading safetensors checkpoint shards: 5% Completed | 1/19 [00:00<00:05, 3.02it/s]
INFO 2024-07-27 18:37:52,083 instructlab.model.backends.backends:336: Waiting for the vLLM server to start at http://127.0.0.1:59587/v1, this might take a moment... Attempt: 6/80
Loading safetensors checkpoint shards: 11% Completed | 2/19 [00:00<00:05, 2.95it/s]
Loading safetensors checkpoint shards: 16% Completed | 3/19 [00:00<00:05, 3.07it/s]
Loading safetensors checkpoint shards: 21% Completed | 4/19 [00:01<00:05, 2.98it/s]
Loading safetensors checkpoint shards: 26% Completed | 5/19 [00:01<00:04, 3.04it/s]
Loading safetensors checkpoint shards: 32% Completed | 6/19 [00:02<00:04, 2.95it/s]
Loading safetensors checkpoint shards: 37% Completed | 7/19 [00:02<00:04, 2.94it/s]
Loading safetensors checkpoint shards: 42% Completed | 8/19 [00:02<00:03, 2.91it/s]
Loading safetensors checkpoint shards: 47% Completed | 9/19 [00:03<00:03, 2.88it/s]
Loading safetensors checkpoint shards: 53% Completed | 10/19 [00:03<00:03, 2.84it/s]
Loading safetensors checkpoint shards: 58% Completed | 11/19 [00:03<00:02, 2.78it/s]
Loading safetensors checkpoint shards: 63% Completed | 12/19 [00:04<00:02, 2.81it/s]
Loading safetensors checkpoint shards: 68% Completed | 13/19 [00:04<00:02, 2.85it/s]
Loading safetensors checkpoint shards: 74% Completed | 14/19 [00:04<00:01, 2.87it/s]
INFO 2024-07-27 18:37:56,538 instructlab.model.backends.backends:336: Waiting for the vLLM server to start at http://127.0.0.1:59587/v1, this might take a moment... Attempt: 7/80
Loading safetensors checkpoint shards: 79% Completed | 15/19 [00:05<00:01, 2.93it/s]
Loading safetensors checkpoint shards: 84% Completed | 16/19 [00:05<00:01, 2.94it/s]
Loading safetensors checkpoint shards: 89% Completed | 17/19 [00:05<00:00, 3.08it/s]
Loading safetensors checkpoint shards: 95% Completed | 18/19 [00:06<00:00, 3.09it/s]
Loading safetensors checkpoint shards: 100% Completed | 19/19 [00:06<00:00, 3.42it/s]
Loading safetensors checkpoint shards: 100% Completed | 19/19 [00:06<00:00, 3.00it/s]
tyler-rhel-newimage:801:1363 [0] NCCL INFO Using non-device net plugin version 0
tyler-rhel-newimage:801:1363 [0] NCCL INFO Using network Socket
tyler-rhel-newimage:830:1364 [7] NCCL INFO Using non-device net plugin version 0
tyler-rhel-newimage:830:1364 [7] NCCL INFO Using network Socket
tyler-rhel-newimage:826:1366 [3] NCCL INFO Using non-device net plugin version 0
tyler-rhel-newimage:826:1366 [3] NCCL INFO Using network Socket
tyler-rhel-newimage:825:1365 [2] NCCL INFO Using non-device net plugin version 0
tyler-rhel-newimage:825:1365 [2] NCCL INFO Using network Socket
tyler-rhel-newimage:824:1367 [1] NCCL INFO Using non-device net plugin version 0
tyler-rhel-newimage:824:1367 [1] NCCL INFO Using network Socket
tyler-rhel-newimage:827:1368 [4] NCCL INFO Using non-device net plugin version 0
tyler-rhel-newimage:827:1368 [4] NCCL INFO Using network Socket
tyler-rhel-newimage:829:1369 [6] NCCL INFO Using non-device net plugin version 0
tyler-rhel-newimage:829:1369 [6] NCCL INFO Using network Socket
tyler-rhel-newimage:828:1370 [5] NCCL INFO Using non-device net plugin version 0
tyler-rhel-newimage:828:1370 [5] NCCL INFO Using network Socket
tyler-rhel-newimage:826:1366 [3] NCCL INFO comm 0x555ba0549a40 rank 3 nranks 8 cudaDev 3 nvmlDev 3 busId a040 commId 0xd56327a586613906 - Init START
tyler-rhel-newimage:829:1369 [6] NCCL INFO comm 0x555ba114a940 rank 6 nranks 8 cudaDev 6 nvmlDev 6 busId e070 commId 0xd56327a586613906 - Init START
tyler-rhel-newimage:827:1368 [4] NCCL INFO comm 0x555b9f749530 rank 4 nranks 8 cudaDev 4 nvmlDev 4 busId c050 commId 0xd56327a586613906 - Init START
tyler-rhel-newimage:825:1365 [2] NCCL INFO comm 0x555ba0547ff0 rank 2 nranks 8 cudaDev 2 nvmlDev 2 busId a030 commId 0xd56327a586613906 - Init START
tyler-rhel-newimage:801:1363 [0] NCCL INFO comm 0x555bab11e710 rank 0 nranks 8 cudaDev 0 nvmlDev 0 busId 8010 commId 0xd56327a586613906 - Init START
tyler-rhel-newimage:824:1367 [1] NCCL INFO comm 0x555b9f748ea0 rank 1 nranks 8 cudaDev 1 nvmlDev 1 busId 8020 commId 0xd56327a586613906 - Init START
tyler-rhel-newimage:830:1364 [7] NCCL INFO comm 0x555ba0534fd0 rank 7 nranks 8 cudaDev 7 nvmlDev 7 busId e080 commId 0xd56327a586613906 - Init START
tyler-rhel-newimage:828:1370 [5] NCCL INFO comm 0x555b9f749730 rank 5 nranks 8 cudaDev 5 nvmlDev 5 busId c060 commId 0xd56327a586613906 - Init START
tyler-rhel-newimage:826:1366 [3] NCCL INFO Setting affinity for GPU 3 to ff,ffffffff
tyler-rhel-newimage:826:1366 [3] NCCL INFO NVLS multicast support is not available on dev 3
tyler-rhel-newimage:829:1369 [6] NCCL INFO Setting affinity for GPU 6 to ffff,ffffff00,00000000
tyler-rhel-newimage:829:1369 [6] NCCL INFO NVLS multicast support is not available on dev 6
tyler-rhel-newimage:824:1367 [1] NCCL INFO Setting affinity for GPU 1 to ff,ffffffff
tyler-rhel-newimage:824:1367 [1] NCCL INFO NVLS multicast support is not available on dev 1
tyler-rhel-newimage:828:1370 [5] NCCL INFO Setting affinity for GPU 5 to ffff,ffffff00,00000000
tyler-rhel-newimage:828:1370 [5] NCCL INFO NVLS multicast support is not available on dev 5
tyler-rhel-newimage:827:1368 [4] NCCL INFO Setting affinity for GPU 4 to ffff,ffffff00,00000000
tyler-rhel-newimage:827:1368 [4] NCCL INFO NVLS multicast support is not available on dev 4
tyler-rhel-newimage:830:1364 [7] NCCL INFO Setting affinity for GPU 7 to ffff,ffffff00,00000000
tyler-rhel-newimage:830:1364 [7] NCCL INFO NVLS multicast support is not available on dev 7
tyler-rhel-newimage:825:1365 [2] NCCL INFO Setting affinity for GPU 2 to ff,ffffffff
tyler-rhel-newimage:825:1365 [2] NCCL INFO NVLS multicast support is not available on dev 2
tyler-rhel-newimage:801:1363 [0] NCCL INFO Setting affinity for GPU 0 to ff,ffffffff
tyler-rhel-newimage:801:1363 [0] NCCL INFO NVLS multicast support is not available on dev 0
tyler-rhel-newimage:827:1368 [4] NCCL INFO comm 0x555b9f749530 rank 4 nRanks 8 nNodes 1 localRanks 8 localRank 4 MNNVL 0
tyler-rhel-newimage:827:1368 [4] NCCL INFO Trees [0] 5/-1/-1->4->3 [1] 5/-1/-1->4->3 [2] 5/-1/-1->4->3 [3] 5/-1/-1->4->3 [4] 5/-1/-1->4->3 [5] 5/-1/-1->4->3 [6] 5/-1/-1->4->3 [7] 5/-1/-1->4->3 [8] 5/-1/-1->4->3 [9] 5/-1/-1->4->3 [10] 5/-1/-1->4->3 [11] 5/-1/-1->4->3 [12] 5/-1/-1->4->3 [13] 5/-1/-1->4->3 [14] 5/-1/-1->4->3 [15] 5/-1/-1->4->3 [16] 5/-1/-1->4->3 [17] 5/-1/-1->4->3 [18] 5/-1/-1->4->3 [19] 5/-1/-1->4->3 [20] 5/-1/-1->4->3 [21] 5/-1/-1->4->3 [22] 5/-1/-1->4->3 [23] 5/-1/-1->4->3
tyler-rhel-newimage:827:1368 [4] NCCL INFO P2P Chunksize set to 524288
tyler-rhel-newimage:830:1364 [7] NCCL INFO comm 0x555ba0534fd0 rank 7 nRanks 8 nNodes 1 localRanks 8 localRank 7 MNNVL 0
tyler-rhel-newimage:829:1369 [6] NCCL INFO comm 0x555ba114a940 rank 6 nRanks 8 nNodes 1 localRanks 8 localRank 6 MNNVL 0
tyler-rhel-newimage:830:1364 [7] NCCL INFO Trees [0] -1/-1/-1->7->6 [1] -1/-1/-1->7->6 [2] -1/-1/-1->7->6 [3] -1/-1/-1->7->6 [4] -1/-1/-1->7->6 [5] -1/-1/-1->7->6 [6] -1/-1/-1->7->6 [7] -1/-1/-1->7->6 [8] -1/-1/-1->7->6 [9] -1/-1/-1->7->6 [10] -1/-1/-1->7->6 [11] -1/-1/-1->7->6 [12] -1/-1/-1->7->6 [13] -1/-1/-1->7->6 [14] -1/-1/-1->7->6 [15] -1/-1/-1->7->6 [16] -1/-1/-1->7->6 [17] -1/-1/-1->7->6 [18] -1/-1/-1->7->6 [19] -1/-1/-1->7->6 [20] -1/-1/-1->7->6 [21] -1/-1/-1->7->6 [22] -1/-1/-1->7->6 [23] -1/-1/-1->7->6
tyler-rhel-newimage:828:1370 [5] NCCL INFO comm 0x555b9f749730 rank 5 nRanks 8 nNodes 1 localRanks 8 localRank 5 MNNVL 0
tyler-rhel-newimage:830:1364 [7] NCCL INFO P2P Chunksize set to 524288
tyler-rhel-newimage:829:1369 [6] NCCL INFO Trees [0] 7/-1/-1->6->5 [1] 7/-1/-1->6->5 [2] 7/-1/-1->6->5 [3] 7/-1/-1->6->5 [4] 7/-1/-1->6->5 [5] 7/-1/-1->6->5 [6] 7/-1/-1->6->5 [7] 7/-1/-1->6->5 [8] 7/-1/-1->6->5 [9] 7/-1/-1->6->5 [10] 7/-1/-1->6->5 [11] 7/-1/-1->6->5 [12] 7/-1/-1->6->5 [13] 7/-1/-1->6->5 [14] 7/-1/-1->6->5 [15] 7/-1/-1->6->5 [16] 7/-1/-1->6->5 [17] 7/-1/-1->6->5 [18] 7/-1/-1->6->5 [19] 7/-1/-1->6->5 [20] 7/-1/-1->6->5 [21] 7/-1/-1->6->5 [22] 7/-1/-1->6->5 [23] 7/-1/-1->6->5
tyler-rhel-newimage:829:1369 [6] NCCL INFO P2P Chunksize set to 524288
tyler-rhel-newimage:828:1370 [5] NCCL INFO Trees [0] 6/-1/-1->5->4 [1] 6/-1/-1->5->4 [2] 6/-1/-1->5->4 [3] 6/-1/-1->5->4 [4] 6/-1/-1->5->4 [5] 6/-1/-1->5->4 [6] 6/-1/-1->5->4 [7] 6/-1/-1->5->4 [8] 6/-1/-1->5->4 [9] 6/-1/-1->5->4 [10] 6/-1/-1->5->4 [11] 6/-1/-1->5->4 [12] 6/-1/-1->5->4 [13] 6/-1/-1->5->4 [14] 6/-1/-1->5->4 [15] 6/-1/-1->5->4 [16] 6/-1/-1->5->4 [17] 6/-1/-1->5->4 [18] 6/-1/-1->5->4 [19] 6/-1/-1->5->4 [20] 6/-1/-1->5->4 [21] 6/-1/-1->5->4 [22] 6/-1/-1->5->4 [23] 6/-1/-1->5->4
tyler-rhel-newimage:828:1370 [5] NCCL INFO P2P Chunksize set to 524288
tyler-rhel-newimage:825:1365 [2] NCCL INFO comm 0x555ba0547ff0 rank 2 nRanks 8 nNodes 1 localRanks 8 localRank 2 MNNVL 0
tyler-rhel-newimage:826:1366 [3] NCCL INFO comm 0x555ba0549a40 rank 3 nRanks 8 nNodes 1 localRanks 8 localRank 3 MNNVL 0
tyler-rhel-newimage:824:1367 [1] NCCL INFO comm 0x555b9f748ea0 rank 1 nRanks 8 nNodes 1 localRanks 8 localRank 1 MNNVL 0
tyler-rhel-newimage:825:1365 [2] NCCL INFO Trees [0] 3/-1/-1->2->1 [1] 3/-1/-1->2->1 [2] 3/-1/-1->2->1 [3] 3/-1/-1->2->1 [4] 3/-1/-1->2->1 [5] 3/-1/-1->2->1 [6] 3/-1/-1->2->1 [7] 3/-1/-1->2->1 [8] 3/-1/-1->2->1 [9] 3/-1/-1->2->1 [10] 3/-1/-1->2->1 [11] 3/-1/-1->2->1 [12] 3/-1/-1->2->1 [13] 3/-1/-1->2->1 [14] 3/-1/-1->2->1 [15] 3/-1/-1->2->1 [16] 3/-1/-1->2->1 [17] 3/-1/-1->2->1 [18] 3/-1/-1->2->1 [19] 3/-1/-1->2->1 [20] 3/-1/-1->2->1 [21] 3/-1/-1->2->1 [22] 3/-1/-1->2->1 [23] 3/-1/-1->2->1
tyler-rhel-newimage:824:1367 [1] NCCL INFO Trees [0] 2/-1/-1->1->0 [1] 2/-1/-1->1->0 [2] 2/-1/-1->1->0 [3] 2/-1/-1->1->0 [4] 2/-1/-1->1->0 [5] 2/-1/-1->1->0 [6] 2/-1/-1->1->0 [7] 2/-1/-1->1->0 [8] 2/-1/-1->1->0 [9] 2/-1/-1->1->0 [10] 2/-1/-1->1->0 [11] 2/-1/-1->1->0 [12] 2/-1/-1->1->0 [13] 2/-1/-1->1->0 [14] 2/-1/-1->1->0 [15] 2/-1/-1->1->0 [16] 2/-1/-1->1->0 [17] 2/-1/-1->1->0 [18] 2/-1/-1->1->0 [19] 2/-1/-1->1->0 [20] 2/-1/-1->1->0 [21] 2/-1/-1->1->0 [22] 2/-1/-1->1->0 [23] 2/-1/-1->1->0
tyler-rhel-newimage:826:1366 [3] NCCL INFO Trees [0] 4/-1/-1->3->2 [1] 4/-1/-1->3->2 [2] 4/-1/-1->3->2 [3] 4/-1/-1->3->2 [4] 4/-1/-1->3->2 [5] 4/-1/-1->3->2 [6] 4/-1/-1->3->2 [7] 4/-1/-1->3->2 [8] 4/-1/-1->3->2 [9] 4/-1/-1->3->2 [10] 4/-1/-1->3->2 [11] 4/-1/-1->3->2 [12] 4/-1/-1->3->2 [13] 4/-1/-1->3->2 [14] 4/-1/-1->3->2 [15] 4/-1/-1->3->2 [16] 4/-1/-1->3->2 [17] 4/-1/-1->3->2 [18] 4/-1/-1->3->2 [19] 4/-1/-1->3->2 [20] 4/-1/-1->3->2 [21] 4/-1/-1->3->2 [22] 4/-1/-1->3->2 [23] 4/-1/-1->3->2
tyler-rhel-newimage:825:1365 [2] NCCL INFO P2P Chunksize set to 524288
tyler-rhel-newimage:824:1367 [1] NCCL INFO P2P Chunksize set to 524288
tyler-rhel-newimage:801:1363 [0] NCCL INFO comm 0x555bab11e710 rank 0 nRanks 8 nNodes 1 localRanks 8 localRank 0 MNNVL 0
tyler-rhel-newimage:826:1366 [3] NCCL INFO P2P Chunksize set to 524288
tyler-rhel-newimage:801:1363 [0] NCCL INFO Channel 00/24 : 0 1 2 3 4 5 6 7
tyler-rhel-newimage:801:1363 [0] NCCL INFO Channel 01/24 : 0 1 2 3 4 5 6 7
tyler-rhel-newimage:801:1363 [0] NCCL INFO Channel 02/24 : 0 1 2 3 4 5 6 7
tyler-rhel-newimage:801:1363 [0] NCCL INFO Channel 03/24 : 0 1 2 3 4 5 6 7
tyler-rhel-newimage:801:1363 [0] NCCL INFO Channel 04/24 : 0 1 2 3 4 5 6 7
tyler-rhel-newimage:801:1363 [0] NCCL INFO Channel 05/24 : 0 1 2 3 4 5 6 7
tyler-rhel-newimage:801:1363 [0] NCCL INFO Channel 06/24 : 0 1 2 3 4 5 6 7
tyler-rhel-newimage:801:1363 [0] NCCL INFO Channel 07/24 : 0 1 2 3 4 5 6 7
tyler-rhel-newimage:801:1363 [0] NCCL INFO Channel 08/24 : 0 1 2 3 4 5 6 7
tyler-rhel-newimage:801:1363 [0] NCCL INFO Channel 09/24 : 0 1 2 3 4 5 6 7
tyler-rhel-newimage:801:1363 [0] NCCL INFO Channel 10/24 : 0 1 2 3 4 5 6 7
tyler-rhel-newimage:801:1363 [0] NCCL INFO Channel 11/24 : 0 1 2 3 4 5 6 7
tyler-rhel-newimage:801:1363 [0] NCCL INFO Channel 12/24 : 0 1 2 3 4 5 6 7
tyler-rhel-newimage:801:1363 [0] NCCL INFO Channel 13/24 : 0 1 2 3 4 5 6 7
tyler-rhel-newimage:801:1363 [0] NCCL INFO Channel 14/24 : 0 1 2 3 4 5 6 7
tyler-rhel-newimage:801:1363 [0] NCCL INFO Channel 15/24 : 0 1 2 3 4 5 6 7
tyler-rhel-newimage:801:1363 [0] NCCL INFO Channel 16/24 : 0 1 2 3 4 5 6 7
tyler-rhel-newimage:801:1363 [0] NCCL INFO Channel 17/24 : 0 1 2 3 4 5 6 7
tyler-rhel-newimage:801:1363 [0] NCCL INFO Channel 18/24 : 0 1 2 3 4 5 6 7
tyler-rhel-newimage:801:1363 [0] NCCL INFO Channel 19/24 : 0 1 2 3 4 5 6 7
tyler-rhel-newimage:801:1363 [0] NCCL INFO Channel 20/24 : 0 1 2 3 4 5 6 7
tyler-rhel-newimage:801:1363 [0] NCCL INFO Channel 21/24 : 0 1 2 3 4 5 6 7
tyler-rhel-newimage:801:1363 [0] NCCL INFO Channel 22/24 : 0 1 2 3 4 5 6 7
tyler-rhel-newimage:801:1363 [0] NCCL INFO Channel 23/24 : 0 1 2 3 4 5 6 7
tyler-rhel-newimage:801:1363 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1 [2] 1/-1/-1->0->-1 [3] 1/-1/-1->0->-1 [4] 1/-1/-1->0->-1 [5] 1/-1/-1->0->-1 [6] 1/-1/-1->0->-1 [7] 1/-1/-1->0->-1 [8] 1/-1/-1->0->-1 [9] 1/-1/-1->0->-1 [10] 1/-1/-1->0->-1 [11] 1/-1/-1->0->-1 [12] 1/-1/-1->0->-1 [13] 1/-1/-1->0->-1 [14] 1/-1/-1->0->-1 [15] 1/-1/-1->0->-1 [16] 1/-1/-1->0->-1 [17] 1/-1/-1->0->-1 [18] 1/-1/-1->0->-1 [19] 1/-1/-1->0->-1 [20] 1/-1/-1->0->-1 [21] 1/-1/-1->0->-1 [22] 1/-1/-1->0->-1 [23] 1/-1/-1->0->-1
tyler-rhel-newimage:801:1363 [0] NCCL INFO P2P Chunksize set to 524288
tyler-rhel-newimage:827:1368 [4] NCCL INFO Channel 00/0 : 4[4] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:827:1368 [4] NCCL INFO Channel 01/0 : 4[4] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:827:1368 [4] NCCL INFO Channel 02/0 : 4[4] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:827:1368 [4] NCCL INFO Channel 03/0 : 4[4] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:827:1368 [4] NCCL INFO Channel 04/0 : 4[4] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:827:1368 [4] NCCL INFO Channel 05/0 : 4[4] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:827:1368 [4] NCCL INFO Channel 06/0 : 4[4] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:830:1364 [7] NCCL INFO Channel 00/0 : 7[7] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:827:1368 [4] NCCL INFO Channel 07/0 : 4[4] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:830:1364 [7] NCCL INFO Channel 01/0 : 7[7] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:827:1368 [4] NCCL INFO Channel 08/0 : 4[4] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:830:1364 [7] NCCL INFO Channel 02/0 : 7[7] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:827:1368 [4] NCCL INFO Channel 09/0 : 4[4] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:830:1364 [7] NCCL INFO Channel 03/0 : 7[7] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:827:1368 [4] NCCL INFO Channel 10/0 : 4[4] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:830:1364 [7] NCCL INFO Channel 04/0 : 7[7] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:827:1368 [4] NCCL INFO Channel 11/0 : 4[4] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:826:1366 [3] NCCL INFO Channel 00/0 : 3[3] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:829:1369 [6] NCCL INFO Channel 00/0 : 6[6] -> 7[7] via P2P/IPC/read
tyler-rhel-newimage:830:1364 [7] NCCL INFO Channel 05/0 : 7[7] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:827:1368 [4] NCCL INFO Channel 12/0 : 4[4] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:828:1370 [5] NCCL INFO Channel 00/0 : 5[5] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:826:1366 [3] NCCL INFO Channel 01/0 : 3[3] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:825:1365 [2] NCCL INFO Channel 00/0 : 2[2] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:829:1369 [6] NCCL INFO Channel 01/0 : 6[6] -> 7[7] via P2P/IPC/read
tyler-rhel-newimage:830:1364 [7] NCCL INFO Channel 06/0 : 7[7] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:827:1368 [4] NCCL INFO Channel 13/0 : 4[4] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:828:1370 [5] NCCL INFO Channel 01/0 : 5[5] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:825:1365 [2] NCCL INFO Channel 01/0 : 2[2] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:826:1366 [3] NCCL INFO Channel 02/0 : 3[3] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:829:1369 [6] NCCL INFO Channel 02/0 : 6[6] -> 7[7] via P2P/IPC/read
tyler-rhel-newimage:830:1364 [7] NCCL INFO Channel 07/0 : 7[7] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:827:1368 [4] NCCL INFO Channel 14/0 : 4[4] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:828:1370 [5] NCCL INFO Channel 02/0 : 5[5] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:825:1365 [2] NCCL INFO Channel 02/0 : 2[2] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:826:1366 [3] NCCL INFO Channel 03/0 : 3[3] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:829:1369 [6] NCCL INFO Channel 03/0 : 6[6] -> 7[7] via P2P/IPC/read
tyler-rhel-newimage:830:1364 [7] NCCL INFO Channel 08/0 : 7[7] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:827:1368 [4] NCCL INFO Channel 15/0 : 4[4] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:825:1365 [2] NCCL INFO Channel 03/0 : 2[2] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:828:1370 [5] NCCL INFO Channel 03/0 : 5[5] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:826:1366 [3] NCCL INFO Channel 04/0 : 3[3] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:824:1367 [1] NCCL INFO Channel 00/0 : 1[1] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:829:1369 [6] NCCL INFO Channel 04/0 : 6[6] -> 7[7] via P2P/IPC/read
tyler-rhel-newimage:830:1364 [7] NCCL INFO Channel 09/0 : 7[7] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:827:1368 [4] NCCL INFO Channel 16/0 : 4[4] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:825:1365 [2] NCCL INFO Channel 04/0 : 2[2] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:826:1366 [3] NCCL INFO Channel 05/0 : 3[3] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:828:1370 [5] NCCL INFO Channel 04/0 : 5[5] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:824:1367 [1] NCCL INFO Channel 01/0 : 1[1] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:829:1369 [6] NCCL INFO Channel 05/0 : 6[6] -> 7[7] via P2P/IPC/read
tyler-rhel-newimage:830:1364 [7] NCCL INFO Channel 10/0 : 7[7] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:801:1363 [0] NCCL INFO Channel 00/0 : 0[0] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:827:1368 [4] NCCL INFO Channel 17/0 : 4[4] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:825:1365 [2] NCCL INFO Channel 05/0 : 2[2] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:826:1366 [3] NCCL INFO Channel 06/0 : 3[3] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:828:1370 [5] NCCL INFO Channel 05/0 : 5[5] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:824:1367 [1] NCCL INFO Channel 02/0 : 1[1] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:829:1369 [6] NCCL INFO Channel 06/0 : 6[6] -> 7[7] via P2P/IPC/read
tyler-rhel-newimage:830:1364 [7] NCCL INFO Channel 11/0 : 7[7] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:801:1363 [0] NCCL INFO Channel 01/0 : 0[0] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:827:1368 [4] NCCL INFO Channel 18/0 : 4[4] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:825:1365 [2] NCCL INFO Channel 06/0 : 2[2] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:826:1366 [3] NCCL INFO Channel 07/0 : 3[3] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:828:1370 [5] NCCL INFO Channel 06/0 : 5[5] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:824:1367 [1] NCCL INFO Channel 03/0 : 1[1] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:829:1369 [6] NCCL INFO Channel 07/0 : 6[6] -> 7[7] via P2P/IPC/read
tyler-rhel-newimage:830:1364 [7] NCCL INFO Channel 12/0 : 7[7] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:801:1363 [0] NCCL INFO Channel 02/0 : 0[0] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:827:1368 [4] NCCL INFO Channel 19/0 : 4[4] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:825:1365 [2] NCCL INFO Channel 07/0 : 2[2] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:826:1366 [3] NCCL INFO Channel 08/0 : 3[3] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:828:1370 [5] NCCL INFO Channel 07/0 : 5[5] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:824:1367 [1] NCCL INFO Channel 04/0 : 1[1] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:829:1369 [6] NCCL INFO Channel 08/0 : 6[6] -> 7[7] via P2P/IPC/read
tyler-rhel-newimage:830:1364 [7] NCCL INFO Channel 13/0 : 7[7] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:801:1363 [0] NCCL INFO Channel 03/0 : 0[0] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:827:1368 [4] NCCL INFO Channel 20/0 : 4[4] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:825:1365 [2] NCCL INFO Channel 08/0 : 2[2] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:826:1366 [3] NCCL INFO Channel 09/0 : 3[3] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:828:1370 [5] NCCL INFO Channel 08/0 : 5[5] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:824:1367 [1] NCCL INFO Channel 05/0 : 1[1] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:830:1364 [7] NCCL INFO Channel 14/0 : 7[7] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:829:1369 [6] NCCL INFO Channel 09/0 : 6[6] -> 7[7] via P2P/IPC/read
tyler-rhel-newimage:801:1363 [0] NCCL INFO Channel 04/0 : 0[0] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:827:1368 [4] NCCL INFO Channel 21/0 : 4[4] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:825:1365 [2] NCCL INFO Channel 09/0 : 2[2] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:826:1366 [3] NCCL INFO Channel 10/0 : 3[3] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:824:1367 [1] NCCL INFO Channel 06/0 : 1[1] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:828:1370 [5] NCCL INFO Channel 09/0 : 5[5] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:830:1364 [7] NCCL INFO Channel 15/0 : 7[7] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:829:1369 [6] NCCL INFO Channel 10/0 : 6[6] -> 7[7] via P2P/IPC/read
tyler-rhel-newimage:801:1363 [0] NCCL INFO Channel 05/0 : 0[0] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:827:1368 [4] NCCL INFO Channel 22/0 : 4[4] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:826:1366 [3] NCCL INFO Channel 11/0 : 3[3] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:825:1365 [2] NCCL INFO Channel 10/0 : 2[2] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:824:1367 [1] NCCL INFO Channel 07/0 : 1[1] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:828:1370 [5] NCCL INFO Channel 10/0 : 5[5] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:830:1364 [7] NCCL INFO Channel 16/0 : 7[7] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:829:1369 [6] NCCL INFO Channel 11/0 : 6[6] -> 7[7] via P2P/IPC/read
tyler-rhel-newimage:801:1363 [0] NCCL INFO Channel 06/0 : 0[0] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:827:1368 [4] NCCL INFO Channel 23/0 : 4[4] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:826:1366 [3] NCCL INFO Channel 12/0 : 3[3] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:825:1365 [2] NCCL INFO Channel 11/0 : 2[2] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:824:1367 [1] NCCL INFO Channel 08/0 : 1[1] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:830:1364 [7] NCCL INFO Channel 17/0 : 7[7] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:828:1370 [5] NCCL INFO Channel 11/0 : 5[5] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:829:1369 [6] NCCL INFO Channel 12/0 : 6[6] -> 7[7] via P2P/IPC/read
tyler-rhel-newimage:801:1363 [0] NCCL INFO Channel 07/0 : 0[0] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:826:1366 [3] NCCL INFO Channel 13/0 : 3[3] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:825:1365 [2] NCCL INFO Channel 12/0 : 2[2] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:824:1367 [1] NCCL INFO Channel 09/0 : 1[1] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:830:1364 [7] NCCL INFO Channel 18/0 : 7[7] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:828:1370 [5] NCCL INFO Channel 12/0 : 5[5] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:829:1369 [6] NCCL INFO Channel 13/0 : 6[6] -> 7[7] via P2P/IPC/read
tyler-rhel-newimage:801:1363 [0] NCCL INFO Channel 08/0 : 0[0] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:826:1366 [3] NCCL INFO Channel 14/0 : 3[3] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:825:1365 [2] NCCL INFO Channel 13/0 : 2[2] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:824:1367 [1] NCCL INFO Channel 10/0 : 1[1] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:830:1364 [7] NCCL INFO Channel 19/0 : 7[7] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:828:1370 [5] NCCL INFO Channel 13/0 : 5[5] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:829:1369 [6] NCCL INFO Channel 14/0 : 6[6] -> 7[7] via P2P/IPC/read
tyler-rhel-newimage:801:1363 [0] NCCL INFO Channel 09/0 : 0[0] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:826:1366 [3] NCCL INFO Channel 15/0 : 3[3] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:825:1365 [2] NCCL INFO Channel 14/0 : 2[2] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:824:1367 [1] NCCL INFO Channel 11/0 : 1[1] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:830:1364 [7] NCCL INFO Channel 20/0 : 7[7] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:828:1370 [5] NCCL INFO Channel 14/0 : 5[5] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:829:1369 [6] NCCL INFO Channel 15/0 : 6[6] -> 7[7] via P2P/IPC/read
tyler-rhel-newimage:801:1363 [0] NCCL INFO Channel 10/0 : 0[0] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:826:1366 [3] NCCL INFO Channel 16/0 : 3[3] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:824:1367 [1] NCCL INFO Channel 12/0 : 1[1] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:825:1365 [2] NCCL INFO Channel 15/0 : 2[2] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:830:1364 [7] NCCL INFO Channel 21/0 : 7[7] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:828:1370 [5] NCCL INFO Channel 15/0 : 5[5] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:829:1369 [6] NCCL INFO Channel 16/0 : 6[6] -> 7[7] via P2P/IPC/read
tyler-rhel-newimage:801:1363 [0] NCCL INFO Channel 11/0 : 0[0] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:826:1366 [3] NCCL INFO Channel 17/0 : 3[3] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:824:1367 [1] NCCL INFO Channel 13/0 : 1[1] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:825:1365 [2] NCCL INFO Channel 16/0 : 2[2] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:830:1364 [7] NCCL INFO Channel 22/0 : 7[7] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:828:1370 [5] NCCL INFO Channel 16/0 : 5[5] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:829:1369 [6] NCCL INFO Channel 17/0 : 6[6] -> 7[7] via P2P/IPC/read
tyler-rhel-newimage:826:1366 [3] NCCL INFO Channel 18/0 : 3[3] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:801:1363 [0] NCCL INFO Channel 12/0 : 0[0] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:824:1367 [1] NCCL INFO Channel 14/0 : 1[1] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:825:1365 [2] NCCL INFO Channel 17/0 : 2[2] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:830:1364 [7] NCCL INFO Channel 23/0 : 7[7] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:828:1370 [5] NCCL INFO Channel 17/0 : 5[5] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:826:1366 [3] NCCL INFO Channel 19/0 : 3[3] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:829:1369 [6] NCCL INFO Channel 18/0 : 6[6] -> 7[7] via P2P/IPC/read
tyler-rhel-newimage:801:1363 [0] NCCL INFO Channel 13/0 : 0[0] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:824:1367 [1] NCCL INFO Channel 15/0 : 1[1] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:825:1365 [2] NCCL INFO Channel 18/0 : 2[2] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:828:1370 [5] NCCL INFO Channel 18/0 : 5[5] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:826:1366 [3] NCCL INFO Channel 20/0 : 3[3] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:824:1367 [1] NCCL INFO Channel 16/0 : 1[1] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:801:1363 [0] NCCL INFO Channel 14/0 : 0[0] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:829:1369 [6] NCCL INFO Channel 19/0 : 6[6] -> 7[7] via P2P/IPC/read
tyler-rhel-newimage:825:1365 [2] NCCL INFO Channel 19/0 : 2[2] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:828:1370 [5] NCCL INFO Channel 19/0 : 5[5] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:826:1366 [3] NCCL INFO Channel 21/0 : 3[3] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:824:1367 [1] NCCL INFO Channel 17/0 : 1[1] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:801:1363 [0] NCCL INFO Channel 15/0 : 0[0] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:829:1369 [6] NCCL INFO Channel 20/0 : 6[6] -> 7[7] via P2P/IPC/read
tyler-rhel-newimage:825:1365 [2] NCCL INFO Channel 20/0 : 2[2] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:828:1370 [5] NCCL INFO Channel 20/0 : 5[5] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:826:1366 [3] NCCL INFO Channel 22/0 : 3[3] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:824:1367 [1] NCCL INFO Channel 18/0 : 1[1] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:801:1363 [0] NCCL INFO Channel 16/0 : 0[0] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:829:1369 [6] NCCL INFO Channel 21/0 : 6[6] -> 7[7] via P2P/IPC/read
tyler-rhel-newimage:825:1365 [2] NCCL INFO Channel 21/0 : 2[2] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:828:1370 [5] NCCL INFO Channel 21/0 : 5[5] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:826:1366 [3] NCCL INFO Channel 23/0 : 3[3] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:824:1367 [1] NCCL INFO Channel 19/0 : 1[1] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:825:1365 [2] NCCL INFO Channel 22/0 : 2[2] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:801:1363 [0] NCCL INFO Channel 17/0 : 0[0] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:829:1369 [6] NCCL INFO Channel 22/0 : 6[6] -> 7[7] via P2P/IPC/read
tyler-rhel-newimage:828:1370 [5] NCCL INFO Channel 22/0 : 5[5] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:824:1367 [1] NCCL INFO Channel 20/0 : 1[1] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:801:1363 [0] NCCL INFO Channel 18/0 : 0[0] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:825:1365 [2] NCCL INFO Channel 23/0 : 2[2] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:824:1367 [1] NCCL INFO Channel 21/0 : 1[1] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:829:1369 [6] NCCL INFO Channel 23/0 : 6[6] -> 7[7] via P2P/IPC/read
tyler-rhel-newimage:828:1370 [5] NCCL INFO Channel 23/0 : 5[5] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:801:1363 [0] NCCL INFO Channel 19/0 : 0[0] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:824:1367 [1] NCCL INFO Channel 22/0 : 1[1] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:801:1363 [0] NCCL INFO Channel 20/0 : 0[0] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:824:1367 [1] NCCL INFO Channel 23/0 : 1[1] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:801:1363 [0] NCCL INFO Channel 21/0 : 0[0] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:801:1363 [0] NCCL INFO Channel 22/0 : 0[0] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:801:1363 [0] NCCL INFO Channel 23/0 : 0[0] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:828:1370 [5] NCCL INFO Connected all rings
tyler-rhel-newimage:826:1366 [3] NCCL INFO Connected all rings
tyler-rhel-newimage:827:1368 [4] NCCL INFO Connected all rings
tyler-rhel-newimage:825:1365 [2] NCCL INFO Connected all rings
tyler-rhel-newimage:824:1367 [1] NCCL INFO Connected all rings
tyler-rhel-newimage:801:1363 [0] NCCL INFO Connected all rings
tyler-rhel-newimage:829:1369 [6] NCCL INFO Connected all rings
tyler-rhel-newimage:830:1364 [7] NCCL INFO Connected all rings
tyler-rhel-newimage:830:1364 [7] NCCL INFO Channel 00/0 : 7[7] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:830:1364 [7] NCCL INFO Channel 01/0 : 7[7] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:830:1364 [7] NCCL INFO Channel 02/0 : 7[7] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:830:1364 [7] NCCL INFO Channel 03/0 : 7[7] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:830:1364 [7] NCCL INFO Channel 04/0 : 7[7] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:830:1364 [7] NCCL INFO Channel 05/0 : 7[7] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:830:1364 [7] NCCL INFO Channel 06/0 : 7[7] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:830:1364 [7] NCCL INFO Channel 07/0 : 7[7] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:830:1364 [7] NCCL INFO Channel 08/0 : 7[7] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:830:1364 [7] NCCL INFO Channel 09/0 : 7[7] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:830:1364 [7] NCCL INFO Channel 10/0 : 7[7] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:830:1364 [7] NCCL INFO Channel 11/0 : 7[7] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:830:1364 [7] NCCL INFO Channel 12/0 : 7[7] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:830:1364 [7] NCCL INFO Channel 13/0 : 7[7] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:830:1364 [7] NCCL INFO Channel 14/0 : 7[7] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:830:1364 [7] NCCL INFO Channel 15/0 : 7[7] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:830:1364 [7] NCCL INFO Channel 16/0 : 7[7] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:830:1364 [7] NCCL INFO Channel 17/0 : 7[7] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:830:1364 [7] NCCL INFO Channel 18/0 : 7[7] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:827:1368 [4] NCCL INFO Channel 00/0 : 4[4] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:830:1364 [7] NCCL INFO Channel 19/0 : 7[7] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:825:1365 [2] NCCL INFO Channel 00/0 : 2[2] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:827:1368 [4] NCCL INFO Channel 01/0 : 4[4] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:830:1364 [7] NCCL INFO Channel 20/0 : 7[7] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:828:1370 [5] NCCL INFO Channel 00/0 : 5[5] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:825:1365 [2] NCCL INFO Channel 01/0 : 2[2] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:824:1367 [1] NCCL INFO Channel 00/0 : 1[1] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:827:1368 [4] NCCL INFO Channel 02/0 : 4[4] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:830:1364 [7] NCCL INFO Channel 21/0 : 7[7] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:828:1370 [5] NCCL INFO Channel 01/0 : 5[5] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:825:1365 [2] NCCL INFO Channel 02/0 : 2[2] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:824:1367 [1] NCCL INFO Channel 01/0 : 1[1] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:827:1368 [4] NCCL INFO Channel 03/0 : 4[4] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:830:1364 [7] NCCL INFO Channel 22/0 : 7[7] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:828:1370 [5] NCCL INFO Channel 02/0 : 5[5] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:825:1365 [2] NCCL INFO Channel 03/0 : 2[2] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:824:1367 [1] NCCL INFO Channel 02/0 : 1[1] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:827:1368 [4] NCCL INFO Channel 04/0 : 4[4] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:830:1364 [7] NCCL INFO Channel 23/0 : 7[7] -> 6[6] via P2P/IPC/read
tyler-rhel-newimage:825:1365 [2] NCCL INFO Channel 04/0 : 2[2] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:828:1370 [5] NCCL INFO Channel 03/0 : 5[5] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:827:1368 [4] NCCL INFO Channel 05/0 : 4[4] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:824:1367 [1] NCCL INFO Channel 03/0 : 1[1] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:829:1369 [6] NCCL INFO Channel 00/0 : 6[6] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:825:1365 [2] NCCL INFO Channel 05/0 : 2[2] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:828:1370 [5] NCCL INFO Channel 04/0 : 5[5] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:826:1366 [3] NCCL INFO Channel 00/0 : 3[3] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:827:1368 [4] NCCL INFO Channel 06/0 : 4[4] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:824:1367 [1] NCCL INFO Channel 04/0 : 1[1] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:825:1365 [2] NCCL INFO Channel 06/0 : 2[2] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:829:1369 [6] NCCL INFO Channel 01/0 : 6[6] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:828:1370 [5] NCCL INFO Channel 05/0 : 5[5] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:826:1366 [3] NCCL INFO Channel 01/0 : 3[3] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:827:1368 [4] NCCL INFO Channel 07/0 : 4[4] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:824:1367 [1] NCCL INFO Channel 05/0 : 1[1] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:825:1365 [2] NCCL INFO Channel 07/0 : 2[2] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:829:1369 [6] NCCL INFO Channel 02/0 : 6[6] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:828:1370 [5] NCCL INFO Channel 06/0 : 5[5] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:826:1366 [3] NCCL INFO Channel 02/0 : 3[3] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:827:1368 [4] NCCL INFO Channel 08/0 : 4[4] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:824:1367 [1] NCCL INFO Channel 06/0 : 1[1] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:825:1365 [2] NCCL INFO Channel 08/0 : 2[2] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:829:1369 [6] NCCL INFO Channel 03/0 : 6[6] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:828:1370 [5] NCCL INFO Channel 07/0 : 5[5] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:826:1366 [3] NCCL INFO Channel 03/0 : 3[3] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:827:1368 [4] NCCL INFO Channel 09/0 : 4[4] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:824:1367 [1] NCCL INFO Channel 07/0 : 1[1] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:829:1369 [6] NCCL INFO Channel 04/0 : 6[6] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:825:1365 [2] NCCL INFO Channel 09/0 : 2[2] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:828:1370 [5] NCCL INFO Channel 08/0 : 5[5] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:826:1366 [3] NCCL INFO Channel 04/0 : 3[3] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:827:1368 [4] NCCL INFO Channel 10/0 : 4[4] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:824:1367 [1] NCCL INFO Channel 08/0 : 1[1] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:829:1369 [6] NCCL INFO Channel 05/0 : 6[6] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:825:1365 [2] NCCL INFO Channel 10/0 : 2[2] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:826:1366 [3] NCCL INFO Channel 05/0 : 3[3] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:828:1370 [5] NCCL INFO Channel 09/0 : 5[5] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:827:1368 [4] NCCL INFO Channel 11/0 : 4[4] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:829:1369 [6] NCCL INFO Channel 06/0 : 6[6] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:824:1367 [1] NCCL INFO Channel 09/0 : 1[1] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:825:1365 [2] NCCL INFO Channel 11/0 : 2[2] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:826:1366 [3] NCCL INFO Channel 06/0 : 3[3] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:828:1370 [5] NCCL INFO Channel 10/0 : 5[5] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:827:1368 [4] NCCL INFO Channel 12/0 : 4[4] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:824:1367 [1] NCCL INFO Channel 10/0 : 1[1] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:829:1369 [6] NCCL INFO Channel 07/0 : 6[6] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:825:1365 [2] NCCL INFO Channel 12/0 : 2[2] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:826:1366 [3] NCCL INFO Channel 07/0 : 3[3] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:828:1370 [5] NCCL INFO Channel 11/0 : 5[5] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:827:1368 [4] NCCL INFO Channel 13/0 : 4[4] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:824:1367 [1] NCCL INFO Channel 11/0 : 1[1] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:825:1365 [2] NCCL INFO Channel 13/0 : 2[2] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:829:1369 [6] NCCL INFO Channel 08/0 : 6[6] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:826:1366 [3] NCCL INFO Channel 08/0 : 3[3] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:828:1370 [5] NCCL INFO Channel 12/0 : 5[5] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:827:1368 [4] NCCL INFO Channel 14/0 : 4[4] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:824:1367 [1] NCCL INFO Channel 12/0 : 1[1] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:825:1365 [2] NCCL INFO Channel 14/0 : 2[2] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:829:1369 [6] NCCL INFO Channel 09/0 : 6[6] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:826:1366 [3] NCCL INFO Channel 09/0 : 3[3] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:827:1368 [4] NCCL INFO Channel 15/0 : 4[4] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:828:1370 [5] NCCL INFO Channel 13/0 : 5[5] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:824:1367 [1] NCCL INFO Channel 13/0 : 1[1] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:825:1365 [2] NCCL INFO Channel 15/0 : 2[2] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:829:1369 [6] NCCL INFO Channel 10/0 : 6[6] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:826:1366 [3] NCCL INFO Channel 10/0 : 3[3] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:827:1368 [4] NCCL INFO Channel 16/0 : 4[4] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:828:1370 [5] NCCL INFO Channel 14/0 : 5[5] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:824:1367 [1] NCCL INFO Channel 14/0 : 1[1] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:825:1365 [2] NCCL INFO Channel 16/0 : 2[2] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:829:1369 [6] NCCL INFO Channel 11/0 : 6[6] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:826:1366 [3] NCCL INFO Channel 11/0 : 3[3] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:827:1368 [4] NCCL INFO Channel 17/0 : 4[4] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:828:1370 [5] NCCL INFO Channel 15/0 : 5[5] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:825:1365 [2] NCCL INFO Channel 17/0 : 2[2] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:824:1367 [1] NCCL INFO Channel 15/0 : 1[1] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:829:1369 [6] NCCL INFO Channel 12/0 : 6[6] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:826:1366 [3] NCCL INFO Channel 12/0 : 3[3] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:827:1368 [4] NCCL INFO Channel 18/0 : 4[4] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:828:1370 [5] NCCL INFO Channel 16/0 : 5[5] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:825:1365 [2] NCCL INFO Channel 18/0 : 2[2] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:824:1367 [1] NCCL INFO Channel 16/0 : 1[1] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:829:1369 [6] NCCL INFO Channel 13/0 : 6[6] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:826:1366 [3] NCCL INFO Channel 13/0 : 3[3] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:827:1368 [4] NCCL INFO Channel 19/0 : 4[4] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:828:1370 [5] NCCL INFO Channel 17/0 : 5[5] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:825:1365 [2] NCCL INFO Channel 19/0 : 2[2] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:824:1367 [1] NCCL INFO Channel 17/0 : 1[1] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:826:1366 [3] NCCL INFO Channel 14/0 : 3[3] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:829:1369 [6] NCCL INFO Channel 14/0 : 6[6] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:827:1368 [4] NCCL INFO Channel 20/0 : 4[4] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:828:1370 [5] NCCL INFO Channel 18/0 : 5[5] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:825:1365 [2] NCCL INFO Channel 20/0 : 2[2] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:824:1367 [1] NCCL INFO Channel 18/0 : 1[1] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:826:1366 [3] NCCL INFO Channel 15/0 : 3[3] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:829:1369 [6] NCCL INFO Channel 15/0 : 6[6] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:827:1368 [4] NCCL INFO Channel 21/0 : 4[4] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:828:1370 [5] NCCL INFO Channel 19/0 : 5[5] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:825:1365 [2] NCCL INFO Channel 21/0 : 2[2] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:824:1367 [1] NCCL INFO Channel 19/0 : 1[1] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:826:1366 [3] NCCL INFO Channel 16/0 : 3[3] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:829:1369 [6] NCCL INFO Channel 16/0 : 6[6] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:827:1368 [4] NCCL INFO Channel 22/0 : 4[4] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:828:1370 [5] NCCL INFO Channel 20/0 : 5[5] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:825:1365 [2] NCCL INFO Channel 22/0 : 2[2] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:824:1367 [1] NCCL INFO Channel 20/0 : 1[1] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:826:1366 [3] NCCL INFO Channel 17/0 : 3[3] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:829:1369 [6] NCCL INFO Channel 17/0 : 6[6] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:827:1368 [4] NCCL INFO Channel 23/0 : 4[4] -> 3[3] via P2P/IPC/read
tyler-rhel-newimage:828:1370 [5] NCCL INFO Channel 21/0 : 5[5] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:825:1365 [2] NCCL INFO Channel 23/0 : 2[2] -> 1[1] via P2P/IPC/read
tyler-rhel-newimage:824:1367 [1] NCCL INFO Channel 21/0 : 1[1] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:826:1366 [3] NCCL INFO Channel 18/0 : 3[3] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:829:1369 [6] NCCL INFO Channel 18/0 : 6[6] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:828:1370 [5] NCCL INFO Channel 22/0 : 5[5] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:824:1367 [1] NCCL INFO Channel 22/0 : 1[1] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:826:1366 [3] NCCL INFO Channel 19/0 : 3[3] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:829:1369 [6] NCCL INFO Channel 19/0 : 6[6] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:828:1370 [5] NCCL INFO Channel 23/0 : 5[5] -> 4[4] via P2P/IPC/read
tyler-rhel-newimage:826:1366 [3] NCCL INFO Channel 20/0 : 3[3] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:824:1367 [1] NCCL INFO Channel 23/0 : 1[1] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:829:1369 [6] NCCL INFO Channel 20/0 : 6[6] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:826:1366 [3] NCCL INFO Channel 21/0 : 3[3] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:829:1369 [6] NCCL INFO Channel 21/0 : 6[6] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:826:1366 [3] NCCL INFO Channel 22/0 : 3[3] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:829:1369 [6] NCCL INFO Channel 22/0 : 6[6] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:826:1366 [3] NCCL INFO Channel 23/0 : 3[3] -> 2[2] via P2P/IPC/read
tyler-rhel-newimage:829:1369 [6] NCCL INFO Channel 23/0 : 6[6] -> 5[5] via P2P/IPC/read
tyler-rhel-newimage:801:1363 [0] NCCL INFO Connected all trees
tyler-rhel-newimage:801:1363 [0] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512
tyler-rhel-newimage:801:1363 [0] NCCL INFO 24 coll channels, 0 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer
tyler-rhel-newimage:824:1367 [1] NCCL INFO Connected all trees
tyler-rhel-newimage:824:1367 [1] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512
tyler-rhel-newimage:824:1367 [1] NCCL INFO 24 coll channels, 0 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer
tyler-rhel-newimage:825:1365 [2] NCCL INFO Connected all trees
tyler-rhel-newimage:825:1365 [2] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512
tyler-rhel-newimage:825:1365 [2] NCCL INFO 24 coll channels, 0 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer
tyler-rhel-newimage:826:1366 [3] NCCL INFO Connected all trees
tyler-rhel-newimage:826:1366 [3] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512
tyler-rhel-newimage:826:1366 [3] NCCL INFO 24 coll channels, 0 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer
tyler-rhel-newimage:827:1368 [4] NCCL INFO Connected all trees
tyler-rhel-newimage:827:1368 [4] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512
tyler-rhel-newimage:827:1368 [4] NCCL INFO 24 coll channels, 0 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer
tyler-rhel-newimage:828:1370 [5] NCCL INFO Connected all trees
tyler-rhel-newimage:828:1370 [5] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512
tyler-rhel-newimage:828:1370 [5] NCCL INFO 24 coll channels, 0 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer
tyler-rhel-newimage:830:1364 [7] NCCL INFO Connected all trees
tyler-rhel-newimage:830:1364 [7] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512
tyler-rhel-newimage:830:1364 [7] NCCL INFO 24 coll channels, 0 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer
tyler-rhel-newimage:829:1369 [6] NCCL INFO Connected all trees
tyler-rhel-newimage:829:1369 [6] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512
tyler-rhel-newimage:829:1369 [6] NCCL INFO 24 coll channels, 0 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer
INFO 2024-07-27 18:38:01,005 instructlab.model.backends.backends:336: Waiting for the vLLM server to start at http://127.0.0.1:59587/v1, this might take a moment... Attempt: 8/80
tyler-rhel-newimage:826:1366 [3] NCCL INFO comm 0x555ba0549a40 rank 3 nranks 8 cudaDev 3 nvmlDev 3 busId a040 commId 0xd56327a586613906 - Init COMPLETE
tyler-rhel-newimage:828:1370 [5] NCCL INFO comm 0x555b9f749730 rank 5 nranks 8 cudaDev 5 nvmlDev 5 busId c060 commId 0xd56327a586613906 - Init COMPLETE
tyler-rhel-newimage:830:1364 [7] NCCL INFO comm 0x555ba0534fd0 rank 7 nranks 8 cudaDev 7 nvmlDev 7 busId e080 commId 0xd56327a586613906 - Init COMPLETE
tyler-rhel-newimage:827:1368 [4] NCCL INFO comm 0x555b9f749530 rank 4 nranks 8 cudaDev 4 nvmlDev 4 busId c050 commId 0xd56327a586613906 - Init COMPLETE
tyler-rhel-newimage:824:1367 [1] NCCL INFO comm 0x555b9f748ea0 rank 1 nranks 8 cudaDev 1 nvmlDev 1 busId 8020 commId 0xd56327a586613906 - Init COMPLETE
tyler-rhel-newimage:829:1369 [6] NCCL INFO comm 0x555ba114a940 rank 6 nranks 8 cudaDev 6 nvmlDev 6 busId e070 commId 0xd56327a586613906 - Init COMPLETE
tyler-rhel-newimage:825:1365 [2] NCCL INFO comm 0x555ba0547ff0 rank 2 nranks 8 cudaDev 2 nvmlDev 2 busId a030 commId 0xd56327a586613906 - Init COMPLETE
tyler-rhel-newimage:801:1363 [0] NCCL INFO comm 0x555bab11e710 rank 0 nranks 8 cudaDev 0 nvmlDev 0 busId 8010 commId 0xd56327a586613906 - Init COMPLETE
tyler-rhel-newimage:827:1603 [4] NCCL INFO Channel 00/1 : 4[4] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:830:1604 [7] NCCL INFO Channel 00/1 : 7[7] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:829:1605 [6] NCCL INFO Channel 00/1 : 6[6] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:827:1603 [4] NCCL INFO Channel 01/1 : 4[4] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:830:1604 [7] NCCL INFO Channel 01/1 : 7[7] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:828:1606 [5] NCCL INFO Channel 00/1 : 5[5] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:827:1603 [4] NCCL INFO Channel 02/1 : 4[4] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:830:1604 [7] NCCL INFO Channel 02/1 : 7[7] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:829:1605 [6] NCCL INFO Channel 01/1 : 6[6] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:827:1603 [4] NCCL INFO Channel 03/1 : 4[4] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:830:1604 [7] NCCL INFO Channel 03/1 : 7[7] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:828:1606 [5] NCCL INFO Channel 01/1 : 5[5] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:829:1605 [6] NCCL INFO Channel 02/1 : 6[6] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:830:1604 [7] NCCL INFO Channel 04/1 : 7[7] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:827:1603 [4] NCCL INFO Channel 04/1 : 4[4] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:828:1606 [5] NCCL INFO Channel 02/1 : 5[5] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:829:1605 [6] NCCL INFO Channel 03/1 : 6[6] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:830:1604 [7] NCCL INFO Channel 05/1 : 7[7] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:827:1603 [4] NCCL INFO Channel 05/1 : 4[4] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:828:1606 [5] NCCL INFO Channel 03/1 : 5[5] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:829:1605 [6] NCCL INFO Channel 04/1 : 6[6] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:830:1604 [7] NCCL INFO Channel 06/1 : 7[7] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:827:1603 [4] NCCL INFO Channel 06/1 : 4[4] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:828:1606 [5] NCCL INFO Channel 04/1 : 5[5] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:830:1604 [7] NCCL INFO Channel 07/1 : 7[7] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:829:1605 [6] NCCL INFO Channel 05/1 : 6[6] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:830:1604 [7] NCCL INFO Channel 08/1 : 7[7] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:829:1605 [6] NCCL INFO Channel 06/1 : 6[6] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:828:1606 [5] NCCL INFO Channel 05/1 : 5[5] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:827:1603 [4] NCCL INFO Channel 07/1 : 4[4] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:830:1604 [7] NCCL INFO Channel 09/1 : 7[7] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:828:1606 [5] NCCL INFO Channel 06/1 : 5[5] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:829:1605 [6] NCCL INFO Channel 07/1 : 6[6] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:827:1603 [4] NCCL INFO Channel 08/1 : 4[4] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:830:1604 [7] NCCL INFO Channel 10/1 : 7[7] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:829:1605 [6] NCCL INFO Channel 08/1 : 6[6] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:828:1606 [5] NCCL INFO Channel 07/1 : 5[5] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:827:1603 [4] NCCL INFO Channel 09/1 : 4[4] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:830:1604 [7] NCCL INFO Channel 11/1 : 7[7] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:828:1606 [5] NCCL INFO Channel 08/1 : 5[5] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:829:1605 [6] NCCL INFO Channel 09/1 : 6[6] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:827:1603 [4] NCCL INFO Channel 10/1 : 4[4] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:830:1604 [7] NCCL INFO Channel 12/1 : 7[7] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:829:1605 [6] NCCL INFO Channel 10/1 : 6[6] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:828:1606 [5] NCCL INFO Channel 09/1 : 5[5] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:827:1603 [4] NCCL INFO Channel 11/1 : 4[4] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:830:1604 [7] NCCL INFO Channel 13/1 : 7[7] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:828:1606 [5] NCCL INFO Channel 10/1 : 5[5] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:829:1605 [6] NCCL INFO Channel 11/1 : 6[6] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:827:1603 [4] NCCL INFO Channel 12/1 : 4[4] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:830:1604 [7] NCCL INFO Channel 14/1 : 7[7] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:828:1606 [5] NCCL INFO Channel 11/1 : 5[5] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:829:1605 [6] NCCL INFO Channel 12/1 : 6[6] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:830:1604 [7] NCCL INFO Channel 15/1 : 7[7] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:827:1603 [4] NCCL INFO Channel 13/1 : 4[4] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:830:1604 [7] NCCL INFO Channel 16/1 : 7[7] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:829:1605 [6] NCCL INFO Channel 13/1 : 6[6] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:828:1606 [5] NCCL INFO Channel 12/1 : 5[5] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:827:1603 [4] NCCL INFO Channel 14/1 : 4[4] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:830:1604 [7] NCCL INFO Channel 17/1 : 7[7] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:829:1605 [6] NCCL INFO Channel 14/1 : 6[6] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:828:1606 [5] NCCL INFO Channel 13/1 : 5[5] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:827:1603 [4] NCCL INFO Channel 15/1 : 4[4] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:830:1604 [7] NCCL INFO Channel 18/1 : 7[7] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:829:1605 [6] NCCL INFO Channel 15/1 : 6[6] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:828:1606 [5] NCCL INFO Channel 14/1 : 5[5] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:827:1603 [4] NCCL INFO Channel 16/1 : 4[4] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:830:1604 [7] NCCL INFO Channel 19/1 : 7[7] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:826:1608 [3] NCCL INFO Channel 00/1 : 3[3] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:824:1609 [1] NCCL INFO Channel 00/1 : 1[1] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:825:1610 [2] NCCL INFO Channel 00/1 : 2[2] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:830:1604 [7] NCCL INFO Channel 20/1 : 7[7] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:828:1606 [5] NCCL INFO Channel 15/1 : 5[5] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:827:1603 [4] NCCL INFO Channel 17/1 : 4[4] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:824:1609 [1] NCCL INFO Channel 01/1 : 1[1] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:829:1605 [6] NCCL INFO Channel 16/1 : 6[6] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:826:1608 [3] NCCL INFO Channel 01/1 : 3[3] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:825:1610 [2] NCCL INFO Channel 01/1 : 2[2] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:830:1604 [7] NCCL INFO Channel 21/1 : 7[7] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:828:1606 [5] NCCL INFO Channel 16/1 : 5[5] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:827:1603 [4] NCCL INFO Channel 18/1 : 4[4] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:824:1609 [1] NCCL INFO Channel 02/1 : 1[1] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:826:1608 [3] NCCL INFO Channel 02/1 : 3[3] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:829:1605 [6] NCCL INFO Channel 17/1 : 6[6] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:825:1610 [2] NCCL INFO Channel 02/1 : 2[2] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:830:1604 [7] NCCL INFO Channel 22/1 : 7[7] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:828:1606 [5] NCCL INFO Channel 17/1 : 5[5] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:827:1603 [4] NCCL INFO Channel 19/1 : 4[4] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:824:1609 [1] NCCL INFO Channel 03/1 : 1[1] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:826:1608 [3] NCCL INFO Channel 03/1 : 3[3] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:829:1605 [6] NCCL INFO Channel 18/1 : 6[6] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:825:1610 [2] NCCL INFO Channel 03/1 : 2[2] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:830:1604 [7] NCCL INFO Channel 23/1 : 7[7] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:828:1606 [5] NCCL INFO Channel 18/1 : 5[5] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:827:1603 [4] NCCL INFO Channel 20/1 : 4[4] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:824:1609 [1] NCCL INFO Channel 04/1 : 1[1] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:826:1608 [3] NCCL INFO Channel 04/1 : 3[3] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:829:1605 [6] NCCL INFO Channel 19/1 : 6[6] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:825:1610 [2] NCCL INFO Channel 04/1 : 2[2] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:830:1604 [7] NCCL INFO Channel 24/1 : 7[7] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:824:1609 [1] NCCL INFO Channel 05/1 : 1[1] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:828:1606 [5] NCCL INFO Channel 19/1 : 5[5] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:826:1608 [3] NCCL INFO Channel 05/1 : 3[3] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:827:1603 [4] NCCL INFO Channel 21/1 : 4[4] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:829:1605 [6] NCCL INFO Channel 20/1 : 6[6] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:825:1610 [2] NCCL INFO Channel 05/1 : 2[2] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:830:1604 [7] NCCL INFO Channel 25/1 : 7[7] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:828:1606 [5] NCCL INFO Channel 20/1 : 5[5] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:824:1609 [1] NCCL INFO Channel 06/1 : 1[1] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:826:1608 [3] NCCL INFO Channel 06/1 : 3[3] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:827:1603 [4] NCCL INFO Channel 22/1 : 4[4] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:829:1605 [6] NCCL INFO Channel 21/1 : 6[6] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:825:1610 [2] NCCL INFO Channel 06/1 : 2[2] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:830:1604 [7] NCCL INFO Channel 26/1 : 7[7] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:826:1608 [3] NCCL INFO Channel 07/1 : 3[3] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:824:1609 [1] NCCL INFO Channel 07/1 : 1[1] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:827:1603 [4] NCCL INFO Channel 23/1 : 4[4] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:828:1606 [5] NCCL INFO Channel 21/1 : 5[5] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:826:1608 [3] NCCL INFO Channel 08/1 : 3[3] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:830:1604 [7] NCCL INFO Channel 27/1 : 7[7] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:829:1605 [6] NCCL INFO Channel 22/1 : 6[6] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:825:1610 [2] NCCL INFO Channel 07/1 : 2[2] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:824:1609 [1] NCCL INFO Channel 08/1 : 1[1] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:827:1603 [4] NCCL INFO Channel 24/1 : 4[4] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:828:1606 [5] NCCL INFO Channel 22/1 : 5[5] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:826:1608 [3] NCCL INFO Channel 09/1 : 3[3] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:830:1604 [7] NCCL INFO Channel 28/1 : 7[7] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:829:1605 [6] NCCL INFO Channel 23/1 : 6[6] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:825:1610 [2] NCCL INFO Channel 08/1 : 2[2] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:824:1609 [1] NCCL INFO Channel 09/1 : 1[1] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:827:1603 [4] NCCL INFO Channel 25/1 : 4[4] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:826:1608 [3] NCCL INFO Channel 10/1 : 3[3] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:830:1604 [7] NCCL INFO Channel 29/1 : 7[7] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:828:1606 [5] NCCL INFO Channel 23/1 : 5[5] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:829:1605 [6] NCCL INFO Channel 24/1 : 6[6] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:824:1609 [1] NCCL INFO Channel 10/1 : 1[1] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:825:1610 [2] NCCL INFO Channel 09/1 : 2[2] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:826:1608 [3] NCCL INFO Channel 11/1 : 3[3] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:827:1603 [4] NCCL INFO Channel 26/1 : 4[4] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:830:1604 [7] NCCL INFO Channel 30/1 : 7[7] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:828:1606 [5] NCCL INFO Channel 24/1 : 5[5] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:829:1605 [6] NCCL INFO Channel 25/1 : 6[6] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:824:1609 [1] NCCL INFO Channel 11/1 : 1[1] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:825:1610 [2] NCCL INFO Channel 10/1 : 2[2] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:826:1608 [3] NCCL INFO Channel 12/1 : 3[3] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:828:1606 [5] NCCL INFO Channel 25/1 : 5[5] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:830:1604 [7] NCCL INFO Channel 31/1 : 7[7] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:829:1605 [6] NCCL INFO Channel 26/1 : 6[6] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:827:1603 [4] NCCL INFO Channel 27/1 : 4[4] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:826:1608 [3] NCCL INFO Channel 13/1 : 3[3] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:824:1609 [1] NCCL INFO Channel 12/1 : 1[1] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:825:1610 [2] NCCL INFO Channel 11/1 : 2[2] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:829:1605 [6] NCCL INFO Channel 27/1 : 6[6] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:828:1606 [5] NCCL INFO Channel 26/1 : 5[5] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:827:1603 [4] NCCL INFO Channel 28/1 : 4[4] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:826:1608 [3] NCCL INFO Channel 14/1 : 3[3] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:824:1609 [1] NCCL INFO Channel 13/1 : 1[1] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:825:1610 [2] NCCL INFO Channel 12/1 : 2[2] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:829:1605 [6] NCCL INFO Channel 28/1 : 6[6] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:828:1606 [5] NCCL INFO Channel 27/1 : 5[5] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:826:1608 [3] NCCL INFO Channel 15/1 : 3[3] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:827:1603 [4] NCCL INFO Channel 29/1 : 4[4] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:824:1609 [1] NCCL INFO Channel 14/1 : 1[1] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:825:1610 [2] NCCL INFO Channel 13/1 : 2[2] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:828:1606 [5] NCCL INFO Channel 28/1 : 5[5] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:827:1603 [4] NCCL INFO Channel 30/1 : 4[4] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:829:1605 [6] NCCL INFO Channel 29/1 : 6[6] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:826:1608 [3] NCCL INFO Channel 16/1 : 3[3] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:824:1609 [1] NCCL INFO Channel 15/1 : 1[1] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:825:1610 [2] NCCL INFO Channel 14/1 : 2[2] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:828:1606 [5] NCCL INFO Channel 29/1 : 5[5] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:827:1603 [4] NCCL INFO Channel 31/1 : 4[4] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:829:1605 [6] NCCL INFO Channel 30/1 : 6[6] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:826:1608 [3] NCCL INFO Channel 17/1 : 3[3] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:824:1609 [1] NCCL INFO Channel 16/1 : 1[1] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:825:1610 [2] NCCL INFO Channel 15/1 : 2[2] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:828:1606 [5] NCCL INFO Channel 30/1 : 5[5] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:826:1608 [3] NCCL INFO Channel 18/1 : 3[3] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:829:1605 [6] NCCL INFO Channel 31/1 : 6[6] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:824:1609 [1] NCCL INFO Channel 17/1 : 1[1] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:825:1610 [2] NCCL INFO Channel 16/1 : 2[2] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:828:1606 [5] NCCL INFO Channel 31/1 : 5[5] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:826:1608 [3] NCCL INFO Channel 19/1 : 3[3] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:824:1609 [1] NCCL INFO Channel 18/1 : 1[1] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:825:1610 [2] NCCL INFO Channel 17/1 : 2[2] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:826:1608 [3] NCCL INFO Channel 20/1 : 3[3] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:824:1609 [1] NCCL INFO Channel 19/1 : 1[1] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:825:1610 [2] NCCL INFO Channel 18/1 : 2[2] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:826:1608 [3] NCCL INFO Channel 21/1 : 3[3] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:824:1609 [1] NCCL INFO Channel 20/1 : 1[1] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:825:1610 [2] NCCL INFO Channel 19/1 : 2[2] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:826:1608 [3] NCCL INFO Channel 22/1 : 3[3] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:824:1609 [1] NCCL INFO Channel 21/1 : 1[1] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:825:1610 [2] NCCL INFO Channel 20/1 : 2[2] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:826:1608 [3] NCCL INFO Channel 23/1 : 3[3] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:824:1609 [1] NCCL INFO Channel 22/1 : 1[1] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:826:1608 [3] NCCL INFO Channel 24/1 : 3[3] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:825:1610 [2] NCCL INFO Channel 21/1 : 2[2] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:824:1609 [1] NCCL INFO Channel 23/1 : 1[1] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:826:1608 [3] NCCL INFO Channel 25/1 : 3[3] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:826:1608 [3] NCCL INFO Channel 26/1 : 3[3] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:824:1609 [1] NCCL INFO Channel 24/1 : 1[1] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:825:1610 [2] NCCL INFO Channel 22/1 : 2[2] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:826:1608 [3] NCCL INFO Channel 27/1 : 3[3] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:824:1609 [1] NCCL INFO Channel 25/1 : 1[1] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:825:1610 [2] NCCL INFO Channel 23/1 : 2[2] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:826:1608 [3] NCCL INFO Channel 28/1 : 3[3] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:825:1610 [2] NCCL INFO Channel 24/1 : 2[2] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:824:1609 [1] NCCL INFO Channel 26/1 : 1[1] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:826:1608 [3] NCCL INFO Channel 29/1 : 3[3] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:824:1609 [1] NCCL INFO Channel 27/1 : 1[1] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:825:1610 [2] NCCL INFO Channel 25/1 : 2[2] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:826:1608 [3] NCCL INFO Channel 30/1 : 3[3] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:824:1609 [1] NCCL INFO Channel 28/1 : 1[1] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:825:1610 [2] NCCL INFO Channel 26/1 : 2[2] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:826:1608 [3] NCCL INFO Channel 31/1 : 3[3] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:824:1609 [1] NCCL INFO Channel 29/1 : 1[1] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:825:1610 [2] NCCL INFO Channel 27/1 : 2[2] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:824:1609 [1] NCCL INFO Channel 30/1 : 1[1] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:825:1610 [2] NCCL INFO Channel 28/1 : 2[2] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:824:1609 [1] NCCL INFO Channel 31/1 : 1[1] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:825:1610 [2] NCCL INFO Channel 29/1 : 2[2] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:825:1610 [2] NCCL INFO Channel 30/1 : 2[2] -> 0[0] via P2P/IPC/read
tyler-rhel-newimage:825:1610 [2] NCCL INFO Channel 31/1 : 2[2] -> 0[0] via P2P/IPC/read
INFO 2024-07-27 18:38:05,545 instructlab.model.backends.backends:336: Waiting for the vLLM server to start at http://127.0.0.1:59587/v1, this might take a moment... Attempt: 9/80
INFO 2024-07-27 18:38:10,412 instructlab.model.backends.backends:336: Waiting for the vLLM server to start at http://127.0.0.1:59587/v1, this might take a moment... Attempt: 10/80
INFO 2024-07-27 18:38:15,137 instructlab.model.backends.backends:336: Waiting for the vLLM server to start at http://127.0.0.1:59587/v1, this might take a moment... Attempt: 11/80
INFO 2024-07-27 18:38:19,746 instructlab.model.backends.backends:336: Waiting for the vLLM server to start at http://127.0.0.1:59587/v1, this might take a moment... Attempt: 12/80
INFO 2024-07-27 18:38:24,535 instructlab.model.backends.backends:336: Waiting for the vLLM server to start at http://127.0.0.1:59587/v1, this might take a moment... Attempt: 13/80
INFO 2024-07-27 18:38:29,159 instructlab.model.backends.backends:336: Waiting for the vLLM server to start at http://127.0.0.1:59587/v1, this might take a moment... Attempt: 14/80
INFO 2024-07-27 18:38:33,998 instructlab.model.backends.backends:336: Waiting for the vLLM server to start at http://127.0.0.1:59587/v1, this might take a moment... Attempt: 15/80
INFO 2024-07-27 18:38:38,748 instructlab.model.backends.backends:336: Waiting for the vLLM server to start at http://127.0.0.1:59587/v1, this might take a moment... Attempt: 16/80
embedding_mode is False. Embedding API will not work.
INFO: Started server process [801]
INFO: Waiting for application startup.
INFO: Application startup complete.
INFO: Uvicorn running on http://127.0.0.1:59587 (Press CTRL+C to quit)
INFO 2024-07-27 18:38:43,526 instructlab.model.backends.backends:336: Waiting for the vLLM server to start at http://127.0.0.1:59587/v1, this might take a moment... Attempt: 17/80
INFO: 127.0.0.1:49680 - "GET /v1/models HTTP/1.1" 200 OK
INFO 2024-07-27 18:38:43,530 instructlab.model.backends.backends:343: vLLM engine successfully started at http://127.0.0.1:59587/v1
0%| | 0/160 [00:00<?, ?it/s]INFO: 127.0.0.1:49756 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:49684 - "POST /v1/chat/completions HTTP/1.1" 200 OK
1%|█▎ | 1/160 [00:05<13:19, 5.03s/it]INFO: 127.0.0.1:49808 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:49794 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:49698 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:49832 - "POST /v1/chat/completions HTTP/1.1" 200 OK
1%|██▌ | 2/160 [00:05<06:49, 2.59s/it]INFO: 127.0.0.1:49824 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:49762 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:49822 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:49778 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:49716 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2%|█████ | 4/160 [00:10<06:07, 2.36s/it]INFO: 127.0.0.1:49836 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:49684 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:49808 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:49716 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:49740 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:49756 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:49794 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:49832 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:49822 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:49778 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:49836 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:49740 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:49756 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:49824 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:49724 - "POST /v1/chat/completions HTTP/1.1" 200 OK
3%|██████▎ | 5/160 [00:19<11:20, 4.39s/it]INFO: 127.0.0.1:49684 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:49836 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:49832 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:49698 - "POST /v1/chat/completions HTTP/1.1" 200 OK
13%|██████████████████████████▎ | 21/160 [00:19<01:20, 1.73it/s]INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:49808 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:49724 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:49832 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:49778 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:49740 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:49716 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:49824 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:49808 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:49836 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:49698 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:49762 - "POST /v1/chat/completions HTTP/1.1" 200 OK
16%|███████████████████████████████▎ | 25/160 [00:23<01:27, 1.55it/s]INFO: 127.0.0.1:49684 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:49756 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:49740 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:49824 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:49822 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:49832 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:49836 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:49716 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:49756 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:49762 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:49724 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:49824 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:49794 - "POST /v1/chat/completions HTTP/1.1" 200 OK
22%|█████████████████████████████████████████████ | 36/160 [00:29<01:14, 1.67it/s]INFO: 127.0.0.1:49836 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:49740 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:49822 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:49684 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:49756 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:49716 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:49824 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK
26%|███████████████████████████████████████████████████▏ | 41/160 [00:31<01:06, 1.78it/s]INFO: 127.0.0.1:49808 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:49794 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:49832 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:49716 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:49824 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:49724 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:49836 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:49778 - "POST /v1/chat/completions HTTP/1.1" 200 OK
34%|████████████████████████████████████████████████████████████████████▊ | 55/160 [00:33<00:37, 2.79it/s]INFO: 127.0.0.1:49698 - "POST /v1/chat/completions HTTP/1.1" 200 OK
39%|█████████████████████████████████████████████████████████████████████████████▌ | 62/160 [00:33<00:26, 3.69it/s]INFO: 127.0.0.1:49756 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:49762 - "POST /v1/chat/completions HTTP/1.1" 200 OK
46%|████████████████████████████████████████████████████████████████████████████████████████████▌ | 74/160 [00:35<00:18, 4.68it/s]INFO: 127.0.0.1:49698 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:49808 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:49756 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:49832 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:49824 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:49724 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:49822 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:49684 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:49794 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:49740 - "POST /v1/chat/completions HTTP/1.1" 200 OK
49%|██████████████████████████████████████████████████████████████████████████████████████████████████▊ | 79/160 [00:39<00:27, 2.89it/s]INFO: 127.0.0.1:49698 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:49762 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:49778 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:49756 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:49824 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK
53%|██████████████████████████████████████████████████████████████████████████████████████████████████████████▎ | 85/160 [00:41<00:23, 3.24it/s]INFO: 127.0.0.1:49832 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:49794 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:49808 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:49822 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:49716 - "POST /v1/chat/completions HTTP/1.1" 200 OK
56%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████▌ | 90/160 [00:43<00:24, 2.82it/s]INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:49762 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:49698 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:49794 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:49756 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:49724 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:49684 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:49808 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:49822 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:49762 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:49778 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:49762 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:49684 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:49832 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:49808 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:49740 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:49836 - "POST /v1/chat/completions HTTP/1.1" 200 OK
58%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▎ | 93/160 [00:48<00:36, 1.83it/s]INFO: 127.0.0.1:49698 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:49684 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:49762 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:49824 - "POST /v1/chat/completions HTTP/1.1" 200 OK
72%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████ | 115/160 [00:50<00:12, 3.59it/s]INFO: 127.0.0.1:49836 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:49778 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:49740 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:49684 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:49698 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:49724 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:49794 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:49836 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:49716 - "POST /v1/chat/completions HTTP/1.1" 200 OK
76%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▋ | 122/160 [00:53<00:11, 3.18it/s]INFO: 127.0.0.1:49684 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:49698 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:49794 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:49740 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:49762 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:49822 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:49716 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:49824 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:49808 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:49698 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:49778 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:49794 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:49684 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:49756 - "POST /v1/chat/completions HTTP/1.1" 200 OK
79%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▉ | 127/160 [00:57<00:13, 2.52it/s]INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:49836 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:49724 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:49832 - "POST /v1/chat/completions HTTP/1.1" 200 OK
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 160/160 [01:11<00:00, 2.24it/s]
INFO: Shutting down
INFO: Waiting for application shutdown.
INFO: Application shutdown complete.
INFO: Finished server process [801]
[rank0]:[W CudaIPCTypes.cpp:16] Producer process has been terminated before all shared CUDA tensors released. See Note [Sharing CUDA tensors]
/usr/lib64/python3.11/multiprocessing/resource_tracker.py:254: UserWarning: resource_tracker: There appear to be 1 leaked shared_memory objects to clean up at shutdown
warnings.warn('resource_tracker: There appear to be %d '
# SKILL EVALUATION REPORT
## MODEL
/var/instructlabbigdisk/instructlab/skillscheckpoints/hf_format/samples_1056/
### AVERAGE:
4.16 (across 158)
### TURN ONE:
5.54
### TURN TWO:
2.77
### ERROR RATE:
0.01
[root@tyler-rhel-newimage instructlab]#
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment