Created
July 27, 2024 17:54
-
-
Save relyt0925/5182f8e3065d45d3a02249ed485d9e6a to your computer and use it in GitHub Desktop.
new mtbench_branch output
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| [root@tyler-rhel-newimage instructlab]# /root/ilab model evaluate --benchmark mt_bench_branch --model /var/instructlabbigdisk/instructlab/skillscheckpoints/hf_format/samples_1056/ --judge-model /var/instructlabbigdisk/instructlab/models/prometheus-eval/prometheus-8x7b-v2.0/ --base-model /var/instructlabbigdisk/instructlab/models/ibm-granite/granite-7b-base/ --output-dir /var/instructlabbigdisk/instructlab/evaltracker/skillscheckpoints/samples_1056/ --gpus 8 --backend vllm --enable-serving-output --taxonomy-path /var/instructlabbigdisk/instructlab/.local/share/instructlab/taxonomy/ --base-branch HEAD --branch HEAD | |
| INFO 2024-07-27 16:30:41,640 numexpr.utils:145: Note: detected 80 virtual cores but NumExpr set to maximum of 64, check "NUMEXPR_MAX_THREADS" environment variable. | |
| INFO 2024-07-27 16:30:41,641 numexpr.utils:148: Note: NumExpr detected 80 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 16. | |
| INFO 2024-07-27 16:30:41,641 numexpr.utils:161: NumExpr defaulting to 16 threads. | |
| Generating questions and reference answers from qna files for branch HEAD... | |
| WARNING 2024-07-27 16:30:41,791 instructlab.model.evaluate:288: Based on your hardware configuration, when using vLLM, we recommend setting max-workers between 53.0 and 160.0 for optimal performance | |
| INFO 2024-07-27 16:30:42,306 instructlab.model.backends.backends:315: Trying to connect to model server at http://127.0.0.1:8000/v1 | |
| INFO 2024-07-27 16:30:44,822 instructlab.model.backends.vllm:196: vLLM starting up on pid 84 at http://127.0.0.1:37393/v1 | |
| INFO 2024-07-27 16:30:44,822 instructlab.model.backends.backends:328: Starting a temporary vLLM server at http://127.0.0.1:37393/v1 | |
| INFO 2024-07-27 16:30:44,823 instructlab.model.backends.backends:336: Waiting for the vLLM server to start at http://127.0.0.1:37393/v1, this might take a moment... Attempt: 1/80 | |
| INFO 2024-07-27 16:30:49,639 instructlab.model.backends.backends:336: Waiting for the vLLM server to start at http://127.0.0.1:37393/v1, this might take a moment... Attempt: 2/80 | |
| Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained. | |
| INFO 2024-07-27 16:30:54,120 instructlab.model.backends.backends:336: Waiting for the vLLM server to start at http://127.0.0.1:37393/v1, this might take a moment... Attempt: 3/80 | |
| INFO 2024-07-27 16:30:58,913 instructlab.model.backends.backends:336: Waiting for the vLLM server to start at http://127.0.0.1:37393/v1, this might take a moment... Attempt: 4/80 | |
| INFO 2024-07-27 16:31:03,320 instructlab.model.backends.backends:336: Waiting for the vLLM server to start at http://127.0.0.1:37393/v1, this might take a moment... Attempt: 5/80 | |
| tyler-rhel-newimage:84:84 [0] NCCL INFO Bootstrap : Using enp8s0:192.168.48.11<0> | |
| tyler-rhel-newimage:84:84 [0] NCCL INFO NET/Plugin : dlerror=libnccl-net.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net.so), using internal implementation | |
| tyler-rhel-newimage:84:84 [0] NCCL INFO cudaDriverVersion 12040 | |
| NCCL version 2.20.5+cuda12.4 | |
| tyler-rhel-newimage:112:112 [6] NCCL INFO cudaDriverVersion 12040 | |
| tyler-rhel-newimage:112:112 [6] NCCL INFO Bootstrap : Using enp8s0:192.168.48.11<0> | |
| tyler-rhel-newimage:112:112 [6] NCCL INFO NET/Plugin : dlerror=libnccl-net.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net.so), using internal implementation | |
| tyler-rhel-newimage:107:107 [1] NCCL INFO cudaDriverVersion 12040 | |
| tyler-rhel-newimage:107:107 [1] NCCL INFO Bootstrap : Using enp8s0:192.168.48.11<0> | |
| tyler-rhel-newimage:107:107 [1] NCCL INFO NET/Plugin : dlerror=libnccl-net.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net.so), using internal implementation | |
| tyler-rhel-newimage:110:110 [4] NCCL INFO cudaDriverVersion 12040 | |
| tyler-rhel-newimage:110:110 [4] NCCL INFO Bootstrap : Using enp8s0:192.168.48.11<0> | |
| tyler-rhel-newimage:110:110 [4] NCCL INFO NET/Plugin : dlerror=libnccl-net.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net.so), using internal implementation | |
| tyler-rhel-newimage:113:113 [7] NCCL INFO cudaDriverVersion 12040 | |
| tyler-rhel-newimage:113:113 [7] NCCL INFO Bootstrap : Using enp8s0:192.168.48.11<0> | |
| tyler-rhel-newimage:113:113 [7] NCCL INFO NET/Plugin : dlerror=libnccl-net.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net.so), using internal implementation | |
| tyler-rhel-newimage:108:108 [2] NCCL INFO cudaDriverVersion 12040 | |
| tyler-rhel-newimage:108:108 [2] NCCL INFO Bootstrap : Using enp8s0:192.168.48.11<0> | |
| tyler-rhel-newimage:108:108 [2] NCCL INFO NET/Plugin : dlerror=libnccl-net.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net.so), using internal implementation | |
| tyler-rhel-newimage:111:111 [5] NCCL INFO cudaDriverVersion 12040 | |
| tyler-rhel-newimage:111:111 [5] NCCL INFO Bootstrap : Using enp8s0:192.168.48.11<0> | |
| tyler-rhel-newimage:111:111 [5] NCCL INFO NET/Plugin : dlerror=libnccl-net.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net.so), using internal implementation | |
| tyler-rhel-newimage:109:109 [3] NCCL INFO cudaDriverVersion 12040 | |
| tyler-rhel-newimage:109:109 [3] NCCL INFO Bootstrap : Using enp8s0:192.168.48.11<0> | |
| tyler-rhel-newimage:109:109 [3] NCCL INFO NET/Plugin : dlerror=libnccl-net.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net.so), using internal implementation | |
| tyler-rhel-newimage:112:112 [6] NCCL INFO NET/IB : No device found. | |
| tyler-rhel-newimage:112:112 [6] NCCL INFO NET/Socket : Using [0]enp8s0:192.168.48.11<0> | |
| tyler-rhel-newimage:112:112 [6] NCCL INFO Using non-device net plugin version 0 | |
| tyler-rhel-newimage:112:112 [6] NCCL INFO Using network Socket | |
| tyler-rhel-newimage:110:110 [4] NCCL INFO NET/IB : No device found. | |
| tyler-rhel-newimage:110:110 [4] NCCL INFO NET/Socket : Using [0]enp8s0:192.168.48.11<0> | |
| tyler-rhel-newimage:110:110 [4] NCCL INFO Using non-device net plugin version 0 | |
| tyler-rhel-newimage:110:110 [4] NCCL INFO Using network Socket | |
| tyler-rhel-newimage:111:111 [5] NCCL INFO NET/IB : No device found. | |
| tyler-rhel-newimage:111:111 [5] NCCL INFO NET/Socket : Using [0]enp8s0:192.168.48.11<0> | |
| tyler-rhel-newimage:111:111 [5] NCCL INFO Using non-device net plugin version 0 | |
| tyler-rhel-newimage:111:111 [5] NCCL INFO Using network Socket | |
| tyler-rhel-newimage:84:84 [0] NCCL INFO NET/IB : No device found. | |
| tyler-rhel-newimage:84:84 [0] NCCL INFO NET/Socket : Using [0]enp8s0:192.168.48.11<0> | |
| tyler-rhel-newimage:113:113 [7] NCCL INFO NET/IB : No device found. | |
| tyler-rhel-newimage:84:84 [0] NCCL INFO Using non-device net plugin version 0 | |
| tyler-rhel-newimage:84:84 [0] NCCL INFO Using network Socket | |
| tyler-rhel-newimage:113:113 [7] NCCL INFO NET/Socket : Using [0]enp8s0:192.168.48.11<0> | |
| tyler-rhel-newimage:113:113 [7] NCCL INFO Using non-device net plugin version 0 | |
| tyler-rhel-newimage:113:113 [7] NCCL INFO Using network Socket | |
| tyler-rhel-newimage:109:109 [3] NCCL INFO NET/IB : No device found. | |
| tyler-rhel-newimage:109:109 [3] NCCL INFO NET/Socket : Using [0]enp8s0:192.168.48.11<0> | |
| tyler-rhel-newimage:109:109 [3] NCCL INFO Using non-device net plugin version 0 | |
| tyler-rhel-newimage:109:109 [3] NCCL INFO Using network Socket | |
| tyler-rhel-newimage:108:108 [2] NCCL INFO NET/IB : No device found. | |
| tyler-rhel-newimage:108:108 [2] NCCL INFO NET/Socket : Using [0]enp8s0:192.168.48.11<0> | |
| tyler-rhel-newimage:108:108 [2] NCCL INFO Using non-device net plugin version 0 | |
| tyler-rhel-newimage:108:108 [2] NCCL INFO Using network Socket | |
| tyler-rhel-newimage:107:107 [1] NCCL INFO NET/IB : No device found. | |
| tyler-rhel-newimage:107:107 [1] NCCL INFO NET/Socket : Using [0]enp8s0:192.168.48.11<0> | |
| tyler-rhel-newimage:107:107 [1] NCCL INFO Using non-device net plugin version 0 | |
| tyler-rhel-newimage:107:107 [1] NCCL INFO Using network Socket | |
| tyler-rhel-newimage:111:111 [5] NCCL INFO comm 0x55cf2d2f6920 rank 5 nranks 8 cudaDev 5 nvmlDev 5 busId c060 commId 0x3a52f5b817e350c3 - Init START | |
| tyler-rhel-newimage:113:113 [7] NCCL INFO comm 0x55cf2d2f7410 rank 7 nranks 8 cudaDev 7 nvmlDev 7 busId e080 commId 0x3a52f5b817e350c3 - Init START | |
| tyler-rhel-newimage:107:107 [1] NCCL INFO comm 0x55cf2d2f5ca0 rank 1 nranks 8 cudaDev 1 nvmlDev 1 busId 8020 commId 0x3a52f5b817e350c3 - Init START | |
| tyler-rhel-newimage:109:109 [3] NCCL INFO comm 0x55cf2d2f6610 rank 3 nranks 8 cudaDev 3 nvmlDev 3 busId a040 commId 0x3a52f5b817e350c3 - Init START | |
| tyler-rhel-newimage:108:108 [2] NCCL INFO comm 0x55cf2d2f6410 rank 2 nranks 8 cudaDev 2 nvmlDev 2 busId a030 commId 0x3a52f5b817e350c3 - Init START | |
| tyler-rhel-newimage:84:84 [0] NCCL INFO comm 0x55cf2d2fe4b0 rank 0 nranks 8 cudaDev 0 nvmlDev 0 busId 8010 commId 0x3a52f5b817e350c3 - Init START | |
| tyler-rhel-newimage:110:110 [4] NCCL INFO comm 0x55cf2d2f6cc0 rank 4 nranks 8 cudaDev 4 nvmlDev 4 busId c050 commId 0x3a52f5b817e350c3 - Init START | |
| tyler-rhel-newimage:112:112 [6] NCCL INFO comm 0x55cf2d2f7630 rank 6 nranks 8 cudaDev 6 nvmlDev 6 busId e070 commId 0x3a52f5b817e350c3 - Init START | |
| tyler-rhel-newimage:110:110 [4] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| tyler-rhel-newimage:112:112 [6] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| tyler-rhel-newimage:107:107 [1] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| tyler-rhel-newimage:108:108 [2] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| tyler-rhel-newimage:109:109 [3] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| tyler-rhel-newimage:84:84 [0] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| tyler-rhel-newimage:111:111 [5] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| tyler-rhel-newimage:110:110 [4] NCCL INFO Setting affinity for GPU 4 to ffff,ffffff00,00000000 | |
| tyler-rhel-newimage:110:110 [4] NCCL INFO NVLS multicast support is not available on dev 4 | |
| tyler-rhel-newimage:113:113 [7] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| tyler-rhel-newimage:108:108 [2] NCCL INFO Setting affinity for GPU 2 to ff,ffffffff | |
| tyler-rhel-newimage:107:107 [1] NCCL INFO Setting affinity for GPU 1 to ff,ffffffff | |
| tyler-rhel-newimage:108:108 [2] NCCL INFO NVLS multicast support is not available on dev 2 | |
| tyler-rhel-newimage:107:107 [1] NCCL INFO NVLS multicast support is not available on dev 1 | |
| tyler-rhel-newimage:84:84 [0] NCCL INFO Setting affinity for GPU 0 to ff,ffffffff | |
| tyler-rhel-newimage:84:84 [0] NCCL INFO NVLS multicast support is not available on dev 0 | |
| tyler-rhel-newimage:109:109 [3] NCCL INFO Setting affinity for GPU 3 to ff,ffffffff | |
| tyler-rhel-newimage:112:112 [6] NCCL INFO Setting affinity for GPU 6 to ffff,ffffff00,00000000 | |
| tyler-rhel-newimage:109:109 [3] NCCL INFO NVLS multicast support is not available on dev 3 | |
| tyler-rhel-newimage:112:112 [6] NCCL INFO NVLS multicast support is not available on dev 6 | |
| tyler-rhel-newimage:111:111 [5] NCCL INFO Setting affinity for GPU 5 to ffff,ffffff00,00000000 | |
| tyler-rhel-newimage:111:111 [5] NCCL INFO NVLS multicast support is not available on dev 5 | |
| tyler-rhel-newimage:113:113 [7] NCCL INFO Setting affinity for GPU 7 to ffff,ffffff00,00000000 | |
| tyler-rhel-newimage:113:113 [7] NCCL INFO NVLS multicast support is not available on dev 7 | |
| tyler-rhel-newimage:107:107 [1] NCCL INFO comm 0x55cf2d2f5ca0 rank 1 nRanks 8 nNodes 1 localRanks 8 localRank 1 MNNVL 0 | |
| tyler-rhel-newimage:108:108 [2] NCCL INFO comm 0x55cf2d2f6410 rank 2 nRanks 8 nNodes 1 localRanks 8 localRank 2 MNNVL 0 | |
| tyler-rhel-newimage:109:109 [3] NCCL INFO comm 0x55cf2d2f6610 rank 3 nRanks 8 nNodes 1 localRanks 8 localRank 3 MNNVL 0 | |
| tyler-rhel-newimage:111:111 [5] NCCL INFO comm 0x55cf2d2f6920 rank 5 nRanks 8 nNodes 1 localRanks 8 localRank 5 MNNVL 0 | |
| tyler-rhel-newimage:110:110 [4] NCCL INFO comm 0x55cf2d2f6cc0 rank 4 nRanks 8 nNodes 1 localRanks 8 localRank 4 MNNVL 0 | |
| tyler-rhel-newimage:112:112 [6] NCCL INFO comm 0x55cf2d2f7630 rank 6 nRanks 8 nNodes 1 localRanks 8 localRank 6 MNNVL 0 | |
| tyler-rhel-newimage:113:113 [7] NCCL INFO comm 0x55cf2d2f7410 rank 7 nRanks 8 nNodes 1 localRanks 8 localRank 7 MNNVL 0 | |
| tyler-rhel-newimage:107:107 [1] NCCL INFO Trees [0] 2/-1/-1->1->0 [1] 2/-1/-1->1->0 [2] 2/-1/-1->1->0 [3] 2/-1/-1->1->0 [4] 2/-1/-1->1->0 [5] 2/-1/-1->1->0 [6] 2/-1/-1->1->0 [7] 2/-1/-1->1->0 [8] 2/-1/-1->1->0 [9] 2/-1/-1->1->0 [10] 2/-1/-1->1->0 [11] 2/-1/-1->1->0 [12] 2/-1/-1->1->0 [13] 2/-1/-1->1->0 [14] 2/-1/-1->1->0 [15] 2/-1/-1->1->0 [16] 2/-1/-1->1->0 [17] 2/-1/-1->1->0 [18] 2/-1/-1->1->0 [19] 2/-1/-1->1->0 [20] 2/-1/-1->1->0 [21] 2/-1/-1->1->0 [22] 2/-1/-1->1->0 [23] 2/-1/-1->1->0 | |
| tyler-rhel-newimage:107:107 [1] NCCL INFO P2P Chunksize set to 524288 | |
| tyler-rhel-newimage:108:108 [2] NCCL INFO Trees [0] 3/-1/-1->2->1 [1] 3/-1/-1->2->1 [2] 3/-1/-1->2->1 [3] 3/-1/-1->2->1 [4] 3/-1/-1->2->1 [5] 3/-1/-1->2->1 [6] 3/-1/-1->2->1 [7] 3/-1/-1->2->1 [8] 3/-1/-1->2->1 [9] 3/-1/-1->2->1 [10] 3/-1/-1->2->1 [11] 3/-1/-1->2->1 [12] 3/-1/-1->2->1 [13] 3/-1/-1->2->1 [14] 3/-1/-1->2->1 [15] 3/-1/-1->2->1 [16] 3/-1/-1->2->1 [17] 3/-1/-1->2->1 [18] 3/-1/-1->2->1 [19] 3/-1/-1->2->1 [20] 3/-1/-1->2->1 [21] 3/-1/-1->2->1 [22] 3/-1/-1->2->1 [23] 3/-1/-1->2->1 | |
| tyler-rhel-newimage:109:109 [3] NCCL INFO Trees [0] 4/-1/-1->3->2 [1] 4/-1/-1->3->2 [2] 4/-1/-1->3->2 [3] 4/-1/-1->3->2 [4] 4/-1/-1->3->2 [5] 4/-1/-1->3->2 [6] 4/-1/-1->3->2 [7] 4/-1/-1->3->2 [8] 4/-1/-1->3->2 [9] 4/-1/-1->3->2 [10] 4/-1/-1->3->2 [11] 4/-1/-1->3->2 [12] 4/-1/-1->3->2 [13] 4/-1/-1->3->2 [14] 4/-1/-1->3->2 [15] 4/-1/-1->3->2 [16] 4/-1/-1->3->2 [17] 4/-1/-1->3->2 [18] 4/-1/-1->3->2 [19] 4/-1/-1->3->2 [20] 4/-1/-1->3->2 [21] 4/-1/-1->3->2 [22] 4/-1/-1->3->2 [23] 4/-1/-1->3->2 | |
| tyler-rhel-newimage:108:108 [2] NCCL INFO P2P Chunksize set to 524288 | |
| tyler-rhel-newimage:84:84 [0] NCCL INFO comm 0x55cf2d2fe4b0 rank 0 nRanks 8 nNodes 1 localRanks 8 localRank 0 MNNVL 0 | |
| tyler-rhel-newimage:109:109 [3] NCCL INFO P2P Chunksize set to 524288 | |
| tyler-rhel-newimage:113:113 [7] NCCL INFO Trees [0] -1/-1/-1->7->6 [1] -1/-1/-1->7->6 [2] -1/-1/-1->7->6 [3] -1/-1/-1->7->6 [4] -1/-1/-1->7->6 [5] -1/-1/-1->7->6 [6] -1/-1/-1->7->6 [7] -1/-1/-1->7->6 [8] -1/-1/-1->7->6 [9] -1/-1/-1->7->6 [10] -1/-1/-1->7->6 [11] -1/-1/-1->7->6 [12] -1/-1/-1->7->6 [13] -1/-1/-1->7->6 [14] -1/-1/-1->7->6 [15] -1/-1/-1->7->6 [16] -1/-1/-1->7->6 [17] -1/-1/-1->7->6 [18] -1/-1/-1->7->6 [19] -1/-1/-1->7->6 [20] -1/-1/-1->7->6 [21] -1/-1/-1->7->6 [22] -1/-1/-1->7->6 [23] -1/-1/-1->7->6 | |
| tyler-rhel-newimage:112:112 [6] NCCL INFO Trees [0] 7/-1/-1->6->5 [1] 7/-1/-1->6->5 [2] 7/-1/-1->6->5 [3] 7/-1/-1->6->5 [4] 7/-1/-1->6->5 [5] 7/-1/-1->6->5 [6] 7/-1/-1->6->5 [7] 7/-1/-1->6->5 [8] 7/-1/-1->6->5 [9] 7/-1/-1->6->5 [10] 7/-1/-1->6->5 [11] 7/-1/-1->6->5 [12] 7/-1/-1->6->5 [13] 7/-1/-1->6->5 [14] 7/-1/-1->6->5 [15] 7/-1/-1->6->5 [16] 7/-1/-1->6->5 [17] 7/-1/-1->6->5 [18] 7/-1/-1->6->5 [19] 7/-1/-1->6->5 [20] 7/-1/-1->6->5 [21] 7/-1/-1->6->5 [22] 7/-1/-1->6->5 [23] 7/-1/-1->6->5 | |
| tyler-rhel-newimage:111:111 [5] NCCL INFO Trees [0] 6/-1/-1->5->4 [1] 6/-1/-1->5->4 [2] 6/-1/-1->5->4 [3] 6/-1/-1->5->4 [4] 6/-1/-1->5->4 [5] 6/-1/-1->5->4 [6] 6/-1/-1->5->4 [7] 6/-1/-1->5->4 [8] 6/-1/-1->5->4 [9] 6/-1/-1->5->4 [10] 6/-1/-1->5->4 [11] 6/-1/-1->5->4 [12] 6/-1/-1->5->4 [13] 6/-1/-1->5->4 [14] 6/-1/-1->5->4 [15] 6/-1/-1->5->4 [16] 6/-1/-1->5->4 [17] 6/-1/-1->5->4 [18] 6/-1/-1->5->4 [19] 6/-1/-1->5->4 [20] 6/-1/-1->5->4 [21] 6/-1/-1->5->4 [22] 6/-1/-1->5->4 [23] 6/-1/-1->5->4 | |
| tyler-rhel-newimage:113:113 [7] NCCL INFO P2P Chunksize set to 524288 | |
| tyler-rhel-newimage:110:110 [4] NCCL INFO Trees [0] 5/-1/-1->4->3 [1] 5/-1/-1->4->3 [2] 5/-1/-1->4->3 [3] 5/-1/-1->4->3 [4] 5/-1/-1->4->3 [5] 5/-1/-1->4->3 [6] 5/-1/-1->4->3 [7] 5/-1/-1->4->3 [8] 5/-1/-1->4->3 [9] 5/-1/-1->4->3 [10] 5/-1/-1->4->3 [11] 5/-1/-1->4->3 [12] 5/-1/-1->4->3 [13] 5/-1/-1->4->3 [14] 5/-1/-1->4->3 [15] 5/-1/-1->4->3 [16] 5/-1/-1->4->3 [17] 5/-1/-1->4->3 [18] 5/-1/-1->4->3 [19] 5/-1/-1->4->3 [20] 5/-1/-1->4->3 [21] 5/-1/-1->4->3 [22] 5/-1/-1->4->3 [23] 5/-1/-1->4->3 | |
| tyler-rhel-newimage:112:112 [6] NCCL INFO P2P Chunksize set to 524288 | |
| tyler-rhel-newimage:111:111 [5] NCCL INFO P2P Chunksize set to 524288 | |
| tyler-rhel-newimage:84:84 [0] NCCL INFO Channel 00/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:110:110 [4] NCCL INFO P2P Chunksize set to 524288 | |
| tyler-rhel-newimage:84:84 [0] NCCL INFO Channel 01/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:84:84 [0] NCCL INFO Channel 02/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:84:84 [0] NCCL INFO Channel 03/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:84:84 [0] NCCL INFO Channel 04/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:84:84 [0] NCCL INFO Channel 05/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:84:84 [0] NCCL INFO Channel 06/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:84:84 [0] NCCL INFO Channel 07/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:84:84 [0] NCCL INFO Channel 08/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:84:84 [0] NCCL INFO Channel 09/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:84:84 [0] NCCL INFO Channel 10/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:84:84 [0] NCCL INFO Channel 11/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:84:84 [0] NCCL INFO Channel 12/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:84:84 [0] NCCL INFO Channel 13/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:84:84 [0] NCCL INFO Channel 14/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:84:84 [0] NCCL INFO Channel 15/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:84:84 [0] NCCL INFO Channel 16/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:84:84 [0] NCCL INFO Channel 17/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:84:84 [0] NCCL INFO Channel 18/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:84:84 [0] NCCL INFO Channel 19/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:84:84 [0] NCCL INFO Channel 20/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:84:84 [0] NCCL INFO Channel 21/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:84:84 [0] NCCL INFO Channel 22/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:84:84 [0] NCCL INFO Channel 23/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:84:84 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1 [2] 1/-1/-1->0->-1 [3] 1/-1/-1->0->-1 [4] 1/-1/-1->0->-1 [5] 1/-1/-1->0->-1 [6] 1/-1/-1->0->-1 [7] 1/-1/-1->0->-1 [8] 1/-1/-1->0->-1 [9] 1/-1/-1->0->-1 [10] 1/-1/-1->0->-1 [11] 1/-1/-1->0->-1 [12] 1/-1/-1->0->-1 [13] 1/-1/-1->0->-1 [14] 1/-1/-1->0->-1 [15] 1/-1/-1->0->-1 [16] 1/-1/-1->0->-1 [17] 1/-1/-1->0->-1 [18] 1/-1/-1->0->-1 [19] 1/-1/-1->0->-1 [20] 1/-1/-1->0->-1 [21] 1/-1/-1->0->-1 [22] 1/-1/-1->0->-1 [23] 1/-1/-1->0->-1 | |
| tyler-rhel-newimage:84:84 [0] NCCL INFO P2P Chunksize set to 524288 | |
| tyler-rhel-newimage:112:112 [6] NCCL INFO Channel 00/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:111:111 [5] NCCL INFO Channel 00/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:109:109 [3] NCCL INFO Channel 00/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:113:113 [7] NCCL INFO Channel 00/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:112:112 [6] NCCL INFO Channel 01/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:111:111 [5] NCCL INFO Channel 01/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:109:109 [3] NCCL INFO Channel 01/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:108:108 [2] NCCL INFO Channel 00/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:113:113 [7] NCCL INFO Channel 01/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:107:107 [1] NCCL INFO Channel 00/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:112:112 [6] NCCL INFO Channel 02/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:111:111 [5] NCCL INFO Channel 02/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:109:109 [3] NCCL INFO Channel 02/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:108:108 [2] NCCL INFO Channel 01/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:113:113 [7] NCCL INFO Channel 02/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:84:84 [0] NCCL INFO Channel 00/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:107:107 [1] NCCL INFO Channel 01/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:112:112 [6] NCCL INFO Channel 03/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:111:111 [5] NCCL INFO Channel 03/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:109:109 [3] NCCL INFO Channel 03/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:108:108 [2] NCCL INFO Channel 02/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:113:113 [7] NCCL INFO Channel 03/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:84:84 [0] NCCL INFO Channel 01/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:107:107 [1] NCCL INFO Channel 02/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:112:112 [6] NCCL INFO Channel 04/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:111:111 [5] NCCL INFO Channel 04/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:109:109 [3] NCCL INFO Channel 04/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:108:108 [2] NCCL INFO Channel 03/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:113:113 [7] NCCL INFO Channel 04/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:84:84 [0] NCCL INFO Channel 02/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:107:107 [1] NCCL INFO Channel 03/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:112:112 [6] NCCL INFO Channel 05/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:111:111 [5] NCCL INFO Channel 05/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:108:108 [2] NCCL INFO Channel 04/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:109:109 [3] NCCL INFO Channel 05/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:113:113 [7] NCCL INFO Channel 05/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:84:84 [0] NCCL INFO Channel 03/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:107:107 [1] NCCL INFO Channel 04/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:112:112 [6] NCCL INFO Channel 06/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:111:111 [5] NCCL INFO Channel 06/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:109:109 [3] NCCL INFO Channel 06/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:108:108 [2] NCCL INFO Channel 05/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:113:113 [7] NCCL INFO Channel 06/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:84:84 [0] NCCL INFO Channel 04/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:110:110 [4] NCCL INFO Channel 00/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:112:112 [6] NCCL INFO Channel 07/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:109:109 [3] NCCL INFO Channel 07/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:107:107 [1] NCCL INFO Channel 05/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:111:111 [5] NCCL INFO Channel 07/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:108:108 [2] NCCL INFO Channel 06/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:113:113 [7] NCCL INFO Channel 07/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:84:84 [0] NCCL INFO Channel 05/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:110:110 [4] NCCL INFO Channel 01/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:112:112 [6] NCCL INFO Channel 08/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:107:107 [1] NCCL INFO Channel 06/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:109:109 [3] NCCL INFO Channel 08/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:111:111 [5] NCCL INFO Channel 08/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:108:108 [2] NCCL INFO Channel 07/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:113:113 [7] NCCL INFO Channel 08/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:84:84 [0] NCCL INFO Channel 06/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:110:110 [4] NCCL INFO Channel 02/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:112:112 [6] NCCL INFO Channel 09/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:107:107 [1] NCCL INFO Channel 07/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:109:109 [3] NCCL INFO Channel 09/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:111:111 [5] NCCL INFO Channel 09/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:108:108 [2] NCCL INFO Channel 08/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:113:113 [7] NCCL INFO Channel 09/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:84:84 [0] NCCL INFO Channel 07/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:110:110 [4] NCCL INFO Channel 03/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:112:112 [6] NCCL INFO Channel 10/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:107:107 [1] NCCL INFO Channel 08/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:109:109 [3] NCCL INFO Channel 10/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:111:111 [5] NCCL INFO Channel 10/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:108:108 [2] NCCL INFO Channel 09/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:113:113 [7] NCCL INFO Channel 10/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:84:84 [0] NCCL INFO Channel 08/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:110:110 [4] NCCL INFO Channel 04/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:112:112 [6] NCCL INFO Channel 11/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:107:107 [1] NCCL INFO Channel 09/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:109:109 [3] NCCL INFO Channel 11/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:111:111 [5] NCCL INFO Channel 11/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:108:108 [2] NCCL INFO Channel 10/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:113:113 [7] NCCL INFO Channel 11/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:84:84 [0] NCCL INFO Channel 09/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:112:112 [6] NCCL INFO Channel 12/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:110:110 [4] NCCL INFO Channel 05/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:107:107 [1] NCCL INFO Channel 10/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:109:109 [3] NCCL INFO Channel 12/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:111:111 [5] NCCL INFO Channel 12/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:113:113 [7] NCCL INFO Channel 12/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:108:108 [2] NCCL INFO Channel 11/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:84:84 [0] NCCL INFO Channel 10/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:110:110 [4] NCCL INFO Channel 06/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:112:112 [6] NCCL INFO Channel 13/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:107:107 [1] NCCL INFO Channel 11/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:109:109 [3] NCCL INFO Channel 13/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:111:111 [5] NCCL INFO Channel 13/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:113:113 [7] NCCL INFO Channel 13/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:108:108 [2] NCCL INFO Channel 12/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:84:84 [0] NCCL INFO Channel 11/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:110:110 [4] NCCL INFO Channel 07/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:112:112 [6] NCCL INFO Channel 14/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:109:109 [3] NCCL INFO Channel 14/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:107:107 [1] NCCL INFO Channel 12/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:111:111 [5] NCCL INFO Channel 14/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:113:113 [7] NCCL INFO Channel 14/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:108:108 [2] NCCL INFO Channel 13/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:84:84 [0] NCCL INFO Channel 12/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:110:110 [4] NCCL INFO Channel 08/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:112:112 [6] NCCL INFO Channel 15/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:109:109 [3] NCCL INFO Channel 15/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:107:107 [1] NCCL INFO Channel 13/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:111:111 [5] NCCL INFO Channel 15/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:113:113 [7] NCCL INFO Channel 15/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:84:84 [0] NCCL INFO Channel 13/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:108:108 [2] NCCL INFO Channel 14/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:110:110 [4] NCCL INFO Channel 09/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:112:112 [6] NCCL INFO Channel 16/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:109:109 [3] NCCL INFO Channel 16/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:107:107 [1] NCCL INFO Channel 14/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:111:111 [5] NCCL INFO Channel 16/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:113:113 [7] NCCL INFO Channel 16/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:84:84 [0] NCCL INFO Channel 14/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:110:110 [4] NCCL INFO Channel 10/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:108:108 [2] NCCL INFO Channel 15/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:112:112 [6] NCCL INFO Channel 17/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:109:109 [3] NCCL INFO Channel 17/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:107:107 [1] NCCL INFO Channel 15/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:111:111 [5] NCCL INFO Channel 17/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:113:113 [7] NCCL INFO Channel 17/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:84:84 [0] NCCL INFO Channel 15/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:110:110 [4] NCCL INFO Channel 11/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:108:108 [2] NCCL INFO Channel 16/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:112:112 [6] NCCL INFO Channel 18/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:109:109 [3] NCCL INFO Channel 18/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:107:107 [1] NCCL INFO Channel 16/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:111:111 [5] NCCL INFO Channel 18/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:113:113 [7] NCCL INFO Channel 18/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:84:84 [0] NCCL INFO Channel 16/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:110:110 [4] NCCL INFO Channel 12/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:108:108 [2] NCCL INFO Channel 17/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:112:112 [6] NCCL INFO Channel 19/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:109:109 [3] NCCL INFO Channel 19/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:107:107 [1] NCCL INFO Channel 17/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:111:111 [5] NCCL INFO Channel 19/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:113:113 [7] NCCL INFO Channel 19/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:84:84 [0] NCCL INFO Channel 17/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:110:110 [4] NCCL INFO Channel 13/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:112:112 [6] NCCL INFO Channel 20/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:108:108 [2] NCCL INFO Channel 18/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:109:109 [3] NCCL INFO Channel 20/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:107:107 [1] NCCL INFO Channel 18/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:111:111 [5] NCCL INFO Channel 20/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:113:113 [7] NCCL INFO Channel 20/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:110:110 [4] NCCL INFO Channel 14/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:84:84 [0] NCCL INFO Channel 18/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:112:112 [6] NCCL INFO Channel 21/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:108:108 [2] NCCL INFO Channel 19/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:109:109 [3] NCCL INFO Channel 21/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:107:107 [1] NCCL INFO Channel 19/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:111:111 [5] NCCL INFO Channel 21/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:113:113 [7] NCCL INFO Channel 21/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:110:110 [4] NCCL INFO Channel 15/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:84:84 [0] NCCL INFO Channel 19/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:112:112 [6] NCCL INFO Channel 22/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:109:109 [3] NCCL INFO Channel 22/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:108:108 [2] NCCL INFO Channel 20/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:107:107 [1] NCCL INFO Channel 20/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:111:111 [5] NCCL INFO Channel 22/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:113:113 [7] NCCL INFO Channel 22/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:110:110 [4] NCCL INFO Channel 16/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:84:84 [0] NCCL INFO Channel 20/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:112:112 [6] NCCL INFO Channel 23/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:109:109 [3] NCCL INFO Channel 23/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:108:108 [2] NCCL INFO Channel 21/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:107:107 [1] NCCL INFO Channel 21/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:111:111 [5] NCCL INFO Channel 23/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:113:113 [7] NCCL INFO Channel 23/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:110:110 [4] NCCL INFO Channel 17/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:84:84 [0] NCCL INFO Channel 21/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:108:108 [2] NCCL INFO Channel 22/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:107:107 [1] NCCL INFO Channel 22/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:110:110 [4] NCCL INFO Channel 18/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:84:84 [0] NCCL INFO Channel 22/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:107:107 [1] NCCL INFO Channel 23/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:108:108 [2] NCCL INFO Channel 23/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:110:110 [4] NCCL INFO Channel 19/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:84:84 [0] NCCL INFO Channel 23/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:110:110 [4] NCCL INFO Channel 20/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:110:110 [4] NCCL INFO Channel 21/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:110:110 [4] NCCL INFO Channel 22/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:110:110 [4] NCCL INFO Channel 23/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:107:107 [1] NCCL INFO Connected all rings | |
| tyler-rhel-newimage:113:113 [7] NCCL INFO Connected all rings | |
| tyler-rhel-newimage:113:113 [7] NCCL INFO Channel 00/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:84:84 [0] NCCL INFO Connected all rings | |
| tyler-rhel-newimage:113:113 [7] NCCL INFO Channel 01/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:113:113 [7] NCCL INFO Channel 02/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:108:108 [2] NCCL INFO Connected all rings | |
| tyler-rhel-newimage:112:112 [6] NCCL INFO Connected all rings | |
| tyler-rhel-newimage:113:113 [7] NCCL INFO Channel 03/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:111:111 [5] NCCL INFO Connected all rings | |
| tyler-rhel-newimage:109:109 [3] NCCL INFO Connected all rings | |
| tyler-rhel-newimage:110:110 [4] NCCL INFO Connected all rings | |
| tyler-rhel-newimage:113:113 [7] NCCL INFO Channel 04/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:113:113 [7] NCCL INFO Channel 05/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:113:113 [7] NCCL INFO Channel 06/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:113:113 [7] NCCL INFO Channel 07/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:113:113 [7] NCCL INFO Channel 08/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:113:113 [7] NCCL INFO Channel 09/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:113:113 [7] NCCL INFO Channel 10/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:113:113 [7] NCCL INFO Channel 11/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:113:113 [7] NCCL INFO Channel 12/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:113:113 [7] NCCL INFO Channel 13/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:113:113 [7] NCCL INFO Channel 14/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:113:113 [7] NCCL INFO Channel 15/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:113:113 [7] NCCL INFO Channel 16/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:113:113 [7] NCCL INFO Channel 17/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:113:113 [7] NCCL INFO Channel 18/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:113:113 [7] NCCL INFO Channel 19/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:113:113 [7] NCCL INFO Channel 20/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:113:113 [7] NCCL INFO Channel 21/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:113:113 [7] NCCL INFO Channel 22/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:113:113 [7] NCCL INFO Channel 23/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:107:107 [1] NCCL INFO Channel 00/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:107:107 [1] NCCL INFO Channel 01/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:109:109 [3] NCCL INFO Channel 00/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:111:111 [5] NCCL INFO Channel 00/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:107:107 [1] NCCL INFO Channel 02/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:109:109 [3] NCCL INFO Channel 01/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:111:111 [5] NCCL INFO Channel 01/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:112:112 [6] NCCL INFO Channel 00/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:107:107 [1] NCCL INFO Channel 03/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:109:109 [3] NCCL INFO Channel 02/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:111:111 [5] NCCL INFO Channel 02/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:112:112 [6] NCCL INFO Channel 01/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:107:107 [1] NCCL INFO Channel 04/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:109:109 [3] NCCL INFO Channel 03/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:111:111 [5] NCCL INFO Channel 03/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:112:112 [6] NCCL INFO Channel 02/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:109:109 [3] NCCL INFO Channel 04/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:107:107 [1] NCCL INFO Channel 05/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:111:111 [5] NCCL INFO Channel 04/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:112:112 [6] NCCL INFO Channel 03/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:107:107 [1] NCCL INFO Channel 06/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:109:109 [3] NCCL INFO Channel 05/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:111:111 [5] NCCL INFO Channel 05/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:112:112 [6] NCCL INFO Channel 04/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:107:107 [1] NCCL INFO Channel 07/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:109:109 [3] NCCL INFO Channel 06/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:111:111 [5] NCCL INFO Channel 06/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:112:112 [6] NCCL INFO Channel 05/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:107:107 [1] NCCL INFO Channel 08/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:109:109 [3] NCCL INFO Channel 07/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:111:111 [5] NCCL INFO Channel 07/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:112:112 [6] NCCL INFO Channel 06/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:107:107 [1] NCCL INFO Channel 09/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:109:109 [3] NCCL INFO Channel 08/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:111:111 [5] NCCL INFO Channel 08/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:112:112 [6] NCCL INFO Channel 07/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:107:107 [1] NCCL INFO Channel 10/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:109:109 [3] NCCL INFO Channel 09/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:108:108 [2] NCCL INFO Channel 00/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:110:110 [4] NCCL INFO Channel 00/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:111:111 [5] NCCL INFO Channel 09/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:112:112 [6] NCCL INFO Channel 08/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:107:107 [1] NCCL INFO Channel 11/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:109:109 [3] NCCL INFO Channel 10/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:108:108 [2] NCCL INFO Channel 01/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:110:110 [4] NCCL INFO Channel 01/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:111:111 [5] NCCL INFO Channel 10/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:109:109 [3] NCCL INFO Channel 11/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:112:112 [6] NCCL INFO Channel 09/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:107:107 [1] NCCL INFO Channel 12/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:108:108 [2] NCCL INFO Channel 02/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:110:110 [4] NCCL INFO Channel 02/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:111:111 [5] NCCL INFO Channel 11/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:109:109 [3] NCCL INFO Channel 12/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:112:112 [6] NCCL INFO Channel 10/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:107:107 [1] NCCL INFO Channel 13/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:108:108 [2] NCCL INFO Channel 03/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:110:110 [4] NCCL INFO Channel 03/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:111:111 [5] NCCL INFO Channel 12/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:109:109 [3] NCCL INFO Channel 13/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:112:112 [6] NCCL INFO Channel 11/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:107:107 [1] NCCL INFO Channel 14/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:108:108 [2] NCCL INFO Channel 04/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:110:110 [4] NCCL INFO Channel 04/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:111:111 [5] NCCL INFO Channel 13/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:109:109 [3] NCCL INFO Channel 14/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:112:112 [6] NCCL INFO Channel 12/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:107:107 [1] NCCL INFO Channel 15/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:108:108 [2] NCCL INFO Channel 05/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:110:110 [4] NCCL INFO Channel 05/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:111:111 [5] NCCL INFO Channel 14/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:109:109 [3] NCCL INFO Channel 15/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:112:112 [6] NCCL INFO Channel 13/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:107:107 [1] NCCL INFO Channel 16/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:108:108 [2] NCCL INFO Channel 06/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:110:110 [4] NCCL INFO Channel 06/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:111:111 [5] NCCL INFO Channel 15/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:109:109 [3] NCCL INFO Channel 16/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:112:112 [6] NCCL INFO Channel 14/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:107:107 [1] NCCL INFO Channel 17/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:108:108 [2] NCCL INFO Channel 07/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:110:110 [4] NCCL INFO Channel 07/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:111:111 [5] NCCL INFO Channel 16/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:109:109 [3] NCCL INFO Channel 17/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:112:112 [6] NCCL INFO Channel 15/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:107:107 [1] NCCL INFO Channel 18/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:108:108 [2] NCCL INFO Channel 08/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:110:110 [4] NCCL INFO Channel 08/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:111:111 [5] NCCL INFO Channel 17/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:109:109 [3] NCCL INFO Channel 18/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:112:112 [6] NCCL INFO Channel 16/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:107:107 [1] NCCL INFO Channel 19/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:108:108 [2] NCCL INFO Channel 09/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:110:110 [4] NCCL INFO Channel 09/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:111:111 [5] NCCL INFO Channel 18/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:109:109 [3] NCCL INFO Channel 19/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:107:107 [1] NCCL INFO Channel 20/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:112:112 [6] NCCL INFO Channel 17/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:108:108 [2] NCCL INFO Channel 10/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:110:110 [4] NCCL INFO Channel 10/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:111:111 [5] NCCL INFO Channel 19/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:109:109 [3] NCCL INFO Channel 20/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:107:107 [1] NCCL INFO Channel 21/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:112:112 [6] NCCL INFO Channel 18/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:108:108 [2] NCCL INFO Channel 11/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:110:110 [4] NCCL INFO Channel 11/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:111:111 [5] NCCL INFO Channel 20/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:109:109 [3] NCCL INFO Channel 21/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:107:107 [1] NCCL INFO Channel 22/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:112:112 [6] NCCL INFO Channel 19/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:108:108 [2] NCCL INFO Channel 12/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:110:110 [4] NCCL INFO Channel 12/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:111:111 [5] NCCL INFO Channel 21/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:109:109 [3] NCCL INFO Channel 22/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:107:107 [1] NCCL INFO Channel 23/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:112:112 [6] NCCL INFO Channel 20/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:108:108 [2] NCCL INFO Channel 13/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:110:110 [4] NCCL INFO Channel 13/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:111:111 [5] NCCL INFO Channel 22/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:109:109 [3] NCCL INFO Channel 23/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:112:112 [6] NCCL INFO Channel 21/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:108:108 [2] NCCL INFO Channel 14/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:110:110 [4] NCCL INFO Channel 14/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:111:111 [5] NCCL INFO Channel 23/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:112:112 [6] NCCL INFO Channel 22/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:108:108 [2] NCCL INFO Channel 15/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:110:110 [4] NCCL INFO Channel 15/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:112:112 [6] NCCL INFO Channel 23/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:108:108 [2] NCCL INFO Channel 16/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:110:110 [4] NCCL INFO Channel 16/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:108:108 [2] NCCL INFO Channel 17/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:110:110 [4] NCCL INFO Channel 17/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:108:108 [2] NCCL INFO Channel 18/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:110:110 [4] NCCL INFO Channel 18/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:108:108 [2] NCCL INFO Channel 19/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:110:110 [4] NCCL INFO Channel 19/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:108:108 [2] NCCL INFO Channel 20/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:110:110 [4] NCCL INFO Channel 20/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:108:108 [2] NCCL INFO Channel 21/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:108:108 [2] NCCL INFO Channel 22/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:108:108 [2] NCCL INFO Channel 23/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:110:110 [4] NCCL INFO Channel 21/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:110:110 [4] NCCL INFO Channel 22/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:110:110 [4] NCCL INFO Channel 23/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:113:113 [7] NCCL INFO Connected all trees | |
| tyler-rhel-newimage:113:113 [7] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512 | |
| tyler-rhel-newimage:113:113 [7] NCCL INFO 24 coll channels, 0 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer | |
| tyler-rhel-newimage:113:113 [7] NCCL INFO NCCL_WORK_FIFO_DEPTH set by environment to 4194304. | |
| tyler-rhel-newimage:84:84 [0] NCCL INFO Connected all trees | |
| tyler-rhel-newimage:84:84 [0] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512 | |
| tyler-rhel-newimage:84:84 [0] NCCL INFO 24 coll channels, 0 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer | |
| tyler-rhel-newimage:107:107 [1] NCCL INFO Connected all trees | |
| tyler-rhel-newimage:107:107 [1] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512 | |
| tyler-rhel-newimage:107:107 [1] NCCL INFO 24 coll channels, 0 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer | |
| tyler-rhel-newimage:111:111 [5] NCCL INFO Connected all trees | |
| tyler-rhel-newimage:112:112 [6] NCCL INFO Connected all trees | |
| tyler-rhel-newimage:111:111 [5] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512 | |
| tyler-rhel-newimage:112:112 [6] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512 | |
| tyler-rhel-newimage:111:111 [5] NCCL INFO 24 coll channels, 0 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer | |
| tyler-rhel-newimage:112:112 [6] NCCL INFO 24 coll channels, 0 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer | |
| tyler-rhel-newimage:108:108 [2] NCCL INFO Connected all trees | |
| tyler-rhel-newimage:108:108 [2] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512 | |
| tyler-rhel-newimage:108:108 [2] NCCL INFO 24 coll channels, 0 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer | |
| tyler-rhel-newimage:110:110 [4] NCCL INFO Connected all trees | |
| tyler-rhel-newimage:109:109 [3] NCCL INFO Connected all trees | |
| tyler-rhel-newimage:110:110 [4] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512 | |
| tyler-rhel-newimage:110:110 [4] NCCL INFO 24 coll channels, 0 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer | |
| tyler-rhel-newimage:109:109 [3] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512 | |
| tyler-rhel-newimage:109:109 [3] NCCL INFO 24 coll channels, 0 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer | |
| tyler-rhel-newimage:107:107 [1] NCCL INFO NCCL_WORK_FIFO_DEPTH set by environment to 4194304. | |
| tyler-rhel-newimage:108:108 [2] NCCL INFO NCCL_WORK_FIFO_DEPTH set by environment to 4194304. | |
| tyler-rhel-newimage:84:84 [0] NCCL INFO NCCL_WORK_FIFO_DEPTH set by environment to 4194304. | |
| tyler-rhel-newimage:112:112 [6] NCCL INFO NCCL_WORK_FIFO_DEPTH set by environment to 4194304. | |
| tyler-rhel-newimage:111:111 [5] NCCL INFO NCCL_WORK_FIFO_DEPTH set by environment to 4194304. | |
| tyler-rhel-newimage:109:109 [3] NCCL INFO NCCL_WORK_FIFO_DEPTH set by environment to 4194304. | |
| tyler-rhel-newimage:110:110 [4] NCCL INFO NCCL_WORK_FIFO_DEPTH set by environment to 4194304. | |
| INFO 2024-07-27 16:31:08,106 instructlab.model.backends.backends:336: Waiting for the vLLM server to start at http://127.0.0.1:37393/v1, this might take a moment... Attempt: 6/80 | |
| tyler-rhel-newimage:113:113 [7] NCCL INFO comm 0x55cf2d2f7410 rank 7 nranks 8 cudaDev 7 nvmlDev 7 busId e080 commId 0x3a52f5b817e350c3 - Init COMPLETE | |
| tyler-rhel-newimage:110:110 [4] NCCL INFO comm 0x55cf2d2f6cc0 rank 4 nranks 8 cudaDev 4 nvmlDev 4 busId c050 commId 0x3a52f5b817e350c3 - Init COMPLETE | |
| tyler-rhel-newimage:84:84 [0] NCCL INFO comm 0x55cf2d2fe4b0 rank 0 nranks 8 cudaDev 0 nvmlDev 0 busId 8010 commId 0x3a52f5b817e350c3 - Init COMPLETE | |
| tyler-rhel-newimage:111:111 [5] NCCL INFO comm 0x55cf2d2f6920 rank 5 nranks 8 cudaDev 5 nvmlDev 5 busId c060 commId 0x3a52f5b817e350c3 - Init COMPLETE | |
| tyler-rhel-newimage:112:112 [6] NCCL INFO comm 0x55cf2d2f7630 rank 6 nranks 8 cudaDev 6 nvmlDev 6 busId e070 commId 0x3a52f5b817e350c3 - Init COMPLETE | |
| tyler-rhel-newimage:107:107 [1] NCCL INFO comm 0x55cf2d2f5ca0 rank 1 nranks 8 cudaDev 1 nvmlDev 1 busId 8020 commId 0x3a52f5b817e350c3 - Init COMPLETE | |
| tyler-rhel-newimage:108:108 [2] NCCL INFO comm 0x55cf2d2f6410 rank 2 nranks 8 cudaDev 2 nvmlDev 2 busId a030 commId 0x3a52f5b817e350c3 - Init COMPLETE | |
| tyler-rhel-newimage:109:109 [3] NCCL INFO comm 0x55cf2d2f6610 rank 3 nranks 8 cudaDev 3 nvmlDev 3 busId a040 commId 0x3a52f5b817e350c3 - Init COMPLETE | |
| INFO 2024-07-27 16:31:12,668 instructlab.model.backends.backends:336: Waiting for the vLLM server to start at http://127.0.0.1:37393/v1, this might take a moment... Attempt: 7/80 | |
| Loading pt checkpoint shards: 0% Completed | 0/1 [00:00<?, ?it/s] | |
| INFO 2024-07-27 16:31:17,441 instructlab.model.backends.backends:336: Waiting for the vLLM server to start at http://127.0.0.1:37393/v1, this might take a moment... Attempt: 8/80 | |
| Loading pt checkpoint shards: 100% Completed | 1/1 [00:06<00:00, 6.05s/it] | |
| Loading pt checkpoint shards: 100% Completed | 1/1 [00:06<00:00, 6.05s/it] | |
| tyler-rhel-newimage:84:646 [0] NCCL INFO Using non-device net plugin version 0 | |
| tyler-rhel-newimage:84:646 [0] NCCL INFO Using network Socket | |
| tyler-rhel-newimage:113:649 [7] NCCL INFO Using non-device net plugin version 0 | |
| tyler-rhel-newimage:113:649 [7] NCCL INFO Using network Socket | |
| tyler-rhel-newimage:112:647 [6] NCCL INFO Using non-device net plugin version 0 | |
| tyler-rhel-newimage:112:647 [6] NCCL INFO Using network Socket | |
| tyler-rhel-newimage:109:648 [3] NCCL INFO Using non-device net plugin version 0 | |
| tyler-rhel-newimage:109:648 [3] NCCL INFO Using network Socket | |
| tyler-rhel-newimage:107:650 [1] NCCL INFO Using non-device net plugin version 0 | |
| tyler-rhel-newimage:107:650 [1] NCCL INFO Using network Socket | |
| tyler-rhel-newimage:111:651 [5] NCCL INFO Using non-device net plugin version 0 | |
| tyler-rhel-newimage:111:651 [5] NCCL INFO Using network Socket | |
| tyler-rhel-newimage:108:652 [2] NCCL INFO Using non-device net plugin version 0 | |
| tyler-rhel-newimage:108:652 [2] NCCL INFO Using network Socket | |
| tyler-rhel-newimage:110:653 [4] NCCL INFO Using non-device net plugin version 0 | |
| tyler-rhel-newimage:110:653 [4] NCCL INFO Using network Socket | |
| tyler-rhel-newimage:110:653 [4] NCCL INFO comm 0x55cf41ff79b0 rank 4 nranks 8 cudaDev 4 nvmlDev 4 busId c050 commId 0xdf19ab53c1b73809 - Init START | |
| tyler-rhel-newimage:84:646 [0] NCCL INFO comm 0x55cf41fd47c0 rank 0 nranks 8 cudaDev 0 nvmlDev 0 busId 8010 commId 0xdf19ab53c1b73809 - Init START | |
| tyler-rhel-newimage:109:648 [3] NCCL INFO comm 0x55cf41ff6a70 rank 3 nranks 8 cudaDev 3 nvmlDev 3 busId a040 commId 0xdf19ab53c1b73809 - Init START | |
| tyler-rhel-newimage:111:651 [5] NCCL INFO comm 0x55cf41ff76e0 rank 5 nranks 8 cudaDev 5 nvmlDev 5 busId c060 commId 0xdf19ab53c1b73809 - Init START | |
| tyler-rhel-newimage:108:652 [2] NCCL INFO comm 0x55cf41ff7720 rank 2 nranks 8 cudaDev 2 nvmlDev 2 busId a030 commId 0xdf19ab53c1b73809 - Init START | |
| tyler-rhel-newimage:107:650 [1] NCCL INFO comm 0x55cf41ff6f70 rank 1 nranks 8 cudaDev 1 nvmlDev 1 busId 8020 commId 0xdf19ab53c1b73809 - Init START | |
| tyler-rhel-newimage:113:649 [7] NCCL INFO comm 0x55cf41fe2540 rank 7 nranks 8 cudaDev 7 nvmlDev 7 busId e080 commId 0xdf19ab53c1b73809 - Init START | |
| tyler-rhel-newimage:112:647 [6] NCCL INFO comm 0x55cf41ff7e60 rank 6 nranks 8 cudaDev 6 nvmlDev 6 busId e070 commId 0xdf19ab53c1b73809 - Init START | |
| INFO 2024-07-27 16:31:21,998 instructlab.model.backends.backends:336: Waiting for the vLLM server to start at http://127.0.0.1:37393/v1, this might take a moment... Attempt: 9/80 | |
| tyler-rhel-newimage:110:653 [4] NCCL INFO Setting affinity for GPU 4 to ffff,ffffff00,00000000 | |
| tyler-rhel-newimage:110:653 [4] NCCL INFO NVLS multicast support is not available on dev 4 | |
| tyler-rhel-newimage:107:650 [1] NCCL INFO Setting affinity for GPU 1 to ff,ffffffff | |
| tyler-rhel-newimage:107:650 [1] NCCL INFO NVLS multicast support is not available on dev 1 | |
| tyler-rhel-newimage:112:647 [6] NCCL INFO Setting affinity for GPU 6 to ffff,ffffff00,00000000 | |
| tyler-rhel-newimage:84:646 [0] NCCL INFO Setting affinity for GPU 0 to ff,ffffffff | |
| tyler-rhel-newimage:112:647 [6] NCCL INFO NVLS multicast support is not available on dev 6 | |
| tyler-rhel-newimage:84:646 [0] NCCL INFO NVLS multicast support is not available on dev 0 | |
| tyler-rhel-newimage:109:648 [3] NCCL INFO Setting affinity for GPU 3 to ff,ffffffff | |
| tyler-rhel-newimage:109:648 [3] NCCL INFO NVLS multicast support is not available on dev 3 | |
| tyler-rhel-newimage:113:649 [7] NCCL INFO Setting affinity for GPU 7 to ffff,ffffff00,00000000 | |
| tyler-rhel-newimage:113:649 [7] NCCL INFO NVLS multicast support is not available on dev 7 | |
| tyler-rhel-newimage:111:651 [5] NCCL INFO Setting affinity for GPU 5 to ffff,ffffff00,00000000 | |
| tyler-rhel-newimage:111:651 [5] NCCL INFO NVLS multicast support is not available on dev 5 | |
| tyler-rhel-newimage:108:652 [2] NCCL INFO Setting affinity for GPU 2 to ff,ffffffff | |
| tyler-rhel-newimage:108:652 [2] NCCL INFO NVLS multicast support is not available on dev 2 | |
| tyler-rhel-newimage:113:649 [7] NCCL INFO comm 0x55cf41fe2540 rank 7 nRanks 8 nNodes 1 localRanks 8 localRank 7 MNNVL 0 | |
| tyler-rhel-newimage:107:650 [1] NCCL INFO comm 0x55cf41ff6f70 rank 1 nRanks 8 nNodes 1 localRanks 8 localRank 1 MNNVL 0 | |
| tyler-rhel-newimage:84:646 [0] NCCL INFO comm 0x55cf41fd47c0 rank 0 nRanks 8 nNodes 1 localRanks 8 localRank 0 MNNVL 0 | |
| tyler-rhel-newimage:108:652 [2] NCCL INFO comm 0x55cf41ff7720 rank 2 nRanks 8 nNodes 1 localRanks 8 localRank 2 MNNVL 0 | |
| tyler-rhel-newimage:111:651 [5] NCCL INFO comm 0x55cf41ff76e0 rank 5 nRanks 8 nNodes 1 localRanks 8 localRank 5 MNNVL 0 | |
| tyler-rhel-newimage:109:648 [3] NCCL INFO comm 0x55cf41ff6a70 rank 3 nRanks 8 nNodes 1 localRanks 8 localRank 3 MNNVL 0 | |
| tyler-rhel-newimage:112:647 [6] NCCL INFO comm 0x55cf41ff7e60 rank 6 nRanks 8 nNodes 1 localRanks 8 localRank 6 MNNVL 0 | |
| tyler-rhel-newimage:113:649 [7] NCCL INFO Trees [0] -1/-1/-1->7->6 [1] -1/-1/-1->7->6 [2] -1/-1/-1->7->6 [3] -1/-1/-1->7->6 [4] -1/-1/-1->7->6 [5] -1/-1/-1->7->6 [6] -1/-1/-1->7->6 [7] -1/-1/-1->7->6 [8] -1/-1/-1->7->6 [9] -1/-1/-1->7->6 [10] -1/-1/-1->7->6 [11] -1/-1/-1->7->6 [12] -1/-1/-1->7->6 [13] -1/-1/-1->7->6 [14] -1/-1/-1->7->6 [15] -1/-1/-1->7->6 [16] -1/-1/-1->7->6 [17] -1/-1/-1->7->6 [18] -1/-1/-1->7->6 [19] -1/-1/-1->7->6 [20] -1/-1/-1->7->6 [21] -1/-1/-1->7->6 [22] -1/-1/-1->7->6 [23] -1/-1/-1->7->6 | |
| tyler-rhel-newimage:110:653 [4] NCCL INFO comm 0x55cf41ff79b0 rank 4 nRanks 8 nNodes 1 localRanks 8 localRank 4 MNNVL 0 | |
| tyler-rhel-newimage:84:646 [0] NCCL INFO Channel 00/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:113:649 [7] NCCL INFO P2P Chunksize set to 524288 | |
| tyler-rhel-newimage:107:650 [1] NCCL INFO Trees [0] 2/-1/-1->1->0 [1] 2/-1/-1->1->0 [2] 2/-1/-1->1->0 [3] 2/-1/-1->1->0 [4] 2/-1/-1->1->0 [5] 2/-1/-1->1->0 [6] 2/-1/-1->1->0 [7] 2/-1/-1->1->0 [8] 2/-1/-1->1->0 [9] 2/-1/-1->1->0 [10] 2/-1/-1->1->0 [11] 2/-1/-1->1->0 [12] 2/-1/-1->1->0 [13] 2/-1/-1->1->0 [14] 2/-1/-1->1->0 [15] 2/-1/-1->1->0 [16] 2/-1/-1->1->0 [17] 2/-1/-1->1->0 [18] 2/-1/-1->1->0 [19] 2/-1/-1->1->0 [20] 2/-1/-1->1->0 [21] 2/-1/-1->1->0 [22] 2/-1/-1->1->0 [23] 2/-1/-1->1->0 | |
| tyler-rhel-newimage:84:646 [0] NCCL INFO Channel 01/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:111:651 [5] NCCL INFO Trees [0] 6/-1/-1->5->4 [1] 6/-1/-1->5->4 [2] 6/-1/-1->5->4 [3] 6/-1/-1->5->4 [4] 6/-1/-1->5->4 [5] 6/-1/-1->5->4 [6] 6/-1/-1->5->4 [7] 6/-1/-1->5->4 [8] 6/-1/-1->5->4 [9] 6/-1/-1->5->4 [10] 6/-1/-1->5->4 [11] 6/-1/-1->5->4 [12] 6/-1/-1->5->4 [13] 6/-1/-1->5->4 [14] 6/-1/-1->5->4 [15] 6/-1/-1->5->4 [16] 6/-1/-1->5->4 [17] 6/-1/-1->5->4 [18] 6/-1/-1->5->4 [19] 6/-1/-1->5->4 [20] 6/-1/-1->5->4 [21] 6/-1/-1->5->4 [22] 6/-1/-1->5->4 [23] 6/-1/-1->5->4 | |
| tyler-rhel-newimage:108:652 [2] NCCL INFO Trees [0] 3/-1/-1->2->1 [1] 3/-1/-1->2->1 [2] 3/-1/-1->2->1 [3] 3/-1/-1->2->1 [4] 3/-1/-1->2->1 [5] 3/-1/-1->2->1 [6] 3/-1/-1->2->1 [7] 3/-1/-1->2->1 [8] 3/-1/-1->2->1 [9] 3/-1/-1->2->1 [10] 3/-1/-1->2->1 [11] 3/-1/-1->2->1 [12] 3/-1/-1->2->1 [13] 3/-1/-1->2->1 [14] 3/-1/-1->2->1 [15] 3/-1/-1->2->1 [16] 3/-1/-1->2->1 [17] 3/-1/-1->2->1 [18] 3/-1/-1->2->1 [19] 3/-1/-1->2->1 [20] 3/-1/-1->2->1 [21] 3/-1/-1->2->1 [22] 3/-1/-1->2->1 [23] 3/-1/-1->2->1 | |
| tyler-rhel-newimage:109:648 [3] NCCL INFO Trees [0] 4/-1/-1->3->2 [1] 4/-1/-1->3->2 [2] 4/-1/-1->3->2 [3] 4/-1/-1->3->2 [4] 4/-1/-1->3->2 [5] 4/-1/-1->3->2 [6] 4/-1/-1->3->2 [7] 4/-1/-1->3->2 [8] 4/-1/-1->3->2 [9] 4/-1/-1->3->2 [10] 4/-1/-1->3->2 [11] 4/-1/-1->3->2 [12] 4/-1/-1->3->2 [13] 4/-1/-1->3->2 [14] 4/-1/-1->3->2 [15] 4/-1/-1->3->2 [16] 4/-1/-1->3->2 [17] 4/-1/-1->3->2 [18] 4/-1/-1->3->2 [19] 4/-1/-1->3->2 [20] 4/-1/-1->3->2 [21] 4/-1/-1->3->2 [22] 4/-1/-1->3->2 [23] 4/-1/-1->3->2 | |
| tyler-rhel-newimage:108:652 [2] NCCL INFO P2P Chunksize set to 524288 | |
| tyler-rhel-newimage:112:647 [6] NCCL INFO Trees [0] 7/-1/-1->6->5 [1] 7/-1/-1->6->5 [2] 7/-1/-1->6->5 [3] 7/-1/-1->6->5 [4] 7/-1/-1->6->5 [5] 7/-1/-1->6->5 [6] 7/-1/-1->6->5 [7] 7/-1/-1->6->5 [8] 7/-1/-1->6->5 [9] 7/-1/-1->6->5 [10] 7/-1/-1->6->5 [11] 7/-1/-1->6->5 [12] 7/-1/-1->6->5 [13] 7/-1/-1->6->5 [14] 7/-1/-1->6->5 [15] 7/-1/-1->6->5 [16] 7/-1/-1->6->5 [17] 7/-1/-1->6->5 [18] 7/-1/-1->6->5 [19] 7/-1/-1->6->5 [20] 7/-1/-1->6->5 [21] 7/-1/-1->6->5 [22] 7/-1/-1->6->5 [23] 7/-1/-1->6->5 | |
| tyler-rhel-newimage:84:646 [0] NCCL INFO Channel 02/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:107:650 [1] NCCL INFO P2P Chunksize set to 524288 | |
| tyler-rhel-newimage:111:651 [5] NCCL INFO P2P Chunksize set to 524288 | |
| tyler-rhel-newimage:110:653 [4] NCCL INFO Trees [0] 5/-1/-1->4->3 [1] 5/-1/-1->4->3 [2] 5/-1/-1->4->3 [3] 5/-1/-1->4->3 [4] 5/-1/-1->4->3 [5] 5/-1/-1->4->3 [6] 5/-1/-1->4->3 [7] 5/-1/-1->4->3 [8] 5/-1/-1->4->3 [9] 5/-1/-1->4->3 [10] 5/-1/-1->4->3 [11] 5/-1/-1->4->3 [12] 5/-1/-1->4->3 [13] 5/-1/-1->4->3 [14] 5/-1/-1->4->3 [15] 5/-1/-1->4->3 [16] 5/-1/-1->4->3 [17] 5/-1/-1->4->3 [18] 5/-1/-1->4->3 [19] 5/-1/-1->4->3 [20] 5/-1/-1->4->3 [21] 5/-1/-1->4->3 [22] 5/-1/-1->4->3 [23] 5/-1/-1->4->3 | |
| tyler-rhel-newimage:109:648 [3] NCCL INFO P2P Chunksize set to 524288 | |
| tyler-rhel-newimage:112:647 [6] NCCL INFO P2P Chunksize set to 524288 | |
| tyler-rhel-newimage:84:646 [0] NCCL INFO Channel 03/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:110:653 [4] NCCL INFO P2P Chunksize set to 524288 | |
| tyler-rhel-newimage:84:646 [0] NCCL INFO Channel 04/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:84:646 [0] NCCL INFO Channel 05/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:84:646 [0] NCCL INFO Channel 06/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:84:646 [0] NCCL INFO Channel 07/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:84:646 [0] NCCL INFO Channel 08/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:84:646 [0] NCCL INFO Channel 09/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:84:646 [0] NCCL INFO Channel 10/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:84:646 [0] NCCL INFO Channel 11/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:84:646 [0] NCCL INFO Channel 12/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:84:646 [0] NCCL INFO Channel 13/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:84:646 [0] NCCL INFO Channel 14/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:84:646 [0] NCCL INFO Channel 15/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:84:646 [0] NCCL INFO Channel 16/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:84:646 [0] NCCL INFO Channel 17/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:84:646 [0] NCCL INFO Channel 18/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:84:646 [0] NCCL INFO Channel 19/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:84:646 [0] NCCL INFO Channel 20/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:84:646 [0] NCCL INFO Channel 21/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:84:646 [0] NCCL INFO Channel 22/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:84:646 [0] NCCL INFO Channel 23/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:84:646 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1 [2] 1/-1/-1->0->-1 [3] 1/-1/-1->0->-1 [4] 1/-1/-1->0->-1 [5] 1/-1/-1->0->-1 [6] 1/-1/-1->0->-1 [7] 1/-1/-1->0->-1 [8] 1/-1/-1->0->-1 [9] 1/-1/-1->0->-1 [10] 1/-1/-1->0->-1 [11] 1/-1/-1->0->-1 [12] 1/-1/-1->0->-1 [13] 1/-1/-1->0->-1 [14] 1/-1/-1->0->-1 [15] 1/-1/-1->0->-1 [16] 1/-1/-1->0->-1 [17] 1/-1/-1->0->-1 [18] 1/-1/-1->0->-1 [19] 1/-1/-1->0->-1 [20] 1/-1/-1->0->-1 [21] 1/-1/-1->0->-1 [22] 1/-1/-1->0->-1 [23] 1/-1/-1->0->-1 | |
| tyler-rhel-newimage:84:646 [0] NCCL INFO P2P Chunksize set to 524288 | |
| tyler-rhel-newimage:111:651 [5] NCCL INFO Channel 00/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:113:649 [7] NCCL INFO Channel 00/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:109:648 [3] NCCL INFO Channel 00/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:111:651 [5] NCCL INFO Channel 01/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:113:649 [7] NCCL INFO Channel 01/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:109:648 [3] NCCL INFO Channel 01/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:111:651 [5] NCCL INFO Channel 02/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:113:649 [7] NCCL INFO Channel 02/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:110:653 [4] NCCL INFO Channel 00/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:112:647 [6] NCCL INFO Channel 00/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:109:648 [3] NCCL INFO Channel 02/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:111:651 [5] NCCL INFO Channel 03/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:113:649 [7] NCCL INFO Channel 03/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:110:653 [4] NCCL INFO Channel 01/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:108:652 [2] NCCL INFO Channel 00/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:107:650 [1] NCCL INFO Channel 00/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:112:647 [6] NCCL INFO Channel 01/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:109:648 [3] NCCL INFO Channel 03/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:111:651 [5] NCCL INFO Channel 04/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:113:649 [7] NCCL INFO Channel 04/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:110:653 [4] NCCL INFO Channel 02/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:108:652 [2] NCCL INFO Channel 01/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:107:650 [1] NCCL INFO Channel 01/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:109:648 [3] NCCL INFO Channel 04/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:112:647 [6] NCCL INFO Channel 02/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:111:651 [5] NCCL INFO Channel 05/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:113:649 [7] NCCL INFO Channel 05/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:110:653 [4] NCCL INFO Channel 03/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:108:652 [2] NCCL INFO Channel 02/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:109:648 [3] NCCL INFO Channel 05/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:107:650 [1] NCCL INFO Channel 02/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:112:647 [6] NCCL INFO Channel 03/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:113:649 [7] NCCL INFO Channel 06/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:111:651 [5] NCCL INFO Channel 06/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:110:653 [4] NCCL INFO Channel 04/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:108:652 [2] NCCL INFO Channel 03/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:109:648 [3] NCCL INFO Channel 06/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:107:650 [1] NCCL INFO Channel 03/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:112:647 [6] NCCL INFO Channel 04/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:111:651 [5] NCCL INFO Channel 07/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:113:649 [7] NCCL INFO Channel 07/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:110:653 [4] NCCL INFO Channel 05/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:108:652 [2] NCCL INFO Channel 04/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:109:648 [3] NCCL INFO Channel 07/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:107:650 [1] NCCL INFO Channel 04/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:84:646 [0] NCCL INFO Channel 00/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:112:647 [6] NCCL INFO Channel 05/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:111:651 [5] NCCL INFO Channel 08/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:113:649 [7] NCCL INFO Channel 08/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:110:653 [4] NCCL INFO Channel 06/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:108:652 [2] NCCL INFO Channel 05/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:109:648 [3] NCCL INFO Channel 08/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:107:650 [1] NCCL INFO Channel 05/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:84:646 [0] NCCL INFO Channel 01/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:112:647 [6] NCCL INFO Channel 06/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:113:649 [7] NCCL INFO Channel 09/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:111:651 [5] NCCL INFO Channel 09/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:110:653 [4] NCCL INFO Channel 07/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:108:652 [2] NCCL INFO Channel 06/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:109:648 [3] NCCL INFO Channel 09/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:107:650 [1] NCCL INFO Channel 06/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:112:647 [6] NCCL INFO Channel 07/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:84:646 [0] NCCL INFO Channel 02/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:110:653 [4] NCCL INFO Channel 08/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:113:649 [7] NCCL INFO Channel 10/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:111:651 [5] NCCL INFO Channel 10/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:108:652 [2] NCCL INFO Channel 07/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:109:648 [3] NCCL INFO Channel 10/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:107:650 [1] NCCL INFO Channel 07/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:112:647 [6] NCCL INFO Channel 08/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:84:646 [0] NCCL INFO Channel 03/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:110:653 [4] NCCL INFO Channel 09/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:113:649 [7] NCCL INFO Channel 11/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:111:651 [5] NCCL INFO Channel 11/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:108:652 [2] NCCL INFO Channel 08/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:109:648 [3] NCCL INFO Channel 11/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:112:647 [6] NCCL INFO Channel 09/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:107:650 [1] NCCL INFO Channel 08/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:84:646 [0] NCCL INFO Channel 04/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:110:653 [4] NCCL INFO Channel 10/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:113:649 [7] NCCL INFO Channel 12/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:111:651 [5] NCCL INFO Channel 12/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:109:648 [3] NCCL INFO Channel 12/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:108:652 [2] NCCL INFO Channel 09/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:112:647 [6] NCCL INFO Channel 10/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:107:650 [1] NCCL INFO Channel 09/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:84:646 [0] NCCL INFO Channel 05/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:110:653 [4] NCCL INFO Channel 11/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:113:649 [7] NCCL INFO Channel 13/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:111:651 [5] NCCL INFO Channel 13/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:109:648 [3] NCCL INFO Channel 13/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:108:652 [2] NCCL INFO Channel 10/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:107:650 [1] NCCL INFO Channel 10/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:112:647 [6] NCCL INFO Channel 11/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:84:646 [0] NCCL INFO Channel 06/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:113:649 [7] NCCL INFO Channel 14/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:110:653 [4] NCCL INFO Channel 12/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:111:651 [5] NCCL INFO Channel 14/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:109:648 [3] NCCL INFO Channel 14/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:108:652 [2] NCCL INFO Channel 11/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:107:650 [1] NCCL INFO Channel 11/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:112:647 [6] NCCL INFO Channel 12/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:84:646 [0] NCCL INFO Channel 07/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:113:649 [7] NCCL INFO Channel 15/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:110:653 [4] NCCL INFO Channel 13/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:111:651 [5] NCCL INFO Channel 15/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:109:648 [3] NCCL INFO Channel 15/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:108:652 [2] NCCL INFO Channel 12/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:107:650 [1] NCCL INFO Channel 12/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:112:647 [6] NCCL INFO Channel 13/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:84:646 [0] NCCL INFO Channel 08/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:113:649 [7] NCCL INFO Channel 16/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:110:653 [4] NCCL INFO Channel 14/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:111:651 [5] NCCL INFO Channel 16/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:109:648 [3] NCCL INFO Channel 16/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:108:652 [2] NCCL INFO Channel 13/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:107:650 [1] NCCL INFO Channel 13/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:112:647 [6] NCCL INFO Channel 14/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:84:646 [0] NCCL INFO Channel 09/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:113:649 [7] NCCL INFO Channel 17/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:110:653 [4] NCCL INFO Channel 15/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:109:648 [3] NCCL INFO Channel 17/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:111:651 [5] NCCL INFO Channel 17/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:108:652 [2] NCCL INFO Channel 14/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:112:647 [6] NCCL INFO Channel 15/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:107:650 [1] NCCL INFO Channel 14/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:84:646 [0] NCCL INFO Channel 10/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:113:649 [7] NCCL INFO Channel 18/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:110:653 [4] NCCL INFO Channel 16/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:109:648 [3] NCCL INFO Channel 18/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:111:651 [5] NCCL INFO Channel 18/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:112:647 [6] NCCL INFO Channel 16/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:108:652 [2] NCCL INFO Channel 15/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:107:650 [1] NCCL INFO Channel 15/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:84:646 [0] NCCL INFO Channel 11/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:109:648 [3] NCCL INFO Channel 19/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:113:649 [7] NCCL INFO Channel 19/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:110:653 [4] NCCL INFO Channel 17/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:111:651 [5] NCCL INFO Channel 19/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:112:647 [6] NCCL INFO Channel 17/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:108:652 [2] NCCL INFO Channel 16/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:107:650 [1] NCCL INFO Channel 16/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:84:646 [0] NCCL INFO Channel 12/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:109:648 [3] NCCL INFO Channel 20/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:113:649 [7] NCCL INFO Channel 20/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:110:653 [4] NCCL INFO Channel 18/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:111:651 [5] NCCL INFO Channel 20/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:112:647 [6] NCCL INFO Channel 18/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:108:652 [2] NCCL INFO Channel 17/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:107:650 [1] NCCL INFO Channel 17/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:109:648 [3] NCCL INFO Channel 21/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:84:646 [0] NCCL INFO Channel 13/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:113:649 [7] NCCL INFO Channel 21/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:110:653 [4] NCCL INFO Channel 19/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:111:651 [5] NCCL INFO Channel 21/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:112:647 [6] NCCL INFO Channel 19/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:108:652 [2] NCCL INFO Channel 18/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:107:650 [1] NCCL INFO Channel 18/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:109:648 [3] NCCL INFO Channel 22/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:84:646 [0] NCCL INFO Channel 14/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:113:649 [7] NCCL INFO Channel 22/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:111:651 [5] NCCL INFO Channel 22/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:110:653 [4] NCCL INFO Channel 20/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:112:647 [6] NCCL INFO Channel 20/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:108:652 [2] NCCL INFO Channel 19/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:107:650 [1] NCCL INFO Channel 19/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:109:648 [3] NCCL INFO Channel 23/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:84:646 [0] NCCL INFO Channel 15/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:110:653 [4] NCCL INFO Channel 21/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:113:649 [7] NCCL INFO Channel 23/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:111:651 [5] NCCL INFO Channel 23/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:112:647 [6] NCCL INFO Channel 21/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:108:652 [2] NCCL INFO Channel 20/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:107:650 [1] NCCL INFO Channel 20/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:84:646 [0] NCCL INFO Channel 16/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:110:653 [4] NCCL INFO Channel 22/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:112:647 [6] NCCL INFO Channel 22/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:108:652 [2] NCCL INFO Channel 21/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:107:650 [1] NCCL INFO Channel 21/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:84:646 [0] NCCL INFO Channel 17/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:110:653 [4] NCCL INFO Channel 23/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:112:647 [6] NCCL INFO Channel 23/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:108:652 [2] NCCL INFO Channel 22/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:107:650 [1] NCCL INFO Channel 22/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:84:646 [0] NCCL INFO Channel 18/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:108:652 [2] NCCL INFO Channel 23/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:107:650 [1] NCCL INFO Channel 23/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:84:646 [0] NCCL INFO Channel 19/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:84:646 [0] NCCL INFO Channel 20/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:84:646 [0] NCCL INFO Channel 21/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:84:646 [0] NCCL INFO Channel 22/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:84:646 [0] NCCL INFO Channel 23/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:111:651 [5] NCCL INFO Connected all rings | |
| tyler-rhel-newimage:110:653 [4] NCCL INFO Connected all rings | |
| tyler-rhel-newimage:109:648 [3] NCCL INFO Connected all rings | |
| tyler-rhel-newimage:112:647 [6] NCCL INFO Connected all rings | |
| tyler-rhel-newimage:113:649 [7] NCCL INFO Connected all rings | |
| tyler-rhel-newimage:113:649 [7] NCCL INFO Channel 00/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:113:649 [7] NCCL INFO Channel 01/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:108:652 [2] NCCL INFO Connected all rings | |
| tyler-rhel-newimage:84:646 [0] NCCL INFO Connected all rings | |
| tyler-rhel-newimage:107:650 [1] NCCL INFO Connected all rings | |
| tyler-rhel-newimage:113:649 [7] NCCL INFO Channel 02/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:113:649 [7] NCCL INFO Channel 03/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:113:649 [7] NCCL INFO Channel 04/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:113:649 [7] NCCL INFO Channel 05/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:113:649 [7] NCCL INFO Channel 06/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:113:649 [7] NCCL INFO Channel 07/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:113:649 [7] NCCL INFO Channel 08/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:113:649 [7] NCCL INFO Channel 09/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:113:649 [7] NCCL INFO Channel 10/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:113:649 [7] NCCL INFO Channel 11/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:113:649 [7] NCCL INFO Channel 12/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:113:649 [7] NCCL INFO Channel 13/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:113:649 [7] NCCL INFO Channel 14/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:113:649 [7] NCCL INFO Channel 15/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:113:649 [7] NCCL INFO Channel 16/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:113:649 [7] NCCL INFO Channel 17/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:113:649 [7] NCCL INFO Channel 18/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:113:649 [7] NCCL INFO Channel 19/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:113:649 [7] NCCL INFO Channel 20/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:111:651 [5] NCCL INFO Channel 00/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:109:648 [3] NCCL INFO Channel 00/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:113:649 [7] NCCL INFO Channel 21/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:111:651 [5] NCCL INFO Channel 01/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:109:648 [3] NCCL INFO Channel 01/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:113:649 [7] NCCL INFO Channel 22/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:110:653 [4] NCCL INFO Channel 00/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:109:648 [3] NCCL INFO Channel 02/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:111:651 [5] NCCL INFO Channel 02/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:113:649 [7] NCCL INFO Channel 23/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:110:653 [4] NCCL INFO Channel 01/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:109:648 [3] NCCL INFO Channel 03/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:111:651 [5] NCCL INFO Channel 03/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:110:653 [4] NCCL INFO Channel 02/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:109:648 [3] NCCL INFO Channel 04/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:111:651 [5] NCCL INFO Channel 04/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:107:650 [1] NCCL INFO Channel 00/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:110:653 [4] NCCL INFO Channel 03/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:109:648 [3] NCCL INFO Channel 05/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:112:647 [6] NCCL INFO Channel 00/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:108:652 [2] NCCL INFO Channel 00/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:111:651 [5] NCCL INFO Channel 05/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:107:650 [1] NCCL INFO Channel 01/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:110:653 [4] NCCL INFO Channel 04/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:109:648 [3] NCCL INFO Channel 06/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:112:647 [6] NCCL INFO Channel 01/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:108:652 [2] NCCL INFO Channel 01/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:107:650 [1] NCCL INFO Channel 02/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:111:651 [5] NCCL INFO Channel 06/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:110:653 [4] NCCL INFO Channel 05/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:109:648 [3] NCCL INFO Channel 07/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:112:647 [6] NCCL INFO Channel 02/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:108:652 [2] NCCL INFO Channel 02/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:107:650 [1] NCCL INFO Channel 03/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:111:651 [5] NCCL INFO Channel 07/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:110:653 [4] NCCL INFO Channel 06/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:109:648 [3] NCCL INFO Channel 08/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:112:647 [6] NCCL INFO Channel 03/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:108:652 [2] NCCL INFO Channel 03/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:107:650 [1] NCCL INFO Channel 04/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:111:651 [5] NCCL INFO Channel 08/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:110:653 [4] NCCL INFO Channel 07/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:109:648 [3] NCCL INFO Channel 09/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:112:647 [6] NCCL INFO Channel 04/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:108:652 [2] NCCL INFO Channel 04/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:107:650 [1] NCCL INFO Channel 05/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:110:653 [4] NCCL INFO Channel 08/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:111:651 [5] NCCL INFO Channel 09/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:109:648 [3] NCCL INFO Channel 10/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:112:647 [6] NCCL INFO Channel 05/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:108:652 [2] NCCL INFO Channel 05/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:107:650 [1] NCCL INFO Channel 06/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:110:653 [4] NCCL INFO Channel 09/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:111:651 [5] NCCL INFO Channel 10/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:109:648 [3] NCCL INFO Channel 11/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:112:647 [6] NCCL INFO Channel 06/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:108:652 [2] NCCL INFO Channel 06/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:107:650 [1] NCCL INFO Channel 07/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:110:653 [4] NCCL INFO Channel 10/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:111:651 [5] NCCL INFO Channel 11/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:109:648 [3] NCCL INFO Channel 12/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:112:647 [6] NCCL INFO Channel 07/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:108:652 [2] NCCL INFO Channel 07/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:107:650 [1] NCCL INFO Channel 08/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:110:653 [4] NCCL INFO Channel 11/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:111:651 [5] NCCL INFO Channel 12/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:109:648 [3] NCCL INFO Channel 13/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:112:647 [6] NCCL INFO Channel 08/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:108:652 [2] NCCL INFO Channel 08/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:107:650 [1] NCCL INFO Channel 09/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:110:653 [4] NCCL INFO Channel 12/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:111:651 [5] NCCL INFO Channel 13/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:109:648 [3] NCCL INFO Channel 14/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:112:647 [6] NCCL INFO Channel 09/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:108:652 [2] NCCL INFO Channel 09/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:107:650 [1] NCCL INFO Channel 10/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:110:653 [4] NCCL INFO Channel 13/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:111:651 [5] NCCL INFO Channel 14/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:109:648 [3] NCCL INFO Channel 15/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:112:647 [6] NCCL INFO Channel 10/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:107:650 [1] NCCL INFO Channel 11/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:108:652 [2] NCCL INFO Channel 10/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:110:653 [4] NCCL INFO Channel 14/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:111:651 [5] NCCL INFO Channel 15/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:109:648 [3] NCCL INFO Channel 16/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:112:647 [6] NCCL INFO Channel 11/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:107:650 [1] NCCL INFO Channel 12/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:108:652 [2] NCCL INFO Channel 11/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:110:653 [4] NCCL INFO Channel 15/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:111:651 [5] NCCL INFO Channel 16/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:109:648 [3] NCCL INFO Channel 17/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:112:647 [6] NCCL INFO Channel 12/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:107:650 [1] NCCL INFO Channel 13/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:108:652 [2] NCCL INFO Channel 12/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:110:653 [4] NCCL INFO Channel 16/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:111:651 [5] NCCL INFO Channel 17/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:109:648 [3] NCCL INFO Channel 18/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:107:650 [1] NCCL INFO Channel 14/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:112:647 [6] NCCL INFO Channel 13/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:108:652 [2] NCCL INFO Channel 13/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:110:653 [4] NCCL INFO Channel 17/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:111:651 [5] NCCL INFO Channel 18/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:109:648 [3] NCCL INFO Channel 19/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:107:650 [1] NCCL INFO Channel 15/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:112:647 [6] NCCL INFO Channel 14/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:108:652 [2] NCCL INFO Channel 14/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:110:653 [4] NCCL INFO Channel 18/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:109:648 [3] NCCL INFO Channel 20/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:111:651 [5] NCCL INFO Channel 19/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:107:650 [1] NCCL INFO Channel 16/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:112:647 [6] NCCL INFO Channel 15/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:110:653 [4] NCCL INFO Channel 19/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:108:652 [2] NCCL INFO Channel 15/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:109:648 [3] NCCL INFO Channel 21/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:111:651 [5] NCCL INFO Channel 20/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:107:650 [1] NCCL INFO Channel 17/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:108:652 [2] NCCL INFO Channel 16/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:112:647 [6] NCCL INFO Channel 16/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:110:653 [4] NCCL INFO Channel 20/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:109:648 [3] NCCL INFO Channel 22/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:107:650 [1] NCCL INFO Channel 18/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:111:651 [5] NCCL INFO Channel 21/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:108:652 [2] NCCL INFO Channel 17/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:112:647 [6] NCCL INFO Channel 17/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:110:653 [4] NCCL INFO Channel 21/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:109:648 [3] NCCL INFO Channel 23/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:107:650 [1] NCCL INFO Channel 19/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:108:652 [2] NCCL INFO Channel 18/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:111:651 [5] NCCL INFO Channel 22/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:112:647 [6] NCCL INFO Channel 18/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:110:653 [4] NCCL INFO Channel 22/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:107:650 [1] NCCL INFO Channel 20/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:108:652 [2] NCCL INFO Channel 19/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:111:651 [5] NCCL INFO Channel 23/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:112:647 [6] NCCL INFO Channel 19/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:110:653 [4] NCCL INFO Channel 23/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:107:650 [1] NCCL INFO Channel 21/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:108:652 [2] NCCL INFO Channel 20/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:112:647 [6] NCCL INFO Channel 20/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:107:650 [1] NCCL INFO Channel 22/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:108:652 [2] NCCL INFO Channel 21/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:112:647 [6] NCCL INFO Channel 21/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:107:650 [1] NCCL INFO Channel 23/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:108:652 [2] NCCL INFO Channel 22/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:112:647 [6] NCCL INFO Channel 22/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:108:652 [2] NCCL INFO Channel 23/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:112:647 [6] NCCL INFO Channel 23/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:111:651 [5] NCCL INFO Connected all trees | |
| tyler-rhel-newimage:111:651 [5] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512 | |
| tyler-rhel-newimage:111:651 [5] NCCL INFO 24 coll channels, 0 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer | |
| tyler-rhel-newimage:113:649 [7] NCCL INFO Connected all trees | |
| tyler-rhel-newimage:113:649 [7] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512 | |
| tyler-rhel-newimage:113:649 [7] NCCL INFO 24 coll channels, 0 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer | |
| tyler-rhel-newimage:112:647 [6] NCCL INFO Connected all trees | |
| tyler-rhel-newimage:112:647 [6] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512 | |
| tyler-rhel-newimage:112:647 [6] NCCL INFO 24 coll channels, 0 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer | |
| tyler-rhel-newimage:84:646 [0] NCCL INFO Connected all trees | |
| tyler-rhel-newimage:84:646 [0] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512 | |
| tyler-rhel-newimage:84:646 [0] NCCL INFO 24 coll channels, 0 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer | |
| tyler-rhel-newimage:107:650 [1] NCCL INFO Connected all trees | |
| tyler-rhel-newimage:107:650 [1] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512 | |
| tyler-rhel-newimage:107:650 [1] NCCL INFO 24 coll channels, 0 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer | |
| tyler-rhel-newimage:108:652 [2] NCCL INFO Connected all trees | |
| tyler-rhel-newimage:108:652 [2] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512 | |
| tyler-rhel-newimage:108:652 [2] NCCL INFO 24 coll channels, 0 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer | |
| tyler-rhel-newimage:109:648 [3] NCCL INFO Connected all trees | |
| tyler-rhel-newimage:110:653 [4] NCCL INFO Connected all trees | |
| tyler-rhel-newimage:109:648 [3] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512 | |
| tyler-rhel-newimage:109:648 [3] NCCL INFO 24 coll channels, 0 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer | |
| tyler-rhel-newimage:110:653 [4] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512 | |
| tyler-rhel-newimage:110:653 [4] NCCL INFO 24 coll channels, 0 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer | |
| tyler-rhel-newimage:108:652 [2] NCCL INFO comm 0x55cf41ff7720 rank 2 nranks 8 cudaDev 2 nvmlDev 2 busId a030 commId 0xdf19ab53c1b73809 - Init COMPLETE | |
| tyler-rhel-newimage:111:651 [5] NCCL INFO comm 0x55cf41ff76e0 rank 5 nranks 8 cudaDev 5 nvmlDev 5 busId c060 commId 0xdf19ab53c1b73809 - Init COMPLETE | |
| tyler-rhel-newimage:113:649 [7] NCCL INFO comm 0x55cf41fe2540 rank 7 nranks 8 cudaDev 7 nvmlDev 7 busId e080 commId 0xdf19ab53c1b73809 - Init COMPLETE | |
| tyler-rhel-newimage:109:648 [3] NCCL INFO comm 0x55cf41ff6a70 rank 3 nranks 8 cudaDev 3 nvmlDev 3 busId a040 commId 0xdf19ab53c1b73809 - Init COMPLETE | |
| tyler-rhel-newimage:112:647 [6] NCCL INFO comm 0x55cf41ff7e60 rank 6 nranks 8 cudaDev 6 nvmlDev 6 busId e070 commId 0xdf19ab53c1b73809 - Init COMPLETE | |
| tyler-rhel-newimage:107:650 [1] NCCL INFO comm 0x55cf41ff6f70 rank 1 nranks 8 cudaDev 1 nvmlDev 1 busId 8020 commId 0xdf19ab53c1b73809 - Init COMPLETE | |
| tyler-rhel-newimage:110:653 [4] NCCL INFO comm 0x55cf41ff79b0 rank 4 nranks 8 cudaDev 4 nvmlDev 4 busId c050 commId 0xdf19ab53c1b73809 - Init COMPLETE | |
| tyler-rhel-newimage:84:646 [0] NCCL INFO comm 0x55cf41fd47c0 rank 0 nranks 8 cudaDev 0 nvmlDev 0 busId 8010 commId 0xdf19ab53c1b73809 - Init COMPLETE | |
| tyler-rhel-newimage:109:678 [3] NCCL INFO Channel 00/1 : 3[3] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:111:679 [5] NCCL INFO Channel 00/1 : 5[5] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:109:678 [3] NCCL INFO Channel 01/1 : 3[3] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:111:679 [5] NCCL INFO Channel 01/1 : 5[5] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:109:678 [3] NCCL INFO Channel 02/1 : 3[3] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:107:680 [1] NCCL INFO Channel 00/1 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:111:679 [5] NCCL INFO Channel 02/1 : 5[5] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:108:682 [2] NCCL INFO Channel 00/1 : 2[2] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:109:678 [3] NCCL INFO Channel 03/1 : 3[3] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:111:679 [5] NCCL INFO Channel 03/1 : 5[5] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:107:680 [1] NCCL INFO Channel 01/1 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:109:678 [3] NCCL INFO Channel 04/1 : 3[3] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:112:683 [6] NCCL INFO Channel 00/1 : 6[6] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:111:679 [5] NCCL INFO Channel 04/1 : 5[5] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:108:682 [2] NCCL INFO Channel 01/1 : 2[2] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:107:680 [1] NCCL INFO Channel 02/1 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:113:684 [7] NCCL INFO Channel 00/1 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:109:678 [3] NCCL INFO Channel 05/1 : 3[3] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:110:685 [4] NCCL INFO Channel 00/1 : 4[4] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:111:679 [5] NCCL INFO Channel 05/1 : 5[5] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:108:682 [2] NCCL INFO Channel 02/1 : 2[2] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:112:683 [6] NCCL INFO Channel 01/1 : 6[6] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:107:680 [1] NCCL INFO Channel 03/1 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:113:684 [7] NCCL INFO Channel 01/1 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:109:678 [3] NCCL INFO Channel 06/1 : 3[3] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:110:685 [4] NCCL INFO Channel 01/1 : 4[4] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:108:682 [2] NCCL INFO Channel 03/1 : 2[2] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:111:679 [5] NCCL INFO Channel 06/1 : 5[5] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:112:683 [6] NCCL INFO Channel 02/1 : 6[6] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:107:680 [1] NCCL INFO Channel 04/1 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:113:684 [7] NCCL INFO Channel 02/1 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:109:678 [3] NCCL INFO Channel 07/1 : 3[3] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:110:685 [4] NCCL INFO Channel 02/1 : 4[4] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:108:682 [2] NCCL INFO Channel 04/1 : 2[2] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:111:679 [5] NCCL INFO Channel 07/1 : 5[5] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:107:680 [1] NCCL INFO Channel 05/1 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:112:683 [6] NCCL INFO Channel 03/1 : 6[6] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:113:684 [7] NCCL INFO Channel 03/1 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:109:678 [3] NCCL INFO Channel 08/1 : 3[3] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:108:682 [2] NCCL INFO Channel 05/1 : 2[2] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:110:685 [4] NCCL INFO Channel 03/1 : 4[4] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:111:679 [5] NCCL INFO Channel 08/1 : 5[5] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:112:683 [6] NCCL INFO Channel 04/1 : 6[6] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:113:684 [7] NCCL INFO Channel 04/1 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:107:680 [1] NCCL INFO Channel 06/1 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:109:678 [3] NCCL INFO Channel 09/1 : 3[3] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:110:685 [4] NCCL INFO Channel 04/1 : 4[4] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:111:679 [5] NCCL INFO Channel 09/1 : 5[5] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:108:682 [2] NCCL INFO Channel 06/1 : 2[2] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:112:683 [6] NCCL INFO Channel 05/1 : 6[6] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:113:684 [7] NCCL INFO Channel 05/1 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:107:680 [1] NCCL INFO Channel 07/1 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:109:678 [3] NCCL INFO Channel 10/1 : 3[3] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:111:679 [5] NCCL INFO Channel 10/1 : 5[5] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:110:685 [4] NCCL INFO Channel 05/1 : 4[4] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:108:682 [2] NCCL INFO Channel 07/1 : 2[2] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:112:683 [6] NCCL INFO Channel 06/1 : 6[6] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:113:684 [7] NCCL INFO Channel 06/1 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:107:680 [1] NCCL INFO Channel 08/1 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:111:679 [5] NCCL INFO Channel 11/1 : 5[5] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:109:678 [3] NCCL INFO Channel 11/1 : 3[3] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:108:682 [2] NCCL INFO Channel 08/1 : 2[2] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:110:685 [4] NCCL INFO Channel 06/1 : 4[4] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:112:683 [6] NCCL INFO Channel 07/1 : 6[6] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:113:684 [7] NCCL INFO Channel 07/1 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:107:680 [1] NCCL INFO Channel 09/1 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:111:679 [5] NCCL INFO Channel 12/1 : 5[5] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:108:682 [2] NCCL INFO Channel 09/1 : 2[2] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:110:685 [4] NCCL INFO Channel 07/1 : 4[4] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:109:678 [3] NCCL INFO Channel 12/1 : 3[3] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:112:683 [6] NCCL INFO Channel 08/1 : 6[6] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:113:684 [7] NCCL INFO Channel 08/1 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:107:680 [1] NCCL INFO Channel 10/1 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:111:679 [5] NCCL INFO Channel 13/1 : 5[5] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:108:682 [2] NCCL INFO Channel 10/1 : 2[2] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:110:685 [4] NCCL INFO Channel 08/1 : 4[4] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:109:678 [3] NCCL INFO Channel 13/1 : 3[3] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:112:683 [6] NCCL INFO Channel 09/1 : 6[6] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:113:684 [7] NCCL INFO Channel 09/1 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:107:680 [1] NCCL INFO Channel 11/1 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:111:679 [5] NCCL INFO Channel 14/1 : 5[5] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:108:682 [2] NCCL INFO Channel 11/1 : 2[2] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:110:685 [4] NCCL INFO Channel 09/1 : 4[4] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:109:678 [3] NCCL INFO Channel 14/1 : 3[3] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:112:683 [6] NCCL INFO Channel 10/1 : 6[6] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:113:684 [7] NCCL INFO Channel 10/1 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:107:680 [1] NCCL INFO Channel 12/1 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:111:679 [5] NCCL INFO Channel 15/1 : 5[5] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:108:682 [2] NCCL INFO Channel 12/1 : 2[2] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:110:685 [4] NCCL INFO Channel 10/1 : 4[4] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:109:678 [3] NCCL INFO Channel 15/1 : 3[3] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:112:683 [6] NCCL INFO Channel 11/1 : 6[6] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:113:684 [7] NCCL INFO Channel 11/1 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:107:680 [1] NCCL INFO Channel 13/1 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:111:679 [5] NCCL INFO Channel 16/1 : 5[5] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:108:682 [2] NCCL INFO Channel 13/1 : 2[2] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:110:685 [4] NCCL INFO Channel 11/1 : 4[4] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:112:683 [6] NCCL INFO Channel 12/1 : 6[6] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:109:678 [3] NCCL INFO Channel 16/1 : 3[3] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:113:684 [7] NCCL INFO Channel 12/1 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:107:680 [1] NCCL INFO Channel 14/1 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:111:679 [5] NCCL INFO Channel 17/1 : 5[5] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:108:682 [2] NCCL INFO Channel 14/1 : 2[2] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:110:685 [4] NCCL INFO Channel 12/1 : 4[4] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:112:683 [6] NCCL INFO Channel 13/1 : 6[6] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:109:678 [3] NCCL INFO Channel 17/1 : 3[3] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:113:684 [7] NCCL INFO Channel 13/1 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:111:679 [5] NCCL INFO Channel 18/1 : 5[5] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:107:680 [1] NCCL INFO Channel 15/1 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:108:682 [2] NCCL INFO Channel 15/1 : 2[2] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:110:685 [4] NCCL INFO Channel 13/1 : 4[4] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:109:678 [3] NCCL INFO Channel 18/1 : 3[3] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:113:684 [7] NCCL INFO Channel 14/1 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:112:683 [6] NCCL INFO Channel 14/1 : 6[6] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:111:679 [5] NCCL INFO Channel 19/1 : 5[5] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:107:680 [1] NCCL INFO Channel 16/1 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:108:682 [2] NCCL INFO Channel 16/1 : 2[2] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:110:685 [4] NCCL INFO Channel 14/1 : 4[4] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:112:683 [6] NCCL INFO Channel 15/1 : 6[6] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:113:684 [7] NCCL INFO Channel 15/1 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:109:678 [3] NCCL INFO Channel 19/1 : 3[3] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:111:679 [5] NCCL INFO Channel 20/1 : 5[5] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:107:680 [1] NCCL INFO Channel 17/1 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:108:682 [2] NCCL INFO Channel 17/1 : 2[2] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:110:685 [4] NCCL INFO Channel 15/1 : 4[4] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:113:684 [7] NCCL INFO Channel 16/1 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:112:683 [6] NCCL INFO Channel 16/1 : 6[6] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:109:678 [3] NCCL INFO Channel 20/1 : 3[3] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:111:679 [5] NCCL INFO Channel 21/1 : 5[5] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:107:680 [1] NCCL INFO Channel 18/1 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:108:682 [2] NCCL INFO Channel 18/1 : 2[2] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:110:685 [4] NCCL INFO Channel 16/1 : 4[4] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:113:684 [7] NCCL INFO Channel 17/1 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:112:683 [6] NCCL INFO Channel 17/1 : 6[6] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:109:678 [3] NCCL INFO Channel 21/1 : 3[3] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:111:679 [5] NCCL INFO Channel 22/1 : 5[5] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:107:680 [1] NCCL INFO Channel 19/1 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:108:682 [2] NCCL INFO Channel 19/1 : 2[2] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:110:685 [4] NCCL INFO Channel 17/1 : 4[4] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:113:684 [7] NCCL INFO Channel 18/1 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:112:683 [6] NCCL INFO Channel 18/1 : 6[6] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:111:679 [5] NCCL INFO Channel 23/1 : 5[5] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:109:678 [3] NCCL INFO Channel 22/1 : 3[3] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:107:680 [1] NCCL INFO Channel 20/1 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:108:682 [2] NCCL INFO Channel 20/1 : 2[2] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:110:685 [4] NCCL INFO Channel 18/1 : 4[4] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:113:684 [7] NCCL INFO Channel 19/1 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:112:683 [6] NCCL INFO Channel 19/1 : 6[6] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:111:679 [5] NCCL INFO Channel 24/1 : 5[5] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:109:678 [3] NCCL INFO Channel 23/1 : 3[3] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:107:680 [1] NCCL INFO Channel 21/1 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:108:682 [2] NCCL INFO Channel 21/1 : 2[2] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:110:685 [4] NCCL INFO Channel 19/1 : 4[4] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:113:684 [7] NCCL INFO Channel 20/1 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:112:683 [6] NCCL INFO Channel 20/1 : 6[6] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:111:679 [5] NCCL INFO Channel 25/1 : 5[5] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:109:678 [3] NCCL INFO Channel 24/1 : 3[3] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:108:682 [2] NCCL INFO Channel 22/1 : 2[2] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:107:680 [1] NCCL INFO Channel 22/1 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:110:685 [4] NCCL INFO Channel 20/1 : 4[4] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:113:684 [7] NCCL INFO Channel 21/1 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:112:683 [6] NCCL INFO Channel 21/1 : 6[6] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:111:679 [5] NCCL INFO Channel 26/1 : 5[5] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:109:678 [3] NCCL INFO Channel 25/1 : 3[3] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:108:682 [2] NCCL INFO Channel 23/1 : 2[2] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:107:680 [1] NCCL INFO Channel 23/1 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:110:685 [4] NCCL INFO Channel 21/1 : 4[4] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:113:684 [7] NCCL INFO Channel 22/1 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:111:679 [5] NCCL INFO Channel 27/1 : 5[5] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:112:683 [6] NCCL INFO Channel 22/1 : 6[6] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:108:682 [2] NCCL INFO Channel 24/1 : 2[2] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:109:678 [3] NCCL INFO Channel 26/1 : 3[3] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:107:680 [1] NCCL INFO Channel 24/1 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:110:685 [4] NCCL INFO Channel 22/1 : 4[4] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:113:684 [7] NCCL INFO Channel 23/1 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:111:679 [5] NCCL INFO Channel 28/1 : 5[5] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:112:683 [6] NCCL INFO Channel 23/1 : 6[6] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:108:682 [2] NCCL INFO Channel 25/1 : 2[2] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:109:678 [3] NCCL INFO Channel 27/1 : 3[3] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:107:680 [1] NCCL INFO Channel 25/1 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:110:685 [4] NCCL INFO Channel 23/1 : 4[4] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:111:679 [5] NCCL INFO Channel 29/1 : 5[5] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:113:684 [7] NCCL INFO Channel 24/1 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:112:683 [6] NCCL INFO Channel 24/1 : 6[6] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:108:682 [2] NCCL INFO Channel 26/1 : 2[2] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:109:678 [3] NCCL INFO Channel 28/1 : 3[3] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:107:680 [1] NCCL INFO Channel 26/1 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:110:685 [4] NCCL INFO Channel 24/1 : 4[4] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:111:679 [5] NCCL INFO Channel 30/1 : 5[5] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:113:684 [7] NCCL INFO Channel 25/1 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:112:683 [6] NCCL INFO Channel 25/1 : 6[6] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:108:682 [2] NCCL INFO Channel 27/1 : 2[2] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:109:678 [3] NCCL INFO Channel 29/1 : 3[3] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:107:680 [1] NCCL INFO Channel 27/1 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:110:685 [4] NCCL INFO Channel 25/1 : 4[4] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:111:679 [5] NCCL INFO Channel 31/1 : 5[5] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:113:684 [7] NCCL INFO Channel 26/1 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:112:683 [6] NCCL INFO Channel 26/1 : 6[6] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:108:682 [2] NCCL INFO Channel 28/1 : 2[2] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:107:680 [1] NCCL INFO Channel 28/1 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:109:678 [3] NCCL INFO Channel 30/1 : 3[3] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:110:685 [4] NCCL INFO Channel 26/1 : 4[4] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:113:684 [7] NCCL INFO Channel 27/1 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:112:683 [6] NCCL INFO Channel 27/1 : 6[6] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:108:682 [2] NCCL INFO Channel 29/1 : 2[2] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:107:680 [1] NCCL INFO Channel 29/1 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:109:678 [3] NCCL INFO Channel 31/1 : 3[3] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:110:685 [4] NCCL INFO Channel 27/1 : 4[4] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:113:684 [7] NCCL INFO Channel 28/1 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:112:683 [6] NCCL INFO Channel 28/1 : 6[6] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:108:682 [2] NCCL INFO Channel 30/1 : 2[2] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:107:680 [1] NCCL INFO Channel 30/1 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:110:685 [4] NCCL INFO Channel 28/1 : 4[4] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:113:684 [7] NCCL INFO Channel 29/1 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:112:683 [6] NCCL INFO Channel 29/1 : 6[6] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:108:682 [2] NCCL INFO Channel 31/1 : 2[2] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:107:680 [1] NCCL INFO Channel 31/1 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:110:685 [4] NCCL INFO Channel 29/1 : 4[4] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:113:684 [7] NCCL INFO Channel 30/1 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:112:683 [6] NCCL INFO Channel 30/1 : 6[6] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:110:685 [4] NCCL INFO Channel 30/1 : 4[4] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:113:684 [7] NCCL INFO Channel 31/1 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:112:683 [6] NCCL INFO Channel 31/1 : 6[6] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:110:685 [4] NCCL INFO Channel 31/1 : 4[4] -> 0[0] via P2P/IPC/read | |
| INFO 2024-07-27 16:31:26,875 instructlab.model.backends.backends:336: Waiting for the vLLM server to start at http://127.0.0.1:37393/v1, this might take a moment... Attempt: 10/80 | |
| INFO 2024-07-27 16:31:31,437 instructlab.model.backends.backends:336: Waiting for the vLLM server to start at http://127.0.0.1:37393/v1, this might take a moment... Attempt: 11/80 | |
| INFO 2024-07-27 16:31:36,046 instructlab.model.backends.backends:336: Waiting for the vLLM server to start at http://127.0.0.1:37393/v1, this might take a moment... Attempt: 12/80 | |
| INFO 2024-07-27 16:31:40,539 instructlab.model.backends.backends:336: Waiting for the vLLM server to start at http://127.0.0.1:37393/v1, this might take a moment... Attempt: 13/80 | |
| INFO 2024-07-27 16:31:44,872 instructlab.model.backends.backends:336: Waiting for the vLLM server to start at http://127.0.0.1:37393/v1, this might take a moment... Attempt: 14/80 | |
| embedding_mode is False. Embedding API will not work. | |
| INFO: Started server process [84] | |
| INFO: Waiting for application startup. | |
| INFO: Application startup complete. | |
| INFO: Uvicorn running on http://127.0.0.1:37393 (Press CTRL+C to quit) | |
| INFO: 127.0.0.1:45378 - "GET /v1/models HTTP/1.1" 200 OK | |
| INFO 2024-07-27 16:31:47,368 instructlab.model.backends.backends:343: vLLM engine successfully started at http://127.0.0.1:37393/v1 | |
| 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 147/147 [00:00<00:00, 157.55it/s] | |
| 0%| | 0/413 [00:00<?, ?it/s]INFO: 127.0.0.1:45440 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45434 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 0%|▉ | 2/413 [00:00<00:37, 10.89it/s]INFO: 127.0.0.1:45428 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45426 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45418 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 1%|██▍ | 5/413 [00:00<00:22, 18.53it/s]INFO: 127.0.0.1:45514 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45440 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45498 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 2%|███▉ | 8/413 [00:00<00:19, 20.76it/s]INFO: 127.0.0.1:45456 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45422 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45468 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45484 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45402 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45388 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 3%|██████▊ | 14/413 [00:00<00:16, 24.79it/s]INFO: 127.0.0.1:45434 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45418 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45428 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45514 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 4%|████████▏ | 17/413 [00:00<00:18, 21.26it/s]INFO: 127.0.0.1:45440 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45498 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45434 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45484 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45388 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45468 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 6%|███████████▌ | 24/413 [00:00<00:13, 29.36it/s]INFO: 127.0.0.1:45480 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45422 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45434 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45402 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 7%|█████████████▌ | 28/413 [00:01<00:14, 26.71it/s]INFO: 127.0.0.1:45514 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45498 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45468 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 8%|███████████████ | 31/413 [00:01<00:15, 23.88it/s]INFO: 127.0.0.1:45440 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45484 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45480 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45514 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45426 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45402 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 9%|█████████████████▍ | 36/413 [00:01<00:14, 26.76it/s]INFO: 127.0.0.1:45456 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45428 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45498 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 10%|███████████████████▎ | 40/413 [00:01<00:13, 27.06it/s]INFO: 127.0.0.1:45434 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45400 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45422 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 10%|████████████████████▊ | 43/413 [00:01<00:14, 24.96it/s]INFO: 127.0.0.1:45388 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45402 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45418 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 11%|██████████████████████▎ | 46/413 [00:01<00:15, 23.24it/s]INFO: 127.0.0.1:45484 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45468 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45426 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45440 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 12%|████████████████████████▏ | 50/413 [00:02<00:15, 23.43it/s]INFO: 127.0.0.1:45514 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45380 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45428 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45422 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 13%|██████████████████████████▏ | 54/413 [00:02<00:14, 25.41it/s]INFO: 127.0.0.1:45456 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45480 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45402 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 14%|███████████████████████████▌ | 57/413 [00:02<00:13, 25.74it/s]INFO: 127.0.0.1:45426 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45388 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45418 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 15%|█████████████████████████████ | 60/413 [00:02<00:18, 19.40it/s]INFO: 127.0.0.1:45400 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45380 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45426 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 15%|██████████████████████████████▌ | 63/413 [00:02<00:16, 21.21it/s]INFO: 127.0.0.1:45468 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45388 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45480 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 16%|███████████████████████████████▉ | 66/413 [00:02<00:17, 19.54it/s]INFO: 127.0.0.1:45514 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45434 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45484 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 17%|█████████████████████████████████▍ | 69/413 [00:02<00:16, 21.42it/s]INFO: 127.0.0.1:45402 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45400 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45426 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 17%|██████████████████████████████████▊ | 72/413 [00:03<00:16, 20.68it/s]INFO: 127.0.0.1:45440 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45418 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45456 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 18%|████████████████████████████████████▎ | 75/413 [00:03<00:15, 21.14it/s]INFO: 127.0.0.1:45428 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45388 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45400 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45456 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 19%|█████████████████████████████████████▊ | 78/413 [00:03<00:19, 16.90it/s]INFO: 127.0.0.1:45422 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45498 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45380 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45418 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 20%|███████████████████████████████████████▋ | 82/413 [00:03<00:16, 20.27it/s]INFO: 127.0.0.1:45480 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45434 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 21%|█████████████████████████████████████████▏ | 85/413 [00:03<00:14, 22.18it/s]INFO: 127.0.0.1:45402 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45426 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45388 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45422 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 21%|██████████████████████████████████████████▌ | 88/413 [00:03<00:16, 20.20it/s]INFO: 127.0.0.1:45468 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45480 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45428 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 22%|████████████████████████████████████████████▌ | 92/413 [00:04<00:14, 22.08it/s]INFO: 127.0.0.1:45484 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45514 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45418 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 23%|██████████████████████████████████████████████ | 95/413 [00:04<00:13, 22.80it/s]INFO: 127.0.0.1:45380 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45402 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45434 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 24%|███████████████████████████████████████████████▍ | 98/413 [00:04<00:13, 22.57it/s]INFO: 127.0.0.1:45440 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45456 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45418 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 24%|████████████████████████████████████████████████▋ | 101/413 [00:04<00:12, 24.06it/s]INFO: 127.0.0.1:45498 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45426 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45388 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45480 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45380 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 26%|███████████████████████████████████████████████████ | 106/413 [00:04<00:10, 30.35it/s]INFO: 127.0.0.1:45468 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45400 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45428 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45484 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 27%|█████████████████████████████████████████████████████ | 110/413 [00:04<00:11, 25.97it/s]INFO: 127.0.0.1:45388 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45514 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45498 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 27%|██████████████████████████████████████████████████████▍ | 113/413 [00:04<00:12, 23.62it/s]INFO: 127.0.0.1:45440 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45418 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45400 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 28%|███████████████████████████████████████████████████████▉ | 116/413 [00:05<00:12, 23.20it/s]INFO: 127.0.0.1:45456 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45480 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45434 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45468 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45402 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 29%|██████████████████████████████████████████████████████████▎ | 121/413 [00:05<00:11, 26.26it/s]INFO: 127.0.0.1:45422 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45428 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45388 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45514 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 30%|████████████████████████████████████████████████████████████▏ | 125/413 [00:05<00:10, 26.43it/s]INFO: 127.0.0.1:45400 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45418 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45484 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 31%|█████████████████████████████████████████████████████████████▋ | 128/413 [00:05<00:10, 26.08it/s]INFO: 127.0.0.1:45456 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45514 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45402 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 32%|███████████████████████████████████████████████████████████████ | 131/413 [00:05<00:11, 24.81it/s]INFO: 127.0.0.1:45434 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45422 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45440 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 32%|████████████████████████████████████████████████████████████████▌ | 134/413 [00:05<00:11, 24.32it/s]INFO: 127.0.0.1:45428 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45484 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45422 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 33%|██████████████████████████████████████████████████████████████████ | 137/413 [00:05<00:11, 24.69it/s]INFO: 127.0.0.1:45426 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45400 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45468 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 34%|███████████████████████████████████████████████████████████████████▍ | 140/413 [00:06<00:11, 23.37it/s]INFO: 127.0.0.1:45434 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45484 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45422 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45434 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 35%|████████████████████████████████████████████████████████████████████▉ | 143/413 [00:06<00:17, 15.23it/s]INFO: 127.0.0.1:45514 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45440 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45388 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 36%|██████████████████████████████████████████████████████████████████████▊ | 147/413 [00:06<00:13, 19.14it/s]INFO: 127.0.0.1:45402 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45480 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45426 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45498 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 37%|████████████████████████████████████████████████████████████████████████▊ | 151/413 [00:06<00:12, 21.07it/s]INFO: 127.0.0.1:45418 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45468 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45484 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 37%|██████████████████████████████████████████████████████████████████████████▏ | 154/413 [00:06<00:11, 22.04it/s]INFO: 127.0.0.1:45400 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45456 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45428 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45514 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45388 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 38%|████████████████████████████████████████████████████████████████████████████▌ | 159/413 [00:06<00:10, 23.62it/s]INFO: 127.0.0.1:45380 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45434 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45480 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 39%|██████████████████████████████████████████████████████████████████████████████ | 162/413 [00:07<00:10, 24.41it/s]INFO: 127.0.0.1:45422 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45402 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45468 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45498 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45418 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 40%|███████████████████████████████████████████████████████████████████████████████▉ | 166/413 [00:07<00:08, 27.47it/s]INFO: 127.0.0.1:45514 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45440 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 41%|█████████████████████████████████████████████████████████████████████████████████▍ | 169/413 [00:07<00:09, 26.34it/s]INFO: 127.0.0.1:45456 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45484 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45480 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 42%|██████████████████████████████████████████████████████████████████████████████████▉ | 172/413 [00:07<00:10, 22.42it/s]INFO: 127.0.0.1:45434 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45400 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45402 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45380 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 43%|████████████████████████████████████████████████████████████████████████████████████▊ | 176/413 [00:07<00:09, 24.72it/s]INFO: 127.0.0.1:45468 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45422 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45440 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45484 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45428 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 44%|███████████████████████████████████████████████████████████████████████████████████████▏ | 181/413 [00:07<00:08, 27.25it/s]INFO: 127.0.0.1:45480 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45402 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45426 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 45%|████████████████████████████████████████████████████████████████████████████████████████▋ | 184/413 [00:07<00:08, 27.17it/s]INFO: 127.0.0.1:45456 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45468 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45418 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45514 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45380 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45484 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 45%|██████████████████████████████████████████████████████████████████████████████████████████ | 187/413 [00:08<00:10, 21.98it/s]INFO: 127.0.0.1:45400 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45480 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45388 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45422 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 47%|████████████████████████████████████████████████████████████████████████████████████████████▉ | 193/413 [00:08<00:07, 28.64it/s]INFO: 127.0.0.1:45440 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45434 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45456 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 48%|██████████████████████████████████████████████████████████████████████████████████████████████▉ | 197/413 [00:08<00:08, 26.12it/s]INFO: 127.0.0.1:45468 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45426 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45498 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 48%|████████████████████████████████████████████████████████████████████████████████████████████████▎ | 200/413 [00:08<00:08, 25.49it/s]INFO: 127.0.0.1:45402 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45484 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45422 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 49%|█████████████████████████████████████████████████████████████████████████████████████████████████▊ | 203/413 [00:08<00:08, 25.86it/s]INFO: 127.0.0.1:45480 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45388 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45428 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 50%|███████████████████████████████████████████████████████████████████████████████████████████████████▎ | 206/413 [00:08<00:08, 23.30it/s]INFO: 127.0.0.1:45468 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45402 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45514 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 51%|████████████████████████████████████████████████████████████████████████████████████████████████████▋ | 209/413 [00:08<00:08, 23.67it/s]INFO: 127.0.0.1:45380 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45426 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45456 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 51%|██████████████████████████████████████████████████████████████████████████████████████████████████████▏ | 212/413 [00:09<00:08, 24.61it/s]INFO: 127.0.0.1:45422 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45498 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45484 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45428 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 52%|███████████████████████████████████████████████████████████████████████████████████████████████████████▌ | 215/413 [00:09<00:09, 21.34it/s]INFO: 127.0.0.1:45434 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45388 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45426 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45440 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45400 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 54%|██████████████████████████████████████████████████████████████████████████████████████████████████████████▍ | 221/413 [00:09<00:07, 25.56it/s]INFO: 127.0.0.1:45514 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45422 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45380 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45498 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 54%|████████████████████████████████████████████████████████████████████████████████████████████████████████████▍ | 225/413 [00:09<00:06, 28.34it/s]INFO: 127.0.0.1:45468 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45418 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45456 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45388 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 55%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████▊ | 228/413 [00:09<00:07, 24.41it/s]INFO: 127.0.0.1:45484 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45380 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 56%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████▎ | 231/413 [00:09<00:07, 23.35it/s]INFO: 127.0.0.1:45426 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45440 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45428 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45434 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 57%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████▊ | 234/413 [00:09<00:08, 22.21it/s]INFO: 127.0.0.1:45388 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45484 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45400 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 58%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████▋ | 238/413 [00:10<00:07, 24.29it/s]INFO: 127.0.0.1:45456 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45514 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45440 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45428 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 58%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████ | 241/413 [00:10<00:07, 24.52it/s]INFO: 127.0.0.1:45422 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45418 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45426 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45400 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 60%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▌ | 246/413 [00:10<00:06, 27.28it/s]INFO: 127.0.0.1:45388 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45380 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45498 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45468 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 60%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▉ | 249/413 [00:10<00:06, 26.67it/s]INFO: 127.0.0.1:45426 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45428 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 61%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▍ | 252/413 [00:10<00:06, 25.70it/s]INFO: 127.0.0.1:45422 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45498 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45484 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 62%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▊ | 255/413 [00:10<00:07, 21.19it/s]INFO: 127.0.0.1:45400 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45418 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45480 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 62%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▎ | 258/413 [00:10<00:06, 22.49it/s]INFO: 127.0.0.1:45426 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45456 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45468 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 63%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▊ | 261/413 [00:11<00:06, 23.37it/s]INFO: 127.0.0.1:45428 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45434 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45400 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 64%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▏ | 264/413 [00:11<00:06, 24.08it/s]INFO: 127.0.0.1:45514 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45498 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45422 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 65%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▋ | 267/413 [00:11<00:06, 22.92it/s]INFO: 127.0.0.1:45418 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45440 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45480 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 65%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████ | 270/413 [00:11<00:05, 24.04it/s]INFO: 127.0.0.1:45380 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45428 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45434 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45484 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 66%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████ | 274/413 [00:11<00:05, 23.98it/s]INFO: 127.0.0.1:45426 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45400 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45498 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 67%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▍ | 277/413 [00:11<00:06, 22.34it/s]INFO: 127.0.0.1:45440 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45456 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45418 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 68%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▉ | 280/413 [00:11<00:06, 21.73it/s]INFO: 127.0.0.1:45468 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45480 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45514 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 69%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▎ | 283/413 [00:12<00:05, 22.09it/s]INFO: 127.0.0.1:45380 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45498 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45434 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45484 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 69%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▊ | 286/413 [00:12<00:05, 23.25it/s]INFO: 127.0.0.1:45422 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45400 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 70%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▎ | 289/413 [00:12<00:05, 22.97it/s]INFO: 127.0.0.1:45428 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45426 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45514 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45440 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45418 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45480 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 71%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▏ | 295/413 [00:12<00:03, 31.75it/s]INFO: 127.0.0.1:45484 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45468 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45380 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45456 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 72%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████ | 299/413 [00:12<00:03, 29.59it/s]INFO: 127.0.0.1:45480 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45400 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45428 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45380 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 73%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▉ | 303/413 [00:12<00:04, 25.18it/s]INFO: 127.0.0.1:45426 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45400 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45388 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 74%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▍ | 306/413 [00:12<00:04, 24.82it/s]INFO: 127.0.0.1:45480 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45440 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45434 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45468 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45514 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 75%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▊ | 311/413 [00:13<00:03, 27.18it/s]INFO: 127.0.0.1:45418 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45456 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45498 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 76%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▎ | 314/413 [00:13<00:03, 26.36it/s]INFO: 127.0.0.1:45428 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45440 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45468 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45426 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 77%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▏ | 318/413 [00:13<00:03, 26.43it/s]INFO: 127.0.0.1:45388 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45480 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45484 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45514 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45498 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45434 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 78%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▏ | 322/413 [00:13<00:03, 26.36it/s]INFO: 127.0.0.1:45468 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 79%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▌ | 325/413 [00:13<00:03, 25.17it/s]INFO: 127.0.0.1:45400 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45440 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45388 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 79%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████ | 328/413 [00:13<00:03, 23.24it/s]INFO: 127.0.0.1:45380 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45456 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45418 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45428 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45456 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 81%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▍ | 333/413 [00:13<00:03, 26.57it/s]INFO: 127.0.0.1:45434 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45440 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45484 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 81%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▉ | 336/413 [00:14<00:03, 23.47it/s]INFO: 127.0.0.1:45400 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45380 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45428 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45456 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 82%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▎ | 339/413 [00:14<00:03, 21.46it/s]INFO: 127.0.0.1:45426 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45418 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45468 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45388 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45498 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45440 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 84%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▏ | 345/413 [00:14<00:02, 23.69it/s]INFO: 127.0.0.1:45434 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45400 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45380 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 85%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▏ | 349/413 [00:14<00:02, 24.00it/s]INFO: 127.0.0.1:45426 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45468 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45480 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45426 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 85%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████ | 353/413 [00:14<00:02, 22.16it/s]INFO: 127.0.0.1:45434 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45480 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45380 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45422 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45400 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 86%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████ | 357/413 [00:15<00:02, 22.39it/s]INFO: 127.0.0.1:45428 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45484 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 87%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▍ | 360/413 [00:15<00:02, 22.19it/s]INFO: 127.0.0.1:45456 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45388 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45440 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45426 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 88%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▉ | 363/413 [00:15<00:02, 23.50it/s]INFO: 127.0.0.1:45434 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45468 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45400 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45418 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 89%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▎ | 368/413 [00:15<00:01, 23.72it/s]INFO: 127.0.0.1:45480 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45440 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45400 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45388 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 90%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▏ | 372/413 [00:15<00:01, 23.98it/s]INFO: 127.0.0.1:45468 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45426 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45498 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45428 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 91%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▋ | 375/413 [00:15<00:01, 23.17it/s]INFO: 127.0.0.1:45480 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45422 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 92%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▏ | 378/413 [00:15<00:01, 22.40it/s]INFO: 127.0.0.1:45388 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45428 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45426 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 92%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▌ | 381/413 [00:16<00:01, 23.52it/s]INFO: 127.0.0.1:45480 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45400 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45422 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 93%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████ | 384/413 [00:16<00:01, 22.33it/s]INFO: 127.0.0.1:45484 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45498 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45388 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45428 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 94%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▉ | 388/413 [00:16<00:00, 26.21it/s]INFO: 127.0.0.1:45468 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45480 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45456 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45484 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 95%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▉ | 392/413 [00:16<00:00, 25.45it/s]INFO: 127.0.0.1:45418 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45514 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45428 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45440 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45498 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 96%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▎ | 397/413 [00:16<00:00, 28.90it/s]INFO: 127.0.0.1:45380 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45426 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45480 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45422 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 97%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▏ | 401/413 [00:16<00:00, 30.26it/s]INFO: 127.0.0.1:45484 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45418 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45434 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45456 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45440 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45514 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 99%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████ | 407/413 [00:16<00:00, 36.73it/s]INFO: 127.0.0.1:45400 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45388 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45498 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45468 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:45428 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▌| 412/413 [00:16<00:00, 37.18it/s]INFO: 127.0.0.1:45402 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 413/413 [00:20<00:00, 20.24it/s] | |
| INFO: Shutting down | |
| INFO: Waiting for application shutdown. | |
| INFO: Application shutdown complete. | |
| INFO: Finished server process [84] | |
| [rank0]:[W CudaIPCTypes.cpp:16] Producer process has been terminated before all shared CUDA tensors released. See Note [Sharing CUDA tensors] | |
| /usr/lib64/python3.11/multiprocessing/resource_tracker.py:254: UserWarning: resource_tracker: There appear to be 1 leaked shared_memory objects to clean up at shutdown | |
| warnings.warn('resource_tracker: There appear to be %d ' | |
| Generating questions and reference answers from qna files for branch HEAD... | |
| WARNING 2024-07-27 16:32:21,241 instructlab.model.evaluate:288: Based on your hardware configuration, when using vLLM, we recommend setting max-workers between 53.0 and 160.0 for optimal performance | |
| INFO 2024-07-27 16:32:21,311 instructlab.model.backends.backends:315: Trying to connect to model server at http://127.0.0.1:8000/v1 | |
| INFO 2024-07-27 16:32:23,996 instructlab.model.backends.vllm:196: vLLM starting up on pid 804 at http://127.0.0.1:44547/v1 | |
| INFO 2024-07-27 16:32:23,996 instructlab.model.backends.backends:328: Starting a temporary vLLM server at http://127.0.0.1:44547/v1 | |
| INFO 2024-07-27 16:32:23,996 instructlab.model.backends.backends:336: Waiting for the vLLM server to start at http://127.0.0.1:44547/v1, this might take a moment... Attempt: 1/80 | |
| You are using the default legacy behaviour of the <class 'transformers.models.llama.tokenization_llama_fast.LlamaTokenizerFast'>. This is expected, and simply means that the `legacy` (previous) behavior will be used so nothing changes for you. If you want to use the new behaviour, set `legacy=False`. This should only be set if you understand what it means, and thoroughly read the reason why this was added as explained in https://github.com/huggingface/transformers/pull/24565 - if you loaded a llama tokenizer from a GGUF file you can ignore this message. | |
| INFO 2024-07-27 16:32:28,780 instructlab.model.backends.backends:336: Waiting for the vLLM server to start at http://127.0.0.1:44547/v1, this might take a moment... Attempt: 2/80 | |
| INFO 2024-07-27 16:32:33,728 instructlab.model.backends.backends:336: Waiting for the vLLM server to start at http://127.0.0.1:44547/v1, this might take a moment... Attempt: 3/80 | |
| INFO 2024-07-27 16:32:38,403 instructlab.model.backends.backends:336: Waiting for the vLLM server to start at http://127.0.0.1:44547/v1, this might take a moment... Attempt: 4/80 | |
| tyler-rhel-newimage:804:804 [0] NCCL INFO Bootstrap : Using enp8s0:192.168.48.11<0> | |
| tyler-rhel-newimage:804:804 [0] NCCL INFO NET/Plugin : dlerror=libnccl-net.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net.so), using internal implementation | |
| tyler-rhel-newimage:804:804 [0] NCCL INFO cudaDriverVersion 12040 | |
| NCCL version 2.20.5+cuda12.4 | |
| tyler-rhel-newimage:832:832 [6] NCCL INFO cudaDriverVersion 12040 | |
| tyler-rhel-newimage:832:832 [6] NCCL INFO Bootstrap : Using enp8s0:192.168.48.11<0> | |
| tyler-rhel-newimage:832:832 [6] NCCL INFO NET/Plugin : dlerror=libnccl-net.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net.so), using internal implementation | |
| tyler-rhel-newimage:827:827 [1] NCCL INFO cudaDriverVersion 12040 | |
| tyler-rhel-newimage:827:827 [1] NCCL INFO Bootstrap : Using enp8s0:192.168.48.11<0> | |
| tyler-rhel-newimage:827:827 [1] NCCL INFO NET/Plugin : dlerror=libnccl-net.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net.so), using internal implementation | |
| tyler-rhel-newimage:828:828 [2] NCCL INFO cudaDriverVersion 12040 | |
| tyler-rhel-newimage:828:828 [2] NCCL INFO Bootstrap : Using enp8s0:192.168.48.11<0> | |
| tyler-rhel-newimage:828:828 [2] NCCL INFO NET/Plugin : dlerror=libnccl-net.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net.so), using internal implementation | |
| tyler-rhel-newimage:833:833 [7] NCCL INFO cudaDriverVersion 12040 | |
| tyler-rhel-newimage:833:833 [7] NCCL INFO Bootstrap : Using enp8s0:192.168.48.11<0> | |
| tyler-rhel-newimage:833:833 [7] NCCL INFO NET/Plugin : dlerror=libnccl-net.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net.so), using internal implementation | |
| tyler-rhel-newimage:831:831 [5] NCCL INFO cudaDriverVersion 12040 | |
| tyler-rhel-newimage:831:831 [5] NCCL INFO Bootstrap : Using enp8s0:192.168.48.11<0> | |
| tyler-rhel-newimage:831:831 [5] NCCL INFO NET/Plugin : dlerror=libnccl-net.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net.so), using internal implementation | |
| tyler-rhel-newimage:830:830 [4] NCCL INFO cudaDriverVersion 12040 | |
| tyler-rhel-newimage:830:830 [4] NCCL INFO Bootstrap : Using enp8s0:192.168.48.11<0> | |
| tyler-rhel-newimage:829:829 [3] NCCL INFO cudaDriverVersion 12040 | |
| tyler-rhel-newimage:829:829 [3] NCCL INFO Bootstrap : Using enp8s0:192.168.48.11<0> | |
| tyler-rhel-newimage:830:830 [4] NCCL INFO NET/Plugin : dlerror=libnccl-net.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net.so), using internal implementation | |
| tyler-rhel-newimage:829:829 [3] NCCL INFO NET/Plugin : dlerror=libnccl-net.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net.so), using internal implementation | |
| tyler-rhel-newimage:804:804 [0] NCCL INFO NET/IB : No device found. | |
| tyler-rhel-newimage:804:804 [0] NCCL INFO NET/Socket : Using [0]enp8s0:192.168.48.11<0> | |
| tyler-rhel-newimage:804:804 [0] NCCL INFO Using non-device net plugin version 0 | |
| tyler-rhel-newimage:804:804 [0] NCCL INFO Using network Socket | |
| tyler-rhel-newimage:832:832 [6] NCCL INFO NET/IB : No device found. | |
| tyler-rhel-newimage:832:832 [6] NCCL INFO NET/Socket : Using [0]enp8s0:192.168.48.11<0> | |
| tyler-rhel-newimage:832:832 [6] NCCL INFO Using non-device net plugin version 0 | |
| tyler-rhel-newimage:832:832 [6] NCCL INFO Using network Socket | |
| tyler-rhel-newimage:827:827 [1] NCCL INFO NET/IB : No device found. | |
| tyler-rhel-newimage:827:827 [1] NCCL INFO NET/Socket : Using [0]enp8s0:192.168.48.11<0> | |
| tyler-rhel-newimage:827:827 [1] NCCL INFO Using non-device net plugin version 0 | |
| tyler-rhel-newimage:827:827 [1] NCCL INFO Using network Socket | |
| tyler-rhel-newimage:831:831 [5] NCCL INFO NET/IB : No device found. | |
| tyler-rhel-newimage:828:828 [2] NCCL INFO NET/IB : No device found. | |
| tyler-rhel-newimage:828:828 [2] NCCL INFO NET/Socket : Using [0]enp8s0:192.168.48.11<0> | |
| tyler-rhel-newimage:831:831 [5] NCCL INFO NET/Socket : Using [0]enp8s0:192.168.48.11<0> | |
| tyler-rhel-newimage:831:831 [5] NCCL INFO Using non-device net plugin version 0 | |
| tyler-rhel-newimage:831:831 [5] NCCL INFO Using network Socket | |
| tyler-rhel-newimage:828:828 [2] NCCL INFO Using non-device net plugin version 0 | |
| tyler-rhel-newimage:828:828 [2] NCCL INFO Using network Socket | |
| tyler-rhel-newimage:829:829 [3] NCCL INFO NET/IB : No device found. | |
| tyler-rhel-newimage:829:829 [3] NCCL INFO NET/Socket : Using [0]enp8s0:192.168.48.11<0> | |
| tyler-rhel-newimage:829:829 [3] NCCL INFO Using non-device net plugin version 0 | |
| tyler-rhel-newimage:829:829 [3] NCCL INFO Using network Socket | |
| tyler-rhel-newimage:833:833 [7] NCCL INFO NET/IB : No device found. | |
| tyler-rhel-newimage:833:833 [7] NCCL INFO NET/Socket : Using [0]enp8s0:192.168.48.11<0> | |
| tyler-rhel-newimage:833:833 [7] NCCL INFO Using non-device net plugin version 0 | |
| tyler-rhel-newimage:833:833 [7] NCCL INFO Using network Socket | |
| tyler-rhel-newimage:830:830 [4] NCCL INFO NET/IB : No device found. | |
| tyler-rhel-newimage:830:830 [4] NCCL INFO NET/Socket : Using [0]enp8s0:192.168.48.11<0> | |
| tyler-rhel-newimage:830:830 [4] NCCL INFO Using non-device net plugin version 0 | |
| tyler-rhel-newimage:830:830 [4] NCCL INFO Using network Socket | |
| tyler-rhel-newimage:829:829 [3] NCCL INFO comm 0x556dc48048f0 rank 3 nranks 8 cudaDev 3 nvmlDev 3 busId a040 commId 0x7ee526e6cd90ff03 - Init START | |
| tyler-rhel-newimage:828:828 [2] NCCL INFO comm 0x556dc48044c0 rank 2 nranks 8 cudaDev 2 nvmlDev 2 busId a030 commId 0x7ee526e6cd90ff03 - Init START | |
| tyler-rhel-newimage:832:832 [6] NCCL INFO comm 0x556dc4805e20 rank 6 nranks 8 cudaDev 6 nvmlDev 6 busId e070 commId 0x7ee526e6cd90ff03 - Init START | |
| tyler-rhel-newimage:830:830 [4] NCCL INFO comm 0x556dc4806330 rank 4 nranks 8 cudaDev 4 nvmlDev 4 busId c050 commId 0x7ee526e6cd90ff03 - Init START | |
| tyler-rhel-newimage:831:831 [5] NCCL INFO comm 0x556dc4806b40 rank 5 nranks 8 cudaDev 5 nvmlDev 5 busId c060 commId 0x7ee526e6cd90ff03 - Init START | |
| tyler-rhel-newimage:804:804 [0] NCCL INFO comm 0x556dc480be40 rank 0 nranks 8 cudaDev 0 nvmlDev 0 busId 8010 commId 0x7ee526e6cd90ff03 - Init START | |
| tyler-rhel-newimage:827:827 [1] NCCL INFO comm 0x556dc48056c0 rank 1 nranks 8 cudaDev 1 nvmlDev 1 busId 8020 commId 0x7ee526e6cd90ff03 - Init START | |
| tyler-rhel-newimage:833:833 [7] NCCL INFO comm 0x556dc4806ea0 rank 7 nranks 8 cudaDev 7 nvmlDev 7 busId e080 commId 0x7ee526e6cd90ff03 - Init START | |
| INFO 2024-07-27 16:32:43,189 instructlab.model.backends.backends:336: Waiting for the vLLM server to start at http://127.0.0.1:44547/v1, this might take a moment... Attempt: 5/80 | |
| tyler-rhel-newimage:827:827 [1] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| tyler-rhel-newimage:832:832 [6] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| tyler-rhel-newimage:828:828 [2] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| tyler-rhel-newimage:804:804 [0] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| tyler-rhel-newimage:830:830 [4] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| tyler-rhel-newimage:833:833 [7] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| tyler-rhel-newimage:831:831 [5] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| tyler-rhel-newimage:830:830 [4] NCCL INFO Setting affinity for GPU 4 to ffff,ffffff00,00000000 | |
| tyler-rhel-newimage:830:830 [4] NCCL INFO NVLS multicast support is not available on dev 4 | |
| tyler-rhel-newimage:827:827 [1] NCCL INFO Setting affinity for GPU 1 to ff,ffffffff | |
| tyler-rhel-newimage:827:827 [1] NCCL INFO NVLS multicast support is not available on dev 1 | |
| tyler-rhel-newimage:832:832 [6] NCCL INFO Setting affinity for GPU 6 to ffff,ffffff00,00000000 | |
| tyler-rhel-newimage:832:832 [6] NCCL INFO NVLS multicast support is not available on dev 6 | |
| tyler-rhel-newimage:804:804 [0] NCCL INFO Setting affinity for GPU 0 to ff,ffffffff | |
| tyler-rhel-newimage:828:828 [2] NCCL INFO Setting affinity for GPU 2 to ff,ffffffff | |
| tyler-rhel-newimage:829:829 [3] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| tyler-rhel-newimage:828:828 [2] NCCL INFO NVLS multicast support is not available on dev 2 | |
| tyler-rhel-newimage:804:804 [0] NCCL INFO NVLS multicast support is not available on dev 0 | |
| tyler-rhel-newimage:833:833 [7] NCCL INFO Setting affinity for GPU 7 to ffff,ffffff00,00000000 | |
| tyler-rhel-newimage:833:833 [7] NCCL INFO NVLS multicast support is not available on dev 7 | |
| tyler-rhel-newimage:831:831 [5] NCCL INFO Setting affinity for GPU 5 to ffff,ffffff00,00000000 | |
| tyler-rhel-newimage:831:831 [5] NCCL INFO NVLS multicast support is not available on dev 5 | |
| tyler-rhel-newimage:829:829 [3] NCCL INFO Setting affinity for GPU 3 to ff,ffffffff | |
| tyler-rhel-newimage:829:829 [3] NCCL INFO NVLS multicast support is not available on dev 3 | |
| tyler-rhel-newimage:829:829 [3] NCCL INFO comm 0x556dc48048f0 rank 3 nRanks 8 nNodes 1 localRanks 8 localRank 3 MNNVL 0 | |
| tyler-rhel-newimage:832:832 [6] NCCL INFO comm 0x556dc4805e20 rank 6 nRanks 8 nNodes 1 localRanks 8 localRank 6 MNNVL 0 | |
| tyler-rhel-newimage:833:833 [7] NCCL INFO comm 0x556dc4806ea0 rank 7 nRanks 8 nNodes 1 localRanks 8 localRank 7 MNNVL 0 | |
| tyler-rhel-newimage:828:828 [2] NCCL INFO comm 0x556dc48044c0 rank 2 nRanks 8 nNodes 1 localRanks 8 localRank 2 MNNVL 0 | |
| tyler-rhel-newimage:830:830 [4] NCCL INFO comm 0x556dc4806330 rank 4 nRanks 8 nNodes 1 localRanks 8 localRank 4 MNNVL 0 | |
| tyler-rhel-newimage:831:831 [5] NCCL INFO comm 0x556dc4806b40 rank 5 nRanks 8 nNodes 1 localRanks 8 localRank 5 MNNVL 0 | |
| tyler-rhel-newimage:804:804 [0] NCCL INFO comm 0x556dc480be40 rank 0 nRanks 8 nNodes 1 localRanks 8 localRank 0 MNNVL 0 | |
| tyler-rhel-newimage:829:829 [3] NCCL INFO Trees [0] 4/-1/-1->3->2 [1] 4/-1/-1->3->2 [2] 4/-1/-1->3->2 [3] 4/-1/-1->3->2 [4] 4/-1/-1->3->2 [5] 4/-1/-1->3->2 [6] 4/-1/-1->3->2 [7] 4/-1/-1->3->2 [8] 4/-1/-1->3->2 [9] 4/-1/-1->3->2 [10] 4/-1/-1->3->2 [11] 4/-1/-1->3->2 [12] 4/-1/-1->3->2 [13] 4/-1/-1->3->2 [14] 4/-1/-1->3->2 [15] 4/-1/-1->3->2 [16] 4/-1/-1->3->2 [17] 4/-1/-1->3->2 [18] 4/-1/-1->3->2 [19] 4/-1/-1->3->2 [20] 4/-1/-1->3->2 [21] 4/-1/-1->3->2 [22] 4/-1/-1->3->2 [23] 4/-1/-1->3->2 | |
| tyler-rhel-newimage:827:827 [1] NCCL INFO comm 0x556dc48056c0 rank 1 nRanks 8 nNodes 1 localRanks 8 localRank 1 MNNVL 0 | |
| tyler-rhel-newimage:832:832 [6] NCCL INFO Trees [0] 7/-1/-1->6->5 [1] 7/-1/-1->6->5 [2] 7/-1/-1->6->5 [3] 7/-1/-1->6->5 [4] 7/-1/-1->6->5 [5] 7/-1/-1->6->5 [6] 7/-1/-1->6->5 [7] 7/-1/-1->6->5 [8] 7/-1/-1->6->5 [9] 7/-1/-1->6->5 [10] 7/-1/-1->6->5 [11] 7/-1/-1->6->5 [12] 7/-1/-1->6->5 [13] 7/-1/-1->6->5 [14] 7/-1/-1->6->5 [15] 7/-1/-1->6->5 [16] 7/-1/-1->6->5 [17] 7/-1/-1->6->5 [18] 7/-1/-1->6->5 [19] 7/-1/-1->6->5 [20] 7/-1/-1->6->5 [21] 7/-1/-1->6->5 [22] 7/-1/-1->6->5 [23] 7/-1/-1->6->5 | |
| tyler-rhel-newimage:829:829 [3] NCCL INFO P2P Chunksize set to 524288 | |
| tyler-rhel-newimage:833:833 [7] NCCL INFO Trees [0] -1/-1/-1->7->6 [1] -1/-1/-1->7->6 [2] -1/-1/-1->7->6 [3] -1/-1/-1->7->6 [4] -1/-1/-1->7->6 [5] -1/-1/-1->7->6 [6] -1/-1/-1->7->6 [7] -1/-1/-1->7->6 [8] -1/-1/-1->7->6 [9] -1/-1/-1->7->6 [10] -1/-1/-1->7->6 [11] -1/-1/-1->7->6 [12] -1/-1/-1->7->6 [13] -1/-1/-1->7->6 [14] -1/-1/-1->7->6 [15] -1/-1/-1->7->6 [16] -1/-1/-1->7->6 [17] -1/-1/-1->7->6 [18] -1/-1/-1->7->6 [19] -1/-1/-1->7->6 [20] -1/-1/-1->7->6 [21] -1/-1/-1->7->6 [22] -1/-1/-1->7->6 [23] -1/-1/-1->7->6 | |
| tyler-rhel-newimage:828:828 [2] NCCL INFO Trees [0] 3/-1/-1->2->1 [1] 3/-1/-1->2->1 [2] 3/-1/-1->2->1 [3] 3/-1/-1->2->1 [4] 3/-1/-1->2->1 [5] 3/-1/-1->2->1 [6] 3/-1/-1->2->1 [7] 3/-1/-1->2->1 [8] 3/-1/-1->2->1 [9] 3/-1/-1->2->1 [10] 3/-1/-1->2->1 [11] 3/-1/-1->2->1 [12] 3/-1/-1->2->1 [13] 3/-1/-1->2->1 [14] 3/-1/-1->2->1 [15] 3/-1/-1->2->1 [16] 3/-1/-1->2->1 [17] 3/-1/-1->2->1 [18] 3/-1/-1->2->1 [19] 3/-1/-1->2->1 [20] 3/-1/-1->2->1 [21] 3/-1/-1->2->1 [22] 3/-1/-1->2->1 [23] 3/-1/-1->2->1 | |
| tyler-rhel-newimage:832:832 [6] NCCL INFO P2P Chunksize set to 524288 | |
| tyler-rhel-newimage:830:830 [4] NCCL INFO Trees [0] 5/-1/-1->4->3 [1] 5/-1/-1->4->3 [2] 5/-1/-1->4->3 [3] 5/-1/-1->4->3 [4] 5/-1/-1->4->3 [5] 5/-1/-1->4->3 [6] 5/-1/-1->4->3 [7] 5/-1/-1->4->3 [8] 5/-1/-1->4->3 [9] 5/-1/-1->4->3 [10] 5/-1/-1->4->3 [11] 5/-1/-1->4->3 [12] 5/-1/-1->4->3 [13] 5/-1/-1->4->3 [14] 5/-1/-1->4->3 [15] 5/-1/-1->4->3 [16] 5/-1/-1->4->3 [17] 5/-1/-1->4->3 [18] 5/-1/-1->4->3 [19] 5/-1/-1->4->3 [20] 5/-1/-1->4->3 [21] 5/-1/-1->4->3 [22] 5/-1/-1->4->3 [23] 5/-1/-1->4->3 | |
| tyler-rhel-newimage:804:804 [0] NCCL INFO Channel 00/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:831:831 [5] NCCL INFO Trees [0] 6/-1/-1->5->4 [1] 6/-1/-1->5->4 [2] 6/-1/-1->5->4 [3] 6/-1/-1->5->4 [4] 6/-1/-1->5->4 [5] 6/-1/-1->5->4 [6] 6/-1/-1->5->4 [7] 6/-1/-1->5->4 [8] 6/-1/-1->5->4 [9] 6/-1/-1->5->4 [10] 6/-1/-1->5->4 [11] 6/-1/-1->5->4 [12] 6/-1/-1->5->4 [13] 6/-1/-1->5->4 [14] 6/-1/-1->5->4 [15] 6/-1/-1->5->4 [16] 6/-1/-1->5->4 [17] 6/-1/-1->5->4 [18] 6/-1/-1->5->4 [19] 6/-1/-1->5->4 [20] 6/-1/-1->5->4 [21] 6/-1/-1->5->4 [22] 6/-1/-1->5->4 [23] 6/-1/-1->5->4 | |
| tyler-rhel-newimage:833:833 [7] NCCL INFO P2P Chunksize set to 524288 | |
| tyler-rhel-newimage:828:828 [2] NCCL INFO P2P Chunksize set to 524288 | |
| tyler-rhel-newimage:831:831 [5] NCCL INFO P2P Chunksize set to 524288 | |
| tyler-rhel-newimage:827:827 [1] NCCL INFO Trees [0] 2/-1/-1->1->0 [1] 2/-1/-1->1->0 [2] 2/-1/-1->1->0 [3] 2/-1/-1->1->0 [4] 2/-1/-1->1->0 [5] 2/-1/-1->1->0 [6] 2/-1/-1->1->0 [7] 2/-1/-1->1->0 [8] 2/-1/-1->1->0 [9] 2/-1/-1->1->0 [10] 2/-1/-1->1->0 [11] 2/-1/-1->1->0 [12] 2/-1/-1->1->0 [13] 2/-1/-1->1->0 [14] 2/-1/-1->1->0 [15] 2/-1/-1->1->0 [16] 2/-1/-1->1->0 [17] 2/-1/-1->1->0 [18] 2/-1/-1->1->0 [19] 2/-1/-1->1->0 [20] 2/-1/-1->1->0 [21] 2/-1/-1->1->0 [22] 2/-1/-1->1->0 [23] 2/-1/-1->1->0 | |
| tyler-rhel-newimage:830:830 [4] NCCL INFO P2P Chunksize set to 524288 | |
| tyler-rhel-newimage:804:804 [0] NCCL INFO Channel 01/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:827:827 [1] NCCL INFO P2P Chunksize set to 524288 | |
| tyler-rhel-newimage:804:804 [0] NCCL INFO Channel 02/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:804:804 [0] NCCL INFO Channel 03/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:804:804 [0] NCCL INFO Channel 04/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:804:804 [0] NCCL INFO Channel 05/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:804:804 [0] NCCL INFO Channel 06/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:804:804 [0] NCCL INFO Channel 07/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:804:804 [0] NCCL INFO Channel 08/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:804:804 [0] NCCL INFO Channel 09/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:804:804 [0] NCCL INFO Channel 10/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:804:804 [0] NCCL INFO Channel 11/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:804:804 [0] NCCL INFO Channel 12/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:804:804 [0] NCCL INFO Channel 13/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:804:804 [0] NCCL INFO Channel 14/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:804:804 [0] NCCL INFO Channel 15/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:804:804 [0] NCCL INFO Channel 16/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:804:804 [0] NCCL INFO Channel 17/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:804:804 [0] NCCL INFO Channel 18/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:804:804 [0] NCCL INFO Channel 19/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:804:804 [0] NCCL INFO Channel 20/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:804:804 [0] NCCL INFO Channel 21/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:804:804 [0] NCCL INFO Channel 22/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:804:804 [0] NCCL INFO Channel 23/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:804:804 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1 [2] 1/-1/-1->0->-1 [3] 1/-1/-1->0->-1 [4] 1/-1/-1->0->-1 [5] 1/-1/-1->0->-1 [6] 1/-1/-1->0->-1 [7] 1/-1/-1->0->-1 [8] 1/-1/-1->0->-1 [9] 1/-1/-1->0->-1 [10] 1/-1/-1->0->-1 [11] 1/-1/-1->0->-1 [12] 1/-1/-1->0->-1 [13] 1/-1/-1->0->-1 [14] 1/-1/-1->0->-1 [15] 1/-1/-1->0->-1 [16] 1/-1/-1->0->-1 [17] 1/-1/-1->0->-1 [18] 1/-1/-1->0->-1 [19] 1/-1/-1->0->-1 [20] 1/-1/-1->0->-1 [21] 1/-1/-1->0->-1 [22] 1/-1/-1->0->-1 [23] 1/-1/-1->0->-1 | |
| tyler-rhel-newimage:804:804 [0] NCCL INFO P2P Chunksize set to 524288 | |
| tyler-rhel-newimage:829:829 [3] NCCL INFO Channel 00/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:831:831 [5] NCCL INFO Channel 00/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:833:833 [7] NCCL INFO Channel 00/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:832:832 [6] NCCL INFO Channel 00/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:828:828 [2] NCCL INFO Channel 00/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:829:829 [3] NCCL INFO Channel 01/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:831:831 [5] NCCL INFO Channel 01/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:804:804 [0] NCCL INFO Channel 00/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:833:833 [7] NCCL INFO Channel 01/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:832:832 [6] NCCL INFO Channel 01/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:828:828 [2] NCCL INFO Channel 01/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:829:829 [3] NCCL INFO Channel 02/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:831:831 [5] NCCL INFO Channel 02/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:833:833 [7] NCCL INFO Channel 02/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:832:832 [6] NCCL INFO Channel 02/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:804:804 [0] NCCL INFO Channel 01/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:828:828 [2] NCCL INFO Channel 02/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:829:829 [3] NCCL INFO Channel 03/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:831:831 [5] NCCL INFO Channel 03/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:833:833 [7] NCCL INFO Channel 03/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:832:832 [6] NCCL INFO Channel 03/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:804:804 [0] NCCL INFO Channel 02/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:828:828 [2] NCCL INFO Channel 03/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:829:829 [3] NCCL INFO Channel 04/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:831:831 [5] NCCL INFO Channel 04/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:833:833 [7] NCCL INFO Channel 04/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:827:827 [1] NCCL INFO Channel 00/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:832:832 [6] NCCL INFO Channel 04/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:804:804 [0] NCCL INFO Channel 03/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:828:828 [2] NCCL INFO Channel 04/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:829:829 [3] NCCL INFO Channel 05/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:831:831 [5] NCCL INFO Channel 05/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:833:833 [7] NCCL INFO Channel 05/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:827:827 [1] NCCL INFO Channel 01/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:832:832 [6] NCCL INFO Channel 05/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:804:804 [0] NCCL INFO Channel 04/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:828:828 [2] NCCL INFO Channel 05/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:829:829 [3] NCCL INFO Channel 06/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:831:831 [5] NCCL INFO Channel 06/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:833:833 [7] NCCL INFO Channel 06/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:827:827 [1] NCCL INFO Channel 02/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:830:830 [4] NCCL INFO Channel 00/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:832:832 [6] NCCL INFO Channel 06/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:804:804 [0] NCCL INFO Channel 05/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:828:828 [2] NCCL INFO Channel 06/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:829:829 [3] NCCL INFO Channel 07/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:831:831 [5] NCCL INFO Channel 07/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:827:827 [1] NCCL INFO Channel 03/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:833:833 [7] NCCL INFO Channel 07/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:830:830 [4] NCCL INFO Channel 01/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:832:832 [6] NCCL INFO Channel 07/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:804:804 [0] NCCL INFO Channel 06/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:828:828 [2] NCCL INFO Channel 07/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:829:829 [3] NCCL INFO Channel 08/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:831:831 [5] NCCL INFO Channel 08/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:827:827 [1] NCCL INFO Channel 04/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:833:833 [7] NCCL INFO Channel 08/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:830:830 [4] NCCL INFO Channel 02/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:832:832 [6] NCCL INFO Channel 08/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:804:804 [0] NCCL INFO Channel 07/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:828:828 [2] NCCL INFO Channel 08/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:829:829 [3] NCCL INFO Channel 09/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:831:831 [5] NCCL INFO Channel 09/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:827:827 [1] NCCL INFO Channel 05/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:833:833 [7] NCCL INFO Channel 09/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:830:830 [4] NCCL INFO Channel 03/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:832:832 [6] NCCL INFO Channel 09/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:804:804 [0] NCCL INFO Channel 08/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:828:828 [2] NCCL INFO Channel 09/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:829:829 [3] NCCL INFO Channel 10/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:831:831 [5] NCCL INFO Channel 10/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:827:827 [1] NCCL INFO Channel 06/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:833:833 [7] NCCL INFO Channel 10/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:830:830 [4] NCCL INFO Channel 04/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:832:832 [6] NCCL INFO Channel 10/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:804:804 [0] NCCL INFO Channel 09/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:828:828 [2] NCCL INFO Channel 10/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:829:829 [3] NCCL INFO Channel 11/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:831:831 [5] NCCL INFO Channel 11/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:833:833 [7] NCCL INFO Channel 11/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:827:827 [1] NCCL INFO Channel 07/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:830:830 [4] NCCL INFO Channel 05/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:832:832 [6] NCCL INFO Channel 11/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:804:804 [0] NCCL INFO Channel 10/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:828:828 [2] NCCL INFO Channel 11/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:829:829 [3] NCCL INFO Channel 12/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:831:831 [5] NCCL INFO Channel 12/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:833:833 [7] NCCL INFO Channel 12/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:830:830 [4] NCCL INFO Channel 06/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:827:827 [1] NCCL INFO Channel 08/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:832:832 [6] NCCL INFO Channel 12/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:828:828 [2] NCCL INFO Channel 12/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:829:829 [3] NCCL INFO Channel 13/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:804:804 [0] NCCL INFO Channel 11/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:831:831 [5] NCCL INFO Channel 13/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:830:830 [4] NCCL INFO Channel 07/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:833:833 [7] NCCL INFO Channel 13/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:827:827 [1] NCCL INFO Channel 09/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:832:832 [6] NCCL INFO Channel 13/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:829:829 [3] NCCL INFO Channel 14/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:828:828 [2] NCCL INFO Channel 13/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:831:831 [5] NCCL INFO Channel 14/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:804:804 [0] NCCL INFO Channel 12/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:833:833 [7] NCCL INFO Channel 14/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:830:830 [4] NCCL INFO Channel 08/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:827:827 [1] NCCL INFO Channel 10/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:832:832 [6] NCCL INFO Channel 14/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:829:829 [3] NCCL INFO Channel 15/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:828:828 [2] NCCL INFO Channel 14/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:831:831 [5] NCCL INFO Channel 15/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:804:804 [0] NCCL INFO Channel 13/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:833:833 [7] NCCL INFO Channel 15/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:830:830 [4] NCCL INFO Channel 09/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:827:827 [1] NCCL INFO Channel 11/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:832:832 [6] NCCL INFO Channel 15/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:829:829 [3] NCCL INFO Channel 16/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:828:828 [2] NCCL INFO Channel 15/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:831:831 [5] NCCL INFO Channel 16/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:833:833 [7] NCCL INFO Channel 16/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:804:804 [0] NCCL INFO Channel 14/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:830:830 [4] NCCL INFO Channel 10/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:827:827 [1] NCCL INFO Channel 12/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:832:832 [6] NCCL INFO Channel 16/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:829:829 [3] NCCL INFO Channel 17/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:831:831 [5] NCCL INFO Channel 17/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:828:828 [2] NCCL INFO Channel 16/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:833:833 [7] NCCL INFO Channel 17/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:804:804 [0] NCCL INFO Channel 15/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:830:830 [4] NCCL INFO Channel 11/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:827:827 [1] NCCL INFO Channel 13/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:832:832 [6] NCCL INFO Channel 17/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:829:829 [3] NCCL INFO Channel 18/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:831:831 [5] NCCL INFO Channel 18/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:828:828 [2] NCCL INFO Channel 17/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:833:833 [7] NCCL INFO Channel 18/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:830:830 [4] NCCL INFO Channel 12/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:827:827 [1] NCCL INFO Channel 14/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:804:804 [0] NCCL INFO Channel 16/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:832:832 [6] NCCL INFO Channel 18/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:829:829 [3] NCCL INFO Channel 19/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:831:831 [5] NCCL INFO Channel 19/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:828:828 [2] NCCL INFO Channel 18/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:833:833 [7] NCCL INFO Channel 19/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:830:830 [4] NCCL INFO Channel 13/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:827:827 [1] NCCL INFO Channel 15/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:832:832 [6] NCCL INFO Channel 19/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:804:804 [0] NCCL INFO Channel 17/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:829:829 [3] NCCL INFO Channel 20/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:831:831 [5] NCCL INFO Channel 20/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:828:828 [2] NCCL INFO Channel 19/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:833:833 [7] NCCL INFO Channel 20/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:830:830 [4] NCCL INFO Channel 14/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:832:832 [6] NCCL INFO Channel 20/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:827:827 [1] NCCL INFO Channel 16/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:804:804 [0] NCCL INFO Channel 18/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:829:829 [3] NCCL INFO Channel 21/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:831:831 [5] NCCL INFO Channel 21/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:828:828 [2] NCCL INFO Channel 20/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:833:833 [7] NCCL INFO Channel 21/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:830:830 [4] NCCL INFO Channel 15/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:832:832 [6] NCCL INFO Channel 21/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:827:827 [1] NCCL INFO Channel 17/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:829:829 [3] NCCL INFO Channel 22/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:804:804 [0] NCCL INFO Channel 19/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:831:831 [5] NCCL INFO Channel 22/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:828:828 [2] NCCL INFO Channel 21/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:833:833 [7] NCCL INFO Channel 22/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:830:830 [4] NCCL INFO Channel 16/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:832:832 [6] NCCL INFO Channel 22/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:827:827 [1] NCCL INFO Channel 18/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:828:828 [2] NCCL INFO Channel 22/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:829:829 [3] NCCL INFO Channel 23/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:831:831 [5] NCCL INFO Channel 23/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:804:804 [0] NCCL INFO Channel 20/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:833:833 [7] NCCL INFO Channel 23/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:830:830 [4] NCCL INFO Channel 17/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:832:832 [6] NCCL INFO Channel 23/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:827:827 [1] NCCL INFO Channel 19/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:828:828 [2] NCCL INFO Channel 23/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:804:804 [0] NCCL INFO Channel 21/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:830:830 [4] NCCL INFO Channel 18/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:827:827 [1] NCCL INFO Channel 20/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:804:804 [0] NCCL INFO Channel 22/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:830:830 [4] NCCL INFO Channel 19/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:827:827 [1] NCCL INFO Channel 21/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:804:804 [0] NCCL INFO Channel 23/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:830:830 [4] NCCL INFO Channel 20/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:827:827 [1] NCCL INFO Channel 22/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:830:830 [4] NCCL INFO Channel 21/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:827:827 [1] NCCL INFO Channel 23/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:830:830 [4] NCCL INFO Channel 22/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:830:830 [4] NCCL INFO Channel 23/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:833:833 [7] NCCL INFO Connected all rings | |
| tyler-rhel-newimage:833:833 [7] NCCL INFO Channel 00/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:833:833 [7] NCCL INFO Channel 01/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:833:833 [7] NCCL INFO Channel 02/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:804:804 [0] NCCL INFO Connected all rings | |
| tyler-rhel-newimage:827:827 [1] NCCL INFO Connected all rings | |
| tyler-rhel-newimage:833:833 [7] NCCL INFO Channel 03/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:832:832 [6] NCCL INFO Connected all rings | |
| tyler-rhel-newimage:828:828 [2] NCCL INFO Connected all rings | |
| tyler-rhel-newimage:830:830 [4] NCCL INFO Connected all rings | |
| tyler-rhel-newimage:831:831 [5] NCCL INFO Connected all rings | |
| tyler-rhel-newimage:829:829 [3] NCCL INFO Connected all rings | |
| tyler-rhel-newimage:833:833 [7] NCCL INFO Channel 04/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:833:833 [7] NCCL INFO Channel 05/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:833:833 [7] NCCL INFO Channel 06/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:833:833 [7] NCCL INFO Channel 07/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:833:833 [7] NCCL INFO Channel 08/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:833:833 [7] NCCL INFO Channel 09/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:833:833 [7] NCCL INFO Channel 10/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:833:833 [7] NCCL INFO Channel 11/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:833:833 [7] NCCL INFO Channel 12/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:833:833 [7] NCCL INFO Channel 13/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:833:833 [7] NCCL INFO Channel 14/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:833:833 [7] NCCL INFO Channel 15/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:833:833 [7] NCCL INFO Channel 16/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:833:833 [7] NCCL INFO Channel 17/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:833:833 [7] NCCL INFO Channel 18/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:833:833 [7] NCCL INFO Channel 19/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:833:833 [7] NCCL INFO Channel 20/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:833:833 [7] NCCL INFO Channel 21/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:833:833 [7] NCCL INFO Channel 22/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:833:833 [7] NCCL INFO Channel 23/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:827:827 [1] NCCL INFO Channel 00/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:831:831 [5] NCCL INFO Channel 00/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:829:829 [3] NCCL INFO Channel 00/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:827:827 [1] NCCL INFO Channel 01/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:831:831 [5] NCCL INFO Channel 01/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:828:828 [2] NCCL INFO Channel 00/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:829:829 [3] NCCL INFO Channel 01/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:830:830 [4] NCCL INFO Channel 00/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:827:827 [1] NCCL INFO Channel 02/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:831:831 [5] NCCL INFO Channel 02/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:828:828 [2] NCCL INFO Channel 01/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:832:832 [6] NCCL INFO Channel 00/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:829:829 [3] NCCL INFO Channel 02/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:830:830 [4] NCCL INFO Channel 01/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:827:827 [1] NCCL INFO Channel 03/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:831:831 [5] NCCL INFO Channel 03/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:828:828 [2] NCCL INFO Channel 02/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:829:829 [3] NCCL INFO Channel 03/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:830:830 [4] NCCL INFO Channel 02/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:832:832 [6] NCCL INFO Channel 01/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:827:827 [1] NCCL INFO Channel 04/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:831:831 [5] NCCL INFO Channel 04/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:828:828 [2] NCCL INFO Channel 03/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:829:829 [3] NCCL INFO Channel 04/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:830:830 [4] NCCL INFO Channel 03/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:827:827 [1] NCCL INFO Channel 05/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:832:832 [6] NCCL INFO Channel 02/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:831:831 [5] NCCL INFO Channel 05/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:828:828 [2] NCCL INFO Channel 04/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:829:829 [3] NCCL INFO Channel 05/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:830:830 [4] NCCL INFO Channel 04/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:827:827 [1] NCCL INFO Channel 06/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:832:832 [6] NCCL INFO Channel 03/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:831:831 [5] NCCL INFO Channel 06/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:828:828 [2] NCCL INFO Channel 05/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:829:829 [3] NCCL INFO Channel 06/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:830:830 [4] NCCL INFO Channel 05/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:827:827 [1] NCCL INFO Channel 07/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:832:832 [6] NCCL INFO Channel 04/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:831:831 [5] NCCL INFO Channel 07/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:828:828 [2] NCCL INFO Channel 06/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:829:829 [3] NCCL INFO Channel 07/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:830:830 [4] NCCL INFO Channel 06/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:827:827 [1] NCCL INFO Channel 08/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:832:832 [6] NCCL INFO Channel 05/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:831:831 [5] NCCL INFO Channel 08/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:828:828 [2] NCCL INFO Channel 07/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:829:829 [3] NCCL INFO Channel 08/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:830:830 [4] NCCL INFO Channel 07/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:827:827 [1] NCCL INFO Channel 09/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:831:831 [5] NCCL INFO Channel 09/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:832:832 [6] NCCL INFO Channel 06/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:828:828 [2] NCCL INFO Channel 08/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:829:829 [3] NCCL INFO Channel 09/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:830:830 [4] NCCL INFO Channel 08/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:827:827 [1] NCCL INFO Channel 10/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:831:831 [5] NCCL INFO Channel 10/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:828:828 [2] NCCL INFO Channel 09/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:832:832 [6] NCCL INFO Channel 07/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:829:829 [3] NCCL INFO Channel 10/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:830:830 [4] NCCL INFO Channel 09/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:827:827 [1] NCCL INFO Channel 11/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:831:831 [5] NCCL INFO Channel 11/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:828:828 [2] NCCL INFO Channel 10/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:829:829 [3] NCCL INFO Channel 11/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:832:832 [6] NCCL INFO Channel 08/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:830:830 [4] NCCL INFO Channel 10/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:827:827 [1] NCCL INFO Channel 12/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:831:831 [5] NCCL INFO Channel 12/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:828:828 [2] NCCL INFO Channel 11/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:829:829 [3] NCCL INFO Channel 12/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:832:832 [6] NCCL INFO Channel 09/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:830:830 [4] NCCL INFO Channel 11/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:831:831 [5] NCCL INFO Channel 13/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:827:827 [1] NCCL INFO Channel 13/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:828:828 [2] NCCL INFO Channel 12/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:829:829 [3] NCCL INFO Channel 13/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:832:832 [6] NCCL INFO Channel 10/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:830:830 [4] NCCL INFO Channel 12/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:831:831 [5] NCCL INFO Channel 14/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:827:827 [1] NCCL INFO Channel 14/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:828:828 [2] NCCL INFO Channel 13/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:829:829 [3] NCCL INFO Channel 14/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:832:832 [6] NCCL INFO Channel 11/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:830:830 [4] NCCL INFO Channel 13/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:831:831 [5] NCCL INFO Channel 15/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:827:827 [1] NCCL INFO Channel 15/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:828:828 [2] NCCL INFO Channel 14/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:829:829 [3] NCCL INFO Channel 15/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:830:830 [4] NCCL INFO Channel 14/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:832:832 [6] NCCL INFO Channel 12/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:827:827 [1] NCCL INFO Channel 16/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:831:831 [5] NCCL INFO Channel 16/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:828:828 [2] NCCL INFO Channel 15/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:829:829 [3] NCCL INFO Channel 16/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:830:830 [4] NCCL INFO Channel 15/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:827:827 [1] NCCL INFO Channel 17/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:832:832 [6] NCCL INFO Channel 13/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:831:831 [5] NCCL INFO Channel 17/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:829:829 [3] NCCL INFO Channel 17/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:828:828 [2] NCCL INFO Channel 16/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:830:830 [4] NCCL INFO Channel 16/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:827:827 [1] NCCL INFO Channel 18/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:832:832 [6] NCCL INFO Channel 14/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:831:831 [5] NCCL INFO Channel 18/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:829:829 [3] NCCL INFO Channel 18/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:828:828 [2] NCCL INFO Channel 17/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:830:830 [4] NCCL INFO Channel 17/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:827:827 [1] NCCL INFO Channel 19/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:832:832 [6] NCCL INFO Channel 15/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:831:831 [5] NCCL INFO Channel 19/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:829:829 [3] NCCL INFO Channel 19/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:828:828 [2] NCCL INFO Channel 18/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:830:830 [4] NCCL INFO Channel 18/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:827:827 [1] NCCL INFO Channel 20/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:829:829 [3] NCCL INFO Channel 20/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:831:831 [5] NCCL INFO Channel 20/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:832:832 [6] NCCL INFO Channel 16/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:828:828 [2] NCCL INFO Channel 19/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:830:830 [4] NCCL INFO Channel 19/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:827:827 [1] NCCL INFO Channel 21/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:829:829 [3] NCCL INFO Channel 21/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:831:831 [5] NCCL INFO Channel 21/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:832:832 [6] NCCL INFO Channel 17/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:828:828 [2] NCCL INFO Channel 20/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:830:830 [4] NCCL INFO Channel 20/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:827:827 [1] NCCL INFO Channel 22/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:829:829 [3] NCCL INFO Channel 22/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:831:831 [5] NCCL INFO Channel 22/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:832:832 [6] NCCL INFO Channel 18/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:828:828 [2] NCCL INFO Channel 21/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:830:830 [4] NCCL INFO Channel 21/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:827:827 [1] NCCL INFO Channel 23/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:829:829 [3] NCCL INFO Channel 23/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:831:831 [5] NCCL INFO Channel 23/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:828:828 [2] NCCL INFO Channel 22/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:832:832 [6] NCCL INFO Channel 19/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:830:830 [4] NCCL INFO Channel 22/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:828:828 [2] NCCL INFO Channel 23/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:830:830 [4] NCCL INFO Channel 23/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:832:832 [6] NCCL INFO Channel 20/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:832:832 [6] NCCL INFO Channel 21/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:832:832 [6] NCCL INFO Channel 22/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:832:832 [6] NCCL INFO Channel 23/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:833:833 [7] NCCL INFO Connected all trees | |
| tyler-rhel-newimage:833:833 [7] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512 | |
| tyler-rhel-newimage:833:833 [7] NCCL INFO 24 coll channels, 0 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer | |
| tyler-rhel-newimage:829:829 [3] NCCL INFO Connected all trees | |
| tyler-rhel-newimage:829:829 [3] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512 | |
| tyler-rhel-newimage:829:829 [3] NCCL INFO 24 coll channels, 0 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer | |
| tyler-rhel-newimage:804:804 [0] NCCL INFO Connected all trees | |
| tyler-rhel-newimage:830:830 [4] NCCL INFO Connected all trees | |
| tyler-rhel-newimage:830:830 [4] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512 | |
| tyler-rhel-newimage:830:830 [4] NCCL INFO 24 coll channels, 0 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer | |
| tyler-rhel-newimage:804:804 [0] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512 | |
| tyler-rhel-newimage:804:804 [0] NCCL INFO 24 coll channels, 0 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer | |
| tyler-rhel-newimage:828:828 [2] NCCL INFO Connected all trees | |
| tyler-rhel-newimage:828:828 [2] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512 | |
| tyler-rhel-newimage:828:828 [2] NCCL INFO 24 coll channels, 0 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer | |
| tyler-rhel-newimage:827:827 [1] NCCL INFO Connected all trees | |
| tyler-rhel-newimage:827:827 [1] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512 | |
| tyler-rhel-newimage:832:832 [6] NCCL INFO Connected all trees | |
| tyler-rhel-newimage:827:827 [1] NCCL INFO 24 coll channels, 0 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer | |
| tyler-rhel-newimage:831:831 [5] NCCL INFO Connected all trees | |
| tyler-rhel-newimage:832:832 [6] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512 | |
| tyler-rhel-newimage:832:832 [6] NCCL INFO 24 coll channels, 0 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer | |
| tyler-rhel-newimage:831:831 [5] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512 | |
| tyler-rhel-newimage:831:831 [5] NCCL INFO 24 coll channels, 0 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer | |
| tyler-rhel-newimage:833:833 [7] NCCL INFO NCCL_WORK_FIFO_DEPTH set by environment to 4194304. | |
| tyler-rhel-newimage:829:829 [3] NCCL INFO NCCL_WORK_FIFO_DEPTH set by environment to 4194304. | |
| tyler-rhel-newimage:804:804 [0] NCCL INFO NCCL_WORK_FIFO_DEPTH set by environment to 4194304. | |
| tyler-rhel-newimage:828:828 [2] NCCL INFO NCCL_WORK_FIFO_DEPTH set by environment to 4194304. | |
| tyler-rhel-newimage:827:827 [1] NCCL INFO NCCL_WORK_FIFO_DEPTH set by environment to 4194304. | |
| tyler-rhel-newimage:830:830 [4] NCCL INFO NCCL_WORK_FIFO_DEPTH set by environment to 4194304. | |
| tyler-rhel-newimage:832:832 [6] NCCL INFO NCCL_WORK_FIFO_DEPTH set by environment to 4194304. | |
| tyler-rhel-newimage:831:831 [5] NCCL INFO NCCL_WORK_FIFO_DEPTH set by environment to 4194304. | |
| tyler-rhel-newimage:829:829 [3] NCCL INFO comm 0x556dc48048f0 rank 3 nranks 8 cudaDev 3 nvmlDev 3 busId a040 commId 0x7ee526e6cd90ff03 - Init COMPLETE | |
| tyler-rhel-newimage:830:830 [4] NCCL INFO comm 0x556dc4806330 rank 4 nranks 8 cudaDev 4 nvmlDev 4 busId c050 commId 0x7ee526e6cd90ff03 - Init COMPLETE | |
| tyler-rhel-newimage:832:832 [6] NCCL INFO comm 0x556dc4805e20 rank 6 nranks 8 cudaDev 6 nvmlDev 6 busId e070 commId 0x7ee526e6cd90ff03 - Init COMPLETE | |
| tyler-rhel-newimage:833:833 [7] NCCL INFO comm 0x556dc4806ea0 rank 7 nranks 8 cudaDev 7 nvmlDev 7 busId e080 commId 0x7ee526e6cd90ff03 - Init COMPLETE | |
| tyler-rhel-newimage:828:828 [2] NCCL INFO comm 0x556dc48044c0 rank 2 nranks 8 cudaDev 2 nvmlDev 2 busId a030 commId 0x7ee526e6cd90ff03 - Init COMPLETE | |
| tyler-rhel-newimage:831:831 [5] NCCL INFO comm 0x556dc4806b40 rank 5 nranks 8 cudaDev 5 nvmlDev 5 busId c060 commId 0x7ee526e6cd90ff03 - Init COMPLETE | |
| tyler-rhel-newimage:827:827 [1] NCCL INFO comm 0x556dc48056c0 rank 1 nranks 8 cudaDev 1 nvmlDev 1 busId 8020 commId 0x7ee526e6cd90ff03 - Init COMPLETE | |
| tyler-rhel-newimage:804:804 [0] NCCL INFO comm 0x556dc480be40 rank 0 nranks 8 cudaDev 0 nvmlDev 0 busId 8010 commId 0x7ee526e6cd90ff03 - Init COMPLETE | |
| Loading safetensors checkpoint shards: 0% Completed | 0/6 [00:00<?, ?it/s] | |
| Loading safetensors checkpoint shards: 17% Completed | 1/6 [00:00<00:03, 1.46it/s] | |
| INFO 2024-07-27 16:32:47,861 instructlab.model.backends.backends:336: Waiting for the vLLM server to start at http://127.0.0.1:44547/v1, this might take a moment... Attempt: 6/80 | |
| Loading safetensors checkpoint shards: 33% Completed | 2/6 [00:01<00:02, 1.35it/s] | |
| Loading safetensors checkpoint shards: 50% Completed | 3/6 [00:02<00:02, 1.34it/s] | |
| Loading safetensors checkpoint shards: 67% Completed | 4/6 [00:02<00:01, 1.64it/s] | |
| Loading safetensors checkpoint shards: 83% Completed | 5/6 [00:03<00:00, 1.46it/s] | |
| Loading safetensors checkpoint shards: 100% Completed | 6/6 [00:03<00:00, 1.66it/s] | |
| Loading safetensors checkpoint shards: 100% Completed | 6/6 [00:03<00:00, 1.55it/s] | |
| tyler-rhel-newimage:804:1366 [0] NCCL INFO Using non-device net plugin version 0 | |
| tyler-rhel-newimage:804:1366 [0] NCCL INFO Using network Socket | |
| tyler-rhel-newimage:832:1367 [6] NCCL INFO Using non-device net plugin version 0 | |
| tyler-rhel-newimage:832:1367 [6] NCCL INFO Using network Socket | |
| tyler-rhel-newimage:827:1368 [1] NCCL INFO Using non-device net plugin version 0 | |
| tyler-rhel-newimage:827:1368 [1] NCCL INFO Using network Socket | |
| tyler-rhel-newimage:829:1369 [3] NCCL INFO Using non-device net plugin version 0 | |
| tyler-rhel-newimage:829:1369 [3] NCCL INFO Using network Socket | |
| tyler-rhel-newimage:830:1370 [4] NCCL INFO Using non-device net plugin version 0 | |
| tyler-rhel-newimage:830:1370 [4] NCCL INFO Using network Socket | |
| tyler-rhel-newimage:831:1371 [5] NCCL INFO Using non-device net plugin version 0 | |
| tyler-rhel-newimage:831:1371 [5] NCCL INFO Using network Socket | |
| tyler-rhel-newimage:828:1372 [2] NCCL INFO Using non-device net plugin version 0 | |
| tyler-rhel-newimage:828:1372 [2] NCCL INFO Using network Socket | |
| tyler-rhel-newimage:833:1373 [7] NCCL INFO Using non-device net plugin version 0 | |
| tyler-rhel-newimage:833:1373 [7] NCCL INFO Using network Socket | |
| tyler-rhel-newimage:833:1373 [7] NCCL INFO comm 0x556ddbc53d80 rank 7 nranks 8 cudaDev 7 nvmlDev 7 busId e080 commId 0x9035024dada8a21d - Init START | |
| tyler-rhel-newimage:832:1367 [6] NCCL INFO comm 0x556ddbc6b1e0 rank 6 nranks 8 cudaDev 6 nvmlDev 6 busId e070 commId 0x9035024dada8a21d - Init START | |
| tyler-rhel-newimage:831:1371 [5] NCCL INFO comm 0x556ddbc6a210 rank 5 nranks 8 cudaDev 5 nvmlDev 5 busId c060 commId 0x9035024dada8a21d - Init START | |
| tyler-rhel-newimage:830:1370 [4] NCCL INFO comm 0x556ddbc69c20 rank 4 nranks 8 cudaDev 4 nvmlDev 4 busId c050 commId 0x9035024dada8a21d - Init START | |
| tyler-rhel-newimage:829:1369 [3] NCCL INFO comm 0x556ddbc68d70 rank 3 nranks 8 cudaDev 3 nvmlDev 3 busId a040 commId 0x9035024dada8a21d - Init START | |
| tyler-rhel-newimage:828:1372 [2] NCCL INFO comm 0x556ddbc69260 rank 2 nranks 8 cudaDev 2 nvmlDev 2 busId a030 commId 0x9035024dada8a21d - Init START | |
| tyler-rhel-newimage:827:1368 [1] NCCL INFO comm 0x556ddbc697b0 rank 1 nranks 8 cudaDev 1 nvmlDev 1 busId 8020 commId 0x9035024dada8a21d - Init START | |
| tyler-rhel-newimage:804:1366 [0] NCCL INFO comm 0x556ddbc45f80 rank 0 nranks 8 cudaDev 0 nvmlDev 0 busId 8010 commId 0x9035024dada8a21d - Init START | |
| INFO 2024-07-27 16:32:52,280 instructlab.model.backends.backends:336: Waiting for the vLLM server to start at http://127.0.0.1:44547/v1, this might take a moment... Attempt: 7/80 | |
| tyler-rhel-newimage:827:1368 [1] NCCL INFO Setting affinity for GPU 1 to ff,ffffffff | |
| tyler-rhel-newimage:827:1368 [1] NCCL INFO NVLS multicast support is not available on dev 1 | |
| tyler-rhel-newimage:830:1370 [4] NCCL INFO Setting affinity for GPU 4 to ffff,ffffff00,00000000 | |
| tyler-rhel-newimage:830:1370 [4] NCCL INFO NVLS multicast support is not available on dev 4 | |
| tyler-rhel-newimage:828:1372 [2] NCCL INFO Setting affinity for GPU 2 to ff,ffffffff | |
| tyler-rhel-newimage:828:1372 [2] NCCL INFO NVLS multicast support is not available on dev 2 | |
| tyler-rhel-newimage:831:1371 [5] NCCL INFO Setting affinity for GPU 5 to ffff,ffffff00,00000000 | |
| tyler-rhel-newimage:831:1371 [5] NCCL INFO NVLS multicast support is not available on dev 5 | |
| tyler-rhel-newimage:832:1367 [6] NCCL INFO Setting affinity for GPU 6 to ffff,ffffff00,00000000 | |
| tyler-rhel-newimage:832:1367 [6] NCCL INFO NVLS multicast support is not available on dev 6 | |
| tyler-rhel-newimage:833:1373 [7] NCCL INFO Setting affinity for GPU 7 to ffff,ffffff00,00000000 | |
| tyler-rhel-newimage:833:1373 [7] NCCL INFO NVLS multicast support is not available on dev 7 | |
| tyler-rhel-newimage:804:1366 [0] NCCL INFO Setting affinity for GPU 0 to ff,ffffffff | |
| tyler-rhel-newimage:804:1366 [0] NCCL INFO NVLS multicast support is not available on dev 0 | |
| tyler-rhel-newimage:829:1369 [3] NCCL INFO Setting affinity for GPU 3 to ff,ffffffff | |
| tyler-rhel-newimage:829:1369 [3] NCCL INFO NVLS multicast support is not available on dev 3 | |
| tyler-rhel-newimage:832:1367 [6] NCCL INFO comm 0x556ddbc6b1e0 rank 6 nRanks 8 nNodes 1 localRanks 8 localRank 6 MNNVL 0 | |
| tyler-rhel-newimage:831:1371 [5] NCCL INFO comm 0x556ddbc6a210 rank 5 nRanks 8 nNodes 1 localRanks 8 localRank 5 MNNVL 0 | |
| tyler-rhel-newimage:830:1370 [4] NCCL INFO comm 0x556ddbc69c20 rank 4 nRanks 8 nNodes 1 localRanks 8 localRank 4 MNNVL 0 | |
| tyler-rhel-newimage:829:1369 [3] NCCL INFO comm 0x556ddbc68d70 rank 3 nRanks 8 nNodes 1 localRanks 8 localRank 3 MNNVL 0 | |
| tyler-rhel-newimage:827:1368 [1] NCCL INFO comm 0x556ddbc697b0 rank 1 nRanks 8 nNodes 1 localRanks 8 localRank 1 MNNVL 0 | |
| tyler-rhel-newimage:828:1372 [2] NCCL INFO comm 0x556ddbc69260 rank 2 nRanks 8 nNodes 1 localRanks 8 localRank 2 MNNVL 0 | |
| tyler-rhel-newimage:832:1367 [6] NCCL INFO Trees [0] 7/-1/-1->6->5 [1] 7/-1/-1->6->5 [2] 7/-1/-1->6->5 [3] 7/-1/-1->6->5 [4] 7/-1/-1->6->5 [5] 7/-1/-1->6->5 [6] 7/-1/-1->6->5 [7] 7/-1/-1->6->5 [8] 7/-1/-1->6->5 [9] 7/-1/-1->6->5 [10] 7/-1/-1->6->5 [11] 7/-1/-1->6->5 [12] 7/-1/-1->6->5 [13] 7/-1/-1->6->5 [14] 7/-1/-1->6->5 [15] 7/-1/-1->6->5 [16] 7/-1/-1->6->5 [17] 7/-1/-1->6->5 [18] 7/-1/-1->6->5 [19] 7/-1/-1->6->5 [20] 7/-1/-1->6->5 [21] 7/-1/-1->6->5 [22] 7/-1/-1->6->5 [23] 7/-1/-1->6->5 | |
| tyler-rhel-newimage:804:1366 [0] NCCL INFO comm 0x556ddbc45f80 rank 0 nRanks 8 nNodes 1 localRanks 8 localRank 0 MNNVL 0 | |
| tyler-rhel-newimage:832:1367 [6] NCCL INFO P2P Chunksize set to 524288 | |
| tyler-rhel-newimage:830:1370 [4] NCCL INFO Trees [0] 5/-1/-1->4->3 [1] 5/-1/-1->4->3 [2] 5/-1/-1->4->3 [3] 5/-1/-1->4->3 [4] 5/-1/-1->4->3 [5] 5/-1/-1->4->3 [6] 5/-1/-1->4->3 [7] 5/-1/-1->4->3 [8] 5/-1/-1->4->3 [9] 5/-1/-1->4->3 [10] 5/-1/-1->4->3 [11] 5/-1/-1->4->3 [12] 5/-1/-1->4->3 [13] 5/-1/-1->4->3 [14] 5/-1/-1->4->3 [15] 5/-1/-1->4->3 [16] 5/-1/-1->4->3 [17] 5/-1/-1->4->3 [18] 5/-1/-1->4->3 [19] 5/-1/-1->4->3 [20] 5/-1/-1->4->3 [21] 5/-1/-1->4->3 [22] 5/-1/-1->4->3 [23] 5/-1/-1->4->3 | |
| tyler-rhel-newimage:829:1369 [3] NCCL INFO Trees [0] 4/-1/-1->3->2 [1] 4/-1/-1->3->2 [2] 4/-1/-1->3->2 [3] 4/-1/-1->3->2 [4] 4/-1/-1->3->2 [5] 4/-1/-1->3->2 [6] 4/-1/-1->3->2 [7] 4/-1/-1->3->2 [8] 4/-1/-1->3->2 [9] 4/-1/-1->3->2 [10] 4/-1/-1->3->2 [11] 4/-1/-1->3->2 [12] 4/-1/-1->3->2 [13] 4/-1/-1->3->2 [14] 4/-1/-1->3->2 [15] 4/-1/-1->3->2 [16] 4/-1/-1->3->2 [17] 4/-1/-1->3->2 [18] 4/-1/-1->3->2 [19] 4/-1/-1->3->2 [20] 4/-1/-1->3->2 [21] 4/-1/-1->3->2 [22] 4/-1/-1->3->2 [23] 4/-1/-1->3->2 | |
| tyler-rhel-newimage:829:1369 [3] NCCL INFO P2P Chunksize set to 524288 | |
| tyler-rhel-newimage:828:1372 [2] NCCL INFO Trees [0] 3/-1/-1->2->1 [1] 3/-1/-1->2->1 [2] 3/-1/-1->2->1 [3] 3/-1/-1->2->1 [4] 3/-1/-1->2->1 [5] 3/-1/-1->2->1 [6] 3/-1/-1->2->1 [7] 3/-1/-1->2->1 [8] 3/-1/-1->2->1 [9] 3/-1/-1->2->1 [10] 3/-1/-1->2->1 [11] 3/-1/-1->2->1 [12] 3/-1/-1->2->1 [13] 3/-1/-1->2->1 [14] 3/-1/-1->2->1 [15] 3/-1/-1->2->1 [16] 3/-1/-1->2->1 [17] 3/-1/-1->2->1 [18] 3/-1/-1->2->1 [19] 3/-1/-1->2->1 [20] 3/-1/-1->2->1 [21] 3/-1/-1->2->1 [22] 3/-1/-1->2->1 [23] 3/-1/-1->2->1 | |
| tyler-rhel-newimage:831:1371 [5] NCCL INFO Trees [0] 6/-1/-1->5->4 [1] 6/-1/-1->5->4 [2] 6/-1/-1->5->4 [3] 6/-1/-1->5->4 [4] 6/-1/-1->5->4 [5] 6/-1/-1->5->4 [6] 6/-1/-1->5->4 [7] 6/-1/-1->5->4 [8] 6/-1/-1->5->4 [9] 6/-1/-1->5->4 [10] 6/-1/-1->5->4 [11] 6/-1/-1->5->4 [12] 6/-1/-1->5->4 [13] 6/-1/-1->5->4 [14] 6/-1/-1->5->4 [15] 6/-1/-1->5->4 [16] 6/-1/-1->5->4 [17] 6/-1/-1->5->4 [18] 6/-1/-1->5->4 [19] 6/-1/-1->5->4 [20] 6/-1/-1->5->4 [21] 6/-1/-1->5->4 [22] 6/-1/-1->5->4 [23] 6/-1/-1->5->4 | |
| tyler-rhel-newimage:804:1366 [0] NCCL INFO Channel 00/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:830:1370 [4] NCCL INFO P2P Chunksize set to 524288 | |
| tyler-rhel-newimage:827:1368 [1] NCCL INFO Trees [0] 2/-1/-1->1->0 [1] 2/-1/-1->1->0 [2] 2/-1/-1->1->0 [3] 2/-1/-1->1->0 [4] 2/-1/-1->1->0 [5] 2/-1/-1->1->0 [6] 2/-1/-1->1->0 [7] 2/-1/-1->1->0 [8] 2/-1/-1->1->0 [9] 2/-1/-1->1->0 [10] 2/-1/-1->1->0 [11] 2/-1/-1->1->0 [12] 2/-1/-1->1->0 [13] 2/-1/-1->1->0 [14] 2/-1/-1->1->0 [15] 2/-1/-1->1->0 [16] 2/-1/-1->1->0 [17] 2/-1/-1->1->0 [18] 2/-1/-1->1->0 [19] 2/-1/-1->1->0 [20] 2/-1/-1->1->0 [21] 2/-1/-1->1->0 [22] 2/-1/-1->1->0 [23] 2/-1/-1->1->0 | |
| tyler-rhel-newimage:833:1373 [7] NCCL INFO comm 0x556ddbc53d80 rank 7 nRanks 8 nNodes 1 localRanks 8 localRank 7 MNNVL 0 | |
| tyler-rhel-newimage:828:1372 [2] NCCL INFO P2P Chunksize set to 524288 | |
| tyler-rhel-newimage:804:1366 [0] NCCL INFO Channel 01/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:831:1371 [5] NCCL INFO P2P Chunksize set to 524288 | |
| tyler-rhel-newimage:827:1368 [1] NCCL INFO P2P Chunksize set to 524288 | |
| tyler-rhel-newimage:804:1366 [0] NCCL INFO Channel 02/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:804:1366 [0] NCCL INFO Channel 03/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:804:1366 [0] NCCL INFO Channel 04/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:804:1366 [0] NCCL INFO Channel 05/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:804:1366 [0] NCCL INFO Channel 06/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:804:1366 [0] NCCL INFO Channel 07/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:804:1366 [0] NCCL INFO Channel 08/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:804:1366 [0] NCCL INFO Channel 09/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:833:1373 [7] NCCL INFO Trees [0] -1/-1/-1->7->6 [1] -1/-1/-1->7->6 [2] -1/-1/-1->7->6 [3] -1/-1/-1->7->6 [4] -1/-1/-1->7->6 [5] -1/-1/-1->7->6 [6] -1/-1/-1->7->6 [7] -1/-1/-1->7->6 [8] -1/-1/-1->7->6 [9] -1/-1/-1->7->6 [10] -1/-1/-1->7->6 [11] -1/-1/-1->7->6 [12] -1/-1/-1->7->6 [13] -1/-1/-1->7->6 [14] -1/-1/-1->7->6 [15] -1/-1/-1->7->6 [16] -1/-1/-1->7->6 [17] -1/-1/-1->7->6 [18] -1/-1/-1->7->6 [19] -1/-1/-1->7->6 [20] -1/-1/-1->7->6 [21] -1/-1/-1->7->6 [22] -1/-1/-1->7->6 [23] -1/-1/-1->7->6 | |
| tyler-rhel-newimage:804:1366 [0] NCCL INFO Channel 10/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:833:1373 [7] NCCL INFO P2P Chunksize set to 524288 | |
| tyler-rhel-newimage:804:1366 [0] NCCL INFO Channel 11/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:804:1366 [0] NCCL INFO Channel 12/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:804:1366 [0] NCCL INFO Channel 13/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:804:1366 [0] NCCL INFO Channel 14/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:804:1366 [0] NCCL INFO Channel 15/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:804:1366 [0] NCCL INFO Channel 16/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:804:1366 [0] NCCL INFO Channel 17/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:804:1366 [0] NCCL INFO Channel 18/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:804:1366 [0] NCCL INFO Channel 19/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:804:1366 [0] NCCL INFO Channel 20/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:804:1366 [0] NCCL INFO Channel 21/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:804:1366 [0] NCCL INFO Channel 22/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:804:1366 [0] NCCL INFO Channel 23/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:804:1366 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1 [2] 1/-1/-1->0->-1 [3] 1/-1/-1->0->-1 [4] 1/-1/-1->0->-1 [5] 1/-1/-1->0->-1 [6] 1/-1/-1->0->-1 [7] 1/-1/-1->0->-1 [8] 1/-1/-1->0->-1 [9] 1/-1/-1->0->-1 [10] 1/-1/-1->0->-1 [11] 1/-1/-1->0->-1 [12] 1/-1/-1->0->-1 [13] 1/-1/-1->0->-1 [14] 1/-1/-1->0->-1 [15] 1/-1/-1->0->-1 [16] 1/-1/-1->0->-1 [17] 1/-1/-1->0->-1 [18] 1/-1/-1->0->-1 [19] 1/-1/-1->0->-1 [20] 1/-1/-1->0->-1 [21] 1/-1/-1->0->-1 [22] 1/-1/-1->0->-1 [23] 1/-1/-1->0->-1 | |
| tyler-rhel-newimage:804:1366 [0] NCCL INFO P2P Chunksize set to 524288 | |
| tyler-rhel-newimage:827:1368 [1] NCCL INFO Channel 00/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:829:1369 [3] NCCL INFO Channel 00/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:827:1368 [1] NCCL INFO Channel 01/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:829:1369 [3] NCCL INFO Channel 01/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:830:1370 [4] NCCL INFO Channel 00/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:827:1368 [1] NCCL INFO Channel 02/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:829:1369 [3] NCCL INFO Channel 02/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:828:1372 [2] NCCL INFO Channel 00/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:830:1370 [4] NCCL INFO Channel 01/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:827:1368 [1] NCCL INFO Channel 03/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:831:1371 [5] NCCL INFO Channel 00/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:829:1369 [3] NCCL INFO Channel 03/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:828:1372 [2] NCCL INFO Channel 01/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:830:1370 [4] NCCL INFO Channel 02/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:832:1367 [6] NCCL INFO Channel 00/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:827:1368 [1] NCCL INFO Channel 04/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:831:1371 [5] NCCL INFO Channel 01/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:829:1369 [3] NCCL INFO Channel 04/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:830:1370 [4] NCCL INFO Channel 03/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:828:1372 [2] NCCL INFO Channel 02/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:832:1367 [6] NCCL INFO Channel 01/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:827:1368 [1] NCCL INFO Channel 05/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:831:1371 [5] NCCL INFO Channel 02/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:829:1369 [3] NCCL INFO Channel 05/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:830:1370 [4] NCCL INFO Channel 04/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:828:1372 [2] NCCL INFO Channel 03/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:833:1373 [7] NCCL INFO Channel 00/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:832:1367 [6] NCCL INFO Channel 02/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:827:1368 [1] NCCL INFO Channel 06/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:831:1371 [5] NCCL INFO Channel 03/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:829:1369 [3] NCCL INFO Channel 06/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:830:1370 [4] NCCL INFO Channel 05/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:828:1372 [2] NCCL INFO Channel 04/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:833:1373 [7] NCCL INFO Channel 01/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:832:1367 [6] NCCL INFO Channel 03/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:827:1368 [1] NCCL INFO Channel 07/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:831:1371 [5] NCCL INFO Channel 04/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:829:1369 [3] NCCL INFO Channel 07/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:830:1370 [4] NCCL INFO Channel 06/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:828:1372 [2] NCCL INFO Channel 05/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:827:1368 [1] NCCL INFO Channel 08/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:833:1373 [7] NCCL INFO Channel 02/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:832:1367 [6] NCCL INFO Channel 04/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:831:1371 [5] NCCL INFO Channel 05/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:829:1369 [3] NCCL INFO Channel 08/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:830:1370 [4] NCCL INFO Channel 07/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:828:1372 [2] NCCL INFO Channel 06/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:827:1368 [1] NCCL INFO Channel 09/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:832:1367 [6] NCCL INFO Channel 05/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:833:1373 [7] NCCL INFO Channel 03/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:829:1369 [3] NCCL INFO Channel 09/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:831:1371 [5] NCCL INFO Channel 06/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:830:1370 [4] NCCL INFO Channel 08/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:828:1372 [2] NCCL INFO Channel 07/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:804:1366 [0] NCCL INFO Channel 00/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:827:1368 [1] NCCL INFO Channel 10/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:832:1367 [6] NCCL INFO Channel 06/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:829:1369 [3] NCCL INFO Channel 10/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:833:1373 [7] NCCL INFO Channel 04/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:831:1371 [5] NCCL INFO Channel 07/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:830:1370 [4] NCCL INFO Channel 09/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:828:1372 [2] NCCL INFO Channel 08/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:804:1366 [0] NCCL INFO Channel 01/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:827:1368 [1] NCCL INFO Channel 11/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:832:1367 [6] NCCL INFO Channel 07/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:829:1369 [3] NCCL INFO Channel 11/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:833:1373 [7] NCCL INFO Channel 05/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:831:1371 [5] NCCL INFO Channel 08/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:830:1370 [4] NCCL INFO Channel 10/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:828:1372 [2] NCCL INFO Channel 09/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:804:1366 [0] NCCL INFO Channel 02/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:827:1368 [1] NCCL INFO Channel 12/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:832:1367 [6] NCCL INFO Channel 08/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:829:1369 [3] NCCL INFO Channel 12/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:833:1373 [7] NCCL INFO Channel 06/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:830:1370 [4] NCCL INFO Channel 11/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:831:1371 [5] NCCL INFO Channel 09/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:804:1366 [0] NCCL INFO Channel 03/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:828:1372 [2] NCCL INFO Channel 10/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:827:1368 [1] NCCL INFO Channel 13/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:832:1367 [6] NCCL INFO Channel 09/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:829:1369 [3] NCCL INFO Channel 13/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:830:1370 [4] NCCL INFO Channel 12/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:833:1373 [7] NCCL INFO Channel 07/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:831:1371 [5] NCCL INFO Channel 10/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:804:1366 [0] NCCL INFO Channel 04/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:828:1372 [2] NCCL INFO Channel 11/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:832:1367 [6] NCCL INFO Channel 10/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:827:1368 [1] NCCL INFO Channel 14/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:829:1369 [3] NCCL INFO Channel 14/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:830:1370 [4] NCCL INFO Channel 13/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:833:1373 [7] NCCL INFO Channel 08/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:831:1371 [5] NCCL INFO Channel 11/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:804:1366 [0] NCCL INFO Channel 05/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:828:1372 [2] NCCL INFO Channel 12/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:832:1367 [6] NCCL INFO Channel 11/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:827:1368 [1] NCCL INFO Channel 15/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:829:1369 [3] NCCL INFO Channel 15/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:830:1370 [4] NCCL INFO Channel 14/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:833:1373 [7] NCCL INFO Channel 09/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:832:1367 [6] NCCL INFO Channel 12/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:831:1371 [5] NCCL INFO Channel 12/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:804:1366 [0] NCCL INFO Channel 06/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:828:1372 [2] NCCL INFO Channel 13/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:827:1368 [1] NCCL INFO Channel 16/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:829:1369 [3] NCCL INFO Channel 16/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:830:1370 [4] NCCL INFO Channel 15/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:833:1373 [7] NCCL INFO Channel 10/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:831:1371 [5] NCCL INFO Channel 13/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:832:1367 [6] NCCL INFO Channel 13/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:804:1366 [0] NCCL INFO Channel 07/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:828:1372 [2] NCCL INFO Channel 14/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:827:1368 [1] NCCL INFO Channel 17/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:829:1369 [3] NCCL INFO Channel 17/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:830:1370 [4] NCCL INFO Channel 16/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:833:1373 [7] NCCL INFO Channel 11/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:832:1367 [6] NCCL INFO Channel 14/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:831:1371 [5] NCCL INFO Channel 14/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:804:1366 [0] NCCL INFO Channel 08/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:827:1368 [1] NCCL INFO Channel 18/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:828:1372 [2] NCCL INFO Channel 15/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:829:1369 [3] NCCL INFO Channel 18/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:830:1370 [4] NCCL INFO Channel 17/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:833:1373 [7] NCCL INFO Channel 12/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:832:1367 [6] NCCL INFO Channel 15/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:831:1371 [5] NCCL INFO Channel 15/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:804:1366 [0] NCCL INFO Channel 09/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:827:1368 [1] NCCL INFO Channel 19/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:828:1372 [2] NCCL INFO Channel 16/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:829:1369 [3] NCCL INFO Channel 19/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:830:1370 [4] NCCL INFO Channel 18/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:832:1367 [6] NCCL INFO Channel 16/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:833:1373 [7] NCCL INFO Channel 13/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:831:1371 [5] NCCL INFO Channel 16/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:804:1366 [0] NCCL INFO Channel 10/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:827:1368 [1] NCCL INFO Channel 20/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:829:1369 [3] NCCL INFO Channel 20/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:828:1372 [2] NCCL INFO Channel 17/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:830:1370 [4] NCCL INFO Channel 19/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:832:1367 [6] NCCL INFO Channel 17/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:833:1373 [7] NCCL INFO Channel 14/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:831:1371 [5] NCCL INFO Channel 17/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:804:1366 [0] NCCL INFO Channel 11/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:827:1368 [1] NCCL INFO Channel 21/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:829:1369 [3] NCCL INFO Channel 21/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:828:1372 [2] NCCL INFO Channel 18/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:830:1370 [4] NCCL INFO Channel 20/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:832:1367 [6] NCCL INFO Channel 18/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:833:1373 [7] NCCL INFO Channel 15/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:831:1371 [5] NCCL INFO Channel 18/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:804:1366 [0] NCCL INFO Channel 12/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:827:1368 [1] NCCL INFO Channel 22/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:829:1369 [3] NCCL INFO Channel 22/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:830:1370 [4] NCCL INFO Channel 21/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:828:1372 [2] NCCL INFO Channel 19/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:832:1367 [6] NCCL INFO Channel 19/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:833:1373 [7] NCCL INFO Channel 16/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:831:1371 [5] NCCL INFO Channel 19/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:827:1368 [1] NCCL INFO Channel 23/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:829:1369 [3] NCCL INFO Channel 23/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:830:1370 [4] NCCL INFO Channel 22/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:804:1366 [0] NCCL INFO Channel 13/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:828:1372 [2] NCCL INFO Channel 20/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:832:1367 [6] NCCL INFO Channel 20/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:833:1373 [7] NCCL INFO Channel 17/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:831:1371 [5] NCCL INFO Channel 20/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:830:1370 [4] NCCL INFO Channel 23/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:804:1366 [0] NCCL INFO Channel 14/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:832:1367 [6] NCCL INFO Channel 21/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:828:1372 [2] NCCL INFO Channel 21/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:833:1373 [7] NCCL INFO Channel 18/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:831:1371 [5] NCCL INFO Channel 21/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:804:1366 [0] NCCL INFO Channel 15/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:832:1367 [6] NCCL INFO Channel 22/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:828:1372 [2] NCCL INFO Channel 22/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:831:1371 [5] NCCL INFO Channel 22/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:833:1373 [7] NCCL INFO Channel 19/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:804:1366 [0] NCCL INFO Channel 16/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:832:1367 [6] NCCL INFO Channel 23/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:828:1372 [2] NCCL INFO Channel 23/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:831:1371 [5] NCCL INFO Channel 23/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:833:1373 [7] NCCL INFO Channel 20/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:804:1366 [0] NCCL INFO Channel 17/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:833:1373 [7] NCCL INFO Channel 21/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:804:1366 [0] NCCL INFO Channel 18/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:833:1373 [7] NCCL INFO Channel 22/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:804:1366 [0] NCCL INFO Channel 19/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:833:1373 [7] NCCL INFO Channel 23/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:804:1366 [0] NCCL INFO Channel 20/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:804:1366 [0] NCCL INFO Channel 21/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:804:1366 [0] NCCL INFO Channel 22/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:804:1366 [0] NCCL INFO Channel 23/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:831:1371 [5] NCCL INFO Connected all rings | |
| tyler-rhel-newimage:830:1370 [4] NCCL INFO Connected all rings | |
| tyler-rhel-newimage:829:1369 [3] NCCL INFO Connected all rings | |
| tyler-rhel-newimage:832:1367 [6] NCCL INFO Connected all rings | |
| tyler-rhel-newimage:833:1373 [7] NCCL INFO Connected all rings | |
| tyler-rhel-newimage:833:1373 [7] NCCL INFO Channel 00/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:833:1373 [7] NCCL INFO Channel 01/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:833:1373 [7] NCCL INFO Channel 02/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:833:1373 [7] NCCL INFO Channel 03/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:833:1373 [7] NCCL INFO Channel 04/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:833:1373 [7] NCCL INFO Channel 05/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:833:1373 [7] NCCL INFO Channel 06/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:833:1373 [7] NCCL INFO Channel 07/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:833:1373 [7] NCCL INFO Channel 08/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:833:1373 [7] NCCL INFO Channel 09/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:828:1372 [2] NCCL INFO Connected all rings | |
| tyler-rhel-newimage:804:1366 [0] NCCL INFO Connected all rings | |
| tyler-rhel-newimage:827:1368 [1] NCCL INFO Connected all rings | |
| tyler-rhel-newimage:833:1373 [7] NCCL INFO Channel 10/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:833:1373 [7] NCCL INFO Channel 11/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:833:1373 [7] NCCL INFO Channel 12/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:833:1373 [7] NCCL INFO Channel 13/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:833:1373 [7] NCCL INFO Channel 14/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:833:1373 [7] NCCL INFO Channel 15/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:833:1373 [7] NCCL INFO Channel 16/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:833:1373 [7] NCCL INFO Channel 17/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:833:1373 [7] NCCL INFO Channel 18/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:833:1373 [7] NCCL INFO Channel 19/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:833:1373 [7] NCCL INFO Channel 20/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:833:1373 [7] NCCL INFO Channel 21/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:833:1373 [7] NCCL INFO Channel 22/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:830:1370 [4] NCCL INFO Channel 00/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:833:1373 [7] NCCL INFO Channel 23/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:832:1367 [6] NCCL INFO Channel 00/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:830:1370 [4] NCCL INFO Channel 01/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:832:1367 [6] NCCL INFO Channel 01/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:829:1369 [3] NCCL INFO Channel 00/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:830:1370 [4] NCCL INFO Channel 02/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:831:1371 [5] NCCL INFO Channel 00/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:832:1367 [6] NCCL INFO Channel 02/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:829:1369 [3] NCCL INFO Channel 01/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:830:1370 [4] NCCL INFO Channel 03/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:831:1371 [5] NCCL INFO Channel 01/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:832:1367 [6] NCCL INFO Channel 03/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:829:1369 [3] NCCL INFO Channel 02/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:830:1370 [4] NCCL INFO Channel 04/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:831:1371 [5] NCCL INFO Channel 02/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:832:1367 [6] NCCL INFO Channel 04/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:829:1369 [3] NCCL INFO Channel 03/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:830:1370 [4] NCCL INFO Channel 05/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:832:1367 [6] NCCL INFO Channel 05/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:831:1371 [5] NCCL INFO Channel 03/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:829:1369 [3] NCCL INFO Channel 04/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:830:1370 [4] NCCL INFO Channel 06/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:832:1367 [6] NCCL INFO Channel 06/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:829:1369 [3] NCCL INFO Channel 05/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:831:1371 [5] NCCL INFO Channel 04/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:830:1370 [4] NCCL INFO Channel 07/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:832:1367 [6] NCCL INFO Channel 07/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:829:1369 [3] NCCL INFO Channel 06/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:831:1371 [5] NCCL INFO Channel 05/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:830:1370 [4] NCCL INFO Channel 08/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:832:1367 [6] NCCL INFO Channel 08/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:829:1369 [3] NCCL INFO Channel 07/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:831:1371 [5] NCCL INFO Channel 06/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:830:1370 [4] NCCL INFO Channel 09/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:832:1367 [6] NCCL INFO Channel 09/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:829:1369 [3] NCCL INFO Channel 08/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:831:1371 [5] NCCL INFO Channel 07/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:830:1370 [4] NCCL INFO Channel 10/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:832:1367 [6] NCCL INFO Channel 10/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:829:1369 [3] NCCL INFO Channel 09/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:827:1368 [1] NCCL INFO Channel 00/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:831:1371 [5] NCCL INFO Channel 08/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:830:1370 [4] NCCL INFO Channel 11/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:832:1367 [6] NCCL INFO Channel 11/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:829:1369 [3] NCCL INFO Channel 10/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:827:1368 [1] NCCL INFO Channel 01/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:831:1371 [5] NCCL INFO Channel 09/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:830:1370 [4] NCCL INFO Channel 12/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:832:1367 [6] NCCL INFO Channel 12/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:829:1369 [3] NCCL INFO Channel 11/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:827:1368 [1] NCCL INFO Channel 02/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:831:1371 [5] NCCL INFO Channel 10/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:830:1370 [4] NCCL INFO Channel 13/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:832:1367 [6] NCCL INFO Channel 13/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:829:1369 [3] NCCL INFO Channel 12/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:827:1368 [1] NCCL INFO Channel 03/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:831:1371 [5] NCCL INFO Channel 11/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:830:1370 [4] NCCL INFO Channel 14/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:832:1367 [6] NCCL INFO Channel 14/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:829:1369 [3] NCCL INFO Channel 13/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:827:1368 [1] NCCL INFO Channel 04/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:831:1371 [5] NCCL INFO Channel 12/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:830:1370 [4] NCCL INFO Channel 15/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:832:1367 [6] NCCL INFO Channel 15/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:829:1369 [3] NCCL INFO Channel 14/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:827:1368 [1] NCCL INFO Channel 05/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:831:1371 [5] NCCL INFO Channel 13/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:830:1370 [4] NCCL INFO Channel 16/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:828:1372 [2] NCCL INFO Channel 00/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:832:1367 [6] NCCL INFO Channel 16/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:829:1369 [3] NCCL INFO Channel 15/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:827:1368 [1] NCCL INFO Channel 06/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:830:1370 [4] NCCL INFO Channel 17/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:831:1371 [5] NCCL INFO Channel 14/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:832:1367 [6] NCCL INFO Channel 17/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:828:1372 [2] NCCL INFO Channel 01/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:829:1369 [3] NCCL INFO Channel 16/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:827:1368 [1] NCCL INFO Channel 07/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:830:1370 [4] NCCL INFO Channel 18/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:831:1371 [5] NCCL INFO Channel 15/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:832:1367 [6] NCCL INFO Channel 18/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:829:1369 [3] NCCL INFO Channel 17/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:828:1372 [2] NCCL INFO Channel 02/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:827:1368 [1] NCCL INFO Channel 08/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:830:1370 [4] NCCL INFO Channel 19/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:831:1371 [5] NCCL INFO Channel 16/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:829:1369 [3] NCCL INFO Channel 18/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:832:1367 [6] NCCL INFO Channel 19/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:828:1372 [2] NCCL INFO Channel 03/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:827:1368 [1] NCCL INFO Channel 09/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:830:1370 [4] NCCL INFO Channel 20/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:831:1371 [5] NCCL INFO Channel 17/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:829:1369 [3] NCCL INFO Channel 19/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:832:1367 [6] NCCL INFO Channel 20/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:828:1372 [2] NCCL INFO Channel 04/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:827:1368 [1] NCCL INFO Channel 10/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:830:1370 [4] NCCL INFO Channel 21/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:831:1371 [5] NCCL INFO Channel 18/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:829:1369 [3] NCCL INFO Channel 20/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:832:1367 [6] NCCL INFO Channel 21/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:828:1372 [2] NCCL INFO Channel 05/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:827:1368 [1] NCCL INFO Channel 11/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:830:1370 [4] NCCL INFO Channel 22/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:831:1371 [5] NCCL INFO Channel 19/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:829:1369 [3] NCCL INFO Channel 21/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:832:1367 [6] NCCL INFO Channel 22/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:828:1372 [2] NCCL INFO Channel 06/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:827:1368 [1] NCCL INFO Channel 12/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:830:1370 [4] NCCL INFO Channel 23/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:831:1371 [5] NCCL INFO Channel 20/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:829:1369 [3] NCCL INFO Channel 22/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:832:1367 [6] NCCL INFO Channel 23/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:828:1372 [2] NCCL INFO Channel 07/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:827:1368 [1] NCCL INFO Channel 13/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:829:1369 [3] NCCL INFO Channel 23/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:831:1371 [5] NCCL INFO Channel 21/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:828:1372 [2] NCCL INFO Channel 08/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:827:1368 [1] NCCL INFO Channel 14/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:831:1371 [5] NCCL INFO Channel 22/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:828:1372 [2] NCCL INFO Channel 09/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:827:1368 [1] NCCL INFO Channel 15/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:831:1371 [5] NCCL INFO Channel 23/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:827:1368 [1] NCCL INFO Channel 16/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:828:1372 [2] NCCL INFO Channel 10/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:827:1368 [1] NCCL INFO Channel 17/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:828:1372 [2] NCCL INFO Channel 11/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:827:1368 [1] NCCL INFO Channel 18/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:828:1372 [2] NCCL INFO Channel 12/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:827:1368 [1] NCCL INFO Channel 19/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:828:1372 [2] NCCL INFO Channel 13/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:827:1368 [1] NCCL INFO Channel 20/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:828:1372 [2] NCCL INFO Channel 14/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:827:1368 [1] NCCL INFO Channel 21/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:828:1372 [2] NCCL INFO Channel 15/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:827:1368 [1] NCCL INFO Channel 22/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:828:1372 [2] NCCL INFO Channel 16/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:827:1368 [1] NCCL INFO Channel 23/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:828:1372 [2] NCCL INFO Channel 17/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:828:1372 [2] NCCL INFO Channel 18/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:828:1372 [2] NCCL INFO Channel 19/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:828:1372 [2] NCCL INFO Channel 20/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:828:1372 [2] NCCL INFO Channel 21/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:828:1372 [2] NCCL INFO Channel 22/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:828:1372 [2] NCCL INFO Channel 23/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:831:1371 [5] NCCL INFO Connected all trees | |
| tyler-rhel-newimage:831:1371 [5] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512 | |
| tyler-rhel-newimage:831:1371 [5] NCCL INFO 24 coll channels, 0 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer | |
| tyler-rhel-newimage:833:1373 [7] NCCL INFO Connected all trees | |
| tyler-rhel-newimage:833:1373 [7] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512 | |
| tyler-rhel-newimage:833:1373 [7] NCCL INFO 24 coll channels, 0 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer | |
| tyler-rhel-newimage:832:1367 [6] NCCL INFO Connected all trees | |
| tyler-rhel-newimage:832:1367 [6] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512 | |
| tyler-rhel-newimage:832:1367 [6] NCCL INFO 24 coll channels, 0 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer | |
| tyler-rhel-newimage:830:1370 [4] NCCL INFO Connected all trees | |
| tyler-rhel-newimage:830:1370 [4] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512 | |
| tyler-rhel-newimage:830:1370 [4] NCCL INFO 24 coll channels, 0 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer | |
| tyler-rhel-newimage:804:1366 [0] NCCL INFO Connected all trees | |
| tyler-rhel-newimage:804:1366 [0] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512 | |
| tyler-rhel-newimage:804:1366 [0] NCCL INFO 24 coll channels, 0 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer | |
| tyler-rhel-newimage:827:1368 [1] NCCL INFO Connected all trees | |
| tyler-rhel-newimage:829:1369 [3] NCCL INFO Connected all trees | |
| tyler-rhel-newimage:827:1368 [1] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512 | |
| tyler-rhel-newimage:827:1368 [1] NCCL INFO 24 coll channels, 0 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer | |
| tyler-rhel-newimage:829:1369 [3] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512 | |
| tyler-rhel-newimage:829:1369 [3] NCCL INFO 24 coll channels, 0 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer | |
| tyler-rhel-newimage:828:1372 [2] NCCL INFO Connected all trees | |
| tyler-rhel-newimage:828:1372 [2] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512 | |
| tyler-rhel-newimage:828:1372 [2] NCCL INFO 24 coll channels, 0 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer | |
| tyler-rhel-newimage:830:1370 [4] NCCL INFO comm 0x556ddbc69c20 rank 4 nranks 8 cudaDev 4 nvmlDev 4 busId c050 commId 0x9035024dada8a21d - Init COMPLETE | |
| tyler-rhel-newimage:831:1371 [5] NCCL INFO comm 0x556ddbc6a210 rank 5 nranks 8 cudaDev 5 nvmlDev 5 busId c060 commId 0x9035024dada8a21d - Init COMPLETE | |
| tyler-rhel-newimage:832:1367 [6] NCCL INFO comm 0x556ddbc6b1e0 rank 6 nranks 8 cudaDev 6 nvmlDev 6 busId e070 commId 0x9035024dada8a21d - Init COMPLETE | |
| tyler-rhel-newimage:829:1369 [3] NCCL INFO comm 0x556ddbc68d70 rank 3 nranks 8 cudaDev 3 nvmlDev 3 busId a040 commId 0x9035024dada8a21d - Init COMPLETE | |
| tyler-rhel-newimage:828:1372 [2] NCCL INFO comm 0x556ddbc69260 rank 2 nranks 8 cudaDev 2 nvmlDev 2 busId a030 commId 0x9035024dada8a21d - Init COMPLETE | |
| tyler-rhel-newimage:827:1368 [1] NCCL INFO comm 0x556ddbc697b0 rank 1 nranks 8 cudaDev 1 nvmlDev 1 busId 8020 commId 0x9035024dada8a21d - Init COMPLETE | |
| tyler-rhel-newimage:804:1366 [0] NCCL INFO comm 0x556ddbc45f80 rank 0 nranks 8 cudaDev 0 nvmlDev 0 busId 8010 commId 0x9035024dada8a21d - Init COMPLETE | |
| tyler-rhel-newimage:833:1373 [7] NCCL INFO comm 0x556ddbc53d80 rank 7 nranks 8 cudaDev 7 nvmlDev 7 busId e080 commId 0x9035024dada8a21d - Init COMPLETE | |
| tyler-rhel-newimage:827:1399 [1] NCCL INFO Channel 00/1 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:829:1398 [3] NCCL INFO Channel 00/1 : 3[3] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:832:1400 [6] NCCL INFO Channel 00/1 : 6[6] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:830:1401 [4] NCCL INFO Channel 00/1 : 4[4] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:828:1402 [2] NCCL INFO Channel 00/1 : 2[2] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:829:1398 [3] NCCL INFO Channel 01/1 : 3[3] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:831:1403 [5] NCCL INFO Channel 00/1 : 5[5] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:827:1399 [1] NCCL INFO Channel 01/1 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:832:1400 [6] NCCL INFO Channel 01/1 : 6[6] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:830:1401 [4] NCCL INFO Channel 01/1 : 4[4] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:832:1400 [6] NCCL INFO Channel 02/1 : 6[6] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:829:1398 [3] NCCL INFO Channel 02/1 : 3[3] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:828:1402 [2] NCCL INFO Channel 01/1 : 2[2] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:833:1404 [7] NCCL INFO Channel 00/1 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:827:1399 [1] NCCL INFO Channel 02/1 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:830:1401 [4] NCCL INFO Channel 02/1 : 4[4] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:831:1403 [5] NCCL INFO Channel 01/1 : 5[5] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:832:1400 [6] NCCL INFO Channel 03/1 : 6[6] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:830:1401 [4] NCCL INFO Channel 03/1 : 4[4] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:833:1404 [7] NCCL INFO Channel 01/1 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:832:1400 [6] NCCL INFO Channel 04/1 : 6[6] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:831:1403 [5] NCCL INFO Channel 02/1 : 5[5] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:830:1401 [4] NCCL INFO Channel 04/1 : 4[4] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:827:1399 [1] NCCL INFO Channel 03/1 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:833:1404 [7] NCCL INFO Channel 02/1 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:832:1400 [6] NCCL INFO Channel 05/1 : 6[6] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:831:1403 [5] NCCL INFO Channel 03/1 : 5[5] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:830:1401 [4] NCCL INFO Channel 05/1 : 4[4] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:827:1399 [1] NCCL INFO Channel 04/1 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:833:1404 [7] NCCL INFO Channel 03/1 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:832:1400 [6] NCCL INFO Channel 06/1 : 6[6] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:831:1403 [5] NCCL INFO Channel 04/1 : 5[5] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:830:1401 [4] NCCL INFO Channel 06/1 : 4[4] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:827:1399 [1] NCCL INFO Channel 05/1 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:833:1404 [7] NCCL INFO Channel 04/1 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:832:1400 [6] NCCL INFO Channel 07/1 : 6[6] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:831:1403 [5] NCCL INFO Channel 05/1 : 5[5] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:830:1401 [4] NCCL INFO Channel 07/1 : 4[4] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:827:1399 [1] NCCL INFO Channel 06/1 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:833:1404 [7] NCCL INFO Channel 05/1 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:829:1398 [3] NCCL INFO Channel 03/1 : 3[3] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:832:1400 [6] NCCL INFO Channel 08/1 : 6[6] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:831:1403 [5] NCCL INFO Channel 06/1 : 5[5] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:830:1401 [4] NCCL INFO Channel 08/1 : 4[4] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:827:1399 [1] NCCL INFO Channel 07/1 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:833:1404 [7] NCCL INFO Channel 06/1 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:829:1398 [3] NCCL INFO Channel 04/1 : 3[3] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:832:1400 [6] NCCL INFO Channel 09/1 : 6[6] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:830:1401 [4] NCCL INFO Channel 09/1 : 4[4] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:831:1403 [5] NCCL INFO Channel 07/1 : 5[5] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:827:1399 [1] NCCL INFO Channel 08/1 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:829:1398 [3] NCCL INFO Channel 05/1 : 3[3] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:833:1404 [7] NCCL INFO Channel 07/1 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:832:1400 [6] NCCL INFO Channel 10/1 : 6[6] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:828:1402 [2] NCCL INFO Channel 02/1 : 2[2] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:830:1401 [4] NCCL INFO Channel 10/1 : 4[4] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:827:1399 [1] NCCL INFO Channel 09/1 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:831:1403 [5] NCCL INFO Channel 08/1 : 5[5] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:829:1398 [3] NCCL INFO Channel 06/1 : 3[3] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:832:1400 [6] NCCL INFO Channel 11/1 : 6[6] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:833:1404 [7] NCCL INFO Channel 08/1 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:828:1402 [2] NCCL INFO Channel 03/1 : 2[2] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:830:1401 [4] NCCL INFO Channel 11/1 : 4[4] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:827:1399 [1] NCCL INFO Channel 10/1 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:829:1398 [3] NCCL INFO Channel 07/1 : 3[3] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:831:1403 [5] NCCL INFO Channel 09/1 : 5[5] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:832:1400 [6] NCCL INFO Channel 12/1 : 6[6] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:833:1404 [7] NCCL INFO Channel 09/1 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:828:1402 [2] NCCL INFO Channel 04/1 : 2[2] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:830:1401 [4] NCCL INFO Channel 12/1 : 4[4] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:827:1399 [1] NCCL INFO Channel 11/1 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:829:1398 [3] NCCL INFO Channel 08/1 : 3[3] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:831:1403 [5] NCCL INFO Channel 10/1 : 5[5] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:832:1400 [6] NCCL INFO Channel 13/1 : 6[6] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:833:1404 [7] NCCL INFO Channel 10/1 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:828:1402 [2] NCCL INFO Channel 05/1 : 2[2] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:830:1401 [4] NCCL INFO Channel 13/1 : 4[4] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:827:1399 [1] NCCL INFO Channel 12/1 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:829:1398 [3] NCCL INFO Channel 09/1 : 3[3] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:831:1403 [5] NCCL INFO Channel 11/1 : 5[5] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:832:1400 [6] NCCL INFO Channel 14/1 : 6[6] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:833:1404 [7] NCCL INFO Channel 11/1 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:830:1401 [4] NCCL INFO Channel 14/1 : 4[4] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:827:1399 [1] NCCL INFO Channel 13/1 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:828:1402 [2] NCCL INFO Channel 06/1 : 2[2] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:829:1398 [3] NCCL INFO Channel 10/1 : 3[3] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:832:1400 [6] NCCL INFO Channel 15/1 : 6[6] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:831:1403 [5] NCCL INFO Channel 12/1 : 5[5] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:833:1404 [7] NCCL INFO Channel 12/1 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:830:1401 [4] NCCL INFO Channel 15/1 : 4[4] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:827:1399 [1] NCCL INFO Channel 14/1 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:829:1398 [3] NCCL INFO Channel 11/1 : 3[3] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:828:1402 [2] NCCL INFO Channel 07/1 : 2[2] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:831:1403 [5] NCCL INFO Channel 13/1 : 5[5] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:832:1400 [6] NCCL INFO Channel 16/1 : 6[6] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:833:1404 [7] NCCL INFO Channel 13/1 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:830:1401 [4] NCCL INFO Channel 16/1 : 4[4] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:827:1399 [1] NCCL INFO Channel 15/1 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:829:1398 [3] NCCL INFO Channel 12/1 : 3[3] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:828:1402 [2] NCCL INFO Channel 08/1 : 2[2] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:831:1403 [5] NCCL INFO Channel 14/1 : 5[5] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:832:1400 [6] NCCL INFO Channel 17/1 : 6[6] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:833:1404 [7] NCCL INFO Channel 14/1 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:830:1401 [4] NCCL INFO Channel 17/1 : 4[4] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:827:1399 [1] NCCL INFO Channel 16/1 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:829:1398 [3] NCCL INFO Channel 13/1 : 3[3] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:828:1402 [2] NCCL INFO Channel 09/1 : 2[2] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:832:1400 [6] NCCL INFO Channel 18/1 : 6[6] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:833:1404 [7] NCCL INFO Channel 15/1 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:831:1403 [5] NCCL INFO Channel 15/1 : 5[5] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:830:1401 [4] NCCL INFO Channel 18/1 : 4[4] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:827:1399 [1] NCCL INFO Channel 17/1 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:829:1398 [3] NCCL INFO Channel 14/1 : 3[3] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:832:1400 [6] NCCL INFO Channel 19/1 : 6[6] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:828:1402 [2] NCCL INFO Channel 10/1 : 2[2] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:833:1404 [7] NCCL INFO Channel 16/1 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:831:1403 [5] NCCL INFO Channel 16/1 : 5[5] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:830:1401 [4] NCCL INFO Channel 19/1 : 4[4] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:827:1399 [1] NCCL INFO Channel 18/1 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:829:1398 [3] NCCL INFO Channel 15/1 : 3[3] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:832:1400 [6] NCCL INFO Channel 20/1 : 6[6] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:833:1404 [7] NCCL INFO Channel 17/1 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:831:1403 [5] NCCL INFO Channel 17/1 : 5[5] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:828:1402 [2] NCCL INFO Channel 11/1 : 2[2] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:830:1401 [4] NCCL INFO Channel 20/1 : 4[4] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:827:1399 [1] NCCL INFO Channel 19/1 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:829:1398 [3] NCCL INFO Channel 16/1 : 3[3] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:832:1400 [6] NCCL INFO Channel 21/1 : 6[6] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:833:1404 [7] NCCL INFO Channel 18/1 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:831:1403 [5] NCCL INFO Channel 18/1 : 5[5] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:828:1402 [2] NCCL INFO Channel 12/1 : 2[2] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:830:1401 [4] NCCL INFO Channel 21/1 : 4[4] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:827:1399 [1] NCCL INFO Channel 20/1 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:829:1398 [3] NCCL INFO Channel 17/1 : 3[3] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:832:1400 [6] NCCL INFO Channel 22/1 : 6[6] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:833:1404 [7] NCCL INFO Channel 19/1 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:831:1403 [5] NCCL INFO Channel 19/1 : 5[5] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:830:1401 [4] NCCL INFO Channel 22/1 : 4[4] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:827:1399 [1] NCCL INFO Channel 21/1 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:829:1398 [3] NCCL INFO Channel 18/1 : 3[3] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:828:1402 [2] NCCL INFO Channel 13/1 : 2[2] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:832:1400 [6] NCCL INFO Channel 23/1 : 6[6] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:833:1404 [7] NCCL INFO Channel 20/1 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:831:1403 [5] NCCL INFO Channel 20/1 : 5[5] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:830:1401 [4] NCCL INFO Channel 23/1 : 4[4] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:827:1399 [1] NCCL INFO Channel 22/1 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:829:1398 [3] NCCL INFO Channel 19/1 : 3[3] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:828:1402 [2] NCCL INFO Channel 14/1 : 2[2] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:832:1400 [6] NCCL INFO Channel 24/1 : 6[6] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:833:1404 [7] NCCL INFO Channel 21/1 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:830:1401 [4] NCCL INFO Channel 24/1 : 4[4] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:827:1399 [1] NCCL INFO Channel 23/1 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:831:1403 [5] NCCL INFO Channel 21/1 : 5[5] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:829:1398 [3] NCCL INFO Channel 20/1 : 3[3] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:828:1402 [2] NCCL INFO Channel 15/1 : 2[2] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:832:1400 [6] NCCL INFO Channel 25/1 : 6[6] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:833:1404 [7] NCCL INFO Channel 22/1 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:830:1401 [4] NCCL INFO Channel 25/1 : 4[4] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:827:1399 [1] NCCL INFO Channel 24/1 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:829:1398 [3] NCCL INFO Channel 21/1 : 3[3] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:831:1403 [5] NCCL INFO Channel 22/1 : 5[5] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:828:1402 [2] NCCL INFO Channel 16/1 : 2[2] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:832:1400 [6] NCCL INFO Channel 26/1 : 6[6] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:833:1404 [7] NCCL INFO Channel 23/1 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:830:1401 [4] NCCL INFO Channel 26/1 : 4[4] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:827:1399 [1] NCCL INFO Channel 25/1 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:829:1398 [3] NCCL INFO Channel 22/1 : 3[3] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:831:1403 [5] NCCL INFO Channel 23/1 : 5[5] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:832:1400 [6] NCCL INFO Channel 27/1 : 6[6] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:828:1402 [2] NCCL INFO Channel 17/1 : 2[2] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:833:1404 [7] NCCL INFO Channel 24/1 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:830:1401 [4] NCCL INFO Channel 27/1 : 4[4] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:827:1399 [1] NCCL INFO Channel 26/1 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:829:1398 [3] NCCL INFO Channel 23/1 : 3[3] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:831:1403 [5] NCCL INFO Channel 24/1 : 5[5] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:832:1400 [6] NCCL INFO Channel 28/1 : 6[6] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:833:1404 [7] NCCL INFO Channel 25/1 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:828:1402 [2] NCCL INFO Channel 18/1 : 2[2] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:830:1401 [4] NCCL INFO Channel 28/1 : 4[4] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:827:1399 [1] NCCL INFO Channel 27/1 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:829:1398 [3] NCCL INFO Channel 24/1 : 3[3] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:831:1403 [5] NCCL INFO Channel 25/1 : 5[5] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:832:1400 [6] NCCL INFO Channel 29/1 : 6[6] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:833:1404 [7] NCCL INFO Channel 26/1 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:830:1401 [4] NCCL INFO Channel 29/1 : 4[4] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:828:1402 [2] NCCL INFO Channel 19/1 : 2[2] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:827:1399 [1] NCCL INFO Channel 28/1 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:829:1398 [3] NCCL INFO Channel 25/1 : 3[3] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:832:1400 [6] NCCL INFO Channel 30/1 : 6[6] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:831:1403 [5] NCCL INFO Channel 26/1 : 5[5] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:830:1401 [4] NCCL INFO Channel 30/1 : 4[4] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:833:1404 [7] NCCL INFO Channel 27/1 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:827:1399 [1] NCCL INFO Channel 29/1 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:828:1402 [2] NCCL INFO Channel 20/1 : 2[2] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:829:1398 [3] NCCL INFO Channel 26/1 : 3[3] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:832:1400 [6] NCCL INFO Channel 31/1 : 6[6] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:831:1403 [5] NCCL INFO Channel 27/1 : 5[5] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:830:1401 [4] NCCL INFO Channel 31/1 : 4[4] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:827:1399 [1] NCCL INFO Channel 30/1 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:833:1404 [7] NCCL INFO Channel 28/1 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:829:1398 [3] NCCL INFO Channel 27/1 : 3[3] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:828:1402 [2] NCCL INFO Channel 21/1 : 2[2] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:831:1403 [5] NCCL INFO Channel 28/1 : 5[5] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:827:1399 [1] NCCL INFO Channel 31/1 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:833:1404 [7] NCCL INFO Channel 29/1 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:829:1398 [3] NCCL INFO Channel 28/1 : 3[3] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:828:1402 [2] NCCL INFO Channel 22/1 : 2[2] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:831:1403 [5] NCCL INFO Channel 29/1 : 5[5] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:833:1404 [7] NCCL INFO Channel 30/1 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:829:1398 [3] NCCL INFO Channel 29/1 : 3[3] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:828:1402 [2] NCCL INFO Channel 23/1 : 2[2] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:831:1403 [5] NCCL INFO Channel 30/1 : 5[5] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:833:1404 [7] NCCL INFO Channel 31/1 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:829:1398 [3] NCCL INFO Channel 30/1 : 3[3] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:828:1402 [2] NCCL INFO Channel 24/1 : 2[2] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:829:1398 [3] NCCL INFO Channel 31/1 : 3[3] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:831:1403 [5] NCCL INFO Channel 31/1 : 5[5] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:828:1402 [2] NCCL INFO Channel 25/1 : 2[2] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:828:1402 [2] NCCL INFO Channel 26/1 : 2[2] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:828:1402 [2] NCCL INFO Channel 27/1 : 2[2] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:828:1402 [2] NCCL INFO Channel 28/1 : 2[2] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:828:1402 [2] NCCL INFO Channel 29/1 : 2[2] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:828:1402 [2] NCCL INFO Channel 30/1 : 2[2] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:828:1402 [2] NCCL INFO Channel 31/1 : 2[2] -> 0[0] via P2P/IPC/read | |
| INFO 2024-07-27 16:32:56,871 instructlab.model.backends.backends:336: Waiting for the vLLM server to start at http://127.0.0.1:44547/v1, this might take a moment... Attempt: 8/80 | |
| INFO 2024-07-27 16:33:01,579 instructlab.model.backends.backends:336: Waiting for the vLLM server to start at http://127.0.0.1:44547/v1, this might take a moment... Attempt: 9/80 | |
| INFO 2024-07-27 16:33:05,965 instructlab.model.backends.backends:336: Waiting for the vLLM server to start at http://127.0.0.1:44547/v1, this might take a moment... Attempt: 10/80 | |
| INFO 2024-07-27 16:33:10,494 instructlab.model.backends.backends:336: Waiting for the vLLM server to start at http://127.0.0.1:44547/v1, this might take a moment... Attempt: 11/80 | |
| INFO 2024-07-27 16:33:15,098 instructlab.model.backends.backends:336: Waiting for the vLLM server to start at http://127.0.0.1:44547/v1, this might take a moment... Attempt: 12/80 | |
| embedding_mode is False. Embedding API will not work. | |
| INFO: Started server process [804] | |
| INFO: Waiting for application startup. | |
| INFO: Application startup complete. | |
| INFO: Uvicorn running on http://127.0.0.1:44547 (Press CTRL+C to quit) | |
| INFO: 127.0.0.1:59398 - "GET /v1/models HTTP/1.1" 200 OK | |
| INFO 2024-07-27 16:33:17,625 instructlab.model.backends.backends:343: vLLM engine successfully started at http://127.0.0.1:44547/v1 | |
| 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 147/147 [00:00<00:00, 158.63it/s] | |
| 0%| | 0/413 [00:00<?, ?it/s]INFO: 127.0.0.1:41294 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41310 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41316 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41346 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41368 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41332 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41384 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41396 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41358 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41408 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41422 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41436 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41430 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41442 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41458 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41464 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 0%|▍ | 1/413 [00:10<1:08:53, 10.03s/it]INFO: 127.0.0.1:41294 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41316 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41310 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41346 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41332 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41358 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41368 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41384 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41396 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41422 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41408 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41430 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41436 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41442 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41458 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41464 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 4%|████████▏ | 17/413 [00:19<06:38, 1.01s/it]INFO: 127.0.0.1:41294 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41310 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41316 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41346 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41332 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41358 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41368 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41384 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41396 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41422 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41408 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41430 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41436 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41442 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41458 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41464 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 8%|███████████████▉ | 33/413 [00:29<05:00, 1.27it/s]INFO: 127.0.0.1:41310 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41294 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41316 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41346 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41332 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41358 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41368 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41384 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41396 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41408 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41422 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41430 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41458 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41442 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41464 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41436 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 12%|███████████████████████▋ | 49/413 [00:39<04:17, 1.41it/s]INFO: 127.0.0.1:41294 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41310 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41316 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41346 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41358 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41332 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41368 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41384 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41396 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41422 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41436 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41430 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41408 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41442 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41464 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41458 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 16%|███████████████████████████████▍ | 65/413 [00:49<03:55, 1.48it/s]INFO: 127.0.0.1:41294 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41310 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41316 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41346 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41332 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41358 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41368 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41384 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41396 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41422 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41408 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41430 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41442 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41436 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41458 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41464 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 20%|███████████████████████████████████████▏ | 81/413 [00:59<03:36, 1.53it/s]INFO: 127.0.0.1:41294 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41316 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41310 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41346 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41368 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41358 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41332 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41384 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41422 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41396 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41408 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41430 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41436 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41442 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41458 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41464 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 23%|██████████████████████████████████████████████▉ | 97/413 [01:09<03:23, 1.55it/s]INFO: 127.0.0.1:41294 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41310 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41316 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41332 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41346 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41358 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41368 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41384 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41396 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41422 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41408 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41430 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41436 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41442 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41458 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41464 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 27%|██████████████████████████████████████████████████████▍ | 113/413 [01:19<03:10, 1.58it/s]INFO: 127.0.0.1:41294 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 31%|██████████████████████████████████████████████████████████████▏ | 129/413 [01:29<02:57, 1.60it/s]INFO: 127.0.0.1:41310 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41316 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41332 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41346 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41358 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41368 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41396 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41384 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41422 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41408 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41430 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41436 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41442 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41458 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41464 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 33%|██████████████████████████████████████████████████████████████████ | 137/413 [01:29<02:21, 1.95it/s]INFO: 127.0.0.1:41294 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 35%|█████████████████████████████████████████████████████████████████████▊ | 145/413 [01:39<02:58, 1.50it/s]INFO: 127.0.0.1:41310 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41316 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41346 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41332 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41368 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41358 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41384 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41396 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41422 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41430 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41408 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41436 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41442 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41458 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41464 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 36%|██████████████████████████████████████████████████████████████████████▊ | 147/413 [01:39<02:45, 1.60it/s]INFO: 127.0.0.1:41294 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 39%|█████████████████████████████████████████████████████████████████████████████▌ | 161/413 [01:48<02:43, 1.54it/s]INFO: 127.0.0.1:41310 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41316 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41346 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41358 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41368 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41332 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41384 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41396 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41422 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41408 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41430 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41436 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41458 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41442 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41464 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 39%|██████████████████████████████████████████████████████████████████████████████▌ | 163/413 [01:49<02:32, 1.64it/s]INFO: 127.0.0.1:41294 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 43%|█████████████████████████████████████████████████████████████████████████████████████▎ | 177/413 [01:58<02:32, 1.55it/s]INFO: 127.0.0.1:41310 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41316 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41346 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41332 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41358 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41368 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41384 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41396 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41422 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41408 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41436 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41430 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41442 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41458 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41464 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 43%|█████████████████████████████████████████████████████████████████████████████████████▊ | 178/413 [01:59<02:28, 1.59it/s]INFO: 127.0.0.1:41294 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 47%|████████████████████████████████████████████████████████████████████████████████████████████▉ | 193/413 [02:08<02:18, 1.59it/s]INFO: 127.0.0.1:41310 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41316 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41346 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41332 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41368 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41358 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41384 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41396 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41422 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41408 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41430 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41436 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41442 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41458 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41464 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 47%|█████████████████████████████████████████████████████████████████████████████████████████████▍ | 194/413 [02:08<02:15, 1.62it/s]INFO: 127.0.0.1:41294 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 51%|████████████████████████████████████████████████████████████████████████████████████████████████████▋ | 209/413 [02:18<02:08, 1.59it/s]INFO: 127.0.0.1:41310 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41316 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41346 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41358 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41332 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41368 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41384 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41396 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41422 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41408 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41430 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41436 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41442 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41458 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41464 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 51%|█████████████████████████████████████████████████████████████████████████████████████████████████████▏ | 210/413 [02:18<02:06, 1.61it/s]INFO: 127.0.0.1:41294 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 54%|████████████████████████████████████████████████████████████████████████████████████████████████████████████▍ | 225/413 [02:28<01:56, 1.61it/s]INFO: 127.0.0.1:41316 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41310 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41346 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41332 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41358 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41368 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41384 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41396 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41422 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41408 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41430 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41436 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41442 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41458 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41464 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 55%|████████████████████████████████████████████████████████████████████████████████████████████████████████████▉ | 226/413 [02:28<01:55, 1.61it/s]INFO: 127.0.0.1:41294 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 58%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████ | 241/413 [02:38<01:47, 1.60it/s]INFO: 127.0.0.1:41310 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41316 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41346 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41358 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41368 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41332 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41384 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41396 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41422 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41408 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41436 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41430 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41442 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41458 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41464 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 59%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▌ | 242/413 [02:38<01:46, 1.60it/s]INFO: 127.0.0.1:41294 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 62%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▊ | 257/413 [02:48<01:37, 1.60it/s]INFO: 127.0.0.1:41358 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41316 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41310 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41346 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41332 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41368 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41384 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41396 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41422 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41408 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41430 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41436 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41442 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41458 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41464 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 62%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▎ | 258/413 [02:48<01:36, 1.60it/s]INFO: 127.0.0.1:41294 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 66%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▌ | 273/413 [02:58<01:28, 1.59it/s]INFO: 127.0.0.1:41310 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41316 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41332 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41346 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41358 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41368 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41384 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41422 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41396 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41408 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41430 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41436 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41442 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41458 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41464 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 66%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████ | 274/413 [02:59<01:27, 1.58it/s]INFO: 127.0.0.1:41294 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 70%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▎ | 289/413 [03:08<01:16, 1.62it/s]INFO: 127.0.0.1:41310 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41316 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41346 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41332 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41358 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41368 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41384 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41422 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41396 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41430 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41408 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41436 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41442 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41458 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41464 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 70%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▋ | 290/413 [03:08<01:16, 1.60it/s]INFO: 127.0.0.1:41294 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 74%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▉ | 305/413 [03:18<01:07, 1.61it/s]INFO: 127.0.0.1:41310 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41316 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41346 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41332 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41358 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41368 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41384 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41422 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41408 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41396 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41430 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41442 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41436 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41458 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41464 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 74%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▍ | 306/413 [03:19<01:07, 1.58it/s]INFO: 127.0.0.1:41294 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 78%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▋ | 321/413 [03:27<00:56, 1.63it/s]INFO: 127.0.0.1:41310 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41346 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41316 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41332 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41358 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41368 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41384 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41422 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41396 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41408 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41430 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41436 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41442 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41464 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41458 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 78%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▏ | 322/413 [03:28<00:56, 1.60it/s]INFO: 127.0.0.1:41294 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 82%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▍ | 337/413 [03:37<00:46, 1.63it/s]INFO: 127.0.0.1:41310 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41316 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41346 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41332 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41358 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41368 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41396 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41384 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41408 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41430 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41436 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41422 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41458 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41442 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41464 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 82%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▊ | 338/413 [03:38<00:47, 1.59it/s]INFO: 127.0.0.1:41294 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 85%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████ | 353/413 [03:47<00:36, 1.65it/s]INFO: 127.0.0.1:41316 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41310 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41332 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41346 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41358 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41368 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41384 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41396 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41422 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41408 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41430 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41436 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41442 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41458 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41464 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 86%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▌ | 354/413 [03:48<00:36, 1.60it/s]INFO: 127.0.0.1:41294 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 89%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▊ | 369/413 [03:57<00:26, 1.64it/s]INFO: 127.0.0.1:41310 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41316 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41346 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41358 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41332 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41368 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41384 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41422 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41396 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41430 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41408 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41436 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41442 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41464 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41458 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 90%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▎ | 370/413 [03:58<00:27, 1.58it/s]INFO: 127.0.0.1:41294 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 93%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▌ | 385/413 [04:07<00:17, 1.64it/s]INFO: 127.0.0.1:41310 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41316 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41346 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41332 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41358 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41368 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41384 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41396 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41408 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41430 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41422 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41436 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41442 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41458 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41464 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 93%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▉ | 386/413 [04:08<00:17, 1.57it/s]INFO: 127.0.0.1:41294 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 97%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▏ | 401/413 [04:16<00:07, 1.68it/s]INFO: 127.0.0.1:41332 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41346 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41316 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41310 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41358 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41368 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41384 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41396 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41422 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41430 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41408 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:41436 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 413/413 [04:18<00:00, 1.60it/s] | |
| INFO: Shutting down | |
| INFO: Waiting for application shutdown. | |
| INFO: Application shutdown complete. | |
| INFO: Finished server process [804] | |
| [rank0]:[W CudaIPCTypes.cpp:16] Producer process has been terminated before all shared CUDA tensors released. See Note [Sharing CUDA tensors] | |
| /usr/lib64/python3.11/multiprocessing/resource_tracker.py:254: UserWarning: resource_tracker: There appear to be 1 leaked shared_memory objects to clean up at shutdown | |
| warnings.warn('resource_tracker: There appear to be %d ' | |
| WARNING 2024-07-27 16:37:53,517 instructlab.model.evaluate:288: Based on your hardware configuration, when using vLLM, we recommend setting max-workers between 53.0 and 160.0 for optimal performance | |
| INFO 2024-07-27 16:37:53,579 instructlab.model.backends.backends:315: Trying to connect to model server at http://127.0.0.1:8000/v1 | |
| INFO 2024-07-27 16:37:56,336 instructlab.model.backends.vllm:196: vLLM starting up on pid 1521 at http://127.0.0.1:57407/v1 | |
| INFO 2024-07-27 16:37:56,336 instructlab.model.backends.backends:328: Starting a temporary vLLM server at http://127.0.0.1:57407/v1 | |
| INFO 2024-07-27 16:37:56,336 instructlab.model.backends.backends:336: Waiting for the vLLM server to start at http://127.0.0.1:57407/v1, this might take a moment... Attempt: 1/80 | |
| INFO 2024-07-27 16:38:00,840 instructlab.model.backends.backends:336: Waiting for the vLLM server to start at http://127.0.0.1:57407/v1, this might take a moment... Attempt: 2/80 | |
| INFO 2024-07-27 16:38:05,498 instructlab.model.backends.backends:336: Waiting for the vLLM server to start at http://127.0.0.1:57407/v1, this might take a moment... Attempt: 3/80 | |
| tyler-rhel-newimage:1521:1521 [0] NCCL INFO Bootstrap : Using enp8s0:192.168.48.11<0> | |
| tyler-rhel-newimage:1521:1521 [0] NCCL INFO NET/Plugin : dlerror=libnccl-net.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net.so), using internal implementation | |
| tyler-rhel-newimage:1521:1521 [0] NCCL INFO cudaDriverVersion 12040 | |
| NCCL version 2.20.5+cuda12.4 | |
| tyler-rhel-newimage:1547:1547 [4] NCCL INFO cudaDriverVersion 12040 | |
| tyler-rhel-newimage:1547:1547 [4] NCCL INFO Bootstrap : Using enp8s0:192.168.48.11<0> | |
| tyler-rhel-newimage:1547:1547 [4] NCCL INFO NET/Plugin : dlerror=libnccl-net.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net.so), using internal implementation | |
| tyler-rhel-newimage:1546:1546 [3] NCCL INFO cudaDriverVersion 12040 | |
| tyler-rhel-newimage:1546:1546 [3] NCCL INFO Bootstrap : Using enp8s0:192.168.48.11<0> | |
| tyler-rhel-newimage:1546:1546 [3] NCCL INFO NET/Plugin : dlerror=libnccl-net.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net.so), using internal implementation | |
| tyler-rhel-newimage:1548:1548 [5] NCCL INFO cudaDriverVersion 12040 | |
| tyler-rhel-newimage:1548:1548 [5] NCCL INFO Bootstrap : Using enp8s0:192.168.48.11<0> | |
| tyler-rhel-newimage:1548:1548 [5] NCCL INFO NET/Plugin : dlerror=libnccl-net.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net.so), using internal implementation | |
| tyler-rhel-newimage:1544:1544 [1] NCCL INFO cudaDriverVersion 12040 | |
| tyler-rhel-newimage:1544:1544 [1] NCCL INFO Bootstrap : Using enp8s0:192.168.48.11<0> | |
| tyler-rhel-newimage:1544:1544 [1] NCCL INFO NET/Plugin : dlerror=libnccl-net.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net.so), using internal implementation | |
| tyler-rhel-newimage:1549:1549 [6] NCCL INFO cudaDriverVersion 12040 | |
| tyler-rhel-newimage:1549:1549 [6] NCCL INFO Bootstrap : Using enp8s0:192.168.48.11<0> | |
| tyler-rhel-newimage:1549:1549 [6] NCCL INFO NET/Plugin : dlerror=libnccl-net.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net.so), using internal implementation | |
| tyler-rhel-newimage:1550:1550 [7] NCCL INFO cudaDriverVersion 12040 | |
| tyler-rhel-newimage:1550:1550 [7] NCCL INFO Bootstrap : Using enp8s0:192.168.48.11<0> | |
| tyler-rhel-newimage:1545:1545 [2] NCCL INFO cudaDriverVersion 12040 | |
| tyler-rhel-newimage:1545:1545 [2] NCCL INFO Bootstrap : Using enp8s0:192.168.48.11<0> | |
| tyler-rhel-newimage:1550:1550 [7] NCCL INFO NET/Plugin : dlerror=libnccl-net.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net.so), using internal implementation | |
| tyler-rhel-newimage:1545:1545 [2] NCCL INFO NET/Plugin : dlerror=libnccl-net.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net.so), using internal implementation | |
| tyler-rhel-newimage:1521:1521 [0] NCCL INFO NET/IB : No device found. | |
| tyler-rhel-newimage:1521:1521 [0] NCCL INFO NET/Socket : Using [0]enp8s0:192.168.48.11<0> | |
| tyler-rhel-newimage:1521:1521 [0] NCCL INFO Using non-device net plugin version 0 | |
| tyler-rhel-newimage:1521:1521 [0] NCCL INFO Using network Socket | |
| tyler-rhel-newimage:1547:1547 [4] NCCL INFO NET/IB : No device found. | |
| tyler-rhel-newimage:1547:1547 [4] NCCL INFO NET/Socket : Using [0]enp8s0:192.168.48.11<0> | |
| tyler-rhel-newimage:1547:1547 [4] NCCL INFO Using non-device net plugin version 0 | |
| tyler-rhel-newimage:1547:1547 [4] NCCL INFO Using network Socket | |
| tyler-rhel-newimage:1548:1548 [5] NCCL INFO NET/IB : No device found. | |
| tyler-rhel-newimage:1548:1548 [5] NCCL INFO NET/Socket : Using [0]enp8s0:192.168.48.11<0> | |
| tyler-rhel-newimage:1548:1548 [5] NCCL INFO Using non-device net plugin version 0 | |
| tyler-rhel-newimage:1548:1548 [5] NCCL INFO Using network Socket | |
| tyler-rhel-newimage:1549:1549 [6] NCCL INFO NET/IB : No device found. | |
| tyler-rhel-newimage:1549:1549 [6] NCCL INFO NET/Socket : Using [0]enp8s0:192.168.48.11<0> | |
| tyler-rhel-newimage:1549:1549 [6] NCCL INFO Using non-device net plugin version 0 | |
| tyler-rhel-newimage:1549:1549 [6] NCCL INFO Using network Socket | |
| tyler-rhel-newimage:1544:1544 [1] NCCL INFO NET/IB : No device found. | |
| tyler-rhel-newimage:1544:1544 [1] NCCL INFO NET/Socket : Using [0]enp8s0:192.168.48.11<0> | |
| tyler-rhel-newimage:1544:1544 [1] NCCL INFO Using non-device net plugin version 0 | |
| tyler-rhel-newimage:1544:1544 [1] NCCL INFO Using network Socket | |
| tyler-rhel-newimage:1550:1550 [7] NCCL INFO NET/IB : No device found. | |
| tyler-rhel-newimage:1550:1550 [7] NCCL INFO NET/Socket : Using [0]enp8s0:192.168.48.11<0> | |
| tyler-rhel-newimage:1550:1550 [7] NCCL INFO Using non-device net plugin version 0 | |
| tyler-rhel-newimage:1550:1550 [7] NCCL INFO Using network Socket | |
| tyler-rhel-newimage:1546:1546 [3] NCCL INFO NET/IB : No device found. | |
| tyler-rhel-newimage:1546:1546 [3] NCCL INFO NET/Socket : Using [0]enp8s0:192.168.48.11<0> | |
| tyler-rhel-newimage:1546:1546 [3] NCCL INFO Using non-device net plugin version 0 | |
| tyler-rhel-newimage:1546:1546 [3] NCCL INFO Using network Socket | |
| tyler-rhel-newimage:1545:1545 [2] NCCL INFO NET/IB : No device found. | |
| tyler-rhel-newimage:1545:1545 [2] NCCL INFO NET/Socket : Using [0]enp8s0:192.168.48.11<0> | |
| tyler-rhel-newimage:1545:1545 [2] NCCL INFO Using non-device net plugin version 0 | |
| tyler-rhel-newimage:1545:1545 [2] NCCL INFO Using network Socket | |
| tyler-rhel-newimage:1550:1550 [7] NCCL INFO comm 0x56095175e8a0 rank 7 nranks 8 cudaDev 7 nvmlDev 7 busId e080 commId 0x2c85857d0fe3f407 - Init START | |
| tyler-rhel-newimage:1544:1544 [1] NCCL INFO comm 0x56095175dfb0 rank 1 nranks 8 cudaDev 1 nvmlDev 1 busId 8020 commId 0x2c85857d0fe3f407 - Init START | |
| tyler-rhel-newimage:1546:1546 [3] NCCL INFO comm 0x56095175d780 rank 3 nranks 8 cudaDev 3 nvmlDev 3 busId a040 commId 0x2c85857d0fe3f407 - Init START | |
| tyler-rhel-newimage:1545:1545 [2] NCCL INFO comm 0x56095175d300 rank 2 nranks 8 cudaDev 2 nvmlDev 2 busId a030 commId 0x2c85857d0fe3f407 - Init START | |
| tyler-rhel-newimage:1521:1521 [0] NCCL INFO comm 0x5609517656d0 rank 0 nranks 8 cudaDev 0 nvmlDev 0 busId 8010 commId 0x2c85857d0fe3f407 - Init START | |
| tyler-rhel-newimage:1549:1549 [6] NCCL INFO comm 0x56095175f2f0 rank 6 nranks 8 cudaDev 6 nvmlDev 6 busId e070 commId 0x2c85857d0fe3f407 - Init START | |
| tyler-rhel-newimage:1547:1547 [4] NCCL INFO comm 0x56095175ec70 rank 4 nranks 8 cudaDev 4 nvmlDev 4 busId c050 commId 0x2c85857d0fe3f407 - Init START | |
| tyler-rhel-newimage:1548:1548 [5] NCCL INFO comm 0x56095175df80 rank 5 nranks 8 cudaDev 5 nvmlDev 5 busId c060 commId 0x2c85857d0fe3f407 - Init START | |
| INFO 2024-07-27 16:38:10,155 instructlab.model.backends.backends:336: Waiting for the vLLM server to start at http://127.0.0.1:57407/v1, this might take a moment... Attempt: 4/80 | |
| tyler-rhel-newimage:1521:1521 [0] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| tyler-rhel-newimage:1547:1547 [4] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| tyler-rhel-newimage:1548:1548 [5] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| tyler-rhel-newimage:1545:1545 [2] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| tyler-rhel-newimage:1544:1544 [1] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| tyler-rhel-newimage:1546:1546 [3] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| tyler-rhel-newimage:1549:1549 [6] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| tyler-rhel-newimage:1521:1521 [0] NCCL INFO Setting affinity for GPU 0 to ff,ffffffff | |
| tyler-rhel-newimage:1521:1521 [0] NCCL INFO NVLS multicast support is not available on dev 0 | |
| tyler-rhel-newimage:1544:1544 [1] NCCL INFO Setting affinity for GPU 1 to ff,ffffffff | |
| tyler-rhel-newimage:1550:1550 [7] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| tyler-rhel-newimage:1544:1544 [1] NCCL INFO NVLS multicast support is not available on dev 1 | |
| tyler-rhel-newimage:1547:1547 [4] NCCL INFO Setting affinity for GPU 4 to ffff,ffffff00,00000000 | |
| tyler-rhel-newimage:1547:1547 [4] NCCL INFO NVLS multicast support is not available on dev 4 | |
| tyler-rhel-newimage:1548:1548 [5] NCCL INFO Setting affinity for GPU 5 to ffff,ffffff00,00000000 | |
| tyler-rhel-newimage:1545:1545 [2] NCCL INFO Setting affinity for GPU 2 to ff,ffffffff | |
| tyler-rhel-newimage:1548:1548 [5] NCCL INFO NVLS multicast support is not available on dev 5 | |
| tyler-rhel-newimage:1545:1545 [2] NCCL INFO NVLS multicast support is not available on dev 2 | |
| tyler-rhel-newimage:1549:1549 [6] NCCL INFO Setting affinity for GPU 6 to ffff,ffffff00,00000000 | |
| tyler-rhel-newimage:1549:1549 [6] NCCL INFO NVLS multicast support is not available on dev 6 | |
| tyler-rhel-newimage:1546:1546 [3] NCCL INFO Setting affinity for GPU 3 to ff,ffffffff | |
| tyler-rhel-newimage:1546:1546 [3] NCCL INFO NVLS multicast support is not available on dev 3 | |
| tyler-rhel-newimage:1550:1550 [7] NCCL INFO Setting affinity for GPU 7 to ffff,ffffff00,00000000 | |
| tyler-rhel-newimage:1550:1550 [7] NCCL INFO NVLS multicast support is not available on dev 7 | |
| tyler-rhel-newimage:1549:1549 [6] NCCL INFO comm 0x56095175f2f0 rank 6 nRanks 8 nNodes 1 localRanks 8 localRank 6 MNNVL 0 | |
| tyler-rhel-newimage:1544:1544 [1] NCCL INFO comm 0x56095175dfb0 rank 1 nRanks 8 nNodes 1 localRanks 8 localRank 1 MNNVL 0 | |
| tyler-rhel-newimage:1550:1550 [7] NCCL INFO comm 0x56095175e8a0 rank 7 nRanks 8 nNodes 1 localRanks 8 localRank 7 MNNVL 0 | |
| tyler-rhel-newimage:1545:1545 [2] NCCL INFO comm 0x56095175d300 rank 2 nRanks 8 nNodes 1 localRanks 8 localRank 2 MNNVL 0 | |
| tyler-rhel-newimage:1548:1548 [5] NCCL INFO comm 0x56095175df80 rank 5 nRanks 8 nNodes 1 localRanks 8 localRank 5 MNNVL 0 | |
| tyler-rhel-newimage:1521:1521 [0] NCCL INFO comm 0x5609517656d0 rank 0 nRanks 8 nNodes 1 localRanks 8 localRank 0 MNNVL 0 | |
| tyler-rhel-newimage:1546:1546 [3] NCCL INFO comm 0x56095175d780 rank 3 nRanks 8 nNodes 1 localRanks 8 localRank 3 MNNVL 0 | |
| tyler-rhel-newimage:1547:1547 [4] NCCL INFO comm 0x56095175ec70 rank 4 nRanks 8 nNodes 1 localRanks 8 localRank 4 MNNVL 0 | |
| tyler-rhel-newimage:1549:1549 [6] NCCL INFO Trees [0] 7/-1/-1->6->5 [1] 7/-1/-1->6->5 [2] 7/-1/-1->6->5 [3] 7/-1/-1->6->5 [4] 7/-1/-1->6->5 [5] 7/-1/-1->6->5 [6] 7/-1/-1->6->5 [7] 7/-1/-1->6->5 [8] 7/-1/-1->6->5 [9] 7/-1/-1->6->5 [10] 7/-1/-1->6->5 [11] 7/-1/-1->6->5 [12] 7/-1/-1->6->5 [13] 7/-1/-1->6->5 [14] 7/-1/-1->6->5 [15] 7/-1/-1->6->5 [16] 7/-1/-1->6->5 [17] 7/-1/-1->6->5 [18] 7/-1/-1->6->5 [19] 7/-1/-1->6->5 [20] 7/-1/-1->6->5 [21] 7/-1/-1->6->5 [22] 7/-1/-1->6->5 [23] 7/-1/-1->6->5 | |
| tyler-rhel-newimage:1544:1544 [1] NCCL INFO Trees [0] 2/-1/-1->1->0 [1] 2/-1/-1->1->0 [2] 2/-1/-1->1->0 [3] 2/-1/-1->1->0 [4] 2/-1/-1->1->0 [5] 2/-1/-1->1->0 [6] 2/-1/-1->1->0 [7] 2/-1/-1->1->0 [8] 2/-1/-1->1->0 [9] 2/-1/-1->1->0 [10] 2/-1/-1->1->0 [11] 2/-1/-1->1->0 [12] 2/-1/-1->1->0 [13] 2/-1/-1->1->0 [14] 2/-1/-1->1->0 [15] 2/-1/-1->1->0 [16] 2/-1/-1->1->0 [17] 2/-1/-1->1->0 [18] 2/-1/-1->1->0 [19] 2/-1/-1->1->0 [20] 2/-1/-1->1->0 [21] 2/-1/-1->1->0 [22] 2/-1/-1->1->0 [23] 2/-1/-1->1->0 | |
| tyler-rhel-newimage:1521:1521 [0] NCCL INFO Channel 00/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:1549:1549 [6] NCCL INFO P2P Chunksize set to 524288 | |
| tyler-rhel-newimage:1544:1544 [1] NCCL INFO P2P Chunksize set to 524288 | |
| tyler-rhel-newimage:1521:1521 [0] NCCL INFO Channel 01/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:1548:1548 [5] NCCL INFO Trees [0] 6/-1/-1->5->4 [1] 6/-1/-1->5->4 [2] 6/-1/-1->5->4 [3] 6/-1/-1->5->4 [4] 6/-1/-1->5->4 [5] 6/-1/-1->5->4 [6] 6/-1/-1->5->4 [7] 6/-1/-1->5->4 [8] 6/-1/-1->5->4 [9] 6/-1/-1->5->4 [10] 6/-1/-1->5->4 [11] 6/-1/-1->5->4 [12] 6/-1/-1->5->4 [13] 6/-1/-1->5->4 [14] 6/-1/-1->5->4 [15] 6/-1/-1->5->4 [16] 6/-1/-1->5->4 [17] 6/-1/-1->5->4 [18] 6/-1/-1->5->4 [19] 6/-1/-1->5->4 [20] 6/-1/-1->5->4 [21] 6/-1/-1->5->4 [22] 6/-1/-1->5->4 [23] 6/-1/-1->5->4 | |
| tyler-rhel-newimage:1545:1545 [2] NCCL INFO Trees [0] 3/-1/-1->2->1 [1] 3/-1/-1->2->1 [2] 3/-1/-1->2->1 [3] 3/-1/-1->2->1 [4] 3/-1/-1->2->1 [5] 3/-1/-1->2->1 [6] 3/-1/-1->2->1 [7] 3/-1/-1->2->1 [8] 3/-1/-1->2->1 [9] 3/-1/-1->2->1 [10] 3/-1/-1->2->1 [11] 3/-1/-1->2->1 [12] 3/-1/-1->2->1 [13] 3/-1/-1->2->1 [14] 3/-1/-1->2->1 [15] 3/-1/-1->2->1 [16] 3/-1/-1->2->1 [17] 3/-1/-1->2->1 [18] 3/-1/-1->2->1 [19] 3/-1/-1->2->1 [20] 3/-1/-1->2->1 [21] 3/-1/-1->2->1 [22] 3/-1/-1->2->1 [23] 3/-1/-1->2->1 | |
| tyler-rhel-newimage:1546:1546 [3] NCCL INFO Trees [0] 4/-1/-1->3->2 [1] 4/-1/-1->3->2 [2] 4/-1/-1->3->2 [3] 4/-1/-1->3->2 [4] 4/-1/-1->3->2 [5] 4/-1/-1->3->2 [6] 4/-1/-1->3->2 [7] 4/-1/-1->3->2 [8] 4/-1/-1->3->2 [9] 4/-1/-1->3->2 [10] 4/-1/-1->3->2 [11] 4/-1/-1->3->2 [12] 4/-1/-1->3->2 [13] 4/-1/-1->3->2 [14] 4/-1/-1->3->2 [15] 4/-1/-1->3->2 [16] 4/-1/-1->3->2 [17] 4/-1/-1->3->2 [18] 4/-1/-1->3->2 [19] 4/-1/-1->3->2 [20] 4/-1/-1->3->2 [21] 4/-1/-1->3->2 [22] 4/-1/-1->3->2 [23] 4/-1/-1->3->2 | |
| tyler-rhel-newimage:1550:1550 [7] NCCL INFO Trees [0] -1/-1/-1->7->6 [1] -1/-1/-1->7->6 [2] -1/-1/-1->7->6 [3] -1/-1/-1->7->6 [4] -1/-1/-1->7->6 [5] -1/-1/-1->7->6 [6] -1/-1/-1->7->6 [7] -1/-1/-1->7->6 [8] -1/-1/-1->7->6 [9] -1/-1/-1->7->6 [10] -1/-1/-1->7->6 [11] -1/-1/-1->7->6 [12] -1/-1/-1->7->6 [13] -1/-1/-1->7->6 [14] -1/-1/-1->7->6 [15] -1/-1/-1->7->6 [16] -1/-1/-1->7->6 [17] -1/-1/-1->7->6 [18] -1/-1/-1->7->6 [19] -1/-1/-1->7->6 [20] -1/-1/-1->7->6 [21] -1/-1/-1->7->6 [22] -1/-1/-1->7->6 [23] -1/-1/-1->7->6 | |
| tyler-rhel-newimage:1547:1547 [4] NCCL INFO Trees [0] 5/-1/-1->4->3 [1] 5/-1/-1->4->3 [2] 5/-1/-1->4->3 [3] 5/-1/-1->4->3 [4] 5/-1/-1->4->3 [5] 5/-1/-1->4->3 [6] 5/-1/-1->4->3 [7] 5/-1/-1->4->3 [8] 5/-1/-1->4->3 [9] 5/-1/-1->4->3 [10] 5/-1/-1->4->3 [11] 5/-1/-1->4->3 [12] 5/-1/-1->4->3 [13] 5/-1/-1->4->3 [14] 5/-1/-1->4->3 [15] 5/-1/-1->4->3 [16] 5/-1/-1->4->3 [17] 5/-1/-1->4->3 [18] 5/-1/-1->4->3 [19] 5/-1/-1->4->3 [20] 5/-1/-1->4->3 [21] 5/-1/-1->4->3 [22] 5/-1/-1->4->3 [23] 5/-1/-1->4->3 | |
| tyler-rhel-newimage:1521:1521 [0] NCCL INFO Channel 02/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:1548:1548 [5] NCCL INFO P2P Chunksize set to 524288 | |
| tyler-rhel-newimage:1521:1521 [0] NCCL INFO Channel 03/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:1545:1545 [2] NCCL INFO P2P Chunksize set to 524288 | |
| tyler-rhel-newimage:1546:1546 [3] NCCL INFO P2P Chunksize set to 524288 | |
| tyler-rhel-newimage:1550:1550 [7] NCCL INFO P2P Chunksize set to 524288 | |
| tyler-rhel-newimage:1547:1547 [4] NCCL INFO P2P Chunksize set to 524288 | |
| tyler-rhel-newimage:1521:1521 [0] NCCL INFO Channel 04/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:1521:1521 [0] NCCL INFO Channel 05/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:1521:1521 [0] NCCL INFO Channel 06/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:1521:1521 [0] NCCL INFO Channel 07/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:1521:1521 [0] NCCL INFO Channel 08/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:1521:1521 [0] NCCL INFO Channel 09/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:1521:1521 [0] NCCL INFO Channel 10/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:1521:1521 [0] NCCL INFO Channel 11/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:1521:1521 [0] NCCL INFO Channel 12/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:1521:1521 [0] NCCL INFO Channel 13/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:1521:1521 [0] NCCL INFO Channel 14/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:1521:1521 [0] NCCL INFO Channel 15/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:1521:1521 [0] NCCL INFO Channel 16/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:1521:1521 [0] NCCL INFO Channel 17/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:1521:1521 [0] NCCL INFO Channel 18/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:1521:1521 [0] NCCL INFO Channel 19/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:1521:1521 [0] NCCL INFO Channel 20/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:1521:1521 [0] NCCL INFO Channel 21/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:1521:1521 [0] NCCL INFO Channel 22/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:1521:1521 [0] NCCL INFO Channel 23/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:1521:1521 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1 [2] 1/-1/-1->0->-1 [3] 1/-1/-1->0->-1 [4] 1/-1/-1->0->-1 [5] 1/-1/-1->0->-1 [6] 1/-1/-1->0->-1 [7] 1/-1/-1->0->-1 [8] 1/-1/-1->0->-1 [9] 1/-1/-1->0->-1 [10] 1/-1/-1->0->-1 [11] 1/-1/-1->0->-1 [12] 1/-1/-1->0->-1 [13] 1/-1/-1->0->-1 [14] 1/-1/-1->0->-1 [15] 1/-1/-1->0->-1 [16] 1/-1/-1->0->-1 [17] 1/-1/-1->0->-1 [18] 1/-1/-1->0->-1 [19] 1/-1/-1->0->-1 [20] 1/-1/-1->0->-1 [21] 1/-1/-1->0->-1 [22] 1/-1/-1->0->-1 [23] 1/-1/-1->0->-1 | |
| tyler-rhel-newimage:1521:1521 [0] NCCL INFO P2P Chunksize set to 524288 | |
| tyler-rhel-newimage:1549:1549 [6] NCCL INFO Channel 00/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:1550:1550 [7] NCCL INFO Channel 00/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1545:1545 [2] NCCL INFO Channel 00/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:1521:1521 [0] NCCL INFO Channel 00/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:1549:1549 [6] NCCL INFO Channel 01/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:1547:1547 [4] NCCL INFO Channel 00/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:1546:1546 [3] NCCL INFO Channel 00/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:1550:1550 [7] NCCL INFO Channel 01/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1544:1544 [1] NCCL INFO Channel 00/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:1545:1545 [2] NCCL INFO Channel 01/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:1521:1521 [0] NCCL INFO Channel 01/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:1549:1549 [6] NCCL INFO Channel 02/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:1547:1547 [4] NCCL INFO Channel 01/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:1548:1548 [5] NCCL INFO Channel 00/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:1546:1546 [3] NCCL INFO Channel 01/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:1550:1550 [7] NCCL INFO Channel 02/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1521:1521 [0] NCCL INFO Channel 02/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:1544:1544 [1] NCCL INFO Channel 01/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:1545:1545 [2] NCCL INFO Channel 02/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:1547:1547 [4] NCCL INFO Channel 02/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:1549:1549 [6] NCCL INFO Channel 03/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:1548:1548 [5] NCCL INFO Channel 01/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:1546:1546 [3] NCCL INFO Channel 02/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:1550:1550 [7] NCCL INFO Channel 03/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1544:1544 [1] NCCL INFO Channel 02/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:1521:1521 [0] NCCL INFO Channel 03/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:1545:1545 [2] NCCL INFO Channel 03/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:1547:1547 [4] NCCL INFO Channel 03/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:1549:1549 [6] NCCL INFO Channel 04/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:1548:1548 [5] NCCL INFO Channel 02/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:1546:1546 [3] NCCL INFO Channel 03/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:1550:1550 [7] NCCL INFO Channel 04/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1521:1521 [0] NCCL INFO Channel 04/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:1544:1544 [1] NCCL INFO Channel 03/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:1545:1545 [2] NCCL INFO Channel 04/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:1547:1547 [4] NCCL INFO Channel 04/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:1549:1549 [6] NCCL INFO Channel 05/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:1548:1548 [5] NCCL INFO Channel 03/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:1546:1546 [3] NCCL INFO Channel 04/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:1550:1550 [7] NCCL INFO Channel 05/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1521:1521 [0] NCCL INFO Channel 05/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:1547:1547 [4] NCCL INFO Channel 05/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:1544:1544 [1] NCCL INFO Channel 04/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:1545:1545 [2] NCCL INFO Channel 05/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:1549:1549 [6] NCCL INFO Channel 06/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:1548:1548 [5] NCCL INFO Channel 04/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:1546:1546 [3] NCCL INFO Channel 05/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:1550:1550 [7] NCCL INFO Channel 06/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1521:1521 [0] NCCL INFO Channel 06/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:1547:1547 [4] NCCL INFO Channel 06/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:1544:1544 [1] NCCL INFO Channel 05/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:1549:1549 [6] NCCL INFO Channel 07/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:1545:1545 [2] NCCL INFO Channel 06/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:1548:1548 [5] NCCL INFO Channel 05/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:1546:1546 [3] NCCL INFO Channel 06/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:1550:1550 [7] NCCL INFO Channel 07/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1521:1521 [0] NCCL INFO Channel 07/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:1547:1547 [4] NCCL INFO Channel 07/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:1544:1544 [1] NCCL INFO Channel 06/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:1549:1549 [6] NCCL INFO Channel 08/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:1545:1545 [2] NCCL INFO Channel 07/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:1546:1546 [3] NCCL INFO Channel 07/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:1548:1548 [5] NCCL INFO Channel 06/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:1550:1550 [7] NCCL INFO Channel 08/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1521:1521 [0] NCCL INFO Channel 08/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:1547:1547 [4] NCCL INFO Channel 08/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:1544:1544 [1] NCCL INFO Channel 07/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:1549:1549 [6] NCCL INFO Channel 09/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:1545:1545 [2] NCCL INFO Channel 08/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:1546:1546 [3] NCCL INFO Channel 08/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:1548:1548 [5] NCCL INFO Channel 07/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:1550:1550 [7] NCCL INFO Channel 09/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1521:1521 [0] NCCL INFO Channel 09/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:1544:1544 [1] NCCL INFO Channel 08/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:1547:1547 [4] NCCL INFO Channel 09/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:1549:1549 [6] NCCL INFO Channel 10/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:1545:1545 [2] NCCL INFO Channel 09/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:1546:1546 [3] NCCL INFO Channel 09/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:1548:1548 [5] NCCL INFO Channel 08/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:1550:1550 [7] NCCL INFO Channel 10/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1521:1521 [0] NCCL INFO Channel 10/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:1544:1544 [1] NCCL INFO Channel 09/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:1547:1547 [4] NCCL INFO Channel 10/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:1549:1549 [6] NCCL INFO Channel 11/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:1545:1545 [2] NCCL INFO Channel 10/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:1546:1546 [3] NCCL INFO Channel 10/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:1548:1548 [5] NCCL INFO Channel 09/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:1521:1521 [0] NCCL INFO Channel 11/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:1550:1550 [7] NCCL INFO Channel 11/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1544:1544 [1] NCCL INFO Channel 10/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:1547:1547 [4] NCCL INFO Channel 11/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:1549:1549 [6] NCCL INFO Channel 12/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:1545:1545 [2] NCCL INFO Channel 11/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:1546:1546 [3] NCCL INFO Channel 11/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:1521:1521 [0] NCCL INFO Channel 12/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:1548:1548 [5] NCCL INFO Channel 10/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:1550:1550 [7] NCCL INFO Channel 12/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1544:1544 [1] NCCL INFO Channel 11/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:1547:1547 [4] NCCL INFO Channel 12/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:1549:1549 [6] NCCL INFO Channel 13/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:1545:1545 [2] NCCL INFO Channel 12/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:1546:1546 [3] NCCL INFO Channel 12/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:1521:1521 [0] NCCL INFO Channel 13/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:1548:1548 [5] NCCL INFO Channel 11/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:1550:1550 [7] NCCL INFO Channel 13/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1544:1544 [1] NCCL INFO Channel 12/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:1547:1547 [4] NCCL INFO Channel 13/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:1549:1549 [6] NCCL INFO Channel 14/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:1545:1545 [2] NCCL INFO Channel 13/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:1546:1546 [3] NCCL INFO Channel 13/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:1521:1521 [0] NCCL INFO Channel 14/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:1548:1548 [5] NCCL INFO Channel 12/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:1550:1550 [7] NCCL INFO Channel 14/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1544:1544 [1] NCCL INFO Channel 13/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:1547:1547 [4] NCCL INFO Channel 14/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:1549:1549 [6] NCCL INFO Channel 15/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:1545:1545 [2] NCCL INFO Channel 14/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:1546:1546 [3] NCCL INFO Channel 14/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:1521:1521 [0] NCCL INFO Channel 15/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:1548:1548 [5] NCCL INFO Channel 13/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:1550:1550 [7] NCCL INFO Channel 15/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1544:1544 [1] NCCL INFO Channel 14/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:1547:1547 [4] NCCL INFO Channel 15/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:1549:1549 [6] NCCL INFO Channel 16/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:1545:1545 [2] NCCL INFO Channel 15/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:1546:1546 [3] NCCL INFO Channel 15/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:1521:1521 [0] NCCL INFO Channel 16/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:1548:1548 [5] NCCL INFO Channel 14/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:1550:1550 [7] NCCL INFO Channel 16/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1544:1544 [1] NCCL INFO Channel 15/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:1547:1547 [4] NCCL INFO Channel 16/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:1549:1549 [6] NCCL INFO Channel 17/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:1546:1546 [3] NCCL INFO Channel 16/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:1545:1545 [2] NCCL INFO Channel 16/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:1521:1521 [0] NCCL INFO Channel 17/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:1548:1548 [5] NCCL INFO Channel 15/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:1550:1550 [7] NCCL INFO Channel 17/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1544:1544 [1] NCCL INFO Channel 16/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:1547:1547 [4] NCCL INFO Channel 17/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:1549:1549 [6] NCCL INFO Channel 18/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:1546:1546 [3] NCCL INFO Channel 17/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:1545:1545 [2] NCCL INFO Channel 17/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:1521:1521 [0] NCCL INFO Channel 18/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:1548:1548 [5] NCCL INFO Channel 16/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:1550:1550 [7] NCCL INFO Channel 18/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1544:1544 [1] NCCL INFO Channel 17/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:1547:1547 [4] NCCL INFO Channel 18/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:1549:1549 [6] NCCL INFO Channel 19/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:1546:1546 [3] NCCL INFO Channel 18/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:1521:1521 [0] NCCL INFO Channel 19/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:1545:1545 [2] NCCL INFO Channel 18/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:1548:1548 [5] NCCL INFO Channel 17/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:1550:1550 [7] NCCL INFO Channel 19/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1544:1544 [1] NCCL INFO Channel 18/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:1547:1547 [4] NCCL INFO Channel 19/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:1546:1546 [3] NCCL INFO Channel 19/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:1549:1549 [6] NCCL INFO Channel 20/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:1521:1521 [0] NCCL INFO Channel 20/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:1545:1545 [2] NCCL INFO Channel 19/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:1548:1548 [5] NCCL INFO Channel 18/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:1550:1550 [7] NCCL INFO Channel 20/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1544:1544 [1] NCCL INFO Channel 19/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:1547:1547 [4] NCCL INFO Channel 20/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:1546:1546 [3] NCCL INFO Channel 20/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:1521:1521 [0] NCCL INFO Channel 21/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:1549:1549 [6] NCCL INFO Channel 21/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:1545:1545 [2] NCCL INFO Channel 20/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:1548:1548 [5] NCCL INFO Channel 19/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:1550:1550 [7] NCCL INFO Channel 21/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1547:1547 [4] NCCL INFO Channel 21/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:1544:1544 [1] NCCL INFO Channel 20/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:1546:1546 [3] NCCL INFO Channel 21/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:1521:1521 [0] NCCL INFO Channel 22/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:1549:1549 [6] NCCL INFO Channel 22/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:1545:1545 [2] NCCL INFO Channel 21/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:1548:1548 [5] NCCL INFO Channel 20/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:1550:1550 [7] NCCL INFO Channel 22/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1547:1547 [4] NCCL INFO Channel 22/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:1544:1544 [1] NCCL INFO Channel 21/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:1546:1546 [3] NCCL INFO Channel 22/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:1521:1521 [0] NCCL INFO Channel 23/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:1549:1549 [6] NCCL INFO Channel 23/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:1545:1545 [2] NCCL INFO Channel 22/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:1548:1548 [5] NCCL INFO Channel 21/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:1550:1550 [7] NCCL INFO Channel 23/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1547:1547 [4] NCCL INFO Channel 23/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:1544:1544 [1] NCCL INFO Channel 22/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:1546:1546 [3] NCCL INFO Channel 23/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:1545:1545 [2] NCCL INFO Channel 23/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:1548:1548 [5] NCCL INFO Channel 22/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:1544:1544 [1] NCCL INFO Channel 23/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:1548:1548 [5] NCCL INFO Channel 23/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:1549:1549 [6] NCCL INFO Connected all rings | |
| tyler-rhel-newimage:1545:1545 [2] NCCL INFO Connected all rings | |
| tyler-rhel-newimage:1548:1548 [5] NCCL INFO Connected all rings | |
| tyler-rhel-newimage:1546:1546 [3] NCCL INFO Connected all rings | |
| tyler-rhel-newimage:1547:1547 [4] NCCL INFO Connected all rings | |
| tyler-rhel-newimage:1544:1544 [1] NCCL INFO Connected all rings | |
| tyler-rhel-newimage:1550:1550 [7] NCCL INFO Connected all rings | |
| tyler-rhel-newimage:1521:1521 [0] NCCL INFO Connected all rings | |
| tyler-rhel-newimage:1550:1550 [7] NCCL INFO Channel 00/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:1550:1550 [7] NCCL INFO Channel 01/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:1550:1550 [7] NCCL INFO Channel 02/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:1550:1550 [7] NCCL INFO Channel 03/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:1550:1550 [7] NCCL INFO Channel 04/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:1550:1550 [7] NCCL INFO Channel 05/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:1550:1550 [7] NCCL INFO Channel 06/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:1550:1550 [7] NCCL INFO Channel 07/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:1550:1550 [7] NCCL INFO Channel 08/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:1550:1550 [7] NCCL INFO Channel 09/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:1550:1550 [7] NCCL INFO Channel 10/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:1550:1550 [7] NCCL INFO Channel 11/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:1550:1550 [7] NCCL INFO Channel 12/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:1550:1550 [7] NCCL INFO Channel 13/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:1545:1545 [2] NCCL INFO Channel 00/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:1548:1548 [5] NCCL INFO Channel 00/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:1545:1545 [2] NCCL INFO Channel 01/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:1547:1547 [4] NCCL INFO Channel 00/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:1546:1546 [3] NCCL INFO Channel 00/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:1548:1548 [5] NCCL INFO Channel 01/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:1545:1545 [2] NCCL INFO Channel 02/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:1547:1547 [4] NCCL INFO Channel 01/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:1546:1546 [3] NCCL INFO Channel 01/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:1550:1550 [7] NCCL INFO Channel 14/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:1548:1548 [5] NCCL INFO Channel 02/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:1545:1545 [2] NCCL INFO Channel 03/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:1547:1547 [4] NCCL INFO Channel 02/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:1546:1546 [3] NCCL INFO Channel 02/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:1550:1550 [7] NCCL INFO Channel 15/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:1548:1548 [5] NCCL INFO Channel 03/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:1549:1549 [6] NCCL INFO Channel 00/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:1545:1545 [2] NCCL INFO Channel 04/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:1547:1547 [4] NCCL INFO Channel 03/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:1546:1546 [3] NCCL INFO Channel 03/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:1550:1550 [7] NCCL INFO Channel 16/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:1548:1548 [5] NCCL INFO Channel 04/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:1549:1549 [6] NCCL INFO Channel 01/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:1545:1545 [2] NCCL INFO Channel 05/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:1547:1547 [4] NCCL INFO Channel 04/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:1546:1546 [3] NCCL INFO Channel 04/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:1550:1550 [7] NCCL INFO Channel 17/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:1544:1544 [1] NCCL INFO Channel 00/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1548:1548 [5] NCCL INFO Channel 05/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:1549:1549 [6] NCCL INFO Channel 02/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:1545:1545 [2] NCCL INFO Channel 06/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:1547:1547 [4] NCCL INFO Channel 05/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:1546:1546 [3] NCCL INFO Channel 05/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:1550:1550 [7] NCCL INFO Channel 18/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:1544:1544 [1] NCCL INFO Channel 01/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1548:1548 [5] NCCL INFO Channel 06/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:1549:1549 [6] NCCL INFO Channel 03/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:1545:1545 [2] NCCL INFO Channel 07/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:1547:1547 [4] NCCL INFO Channel 06/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:1546:1546 [3] NCCL INFO Channel 06/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:1550:1550 [7] NCCL INFO Channel 19/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:1544:1544 [1] NCCL INFO Channel 02/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1548:1548 [5] NCCL INFO Channel 07/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:1549:1549 [6] NCCL INFO Channel 04/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:1545:1545 [2] NCCL INFO Channel 08/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:1547:1547 [4] NCCL INFO Channel 07/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:1546:1546 [3] NCCL INFO Channel 07/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:1550:1550 [7] NCCL INFO Channel 20/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:1544:1544 [1] NCCL INFO Channel 03/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1548:1548 [5] NCCL INFO Channel 08/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:1549:1549 [6] NCCL INFO Channel 05/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:1545:1545 [2] NCCL INFO Channel 09/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:1547:1547 [4] NCCL INFO Channel 08/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:1546:1546 [3] NCCL INFO Channel 08/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:1550:1550 [7] NCCL INFO Channel 21/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:1544:1544 [1] NCCL INFO Channel 04/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1548:1548 [5] NCCL INFO Channel 09/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:1549:1549 [6] NCCL INFO Channel 06/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:1545:1545 [2] NCCL INFO Channel 10/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:1547:1547 [4] NCCL INFO Channel 09/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:1546:1546 [3] NCCL INFO Channel 09/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:1550:1550 [7] NCCL INFO Channel 22/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:1544:1544 [1] NCCL INFO Channel 05/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1548:1548 [5] NCCL INFO Channel 10/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:1549:1549 [6] NCCL INFO Channel 07/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:1545:1545 [2] NCCL INFO Channel 11/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:1547:1547 [4] NCCL INFO Channel 10/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:1546:1546 [3] NCCL INFO Channel 10/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:1550:1550 [7] NCCL INFO Channel 23/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:1544:1544 [1] NCCL INFO Channel 06/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1548:1548 [5] NCCL INFO Channel 11/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:1549:1549 [6] NCCL INFO Channel 08/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:1545:1545 [2] NCCL INFO Channel 12/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:1547:1547 [4] NCCL INFO Channel 11/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:1546:1546 [3] NCCL INFO Channel 11/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:1544:1544 [1] NCCL INFO Channel 07/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1545:1545 [2] NCCL INFO Channel 13/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:1548:1548 [5] NCCL INFO Channel 12/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:1549:1549 [6] NCCL INFO Channel 09/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:1547:1547 [4] NCCL INFO Channel 12/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:1546:1546 [3] NCCL INFO Channel 12/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:1544:1544 [1] NCCL INFO Channel 08/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1545:1545 [2] NCCL INFO Channel 14/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:1548:1548 [5] NCCL INFO Channel 13/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:1549:1549 [6] NCCL INFO Channel 10/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:1547:1547 [4] NCCL INFO Channel 13/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:1546:1546 [3] NCCL INFO Channel 13/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:1544:1544 [1] NCCL INFO Channel 09/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1545:1545 [2] NCCL INFO Channel 15/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:1548:1548 [5] NCCL INFO Channel 14/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:1549:1549 [6] NCCL INFO Channel 11/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:1547:1547 [4] NCCL INFO Channel 14/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:1546:1546 [3] NCCL INFO Channel 14/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:1544:1544 [1] NCCL INFO Channel 10/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1545:1545 [2] NCCL INFO Channel 16/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:1548:1548 [5] NCCL INFO Channel 15/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:1549:1549 [6] NCCL INFO Channel 12/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:1547:1547 [4] NCCL INFO Channel 15/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:1546:1546 [3] NCCL INFO Channel 15/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:1544:1544 [1] NCCL INFO Channel 11/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1545:1545 [2] NCCL INFO Channel 17/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:1548:1548 [5] NCCL INFO Channel 16/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:1549:1549 [6] NCCL INFO Channel 13/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:1547:1547 [4] NCCL INFO Channel 16/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:1546:1546 [3] NCCL INFO Channel 16/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:1544:1544 [1] NCCL INFO Channel 12/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1545:1545 [2] NCCL INFO Channel 18/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:1548:1548 [5] NCCL INFO Channel 17/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:1549:1549 [6] NCCL INFO Channel 14/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:1547:1547 [4] NCCL INFO Channel 17/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:1546:1546 [3] NCCL INFO Channel 17/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:1544:1544 [1] NCCL INFO Channel 13/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1545:1545 [2] NCCL INFO Channel 19/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:1548:1548 [5] NCCL INFO Channel 18/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:1549:1549 [6] NCCL INFO Channel 15/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:1547:1547 [4] NCCL INFO Channel 18/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:1544:1544 [1] NCCL INFO Channel 14/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1546:1546 [3] NCCL INFO Channel 18/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:1545:1545 [2] NCCL INFO Channel 20/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:1548:1548 [5] NCCL INFO Channel 19/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:1549:1549 [6] NCCL INFO Channel 16/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:1547:1547 [4] NCCL INFO Channel 19/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:1544:1544 [1] NCCL INFO Channel 15/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1546:1546 [3] NCCL INFO Channel 19/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:1545:1545 [2] NCCL INFO Channel 21/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:1548:1548 [5] NCCL INFO Channel 20/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:1549:1549 [6] NCCL INFO Channel 17/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:1547:1547 [4] NCCL INFO Channel 20/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:1544:1544 [1] NCCL INFO Channel 16/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1546:1546 [3] NCCL INFO Channel 20/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:1545:1545 [2] NCCL INFO Channel 22/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:1548:1548 [5] NCCL INFO Channel 21/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:1549:1549 [6] NCCL INFO Channel 18/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:1547:1547 [4] NCCL INFO Channel 21/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:1544:1544 [1] NCCL INFO Channel 17/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1546:1546 [3] NCCL INFO Channel 21/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:1545:1545 [2] NCCL INFO Channel 23/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:1548:1548 [5] NCCL INFO Channel 22/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:1549:1549 [6] NCCL INFO Channel 19/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:1547:1547 [4] NCCL INFO Channel 22/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:1544:1544 [1] NCCL INFO Channel 18/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1546:1546 [3] NCCL INFO Channel 22/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:1548:1548 [5] NCCL INFO Channel 23/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:1549:1549 [6] NCCL INFO Channel 20/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:1547:1547 [4] NCCL INFO Channel 23/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:1544:1544 [1] NCCL INFO Channel 19/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1546:1546 [3] NCCL INFO Channel 23/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:1549:1549 [6] NCCL INFO Channel 21/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:1544:1544 [1] NCCL INFO Channel 20/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1549:1549 [6] NCCL INFO Channel 22/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:1544:1544 [1] NCCL INFO Channel 21/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1549:1549 [6] NCCL INFO Channel 23/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:1544:1544 [1] NCCL INFO Channel 22/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1544:1544 [1] NCCL INFO Channel 23/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1550:1550 [7] NCCL INFO Connected all trees | |
| tyler-rhel-newimage:1550:1550 [7] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512 | |
| tyler-rhel-newimage:1550:1550 [7] NCCL INFO 24 coll channels, 0 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer | |
| tyler-rhel-newimage:1547:1547 [4] NCCL INFO Connected all trees | |
| tyler-rhel-newimage:1547:1547 [4] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512 | |
| tyler-rhel-newimage:1547:1547 [4] NCCL INFO 24 coll channels, 0 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer | |
| tyler-rhel-newimage:1549:1549 [6] NCCL INFO Connected all trees | |
| tyler-rhel-newimage:1549:1549 [6] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512 | |
| tyler-rhel-newimage:1549:1549 [6] NCCL INFO 24 coll channels, 0 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer | |
| tyler-rhel-newimage:1548:1548 [5] NCCL INFO Connected all trees | |
| tyler-rhel-newimage:1548:1548 [5] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512 | |
| tyler-rhel-newimage:1548:1548 [5] NCCL INFO 24 coll channels, 0 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer | |
| tyler-rhel-newimage:1546:1546 [3] NCCL INFO Connected all trees | |
| tyler-rhel-newimage:1521:1521 [0] NCCL INFO Connected all trees | |
| tyler-rhel-newimage:1546:1546 [3] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512 | |
| tyler-rhel-newimage:1546:1546 [3] NCCL INFO 24 coll channels, 0 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer | |
| tyler-rhel-newimage:1545:1545 [2] NCCL INFO Connected all trees | |
| tyler-rhel-newimage:1521:1521 [0] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512 | |
| tyler-rhel-newimage:1521:1521 [0] NCCL INFO 24 coll channels, 0 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer | |
| tyler-rhel-newimage:1544:1544 [1] NCCL INFO Connected all trees | |
| tyler-rhel-newimage:1545:1545 [2] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512 | |
| tyler-rhel-newimage:1545:1545 [2] NCCL INFO 24 coll channels, 0 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer | |
| tyler-rhel-newimage:1544:1544 [1] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512 | |
| tyler-rhel-newimage:1544:1544 [1] NCCL INFO 24 coll channels, 0 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer | |
| tyler-rhel-newimage:1550:1550 [7] NCCL INFO NCCL_WORK_FIFO_DEPTH set by environment to 4194304. | |
| tyler-rhel-newimage:1547:1547 [4] NCCL INFO NCCL_WORK_FIFO_DEPTH set by environment to 4194304. | |
| tyler-rhel-newimage:1546:1546 [3] NCCL INFO NCCL_WORK_FIFO_DEPTH set by environment to 4194304. | |
| tyler-rhel-newimage:1545:1545 [2] NCCL INFO NCCL_WORK_FIFO_DEPTH set by environment to 4194304. | |
| tyler-rhel-newimage:1544:1544 [1] NCCL INFO NCCL_WORK_FIFO_DEPTH set by environment to 4194304. | |
| tyler-rhel-newimage:1549:1549 [6] NCCL INFO NCCL_WORK_FIFO_DEPTH set by environment to 4194304. | |
| tyler-rhel-newimage:1548:1548 [5] NCCL INFO NCCL_WORK_FIFO_DEPTH set by environment to 4194304. | |
| tyler-rhel-newimage:1521:1521 [0] NCCL INFO NCCL_WORK_FIFO_DEPTH set by environment to 4194304. | |
| tyler-rhel-newimage:1549:1549 [6] NCCL INFO comm 0x56095175f2f0 rank 6 nranks 8 cudaDev 6 nvmlDev 6 busId e070 commId 0x2c85857d0fe3f407 - Init COMPLETE | |
| tyler-rhel-newimage:1550:1550 [7] NCCL INFO comm 0x56095175e8a0 rank 7 nranks 8 cudaDev 7 nvmlDev 7 busId e080 commId 0x2c85857d0fe3f407 - Init COMPLETE | |
| tyler-rhel-newimage:1547:1547 [4] NCCL INFO comm 0x56095175ec70 rank 4 nranks 8 cudaDev 4 nvmlDev 4 busId c050 commId 0x2c85857d0fe3f407 - Init COMPLETE | |
| tyler-rhel-newimage:1548:1548 [5] NCCL INFO comm 0x56095175df80 rank 5 nranks 8 cudaDev 5 nvmlDev 5 busId c060 commId 0x2c85857d0fe3f407 - Init COMPLETE | |
| tyler-rhel-newimage:1546:1546 [3] NCCL INFO comm 0x56095175d780 rank 3 nranks 8 cudaDev 3 nvmlDev 3 busId a040 commId 0x2c85857d0fe3f407 - Init COMPLETE | |
| tyler-rhel-newimage:1521:1521 [0] NCCL INFO comm 0x5609517656d0 rank 0 nranks 8 cudaDev 0 nvmlDev 0 busId 8010 commId 0x2c85857d0fe3f407 - Init COMPLETE | |
| tyler-rhel-newimage:1545:1545 [2] NCCL INFO comm 0x56095175d300 rank 2 nranks 8 cudaDev 2 nvmlDev 2 busId a030 commId 0x2c85857d0fe3f407 - Init COMPLETE | |
| tyler-rhel-newimage:1544:1544 [1] NCCL INFO comm 0x56095175dfb0 rank 1 nranks 8 cudaDev 1 nvmlDev 1 busId 8020 commId 0x2c85857d0fe3f407 - Init COMPLETE | |
| Loading safetensors checkpoint shards: 0% Completed | 0/19 [00:00<?, ?it/s] | |
| Loading safetensors checkpoint shards: 5% Completed | 1/19 [00:00<00:05, 3.37it/s] | |
| Loading safetensors checkpoint shards: 11% Completed | 2/19 [00:00<00:05, 3.07it/s] | |
| INFO 2024-07-27 16:38:14,854 instructlab.model.backends.backends:336: Waiting for the vLLM server to start at http://127.0.0.1:57407/v1, this might take a moment... Attempt: 5/80 | |
| Loading safetensors checkpoint shards: 16% Completed | 3/19 [00:00<00:05, 3.08it/s] | |
| Loading safetensors checkpoint shards: 21% Completed | 4/19 [00:01<00:04, 3.03it/s] | |
| Loading safetensors checkpoint shards: 26% Completed | 5/19 [00:01<00:04, 3.15it/s] | |
| Loading safetensors checkpoint shards: 32% Completed | 6/19 [00:01<00:04, 3.20it/s] | |
| Loading safetensors checkpoint shards: 37% Completed | 7/19 [00:02<00:03, 3.12it/s] | |
| Loading safetensors checkpoint shards: 42% Completed | 8/19 [00:02<00:03, 3.02it/s] | |
| Loading safetensors checkpoint shards: 47% Completed | 9/19 [00:02<00:03, 3.01it/s] | |
| Loading safetensors checkpoint shards: 53% Completed | 10/19 [00:03<00:03, 2.94it/s] | |
| Loading safetensors checkpoint shards: 58% Completed | 11/19 [00:03<00:02, 2.97it/s] | |
| Loading safetensors checkpoint shards: 63% Completed | 12/19 [00:03<00:02, 2.98it/s] | |
| Loading safetensors checkpoint shards: 68% Completed | 13/19 [00:04<00:02, 2.89it/s] | |
| Loading safetensors checkpoint shards: 74% Completed | 14/19 [00:04<00:01, 2.93it/s] | |
| Loading safetensors checkpoint shards: 79% Completed | 15/19 [00:04<00:01, 2.95it/s] | |
| Loading safetensors checkpoint shards: 84% Completed | 16/19 [00:05<00:01, 2.87it/s] | |
| INFO 2024-07-27 16:38:19,496 instructlab.model.backends.backends:336: Waiting for the vLLM server to start at http://127.0.0.1:57407/v1, this might take a moment... Attempt: 6/80 | |
| Loading safetensors checkpoint shards: 89% Completed | 17/19 [00:05<00:00, 2.88it/s] | |
| Loading safetensors checkpoint shards: 95% Completed | 18/19 [00:05<00:00, 3.03it/s] | |
| Loading safetensors checkpoint shards: 100% Completed | 19/19 [00:06<00:00, 3.39it/s] | |
| Loading safetensors checkpoint shards: 100% Completed | 19/19 [00:06<00:00, 3.07it/s] | |
| tyler-rhel-newimage:1521:2083 [0] NCCL INFO Using non-device net plugin version 0 | |
| tyler-rhel-newimage:1521:2083 [0] NCCL INFO Using network Socket | |
| tyler-rhel-newimage:1550:2084 [7] NCCL INFO Using non-device net plugin version 0 | |
| tyler-rhel-newimage:1550:2084 [7] NCCL INFO Using network Socket | |
| tyler-rhel-newimage:1548:2085 [5] NCCL INFO Using non-device net plugin version 0 | |
| tyler-rhel-newimage:1548:2085 [5] NCCL INFO Using network Socket | |
| tyler-rhel-newimage:1546:2087 [3] NCCL INFO Using non-device net plugin version 0 | |
| tyler-rhel-newimage:1546:2087 [3] NCCL INFO Using network Socket | |
| tyler-rhel-newimage:1547:2086 [4] NCCL INFO Using non-device net plugin version 0 | |
| tyler-rhel-newimage:1547:2086 [4] NCCL INFO Using network Socket | |
| tyler-rhel-newimage:1549:2088 [6] NCCL INFO Using non-device net plugin version 0 | |
| tyler-rhel-newimage:1549:2088 [6] NCCL INFO Using network Socket | |
| tyler-rhel-newimage:1544:2089 [1] NCCL INFO Using non-device net plugin version 0 | |
| tyler-rhel-newimage:1544:2089 [1] NCCL INFO Using network Socket | |
| tyler-rhel-newimage:1545:2090 [2] NCCL INFO Using non-device net plugin version 0 | |
| tyler-rhel-newimage:1545:2090 [2] NCCL INFO Using network Socket | |
| tyler-rhel-newimage:1545:2090 [2] NCCL INFO comm 0x56095bf30100 rank 2 nranks 8 cudaDev 2 nvmlDev 2 busId a030 commId 0xaf55c5d9f1b4fe49 - Init START | |
| tyler-rhel-newimage:1546:2087 [3] NCCL INFO comm 0x56095cd2ffd0 rank 3 nranks 8 cudaDev 3 nvmlDev 3 busId a040 commId 0xaf55c5d9f1b4fe49 - Init START | |
| tyler-rhel-newimage:1550:2084 [7] NCCL INFO comm 0x56095cd1bbb0 rank 7 nranks 8 cudaDev 7 nvmlDev 7 busId e080 commId 0xaf55c5d9f1b4fe49 - Init START | |
| tyler-rhel-newimage:1549:2088 [6] NCCL INFO comm 0x56095bf31a90 rank 6 nranks 8 cudaDev 6 nvmlDev 6 busId e070 commId 0xaf55c5d9f1b4fe49 - Init START | |
| tyler-rhel-newimage:1548:2085 [5] NCCL INFO comm 0x56095bf305c0 rank 5 nranks 8 cudaDev 5 nvmlDev 5 busId c060 commId 0xaf55c5d9f1b4fe49 - Init START | |
| tyler-rhel-newimage:1547:2086 [4] NCCL INFO comm 0x56095cb310c0 rank 4 nranks 8 cudaDev 4 nvmlDev 4 busId c050 commId 0xaf55c5d9f1b4fe49 - Init START | |
| tyler-rhel-newimage:1544:2089 [1] NCCL INFO comm 0x56095bf30330 rank 1 nranks 8 cudaDev 1 nvmlDev 1 busId 8020 commId 0xaf55c5d9f1b4fe49 - Init START | |
| tyler-rhel-newimage:1521:2083 [0] NCCL INFO comm 0x56096637cfa0 rank 0 nranks 8 cudaDev 0 nvmlDev 0 busId 8010 commId 0xaf55c5d9f1b4fe49 - Init START | |
| tyler-rhel-newimage:1521:2083 [0] NCCL INFO Setting affinity for GPU 0 to ff,ffffffff | |
| tyler-rhel-newimage:1521:2083 [0] NCCL INFO NVLS multicast support is not available on dev 0 | |
| tyler-rhel-newimage:1544:2089 [1] NCCL INFO Setting affinity for GPU 1 to ff,ffffffff | |
| tyler-rhel-newimage:1544:2089 [1] NCCL INFO NVLS multicast support is not available on dev 1 | |
| tyler-rhel-newimage:1546:2087 [3] NCCL INFO Setting affinity for GPU 3 to ff,ffffffff | |
| tyler-rhel-newimage:1546:2087 [3] NCCL INFO NVLS multicast support is not available on dev 3 | |
| tyler-rhel-newimage:1549:2088 [6] NCCL INFO Setting affinity for GPU 6 to ffff,ffffff00,00000000 | |
| tyler-rhel-newimage:1549:2088 [6] NCCL INFO NVLS multicast support is not available on dev 6 | |
| tyler-rhel-newimage:1550:2084 [7] NCCL INFO Setting affinity for GPU 7 to ffff,ffffff00,00000000 | |
| tyler-rhel-newimage:1550:2084 [7] NCCL INFO NVLS multicast support is not available on dev 7 | |
| tyler-rhel-newimage:1547:2086 [4] NCCL INFO Setting affinity for GPU 4 to ffff,ffffff00,00000000 | |
| tyler-rhel-newimage:1547:2086 [4] NCCL INFO NVLS multicast support is not available on dev 4 | |
| tyler-rhel-newimage:1548:2085 [5] NCCL INFO Setting affinity for GPU 5 to ffff,ffffff00,00000000 | |
| tyler-rhel-newimage:1548:2085 [5] NCCL INFO NVLS multicast support is not available on dev 5 | |
| tyler-rhel-newimage:1545:2090 [2] NCCL INFO Setting affinity for GPU 2 to ff,ffffffff | |
| tyler-rhel-newimage:1545:2090 [2] NCCL INFO NVLS multicast support is not available on dev 2 | |
| tyler-rhel-newimage:1545:2090 [2] NCCL INFO comm 0x56095bf30100 rank 2 nRanks 8 nNodes 1 localRanks 8 localRank 2 MNNVL 0 | |
| tyler-rhel-newimage:1545:2090 [2] NCCL INFO Trees [0] 3/-1/-1->2->1 [1] 3/-1/-1->2->1 [2] 3/-1/-1->2->1 [3] 3/-1/-1->2->1 [4] 3/-1/-1->2->1 [5] 3/-1/-1->2->1 [6] 3/-1/-1->2->1 [7] 3/-1/-1->2->1 [8] 3/-1/-1->2->1 [9] 3/-1/-1->2->1 [10] 3/-1/-1->2->1 [11] 3/-1/-1->2->1 [12] 3/-1/-1->2->1 [13] 3/-1/-1->2->1 [14] 3/-1/-1->2->1 [15] 3/-1/-1->2->1 [16] 3/-1/-1->2->1 [17] 3/-1/-1->2->1 [18] 3/-1/-1->2->1 [19] 3/-1/-1->2->1 [20] 3/-1/-1->2->1 [21] 3/-1/-1->2->1 [22] 3/-1/-1->2->1 [23] 3/-1/-1->2->1 | |
| tyler-rhel-newimage:1545:2090 [2] NCCL INFO P2P Chunksize set to 524288 | |
| tyler-rhel-newimage:1550:2084 [7] NCCL INFO comm 0x56095cd1bbb0 rank 7 nRanks 8 nNodes 1 localRanks 8 localRank 7 MNNVL 0 | |
| tyler-rhel-newimage:1549:2088 [6] NCCL INFO comm 0x56095bf31a90 rank 6 nRanks 8 nNodes 1 localRanks 8 localRank 6 MNNVL 0 | |
| tyler-rhel-newimage:1544:2089 [1] NCCL INFO comm 0x56095bf30330 rank 1 nRanks 8 nNodes 1 localRanks 8 localRank 1 MNNVL 0 | |
| tyler-rhel-newimage:1521:2083 [0] NCCL INFO comm 0x56096637cfa0 rank 0 nRanks 8 nNodes 1 localRanks 8 localRank 0 MNNVL 0 | |
| tyler-rhel-newimage:1547:2086 [4] NCCL INFO comm 0x56095cb310c0 rank 4 nRanks 8 nNodes 1 localRanks 8 localRank 4 MNNVL 0 | |
| tyler-rhel-newimage:1546:2087 [3] NCCL INFO comm 0x56095cd2ffd0 rank 3 nRanks 8 nNodes 1 localRanks 8 localRank 3 MNNVL 0 | |
| tyler-rhel-newimage:1548:2085 [5] NCCL INFO comm 0x56095bf305c0 rank 5 nRanks 8 nNodes 1 localRanks 8 localRank 5 MNNVL 0 | |
| tyler-rhel-newimage:1550:2084 [7] NCCL INFO Trees [0] -1/-1/-1->7->6 [1] -1/-1/-1->7->6 [2] -1/-1/-1->7->6 [3] -1/-1/-1->7->6 [4] -1/-1/-1->7->6 [5] -1/-1/-1->7->6 [6] -1/-1/-1->7->6 [7] -1/-1/-1->7->6 [8] -1/-1/-1->7->6 [9] -1/-1/-1->7->6 [10] -1/-1/-1->7->6 [11] -1/-1/-1->7->6 [12] -1/-1/-1->7->6 [13] -1/-1/-1->7->6 [14] -1/-1/-1->7->6 [15] -1/-1/-1->7->6 [16] -1/-1/-1->7->6 [17] -1/-1/-1->7->6 [18] -1/-1/-1->7->6 [19] -1/-1/-1->7->6 [20] -1/-1/-1->7->6 [21] -1/-1/-1->7->6 [22] -1/-1/-1->7->6 [23] -1/-1/-1->7->6 | |
| tyler-rhel-newimage:1521:2083 [0] NCCL INFO Channel 00/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:1550:2084 [7] NCCL INFO P2P Chunksize set to 524288 | |
| tyler-rhel-newimage:1549:2088 [6] NCCL INFO Trees [0] 7/-1/-1->6->5 [1] 7/-1/-1->6->5 [2] 7/-1/-1->6->5 [3] 7/-1/-1->6->5 [4] 7/-1/-1->6->5 [5] 7/-1/-1->6->5 [6] 7/-1/-1->6->5 [7] 7/-1/-1->6->5 [8] 7/-1/-1->6->5 [9] 7/-1/-1->6->5 [10] 7/-1/-1->6->5 [11] 7/-1/-1->6->5 [12] 7/-1/-1->6->5 [13] 7/-1/-1->6->5 [14] 7/-1/-1->6->5 [15] 7/-1/-1->6->5 [16] 7/-1/-1->6->5 [17] 7/-1/-1->6->5 [18] 7/-1/-1->6->5 [19] 7/-1/-1->6->5 [20] 7/-1/-1->6->5 [21] 7/-1/-1->6->5 [22] 7/-1/-1->6->5 [23] 7/-1/-1->6->5 | |
| tyler-rhel-newimage:1521:2083 [0] NCCL INFO Channel 01/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:1544:2089 [1] NCCL INFO Trees [0] 2/-1/-1->1->0 [1] 2/-1/-1->1->0 [2] 2/-1/-1->1->0 [3] 2/-1/-1->1->0 [4] 2/-1/-1->1->0 [5] 2/-1/-1->1->0 [6] 2/-1/-1->1->0 [7] 2/-1/-1->1->0 [8] 2/-1/-1->1->0 [9] 2/-1/-1->1->0 [10] 2/-1/-1->1->0 [11] 2/-1/-1->1->0 [12] 2/-1/-1->1->0 [13] 2/-1/-1->1->0 [14] 2/-1/-1->1->0 [15] 2/-1/-1->1->0 [16] 2/-1/-1->1->0 [17] 2/-1/-1->1->0 [18] 2/-1/-1->1->0 [19] 2/-1/-1->1->0 [20] 2/-1/-1->1->0 [21] 2/-1/-1->1->0 [22] 2/-1/-1->1->0 [23] 2/-1/-1->1->0 | |
| tyler-rhel-newimage:1549:2088 [6] NCCL INFO P2P Chunksize set to 524288 | |
| tyler-rhel-newimage:1548:2085 [5] NCCL INFO Trees [0] 6/-1/-1->5->4 [1] 6/-1/-1->5->4 [2] 6/-1/-1->5->4 [3] 6/-1/-1->5->4 [4] 6/-1/-1->5->4 [5] 6/-1/-1->5->4 [6] 6/-1/-1->5->4 [7] 6/-1/-1->5->4 [8] 6/-1/-1->5->4 [9] 6/-1/-1->5->4 [10] 6/-1/-1->5->4 [11] 6/-1/-1->5->4 [12] 6/-1/-1->5->4 [13] 6/-1/-1->5->4 [14] 6/-1/-1->5->4 [15] 6/-1/-1->5->4 [16] 6/-1/-1->5->4 [17] 6/-1/-1->5->4 [18] 6/-1/-1->5->4 [19] 6/-1/-1->5->4 [20] 6/-1/-1->5->4 [21] 6/-1/-1->5->4 [22] 6/-1/-1->5->4 [23] 6/-1/-1->5->4 | |
| tyler-rhel-newimage:1521:2083 [0] NCCL INFO Channel 02/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:1547:2086 [4] NCCL INFO Trees [0] 5/-1/-1->4->3 [1] 5/-1/-1->4->3 [2] 5/-1/-1->4->3 [3] 5/-1/-1->4->3 [4] 5/-1/-1->4->3 [5] 5/-1/-1->4->3 [6] 5/-1/-1->4->3 [7] 5/-1/-1->4->3 [8] 5/-1/-1->4->3 [9] 5/-1/-1->4->3 [10] 5/-1/-1->4->3 [11] 5/-1/-1->4->3 [12] 5/-1/-1->4->3 [13] 5/-1/-1->4->3 [14] 5/-1/-1->4->3 [15] 5/-1/-1->4->3 [16] 5/-1/-1->4->3 [17] 5/-1/-1->4->3 [18] 5/-1/-1->4->3 [19] 5/-1/-1->4->3 [20] 5/-1/-1->4->3 [21] 5/-1/-1->4->3 [22] 5/-1/-1->4->3 [23] 5/-1/-1->4->3 | |
| tyler-rhel-newimage:1546:2087 [3] NCCL INFO Trees [0] 4/-1/-1->3->2 [1] 4/-1/-1->3->2 [2] 4/-1/-1->3->2 [3] 4/-1/-1->3->2 [4] 4/-1/-1->3->2 [5] 4/-1/-1->3->2 [6] 4/-1/-1->3->2 [7] 4/-1/-1->3->2 [8] 4/-1/-1->3->2 [9] 4/-1/-1->3->2 [10] 4/-1/-1->3->2 [11] 4/-1/-1->3->2 [12] 4/-1/-1->3->2 [13] 4/-1/-1->3->2 [14] 4/-1/-1->3->2 [15] 4/-1/-1->3->2 [16] 4/-1/-1->3->2 [17] 4/-1/-1->3->2 [18] 4/-1/-1->3->2 [19] 4/-1/-1->3->2 [20] 4/-1/-1->3->2 [21] 4/-1/-1->3->2 [22] 4/-1/-1->3->2 [23] 4/-1/-1->3->2 | |
| tyler-rhel-newimage:1544:2089 [1] NCCL INFO P2P Chunksize set to 524288 | |
| tyler-rhel-newimage:1548:2085 [5] NCCL INFO P2P Chunksize set to 524288 | |
| tyler-rhel-newimage:1521:2083 [0] NCCL INFO Channel 03/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:1547:2086 [4] NCCL INFO P2P Chunksize set to 524288 | |
| tyler-rhel-newimage:1546:2087 [3] NCCL INFO P2P Chunksize set to 524288 | |
| tyler-rhel-newimage:1521:2083 [0] NCCL INFO Channel 04/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:1521:2083 [0] NCCL INFO Channel 05/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:1521:2083 [0] NCCL INFO Channel 06/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:1521:2083 [0] NCCL INFO Channel 07/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:1521:2083 [0] NCCL INFO Channel 08/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:1521:2083 [0] NCCL INFO Channel 09/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:1521:2083 [0] NCCL INFO Channel 10/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:1521:2083 [0] NCCL INFO Channel 11/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:1521:2083 [0] NCCL INFO Channel 12/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:1521:2083 [0] NCCL INFO Channel 13/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:1521:2083 [0] NCCL INFO Channel 14/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:1521:2083 [0] NCCL INFO Channel 15/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:1521:2083 [0] NCCL INFO Channel 16/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:1521:2083 [0] NCCL INFO Channel 17/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:1521:2083 [0] NCCL INFO Channel 18/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:1521:2083 [0] NCCL INFO Channel 19/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:1521:2083 [0] NCCL INFO Channel 20/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:1521:2083 [0] NCCL INFO Channel 21/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:1521:2083 [0] NCCL INFO Channel 22/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:1521:2083 [0] NCCL INFO Channel 23/24 : 0 1 2 3 4 5 6 7 | |
| tyler-rhel-newimage:1521:2083 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1 [2] 1/-1/-1->0->-1 [3] 1/-1/-1->0->-1 [4] 1/-1/-1->0->-1 [5] 1/-1/-1->0->-1 [6] 1/-1/-1->0->-1 [7] 1/-1/-1->0->-1 [8] 1/-1/-1->0->-1 [9] 1/-1/-1->0->-1 [10] 1/-1/-1->0->-1 [11] 1/-1/-1->0->-1 [12] 1/-1/-1->0->-1 [13] 1/-1/-1->0->-1 [14] 1/-1/-1->0->-1 [15] 1/-1/-1->0->-1 [16] 1/-1/-1->0->-1 [17] 1/-1/-1->0->-1 [18] 1/-1/-1->0->-1 [19] 1/-1/-1->0->-1 [20] 1/-1/-1->0->-1 [21] 1/-1/-1->0->-1 [22] 1/-1/-1->0->-1 [23] 1/-1/-1->0->-1 | |
| tyler-rhel-newimage:1521:2083 [0] NCCL INFO P2P Chunksize set to 524288 | |
| tyler-rhel-newimage:1545:2090 [2] NCCL INFO Channel 00/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:1545:2090 [2] NCCL INFO Channel 01/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:1545:2090 [2] NCCL INFO Channel 02/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:1545:2090 [2] NCCL INFO Channel 03/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:1545:2090 [2] NCCL INFO Channel 04/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:1545:2090 [2] NCCL INFO Channel 05/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:1550:2084 [7] NCCL INFO Channel 00/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1549:2088 [6] NCCL INFO Channel 00/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:1545:2090 [2] NCCL INFO Channel 06/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:1548:2085 [5] NCCL INFO Channel 00/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:1550:2084 [7] NCCL INFO Channel 01/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1549:2088 [6] NCCL INFO Channel 01/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:1547:2086 [4] NCCL INFO Channel 00/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:1545:2090 [2] NCCL INFO Channel 07/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:1548:2085 [5] NCCL INFO Channel 01/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:1550:2084 [7] NCCL INFO Channel 02/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1544:2089 [1] NCCL INFO Channel 00/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:1549:2088 [6] NCCL INFO Channel 02/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:1547:2086 [4] NCCL INFO Channel 01/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:1545:2090 [2] NCCL INFO Channel 08/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:1546:2087 [3] NCCL INFO Channel 00/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:1521:2083 [0] NCCL INFO Channel 00/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:1548:2085 [5] NCCL INFO Channel 02/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:1550:2084 [7] NCCL INFO Channel 03/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1544:2089 [1] NCCL INFO Channel 01/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:1549:2088 [6] NCCL INFO Channel 03/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:1545:2090 [2] NCCL INFO Channel 09/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:1547:2086 [4] NCCL INFO Channel 02/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:1521:2083 [0] NCCL INFO Channel 01/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:1546:2087 [3] NCCL INFO Channel 01/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:1548:2085 [5] NCCL INFO Channel 03/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:1550:2084 [7] NCCL INFO Channel 04/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1544:2089 [1] NCCL INFO Channel 02/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:1549:2088 [6] NCCL INFO Channel 04/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:1545:2090 [2] NCCL INFO Channel 10/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:1547:2086 [4] NCCL INFO Channel 03/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:1521:2083 [0] NCCL INFO Channel 02/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:1546:2087 [3] NCCL INFO Channel 02/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:1548:2085 [5] NCCL INFO Channel 04/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:1550:2084 [7] NCCL INFO Channel 05/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1544:2089 [1] NCCL INFO Channel 03/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:1549:2088 [6] NCCL INFO Channel 05/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:1545:2090 [2] NCCL INFO Channel 11/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:1547:2086 [4] NCCL INFO Channel 04/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:1521:2083 [0] NCCL INFO Channel 03/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:1546:2087 [3] NCCL INFO Channel 03/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:1548:2085 [5] NCCL INFO Channel 05/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:1550:2084 [7] NCCL INFO Channel 06/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1544:2089 [1] NCCL INFO Channel 04/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:1549:2088 [6] NCCL INFO Channel 06/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:1545:2090 [2] NCCL INFO Channel 12/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:1547:2086 [4] NCCL INFO Channel 05/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:1521:2083 [0] NCCL INFO Channel 04/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:1548:2085 [5] NCCL INFO Channel 06/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:1546:2087 [3] NCCL INFO Channel 04/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:1550:2084 [7] NCCL INFO Channel 07/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1544:2089 [1] NCCL INFO Channel 05/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:1549:2088 [6] NCCL INFO Channel 07/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:1545:2090 [2] NCCL INFO Channel 13/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:1547:2086 [4] NCCL INFO Channel 06/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:1521:2083 [0] NCCL INFO Channel 05/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:1548:2085 [5] NCCL INFO Channel 07/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:1546:2087 [3] NCCL INFO Channel 05/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:1550:2084 [7] NCCL INFO Channel 08/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1549:2088 [6] NCCL INFO Channel 08/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:1544:2089 [1] NCCL INFO Channel 06/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:1545:2090 [2] NCCL INFO Channel 14/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:1547:2086 [4] NCCL INFO Channel 07/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:1521:2083 [0] NCCL INFO Channel 06/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:1548:2085 [5] NCCL INFO Channel 08/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:1550:2084 [7] NCCL INFO Channel 09/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1546:2087 [3] NCCL INFO Channel 06/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:1549:2088 [6] NCCL INFO Channel 09/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:1544:2089 [1] NCCL INFO Channel 07/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:1545:2090 [2] NCCL INFO Channel 15/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:1547:2086 [4] NCCL INFO Channel 08/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:1521:2083 [0] NCCL INFO Channel 07/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:1548:2085 [5] NCCL INFO Channel 09/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:1550:2084 [7] NCCL INFO Channel 10/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1546:2087 [3] NCCL INFO Channel 07/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:1549:2088 [6] NCCL INFO Channel 10/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:1544:2089 [1] NCCL INFO Channel 08/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:1545:2090 [2] NCCL INFO Channel 16/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:1547:2086 [4] NCCL INFO Channel 09/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:1521:2083 [0] NCCL INFO Channel 08/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:1548:2085 [5] NCCL INFO Channel 10/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:1550:2084 [7] NCCL INFO Channel 11/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1546:2087 [3] NCCL INFO Channel 08/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:1549:2088 [6] NCCL INFO Channel 11/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:1544:2089 [1] NCCL INFO Channel 09/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:1545:2090 [2] NCCL INFO Channel 17/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:1547:2086 [4] NCCL INFO Channel 10/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:1548:2085 [5] NCCL INFO Channel 11/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:1521:2083 [0] NCCL INFO Channel 09/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:1550:2084 [7] NCCL INFO Channel 12/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1549:2088 [6] NCCL INFO Channel 12/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:1546:2087 [3] NCCL INFO Channel 09/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:1547:2086 [4] NCCL INFO Channel 11/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:1544:2089 [1] NCCL INFO Channel 10/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:1545:2090 [2] NCCL INFO Channel 18/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:1548:2085 [5] NCCL INFO Channel 12/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:1521:2083 [0] NCCL INFO Channel 10/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:1550:2084 [7] NCCL INFO Channel 13/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1549:2088 [6] NCCL INFO Channel 13/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:1546:2087 [3] NCCL INFO Channel 10/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:1547:2086 [4] NCCL INFO Channel 12/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:1544:2089 [1] NCCL INFO Channel 11/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:1545:2090 [2] NCCL INFO Channel 19/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:1548:2085 [5] NCCL INFO Channel 13/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:1521:2083 [0] NCCL INFO Channel 11/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:1550:2084 [7] NCCL INFO Channel 14/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1549:2088 [6] NCCL INFO Channel 14/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:1546:2087 [3] NCCL INFO Channel 11/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:1547:2086 [4] NCCL INFO Channel 13/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:1544:2089 [1] NCCL INFO Channel 12/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:1545:2090 [2] NCCL INFO Channel 20/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:1548:2085 [5] NCCL INFO Channel 14/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:1521:2083 [0] NCCL INFO Channel 12/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:1550:2084 [7] NCCL INFO Channel 15/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1549:2088 [6] NCCL INFO Channel 15/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:1546:2087 [3] NCCL INFO Channel 12/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:1547:2086 [4] NCCL INFO Channel 14/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:1544:2089 [1] NCCL INFO Channel 13/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:1545:2090 [2] NCCL INFO Channel 21/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:1548:2085 [5] NCCL INFO Channel 15/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:1521:2083 [0] NCCL INFO Channel 13/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:1550:2084 [7] NCCL INFO Channel 16/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1549:2088 [6] NCCL INFO Channel 16/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:1546:2087 [3] NCCL INFO Channel 13/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:1547:2086 [4] NCCL INFO Channel 15/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:1545:2090 [2] NCCL INFO Channel 22/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:1544:2089 [1] NCCL INFO Channel 14/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:1548:2085 [5] NCCL INFO Channel 16/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:1521:2083 [0] NCCL INFO Channel 14/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:1550:2084 [7] NCCL INFO Channel 17/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1549:2088 [6] NCCL INFO Channel 17/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:1546:2087 [3] NCCL INFO Channel 14/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:1547:2086 [4] NCCL INFO Channel 16/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:1544:2089 [1] NCCL INFO Channel 15/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:1545:2090 [2] NCCL INFO Channel 23/0 : 2[2] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:1548:2085 [5] NCCL INFO Channel 17/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:1521:2083 [0] NCCL INFO Channel 15/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:1550:2084 [7] NCCL INFO Channel 18/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1549:2088 [6] NCCL INFO Channel 18/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:1546:2087 [3] NCCL INFO Channel 15/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:1547:2086 [4] NCCL INFO Channel 17/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:1544:2089 [1] NCCL INFO Channel 16/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:1548:2085 [5] NCCL INFO Channel 18/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:1521:2083 [0] NCCL INFO Channel 16/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:1550:2084 [7] NCCL INFO Channel 19/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1549:2088 [6] NCCL INFO Channel 19/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:1546:2087 [3] NCCL INFO Channel 16/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:1547:2086 [4] NCCL INFO Channel 18/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:1548:2085 [5] NCCL INFO Channel 19/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:1544:2089 [1] NCCL INFO Channel 17/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:1521:2083 [0] NCCL INFO Channel 17/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:1550:2084 [7] NCCL INFO Channel 20/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1549:2088 [6] NCCL INFO Channel 20/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:1548:2085 [5] NCCL INFO Channel 20/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:1547:2086 [4] NCCL INFO Channel 19/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:1546:2087 [3] NCCL INFO Channel 17/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:1544:2089 [1] NCCL INFO Channel 18/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:1550:2084 [7] NCCL INFO Channel 21/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1521:2083 [0] NCCL INFO Channel 18/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:1549:2088 [6] NCCL INFO Channel 21/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:1548:2085 [5] NCCL INFO Channel 21/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:1547:2086 [4] NCCL INFO Channel 20/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:1546:2087 [3] NCCL INFO Channel 18/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:1544:2089 [1] NCCL INFO Channel 19/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:1550:2084 [7] NCCL INFO Channel 22/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1549:2088 [6] NCCL INFO Channel 22/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:1521:2083 [0] NCCL INFO Channel 19/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:1548:2085 [5] NCCL INFO Channel 22/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:1547:2086 [4] NCCL INFO Channel 21/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:1546:2087 [3] NCCL INFO Channel 19/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:1544:2089 [1] NCCL INFO Channel 20/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:1549:2088 [6] NCCL INFO Channel 23/0 : 6[6] -> 7[7] via P2P/IPC/read | |
| tyler-rhel-newimage:1550:2084 [7] NCCL INFO Channel 23/0 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1521:2083 [0] NCCL INFO Channel 20/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:1548:2085 [5] NCCL INFO Channel 23/0 : 5[5] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:1547:2086 [4] NCCL INFO Channel 22/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:1546:2087 [3] NCCL INFO Channel 20/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:1544:2089 [1] NCCL INFO Channel 21/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:1521:2083 [0] NCCL INFO Channel 21/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:1546:2087 [3] NCCL INFO Channel 21/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:1547:2086 [4] NCCL INFO Channel 23/0 : 4[4] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:1544:2089 [1] NCCL INFO Channel 22/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:1521:2083 [0] NCCL INFO Channel 22/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:1546:2087 [3] NCCL INFO Channel 22/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:1544:2089 [1] NCCL INFO Channel 23/0 : 1[1] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:1521:2083 [0] NCCL INFO Channel 23/0 : 0[0] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:1546:2087 [3] NCCL INFO Channel 23/0 : 3[3] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:1549:2088 [6] NCCL INFO Connected all rings | |
| tyler-rhel-newimage:1545:2090 [2] NCCL INFO Connected all rings | |
| tyler-rhel-newimage:1548:2085 [5] NCCL INFO Connected all rings | |
| tyler-rhel-newimage:1546:2087 [3] NCCL INFO Connected all rings | |
| tyler-rhel-newimage:1547:2086 [4] NCCL INFO Connected all rings | |
| tyler-rhel-newimage:1544:2089 [1] NCCL INFO Connected all rings | |
| tyler-rhel-newimage:1550:2084 [7] NCCL INFO Connected all rings | |
| tyler-rhel-newimage:1521:2083 [0] NCCL INFO Connected all rings | |
| tyler-rhel-newimage:1550:2084 [7] NCCL INFO Channel 00/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:1550:2084 [7] NCCL INFO Channel 01/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:1550:2084 [7] NCCL INFO Channel 02/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:1550:2084 [7] NCCL INFO Channel 03/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:1550:2084 [7] NCCL INFO Channel 04/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:1550:2084 [7] NCCL INFO Channel 05/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:1550:2084 [7] NCCL INFO Channel 06/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:1550:2084 [7] NCCL INFO Channel 07/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:1550:2084 [7] NCCL INFO Channel 08/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:1550:2084 [7] NCCL INFO Channel 09/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:1550:2084 [7] NCCL INFO Channel 10/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:1550:2084 [7] NCCL INFO Channel 11/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:1550:2084 [7] NCCL INFO Channel 12/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:1550:2084 [7] NCCL INFO Channel 13/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:1550:2084 [7] NCCL INFO Channel 14/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:1549:2088 [6] NCCL INFO Channel 00/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:1550:2084 [7] NCCL INFO Channel 15/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:1549:2088 [6] NCCL INFO Channel 01/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:1550:2084 [7] NCCL INFO Channel 16/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:1550:2084 [7] NCCL INFO Channel 17/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:1549:2088 [6] NCCL INFO Channel 02/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:1550:2084 [7] NCCL INFO Channel 18/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:1549:2088 [6] NCCL INFO Channel 03/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:1545:2090 [2] NCCL INFO Channel 00/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:1550:2084 [7] NCCL INFO Channel 19/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:1549:2088 [6] NCCL INFO Channel 04/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:1548:2085 [5] NCCL INFO Channel 00/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:1546:2087 [3] NCCL INFO Channel 00/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:1545:2090 [2] NCCL INFO Channel 01/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:1549:2088 [6] NCCL INFO Channel 05/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:1550:2084 [7] NCCL INFO Channel 20/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:1547:2086 [4] NCCL INFO Channel 00/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:1548:2085 [5] NCCL INFO Channel 01/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:1546:2087 [3] NCCL INFO Channel 01/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:1545:2090 [2] NCCL INFO Channel 02/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:1549:2088 [6] NCCL INFO Channel 06/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:1544:2089 [1] NCCL INFO Channel 00/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1550:2084 [7] NCCL INFO Channel 21/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:1547:2086 [4] NCCL INFO Channel 01/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:1548:2085 [5] NCCL INFO Channel 02/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:1546:2087 [3] NCCL INFO Channel 02/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:1545:2090 [2] NCCL INFO Channel 03/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:1549:2088 [6] NCCL INFO Channel 07/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:1544:2089 [1] NCCL INFO Channel 01/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1550:2084 [7] NCCL INFO Channel 22/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:1547:2086 [4] NCCL INFO Channel 02/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:1548:2085 [5] NCCL INFO Channel 03/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:1546:2087 [3] NCCL INFO Channel 03/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:1545:2090 [2] NCCL INFO Channel 04/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:1549:2088 [6] NCCL INFO Channel 08/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:1544:2089 [1] NCCL INFO Channel 02/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1550:2084 [7] NCCL INFO Channel 23/0 : 7[7] -> 6[6] via P2P/IPC/read | |
| tyler-rhel-newimage:1547:2086 [4] NCCL INFO Channel 03/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:1548:2085 [5] NCCL INFO Channel 04/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:1546:2087 [3] NCCL INFO Channel 04/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:1545:2090 [2] NCCL INFO Channel 05/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:1549:2088 [6] NCCL INFO Channel 09/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:1544:2089 [1] NCCL INFO Channel 03/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1547:2086 [4] NCCL INFO Channel 04/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:1548:2085 [5] NCCL INFO Channel 05/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:1546:2087 [3] NCCL INFO Channel 05/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:1545:2090 [2] NCCL INFO Channel 06/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:1549:2088 [6] NCCL INFO Channel 10/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:1544:2089 [1] NCCL INFO Channel 04/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1547:2086 [4] NCCL INFO Channel 05/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:1548:2085 [5] NCCL INFO Channel 06/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:1546:2087 [3] NCCL INFO Channel 06/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:1545:2090 [2] NCCL INFO Channel 07/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:1549:2088 [6] NCCL INFO Channel 11/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:1544:2089 [1] NCCL INFO Channel 05/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1547:2086 [4] NCCL INFO Channel 06/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:1548:2085 [5] NCCL INFO Channel 07/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:1546:2087 [3] NCCL INFO Channel 07/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:1545:2090 [2] NCCL INFO Channel 08/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:1549:2088 [6] NCCL INFO Channel 12/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:1544:2089 [1] NCCL INFO Channel 06/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1547:2086 [4] NCCL INFO Channel 07/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:1548:2085 [5] NCCL INFO Channel 08/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:1546:2087 [3] NCCL INFO Channel 08/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:1545:2090 [2] NCCL INFO Channel 09/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:1549:2088 [6] NCCL INFO Channel 13/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:1544:2089 [1] NCCL INFO Channel 07/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1547:2086 [4] NCCL INFO Channel 08/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:1548:2085 [5] NCCL INFO Channel 09/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:1546:2087 [3] NCCL INFO Channel 09/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:1545:2090 [2] NCCL INFO Channel 10/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:1549:2088 [6] NCCL INFO Channel 14/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:1544:2089 [1] NCCL INFO Channel 08/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1548:2085 [5] NCCL INFO Channel 10/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:1547:2086 [4] NCCL INFO Channel 09/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:1546:2087 [3] NCCL INFO Channel 10/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:1545:2090 [2] NCCL INFO Channel 11/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:1549:2088 [6] NCCL INFO Channel 15/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:1544:2089 [1] NCCL INFO Channel 09/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1548:2085 [5] NCCL INFO Channel 11/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:1547:2086 [4] NCCL INFO Channel 10/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:1546:2087 [3] NCCL INFO Channel 11/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:1545:2090 [2] NCCL INFO Channel 12/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:1549:2088 [6] NCCL INFO Channel 16/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:1544:2089 [1] NCCL INFO Channel 10/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1548:2085 [5] NCCL INFO Channel 12/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:1546:2087 [3] NCCL INFO Channel 12/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:1547:2086 [4] NCCL INFO Channel 11/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:1545:2090 [2] NCCL INFO Channel 13/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:1549:2088 [6] NCCL INFO Channel 17/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:1544:2089 [1] NCCL INFO Channel 11/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1547:2086 [4] NCCL INFO Channel 12/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:1548:2085 [5] NCCL INFO Channel 13/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:1546:2087 [3] NCCL INFO Channel 13/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:1545:2090 [2] NCCL INFO Channel 14/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:1549:2088 [6] NCCL INFO Channel 18/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:1544:2089 [1] NCCL INFO Channel 12/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1548:2085 [5] NCCL INFO Channel 14/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:1547:2086 [4] NCCL INFO Channel 13/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:1546:2087 [3] NCCL INFO Channel 14/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:1549:2088 [6] NCCL INFO Channel 19/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:1545:2090 [2] NCCL INFO Channel 15/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:1544:2089 [1] NCCL INFO Channel 13/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1548:2085 [5] NCCL INFO Channel 15/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:1547:2086 [4] NCCL INFO Channel 14/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:1546:2087 [3] NCCL INFO Channel 15/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:1549:2088 [6] NCCL INFO Channel 20/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:1545:2090 [2] NCCL INFO Channel 16/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:1544:2089 [1] NCCL INFO Channel 14/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1548:2085 [5] NCCL INFO Channel 16/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:1547:2086 [4] NCCL INFO Channel 15/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:1546:2087 [3] NCCL INFO Channel 16/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:1549:2088 [6] NCCL INFO Channel 21/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:1545:2090 [2] NCCL INFO Channel 17/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:1544:2089 [1] NCCL INFO Channel 15/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1548:2085 [5] NCCL INFO Channel 17/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:1547:2086 [4] NCCL INFO Channel 16/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:1546:2087 [3] NCCL INFO Channel 17/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:1549:2088 [6] NCCL INFO Channel 22/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:1544:2089 [1] NCCL INFO Channel 16/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1545:2090 [2] NCCL INFO Channel 18/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:1547:2086 [4] NCCL INFO Channel 17/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:1548:2085 [5] NCCL INFO Channel 18/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:1546:2087 [3] NCCL INFO Channel 18/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:1544:2089 [1] NCCL INFO Channel 17/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1549:2088 [6] NCCL INFO Channel 23/0 : 6[6] -> 5[5] via P2P/IPC/read | |
| tyler-rhel-newimage:1545:2090 [2] NCCL INFO Channel 19/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:1547:2086 [4] NCCL INFO Channel 18/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:1548:2085 [5] NCCL INFO Channel 19/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:1546:2087 [3] NCCL INFO Channel 19/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:1544:2089 [1] NCCL INFO Channel 18/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1545:2090 [2] NCCL INFO Channel 20/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:1548:2085 [5] NCCL INFO Channel 20/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:1547:2086 [4] NCCL INFO Channel 19/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:1546:2087 [3] NCCL INFO Channel 20/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:1544:2089 [1] NCCL INFO Channel 19/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1545:2090 [2] NCCL INFO Channel 21/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:1547:2086 [4] NCCL INFO Channel 20/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:1548:2085 [5] NCCL INFO Channel 21/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:1546:2087 [3] NCCL INFO Channel 21/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:1544:2089 [1] NCCL INFO Channel 20/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1545:2090 [2] NCCL INFO Channel 22/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:1548:2085 [5] NCCL INFO Channel 22/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:1547:2086 [4] NCCL INFO Channel 21/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:1546:2087 [3] NCCL INFO Channel 22/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:1544:2089 [1] NCCL INFO Channel 21/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1545:2090 [2] NCCL INFO Channel 23/0 : 2[2] -> 1[1] via P2P/IPC/read | |
| tyler-rhel-newimage:1548:2085 [5] NCCL INFO Channel 23/0 : 5[5] -> 4[4] via P2P/IPC/read | |
| tyler-rhel-newimage:1547:2086 [4] NCCL INFO Channel 22/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:1546:2087 [3] NCCL INFO Channel 23/0 : 3[3] -> 2[2] via P2P/IPC/read | |
| tyler-rhel-newimage:1544:2089 [1] NCCL INFO Channel 22/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1547:2086 [4] NCCL INFO Channel 23/0 : 4[4] -> 3[3] via P2P/IPC/read | |
| tyler-rhel-newimage:1544:2089 [1] NCCL INFO Channel 23/0 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1550:2084 [7] NCCL INFO Connected all trees | |
| tyler-rhel-newimage:1549:2088 [6] NCCL INFO Connected all trees | |
| tyler-rhel-newimage:1550:2084 [7] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512 | |
| tyler-rhel-newimage:1550:2084 [7] NCCL INFO 24 coll channels, 0 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer | |
| tyler-rhel-newimage:1549:2088 [6] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512 | |
| tyler-rhel-newimage:1549:2088 [6] NCCL INFO 24 coll channels, 0 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer | |
| tyler-rhel-newimage:1548:2085 [5] NCCL INFO Connected all trees | |
| tyler-rhel-newimage:1547:2086 [4] NCCL INFO Connected all trees | |
| tyler-rhel-newimage:1548:2085 [5] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512 | |
| tyler-rhel-newimage:1548:2085 [5] NCCL INFO 24 coll channels, 0 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer | |
| tyler-rhel-newimage:1547:2086 [4] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512 | |
| tyler-rhel-newimage:1547:2086 [4] NCCL INFO 24 coll channels, 0 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer | |
| tyler-rhel-newimage:1544:2089 [1] NCCL INFO Connected all trees | |
| tyler-rhel-newimage:1544:2089 [1] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512 | |
| tyler-rhel-newimage:1544:2089 [1] NCCL INFO 24 coll channels, 0 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer | |
| tyler-rhel-newimage:1546:2087 [3] NCCL INFO Connected all trees | |
| tyler-rhel-newimage:1546:2087 [3] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512 | |
| tyler-rhel-newimage:1546:2087 [3] NCCL INFO 24 coll channels, 0 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer | |
| tyler-rhel-newimage:1545:2090 [2] NCCL INFO Connected all trees | |
| tyler-rhel-newimage:1545:2090 [2] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512 | |
| tyler-rhel-newimage:1545:2090 [2] NCCL INFO 24 coll channels, 0 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer | |
| tyler-rhel-newimage:1521:2083 [0] NCCL INFO Connected all trees | |
| tyler-rhel-newimage:1521:2083 [0] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512 | |
| tyler-rhel-newimage:1521:2083 [0] NCCL INFO 24 coll channels, 0 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer | |
| INFO 2024-07-27 16:38:24,241 instructlab.model.backends.backends:336: Waiting for the vLLM server to start at http://127.0.0.1:57407/v1, this might take a moment... Attempt: 7/80 | |
| tyler-rhel-newimage:1548:2085 [5] NCCL INFO comm 0x56095bf305c0 rank 5 nranks 8 cudaDev 5 nvmlDev 5 busId c060 commId 0xaf55c5d9f1b4fe49 - Init COMPLETE | |
| tyler-rhel-newimage:1550:2084 [7] NCCL INFO comm 0x56095cd1bbb0 rank 7 nranks 8 cudaDev 7 nvmlDev 7 busId e080 commId 0xaf55c5d9f1b4fe49 - Init COMPLETE | |
| tyler-rhel-newimage:1544:2089 [1] NCCL INFO comm 0x56095bf30330 rank 1 nranks 8 cudaDev 1 nvmlDev 1 busId 8020 commId 0xaf55c5d9f1b4fe49 - Init COMPLETE | |
| tyler-rhel-newimage:1546:2087 [3] NCCL INFO comm 0x56095cd2ffd0 rank 3 nranks 8 cudaDev 3 nvmlDev 3 busId a040 commId 0xaf55c5d9f1b4fe49 - Init COMPLETE | |
| tyler-rhel-newimage:1547:2086 [4] NCCL INFO comm 0x56095cb310c0 rank 4 nranks 8 cudaDev 4 nvmlDev 4 busId c050 commId 0xaf55c5d9f1b4fe49 - Init COMPLETE | |
| tyler-rhel-newimage:1549:2088 [6] NCCL INFO comm 0x56095bf31a90 rank 6 nranks 8 cudaDev 6 nvmlDev 6 busId e070 commId 0xaf55c5d9f1b4fe49 - Init COMPLETE | |
| tyler-rhel-newimage:1521:2083 [0] NCCL INFO comm 0x56096637cfa0 rank 0 nranks 8 cudaDev 0 nvmlDev 0 busId 8010 commId 0xaf55c5d9f1b4fe49 - Init COMPLETE | |
| tyler-rhel-newimage:1545:2090 [2] NCCL INFO comm 0x56095bf30100 rank 2 nranks 8 cudaDev 2 nvmlDev 2 busId a030 commId 0xaf55c5d9f1b4fe49 - Init COMPLETE | |
| tyler-rhel-newimage:1549:2195 [6] NCCL INFO Channel 00/1 : 6[6] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1549:2195 [6] NCCL INFO Channel 01/1 : 6[6] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1545:2196 [2] NCCL INFO Channel 00/1 : 2[2] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1549:2195 [6] NCCL INFO Channel 02/1 : 6[6] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1545:2196 [2] NCCL INFO Channel 01/1 : 2[2] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1549:2195 [6] NCCL INFO Channel 03/1 : 6[6] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1545:2196 [2] NCCL INFO Channel 02/1 : 2[2] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1549:2195 [6] NCCL INFO Channel 04/1 : 6[6] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1545:2196 [2] NCCL INFO Channel 03/1 : 2[2] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1549:2195 [6] NCCL INFO Channel 05/1 : 6[6] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1545:2196 [2] NCCL INFO Channel 04/1 : 2[2] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1549:2195 [6] NCCL INFO Channel 06/1 : 6[6] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1550:2198 [7] NCCL INFO Channel 00/1 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1544:2199 [1] NCCL INFO Channel 00/1 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1548:2200 [5] NCCL INFO Channel 00/1 : 5[5] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1546:2201 [3] NCCL INFO Channel 00/1 : 3[3] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1547:2202 [4] NCCL INFO Channel 00/1 : 4[4] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1545:2196 [2] NCCL INFO Channel 05/1 : 2[2] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1548:2200 [5] NCCL INFO Channel 01/1 : 5[5] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1546:2201 [3] NCCL INFO Channel 01/1 : 3[3] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1544:2199 [1] NCCL INFO Channel 01/1 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1550:2198 [7] NCCL INFO Channel 01/1 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1547:2202 [4] NCCL INFO Channel 01/1 : 4[4] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1549:2195 [6] NCCL INFO Channel 07/1 : 6[6] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1545:2196 [2] NCCL INFO Channel 06/1 : 2[2] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1544:2199 [1] NCCL INFO Channel 02/1 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1548:2200 [5] NCCL INFO Channel 02/1 : 5[5] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1546:2201 [3] NCCL INFO Channel 02/1 : 3[3] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1550:2198 [7] NCCL INFO Channel 02/1 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1547:2202 [4] NCCL INFO Channel 02/1 : 4[4] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1549:2195 [6] NCCL INFO Channel 08/1 : 6[6] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1545:2196 [2] NCCL INFO Channel 07/1 : 2[2] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1548:2200 [5] NCCL INFO Channel 03/1 : 5[5] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1546:2201 [3] NCCL INFO Channel 03/1 : 3[3] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1544:2199 [1] NCCL INFO Channel 03/1 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1550:2198 [7] NCCL INFO Channel 03/1 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1547:2202 [4] NCCL INFO Channel 03/1 : 4[4] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1549:2195 [6] NCCL INFO Channel 09/1 : 6[6] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1545:2196 [2] NCCL INFO Channel 08/1 : 2[2] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1548:2200 [5] NCCL INFO Channel 04/1 : 5[5] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1546:2201 [3] NCCL INFO Channel 04/1 : 3[3] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1544:2199 [1] NCCL INFO Channel 04/1 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1550:2198 [7] NCCL INFO Channel 04/1 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1547:2202 [4] NCCL INFO Channel 04/1 : 4[4] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1549:2195 [6] NCCL INFO Channel 10/1 : 6[6] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1545:2196 [2] NCCL INFO Channel 09/1 : 2[2] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1544:2199 [1] NCCL INFO Channel 05/1 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1546:2201 [3] NCCL INFO Channel 05/1 : 3[3] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1550:2198 [7] NCCL INFO Channel 05/1 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1548:2200 [5] NCCL INFO Channel 05/1 : 5[5] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1547:2202 [4] NCCL INFO Channel 05/1 : 4[4] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1549:2195 [6] NCCL INFO Channel 11/1 : 6[6] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1545:2196 [2] NCCL INFO Channel 10/1 : 2[2] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1544:2199 [1] NCCL INFO Channel 06/1 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1550:2198 [7] NCCL INFO Channel 06/1 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1546:2201 [3] NCCL INFO Channel 06/1 : 3[3] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1548:2200 [5] NCCL INFO Channel 06/1 : 5[5] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1547:2202 [4] NCCL INFO Channel 06/1 : 4[4] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1549:2195 [6] NCCL INFO Channel 12/1 : 6[6] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1544:2199 [1] NCCL INFO Channel 07/1 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1545:2196 [2] NCCL INFO Channel 11/1 : 2[2] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1546:2201 [3] NCCL INFO Channel 07/1 : 3[3] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1548:2200 [5] NCCL INFO Channel 07/1 : 5[5] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1547:2202 [4] NCCL INFO Channel 07/1 : 4[4] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1550:2198 [7] NCCL INFO Channel 07/1 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1549:2195 [6] NCCL INFO Channel 13/1 : 6[6] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1545:2196 [2] NCCL INFO Channel 12/1 : 2[2] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1544:2199 [1] NCCL INFO Channel 08/1 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1546:2201 [3] NCCL INFO Channel 08/1 : 3[3] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1548:2200 [5] NCCL INFO Channel 08/1 : 5[5] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1547:2202 [4] NCCL INFO Channel 08/1 : 4[4] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1550:2198 [7] NCCL INFO Channel 08/1 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1549:2195 [6] NCCL INFO Channel 14/1 : 6[6] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1545:2196 [2] NCCL INFO Channel 13/1 : 2[2] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1546:2201 [3] NCCL INFO Channel 09/1 : 3[3] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1544:2199 [1] NCCL INFO Channel 09/1 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1548:2200 [5] NCCL INFO Channel 09/1 : 5[5] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1550:2198 [7] NCCL INFO Channel 09/1 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1547:2202 [4] NCCL INFO Channel 09/1 : 4[4] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1549:2195 [6] NCCL INFO Channel 15/1 : 6[6] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1545:2196 [2] NCCL INFO Channel 14/1 : 2[2] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1544:2199 [1] NCCL INFO Channel 10/1 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1546:2201 [3] NCCL INFO Channel 10/1 : 3[3] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1548:2200 [5] NCCL INFO Channel 10/1 : 5[5] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1550:2198 [7] NCCL INFO Channel 10/1 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1547:2202 [4] NCCL INFO Channel 10/1 : 4[4] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1549:2195 [6] NCCL INFO Channel 16/1 : 6[6] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1545:2196 [2] NCCL INFO Channel 15/1 : 2[2] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1544:2199 [1] NCCL INFO Channel 11/1 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1546:2201 [3] NCCL INFO Channel 11/1 : 3[3] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1550:2198 [7] NCCL INFO Channel 11/1 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1548:2200 [5] NCCL INFO Channel 11/1 : 5[5] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1547:2202 [4] NCCL INFO Channel 11/1 : 4[4] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1549:2195 [6] NCCL INFO Channel 17/1 : 6[6] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1545:2196 [2] NCCL INFO Channel 16/1 : 2[2] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1544:2199 [1] NCCL INFO Channel 12/1 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1546:2201 [3] NCCL INFO Channel 12/1 : 3[3] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1548:2200 [5] NCCL INFO Channel 12/1 : 5[5] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1547:2202 [4] NCCL INFO Channel 12/1 : 4[4] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1549:2195 [6] NCCL INFO Channel 18/1 : 6[6] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1545:2196 [2] NCCL INFO Channel 17/1 : 2[2] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1550:2198 [7] NCCL INFO Channel 12/1 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1544:2199 [1] NCCL INFO Channel 13/1 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1546:2201 [3] NCCL INFO Channel 13/1 : 3[3] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1547:2202 [4] NCCL INFO Channel 13/1 : 4[4] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1548:2200 [5] NCCL INFO Channel 13/1 : 5[5] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1545:2196 [2] NCCL INFO Channel 18/1 : 2[2] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1549:2195 [6] NCCL INFO Channel 19/1 : 6[6] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1550:2198 [7] NCCL INFO Channel 13/1 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1544:2199 [1] NCCL INFO Channel 14/1 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1546:2201 [3] NCCL INFO Channel 14/1 : 3[3] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1547:2202 [4] NCCL INFO Channel 14/1 : 4[4] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1545:2196 [2] NCCL INFO Channel 19/1 : 2[2] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1548:2200 [5] NCCL INFO Channel 14/1 : 5[5] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1549:2195 [6] NCCL INFO Channel 20/1 : 6[6] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1550:2198 [7] NCCL INFO Channel 14/1 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1544:2199 [1] NCCL INFO Channel 15/1 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1545:2196 [2] NCCL INFO Channel 20/1 : 2[2] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1546:2201 [3] NCCL INFO Channel 15/1 : 3[3] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1548:2200 [5] NCCL INFO Channel 15/1 : 5[5] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1544:2199 [1] NCCL INFO Channel 16/1 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1547:2202 [4] NCCL INFO Channel 15/1 : 4[4] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1550:2198 [7] NCCL INFO Channel 15/1 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1549:2195 [6] NCCL INFO Channel 21/1 : 6[6] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1545:2196 [2] NCCL INFO Channel 21/1 : 2[2] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1546:2201 [3] NCCL INFO Channel 16/1 : 3[3] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1548:2200 [5] NCCL INFO Channel 16/1 : 5[5] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1544:2199 [1] NCCL INFO Channel 17/1 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1547:2202 [4] NCCL INFO Channel 16/1 : 4[4] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1550:2198 [7] NCCL INFO Channel 16/1 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1549:2195 [6] NCCL INFO Channel 22/1 : 6[6] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1545:2196 [2] NCCL INFO Channel 22/1 : 2[2] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1546:2201 [3] NCCL INFO Channel 17/1 : 3[3] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1548:2200 [5] NCCL INFO Channel 17/1 : 5[5] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1544:2199 [1] NCCL INFO Channel 18/1 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1547:2202 [4] NCCL INFO Channel 17/1 : 4[4] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1550:2198 [7] NCCL INFO Channel 17/1 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1549:2195 [6] NCCL INFO Channel 23/1 : 6[6] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1545:2196 [2] NCCL INFO Channel 23/1 : 2[2] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1546:2201 [3] NCCL INFO Channel 18/1 : 3[3] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1548:2200 [5] NCCL INFO Channel 18/1 : 5[5] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1544:2199 [1] NCCL INFO Channel 19/1 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1547:2202 [4] NCCL INFO Channel 18/1 : 4[4] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1550:2198 [7] NCCL INFO Channel 18/1 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1549:2195 [6] NCCL INFO Channel 24/1 : 6[6] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1545:2196 [2] NCCL INFO Channel 24/1 : 2[2] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1546:2201 [3] NCCL INFO Channel 19/1 : 3[3] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1548:2200 [5] NCCL INFO Channel 19/1 : 5[5] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1544:2199 [1] NCCL INFO Channel 20/1 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1547:2202 [4] NCCL INFO Channel 19/1 : 4[4] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1550:2198 [7] NCCL INFO Channel 19/1 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1549:2195 [6] NCCL INFO Channel 25/1 : 6[6] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1545:2196 [2] NCCL INFO Channel 25/1 : 2[2] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1546:2201 [3] NCCL INFO Channel 20/1 : 3[3] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1548:2200 [5] NCCL INFO Channel 20/1 : 5[5] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1544:2199 [1] NCCL INFO Channel 21/1 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1547:2202 [4] NCCL INFO Channel 20/1 : 4[4] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1550:2198 [7] NCCL INFO Channel 20/1 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1549:2195 [6] NCCL INFO Channel 26/1 : 6[6] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1545:2196 [2] NCCL INFO Channel 26/1 : 2[2] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1546:2201 [3] NCCL INFO Channel 21/1 : 3[3] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1545:2196 [2] NCCL INFO Channel 27/1 : 2[2] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1546:2201 [3] NCCL INFO Channel 22/1 : 3[3] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1544:2199 [1] NCCL INFO Channel 22/1 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1547:2202 [4] NCCL INFO Channel 21/1 : 4[4] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1548:2200 [5] NCCL INFO Channel 21/1 : 5[5] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1550:2198 [7] NCCL INFO Channel 21/1 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1549:2195 [6] NCCL INFO Channel 27/1 : 6[6] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1545:2196 [2] NCCL INFO Channel 28/1 : 2[2] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1547:2202 [4] NCCL INFO Channel 22/1 : 4[4] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1544:2199 [1] NCCL INFO Channel 23/1 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1546:2201 [3] NCCL INFO Channel 23/1 : 3[3] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1548:2200 [5] NCCL INFO Channel 22/1 : 5[5] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1550:2198 [7] NCCL INFO Channel 22/1 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1549:2195 [6] NCCL INFO Channel 28/1 : 6[6] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1545:2196 [2] NCCL INFO Channel 29/1 : 2[2] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1546:2201 [3] NCCL INFO Channel 24/1 : 3[3] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1544:2199 [1] NCCL INFO Channel 24/1 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1547:2202 [4] NCCL INFO Channel 23/1 : 4[4] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1548:2200 [5] NCCL INFO Channel 23/1 : 5[5] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1550:2198 [7] NCCL INFO Channel 23/1 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1549:2195 [6] NCCL INFO Channel 29/1 : 6[6] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1545:2196 [2] NCCL INFO Channel 30/1 : 2[2] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1544:2199 [1] NCCL INFO Channel 25/1 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1546:2201 [3] NCCL INFO Channel 25/1 : 3[3] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1547:2202 [4] NCCL INFO Channel 24/1 : 4[4] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1548:2200 [5] NCCL INFO Channel 24/1 : 5[5] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1550:2198 [7] NCCL INFO Channel 24/1 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1549:2195 [6] NCCL INFO Channel 30/1 : 6[6] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1545:2196 [2] NCCL INFO Channel 31/1 : 2[2] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1544:2199 [1] NCCL INFO Channel 26/1 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1547:2202 [4] NCCL INFO Channel 25/1 : 4[4] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1546:2201 [3] NCCL INFO Channel 26/1 : 3[3] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1550:2198 [7] NCCL INFO Channel 25/1 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1548:2200 [5] NCCL INFO Channel 25/1 : 5[5] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1549:2195 [6] NCCL INFO Channel 31/1 : 6[6] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1544:2199 [1] NCCL INFO Channel 27/1 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1546:2201 [3] NCCL INFO Channel 27/1 : 3[3] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1544:2199 [1] NCCL INFO Channel 28/1 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1548:2200 [5] NCCL INFO Channel 26/1 : 5[5] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1547:2202 [4] NCCL INFO Channel 26/1 : 4[4] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1550:2198 [7] NCCL INFO Channel 26/1 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1544:2199 [1] NCCL INFO Channel 29/1 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1546:2201 [3] NCCL INFO Channel 28/1 : 3[3] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1550:2198 [7] NCCL INFO Channel 27/1 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1547:2202 [4] NCCL INFO Channel 27/1 : 4[4] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1548:2200 [5] NCCL INFO Channel 27/1 : 5[5] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1544:2199 [1] NCCL INFO Channel 30/1 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1546:2201 [3] NCCL INFO Channel 29/1 : 3[3] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1547:2202 [4] NCCL INFO Channel 28/1 : 4[4] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1550:2198 [7] NCCL INFO Channel 28/1 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1548:2200 [5] NCCL INFO Channel 28/1 : 5[5] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1544:2199 [1] NCCL INFO Channel 31/1 : 1[1] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1546:2201 [3] NCCL INFO Channel 30/1 : 3[3] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1547:2202 [4] NCCL INFO Channel 29/1 : 4[4] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1550:2198 [7] NCCL INFO Channel 29/1 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1548:2200 [5] NCCL INFO Channel 29/1 : 5[5] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1548:2200 [5] NCCL INFO Channel 30/1 : 5[5] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1548:2200 [5] NCCL INFO Channel 31/1 : 5[5] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1546:2201 [3] NCCL INFO Channel 31/1 : 3[3] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1550:2198 [7] NCCL INFO Channel 30/1 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1547:2202 [4] NCCL INFO Channel 30/1 : 4[4] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1550:2198 [7] NCCL INFO Channel 31/1 : 7[7] -> 0[0] via P2P/IPC/read | |
| tyler-rhel-newimage:1547:2202 [4] NCCL INFO Channel 31/1 : 4[4] -> 0[0] via P2P/IPC/read | |
| INFO 2024-07-27 16:38:29,129 instructlab.model.backends.backends:336: Waiting for the vLLM server to start at http://127.0.0.1:57407/v1, this might take a moment... Attempt: 8/80 | |
| INFO 2024-07-27 16:38:33,671 instructlab.model.backends.backends:336: Waiting for the vLLM server to start at http://127.0.0.1:57407/v1, this might take a moment... Attempt: 9/80 | |
| INFO 2024-07-27 16:38:38,328 instructlab.model.backends.backends:336: Waiting for the vLLM server to start at http://127.0.0.1:57407/v1, this might take a moment... Attempt: 10/80 | |
| INFO 2024-07-27 16:38:42,774 instructlab.model.backends.backends:336: Waiting for the vLLM server to start at http://127.0.0.1:57407/v1, this might take a moment... Attempt: 11/80 | |
| INFO 2024-07-27 16:38:47,305 instructlab.model.backends.backends:336: Waiting for the vLLM server to start at http://127.0.0.1:57407/v1, this might take a moment... Attempt: 12/80 | |
| INFO 2024-07-27 16:38:52,127 instructlab.model.backends.backends:336: Waiting for the vLLM server to start at http://127.0.0.1:57407/v1, this might take a moment... Attempt: 13/80 | |
| INFO 2024-07-27 16:38:56,614 instructlab.model.backends.backends:336: Waiting for the vLLM server to start at http://127.0.0.1:57407/v1, this might take a moment... Attempt: 14/80 | |
| embedding_mode is False. Embedding API will not work. | |
| INFO: Started server process [1521] | |
| INFO: Waiting for application startup. | |
| INFO: Application startup complete. | |
| INFO: Uvicorn running on http://127.0.0.1:57407 (Press CTRL+C to quit) | |
| INFO 2024-07-27 16:39:01,416 instructlab.model.backends.backends:336: Waiting for the vLLM server to start at http://127.0.0.1:57407/v1, this might take a moment... Attempt: 15/80 | |
| INFO: 127.0.0.1:56100 - "GET /v1/models HTTP/1.1" 200 OK | |
| INFO 2024-07-27 16:39:01,420 instructlab.model.backends.backends:343: vLLM engine successfully started at http://127.0.0.1:57407/v1 | |
| Evaluating answers for branch HEAD... | |
| 0%| | 0/826 [00:00<?, ?it/s]INFO: 127.0.0.1:56138 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56150 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56128 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56230 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56244 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56114 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56186 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56220 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56240 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56230 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56182 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56166 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56186 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56178 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56114 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56138 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56108 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 0%|▏ | 1/826 [00:12<2:53:23, 12.61s/it]INFO: 127.0.0.1:56128 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56204 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56150 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56240 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56182 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56128 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56230 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56150 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56178 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56138 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56204 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56166 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56128 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56182 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56230 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56150 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56108 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56166 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56204 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56240 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56138 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56178 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56166 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56128 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56230 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56150 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56204 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56108 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56114 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56178 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56138 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56230 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56240 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56138 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56108 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56128 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56178 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56138 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56126 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 0%|▉ | 4/826 [00:29<1:33:41, 6.84s/it]INFO: 127.0.0.1:56166 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56204 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56128 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56240 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56204 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56114 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56166 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56108 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56114 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56230 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56128 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56204 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56108 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56240 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56230 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56114 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56166 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56128 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56240 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56204 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56230 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56166 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56166 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56240 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56230 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56204 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56202 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 1%|██▋ | 11/826 [00:47<50:53, 3.75s/it]INFO: 127.0.0.1:56202 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56240 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56204 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56244 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 3%|█████ | 21/826 [00:49<22:08, 1.65s/it]INFO: 127.0.0.1:56114 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56230 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56220 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 3%|█████▊ | 24/826 [00:50<18:11, 1.36s/it]INFO: 127.0.0.1:56240 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56186 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 4%|███████ | 29/826 [00:51<13:24, 1.01s/it]INFO: 127.0.0.1:56128 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56166 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56244 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56114 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56204 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56128 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56230 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56240 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56166 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56244 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56202 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56186 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56114 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56240 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56182 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 6%|███████████▍ | 47/826 [00:59<08:09, 1.59it/s]INFO: 127.0.0.1:56186 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56166 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56244 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56204 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56186 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56230 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56150 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 7%|██████████████▎ | 59/826 [01:03<06:48, 1.88it/s]INFO: 127.0.0.1:56244 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56166 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56240 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56182 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56150 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56186 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56204 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56182 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56178 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 8%|████████████████▉ | 70/826 [01:09<06:53, 1.83it/s]INFO: 127.0.0.1:56138 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56126 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 9%|█████████████████▏ | 71/826 [01:10<06:43, 1.87it/s]INFO: 127.0.0.1:56240 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56166 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56230 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56128 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56182 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56186 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56150 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56240 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56166 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56178 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56244 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56166 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56138 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56182 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56108 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 10%|████████████████████▌ | 85/826 [01:17<06:27, 1.91it/s]INFO: 127.0.0.1:56244 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56186 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56178 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56240 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56166 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56204 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56150 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56204 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56108 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56138 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56182 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56150 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56166 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56204 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56108 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56178 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56150 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56138 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56230 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56178 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56108 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56150 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56138 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56230 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56204 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56204 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56108 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56220 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 13%|█████████████████████████▌ | 106/826 [01:32<07:25, 1.62it/s]INFO: 127.0.0.1:56230 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56138 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56204 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56220 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56108 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56178 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56138 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56230 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56202 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 14%|████████████████████████████▋ | 119/826 [01:38<06:48, 1.73it/s]INFO: 127.0.0.1:56108 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56114 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 15%|█████████████████████████████▏ | 121/826 [01:39<06:50, 1.72it/s]INFO: 127.0.0.1:56230 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56178 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56114 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56230 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56108 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56114 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56202 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56108 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56202 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56230 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56114 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56108 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56230 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56178 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56202 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56126 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 17%|█████████████████████████████████▋ | 140/826 [01:50<06:24, 1.78it/s]INFO: 127.0.0.1:56114 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56178 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56108 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56128 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 18%|██████████████████████████████████▉ | 145/826 [01:51<05:54, 1.92it/s]INFO: 127.0.0.1:56138 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56230 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56114 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56178 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56126 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56108 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56128 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56230 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56126 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56138 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56244 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56186 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 19%|█████████████████████████████████████▊ | 157/826 [01:58<05:58, 1.87it/s]INFO: 127.0.0.1:56128 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56108 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56240 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 19%|██████████████████████████████████████▌ | 160/826 [01:58<05:30, 2.02it/s]INFO: 127.0.0.1:56138 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56230 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56178 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56182 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 20%|████████████████████████████████████████▏ | 167/826 [02:01<04:57, 2.21it/s]INFO: 127.0.0.1:56244 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56108 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56114 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56178 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56230 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56166 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 20%|████████████████████████████████████████▋ | 169/826 [02:03<05:59, 1.83it/s]INFO: 127.0.0.1:56126 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56186 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56108 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56114 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56128 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56138 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56186 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56182 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56244 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56230 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56138 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56240 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56128 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56186 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56150 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 22%|██████████████████████████████████████████▉ | 178/826 [02:09<06:07, 1.77it/s]INFO: 127.0.0.1:56166 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56230 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56178 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56114 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56138 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56240 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56182 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56230 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56186 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56150 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56240 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56128 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56178 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56204 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 23%|█████████████████████████████████████████████ | 187/826 [02:13<05:35, 1.91it/s]INFO: 127.0.0.1:56108 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56138 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56166 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56220 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 23%|█████████████████████████████████████████████▎ | 188/826 [02:14<06:04, 1.75it/s]INFO: 127.0.0.1:56114 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56220 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56204 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56182 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56166 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56128 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56150 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56240 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56204 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56108 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56244 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56114 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56182 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56128 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56178 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56166 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56204 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56108 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56114 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56182 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56244 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56128 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56240 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56178 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56114 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56128 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56244 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56240 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56178 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56244 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56204 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56128 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56240 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56108 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56182 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56166 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56202 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 25%|██████████████████████████████████████████████████▌ | 210/826 [02:30<06:54, 1.49it/s]INFO: 127.0.0.1:56240 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56114 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56128 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56108 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56244 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56178 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56166 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56204 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56128 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56108 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56244 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56166 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56240 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56128 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56204 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56182 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56178 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56202 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56166 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56204 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56128 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56244 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56182 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56108 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56202 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56166 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56204 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56178 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56126 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 29%|██████████████████████████████████████████████████████████ | 241/826 [02:43<05:06, 1.91it/s]INFO: 127.0.0.1:56182 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56202 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56128 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56204 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56182 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56166 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56244 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56178 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56128 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56182 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56126 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56240 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56166 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56108 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56230 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56178 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 32%|███████████████████████████████████████████████████████████████▎ | 263/826 [02:51<04:23, 2.14it/s]INFO: 127.0.0.1:56186 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 32%|███████████████████████████████████████████████████████████████▌ | 264/826 [02:51<04:18, 2.18it/s]INFO: 127.0.0.1:56128 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56244 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56138 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 33%|█████████████████████████████████████████████████████████████████▎ | 271/826 [02:53<03:53, 2.38it/s]INFO: 127.0.0.1:56178 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56204 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56128 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56230 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56126 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56220 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56204 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 33%|██████████████████████████████████████████████████████████████████▎ | 275/826 [02:55<04:06, 2.24it/s]INFO: 127.0.0.1:56182 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56108 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56138 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56150 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 34%|███████████████████████████████████████████████████████████████████▍ | 280/826 [02:57<03:45, 2.43it/s]INFO: 127.0.0.1:56230 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56128 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56178 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56182 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56108 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56138 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56230 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56244 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56128 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56186 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56138 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56204 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56220 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56128 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56186 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56220 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56230 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56178 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56244 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56138 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56182 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56204 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56186 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56240 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56128 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56244 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56138 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56204 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56230 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56186 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56178 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56128 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56108 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56240 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56114 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 38%|███████████████████████████████████████████████████████████████████████████▏ | 312/826 [03:11<03:42, 2.31it/s]INFO: 127.0.0.1:56204 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56182 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56138 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56186 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56240 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56230 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56244 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56150 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56128 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56138 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56204 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56150 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56230 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56108 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56138 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56186 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56150 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56182 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56114 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56138 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56150 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56230 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56114 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56182 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56150 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56230 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56202 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 41%|██████████████████████████████████████████████████████████████████████████████████▏ | 341/826 [03:23<03:23, 2.38it/s]INFO: 127.0.0.1:56240 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56182 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56114 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56230 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56138 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56202 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56182 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56150 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56240 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56186 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56230 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56202 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56138 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56150 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56182 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56230 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56166 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 43%|████████████████████████████████████████████████████████████████████████████████████▊ | 352/826 [03:30<03:42, 2.13it/s]INFO: 127.0.0.1:56240 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56186 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56138 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56182 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56202 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56150 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56166 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56126 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 44%|███████████████████████████████████████████████████████████████████████████████████████▋ | 364/826 [03:34<03:19, 2.32it/s]INFO: 127.0.0.1:56138 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56240 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56126 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56186 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56150 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56186 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56202 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56182 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56138 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56166 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56126 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56202 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56114 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56166 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56240 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56138 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56126 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56220 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56240 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 47%|████████████████████████████████████████████████████████████████████████████████████████████▉ | 386/826 [03:44<03:13, 2.27it/s]INFO: 127.0.0.1:56182 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56166 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56186 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56126 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56240 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56186 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56138 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56178 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 49%|████████████████████████████████████████████████████████████████████████████████████████████████▌ | 401/826 [03:49<02:57, 2.39it/s]INFO: 127.0.0.1:56166 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56240 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56182 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56138 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56186 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56178 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56220 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56244 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 50%|███████████████████████████████████████████████████████████████████████████████████████████████████▎ | 412/826 [03:52<02:38, 2.61it/s]INFO: 127.0.0.1:56202 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56128 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 50%|███████████████████████████████████████████████████████████████████████████████████████████████████▋ | 414/826 [03:53<02:43, 2.52it/s]INFO: 127.0.0.1:56166 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56220 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56204 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 50%|████████████████████████████████████████████████████████████████████████████████████████████████████▏ | 416/826 [03:54<02:39, 2.57it/s]INFO: 127.0.0.1:56240 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56108 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 51%|████████████████████████████████████████████████████████████████████████████████████████████████████▉ | 419/826 [03:55<02:32, 2.67it/s]INFO: 127.0.0.1:56178 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56182 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56220 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56202 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56204 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56128 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56240 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56182 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56108 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56244 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56220 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56138 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56204 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56240 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56244 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56182 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56202 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56220 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56128 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56244 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56240 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56202 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56220 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56204 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56182 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56240 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56138 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56230 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 54%|███████████████████████████████████████████████████████████████████████████████████████████████████████████▉ | 448/826 [04:09<02:47, 2.26it/s]INFO: 127.0.0.1:56204 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56244 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56220 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56230 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56182 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56128 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56240 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56138 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56202 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56220 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56240 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56150 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 56%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████▎ | 462/826 [04:16<02:47, 2.18it/s]INFO: 127.0.0.1:56230 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56138 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56202 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56204 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56128 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56138 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56230 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56204 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56240 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56114 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 57%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████▏ | 470/826 [04:20<02:48, 2.11it/s]INFO: 127.0.0.1:56202 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56150 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56114 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56128 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56126 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 58%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████▋ | 480/826 [04:24<02:37, 2.20it/s]INFO: 127.0.0.1:56230 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56150 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56204 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56128 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56114 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56150 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56126 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56114 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56230 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56204 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56128 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56114 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56186 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 59%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▊ | 489/826 [04:30<02:51, 1.97it/s]INFO: 127.0.0.1:56230 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56114 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56186 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56166 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 60%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▎ | 495/826 [04:32<02:41, 2.05it/s]INFO: 127.0.0.1:56230 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56178 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 61%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▍ | 500/826 [04:34<02:30, 2.17it/s]INFO: 127.0.0.1:56128 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56114 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56186 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56230 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56178 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56108 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 62%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▍ | 508/826 [04:38<02:27, 2.15it/s]INFO: 127.0.0.1:56204 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56178 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56114 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56128 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56178 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56204 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56128 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56204 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56186 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56114 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56204 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56178 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56114 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56244 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 64%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▍ | 529/826 [04:48<02:22, 2.09it/s]INFO: 127.0.0.1:56186 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56186 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56178 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56128 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56182 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 64%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▏ | 532/826 [04:51<02:34, 1.91it/s]INFO: 127.0.0.1:56204 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56244 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56220 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 65%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▎ | 537/826 [04:52<02:19, 2.08it/s]INFO: 127.0.0.1:56178 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56204 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56220 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56114 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56128 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56230 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56128 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56138 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 66%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▎ | 545/826 [04:57<02:18, 2.03it/s]INFO: 127.0.0.1:56220 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56204 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56244 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56114 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56240 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 66%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████ | 548/826 [04:58<02:15, 2.04it/s]INFO: 127.0.0.1:56202 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 67%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▌ | 550/826 [04:58<02:06, 2.18it/s]INFO: 127.0.0.1:56128 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56182 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56230 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56114 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56202 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56240 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56138 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56182 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56128 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56244 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56204 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56202 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56138 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56114 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56182 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56230 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56244 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56150 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 68%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▉ | 560/826 [05:05<02:29, 1.78it/s]INFO: 127.0.0.1:56126 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 68%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▏ | 561/826 [05:06<02:27, 1.79it/s]INFO: 127.0.0.1:56240 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56138 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56182 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56220 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56114 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56150 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56240 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56244 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56182 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56128 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56126 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56150 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56166 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 69%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▌ | 571/826 [05:11<02:17, 1.86it/s]INFO: 127.0.0.1:56244 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56220 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56126 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56138 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56114 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56128 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56126 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56150 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56220 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56114 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56244 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56108 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 70%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▍ | 579/826 [05:17<02:27, 1.68it/s]INFO: 127.0.0.1:56166 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56126 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56182 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56244 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56150 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56166 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56182 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56220 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56108 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56126 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56138 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56244 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56150 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56166 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56220 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56138 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56108 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56244 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56126 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56138 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56182 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56220 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56150 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56166 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56128 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56108 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56138 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56186 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 72%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▎ | 595/826 [05:30<02:46, 1.39it/s]INFO: 127.0.0.1:56244 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56126 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56182 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56128 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56166 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56178 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 73%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████ | 602/826 [05:32<02:17, 1.63it/s]INFO: 127.0.0.1:56186 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56220 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56108 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56244 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56182 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56126 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56150 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56108 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56166 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56128 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56244 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56108 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56178 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56182 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56126 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56166 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56150 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56128 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56178 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56108 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56204 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 76%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▊ | 626/826 [05:41<01:39, 2.02it/s]INFO: 127.0.0.1:56202 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 76%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████ | 627/826 [05:42<01:36, 2.07it/s]INFO: 127.0.0.1:56150 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56126 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56128 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56204 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56230 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 76%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████ | 631/826 [05:44<01:38, 1.98it/s]INFO: 127.0.0.1:56126 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56202 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56150 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56178 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56186 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56240 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 78%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▍ | 641/826 [05:47<01:21, 2.26it/s]INFO: 127.0.0.1:56128 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56126 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56204 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56230 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56108 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56178 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56202 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56108 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56150 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56204 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56230 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56186 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56128 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56114 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 80%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▎ | 657/826 [05:54<01:12, 2.33it/s]INFO: 127.0.0.1:56108 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56204 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56150 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56178 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56202 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56230 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56114 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56204 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56186 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56178 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56202 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56128 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56186 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56150 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56114 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56108 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56230 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56178 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56186 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56230 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56178 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56202 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56178 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56230 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56186 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56150 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56202 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56108 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56138 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 83%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▎ | 686/826 [06:09<01:05, 2.13it/s]INFO: 127.0.0.1:56202 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56220 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 84%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▍ | 695/826 [06:11<00:56, 2.31it/s]INFO: 127.0.0.1:56186 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56108 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56230 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56138 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56220 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56244 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 85%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▌ | 704/826 [06:15<00:53, 2.30it/s]INFO: 127.0.0.1:56150 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56230 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56202 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56108 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56182 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 86%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▎ | 707/826 [06:17<00:52, 2.27it/s]INFO: 127.0.0.1:56166 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 86%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▊ | 709/826 [06:17<00:49, 2.36it/s]INFO: 127.0.0.1:56244 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56220 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56138 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56230 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56150 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56108 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56138 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56182 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56244 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56166 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56138 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56244 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56150 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56220 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56108 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56138 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56220 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56240 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 88%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▉ | 726/826 [06:26<00:47, 2.10it/s]INFO: 127.0.0.1:56166 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56244 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56126 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 88%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▍ | 728/826 [06:27<00:46, 2.12it/s]INFO: 127.0.0.1:56230 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56150 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56108 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56240 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56230 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56150 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56166 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56244 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56126 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56230 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56230 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56204 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 91%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▏ | 748/826 [06:35<00:34, 2.23it/s]INFO: 127.0.0.1:56150 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56220 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56128 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 91%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▏ | 752/826 [06:37<00:32, 2.31it/s]INFO: 127.0.0.1:56114 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 91%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▉ | 755/826 [06:38<00:29, 2.37it/s]INFO: 127.0.0.1:56230 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56244 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56128 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56240 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56114 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56230 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56150 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56220 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56244 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56114 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56138 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56178 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 92%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▊ | 763/826 [06:44<00:34, 1.81it/s]INFO: 127.0.0.1:56240 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:56186 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 93%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▉ | 772/826 [06:49<00:28, 1.88it/s]INFO: 127.0.0.1:56202 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 94%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▉ | 780/826 [06:51<00:21, 2.12it/s]INFO: 127.0.0.1:56182 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 96%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▌ | 791/826 [06:55<00:14, 2.36it/s]INFO: 127.0.0.1:56108 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 98%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▍ | 807/826 [07:02<00:07, 2.40it/s]INFO: 127.0.0.1:56166 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 98%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▍ | 811/826 [07:04<00:06, 2.34it/s]INFO: 127.0.0.1:56126 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 98%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▊ | 813/826 [07:04<00:05, 2.51it/s]INFO: 127.0.0.1:56204 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 99%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▌ | 816/826 [07:05<00:04, 2.40it/s]INFO: 127.0.0.1:56128 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▎| 823/826 [07:08<00:01, 2.64it/s]INFO: 127.0.0.1:56230 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 826/826 [07:08<00:00, 1.93it/s] | |
| Evaluating answers for branch HEAD... | |
| 0%| | 0/826 [00:00<?, ?it/s]INFO: 127.0.0.1:60440 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60542 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60562 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60476 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60512 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60514 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60528 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60476 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60450 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60488 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60426 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 0%|▏ | 1/826 [00:04<1:07:04, 4.88s/it]INFO: 127.0.0.1:60498 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60528 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60558 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60514 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60440 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60562 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60526 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60542 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60450 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60426 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60468 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60526 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60488 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60440 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60450 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60558 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60498 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60542 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60526 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60468 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60488 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60440 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60562 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60542 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60498 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60558 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60426 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60542 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60450 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60526 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60440 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60488 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60498 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60562 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60514 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60542 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60426 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60440 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60558 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60426 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60562 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60464 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 0%|▉ | 4/826 [00:21<1:13:05, 5.34s/it]INFO: 127.0.0.1:60526 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60542 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60426 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60450 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60498 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60464 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60514 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60498 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60558 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60450 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60562 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60558 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60464 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60440 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60498 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60562 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60514 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60464 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60558 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60450 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60440 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60464 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60498 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60450 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60498 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60464 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60450 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60498 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60524 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 1%|██▋ | 11/826 [00:38<44:41, 3.29s/it]INFO: 127.0.0.1:60514 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60512 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 3%|█████ | 21/826 [00:41<19:44, 1.47s/it]INFO: 127.0.0.1:60514 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60450 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60558 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60476 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 3%|█████▊ | 24/826 [00:42<16:48, 1.26s/it]INFO: 127.0.0.1:60524 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60528 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 4%|███████ | 29/826 [00:43<12:17, 1.08it/s]INFO: 127.0.0.1:60498 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60514 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60440 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60476 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60558 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60464 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60450 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60440 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60498 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60524 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60476 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60558 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60514 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60512 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60464 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60524 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60468 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 6%|███████████▍ | 47/826 [00:51<08:03, 1.61it/s]INFO: 127.0.0.1:60476 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60512 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60558 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60450 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60498 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60476 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60488 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 7%|██████████████▎ | 59/826 [00:55<06:26, 1.98it/s]INFO: 127.0.0.1:60558 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60512 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60524 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60468 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60488 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60450 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60498 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60526 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 8%|████████████████▉ | 70/826 [01:00<06:06, 2.06it/s]INFO: 127.0.0.1:60542 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 9%|█████████████████▏ | 71/826 [01:00<05:55, 2.12it/s]INFO: 127.0.0.1:60468 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60426 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 9%|█████████████████▍ | 72/826 [01:01<05:53, 2.13it/s]INFO: 127.0.0.1:60512 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60524 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60476 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60440 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60526 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60450 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60426 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60488 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60524 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60512 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60542 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60558 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60524 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60526 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60562 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 10%|████████████████████▌ | 85/826 [01:07<06:09, 2.00it/s]INFO: 127.0.0.1:60542 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60512 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60488 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60558 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60450 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60498 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60426 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60562 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60498 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60524 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60526 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60426 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60450 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60562 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60512 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60498 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60426 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60524 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60476 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60498 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60512 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60426 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60524 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60562 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60476 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60476 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60512 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60562 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60476 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60524 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60528 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 13%|█████████████████████████▌ | 106/826 [01:22<07:21, 1.63it/s]INFO: 127.0.0.1:60562 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60512 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60498 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60524 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60476 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60512 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60514 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 14%|████████████████████████████▋ | 119/826 [01:28<06:30, 1.81it/s]INFO: 127.0.0.1:60464 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 15%|█████████████████████████████▏ | 121/826 [01:29<06:31, 1.80it/s]INFO: 127.0.0.1:60476 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60498 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60464 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60476 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60512 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60514 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60464 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60512 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60464 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60476 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60514 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60512 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60476 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60468 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 17%|█████████████████████████████████▋ | 140/826 [01:38<05:59, 1.91it/s]INFO: 127.0.0.1:60498 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60464 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60514 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60440 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60468 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 18%|██████████████████████████████████▉ | 145/826 [01:40<05:42, 1.99it/s]INFO: 127.0.0.1:60512 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60524 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60476 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60514 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60440 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60464 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60512 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60468 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60476 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60464 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60524 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60542 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 19%|█████████████████████████████████████▊ | 157/826 [01:46<05:34, 2.00it/s]INFO: 127.0.0.1:60488 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 19%|██████████████████████████████████████▎ | 159/826 [01:47<05:21, 2.08it/s]INFO: 127.0.0.1:60558 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 19%|██████████████████████████████████████▌ | 160/826 [01:47<05:21, 2.07it/s]INFO: 127.0.0.1:60512 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60468 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60524 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60476 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60440 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60526 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 20%|████████████████████████████████████████▏ | 167/826 [01:50<04:54, 2.24it/s]INFO: 127.0.0.1:60440 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60512 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60514 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60488 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60450 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 20%|████████████████████████████████████████▋ | 169/826 [01:51<05:21, 2.04it/s]INFO: 127.0.0.1:60476 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60464 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60440 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60542 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60558 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60524 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60514 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60440 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60488 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60526 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60450 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60512 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60440 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60426 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 22%|██████████████████████████████████████████▉ | 178/826 [01:57<05:40, 1.90it/s]INFO: 127.0.0.1:60524 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60468 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60476 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60450 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60558 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60512 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60488 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60440 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60524 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60468 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60450 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60514 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60440 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60426 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60528 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 23%|█████████████████████████████████████████████ | 187/826 [02:01<05:36, 1.90it/s]INFO: 127.0.0.1:60558 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60542 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60476 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60488 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60562 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 23%|█████████████████████████████████████████████▎ | 188/826 [02:02<05:56, 1.79it/s]INFO: 127.0.0.1:60512 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60562 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60524 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60488 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60426 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60558 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60514 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60440 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60524 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60526 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60542 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60488 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60528 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60512 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60558 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60426 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60524 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60526 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60542 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60528 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60488 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60512 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60440 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60558 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60542 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60512 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60440 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60488 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60558 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60440 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60512 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60524 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60488 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60526 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60498 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 25%|██████████████████████████████████████████████████▌ | 210/826 [02:18<06:45, 1.52it/s]INFO: 127.0.0.1:60426 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60528 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60488 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60542 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60524 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60526 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60440 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60558 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60426 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60512 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60524 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60526 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60440 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60488 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60524 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60512 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60498 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60558 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60426 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60488 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60528 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60512 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60498 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60440 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60526 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60558 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60488 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60528 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60464 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 29%|██████████████████████████████████████████████████████████ | 241/826 [02:31<05:03, 1.93it/s]INFO: 127.0.0.1:60426 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60512 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60526 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60488 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60498 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60464 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60526 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60440 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60528 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60426 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60498 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60512 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60526 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60468 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 32%|███████████████████████████████████████████████████████████████▎ | 263/826 [02:39<04:19, 2.17it/s]INFO: 127.0.0.1:60440 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60450 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 32%|███████████████████████████████████████████████████████████████▌ | 264/826 [02:39<04:14, 2.21it/s]INFO: 127.0.0.1:60558 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60426 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60524 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60498 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60476 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 33%|█████████████████████████████████████████████████████████████████▎ | 271/826 [02:41<03:53, 2.38it/s]INFO: 127.0.0.1:60528 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60558 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60464 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60498 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60426 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60526 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60562 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 33%|██████████████████████████████████████████████████████████████████▎ | 275/826 [02:43<04:06, 2.23it/s]INFO: 127.0.0.1:60464 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60450 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60528 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60512 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60514 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 34%|███████████████████████████████████████████████████████████████████▍ | 280/826 [02:45<03:48, 2.39it/s]INFO: 127.0.0.1:60426 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60498 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60528 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60558 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60450 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60512 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60426 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60476 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60498 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60512 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60524 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60464 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60498 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60426 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60512 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60562 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60528 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60476 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60498 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60524 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60450 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60512 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60464 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60468 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60426 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60524 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60512 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60498 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60528 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60476 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60464 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60558 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60426 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60542 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 38%|███████████████████████████████████████████████████████████████████████████▏ | 312/826 [02:59<03:41, 2.32it/s]INFO: 127.0.0.1:60468 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60498 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60450 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60512 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60476 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60542 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60528 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60524 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60514 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60558 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60514 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60512 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60528 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60498 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60426 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60514 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60528 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60476 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60450 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60468 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60514 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60476 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60498 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60468 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60450 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60476 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60498 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60488 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 41%|██████████████████████████████████████████████████████████████████████████████████▏ | 341/826 [03:11<03:25, 2.36it/s]INFO: 127.0.0.1:60450 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60542 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60468 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60498 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60514 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60488 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60450 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60542 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60476 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60528 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60498 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60488 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60514 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60542 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60450 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60498 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60440 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 43%|████████████████████████████████████████████████████████████████████████████████████▊ | 352/826 [03:18<03:41, 2.14it/s]INFO: 127.0.0.1:60476 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60528 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60514 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60450 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60488 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60440 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60542 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60526 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 44%|███████████████████████████████████████████████████████████████████████████████████████▋ | 364/826 [03:22<03:22, 2.28it/s]INFO: 127.0.0.1:60514 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60476 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60528 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60526 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60440 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60526 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60488 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60542 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60450 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60514 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60528 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60488 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60468 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60514 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60476 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60450 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60562 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 47%|████████████████████████████████████████████████████████████████████████████████████████████▉ | 386/826 [03:32<03:11, 2.30it/s]INFO: 127.0.0.1:60542 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60528 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60476 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60514 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60526 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60562 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60528 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60450 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60526 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60514 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60464 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 49%|████████████████████████████████████████████████████████████████████████████████████████████████▌ | 401/826 [03:37<02:57, 2.39it/s]INFO: 127.0.0.1:60528 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60476 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60526 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60450 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60514 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60542 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60524 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 50%|███████████████████████████████████████████████████████████████████████████████████████████████████▎ | 412/826 [03:40<02:39, 2.60it/s]INFO: 127.0.0.1:60488 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60464 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60558 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 50%|███████████████████████████████████████████████████████████████████████████████████████████████████▋ | 414/826 [03:41<02:43, 2.52it/s]INFO: 127.0.0.1:60542 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60512 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 50%|████████████████████████████████████████████████████████████████████████████████████████████████████▏ | 416/826 [03:42<02:39, 2.57it/s]INFO: 127.0.0.1:60528 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60426 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 51%|████████████████████████████████████████████████████████████████████████████████████████████████████▉ | 419/826 [03:43<02:35, 2.62it/s]INFO: 127.0.0.1:60514 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60476 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60542 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60512 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60488 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60464 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60528 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60476 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60542 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60524 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60426 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60526 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60488 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60528 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60524 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60476 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60512 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60426 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60464 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60524 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60528 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60488 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60512 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60426 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60476 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60528 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60498 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 54%|███████████████████████████████████████████████████████████████████████████████████████████████████████████▉ | 448/826 [03:57<02:46, 2.27it/s]INFO: 127.0.0.1:60526 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60426 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60524 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60512 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60526 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60476 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60464 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60528 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60488 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60498 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60512 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60528 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60440 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60526 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 56%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████▎ | 462/826 [04:04<02:48, 2.16it/s]INFO: 127.0.0.1:60488 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60498 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60426 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60464 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60488 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60426 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60526 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60528 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60468 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 57%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████▏ | 470/826 [04:08<02:46, 2.14it/s]INFO: 127.0.0.1:60498 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60468 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60464 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60440 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60562 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 58%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████▋ | 480/826 [04:12<02:34, 2.23it/s]INFO: 127.0.0.1:60426 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60526 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60440 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60464 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60468 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60526 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60562 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60468 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60426 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60440 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60464 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60450 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 59%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▊ | 489/826 [04:17<02:47, 2.01it/s]INFO: 127.0.0.1:60468 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60426 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60558 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 60%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▎ | 495/826 [04:19<02:35, 2.13it/s]INFO: 127.0.0.1:60450 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60468 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60514 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 61%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▍ | 500/826 [04:22<02:29, 2.18it/s]INFO: 127.0.0.1:60426 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60464 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60558 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60450 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60514 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60426 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60542 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 62%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▍ | 508/826 [04:26<02:30, 2.12it/s]INFO: 127.0.0.1:60440 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60426 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60558 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60464 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60426 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60440 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60464 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60450 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60440 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60558 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60450 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60426 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60558 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60524 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 64%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▍ | 529/826 [04:36<02:22, 2.08it/s]INFO: 127.0.0.1:60440 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60476 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60426 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 64%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▏ | 532/826 [04:38<02:30, 1.95it/s]INFO: 127.0.0.1:60450 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60440 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60464 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60524 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60512 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 65%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▎ | 537/826 [04:41<02:31, 1.90it/s]INFO: 127.0.0.1:60476 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60512 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60464 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60558 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60450 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60514 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60450 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60488 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 66%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▎ | 545/826 [04:46<02:30, 1.87it/s]INFO: 127.0.0.1:60464 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60512 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60524 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60558 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60528 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 66%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████ | 548/826 [04:47<02:21, 1.97it/s]INFO: 127.0.0.1:60450 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60440 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60514 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60498 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 67%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▌ | 550/826 [04:48<02:33, 1.79it/s]INFO: 127.0.0.1:60558 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60528 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60488 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60450 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60524 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60440 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60514 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60512 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60528 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60558 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60450 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60524 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60498 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60514 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60526 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 68%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▉ | 560/826 [04:54<02:29, 1.78it/s]INFO: 127.0.0.1:60562 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 68%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▏ | 561/826 [04:54<02:22, 1.85it/s]INFO: 127.0.0.1:60488 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60558 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60464 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60524 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60450 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60526 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60488 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60514 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60562 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60464 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60440 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60526 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60514 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60468 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 69%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▌ | 571/826 [05:00<02:25, 1.76it/s]INFO: 127.0.0.1:60524 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60440 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60558 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60450 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60464 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60524 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60526 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60450 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60440 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60468 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60542 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 70%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▍ | 579/826 [05:05<02:22, 1.73it/s]INFO: 127.0.0.1:60514 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60524 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60562 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60526 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60468 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60514 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60562 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60542 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60450 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60524 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60558 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60526 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60468 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60542 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60558 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60514 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60450 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60526 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60524 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60514 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60562 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60558 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60464 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60468 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60542 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60450 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60426 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 72%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▎ | 595/826 [05:18<02:42, 1.42it/s]INFO: 127.0.0.1:60514 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60526 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60524 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60562 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60542 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60476 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 73%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████ | 602/826 [05:20<02:13, 1.67it/s]INFO: 127.0.0.1:60514 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60468 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60558 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60450 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60526 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60562 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60524 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60464 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60476 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60450 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60542 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60526 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60476 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60514 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60562 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60524 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60464 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60450 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60542 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60514 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60476 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60562 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60512 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60528 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 76%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▊ | 626/826 [05:30<01:38, 2.04it/s]INFO: 127.0.0.1:60450 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60562 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60542 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60524 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60498 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 76%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████ | 631/826 [05:32<01:35, 2.03it/s]INFO: 127.0.0.1:60450 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60512 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60528 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60514 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60468 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60498 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60562 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60488 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 78%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▍ | 641/826 [05:36<01:24, 2.18it/s]INFO: 127.0.0.1:60524 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60542 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60450 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60476 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60528 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60542 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60512 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60488 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60524 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60528 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60514 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60498 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60512 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60542 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60440 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 80%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▎ | 657/826 [05:43<01:14, 2.28it/s]INFO: 127.0.0.1:60476 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60488 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60514 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60450 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60524 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60512 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60440 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60476 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60528 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60542 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60524 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60514 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60498 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60528 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60488 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60512 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60498 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60528 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60512 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60440 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60528 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60498 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60488 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60542 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60514 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60488 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60426 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60528 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 83%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▎ | 686/826 [05:57<01:05, 2.15it/s]INFO: 127.0.0.1:60440 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60542 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60426 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60558 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 84%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▍ | 695/826 [05:59<00:55, 2.34it/s]INFO: 127.0.0.1:60498 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60514 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60542 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60488 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60440 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60498 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60526 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 85%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▌ | 704/826 [06:04<00:53, 2.27it/s]INFO: 127.0.0.1:60558 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60426 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60514 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60542 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60464 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 86%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▊ | 709/826 [06:05<00:49, 2.34it/s]INFO: 127.0.0.1:60440 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60498 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60514 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60528 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60526 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60558 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60498 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60464 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60426 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60528 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60440 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60558 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60498 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60426 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60526 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60468 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 88%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▉ | 726/826 [06:13<00:44, 2.27it/s]INFO: 127.0.0.1:60542 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60562 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60528 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 88%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▍ | 728/826 [06:14<00:43, 2.23it/s]INFO: 127.0.0.1:60464 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60498 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60542 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60528 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60498 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60426 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60468 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60526 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60542 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60558 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60468 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60450 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 91%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▏ | 748/826 [06:22<00:32, 2.41it/s]INFO: 127.0.0.1:60526 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60476 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 91%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▏ | 752/826 [06:23<00:29, 2.51it/s]INFO: 127.0.0.1:60498 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60524 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 91%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▉ | 755/826 [06:24<00:27, 2.59it/s]INFO: 127.0.0.1:60526 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60464 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60450 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60476 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60464 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60558 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60498 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60468 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60440 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60512 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 92%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▊ | 763/826 [06:29<00:29, 2.15it/s]INFO: 127.0.0.1:60526 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| INFO: 127.0.0.1:60488 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 94%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▉ | 780/826 [06:36<00:20, 2.27it/s]INFO: 127.0.0.1:60514 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 96%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▌ | 791/826 [06:41<00:15, 2.29it/s]INFO: 127.0.0.1:60562 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 98%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▍ | 807/826 [06:46<00:07, 2.56it/s]INFO: 127.0.0.1:60528 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 98%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▍ | 811/826 [06:48<00:06, 2.47it/s]INFO: 127.0.0.1:60426 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 98%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▊ | 813/826 [06:48<00:04, 2.63it/s]INFO: 127.0.0.1:60542 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 99%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▌ | 816/826 [06:49<00:03, 2.62it/s]INFO: 127.0.0.1:60524 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▎| 823/826 [06:51<00:01, 2.77it/s]INFO: 127.0.0.1:60450 - "POST /v1/chat/completions HTTP/1.1" 200 OK | |
| 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 826/826 [06:52<00:00, 2.00it/s] | |
| INFO: Shutting down | |
| INFO: Waiting for application shutdown. | |
| INFO: Application shutdown complete. | |
| INFO: Finished server process [1521] | |
| [rank0]:[W CudaIPCTypes.cpp:16] Producer process has been terminated before all shared CUDA tensors released. See Note [Sharing CUDA tensors] | |
| /usr/lib64/python3.11/multiprocessing/resource_tracker.py:254: UserWarning: resource_tracker: There appear to be 1 leaked shared_memory objects to clean up at shutdown | |
| warnings.warn('resource_tracker: There appear to be %d ' | |
| # SKILL EVALUATION REPORT | |
| ## BASE MODEL | |
| /var/instructlabbigdisk/instructlab/models/ibm-granite/granite-7b-base/ | |
| ## MODEL | |
| /var/instructlabbigdisk/instructlab/skillscheckpoints/hf_format/samples_1056/ | |
| ### IMPROVEMENTS: | |
| 1. ompositional_skills/writing/grounded/editing/punctuation/qna.yaml (+4.5) | |
| 2. ompositional_skills/linguistics/rhyming_words/qna.yaml (+0.67) | |
| 3. ompositional_skills/extraction/technical_paper/equations/bullet_points/qna.yaml (+0.5) | |
| 4. ompositional_skills/roleplay/explain_like_you_are/fictional/movies/qna.yaml (+0.2) | |
| 5. oundational_skills/reasoning/logical_reasoning/tabular/qna.yaml (+0.2) | |
| 6. ompositional_skills/STEM/math/arithmetic_w_grammar/qna.yaml (+0.17) | |
| 7. oundational_skills/reasoning/theory_of_mind/qna.yaml (+0.07) | |
| 8. ompositional_skills/roleplay/explain_like_you_are/non_fictional/popular_personalities/qna.yaml (+0.07) | |
| ### REGRESSIONS: | |
| 1. ompositional_skills/STEM/math/reasoning/qna.yaml (-1.0) | |
| 2. ompositional_skills/linguistics/bullet_lists/qna.yaml (-0.67) | |
| 3. ompositional_skills/extraction/technical_paper/equations/plain_text/qna.yaml (-0.6) | |
| 4. ompositional_skills/writing/freeform/prose/stories/qna.yaml (-0.5) | |
| 5. ompositional_skills/extraction/technical_paper/tables/bullet_points/qna.yaml (-0.4) | |
| 6. ompositional_skills/extraction/technical_paper/abstract/plain_text/qna.yaml (-0.4) | |
| 7. ompositional_skills/roleplay/explain_like_you_are/abstract/qna.yaml (-0.33) | |
| 8. ompositional_skills/linguistics/complete_common_expressions/qna.yaml (-0.25) | |
| 9. oundational_skills/reasoning/linguistics_reasoning/object_identification/qna.yaml (-0.17) | |
| 10. ompositional_skills/extraction/receipt/plain_text/qna.yaml (-0.17) | |
| 11. ompositional_skills/roleplay/explain_like_i_am/graduate/qna.yaml (-0.17) | |
| 12. oundational_skills/reasoning/logical_reasoning/general/qna.yaml (-0.04) | |
| ### NO CHANGE: | |
| 1. ompositional_skills/linguistics/organize_lists/qna.yaml | |
| 2. ompositional_skills/extraction/services_agreement/reasoning/qna.yaml | |
| 3. ompositional_skills/extraction/technical_paper/equations/csv/qna.yaml | |
| 4. oundational_skills/reasoning/temporal_reasoning/qna.yaml | |
| 5. ompositional_skills/writing/grounded/summarization/wiki_insights/five_point/qna.yaml | |
| 6. ompositional_skills/extraction/fda_filing/csv/qna.yaml | |
| 7. oundational_skills/reasoning/linguistics_reasoning/odd_one_out/qna.yaml | |
| 8. ompositional_skills/linguistics/pattern_recognition/qna.yaml | |
| 9. ompositional_skills/writing/freeform/grammar/basic_grammer_tests/qna.yaml | |
| 10. oundational_skills/reasoning/unconventional_reasoning/lower_score_wins/qna.yaml | |
| 11. ompositional_skills/extraction/invoice/markdown/qna.yaml | |
| 12. ompositional_skills/writing/freeform/poetry/narrative_poetry/qna.yaml | |
| 13. ompositional_skills/extraction/commercial_lease_agreement/markdown/qna.yaml | |
| 14. ompositional_skills/extraction/invoice/bullet_points/qna.yaml | |
| 15. ompositional_skills/extraction/invoice/csv/qna.yaml | |
| 16. ompositional_skills/STEM/science/units_conversion/temperature_conversion/qna.yaml | |
| 17. ompositional_skills/writing/freeform/emoji/qna.yaml | |
| 18. ompositional_skills/extraction/services_agreement/plain_text/qna.yaml | |
| 19. ompositional_skills/writing/freeform/social_media/instagram/qna.yaml | |
| 20. ompositional_skills/extraction/receipt/bullet_points/qna.yaml | |
| 21. ompositional_skills/extraction/technical_paper/tables/plain_text/qna.yaml | |
| 22. ompositional_skills/extraction/fda_filing/bullet_points/qna.yaml | |
| 23. ompositional_skills/extraction/email/reasoning/qna.yaml | |
| 24. oundational_skills/reasoning/logical_reasoning/causal/qna.yaml | |
| 25. ompositional_skills/STEM/science/geography/qna.yaml | |
| 26. ompositional_skills/linguistics/word_gen/qna.yaml | |
| 27. ompositional_skills/writing/freeform/prose/emails/informal/qna.yaml | |
| 28. ompositional_skills/general/tables/editing/combining_altering/qna.yaml | |
| 29. oundational_skills/reasoning/mathematical_reasoning/qna.yaml | |
| 30. ompositional_skills/extraction/technical_paper/equations/markdown/qna.yaml | |
| 31. ompositional_skills/writing/freeform/prose/emails/formal/qna.yaml | |
| 32. ompositional_skills/writing/freeform/legal/agreement/qna.yaml | |
| 33. ompositional_skills/extraction/information/named_entities/person_names/qna.yaml | |
| 34. ompositional_skills/extraction/technical_paper/abstract/csv/qna.yaml | |
| 35. ompositional_skills/extraction/email/bullet_points/qna.yaml | |
| 36. ompositional_skills/writing/grounded/editing/grammar/qna.yaml | |
| 37. ompositional_skills/extraction/annual_report/reasoning/qna.yaml | |
| 38. ompositional_skills/extraction/technical_paper/abstract/bullet_points/qna.yaml | |
| 39. ompositional_skills/extraction/annual_report/csv/qna.yaml | |
| 40. ompositional_skills/writing/freeform/poetry/sonnet/qna.yaml | |
| 41. ompositional_skills/roleplay/explain_like_you_are/fictional/tv_shows/qna.yaml | |
| 42. ompositional_skills/writing/freeform/riddles/qna.yaml | |
| 43. ompositional_skills/extraction/email/markdown/qna.yaml | |
| 44. ompositional_skills/STEM/math/distance_conversion/qna.yaml | |
| 45. ompositional_skills/extraction/abstractive/key_points/qna.yaml | |
| 46. nowledge/textbook/history/ibm_history/qna.yaml | |
| 47. ompositional_skills/writing/freeform/brainstorming/idea_generation/qna.yaml | |
| 48. ompositional_skills/STEM/science/units_conversion/distance_conversion/qna.yaml | |
| 49. ompositional_skills/extraction/invoice/reasoning/qna.yaml | |
| 50. ompositional_skills/writing/freeform/poetry/epic_poetry/qna.yaml | |
| 51. ompositional_skills/roleplay/explain_like_i_am/primary_schooler/qna.yaml | |
| 52. ompositional_skills/writing/freeform/poetry/freeverse/qna.yaml | |
| 53. ompositional_skills/extraction/invoice/plain_text/qna.yaml | |
| 54. ompositional_skills/writing/freeform/social_media/facebook/qna.yaml | |
| 55. ompositional_skills/extraction/annual_report/plain_text/qna.yaml | |
| 56. ompositional_skills/extraction/fda_filing/plain_text/qna.yaml | |
| 57. ompositional_skills/extraction/inference/quantitative/asciidoc/tables/qna.yaml | |
| 58. ompositional_skills/extraction/technical_paper/abstract/reasoning/qna.yaml | |
| 59. ompositional_skills/extraction/annual_report/bullet_points/qna.yaml | |
| 60. ompositional_skills/extraction/receipt/csv/qna.yaml | |
| 61. ompositional_skills/extraction/abstractive/main_takeaway/qna.yaml | |
| 62. ompositional_skills/extraction/commercial_lease_agreement/bullet_points/qna.yaml | |
| 63. oundational_skills/reasoning/linguistics_reasoning/logical_sequence_of_words/qna.yaml | |
| 64. ompositional_skills/STEM/math/pattern_recognition/qna.yaml | |
| 65. ompositional_skills/writing/freeform/brainstorming/support_claim/qna.yaml | |
| 66. ompositional_skills/writing/grounded/meeting_insights/corporate_email/qna.yaml | |
| 67. ompositional_skills/general/tables/empty/qna.yaml | |
| 68. ompositional_skills/extraction/technical_paper/tables/csv/qna.yaml | |
| 69. ompositional_skills/extraction/inference/quantitative/table_analaysis/qna.yaml | |
| 70. ompositional_skills/STEM/math/time_series/qna.yaml | |
| 71. ompositional_skills/writing/grounded/summarization/wiki_insights/high_level_outline/qna.yaml | |
| 72. ompositional_skills/extraction/email/plain_text/qna.yaml | |
| 73. ompositional_skills/STEM/math/arithmetic_reasoning/qna.yaml | |
| 74. ompositional_skills/extraction/technical_paper/abstract/markdown/qna.yaml | |
| 75. ompositional_skills/writing/freeform/technical/product_description/qna.yaml | |
| 76. ompositional_skills/extraction/commercial_lease_agreement/reasoning/qna.yaml | |
| 77. ompositional_skills/STEM/math/area/qna.yaml | |
| 78. ompositional_skills/writing/freeform/legal/contracts/qna.yaml | |
| 79. ompositional_skills/extraction/commercial_lease_agreement/plain_text/qna.yaml | |
| 80. oundational_skills/reasoning/common_sense_reasoning/qna.yaml | |
| 81. ompositional_skills/extraction/commercial_lease_agreement/csv/qna.yaml | |
| 82. ompositional_skills/writing/grounded/summarization/wiki_insights/one_line/qna.yaml | |
| 83. ompositional_skills/extraction/annual_report/markdown/qna.yaml | |
| 84. ompositional_skills/linguistics/reversing_string/qna.yaml | |
| 85. ompositional_skills/extraction/technical_paper/equations/reasoning/qna.yaml | |
| 86. ompositional_skills/writing/freeform/social_media/twitter/qna.yaml | |
| 87. ompositional_skills/writing/freeform/technical/proposal/qna.yaml | |
| 88. ompositional_skills/writing/freeform/prose/articles/qna.yaml | |
| 89. ompositional_skills/writing/freeform/poetry/ballad/qna.yaml | |
| 90. ompositional_skills/writing/freeform/jokes/puns/general/qna.yaml | |
| 91. ompositional_skills/writing/freeform/poetry/limerick/qna.yaml | |
| 92. ompositional_skills/extraction/technical_paper/tables/reasoning/qna.yaml | |
| 93. ompositional_skills/extraction/receipt/markdown/qna.yaml | |
| 94. ompositional_skills/roleplay/explain_like_you_are/fictional/video_games/qna.yaml | |
| 95. ompositional_skills/extraction/services_agreement/bullet_points/qna.yaml | |
| 96. ompositional_skills/extraction/fda_filing/reasoning/qna.yaml | |
| 97. ompositional_skills/extraction/abstractive/abstract/qna.yaml | |
| 98. ompositional_skills/writing/freeform/technical/report/qna.yaml | |
| 99. ompositional_skills/writing/freeform/prose/screenplay/qna.yaml | |
| 100. ompositional_skills/extraction/fda_filing/markdown/qna.yaml | |
| 101. ompositional_skills/writing/grounded/meeting_insights/action_items/qna.yaml | |
| 102. ompositional_skills/general/tables/editing/add_remove/qna.yaml | |
| 103. ompositional_skills/linguistics/summarization/ignore_pii/qna.yaml | |
| 104. ompositional_skills/STEM/math/mensurational/qna.yaml | |
| 105. ompositional_skills/linguistics/classification/agent_classification/qna.yaml | |
| 106. ompositional_skills/roleplay/explain_like_you_are/non_fictional/historical_figures/qna.yaml | |
| 107. ompositional_skills/writing/grounded/editing/spelling/qna.yaml | |
| 108. ompositional_skills/extraction/abstractive/outline/qna.yaml | |
| 109. ompositional_skills/extraction/information/named_entities/dates_and_events/qna.yaml | |
| 110. ompositional_skills/writing/freeform/brainstorming/refute_claim/qna.yaml | |
| 111. ompositional_skills/writing/grounded/summarization/wiki_insights/concise/qna.yaml | |
| 112. ompositional_skills/extraction/receipt/reasoning/qna.yaml | |
| 113. ompositional_skills/extraction/information/named_entities/places/qna.yaml | |
| 114. ompositional_skills/linguistics/summarization/list_of_sentences/qna.yaml | |
| 115. ompositional_skills/writing/freeform/social_media/linkedin/qna.yaml | |
| 116. ompositional_skills/writing/freeform/poetry/haiku/qna.yaml | |
| 117. ompositional_skills/writing/freeform/debate/qna.yaml | |
| 118. ompositional_skills/linguistics/jumbled_sentences/qna.yaml | |
| 119. ompositional_skills/writing/freeform/poetry/ode/qna.yaml | |
| 120. ompositional_skills/writing/freeform/technical/guide/qna.yaml | |
| 121. ompositional_skills/writing/grounded/meeting_insights/minutes_of_meeting/qna.yaml | |
| 122. ompositional_skills/extraction/inference/qualitative/sentiment/qna.yaml | |
| 123. ompositional_skills/writing/grounded/summarization/wiki_insights/detailed/qna.yaml | |
| 124. ompositional_skills/extraction/abstractive/title/qna.yaml | |
| 125. ompositional_skills/writing/freeform/technical/user_manual/qna.yaml | |
| 126. ompositional_skills/writing/grounded/meeting_insights/executive_summaries/qna.yaml | |
| 127. ompositional_skills/writing/freeform/technical/specification/qna.yaml | |
| ### ERROR RATE: | |
| 0.11 | |
| [root@tyler-rhel-newimage instructlab]# |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment