Created
January 26, 2024 19:05
-
-
Save OguzPastirmaci/c4b6584b5a92b80ed56dc2ad81738403 to your computer and use it in GitHub Desktop.
nccl.log
This file has been truncated, but you can view the full file.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Warning: Permanently added 'nccl-alltoall-job-oguz-mpiworker-0.nccl-alltoall-job-oguz' (ED25519) to the list of known hosts. | |
| Warning: Permanently added 'nccl-alltoall-job-oguz-mpiworker-1.nccl-alltoall-job-oguz' (ED25519) to the list of known hosts. | |
| Warning: Permanently added 'nccl-alltoall-job-oguz-mpiworker-2.nccl-alltoall-job-oguz' (ED25519) to the list of known hosts. | |
| Warning: Permanently added 'nccl-alltoall-job-oguz-mpiworker-4.nccl-alltoall-job-oguz' (ED25519) to the list of known hosts. | |
| Warning: Permanently added 'nccl-alltoall-job-oguz-mpiworker-3.nccl-alltoall-job-oguz' (ED25519) to the list of known hosts. | |
| Warning: Permanently added 'nccl-alltoall-job-oguz-mpiworker-5.nccl-alltoall-job-oguz' (ED25519) to the list of known hosts. | |
| Warning: Permanently added 'nccl-alltoall-job-oguz-mpiworker-6.nccl-alltoall-job-oguz' (ED25519) to the list of known hosts. | |
| Warning: Permanently added 'nccl-alltoall-job-oguz-mpiworker-12.nccl-alltoall-job-oguz' (ED25519) to the list of known hosts. | |
| Warning: Permanently added 'nccl-alltoall-job-oguz-mpiworker-13.nccl-alltoall-job-oguz' (ED25519) to the list of known hosts. | |
| Warning: Permanently added 'nccl-alltoall-job-oguz-mpiworker-8.nccl-alltoall-job-oguz' (ED25519) to the list of known hosts. | |
| Warning: Permanently added 'nccl-alltoall-job-oguz-mpiworker-7.nccl-alltoall-job-oguz' (ED25519) to the list of known hosts. | |
| Warning: Permanently added 'nccl-alltoall-job-oguz-mpiworker-9.nccl-alltoall-job-oguz' (ED25519) to the list of known hosts. | |
| Warning: Permanently added 'nccl-alltoall-job-oguz-mpiworker-10.nccl-alltoall-job-oguz' (ED25519) to the list of known hosts. | |
| Warning: Permanently added 'nccl-alltoall-job-oguz-mpiworker-11.nccl-alltoall-job-oguz' (ED25519) to the list of known hosts. | |
| Warning: Permanently added 'nccl-alltoall-job-oguz-mpiworker-26.nccl-alltoall-job-oguz' (ED25519) to the list of known hosts. | |
| Warning: Permanently added 'nccl-alltoall-job-oguz-mpiworker-27.nccl-alltoall-job-oguz' (ED25519) to the list of known hosts. | |
| Warning: Permanently added 'nccl-alltoall-job-oguz-mpiworker-17.nccl-alltoall-job-oguz' (ED25519) to the list of known hosts. | |
| Warning: Permanently added 'nccl-alltoall-job-oguz-mpiworker-30.nccl-alltoall-job-oguz' (ED25519) to the list of known hosts. | |
| Warning: Permanently added 'nccl-alltoall-job-oguz-mpiworker-31.nccl-alltoall-job-oguz' (ED25519) to the list of known hosts. | |
| Warning: Permanently added 'nccl-alltoall-job-oguz-mpiworker-28.nccl-alltoall-job-oguz' (ED25519) to the list of known hosts. | |
| Warning: Permanently added 'nccl-alltoall-job-oguz-mpiworker-16.nccl-alltoall-job-oguz' (ED25519) to the list of known hosts. | |
| Warning: Permanently added 'nccl-alltoall-job-oguz-mpiworker-29.nccl-alltoall-job-oguz' (ED25519) to the list of known hosts. | |
| Warning: Permanently added 'nccl-alltoall-job-oguz-mpiworker-15.nccl-alltoall-job-oguz' (ED25519) to the list of known hosts. | |
| Warning: Permanently added 'nccl-alltoall-job-oguz-mpiworker-20.nccl-alltoall-job-oguz' (ED25519) to the list of known hosts. | |
| Warning: Permanently added 'nccl-alltoall-job-oguz-mpiworker-18.nccl-alltoall-job-oguz' (ED25519) to the list of known hosts. | |
| Warning: Permanently added 'nccl-alltoall-job-oguz-mpiworker-19.nccl-alltoall-job-oguz' (ED25519) to the list of known hosts. | |
| Warning: Permanently added 'nccl-alltoall-job-oguz-mpiworker-14.nccl-alltoall-job-oguz' (ED25519) to the list of known hosts. | |
| Warning: Permanently added 'nccl-alltoall-job-oguz-mpiworker-21.nccl-alltoall-job-oguz' (ED25519) to the list of known hosts. | |
| Warning: Permanently added 'nccl-alltoall-job-oguz-mpiworker-25.nccl-alltoall-job-oguz' (ED25519) to the list of known hosts. | |
| Warning: Permanently added 'nccl-alltoall-job-oguz-mpiworker-22.nccl-alltoall-job-oguz' (ED25519) to the list of known hosts. | |
| Warning: Permanently added 'nccl-alltoall-job-oguz-mpiworker-24.nccl-alltoall-job-oguz' (ED25519) to the list of known hosts. | |
| Warning: Permanently added 'nccl-alltoall-job-oguz-mpiworker-23.nccl-alltoall-job-oguz' (ED25519) to the list of known hosts. | |
| # nThread 1 nGpus 1 minBytes 8 maxBytes 4294967296 step: 2(factor) warmup iters: 5 iters: 20 agg iters: 1 validation: 1 graph: 0 | |
| # | |
| # Using devices | |
| # Rank 0 Group 0 Pid 46 on nccl-alltoall-job-oguz-mpiworker-0 device 0 [0x0f] NVIDIA H100 80GB HBM3 | |
| # Rank 1 Group 0 Pid 47 on nccl-alltoall-job-oguz-mpiworker-0 device 1 [0x2d] NVIDIA H100 80GB HBM3 | |
| # Rank 2 Group 0 Pid 48 on nccl-alltoall-job-oguz-mpiworker-0 device 2 [0x44] NVIDIA H100 80GB HBM3 | |
| # Rank 3 Group 0 Pid 49 on nccl-alltoall-job-oguz-mpiworker-0 device 3 [0x5b] NVIDIA H100 80GB HBM3 | |
| # Rank 4 Group 0 Pid 50 on nccl-alltoall-job-oguz-mpiworker-0 device 4 [0x89] NVIDIA H100 80GB HBM3 | |
| # Rank 5 Group 0 Pid 51 on nccl-alltoall-job-oguz-mpiworker-0 device 5 [0xa8] NVIDIA H100 80GB HBM3 | |
| # Rank 6 Group 0 Pid 52 on nccl-alltoall-job-oguz-mpiworker-0 device 6 [0xc0] NVIDIA H100 80GB HBM3 | |
| # Rank 7 Group 0 Pid 53 on nccl-alltoall-job-oguz-mpiworker-0 device 7 [0xd8] NVIDIA H100 80GB HBM3 | |
| # Rank 8 Group 0 Pid 46 on nccl-alltoall-job-oguz-mpiworker-1 device 0 [0x0f] NVIDIA H100 80GB HBM3 | |
| # Rank 9 Group 0 Pid 47 on nccl-alltoall-job-oguz-mpiworker-1 device 1 [0x2d] NVIDIA H100 80GB HBM3 | |
| # Rank 10 Group 0 Pid 48 on nccl-alltoall-job-oguz-mpiworker-1 device 2 [0x44] NVIDIA H100 80GB HBM3 | |
| # Rank 11 Group 0 Pid 49 on nccl-alltoall-job-oguz-mpiworker-1 device 3 [0x5b] NVIDIA H100 80GB HBM3 | |
| # Rank 12 Group 0 Pid 50 on nccl-alltoall-job-oguz-mpiworker-1 device 4 [0x89] NVIDIA H100 80GB HBM3 | |
| # Rank 13 Group 0 Pid 51 on nccl-alltoall-job-oguz-mpiworker-1 device 5 [0xa8] NVIDIA H100 80GB HBM3 | |
| # Rank 14 Group 0 Pid 52 on nccl-alltoall-job-oguz-mpiworker-1 device 6 [0xc0] NVIDIA H100 80GB HBM3 | |
| # Rank 15 Group 0 Pid 54 on nccl-alltoall-job-oguz-mpiworker-1 device 7 [0xd8] NVIDIA H100 80GB HBM3 | |
| # Rank 16 Group 0 Pid 46 on nccl-alltoall-job-oguz-mpiworker-2 device 0 [0x0f] NVIDIA H100 80GB HBM3 | |
| # Rank 17 Group 0 Pid 47 on nccl-alltoall-job-oguz-mpiworker-2 device 1 [0x2d] NVIDIA H100 80GB HBM3 | |
| # Rank 18 Group 0 Pid 48 on nccl-alltoall-job-oguz-mpiworker-2 device 2 [0x44] NVIDIA H100 80GB HBM3 | |
| # Rank 19 Group 0 Pid 49 on nccl-alltoall-job-oguz-mpiworker-2 device 3 [0x5b] NVIDIA H100 80GB HBM3 | |
| # Rank 20 Group 0 Pid 50 on nccl-alltoall-job-oguz-mpiworker-2 device 4 [0x89] NVIDIA H100 80GB HBM3 | |
| # Rank 21 Group 0 Pid 51 on nccl-alltoall-job-oguz-mpiworker-2 device 5 [0xa8] NVIDIA H100 80GB HBM3 | |
| # Rank 22 Group 0 Pid 52 on nccl-alltoall-job-oguz-mpiworker-2 device 6 [0xc0] NVIDIA H100 80GB HBM3 | |
| # Rank 23 Group 0 Pid 53 on nccl-alltoall-job-oguz-mpiworker-2 device 7 [0xd8] NVIDIA H100 80GB HBM3 | |
| # Rank 24 Group 0 Pid 46 on nccl-alltoall-job-oguz-mpiworker-3 device 0 [0x0f] NVIDIA H100 80GB HBM3 | |
| # Rank 25 Group 0 Pid 47 on nccl-alltoall-job-oguz-mpiworker-3 device 1 [0x2d] NVIDIA H100 80GB HBM3 | |
| # Rank 26 Group 0 Pid 48 on nccl-alltoall-job-oguz-mpiworker-3 device 2 [0x44] NVIDIA H100 80GB HBM3 | |
| # Rank 27 Group 0 Pid 49 on nccl-alltoall-job-oguz-mpiworker-3 device 3 [0x5b] NVIDIA H100 80GB HBM3 | |
| # Rank 28 Group 0 Pid 50 on nccl-alltoall-job-oguz-mpiworker-3 device 4 [0x89] NVIDIA H100 80GB HBM3 | |
| # Rank 29 Group 0 Pid 51 on nccl-alltoall-job-oguz-mpiworker-3 device 5 [0xa8] NVIDIA H100 80GB HBM3 | |
| # Rank 30 Group 0 Pid 52 on nccl-alltoall-job-oguz-mpiworker-3 device 6 [0xc0] NVIDIA H100 80GB HBM3 | |
| # Rank 31 Group 0 Pid 53 on nccl-alltoall-job-oguz-mpiworker-3 device 7 [0xd8] NVIDIA H100 80GB HBM3 | |
| # Rank 32 Group 0 Pid 46 on nccl-alltoall-job-oguz-mpiworker-4 device 0 [0x0f] NVIDIA H100 80GB HBM3 | |
| # Rank 33 Group 0 Pid 47 on nccl-alltoall-job-oguz-mpiworker-4 device 1 [0x2d] NVIDIA H100 80GB HBM3 | |
| # Rank 34 Group 0 Pid 48 on nccl-alltoall-job-oguz-mpiworker-4 device 2 [0x44] NVIDIA H100 80GB HBM3 | |
| # Rank 35 Group 0 Pid 49 on nccl-alltoall-job-oguz-mpiworker-4 device 3 [0x5b] NVIDIA H100 80GB HBM3 | |
| # Rank 36 Group 0 Pid 50 on nccl-alltoall-job-oguz-mpiworker-4 device 4 [0x89] NVIDIA H100 80GB HBM3 | |
| # Rank 37 Group 0 Pid 51 on nccl-alltoall-job-oguz-mpiworker-4 device 5 [0xa8] NVIDIA H100 80GB HBM3 | |
| # Rank 38 Group 0 Pid 52 on nccl-alltoall-job-oguz-mpiworker-4 device 6 [0xc0] NVIDIA H100 80GB HBM3 | |
| # Rank 39 Group 0 Pid 53 on nccl-alltoall-job-oguz-mpiworker-4 device 7 [0xd8] NVIDIA H100 80GB HBM3 | |
| # Rank 40 Group 0 Pid 46 on nccl-alltoall-job-oguz-mpiworker-5 device 0 [0x0f] NVIDIA H100 80GB HBM3 | |
| # Rank 41 Group 0 Pid 47 on nccl-alltoall-job-oguz-mpiworker-5 device 1 [0x2d] NVIDIA H100 80GB HBM3 | |
| # Rank 42 Group 0 Pid 48 on nccl-alltoall-job-oguz-mpiworker-5 device 2 [0x44] NVIDIA H100 80GB HBM3 | |
| # Rank 43 Group 0 Pid 49 on nccl-alltoall-job-oguz-mpiworker-5 device 3 [0x5b] NVIDIA H100 80GB HBM3 | |
| # Rank 44 Group 0 Pid 50 on nccl-alltoall-job-oguz-mpiworker-5 device 4 [0x89] NVIDIA H100 80GB HBM3 | |
| # Rank 45 Group 0 Pid 51 on nccl-alltoall-job-oguz-mpiworker-5 device 5 [0xa8] NVIDIA H100 80GB HBM3 | |
| # Rank 46 Group 0 Pid 52 on nccl-alltoall-job-oguz-mpiworker-5 device 6 [0xc0] NVIDIA H100 80GB HBM3 | |
| # Rank 47 Group 0 Pid 53 on nccl-alltoall-job-oguz-mpiworker-5 device 7 [0xd8] NVIDIA H100 80GB HBM3 | |
| # Rank 48 Group 0 Pid 46 on nccl-alltoall-job-oguz-mpiworker-6 device 0 [0x0f] NVIDIA H100 80GB HBM3 | |
| # Rank 49 Group 0 Pid 47 on nccl-alltoall-job-oguz-mpiworker-6 device 1 [0x2d] NVIDIA H100 80GB HBM3 | |
| # Rank 50 Group 0 Pid 48 on nccl-alltoall-job-oguz-mpiworker-6 device 2 [0x44] NVIDIA H100 80GB HBM3 | |
| # Rank 51 Group 0 Pid 49 on nccl-alltoall-job-oguz-mpiworker-6 device 3 [0x5b] NVIDIA H100 80GB HBM3 | |
| # Rank 52 Group 0 Pid 50 on nccl-alltoall-job-oguz-mpiworker-6 device 4 [0x89] NVIDIA H100 80GB HBM3 | |
| # Rank 53 Group 0 Pid 51 on nccl-alltoall-job-oguz-mpiworker-6 device 5 [0xa8] NVIDIA H100 80GB HBM3 | |
| # Rank 54 Group 0 Pid 52 on nccl-alltoall-job-oguz-mpiworker-6 device 6 [0xc0] NVIDIA H100 80GB HBM3 | |
| # Rank 55 Group 0 Pid 53 on nccl-alltoall-job-oguz-mpiworker-6 device 7 [0xd8] NVIDIA H100 80GB HBM3 | |
| # Rank 56 Group 0 Pid 46 on nccl-alltoall-job-oguz-mpiworker-7 device 0 [0x0f] NVIDIA H100 80GB HBM3 | |
| # Rank 57 Group 0 Pid 47 on nccl-alltoall-job-oguz-mpiworker-7 device 1 [0x2d] NVIDIA H100 80GB HBM3 | |
| # Rank 58 Group 0 Pid 48 on nccl-alltoall-job-oguz-mpiworker-7 device 2 [0x44] NVIDIA H100 80GB HBM3 | |
| # Rank 59 Group 0 Pid 49 on nccl-alltoall-job-oguz-mpiworker-7 device 3 [0x5b] NVIDIA H100 80GB HBM3 | |
| # Rank 60 Group 0 Pid 50 on nccl-alltoall-job-oguz-mpiworker-7 device 4 [0x89] NVIDIA H100 80GB HBM3 | |
| # Rank 61 Group 0 Pid 51 on nccl-alltoall-job-oguz-mpiworker-7 device 5 [0xa8] NVIDIA H100 80GB HBM3 | |
| # Rank 62 Group 0 Pid 52 on nccl-alltoall-job-oguz-mpiworker-7 device 6 [0xc0] NVIDIA H100 80GB HBM3 | |
| # Rank 63 Group 0 Pid 54 on nccl-alltoall-job-oguz-mpiworker-7 device 7 [0xd8] NVIDIA H100 80GB HBM3 | |
| # Rank 64 Group 0 Pid 46 on nccl-alltoall-job-oguz-mpiworker-8 device 0 [0x0f] NVIDIA H100 80GB HBM3 | |
| # Rank 65 Group 0 Pid 47 on nccl-alltoall-job-oguz-mpiworker-8 device 1 [0x2d] NVIDIA H100 80GB HBM3 | |
| # Rank 66 Group 0 Pid 48 on nccl-alltoall-job-oguz-mpiworker-8 device 2 [0x44] NVIDIA H100 80GB HBM3 | |
| # Rank 67 Group 0 Pid 49 on nccl-alltoall-job-oguz-mpiworker-8 device 3 [0x5b] NVIDIA H100 80GB HBM3 | |
| # Rank 68 Group 0 Pid 50 on nccl-alltoall-job-oguz-mpiworker-8 device 4 [0x89] NVIDIA H100 80GB HBM3 | |
| # Rank 69 Group 0 Pid 51 on nccl-alltoall-job-oguz-mpiworker-8 device 5 [0xa8] NVIDIA H100 80GB HBM3 | |
| # Rank 70 Group 0 Pid 52 on nccl-alltoall-job-oguz-mpiworker-8 device 6 [0xc0] NVIDIA H100 80GB HBM3 | |
| # Rank 71 Group 0 Pid 53 on nccl-alltoall-job-oguz-mpiworker-8 device 7 [0xd8] NVIDIA H100 80GB HBM3 | |
| # Rank 72 Group 0 Pid 46 on nccl-alltoall-job-oguz-mpiworker-9 device 0 [0x0f] NVIDIA H100 80GB HBM3 | |
| # Rank 73 Group 0 Pid 47 on nccl-alltoall-job-oguz-mpiworker-9 device 1 [0x2d] NVIDIA H100 80GB HBM3 | |
| # Rank 74 Group 0 Pid 48 on nccl-alltoall-job-oguz-mpiworker-9 device 2 [0x44] NVIDIA H100 80GB HBM3 | |
| # Rank 75 Group 0 Pid 49 on nccl-alltoall-job-oguz-mpiworker-9 device 3 [0x5b] NVIDIA H100 80GB HBM3 | |
| # Rank 76 Group 0 Pid 50 on nccl-alltoall-job-oguz-mpiworker-9 device 4 [0x89] NVIDIA H100 80GB HBM3 | |
| # Rank 77 Group 0 Pid 51 on nccl-alltoall-job-oguz-mpiworker-9 device 5 [0xa8] NVIDIA H100 80GB HBM3 | |
| # Rank 78 Group 0 Pid 52 on nccl-alltoall-job-oguz-mpiworker-9 device 6 [0xc0] NVIDIA H100 80GB HBM3 | |
| # Rank 79 Group 0 Pid 53 on nccl-alltoall-job-oguz-mpiworker-9 device 7 [0xd8] NVIDIA H100 80GB HBM3 | |
| # Rank 80 Group 0 Pid 46 on nccl-alltoall-job-oguz-mpiworker-10 device 0 [0x0f] NVIDIA H100 80GB HBM3 | |
| # Rank 81 Group 0 Pid 47 on nccl-alltoall-job-oguz-mpiworker-10 device 1 [0x2d] NVIDIA H100 80GB HBM3 | |
| # Rank 82 Group 0 Pid 48 on nccl-alltoall-job-oguz-mpiworker-10 device 2 [0x44] NVIDIA H100 80GB HBM3 | |
| # Rank 83 Group 0 Pid 49 on nccl-alltoall-job-oguz-mpiworker-10 device 3 [0x5b] NVIDIA H100 80GB HBM3 | |
| # Rank 84 Group 0 Pid 50 on nccl-alltoall-job-oguz-mpiworker-10 device 4 [0x89] NVIDIA H100 80GB HBM3 | |
| # Rank 85 Group 0 Pid 51 on nccl-alltoall-job-oguz-mpiworker-10 device 5 [0xa8] NVIDIA H100 80GB HBM3 | |
| # Rank 86 Group 0 Pid 52 on nccl-alltoall-job-oguz-mpiworker-10 device 6 [0xc0] NVIDIA H100 80GB HBM3 | |
| # Rank 87 Group 0 Pid 53 on nccl-alltoall-job-oguz-mpiworker-10 device 7 [0xd8] NVIDIA H100 80GB HBM3 | |
| # Rank 88 Group 0 Pid 46 on nccl-alltoall-job-oguz-mpiworker-11 device 0 [0x0f] NVIDIA H100 80GB HBM3 | |
| # Rank 89 Group 0 Pid 47 on nccl-alltoall-job-oguz-mpiworker-11 device 1 [0x2d] NVIDIA H100 80GB HBM3 | |
| # Rank 90 Group 0 Pid 48 on nccl-alltoall-job-oguz-mpiworker-11 device 2 [0x44] NVIDIA H100 80GB HBM3 | |
| # Rank 91 Group 0 Pid 49 on nccl-alltoall-job-oguz-mpiworker-11 device 3 [0x5b] NVIDIA H100 80GB HBM3 | |
| # Rank 92 Group 0 Pid 50 on nccl-alltoall-job-oguz-mpiworker-11 device 4 [0x89] NVIDIA H100 80GB HBM3 | |
| # Rank 93 Group 0 Pid 51 on nccl-alltoall-job-oguz-mpiworker-11 device 5 [0xa8] NVIDIA H100 80GB HBM3 | |
| # Rank 94 Group 0 Pid 52 on nccl-alltoall-job-oguz-mpiworker-11 device 6 [0xc0] NVIDIA H100 80GB HBM3 | |
| # Rank 95 Group 0 Pid 53 on nccl-alltoall-job-oguz-mpiworker-11 device 7 [0xd8] NVIDIA H100 80GB HBM3 | |
| # Rank 96 Group 0 Pid 46 on nccl-alltoall-job-oguz-mpiworker-12 device 0 [0x0f] NVIDIA H100 80GB HBM3 | |
| # Rank 97 Group 0 Pid 47 on nccl-alltoall-job-oguz-mpiworker-12 device 1 [0x2d] NVIDIA H100 80GB HBM3 | |
| # Rank 98 Group 0 Pid 48 on nccl-alltoall-job-oguz-mpiworker-12 device 2 [0x44] NVIDIA H100 80GB HBM3 | |
| # Rank 99 Group 0 Pid 49 on nccl-alltoall-job-oguz-mpiworker-12 device 3 [0x5b] NVIDIA H100 80GB HBM3 | |
| # Rank 100 Group 0 Pid 50 on nccl-alltoall-job-oguz-mpiworker-12 device 4 [0x89] NVIDIA H100 80GB HBM3 | |
| # Rank 101 Group 0 Pid 51 on nccl-alltoall-job-oguz-mpiworker-12 device 5 [0xa8] NVIDIA H100 80GB HBM3 | |
| # Rank 102 Group 0 Pid 52 on nccl-alltoall-job-oguz-mpiworker-12 device 6 [0xc0] NVIDIA H100 80GB HBM3 | |
| # Rank 103 Group 0 Pid 54 on nccl-alltoall-job-oguz-mpiworker-12 device 7 [0xd8] NVIDIA H100 80GB HBM3 | |
| # Rank 104 Group 0 Pid 46 on nccl-alltoall-job-oguz-mpiworker-13 device 0 [0x0f] NVIDIA H100 80GB HBM3 | |
| # Rank 105 Group 0 Pid 47 on nccl-alltoall-job-oguz-mpiworker-13 device 1 [0x2d] NVIDIA H100 80GB HBM3 | |
| # Rank 106 Group 0 Pid 48 on nccl-alltoall-job-oguz-mpiworker-13 device 2 [0x44] NVIDIA H100 80GB HBM3 | |
| # Rank 107 Group 0 Pid 49 on nccl-alltoall-job-oguz-mpiworker-13 device 3 [0x5b] NVIDIA H100 80GB HBM3 | |
| # Rank 108 Group 0 Pid 50 on nccl-alltoall-job-oguz-mpiworker-13 device 4 [0x89] NVIDIA H100 80GB HBM3 | |
| # Rank 109 Group 0 Pid 51 on nccl-alltoall-job-oguz-mpiworker-13 device 5 [0xa8] NVIDIA H100 80GB HBM3 | |
| # Rank 110 Group 0 Pid 52 on nccl-alltoall-job-oguz-mpiworker-13 device 6 [0xc0] NVIDIA H100 80GB HBM3 | |
| # Rank 111 Group 0 Pid 53 on nccl-alltoall-job-oguz-mpiworker-13 device 7 [0xd8] NVIDIA H100 80GB HBM3 | |
| # Rank 112 Group 0 Pid 46 on nccl-alltoall-job-oguz-mpiworker-14 device 0 [0x0f] NVIDIA H100 80GB HBM3 | |
| # Rank 113 Group 0 Pid 47 on nccl-alltoall-job-oguz-mpiworker-14 device 1 [0x2d] NVIDIA H100 80GB HBM3 | |
| # Rank 114 Group 0 Pid 48 on nccl-alltoall-job-oguz-mpiworker-14 device 2 [0x44] NVIDIA H100 80GB HBM3 | |
| # Rank 115 Group 0 Pid 49 on nccl-alltoall-job-oguz-mpiworker-14 device 3 [0x5b] NVIDIA H100 80GB HBM3 | |
| # Rank 116 Group 0 Pid 50 on nccl-alltoall-job-oguz-mpiworker-14 device 4 [0x89] NVIDIA H100 80GB HBM3 | |
| # Rank 117 Group 0 Pid 51 on nccl-alltoall-job-oguz-mpiworker-14 device 5 [0xa8] NVIDIA H100 80GB HBM3 | |
| # Rank 118 Group 0 Pid 52 on nccl-alltoall-job-oguz-mpiworker-14 device 6 [0xc0] NVIDIA H100 80GB HBM3 | |
| # Rank 119 Group 0 Pid 53 on nccl-alltoall-job-oguz-mpiworker-14 device 7 [0xd8] NVIDIA H100 80GB HBM3 | |
| # Rank 120 Group 0 Pid 46 on nccl-alltoall-job-oguz-mpiworker-15 device 0 [0x0f] NVIDIA H100 80GB HBM3 | |
| # Rank 121 Group 0 Pid 47 on nccl-alltoall-job-oguz-mpiworker-15 device 1 [0x2d] NVIDIA H100 80GB HBM3 | |
| # Rank 122 Group 0 Pid 48 on nccl-alltoall-job-oguz-mpiworker-15 device 2 [0x44] NVIDIA H100 80GB HBM3 | |
| # Rank 123 Group 0 Pid 49 on nccl-alltoall-job-oguz-mpiworker-15 device 3 [0x5b] NVIDIA H100 80GB HBM3 | |
| # Rank 124 Group 0 Pid 50 on nccl-alltoall-job-oguz-mpiworker-15 device 4 [0x89] NVIDIA H100 80GB HBM3 | |
| # Rank 125 Group 0 Pid 51 on nccl-alltoall-job-oguz-mpiworker-15 device 5 [0xa8] NVIDIA H100 80GB HBM3 | |
| # Rank 126 Group 0 Pid 52 on nccl-alltoall-job-oguz-mpiworker-15 device 6 [0xc0] NVIDIA H100 80GB HBM3 | |
| # Rank 127 Group 0 Pid 53 on nccl-alltoall-job-oguz-mpiworker-15 device 7 [0xd8] NVIDIA H100 80GB HBM3 | |
| # Rank 128 Group 0 Pid 46 on nccl-alltoall-job-oguz-mpiworker-16 device 0 [0x0f] NVIDIA H100 80GB HBM3 | |
| # Rank 129 Group 0 Pid 47 on nccl-alltoall-job-oguz-mpiworker-16 device 1 [0x2d] NVIDIA H100 80GB HBM3 | |
| # Rank 130 Group 0 Pid 48 on nccl-alltoall-job-oguz-mpiworker-16 device 2 [0x44] NVIDIA H100 80GB HBM3 | |
| # Rank 131 Group 0 Pid 49 on nccl-alltoall-job-oguz-mpiworker-16 device 3 [0x5b] NVIDIA H100 80GB HBM3 | |
| # Rank 132 Group 0 Pid 50 on nccl-alltoall-job-oguz-mpiworker-16 device 4 [0x89] NVIDIA H100 80GB HBM3 | |
| # Rank 133 Group 0 Pid 51 on nccl-alltoall-job-oguz-mpiworker-16 device 5 [0xa8] NVIDIA H100 80GB HBM3 | |
| # Rank 134 Group 0 Pid 52 on nccl-alltoall-job-oguz-mpiworker-16 device 6 [0xc0] NVIDIA H100 80GB HBM3 | |
| # Rank 135 Group 0 Pid 53 on nccl-alltoall-job-oguz-mpiworker-16 device 7 [0xd8] NVIDIA H100 80GB HBM3 | |
| # Rank 136 Group 0 Pid 46 on nccl-alltoall-job-oguz-mpiworker-17 device 0 [0x0f] NVIDIA H100 80GB HBM3 | |
| # Rank 137 Group 0 Pid 47 on nccl-alltoall-job-oguz-mpiworker-17 device 1 [0x2d] NVIDIA H100 80GB HBM3 | |
| # Rank 138 Group 0 Pid 48 on nccl-alltoall-job-oguz-mpiworker-17 device 2 [0x44] NVIDIA H100 80GB HBM3 | |
| # Rank 139 Group 0 Pid 49 on nccl-alltoall-job-oguz-mpiworker-17 device 3 [0x5b] NVIDIA H100 80GB HBM3 | |
| # Rank 140 Group 0 Pid 50 on nccl-alltoall-job-oguz-mpiworker-17 device 4 [0x89] NVIDIA H100 80GB HBM3 | |
| # Rank 141 Group 0 Pid 51 on nccl-alltoall-job-oguz-mpiworker-17 device 5 [0xa8] NVIDIA H100 80GB HBM3 | |
| # Rank 142 Group 0 Pid 52 on nccl-alltoall-job-oguz-mpiworker-17 device 6 [0xc0] NVIDIA H100 80GB HBM3 | |
| # Rank 143 Group 0 Pid 53 on nccl-alltoall-job-oguz-mpiworker-17 device 7 [0xd8] NVIDIA H100 80GB HBM3 | |
| # Rank 144 Group 0 Pid 46 on nccl-alltoall-job-oguz-mpiworker-18 device 0 [0x0f] NVIDIA H100 80GB HBM3 | |
| # Rank 145 Group 0 Pid 47 on nccl-alltoall-job-oguz-mpiworker-18 device 1 [0x2d] NVIDIA H100 80GB HBM3 | |
| # Rank 146 Group 0 Pid 48 on nccl-alltoall-job-oguz-mpiworker-18 device 2 [0x44] NVIDIA H100 80GB HBM3 | |
| # Rank 147 Group 0 Pid 49 on nccl-alltoall-job-oguz-mpiworker-18 device 3 [0x5b] NVIDIA H100 80GB HBM3 | |
| # Rank 148 Group 0 Pid 50 on nccl-alltoall-job-oguz-mpiworker-18 device 4 [0x89] NVIDIA H100 80GB HBM3 | |
| # Rank 149 Group 0 Pid 51 on nccl-alltoall-job-oguz-mpiworker-18 device 5 [0xa8] NVIDIA H100 80GB HBM3 | |
| # Rank 150 Group 0 Pid 52 on nccl-alltoall-job-oguz-mpiworker-18 device 6 [0xc0] NVIDIA H100 80GB HBM3 | |
| # Rank 151 Group 0 Pid 53 on nccl-alltoall-job-oguz-mpiworker-18 device 7 [0xd8] NVIDIA H100 80GB HBM3 | |
| # Rank 152 Group 0 Pid 46 on nccl-alltoall-job-oguz-mpiworker-19 device 0 [0x0f] NVIDIA H100 80GB HBM3 | |
| # Rank 153 Group 0 Pid 47 on nccl-alltoall-job-oguz-mpiworker-19 device 1 [0x2d] NVIDIA H100 80GB HBM3 | |
| # Rank 154 Group 0 Pid 48 on nccl-alltoall-job-oguz-mpiworker-19 device 2 [0x44] NVIDIA H100 80GB HBM3 | |
| # Rank 155 Group 0 Pid 49 on nccl-alltoall-job-oguz-mpiworker-19 device 3 [0x5b] NVIDIA H100 80GB HBM3 | |
| # Rank 156 Group 0 Pid 50 on nccl-alltoall-job-oguz-mpiworker-19 device 4 [0x89] NVIDIA H100 80GB HBM3 | |
| # Rank 157 Group 0 Pid 51 on nccl-alltoall-job-oguz-mpiworker-19 device 5 [0xa8] NVIDIA H100 80GB HBM3 | |
| # Rank 158 Group 0 Pid 52 on nccl-alltoall-job-oguz-mpiworker-19 device 6 [0xc0] NVIDIA H100 80GB HBM3 | |
| # Rank 159 Group 0 Pid 53 on nccl-alltoall-job-oguz-mpiworker-19 device 7 [0xd8] NVIDIA H100 80GB HBM3 | |
| # Rank 160 Group 0 Pid 46 on nccl-alltoall-job-oguz-mpiworker-20 device 0 [0x0f] NVIDIA H100 80GB HBM3 | |
| # Rank 161 Group 0 Pid 47 on nccl-alltoall-job-oguz-mpiworker-20 device 1 [0x2d] NVIDIA H100 80GB HBM3 | |
| # Rank 162 Group 0 Pid 48 on nccl-alltoall-job-oguz-mpiworker-20 device 2 [0x44] NVIDIA H100 80GB HBM3 | |
| # Rank 163 Group 0 Pid 49 on nccl-alltoall-job-oguz-mpiworker-20 device 3 [0x5b] NVIDIA H100 80GB HBM3 | |
| # Rank 164 Group 0 Pid 50 on nccl-alltoall-job-oguz-mpiworker-20 device 4 [0x89] NVIDIA H100 80GB HBM3 | |
| # Rank 165 Group 0 Pid 51 on nccl-alltoall-job-oguz-mpiworker-20 device 5 [0xa8] NVIDIA H100 80GB HBM3 | |
| # Rank 166 Group 0 Pid 52 on nccl-alltoall-job-oguz-mpiworker-20 device 6 [0xc0] NVIDIA H100 80GB HBM3 | |
| # Rank 167 Group 0 Pid 53 on nccl-alltoall-job-oguz-mpiworker-20 device 7 [0xd8] NVIDIA H100 80GB HBM3 | |
| # Rank 168 Group 0 Pid 46 on nccl-alltoall-job-oguz-mpiworker-21 device 0 [0x0f] NVIDIA H100 80GB HBM3 | |
| # Rank 169 Group 0 Pid 47 on nccl-alltoall-job-oguz-mpiworker-21 device 1 [0x2d] NVIDIA H100 80GB HBM3 | |
| # Rank 170 Group 0 Pid 48 on nccl-alltoall-job-oguz-mpiworker-21 device 2 [0x44] NVIDIA H100 80GB HBM3 | |
| # Rank 171 Group 0 Pid 49 on nccl-alltoall-job-oguz-mpiworker-21 device 3 [0x5b] NVIDIA H100 80GB HBM3 | |
| # Rank 172 Group 0 Pid 50 on nccl-alltoall-job-oguz-mpiworker-21 device 4 [0x89] NVIDIA H100 80GB HBM3 | |
| # Rank 173 Group 0 Pid 51 on nccl-alltoall-job-oguz-mpiworker-21 device 5 [0xa8] NVIDIA H100 80GB HBM3 | |
| # Rank 174 Group 0 Pid 52 on nccl-alltoall-job-oguz-mpiworker-21 device 6 [0xc0] NVIDIA H100 80GB HBM3 | |
| # Rank 175 Group 0 Pid 53 on nccl-alltoall-job-oguz-mpiworker-21 device 7 [0xd8] NVIDIA H100 80GB HBM3 | |
| # Rank 176 Group 0 Pid 46 on nccl-alltoall-job-oguz-mpiworker-22 device 0 [0x0f] NVIDIA H100 80GB HBM3 | |
| # Rank 177 Group 0 Pid 47 on nccl-alltoall-job-oguz-mpiworker-22 device 1 [0x2d] NVIDIA H100 80GB HBM3 | |
| # Rank 178 Group 0 Pid 48 on nccl-alltoall-job-oguz-mpiworker-22 device 2 [0x44] NVIDIA H100 80GB HBM3 | |
| # Rank 179 Group 0 Pid 49 on nccl-alltoall-job-oguz-mpiworker-22 device 3 [0x5b] NVIDIA H100 80GB HBM3 | |
| # Rank 180 Group 0 Pid 50 on nccl-alltoall-job-oguz-mpiworker-22 device 4 [0x89] NVIDIA H100 80GB HBM3 | |
| # Rank 181 Group 0 Pid 51 on nccl-alltoall-job-oguz-mpiworker-22 device 5 [0xa8] NVIDIA H100 80GB HBM3 | |
| # Rank 182 Group 0 Pid 52 on nccl-alltoall-job-oguz-mpiworker-22 device 6 [0xc0] NVIDIA H100 80GB HBM3 | |
| # Rank 183 Group 0 Pid 53 on nccl-alltoall-job-oguz-mpiworker-22 device 7 [0xd8] NVIDIA H100 80GB HBM3 | |
| # Rank 184 Group 0 Pid 46 on nccl-alltoall-job-oguz-mpiworker-23 device 0 [0x0f] NVIDIA H100 80GB HBM3 | |
| # Rank 185 Group 0 Pid 47 on nccl-alltoall-job-oguz-mpiworker-23 device 1 [0x2d] NVIDIA H100 80GB HBM3 | |
| # Rank 186 Group 0 Pid 48 on nccl-alltoall-job-oguz-mpiworker-23 device 2 [0x44] NVIDIA H100 80GB HBM3 | |
| # Rank 187 Group 0 Pid 49 on nccl-alltoall-job-oguz-mpiworker-23 device 3 [0x5b] NVIDIA H100 80GB HBM3 | |
| # Rank 188 Group 0 Pid 50 on nccl-alltoall-job-oguz-mpiworker-23 device 4 [0x89] NVIDIA H100 80GB HBM3 | |
| # Rank 189 Group 0 Pid 51 on nccl-alltoall-job-oguz-mpiworker-23 device 5 [0xa8] NVIDIA H100 80GB HBM3 | |
| # Rank 190 Group 0 Pid 52 on nccl-alltoall-job-oguz-mpiworker-23 device 6 [0xc0] NVIDIA H100 80GB HBM3 | |
| # Rank 191 Group 0 Pid 53 on nccl-alltoall-job-oguz-mpiworker-23 device 7 [0xd8] NVIDIA H100 80GB HBM3 | |
| # Rank 192 Group 0 Pid 46 on nccl-alltoall-job-oguz-mpiworker-24 device 0 [0x0f] NVIDIA H100 80GB HBM3 | |
| # Rank 193 Group 0 Pid 47 on nccl-alltoall-job-oguz-mpiworker-24 device 1 [0x2d] NVIDIA H100 80GB HBM3 | |
| # Rank 194 Group 0 Pid 48 on nccl-alltoall-job-oguz-mpiworker-24 device 2 [0x44] NVIDIA H100 80GB HBM3 | |
| # Rank 195 Group 0 Pid 49 on nccl-alltoall-job-oguz-mpiworker-24 device 3 [0x5b] NVIDIA H100 80GB HBM3 | |
| # Rank 196 Group 0 Pid 50 on nccl-alltoall-job-oguz-mpiworker-24 device 4 [0x89] NVIDIA H100 80GB HBM3 | |
| # Rank 197 Group 0 Pid 51 on nccl-alltoall-job-oguz-mpiworker-24 device 5 [0xa8] NVIDIA H100 80GB HBM3 | |
| # Rank 198 Group 0 Pid 52 on nccl-alltoall-job-oguz-mpiworker-24 device 6 [0xc0] NVIDIA H100 80GB HBM3 | |
| # Rank 199 Group 0 Pid 53 on nccl-alltoall-job-oguz-mpiworker-24 device 7 [0xd8] NVIDIA H100 80GB HBM3 | |
| # Rank 200 Group 0 Pid 46 on nccl-alltoall-job-oguz-mpiworker-25 device 0 [0x0f] NVIDIA H100 80GB HBM3 | |
| # Rank 201 Group 0 Pid 47 on nccl-alltoall-job-oguz-mpiworker-25 device 1 [0x2d] NVIDIA H100 80GB HBM3 | |
| # Rank 202 Group 0 Pid 48 on nccl-alltoall-job-oguz-mpiworker-25 device 2 [0x44] NVIDIA H100 80GB HBM3 | |
| # Rank 203 Group 0 Pid 49 on nccl-alltoall-job-oguz-mpiworker-25 device 3 [0x5b] NVIDIA H100 80GB HBM3 | |
| # Rank 204 Group 0 Pid 50 on nccl-alltoall-job-oguz-mpiworker-25 device 4 [0x89] NVIDIA H100 80GB HBM3 | |
| # Rank 205 Group 0 Pid 51 on nccl-alltoall-job-oguz-mpiworker-25 device 5 [0xa8] NVIDIA H100 80GB HBM3 | |
| # Rank 206 Group 0 Pid 52 on nccl-alltoall-job-oguz-mpiworker-25 device 6 [0xc0] NVIDIA H100 80GB HBM3 | |
| # Rank 207 Group 0 Pid 53 on nccl-alltoall-job-oguz-mpiworker-25 device 7 [0xd8] NVIDIA H100 80GB HBM3 | |
| # Rank 208 Group 0 Pid 46 on nccl-alltoall-job-oguz-mpiworker-26 device 0 [0x0f] NVIDIA H100 80GB HBM3 | |
| # Rank 209 Group 0 Pid 47 on nccl-alltoall-job-oguz-mpiworker-26 device 1 [0x2d] NVIDIA H100 80GB HBM3 | |
| # Rank 210 Group 0 Pid 48 on nccl-alltoall-job-oguz-mpiworker-26 device 2 [0x44] NVIDIA H100 80GB HBM3 | |
| # Rank 211 Group 0 Pid 49 on nccl-alltoall-job-oguz-mpiworker-26 device 3 [0x5b] NVIDIA H100 80GB HBM3 | |
| # Rank 212 Group 0 Pid 50 on nccl-alltoall-job-oguz-mpiworker-26 device 4 [0x89] NVIDIA H100 80GB HBM3 | |
| # Rank 213 Group 0 Pid 51 on nccl-alltoall-job-oguz-mpiworker-26 device 5 [0xa8] NVIDIA H100 80GB HBM3 | |
| # Rank 214 Group 0 Pid 52 on nccl-alltoall-job-oguz-mpiworker-26 device 6 [0xc0] NVIDIA H100 80GB HBM3 | |
| # Rank 215 Group 0 Pid 53 on nccl-alltoall-job-oguz-mpiworker-26 device 7 [0xd8] NVIDIA H100 80GB HBM3 | |
| # Rank 216 Group 0 Pid 46 on nccl-alltoall-job-oguz-mpiworker-27 device 0 [0x0f] NVIDIA H100 80GB HBM3 | |
| # Rank 217 Group 0 Pid 47 on nccl-alltoall-job-oguz-mpiworker-27 device 1 [0x2d] NVIDIA H100 80GB HBM3 | |
| # Rank 218 Group 0 Pid 48 on nccl-alltoall-job-oguz-mpiworker-27 device 2 [0x44] NVIDIA H100 80GB HBM3 | |
| # Rank 219 Group 0 Pid 49 on nccl-alltoall-job-oguz-mpiworker-27 device 3 [0x5b] NVIDIA H100 80GB HBM3 | |
| # Rank 220 Group 0 Pid 50 on nccl-alltoall-job-oguz-mpiworker-27 device 4 [0x89] NVIDIA H100 80GB HBM3 | |
| # Rank 221 Group 0 Pid 51 on nccl-alltoall-job-oguz-mpiworker-27 device 5 [0xa8] NVIDIA H100 80GB HBM3 | |
| # Rank 222 Group 0 Pid 52 on nccl-alltoall-job-oguz-mpiworker-27 device 6 [0xc0] NVIDIA H100 80GB HBM3 | |
| # Rank 223 Group 0 Pid 53 on nccl-alltoall-job-oguz-mpiworker-27 device 7 [0xd8] NVIDIA H100 80GB HBM3 | |
| # Rank 224 Group 0 Pid 46 on nccl-alltoall-job-oguz-mpiworker-28 device 0 [0x0f] NVIDIA H100 80GB HBM3 | |
| # Rank 225 Group 0 Pid 47 on nccl-alltoall-job-oguz-mpiworker-28 device 1 [0x2d] NVIDIA H100 80GB HBM3 | |
| # Rank 226 Group 0 Pid 48 on nccl-alltoall-job-oguz-mpiworker-28 device 2 [0x44] NVIDIA H100 80GB HBM3 | |
| # Rank 227 Group 0 Pid 49 on nccl-alltoall-job-oguz-mpiworker-28 device 3 [0x5b] NVIDIA H100 80GB HBM3 | |
| # Rank 228 Group 0 Pid 50 on nccl-alltoall-job-oguz-mpiworker-28 device 4 [0x89] NVIDIA H100 80GB HBM3 | |
| # Rank 229 Group 0 Pid 51 on nccl-alltoall-job-oguz-mpiworker-28 device 5 [0xa8] NVIDIA H100 80GB HBM3 | |
| # Rank 230 Group 0 Pid 52 on nccl-alltoall-job-oguz-mpiworker-28 device 6 [0xc0] NVIDIA H100 80GB HBM3 | |
| # Rank 231 Group 0 Pid 53 on nccl-alltoall-job-oguz-mpiworker-28 device 7 [0xd8] NVIDIA H100 80GB HBM3 | |
| # Rank 232 Group 0 Pid 46 on nccl-alltoall-job-oguz-mpiworker-29 device 0 [0x0f] NVIDIA H100 80GB HBM3 | |
| # Rank 233 Group 0 Pid 47 on nccl-alltoall-job-oguz-mpiworker-29 device 1 [0x2d] NVIDIA H100 80GB HBM3 | |
| # Rank 234 Group 0 Pid 48 on nccl-alltoall-job-oguz-mpiworker-29 device 2 [0x44] NVIDIA H100 80GB HBM3 | |
| # Rank 235 Group 0 Pid 49 on nccl-alltoall-job-oguz-mpiworker-29 device 3 [0x5b] NVIDIA H100 80GB HBM3 | |
| # Rank 236 Group 0 Pid 50 on nccl-alltoall-job-oguz-mpiworker-29 device 4 [0x89] NVIDIA H100 80GB HBM3 | |
| # Rank 237 Group 0 Pid 51 on nccl-alltoall-job-oguz-mpiworker-29 device 5 [0xa8] NVIDIA H100 80GB HBM3 | |
| # Rank 238 Group 0 Pid 52 on nccl-alltoall-job-oguz-mpiworker-29 device 6 [0xc0] NVIDIA H100 80GB HBM3 | |
| # Rank 239 Group 0 Pid 53 on nccl-alltoall-job-oguz-mpiworker-29 device 7 [0xd8] NVIDIA H100 80GB HBM3 | |
| # Rank 240 Group 0 Pid 46 on nccl-alltoall-job-oguz-mpiworker-30 device 0 [0x0f] NVIDIA H100 80GB HBM3 | |
| # Rank 241 Group 0 Pid 47 on nccl-alltoall-job-oguz-mpiworker-30 device 1 [0x2d] NVIDIA H100 80GB HBM3 | |
| # Rank 242 Group 0 Pid 48 on nccl-alltoall-job-oguz-mpiworker-30 device 2 [0x44] NVIDIA H100 80GB HBM3 | |
| # Rank 243 Group 0 Pid 49 on nccl-alltoall-job-oguz-mpiworker-30 device 3 [0x5b] NVIDIA H100 80GB HBM3 | |
| # Rank 244 Group 0 Pid 50 on nccl-alltoall-job-oguz-mpiworker-30 device 4 [0x89] NVIDIA H100 80GB HBM3 | |
| # Rank 245 Group 0 Pid 51 on nccl-alltoall-job-oguz-mpiworker-30 device 5 [0xa8] NVIDIA H100 80GB HBM3 | |
| # Rank 246 Group 0 Pid 52 on nccl-alltoall-job-oguz-mpiworker-30 device 6 [0xc0] NVIDIA H100 80GB HBM3 | |
| # Rank 247 Group 0 Pid 53 on nccl-alltoall-job-oguz-mpiworker-30 device 7 [0xd8] NVIDIA H100 80GB HBM3 | |
| # Rank 248 Group 0 Pid 46 on nccl-alltoall-job-oguz-mpiworker-31 device 0 [0x0f] NVIDIA H100 80GB HBM3 | |
| # Rank 249 Group 0 Pid 47 on nccl-alltoall-job-oguz-mpiworker-31 device 1 [0x2d] NVIDIA H100 80GB HBM3 | |
| # Rank 250 Group 0 Pid 48 on nccl-alltoall-job-oguz-mpiworker-31 device 2 [0x44] NVIDIA H100 80GB HBM3 | |
| # Rank 251 Group 0 Pid 49 on nccl-alltoall-job-oguz-mpiworker-31 device 3 [0x5b] NVIDIA H100 80GB HBM3 | |
| # Rank 252 Group 0 Pid 50 on nccl-alltoall-job-oguz-mpiworker-31 device 4 [0x89] NVIDIA H100 80GB HBM3 | |
| # Rank 253 Group 0 Pid 51 on nccl-alltoall-job-oguz-mpiworker-31 device 5 [0xa8] NVIDIA H100 80GB HBM3 | |
| # Rank 254 Group 0 Pid 52 on nccl-alltoall-job-oguz-mpiworker-31 device 6 [0xc0] NVIDIA H100 80GB HBM3 | |
| # Rank 255 Group 0 Pid 53 on nccl-alltoall-job-oguz-mpiworker-31 device 7 [0xd8] NVIDIA H100 80GB HBM3 | |
| nccl-alltoall-job-oguz-mpiworker-0:46:46 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-0:46:46 [0] NCCL INFO Bootstrap : Using eth0:10.244.72.84<0> | |
| nccl-alltoall-job-oguz-mpiworker-0:46:46 [0] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-0:46:46 [0] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-10:46:46 [0] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-10:46:46 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-10:46:46 [0] NCCL INFO Bootstrap : Using eth0:10.244.150.6<0> | |
| nccl-alltoall-job-oguz-mpiworker-10:46:46 [0] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-10:46:46 [0] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-24:48:48 [2] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-24:48:48 [2] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-24:48:48 [2] NCCL INFO Bootstrap : Using eth0:10.244.172.120<0> | |
| nccl-alltoall-job-oguz-mpiworker-24:48:48 [2] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-24:48:48 [2] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-20:51:51 [5] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-20:51:51 [5] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-20:51:51 [5] NCCL INFO Bootstrap : Using eth0:10.244.124.65<0> | |
| nccl-alltoall-job-oguz-mpiworker-20:51:51 [5] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-20:51:51 [5] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-27:52:52 [6] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-27:52:52 [6] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-7:46:46 [0] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-7:46:46 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-27:52:52 [6] NCCL INFO Bootstrap : Using eth0:10.244.144.145<0> | |
| nccl-alltoall-job-oguz-mpiworker-27:52:52 [6] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-27:52:52 [6] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-7:46:46 [0] NCCL INFO Bootstrap : Using eth0:10.244.115.133<0> | |
| nccl-alltoall-job-oguz-mpiworker-7:46:46 [0] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-7:46:46 [0] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-26:47:47 [1] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-26:47:47 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-26:47:47 [1] NCCL INFO Bootstrap : Using eth0:10.244.157.211<0> | |
| nccl-alltoall-job-oguz-mpiworker-26:47:47 [1] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-26:47:47 [1] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-26:46:46 [0] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-26:46:46 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-26:46:46 [0] NCCL INFO Bootstrap : Using eth0:10.244.157.211<0> | |
| nccl-alltoall-job-oguz-mpiworker-26:46:46 [0] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-26:46:46 [0] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-7:52:52 [6] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-7:52:52 [6] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-7:52:52 [6] NCCL INFO Bootstrap : Using eth0:10.244.115.133<0> | |
| nccl-alltoall-job-oguz-mpiworker-7:52:52 [6] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-7:52:52 [6] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-25:46:46 [0] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-25:46:46 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-25:46:46 [0] NCCL INFO Bootstrap : Using eth0:10.244.121.49<0> | |
| nccl-alltoall-job-oguz-mpiworker-25:46:46 [0] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-25:46:46 [0] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-7:51:51 [5] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-7:51:51 [5] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-7:51:51 [5] NCCL INFO Bootstrap : Using eth0:10.244.115.133<0> | |
| nccl-alltoall-job-oguz-mpiworker-7:51:51 [5] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-7:51:51 [5] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-10:48:48 [2] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-10:48:48 [2] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-30:46:46 [0] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-30:46:46 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-10:48:48 [2] NCCL INFO Bootstrap : Using eth0:10.244.150.6<0> | |
| nccl-alltoall-job-oguz-mpiworker-10:48:48 [2] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-10:48:48 [2] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-27:46:46 [0] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-27:46:46 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-30:46:46 [0] NCCL INFO Bootstrap : Using eth0:10.244.155.168<0> | |
| nccl-alltoall-job-oguz-mpiworker-30:46:46 [0] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-30:46:46 [0] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-27:46:46 [0] NCCL INFO Bootstrap : Using eth0:10.244.144.145<0> | |
| nccl-alltoall-job-oguz-mpiworker-27:46:46 [0] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-21:46:46 [0] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-21:46:46 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-27:46:46 [0] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-0:46:46 [0] NCCL INFO cudaDriverVersion 12020 | |
| NCCL version 2.19.3+cuda12.2 | |
| nccl-alltoall-job-oguz-mpiworker-7:47:47 [1] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-7:47:47 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-21:46:46 [0] NCCL INFO Bootstrap : Using eth0:10.244.149.199<0> | |
| nccl-alltoall-job-oguz-mpiworker-21:46:46 [0] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-21:46:46 [0] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-7:47:47 [1] NCCL INFO Bootstrap : Using eth0:10.244.115.133<0> | |
| nccl-alltoall-job-oguz-mpiworker-7:47:47 [1] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-7:47:47 [1] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-27:50:50 [4] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-27:50:50 [4] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-7:50:50 [4] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-7:50:50 [4] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-27:50:50 [4] NCCL INFO Bootstrap : Using eth0:10.244.144.145<0> | |
| nccl-alltoall-job-oguz-mpiworker-27:50:50 [4] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-27:50:50 [4] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-7:50:50 [4] NCCL INFO Bootstrap : Using eth0:10.244.115.133<0> | |
| nccl-alltoall-job-oguz-mpiworker-7:50:50 [4] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-7:50:50 [4] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-7:49:49 [3] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-7:49:49 [3] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-7:49:49 [3] NCCL INFO Bootstrap : Using eth0:10.244.115.133<0> | |
| nccl-alltoall-job-oguz-mpiworker-7:49:49 [3] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-7:49:49 [3] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-7:54:54 [7] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-7:54:54 [7] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-7:48:48 [2] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-7:48:48 [2] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-16:46:46 [0] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-16:46:46 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-7:54:54 [7] NCCL INFO Bootstrap : Using eth0:10.244.115.133<0> | |
| nccl-alltoall-job-oguz-mpiworker-7:54:54 [7] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-7:48:48 [2] NCCL INFO Bootstrap : Using eth0:10.244.115.133<0> | |
| nccl-alltoall-job-oguz-mpiworker-7:48:48 [2] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-7:54:54 [7] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-7:48:48 [2] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-16:46:46 [0] NCCL INFO Bootstrap : Using eth0:10.244.16.145<0> | |
| nccl-alltoall-job-oguz-mpiworker-16:46:46 [0] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-16:46:46 [0] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-10:51:51 [5] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-10:51:51 [5] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-10:51:51 [5] NCCL INFO Bootstrap : Using eth0:10.244.150.6<0> | |
| nccl-alltoall-job-oguz-mpiworker-10:51:51 [5] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-10:51:51 [5] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-10:52:52 [6] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-10:52:52 [6] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-0:51:51 [5] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-0:51:51 [5] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-10:50:50 [4] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-10:50:50 [4] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-0:51:51 [5] NCCL INFO Bootstrap : Using eth0:10.244.72.84<0> | |
| nccl-alltoall-job-oguz-mpiworker-0:51:51 [5] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-0:51:51 [5] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-10:52:52 [6] NCCL INFO Bootstrap : Using eth0:10.244.150.6<0> | |
| nccl-alltoall-job-oguz-mpiworker-10:52:52 [6] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-10:50:50 [4] NCCL INFO Bootstrap : Using eth0:10.244.150.6<0> | |
| nccl-alltoall-job-oguz-mpiworker-10:50:50 [4] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-10:52:52 [6] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-10:50:50 [4] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-10:53:53 [7] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-10:53:53 [7] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-10:47:47 [1] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-10:47:47 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-13:48:48 [2] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-13:48:48 [2] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-10:49:49 [3] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-10:49:49 [3] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-19:46:46 [0] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-19:46:46 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-10:53:53 [7] NCCL INFO Bootstrap : Using eth0:10.244.150.6<0> | |
| nccl-alltoall-job-oguz-mpiworker-10:53:53 [7] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-10:53:53 [7] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-10:47:47 [1] NCCL INFO Bootstrap : Using eth0:10.244.150.6<0> | |
| nccl-alltoall-job-oguz-mpiworker-10:47:47 [1] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-10:47:47 [1] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-13:48:48 [2] NCCL INFO Bootstrap : Using eth0:10.244.125.171<0> | |
| nccl-alltoall-job-oguz-mpiworker-13:48:48 [2] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-10:49:49 [3] NCCL INFO Bootstrap : Using eth0:10.244.150.6<0> | |
| nccl-alltoall-job-oguz-mpiworker-10:49:49 [3] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-13:48:48 [2] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-10:49:49 [3] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-24:47:47 [1] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-24:47:47 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-19:46:46 [0] NCCL INFO Bootstrap : Using eth0:10.244.117.132<0> | |
| nccl-alltoall-job-oguz-mpiworker-19:46:46 [0] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-23:46:46 [0] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-23:46:46 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-19:46:46 [0] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-13:47:47 [1] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-13:47:47 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-24:47:47 [1] NCCL INFO Bootstrap : Using eth0:10.244.172.120<0> | |
| nccl-alltoall-job-oguz-mpiworker-24:47:47 [1] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-29:51:51 [5] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-13:47:47 [1] NCCL INFO Bootstrap : Using eth0:10.244.125.171<0> | |
| nccl-alltoall-job-oguz-mpiworker-13:47:47 [1] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-29:51:51 [5] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-24:47:47 [1] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-13:47:47 [1] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-5:46:46 [0] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-5:46:46 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-23:46:46 [0] NCCL INFO Bootstrap : Using eth0:10.244.32.41<0> | |
| nccl-alltoall-job-oguz-mpiworker-23:46:46 [0] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-23:46:46 [0] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-24:52:52 [6] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-24:52:52 [6] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-25:53:53 [7] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-25:53:53 [7] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-29:51:51 [5] NCCL INFO Bootstrap : Using eth0:10.244.90.183<0> | |
| nccl-alltoall-job-oguz-mpiworker-29:51:51 [5] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-29:51:51 [5] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-27:51:51 [5] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-27:51:51 [5] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-5:46:46 [0] NCCL INFO Bootstrap : Using eth0:10.244.169.8<0> | |
| nccl-alltoall-job-oguz-mpiworker-5:46:46 [0] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-27:49:49 [3] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-27:49:49 [3] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-5:46:46 [0] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-24:52:52 [6] NCCL INFO Bootstrap : Using eth0:10.244.172.120<0> | |
| nccl-alltoall-job-oguz-mpiworker-24:52:52 [6] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-24:52:52 [6] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-25:53:53 [7] NCCL INFO Bootstrap : Using eth0:10.244.121.49<0> | |
| nccl-alltoall-job-oguz-mpiworker-25:53:53 [7] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-25:53:53 [7] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-29:48:48 [2] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-29:48:48 [2] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-27:51:51 [5] NCCL INFO Bootstrap : Using eth0:10.244.144.145<0> | |
| nccl-alltoall-job-oguz-mpiworker-27:51:51 [5] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-27:49:49 [3] NCCL INFO Bootstrap : Using eth0:10.244.144.145<0> | |
| nccl-alltoall-job-oguz-mpiworker-27:49:49 [3] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-27:49:49 [3] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-27:51:51 [5] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-27:48:48 [2] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-27:48:48 [2] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-29:48:48 [2] NCCL INFO Bootstrap : Using eth0:10.244.90.183<0> | |
| nccl-alltoall-job-oguz-mpiworker-29:48:48 [2] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-29:48:48 [2] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-27:48:48 [2] NCCL INFO Bootstrap : Using eth0:10.244.144.145<0> | |
| nccl-alltoall-job-oguz-mpiworker-27:48:48 [2] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-27:48:48 [2] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-30:53:53 [7] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-30:53:53 [7] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-30:47:47 [1] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-30:47:47 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-16:48:48 [2] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-16:48:48 [2] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-30:53:53 [7] NCCL INFO Bootstrap : Using eth0:10.244.155.168<0> | |
| nccl-alltoall-job-oguz-mpiworker-30:53:53 [7] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-30:53:53 [7] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-16:50:50 [4] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-16:50:50 [4] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-27:47:47 [1] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-27:47:47 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-30:47:47 [1] NCCL INFO Bootstrap : Using eth0:10.244.155.168<0> | |
| nccl-alltoall-job-oguz-mpiworker-30:47:47 [1] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-30:47:47 [1] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-16:49:49 [3] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-16:49:49 [3] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-27:47:47 [1] NCCL INFO Bootstrap : Using eth0:10.244.144.145<0> | |
| nccl-alltoall-job-oguz-mpiworker-27:47:47 [1] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-27:47:47 [1] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-4:50:50 [4] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-4:50:50 [4] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-16:48:48 [2] NCCL INFO Bootstrap : Using eth0:10.244.16.145<0> | |
| nccl-alltoall-job-oguz-mpiworker-16:48:48 [2] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-16:48:48 [2] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-16:50:50 [4] NCCL INFO Bootstrap : Using eth0:10.244.16.145<0> | |
| nccl-alltoall-job-oguz-mpiworker-16:50:50 [4] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-16:50:50 [4] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-27:53:53 [7] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-27:53:53 [7] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-16:49:49 [3] NCCL INFO Bootstrap : Using eth0:10.244.16.145<0> | |
| nccl-alltoall-job-oguz-mpiworker-16:49:49 [3] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-16:49:49 [3] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-4:50:50 [4] NCCL INFO Bootstrap : Using eth0:10.244.80.162<0> | |
| nccl-alltoall-job-oguz-mpiworker-4:50:50 [4] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-16:47:47 [1] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-16:47:47 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-4:50:50 [4] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-16:51:51 [5] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-16:51:51 [5] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-4:46:46 [0] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-4:46:46 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-27:53:53 [7] NCCL INFO Bootstrap : Using eth0:10.244.144.145<0> | |
| nccl-alltoall-job-oguz-mpiworker-27:53:53 [7] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-4:47:47 [1] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-27:53:53 [7] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-4:47:47 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-16:47:47 [1] NCCL INFO Bootstrap : Using eth0:10.244.16.145<0> | |
| nccl-alltoall-job-oguz-mpiworker-16:47:47 [1] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-16:51:51 [5] NCCL INFO Bootstrap : Using eth0:10.244.16.145<0> | |
| nccl-alltoall-job-oguz-mpiworker-16:51:51 [5] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-16:47:47 [1] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-16:51:51 [5] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-16:53:53 [7] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-16:53:53 [7] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-4:46:46 [0] NCCL INFO Bootstrap : Using eth0:10.244.80.162<0> | |
| nccl-alltoall-job-oguz-mpiworker-4:46:46 [0] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-4:47:47 [1] NCCL INFO Bootstrap : Using eth0:10.244.80.162<0> | |
| nccl-alltoall-job-oguz-mpiworker-4:47:47 [1] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-4:46:46 [0] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-4:47:47 [1] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-16:53:53 [7] NCCL INFO Bootstrap : Using eth0:10.244.16.145<0> | |
| nccl-alltoall-job-oguz-mpiworker-16:53:53 [7] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-16:53:53 [7] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-29:46:46 [0] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-29:46:46 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-29:46:46 [0] NCCL INFO Bootstrap : Using eth0:10.244.90.183<0> | |
| nccl-alltoall-job-oguz-mpiworker-29:46:46 [0] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-29:46:46 [0] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-26:53:53 [7] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-26:53:53 [7] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-29:47:47 [1] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-29:47:47 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-26:53:53 [7] NCCL INFO Bootstrap : Using eth0:10.244.157.211<0> | |
| nccl-alltoall-job-oguz-mpiworker-26:53:53 [7] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-29:47:47 [1] NCCL INFO Bootstrap : Using eth0:10.244.90.183<0> | |
| nccl-alltoall-job-oguz-mpiworker-29:47:47 [1] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-26:53:53 [7] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-29:47:47 [1] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-23:53:53 [7] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-23:53:53 [7] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-0:48:48 [2] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-0:48:48 [2] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-23:53:53 [7] NCCL INFO Bootstrap : Using eth0:10.244.32.41<0> | |
| nccl-alltoall-job-oguz-mpiworker-23:53:53 [7] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-23:53:53 [7] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-23:48:48 [2] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-23:48:48 [2] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-0:48:48 [2] NCCL INFO Bootstrap : Using eth0:10.244.72.84<0> | |
| nccl-alltoall-job-oguz-mpiworker-0:48:48 [2] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-0:48:48 [2] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-29:50:50 [4] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-29:50:50 [4] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-23:48:48 [2] NCCL INFO Bootstrap : Using eth0:10.244.32.41<0> | |
| nccl-alltoall-job-oguz-mpiworker-23:48:48 [2] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-23:48:48 [2] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-29:50:50 [4] NCCL INFO Bootstrap : Using eth0:10.244.90.183<0> | |
| nccl-alltoall-job-oguz-mpiworker-29:50:50 [4] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-29:50:50 [4] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-1:46:46 [0] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-1:46:46 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-1:48:48 [2] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-1:48:48 [2] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-29:53:53 [7] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-29:53:53 [7] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-29:49:49 [3] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-29:49:49 [3] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-23:50:50 [4] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-23:50:50 [4] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-29:52:52 [6] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-29:52:52 [6] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-23:47:47 [1] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-23:47:47 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-13:53:53 [7] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-13:53:53 [7] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-13:52:52 [6] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-13:52:52 [6] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-1:46:46 [0] NCCL INFO Bootstrap : Using eth0:10.244.130.46<0> | |
| nccl-alltoall-job-oguz-mpiworker-1:46:46 [0] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-1:48:48 [2] NCCL INFO Bootstrap : Using eth0:10.244.130.46<0> | |
| nccl-alltoall-job-oguz-mpiworker-1:48:48 [2] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-1:46:46 [0] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-13:46:46 [0] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-1:48:48 [2] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-13:46:46 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-0:53:53 [7] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-13:49:49 [3] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-0:53:53 [7] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-13:49:49 [3] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-29:53:53 [7] NCCL INFO Bootstrap : Using eth0:10.244.90.183<0> | |
| nccl-alltoall-job-oguz-mpiworker-29:53:53 [7] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-29:53:53 [7] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-16:52:52 [6] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-16:52:52 [6] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-29:49:49 [3] NCCL INFO Bootstrap : Using eth0:10.244.90.183<0> | |
| nccl-alltoall-job-oguz-mpiworker-29:49:49 [3] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-29:52:52 [6] NCCL INFO Bootstrap : Using eth0:10.244.90.183<0> | |
| nccl-alltoall-job-oguz-mpiworker-29:52:52 [6] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-29:49:49 [3] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-29:52:52 [6] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-23:50:50 [4] NCCL INFO Bootstrap : Using eth0:10.244.32.41<0> | |
| nccl-alltoall-job-oguz-mpiworker-23:50:50 [4] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-23:47:47 [1] NCCL INFO Bootstrap : Using eth0:10.244.32.41<0> | |
| nccl-alltoall-job-oguz-mpiworker-23:47:47 [1] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-23:51:51 [5] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-23:50:50 [4] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-23:47:47 [1] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-23:51:51 [5] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-25:51:51 [5] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-0:53:53 [7] NCCL INFO Bootstrap : Using eth0:10.244.72.84<0> | |
| nccl-alltoall-job-oguz-mpiworker-0:53:53 [7] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-25:51:51 [5] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-0:53:53 [7] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-23:49:49 [3] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-23:49:49 [3] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-16:52:52 [6] NCCL INFO Bootstrap : Using eth0:10.244.16.145<0> | |
| nccl-alltoall-job-oguz-mpiworker-16:52:52 [6] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-16:52:52 [6] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-23:52:52 [6] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-23:52:52 [6] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-25:51:51 [5] NCCL INFO Bootstrap : Using eth0:10.244.121.49<0> | |
| nccl-alltoall-job-oguz-mpiworker-25:51:51 [5] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-25:51:51 [5] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-13:50:50 [4] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-13:50:50 [4] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-3:47:47 [1] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-3:47:47 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-13:53:53 [7] NCCL INFO Bootstrap : Using eth0:10.244.125.171<0> | |
| nccl-alltoall-job-oguz-mpiworker-13:53:53 [7] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-13:53:53 [7] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-13:52:52 [6] NCCL INFO Bootstrap : Using eth0:10.244.125.171<0> | |
| nccl-alltoall-job-oguz-mpiworker-13:52:52 [6] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-23:51:51 [5] NCCL INFO Bootstrap : Using eth0:10.244.32.41<0> | |
| nccl-alltoall-job-oguz-mpiworker-23:51:51 [5] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-13:52:52 [6] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-23:51:51 [5] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-13:49:49 [3] NCCL INFO Bootstrap : Using eth0:10.244.125.171<0> | |
| nccl-alltoall-job-oguz-mpiworker-13:49:49 [3] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-13:46:46 [0] NCCL INFO Bootstrap : Using eth0:10.244.125.171<0> | |
| nccl-alltoall-job-oguz-mpiworker-13:46:46 [0] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-13:46:46 [0] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-13:49:49 [3] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-23:49:49 [3] NCCL INFO Bootstrap : Using eth0:10.244.32.41<0> | |
| nccl-alltoall-job-oguz-mpiworker-23:49:49 [3] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-23:52:52 [6] NCCL INFO Bootstrap : Using eth0:10.244.32.41<0> | |
| nccl-alltoall-job-oguz-mpiworker-23:52:52 [6] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-23:52:52 [6] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-13:50:50 [4] NCCL INFO Bootstrap : Using eth0:10.244.125.171<0> | |
| nccl-alltoall-job-oguz-mpiworker-13:50:50 [4] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-23:49:49 [3] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-13:50:50 [4] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-0:52:52 [6] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-0:52:52 [6] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-3:47:47 [1] NCCL INFO Bootstrap : Using eth0:10.244.129.187<0> | |
| nccl-alltoall-job-oguz-mpiworker-3:47:47 [1] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-3:47:47 [1] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-13:51:51 [5] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-13:51:51 [5] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-0:52:52 [6] NCCL INFO Bootstrap : Using eth0:10.244.72.84<0> | |
| nccl-alltoall-job-oguz-mpiworker-0:52:52 [6] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-0:52:52 [6] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-13:51:51 [5] NCCL INFO Bootstrap : Using eth0:10.244.125.171<0> | |
| nccl-alltoall-job-oguz-mpiworker-13:51:51 [5] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-13:51:51 [5] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-24:49:49 [3] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-24:49:49 [3] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-0:49:49 [3] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-0:49:49 [3] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-0:47:47 [1] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-0:47:47 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-0:50:50 [4] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-24:49:49 [3] NCCL INFO Bootstrap : Using eth0:10.244.172.120<0> | |
| nccl-alltoall-job-oguz-mpiworker-24:49:49 [3] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-0:50:50 [4] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-24:49:49 [3] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-24:51:51 [5] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-24:51:51 [5] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-24:51:51 [5] NCCL INFO Bootstrap : Using eth0:10.244.172.120<0> | |
| nccl-alltoall-job-oguz-mpiworker-24:51:51 [5] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-0:49:49 [3] NCCL INFO Bootstrap : Using eth0:10.244.72.84<0> | |
| nccl-alltoall-job-oguz-mpiworker-0:49:49 [3] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-24:51:51 [5] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-0:49:49 [3] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-4:49:49 [3] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-4:49:49 [3] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-0:47:47 [1] NCCL INFO Bootstrap : Using eth0:10.244.72.84<0> | |
| nccl-alltoall-job-oguz-mpiworker-0:47:47 [1] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-0:50:50 [4] NCCL INFO Bootstrap : Using eth0:10.244.72.84<0> | |
| nccl-alltoall-job-oguz-mpiworker-0:50:50 [4] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-0:47:47 [1] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-0:50:50 [4] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-25:49:49 [3] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-25:49:49 [3] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-4:49:49 [3] NCCL INFO Bootstrap : Using eth0:10.244.80.162<0> | |
| nccl-alltoall-job-oguz-mpiworker-4:49:49 [3] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-25:47:47 [1] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-25:47:47 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-4:49:49 [3] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-25:48:48 [2] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-25:48:48 [2] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-25:52:52 [6] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-25:52:52 [6] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-25:50:50 [4] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-25:50:50 [4] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-25:49:49 [3] NCCL INFO Bootstrap : Using eth0:10.244.121.49<0> | |
| nccl-alltoall-job-oguz-mpiworker-25:49:49 [3] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-25:49:49 [3] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-24:53:53 [7] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-24:53:53 [7] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-24:53:53 [7] NCCL INFO Bootstrap : Using eth0:10.244.172.120<0> | |
| nccl-alltoall-job-oguz-mpiworker-24:53:53 [7] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-24:53:53 [7] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-18:46:46 [0] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-18:46:46 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-30:51:51 [5] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-30:51:51 [5] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-25:47:47 [1] NCCL INFO Bootstrap : Using eth0:10.244.121.49<0> | |
| nccl-alltoall-job-oguz-mpiworker-25:47:47 [1] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-25:50:50 [4] NCCL INFO Bootstrap : Using eth0:10.244.121.49<0> | |
| nccl-alltoall-job-oguz-mpiworker-25:50:50 [4] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-25:48:48 [2] NCCL INFO Bootstrap : Using eth0:10.244.121.49<0> | |
| nccl-alltoall-job-oguz-mpiworker-25:48:48 [2] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-25:52:52 [6] NCCL INFO Bootstrap : Using eth0:10.244.121.49<0> | |
| nccl-alltoall-job-oguz-mpiworker-25:52:52 [6] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-30:51:51 [5] NCCL INFO Bootstrap : Using eth0:10.244.155.168<0> | |
| nccl-alltoall-job-oguz-mpiworker-30:51:51 [5] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-25:47:47 [1] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-25:50:50 [4] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-25:52:52 [6] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-26:49:49 [3] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-25:48:48 [2] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-30:51:51 [5] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-26:49:49 [3] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-18:46:46 [0] NCCL INFO Bootstrap : Using eth0:10.244.148.215<0> | |
| nccl-alltoall-job-oguz-mpiworker-18:46:46 [0] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-18:46:46 [0] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-21:52:52 [6] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-21:52:52 [6] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-21:48:48 [2] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-21:48:48 [2] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-28:47:47 [1] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-28:47:47 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-26:49:49 [3] NCCL INFO Bootstrap : Using eth0:10.244.157.211<0> | |
| nccl-alltoall-job-oguz-mpiworker-26:49:49 [3] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-21:53:53 [7] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-21:53:53 [7] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-26:49:49 [3] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-24:50:50 [4] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-24:50:50 [4] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-24:46:46 [0] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-26:50:50 [4] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-4:48:48 [2] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-26:50:50 [4] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-24:46:46 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-4:48:48 [2] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-28:47:47 [1] NCCL INFO Bootstrap : Using eth0:10.244.168.119<0> | |
| nccl-alltoall-job-oguz-mpiworker-28:47:47 [1] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-28:47:47 [1] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-4:48:48 [2] NCCL INFO Bootstrap : Using eth0:10.244.80.162<0> | |
| nccl-alltoall-job-oguz-mpiworker-4:48:48 [2] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-26:50:50 [4] NCCL INFO Bootstrap : Using eth0:10.244.157.211<0> | |
| nccl-alltoall-job-oguz-mpiworker-26:50:50 [4] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-4:48:48 [2] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-26:50:50 [4] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-21:52:52 [6] NCCL INFO Bootstrap : Using eth0:10.244.149.199<0> | |
| nccl-alltoall-job-oguz-mpiworker-21:52:52 [6] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-21:48:48 [2] NCCL INFO Bootstrap : Using eth0:10.244.149.199<0> | |
| nccl-alltoall-job-oguz-mpiworker-21:48:48 [2] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-21:52:52 [6] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-21:53:53 [7] NCCL INFO Bootstrap : Using eth0:10.244.149.199<0> | |
| nccl-alltoall-job-oguz-mpiworker-21:53:53 [7] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-21:48:48 [2] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-21:53:53 [7] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-24:46:46 [0] NCCL INFO Bootstrap : Using eth0:10.244.172.120<0> | |
| nccl-alltoall-job-oguz-mpiworker-24:46:46 [0] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-24:50:50 [4] NCCL INFO Bootstrap : Using eth0:10.244.172.120<0> | |
| nccl-alltoall-job-oguz-mpiworker-24:50:50 [4] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-24:46:46 [0] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-24:50:50 [4] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-4:53:53 [7] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-4:53:53 [7] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-20:53:53 [7] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-20:53:53 [7] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-4:53:53 [7] NCCL INFO Bootstrap : Using eth0:10.244.80.162<0> | |
| nccl-alltoall-job-oguz-mpiworker-4:53:53 [7] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-4:53:53 [7] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-4:51:51 [5] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-20:53:53 [7] NCCL INFO Bootstrap : Using eth0:10.244.124.65<0> | |
| nccl-alltoall-job-oguz-mpiworker-20:53:53 [7] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-20:53:53 [7] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-4:51:51 [5] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-21:47:47 [1] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-21:47:47 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-4:51:51 [5] NCCL INFO Bootstrap : Using eth0:10.244.80.162<0> | |
| nccl-alltoall-job-oguz-mpiworker-4:51:51 [5] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-4:51:51 [5] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-21:47:47 [1] NCCL INFO Bootstrap : Using eth0:10.244.149.199<0> | |
| nccl-alltoall-job-oguz-mpiworker-21:47:47 [1] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-21:47:47 [1] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-26:48:48 [2] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-26:48:48 [2] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-4:52:52 [6] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-26:51:51 [5] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-26:51:51 [5] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-4:52:52 [6] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-12:47:47 [1] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-26:52:52 [6] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-12:47:47 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-26:52:52 [6] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-4:52:52 [6] NCCL INFO Bootstrap : Using eth0:10.244.80.162<0> | |
| nccl-alltoall-job-oguz-mpiworker-4:52:52 [6] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-4:52:52 [6] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-12:47:47 [1] NCCL INFO Bootstrap : Using eth0:10.244.95.47<0> | |
| nccl-alltoall-job-oguz-mpiworker-12:47:47 [1] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-12:47:47 [1] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-26:48:48 [2] NCCL INFO Bootstrap : Using eth0:10.244.157.211<0> | |
| nccl-alltoall-job-oguz-mpiworker-26:48:48 [2] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-26:48:48 [2] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-26:51:51 [5] NCCL INFO Bootstrap : Using eth0:10.244.157.211<0> | |
| nccl-alltoall-job-oguz-mpiworker-26:51:51 [5] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-26:52:52 [6] NCCL INFO Bootstrap : Using eth0:10.244.157.211<0> | |
| nccl-alltoall-job-oguz-mpiworker-26:52:52 [6] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-26:51:51 [5] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-26:52:52 [6] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-20:52:52 [6] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-20:52:52 [6] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-21:51:51 [5] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-21:51:51 [5] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-20:46:46 [0] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-20:46:46 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-20:47:47 [1] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-21:51:51 [5] NCCL INFO Bootstrap : Using eth0:10.244.149.199<0> | |
| nccl-alltoall-job-oguz-mpiworker-21:51:51 [5] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-20:47:47 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-21:51:51 [5] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-30:49:49 [3] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-30:49:49 [3] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-21:49:49 [3] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-21:49:49 [3] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-20:49:49 [3] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-20:49:49 [3] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-18:47:47 [1] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-18:47:47 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-20:48:48 [2] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-18:53:53 [7] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-18:53:53 [7] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-20:48:48 [2] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-18:48:48 [2] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-30:49:49 [3] NCCL INFO Bootstrap : Using eth0:10.244.155.168<0> | |
| nccl-alltoall-job-oguz-mpiworker-30:49:49 [3] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-18:48:48 [2] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-21:49:49 [3] NCCL INFO Bootstrap : Using eth0:10.244.149.199<0> | |
| nccl-alltoall-job-oguz-mpiworker-21:49:49 [3] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-30:49:49 [3] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-22:47:47 [1] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-21:49:49 [3] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-22:47:47 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-20:52:52 [6] NCCL INFO Bootstrap : Using eth0:10.244.124.65<0> | |
| nccl-alltoall-job-oguz-mpiworker-20:52:52 [6] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-18:50:50 [4] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-18:50:50 [4] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-20:52:52 [6] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-20:50:50 [4] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-20:50:50 [4] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-30:52:52 [6] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-30:52:52 [6] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-21:50:50 [4] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-18:51:51 [5] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-21:50:50 [4] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-18:51:51 [5] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-28:48:48 [2] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-28:48:48 [2] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-22:47:47 [1] NCCL INFO Bootstrap : Using eth0:10.244.168.242<0> | |
| nccl-alltoall-job-oguz-mpiworker-22:47:47 [1] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-22:47:47 [1] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-28:46:46 [0] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-28:46:46 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-20:46:46 [0] NCCL INFO Bootstrap : Using eth0:10.244.124.65<0> | |
| nccl-alltoall-job-oguz-mpiworker-20:46:46 [0] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-18:49:49 [3] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-30:52:52 [6] NCCL INFO Bootstrap : Using eth0:10.244.155.168<0> | |
| nccl-alltoall-job-oguz-mpiworker-30:52:52 [6] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-21:50:50 [4] NCCL INFO Bootstrap : Using eth0:10.244.149.199<0> | |
| nccl-alltoall-job-oguz-mpiworker-21:50:50 [4] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-18:49:49 [3] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-20:46:46 [0] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-30:52:52 [6] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-21:50:50 [4] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-30:50:50 [4] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-30:50:50 [4] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-20:47:47 [1] NCCL INFO Bootstrap : Using eth0:10.244.124.65<0> | |
| nccl-alltoall-job-oguz-mpiworker-20:47:47 [1] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-28:52:52 [6] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-20:47:47 [1] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-28:52:52 [6] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-30:48:48 [2] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-30:48:48 [2] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-20:49:49 [3] NCCL INFO Bootstrap : Using eth0:10.244.124.65<0> | |
| nccl-alltoall-job-oguz-mpiworker-20:49:49 [3] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-20:49:49 [3] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-20:48:48 [2] NCCL INFO Bootstrap : Using eth0:10.244.124.65<0> | |
| nccl-alltoall-job-oguz-mpiworker-20:48:48 [2] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-20:50:50 [4] NCCL INFO Bootstrap : Using eth0:10.244.124.65<0> | |
| nccl-alltoall-job-oguz-mpiworker-20:50:50 [4] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-20:50:50 [4] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-20:48:48 [2] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-28:48:48 [2] NCCL INFO Bootstrap : Using eth0:10.244.168.119<0> | |
| nccl-alltoall-job-oguz-mpiworker-28:48:48 [2] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-28:48:48 [2] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-30:50:50 [4] NCCL INFO Bootstrap : Using eth0:10.244.155.168<0> | |
| nccl-alltoall-job-oguz-mpiworker-30:50:50 [4] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-30:50:50 [4] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-28:46:46 [0] NCCL INFO Bootstrap : Using eth0:10.244.168.119<0> | |
| nccl-alltoall-job-oguz-mpiworker-28:46:46 [0] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-30:48:48 [2] NCCL INFO Bootstrap : Using eth0:10.244.155.168<0> | |
| nccl-alltoall-job-oguz-mpiworker-30:48:48 [2] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-28:46:46 [0] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-30:48:48 [2] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-28:52:52 [6] NCCL INFO Bootstrap : Using eth0:10.244.168.119<0> | |
| nccl-alltoall-job-oguz-mpiworker-28:52:52 [6] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-28:52:52 [6] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-28:50:50 [4] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-28:50:50 [4] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-18:47:47 [1] NCCL INFO Bootstrap : Using eth0:10.244.148.215<0> | |
| nccl-alltoall-job-oguz-mpiworker-18:47:47 [1] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-18:47:47 [1] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-18:53:53 [7] NCCL INFO Bootstrap : Using eth0:10.244.148.215<0> | |
| nccl-alltoall-job-oguz-mpiworker-18:53:53 [7] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-18:48:48 [2] NCCL INFO Bootstrap : Using eth0:10.244.148.215<0> | |
| nccl-alltoall-job-oguz-mpiworker-18:48:48 [2] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-18:53:53 [7] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-18:48:48 [2] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-18:50:50 [4] NCCL INFO Bootstrap : Using eth0:10.244.148.215<0> | |
| nccl-alltoall-job-oguz-mpiworker-18:50:50 [4] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-18:51:51 [5] NCCL INFO Bootstrap : Using eth0:10.244.148.215<0> | |
| nccl-alltoall-job-oguz-mpiworker-18:51:51 [5] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-18:49:49 [3] NCCL INFO Bootstrap : Using eth0:10.244.148.215<0> | |
| nccl-alltoall-job-oguz-mpiworker-18:49:49 [3] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-18:50:50 [4] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-18:51:51 [5] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-18:49:49 [3] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-28:50:50 [4] NCCL INFO Bootstrap : Using eth0:10.244.168.119<0> | |
| nccl-alltoall-job-oguz-mpiworker-28:50:50 [4] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-28:50:50 [4] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-28:53:53 [7] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-28:53:53 [7] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-14:51:51 [5] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-14:51:51 [5] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-28:53:53 [7] NCCL INFO Bootstrap : Using eth0:10.244.168.119<0> | |
| nccl-alltoall-job-oguz-mpiworker-28:53:53 [7] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-28:53:53 [7] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-28:51:51 [5] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-28:51:51 [5] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-14:51:51 [5] NCCL INFO Bootstrap : Using eth0:10.244.105.20<0> | |
| nccl-alltoall-job-oguz-mpiworker-14:51:51 [5] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-14:51:51 [5] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-28:51:51 [5] NCCL INFO Bootstrap : Using eth0:10.244.168.119<0> | |
| nccl-alltoall-job-oguz-mpiworker-28:51:51 [5] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-28:51:51 [5] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-9:47:47 [1] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-9:47:47 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-28:49:49 [3] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-28:49:49 [3] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-9:47:47 [1] NCCL INFO Bootstrap : Using eth0:10.244.107.96<0> | |
| nccl-alltoall-job-oguz-mpiworker-9:47:47 [1] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-9:47:47 [1] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-28:49:49 [3] NCCL INFO Bootstrap : Using eth0:10.244.168.119<0> | |
| nccl-alltoall-job-oguz-mpiworker-28:49:49 [3] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-28:49:49 [3] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-8:53:53 [7] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-8:53:53 [7] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-9:46:46 [0] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-9:46:46 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-8:53:53 [7] NCCL INFO Bootstrap : Using eth0:10.244.147.49<0> | |
| nccl-alltoall-job-oguz-mpiworker-8:53:53 [7] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-8:53:53 [7] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-9:46:46 [0] NCCL INFO Bootstrap : Using eth0:10.244.107.96<0> | |
| nccl-alltoall-job-oguz-mpiworker-9:46:46 [0] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-9:46:46 [0] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-14:53:53 [7] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-14:53:53 [7] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-14:53:53 [7] NCCL INFO Bootstrap : Using eth0:10.244.105.20<0> | |
| nccl-alltoall-job-oguz-mpiworker-14:53:53 [7] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-14:53:53 [7] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-3:48:48 [2] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-3:48:48 [2] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-22:46:46 [0] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-22:46:46 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-3:48:48 [2] NCCL INFO Bootstrap : Using eth0:10.244.129.187<0> | |
| nccl-alltoall-job-oguz-mpiworker-3:48:48 [2] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-3:48:48 [2] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-22:46:46 [0] NCCL INFO Bootstrap : Using eth0:10.244.168.242<0> | |
| nccl-alltoall-job-oguz-mpiworker-22:46:46 [0] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-22:46:46 [0] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-22:50:50 [4] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-22:50:50 [4] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-14:49:49 [3] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-14:49:49 [3] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-22:51:51 [5] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-22:51:51 [5] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-14:49:49 [3] NCCL INFO Bootstrap : Using eth0:10.244.105.20<0> | |
| nccl-alltoall-job-oguz-mpiworker-14:49:49 [3] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-14:49:49 [3] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-31:48:48 [2] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-31:48:48 [2] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-8:46:46 [0] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-8:46:46 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-22:50:50 [4] NCCL INFO Bootstrap : Using eth0:10.244.168.242<0> | |
| nccl-alltoall-job-oguz-mpiworker-22:50:50 [4] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-22:50:50 [4] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-22:51:51 [5] NCCL INFO Bootstrap : Using eth0:10.244.168.242<0> | |
| nccl-alltoall-job-oguz-mpiworker-22:51:51 [5] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-22:51:51 [5] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-8:46:46 [0] NCCL INFO Bootstrap : Using eth0:10.244.147.49<0> | |
| nccl-alltoall-job-oguz-mpiworker-8:46:46 [0] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-8:46:46 [0] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-31:48:48 [2] NCCL INFO Bootstrap : Using eth0:10.244.146.72<0> | |
| nccl-alltoall-job-oguz-mpiworker-31:48:48 [2] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-31:48:48 [2] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-31:53:53 [7] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-31:53:53 [7] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-31:47:47 [1] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-31:47:47 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-8:47:47 [1] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-8:47:47 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-18:52:52 [6] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-18:52:52 [6] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-8:47:47 [1] NCCL INFO Bootstrap : Using eth0:10.244.147.49<0> | |
| nccl-alltoall-job-oguz-mpiworker-8:47:47 [1] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-8:47:47 [1] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-19:53:53 [7] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-19:53:53 [7] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-18:52:52 [6] NCCL INFO Bootstrap : Using eth0:10.244.148.215<0> | |
| nccl-alltoall-job-oguz-mpiworker-18:52:52 [6] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-18:52:52 [6] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-31:53:53 [7] NCCL INFO Bootstrap : Using eth0:10.244.146.72<0> | |
| nccl-alltoall-job-oguz-mpiworker-31:53:53 [7] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-31:47:47 [1] NCCL INFO Bootstrap : Using eth0:10.244.146.72<0> | |
| nccl-alltoall-job-oguz-mpiworker-31:47:47 [1] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-22:53:53 [7] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-22:53:53 [7] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-31:53:53 [7] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-31:47:47 [1] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-8:50:50 [4] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-8:50:50 [4] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-6:53:53 [7] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-6:53:53 [7] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-19:47:47 [1] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-19:47:47 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-22:53:53 [7] NCCL INFO Bootstrap : Using eth0:10.244.168.242<0> | |
| nccl-alltoall-job-oguz-mpiworker-22:53:53 [7] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-31:52:52 [6] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-31:52:52 [6] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-22:53:53 [7] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-22:52:52 [6] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-8:50:50 [4] NCCL INFO Bootstrap : Using eth0:10.244.147.49<0> | |
| nccl-alltoall-job-oguz-mpiworker-8:50:50 [4] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-22:52:52 [6] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-31:46:46 [0] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-31:46:46 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-8:50:50 [4] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-5:50:50 [4] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-22:48:48 [2] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-5:50:50 [4] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-22:48:48 [2] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-19:53:53 [7] NCCL INFO Bootstrap : Using eth0:10.244.117.132<0> | |
| nccl-alltoall-job-oguz-mpiworker-19:53:53 [7] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-14:46:46 [0] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-14:46:46 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-19:53:53 [7] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-6:53:53 [7] NCCL INFO Bootstrap : Using eth0:10.244.154.237<0> | |
| nccl-alltoall-job-oguz-mpiworker-6:53:53 [7] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-6:53:53 [7] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-19:47:47 [1] NCCL INFO Bootstrap : Using eth0:10.244.117.132<0> | |
| nccl-alltoall-job-oguz-mpiworker-19:47:47 [1] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-14:47:47 [1] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-19:47:47 [1] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-14:47:47 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-22:49:49 [3] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-5:49:49 [3] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-22:49:49 [3] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-5:50:50 [4] NCCL INFO Bootstrap : Using eth0:10.244.169.8<0> | |
| nccl-alltoall-job-oguz-mpiworker-5:50:50 [4] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-5:49:49 [3] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-5:50:50 [4] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-31:49:49 [3] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-31:49:49 [3] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-8:48:48 [2] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-8:48:48 [2] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-31:52:52 [6] NCCL INFO Bootstrap : Using eth0:10.244.146.72<0> | |
| nccl-alltoall-job-oguz-mpiworker-31:52:52 [6] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-31:52:52 [6] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-31:46:46 [0] NCCL INFO Bootstrap : Using eth0:10.244.146.72<0> | |
| nccl-alltoall-job-oguz-mpiworker-31:46:46 [0] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-22:52:52 [6] NCCL INFO Bootstrap : Using eth0:10.244.168.242<0> | |
| nccl-alltoall-job-oguz-mpiworker-22:52:52 [6] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-31:46:46 [0] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-22:52:52 [6] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-22:48:48 [2] NCCL INFO Bootstrap : Using eth0:10.244.168.242<0> | |
| nccl-alltoall-job-oguz-mpiworker-22:48:48 [2] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-5:49:49 [3] NCCL INFO Bootstrap : Using eth0:10.244.169.8<0> | |
| nccl-alltoall-job-oguz-mpiworker-5:49:49 [3] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-31:51:51 [5] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-31:51:51 [5] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-22:48:48 [2] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-5:49:49 [3] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-22:49:49 [3] NCCL INFO Bootstrap : Using eth0:10.244.168.242<0> | |
| nccl-alltoall-job-oguz-mpiworker-22:49:49 [3] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-8:48:48 [2] NCCL INFO Bootstrap : Using eth0:10.244.147.49<0> | |
| nccl-alltoall-job-oguz-mpiworker-8:48:48 [2] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-22:49:49 [3] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-8:51:51 [5] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-3:49:49 [3] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-8:48:48 [2] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-8:51:51 [5] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-3:49:49 [3] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-14:46:46 [0] NCCL INFO Bootstrap : Using eth0:10.244.105.20<0> | |
| nccl-alltoall-job-oguz-mpiworker-14:46:46 [0] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-14:47:47 [1] NCCL INFO Bootstrap : Using eth0:10.244.105.20<0> | |
| nccl-alltoall-job-oguz-mpiworker-14:47:47 [1] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-14:46:46 [0] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-14:47:47 [1] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-31:50:50 [4] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-31:49:49 [3] NCCL INFO Bootstrap : Using eth0:10.244.146.72<0> | |
| nccl-alltoall-job-oguz-mpiworker-31:49:49 [3] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-31:50:50 [4] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-31:49:49 [3] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-12:49:49 [3] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-12:49:49 [3] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-8:51:51 [5] NCCL INFO Bootstrap : Using eth0:10.244.147.49<0> | |
| nccl-alltoall-job-oguz-mpiworker-8:51:51 [5] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-8:51:51 [5] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-3:49:49 [3] NCCL INFO Bootstrap : Using eth0:10.244.129.187<0> | |
| nccl-alltoall-job-oguz-mpiworker-3:49:49 [3] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-6:48:48 [2] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-3:49:49 [3] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-6:48:48 [2] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-12:49:49 [3] NCCL INFO Bootstrap : Using eth0:10.244.95.47<0> | |
| nccl-alltoall-job-oguz-mpiworker-12:49:49 [3] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-31:51:51 [5] NCCL INFO Bootstrap : Using eth0:10.244.146.72<0> | |
| nccl-alltoall-job-oguz-mpiworker-31:51:51 [5] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-12:49:49 [3] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-5:53:53 [7] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-31:51:51 [5] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-5:53:53 [7] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-31:50:50 [4] NCCL INFO Bootstrap : Using eth0:10.244.146.72<0> | |
| nccl-alltoall-job-oguz-mpiworker-31:50:50 [4] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-31:50:50 [4] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-8:52:52 [6] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-8:52:52 [6] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-14:50:50 [4] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-14:50:50 [4] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-14:48:48 [2] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-6:48:48 [2] NCCL INFO Bootstrap : Using eth0:10.244.154.237<0> | |
| nccl-alltoall-job-oguz-mpiworker-6:48:48 [2] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-14:48:48 [2] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-6:48:48 [2] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-5:53:53 [7] NCCL INFO Bootstrap : Using eth0:10.244.169.8<0> | |
| nccl-alltoall-job-oguz-mpiworker-5:53:53 [7] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-8:49:49 [3] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-8:49:49 [3] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-5:53:53 [7] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-14:52:52 [6] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-14:52:52 [6] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-8:52:52 [6] NCCL INFO Bootstrap : Using eth0:10.244.147.49<0> | |
| nccl-alltoall-job-oguz-mpiworker-8:52:52 [6] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-8:52:52 [6] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-5:48:48 [2] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-5:48:48 [2] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-8:49:49 [3] NCCL INFO Bootstrap : Using eth0:10.244.147.49<0> | |
| nccl-alltoall-job-oguz-mpiworker-8:49:49 [3] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-8:49:49 [3] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-5:47:47 [1] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-5:47:47 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-19:48:48 [2] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-19:48:48 [2] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-19:52:52 [6] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-19:52:52 [6] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-14:50:50 [4] NCCL INFO Bootstrap : Using eth0:10.244.105.20<0> | |
| nccl-alltoall-job-oguz-mpiworker-14:50:50 [4] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-14:48:48 [2] NCCL INFO Bootstrap : Using eth0:10.244.105.20<0> | |
| nccl-alltoall-job-oguz-mpiworker-14:48:48 [2] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-14:52:52 [6] NCCL INFO Bootstrap : Using eth0:10.244.105.20<0> | |
| nccl-alltoall-job-oguz-mpiworker-14:52:52 [6] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-5:48:48 [2] NCCL INFO Bootstrap : Using eth0:10.244.169.8<0> | |
| nccl-alltoall-job-oguz-mpiworker-5:48:48 [2] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-14:50:50 [4] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-14:48:48 [2] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-14:52:52 [6] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-5:48:48 [2] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-5:47:47 [1] NCCL INFO Bootstrap : Using eth0:10.244.169.8<0> | |
| nccl-alltoall-job-oguz-mpiworker-5:47:47 [1] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-5:52:52 [6] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-5:47:47 [1] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-5:52:52 [6] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-19:51:51 [5] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-19:51:51 [5] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-5:51:51 [5] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-5:51:51 [5] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-19:48:48 [2] NCCL INFO Bootstrap : Using eth0:10.244.117.132<0> | |
| nccl-alltoall-job-oguz-mpiworker-19:48:48 [2] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-6:47:47 [1] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-6:47:47 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-19:48:48 [2] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-6:52:52 [6] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-6:52:52 [6] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-6:46:46 [0] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-6:46:46 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-19:52:52 [6] NCCL INFO Bootstrap : Using eth0:10.244.117.132<0> | |
| nccl-alltoall-job-oguz-mpiworker-19:52:52 [6] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-6:49:49 [3] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-1:51:51 [5] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-1:51:51 [5] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-19:52:52 [6] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-6:49:49 [3] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-19:51:51 [5] NCCL INFO Bootstrap : Using eth0:10.244.117.132<0> | |
| nccl-alltoall-job-oguz-mpiworker-19:51:51 [5] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-19:51:51 [5] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-5:52:52 [6] NCCL INFO Bootstrap : Using eth0:10.244.169.8<0> | |
| nccl-alltoall-job-oguz-mpiworker-5:52:52 [6] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-5:51:51 [5] NCCL INFO Bootstrap : Using eth0:10.244.169.8<0> | |
| nccl-alltoall-job-oguz-mpiworker-5:51:51 [5] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-6:51:51 [5] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-5:52:52 [6] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-5:51:51 [5] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-2:53:53 [7] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-6:51:51 [5] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-2:53:53 [7] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-1:51:51 [5] NCCL INFO Bootstrap : Using eth0:10.244.130.46<0> | |
| nccl-alltoall-job-oguz-mpiworker-1:51:51 [5] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-2:50:50 [4] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-1:51:51 [5] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-2:50:50 [4] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-19:49:49 [3] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-19:49:49 [3] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-6:50:50 [4] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-6:50:50 [4] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-9:49:49 [3] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-9:49:49 [3] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-19:50:50 [4] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-19:50:50 [4] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-2:53:53 [7] NCCL INFO Bootstrap : Using eth0:10.244.96.180<0> | |
| nccl-alltoall-job-oguz-mpiworker-2:53:53 [7] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-2:50:50 [4] NCCL INFO Bootstrap : Using eth0:10.244.96.180<0> | |
| nccl-alltoall-job-oguz-mpiworker-2:50:50 [4] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-2:53:53 [7] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-2:50:50 [4] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-9:49:49 [3] NCCL INFO Bootstrap : Using eth0:10.244.107.96<0> | |
| nccl-alltoall-job-oguz-mpiworker-9:49:49 [3] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-19:49:49 [3] NCCL INFO Bootstrap : Using eth0:10.244.117.132<0> | |
| nccl-alltoall-job-oguz-mpiworker-19:49:49 [3] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-9:49:49 [3] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-19:49:49 [3] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-19:50:50 [4] NCCL INFO Bootstrap : Using eth0:10.244.117.132<0> | |
| nccl-alltoall-job-oguz-mpiworker-19:50:50 [4] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-19:50:50 [4] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-3:46:46 [0] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-3:46:46 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-6:47:47 [1] NCCL INFO Bootstrap : Using eth0:10.244.154.237<0> | |
| nccl-alltoall-job-oguz-mpiworker-6:47:47 [1] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-6:47:47 [1] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-6:52:52 [6] NCCL INFO Bootstrap : Using eth0:10.244.154.237<0> | |
| nccl-alltoall-job-oguz-mpiworker-6:52:52 [6] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-6:52:52 [6] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-6:46:46 [0] NCCL INFO Bootstrap : Using eth0:10.244.154.237<0> | |
| nccl-alltoall-job-oguz-mpiworker-6:46:46 [0] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-6:46:46 [0] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-3:46:46 [0] NCCL INFO Bootstrap : Using eth0:10.244.129.187<0> | |
| nccl-alltoall-job-oguz-mpiworker-3:46:46 [0] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-6:49:49 [3] NCCL INFO Bootstrap : Using eth0:10.244.154.237<0> | |
| nccl-alltoall-job-oguz-mpiworker-6:49:49 [3] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-3:46:46 [0] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-6:49:49 [3] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-15:46:46 [0] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-15:46:46 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-3:51:51 [5] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-6:51:51 [5] NCCL INFO Bootstrap : Using eth0:10.244.154.237<0> | |
| nccl-alltoall-job-oguz-mpiworker-6:51:51 [5] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-6:50:50 [4] NCCL INFO Bootstrap : Using eth0:10.244.154.237<0> | |
| nccl-alltoall-job-oguz-mpiworker-6:50:50 [4] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-11:51:51 [5] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-11:51:51 [5] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-3:51:51 [5] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-6:51:51 [5] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-6:50:50 [4] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-2:46:46 [0] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-2:46:46 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-11:49:49 [3] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-11:49:49 [3] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-3:51:51 [5] NCCL INFO Bootstrap : Using eth0:10.244.129.187<0> | |
| nccl-alltoall-job-oguz-mpiworker-3:51:51 [5] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-3:51:51 [5] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-2:46:46 [0] NCCL INFO Bootstrap : Using eth0:10.244.96.180<0> | |
| nccl-alltoall-job-oguz-mpiworker-2:46:46 [0] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-2:46:46 [0] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-15:46:46 [0] NCCL INFO Bootstrap : Using eth0:10.244.25.48<0> | |
| nccl-alltoall-job-oguz-mpiworker-15:46:46 [0] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-15:46:46 [0] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-11:46:46 [0] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-11:46:46 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-12:50:50 [4] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-12:50:50 [4] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-11:51:51 [5] NCCL INFO Bootstrap : Using eth0:10.244.113.218<0> | |
| nccl-alltoall-job-oguz-mpiworker-11:51:51 [5] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-11:49:49 [3] NCCL INFO Bootstrap : Using eth0:10.244.113.218<0> | |
| nccl-alltoall-job-oguz-mpiworker-11:49:49 [3] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-11:51:51 [5] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-11:49:49 [3] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-11:46:46 [0] NCCL INFO Bootstrap : Using eth0:10.244.113.218<0> | |
| nccl-alltoall-job-oguz-mpiworker-11:46:46 [0] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-11:46:46 [0] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-11:48:48 [2] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-11:48:48 [2] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-15:48:48 [2] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-15:48:48 [2] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-3:53:53 [7] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-3:53:53 [7] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-12:50:50 [4] NCCL INFO Bootstrap : Using eth0:10.244.95.47<0> | |
| nccl-alltoall-job-oguz-mpiworker-12:50:50 [4] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-12:50:50 [4] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-11:48:48 [2] NCCL INFO Bootstrap : Using eth0:10.244.113.218<0> | |
| nccl-alltoall-job-oguz-mpiworker-11:48:48 [2] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-11:48:48 [2] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-15:48:48 [2] NCCL INFO Bootstrap : Using eth0:10.244.25.48<0> | |
| nccl-alltoall-job-oguz-mpiworker-15:48:48 [2] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-15:48:48 [2] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-3:53:53 [7] NCCL INFO Bootstrap : Using eth0:10.244.129.187<0> | |
| nccl-alltoall-job-oguz-mpiworker-3:53:53 [7] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-3:53:53 [7] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-15:47:47 [1] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-15:47:47 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-2:52:52 [6] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-2:52:52 [6] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-15:51:51 [5] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-15:51:51 [5] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-15:50:50 [4] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-15:50:50 [4] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-2:52:52 [6] NCCL INFO Bootstrap : Using eth0:10.244.96.180<0> | |
| nccl-alltoall-job-oguz-mpiworker-2:52:52 [6] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-3:50:50 [4] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-2:51:51 [5] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-3:50:50 [4] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-2:51:51 [5] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-2:52:52 [6] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-3:52:52 [6] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-3:52:52 [6] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-11:52:52 [6] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-10:46:230 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-11:52:52 [6] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-2:51:51 [5] NCCL INFO Bootstrap : Using eth0:10.244.96.180<0> | |
| nccl-alltoall-job-oguz-mpiworker-2:51:51 [5] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-2:51:51 [5] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-15:49:49 [3] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-15:52:52 [6] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-15:49:49 [3] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-15:52:52 [6] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-15:47:47 [1] NCCL INFO Bootstrap : Using eth0:10.244.25.48<0> | |
| nccl-alltoall-job-oguz-mpiworker-15:47:47 [1] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-15:47:47 [1] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-11:52:52 [6] NCCL INFO Bootstrap : Using eth0:10.244.113.218<0> | |
| nccl-alltoall-job-oguz-mpiworker-11:52:52 [6] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-10:46:230 [0] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-3:50:50 [4] NCCL INFO Bootstrap : Using eth0:10.244.129.187<0> | |
| nccl-alltoall-job-oguz-mpiworker-3:50:50 [4] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-11:52:52 [6] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-15:53:53 [7] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-15:53:53 [7] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-3:50:50 [4] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-3:52:52 [6] NCCL INFO Bootstrap : Using eth0:10.244.129.187<0> | |
| nccl-alltoall-job-oguz-mpiworker-3:52:52 [6] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-3:52:52 [6] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-15:51:51 [5] NCCL INFO Bootstrap : Using eth0:10.244.25.48<0> | |
| nccl-alltoall-job-oguz-mpiworker-15:51:51 [5] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-15:51:51 [5] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-2:47:47 [1] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-2:47:47 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-2:47:47 [1] NCCL INFO Bootstrap : Using eth0:10.244.96.180<0> | |
| nccl-alltoall-job-oguz-mpiworker-2:47:47 [1] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-15:50:50 [4] NCCL INFO Bootstrap : Using eth0:10.244.25.48<0> | |
| nccl-alltoall-job-oguz-mpiworker-15:50:50 [4] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-2:47:47 [1] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-15:50:50 [4] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-15:49:49 [3] NCCL INFO Bootstrap : Using eth0:10.244.25.48<0> | |
| nccl-alltoall-job-oguz-mpiworker-15:49:49 [3] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-15:52:52 [6] NCCL INFO Bootstrap : Using eth0:10.244.25.48<0> | |
| nccl-alltoall-job-oguz-mpiworker-15:52:52 [6] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-15:53:53 [7] NCCL INFO Bootstrap : Using eth0:10.244.25.48<0> | |
| nccl-alltoall-job-oguz-mpiworker-15:53:53 [7] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-15:49:49 [3] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-15:52:52 [6] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-15:53:53 [7] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-2:49:49 [3] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-2:49:49 [3] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-11:47:47 [1] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-11:47:47 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-1:50:50 [4] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-1:50:50 [4] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-11:50:50 [4] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-11:50:50 [4] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-2:48:48 [2] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-2:48:48 [2] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-11:53:53 [7] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-11:53:53 [7] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-9:48:48 [2] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-1:50:50 [4] NCCL INFO Bootstrap : Using eth0:10.244.130.46<0> | |
| nccl-alltoall-job-oguz-mpiworker-1:50:50 [4] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-9:48:48 [2] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-1:50:50 [4] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-1:54:54 [7] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-1:54:54 [7] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-2:49:49 [3] NCCL INFO Bootstrap : Using eth0:10.244.96.180<0> | |
| nccl-alltoall-job-oguz-mpiworker-2:49:49 [3] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-2:49:49 [3] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-1:52:52 [6] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-1:52:52 [6] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-2:48:48 [2] NCCL INFO Bootstrap : Using eth0:10.244.96.180<0> | |
| nccl-alltoall-job-oguz-mpiworker-2:48:48 [2] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-2:48:48 [2] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-9:48:48 [2] NCCL INFO Bootstrap : Using eth0:10.244.107.96<0> | |
| nccl-alltoall-job-oguz-mpiworker-9:48:48 [2] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-9:48:48 [2] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-1:49:49 [3] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-1:49:49 [3] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-11:47:47 [1] NCCL INFO Bootstrap : Using eth0:10.244.113.218<0> | |
| nccl-alltoall-job-oguz-mpiworker-11:47:47 [1] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-11:50:50 [4] NCCL INFO Bootstrap : Using eth0:10.244.113.218<0> | |
| nccl-alltoall-job-oguz-mpiworker-11:50:50 [4] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-11:53:53 [7] NCCL INFO Bootstrap : Using eth0:10.244.113.218<0> | |
| nccl-alltoall-job-oguz-mpiworker-11:53:53 [7] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-11:47:47 [1] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-11:50:50 [4] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-11:53:53 [7] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-1:47:47 [1] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-1:47:47 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-1:54:54 [7] NCCL INFO Bootstrap : Using eth0:10.244.130.46<0> | |
| nccl-alltoall-job-oguz-mpiworker-1:54:54 [7] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-1:54:54 [7] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-1:52:52 [6] NCCL INFO Bootstrap : Using eth0:10.244.130.46<0> | |
| nccl-alltoall-job-oguz-mpiworker-1:52:52 [6] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-1:52:52 [6] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-1:49:49 [3] NCCL INFO Bootstrap : Using eth0:10.244.130.46<0> | |
| nccl-alltoall-job-oguz-mpiworker-1:49:49 [3] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-1:47:47 [1] NCCL INFO Bootstrap : Using eth0:10.244.130.46<0> | |
| nccl-alltoall-job-oguz-mpiworker-1:47:47 [1] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-1:49:49 [3] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-1:47:47 [1] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-9:52:52 [6] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-9:52:52 [6] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-9:53:53 [7] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-9:53:53 [7] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-9:52:52 [6] NCCL INFO Bootstrap : Using eth0:10.244.107.96<0> | |
| nccl-alltoall-job-oguz-mpiworker-9:52:52 [6] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-9:52:52 [6] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-12:54:54 [7] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-12:54:54 [7] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-12:51:51 [5] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-12:51:51 [5] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-9:53:53 [7] NCCL INFO Bootstrap : Using eth0:10.244.107.96<0> | |
| nccl-alltoall-job-oguz-mpiworker-9:53:53 [7] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-12:48:48 [2] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-9:53:53 [7] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-12:48:48 [2] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-12:52:52 [6] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-12:52:52 [6] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-12:46:46 [0] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-12:46:46 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-9:51:51 [5] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-9:51:51 [5] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-9:50:50 [4] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-9:50:50 [4] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-12:54:54 [7] NCCL INFO Bootstrap : Using eth0:10.244.95.47<0> | |
| nccl-alltoall-job-oguz-mpiworker-12:54:54 [7] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-17:48:48 [2] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-17:48:48 [2] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-12:54:54 [7] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-12:51:51 [5] NCCL INFO Bootstrap : Using eth0:10.244.95.47<0> | |
| nccl-alltoall-job-oguz-mpiworker-12:51:51 [5] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-12:48:48 [2] NCCL INFO Bootstrap : Using eth0:10.244.95.47<0> | |
| nccl-alltoall-job-oguz-mpiworker-12:48:48 [2] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-12:52:52 [6] NCCL INFO Bootstrap : Using eth0:10.244.95.47<0> | |
| nccl-alltoall-job-oguz-mpiworker-12:52:52 [6] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-12:46:46 [0] NCCL INFO Bootstrap : Using eth0:10.244.95.47<0> | |
| nccl-alltoall-job-oguz-mpiworker-12:46:46 [0] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-12:51:51 [5] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-12:48:48 [2] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-12:52:52 [6] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-12:46:46 [0] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-9:51:51 [5] NCCL INFO Bootstrap : Using eth0:10.244.107.96<0> | |
| nccl-alltoall-job-oguz-mpiworker-9:51:51 [5] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-9:50:50 [4] NCCL INFO Bootstrap : Using eth0:10.244.107.96<0> | |
| nccl-alltoall-job-oguz-mpiworker-9:50:50 [4] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-9:51:51 [5] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-9:50:50 [4] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-17:48:48 [2] NCCL INFO Bootstrap : Using eth0:10.244.133.20<0> | |
| nccl-alltoall-job-oguz-mpiworker-17:48:48 [2] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-17:48:48 [2] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-17:47:47 [1] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-17:47:47 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-17:51:51 [5] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-17:51:51 [5] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-17:49:49 [3] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-17:49:49 [3] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-17:46:46 [0] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-17:46:46 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-17:50:50 [4] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-17:50:50 [4] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-17:53:53 [7] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-17:53:53 [7] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-17:52:52 [6] NCCL INFO cudaDriverVersion 12020 | |
| nccl-alltoall-job-oguz-mpiworker-17:52:52 [6] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-17:47:47 [1] NCCL INFO Bootstrap : Using eth0:10.244.133.20<0> | |
| nccl-alltoall-job-oguz-mpiworker-17:47:47 [1] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-17:47:47 [1] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-17:51:51 [5] NCCL INFO Bootstrap : Using eth0:10.244.133.20<0> | |
| nccl-alltoall-job-oguz-mpiworker-17:51:51 [5] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-17:51:51 [5] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-17:49:49 [3] NCCL INFO Bootstrap : Using eth0:10.244.133.20<0> | |
| nccl-alltoall-job-oguz-mpiworker-17:49:49 [3] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-17:49:49 [3] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-17:46:46 [0] NCCL INFO Bootstrap : Using eth0:10.244.133.20<0> | |
| nccl-alltoall-job-oguz-mpiworker-17:46:46 [0] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-17:50:50 [4] NCCL INFO Bootstrap : Using eth0:10.244.133.20<0> | |
| nccl-alltoall-job-oguz-mpiworker-17:50:50 [4] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-17:53:53 [7] NCCL INFO Bootstrap : Using eth0:10.244.133.20<0> | |
| nccl-alltoall-job-oguz-mpiworker-17:53:53 [7] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-17:52:52 [6] NCCL INFO Bootstrap : Using eth0:10.244.133.20<0> | |
| nccl-alltoall-job-oguz-mpiworker-17:52:52 [6] NCCL INFO Plugin name set by env to libnccl-net-none.so | |
| nccl-alltoall-job-oguz-mpiworker-17:46:46 [0] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-17:50:50 [4] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-17:53:53 [7] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-17:52:52 [6] NCCL INFO NET/Plugin : dlerror=libnccl-net-none.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net-none.so), using internal implementation | |
| nccl-alltoall-job-oguz-mpiworker-24:48:230 [2] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-24:48:230 [2] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-20:51:230 [5] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-20:51:230 [5] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-7:46:230 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-7:46:230 [0] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-26:46:231 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-26:47:230 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-26:47:230 [1] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-26:46:231 [0] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-10:48:231 [2] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-10:48:231 [2] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-21:46:230 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-7:52:231 [6] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-21:46:230 [0] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-7:52:231 [6] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-30:46:230 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-30:46:230 [0] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-7:51:232 [5] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-27:52:230 [6] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-7:51:232 [5] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-25:46:230 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-27:52:230 [6] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-25:46:230 [0] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-7:50:234 [4] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-7:47:233 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-7:50:234 [4] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-7:47:233 [1] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-7:48:237 [2] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-7:54:236 [7] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-16:46:230 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-0:46:231 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-7:49:235 [3] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-16:46:230 [0] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-0:46:231 [0] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-7:48:237 [2] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-7:54:236 [7] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-7:49:235 [3] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-5:46:230 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-5:46:230 [0] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-10:52:233 [6] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-10:51:232 [5] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-10:50:234 [4] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-10:52:233 [6] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-10:51:232 [5] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-10:50:234 [4] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-24:47:231 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-19:46:230 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-10:53:235 [7] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-10:47:236 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-24:47:231 [1] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-19:46:230 [0] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-10:53:235 [7] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-10:47:236 [1] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-27:46:231 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-27:46:231 [0] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-10:49:237 [3] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-10:49:237 [3] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-13:48:230 [2] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-27:50:232 [4] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-13:48:230 [2] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-27:50:232 [4] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-29:51:230 [5] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-13:47:231 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-29:51:230 [5] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-0:51:232 [5] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-30:53:231 [7] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-13:47:231 [1] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-29:48:231 [2] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-23:46:230 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-0:51:232 [5] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-30:53:231 [7] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-24:52:232 [6] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-29:48:231 [2] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-23:46:230 [0] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-24:52:232 [6] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-30:47:232 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-30:47:232 [1] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-16:48:231 [2] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-16:50:232 [4] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-16:48:231 [2] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-16:50:232 [4] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-16:49:233 [3] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-16:49:233 [3] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-16:51:235 [5] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-16:47:234 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-16:51:235 [5] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-25:53:231 [7] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-16:47:234 [1] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-16:53:236 [7] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-25:53:231 [7] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-16:53:236 [7] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-29:46:232 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-26:53:232 [7] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-29:46:232 [0] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-26:53:232 [7] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-27:51:234 [5] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-27:51:234 [5] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-4:47:232 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-27:53:237 [7] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-27:48:235 [2] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-27:49:233 [3] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-16:52:237 [6] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-4:47:232 [1] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-4:50:230 [4] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-27:47:236 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-16:52:237 [6] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-4:50:230 [4] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-1:46:230 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-1:48:231 [2] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-27:53:237 [7] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-27:48:235 [2] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-27:49:233 [3] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-27:47:236 [1] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-4:46:231 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-29:47:233 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-4:46:231 [0] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-1:46:230 [0] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-1:48:231 [2] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-23:53:231 [7] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-23:48:232 [2] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-29:47:233 [1] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-3:47:230 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-23:48:232 [2] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-23:53:231 [7] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-3:47:230 [1] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-29:53:235 [7] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-29:52:237 [6] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-25:51:232 [5] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-29:49:236 [3] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-24:51:234 [5] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-25:51:232 [5] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-24:51:234 [5] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-24:53:235 [7] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-24:49:233 [3] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-23:50:234 [4] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-29:53:235 [7] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-29:49:236 [3] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-29:52:237 [6] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-24:53:235 [7] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-24:49:233 [3] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-23:51:235 [5] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-23:50:234 [4] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-13:46:235 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-23:47:233 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-23:51:235 [5] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-13:46:235 [0] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-23:47:233 [1] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-23:49:236 [3] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-23:49:236 [3] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-0:53:234 [7] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-23:52:237 [6] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-13:52:233 [6] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-0:53:234 [7] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-13:50:236 [4] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-0:48:233 [2] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-23:52:237 [6] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-13:52:233 [6] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-0:48:233 [2] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-13:50:236 [4] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-30:51:233 [5] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-13:51:237 [5] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-13:53:232 [7] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-13:49:234 [3] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-25:47:234 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-30:51:233 [5] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-0:52:235 [6] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-25:47:234 [1] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-26:49:233 [3] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-12:47:230 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-0:52:235 [6] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-26:49:233 [3] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-18:46:230 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-12:47:230 [1] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-26:50:234 [4] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-25:49:233 [3] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-25:50:235 [4] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-13:51:237 [5] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-13:53:232 [7] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-13:49:234 [3] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-18:46:230 [0] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-26:50:234 [4] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-25:48:237 [2] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-21:48:232 [2] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-21:52:231 [6] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-0:49:236 [3] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-0:50:238 [4] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-25:49:233 [3] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-25:50:235 [4] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-25:48:237 [2] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-26:51:236 [5] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-21:48:232 [2] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-21:52:231 [6] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-0:50:238 [4] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-0:49:236 [3] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-26:51:236 [5] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-0:47:237 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-26:48:235 [2] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-0:47:237 [1] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-26:48:235 [2] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-20:53:231 [7] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-29:50:234 [4] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-26:52:237 [6] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-20:53:231 [7] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-29:50:234 [4] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-26:52:237 [6] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-21:53:233 [7] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-21:53:233 [7] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-4:53:235 [7] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-30:48:237 [2] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-30:52:235 [6] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-4:53:235 [7] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-21:50:237 [4] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-30:49:234 [3] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-21:50:237 [4] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-4:51:236 [5] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-4:48:233 [2] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-30:48:237 [2] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-20:52:232 [6] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-20:49:235 [3] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-21:51:235 [5] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-30:52:235 [6] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-4:49:234 [3] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-30:49:234 [3] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-30:50:236 [4] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-21:51:235 [5] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-22:47:230 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-21:47:234 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-21:49:236 [3] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-30:50:236 [4] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-20:52:232 [6] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-20:49:235 [3] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-4:51:236 [5] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-4:48:233 [2] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-4:49:234 [3] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-22:47:230 [1] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-25:52:236 [6] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-28:47:230 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-20:47:234 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-20:46:233 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-4:52:237 [6] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-21:49:236 [3] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-21:47:234 [1] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-25:52:236 [6] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-28:47:230 [1] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-4:52:237 [6] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-9:47:230 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-20:47:234 [1] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-20:46:233 [0] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-20:48:236 [2] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-9:47:230 [1] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-20:48:236 [2] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-28:48:231 [2] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-28:48:231 [2] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-28:46:233 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-28:46:233 [0] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-28:50:234 [4] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-28:50:234 [4] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-14:51:230 [5] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-28:52:232 [6] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-8:53:230 [7] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-14:51:230 [5] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-28:52:232 [6] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-8:53:230 [7] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-3:48:231 [2] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-18:48:233 [2] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-9:46:231 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-3:48:231 [2] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-18:48:233 [2] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-28:53:235 [7] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-18:51:235 [5] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-14:53:231 [7] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-9:46:231 [0] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-28:53:235 [7] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-28:51:236 [5] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-14:53:231 [7] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-18:51:235 [5] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-28:51:236 [5] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-18:53:231 [7] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-8:46:231 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-18:50:236 [4] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-18:47:232 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-8:46:231 [0] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-22:46:231 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-22:50:232 [4] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-22:51:233 [5] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-22:52:235 [6] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-22:53:234 [7] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-22:48:236 [2] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-18:53:231 [7] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-14:49:232 [3] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-18:50:236 [4] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-18:47:232 [1] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-24:46:237 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-20:50:237 [4] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-24:50:236 [4] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-6:53:230 [7] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-14:49:232 [3] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-28:49:237 [3] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-20:50:237 [4] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-6:53:230 [7] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-8:47:232 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-28:49:237 [3] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-24:46:237 [0] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-18:49:234 [3] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-24:50:236 [4] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-8:47:232 [1] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-18:49:234 [3] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-31:47:232 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-31:53:231 [7] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-6:48:231 [2] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-22:49:237 [3] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-18:52:237 [6] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-8:51:234 [5] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-22:46:231 [0] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-22:51:233 [5] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-22:50:232 [4] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-22:52:235 [6] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-5:50:231 [4] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-22:53:234 [7] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-22:48:236 [2] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-6:48:231 [2] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-18:52:237 [6] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-31:47:232 [1] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-31:53:231 [7] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-22:49:237 [3] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-8:51:234 [5] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-5:50:231 [4] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-31:51:236 [5] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-8:50:233 [4] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-31:46:234 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-31:49:235 [3] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-12:49:231 [3] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-8:50:233 [4] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-3:49:232 [3] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-8:48:235 [2] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-5:49:232 [3] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-19:53:231 [7] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-31:51:236 [5] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-12:49:231 [3] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-3:49:232 [3] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-31:52:233 [6] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-8:48:235 [2] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-14:47:233 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-19:48:233 [2] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-19:47:232 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-19:52:234 [6] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-5:49:232 [3] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-31:46:234 [0] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-31:49:235 [3] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-14:47:233 [1] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-31:52:233 [6] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-8:52:236 [6] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-10:46:230 [0] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.150.6<0> | |
| nccl-alltoall-job-oguz-mpiworker-10:46:230 [0] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-10:46:230 [0] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-8:49:237 [3] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-19:53:231 [7] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-5:53:233 [7] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-5:48:234 [2] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-14:46:234 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-19:52:234 [6] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-19:47:232 [1] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-19:48:233 [2] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-8:52:236 [6] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-8:49:237 [3] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-3:51:234 [5] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-14:46:234 [0] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-5:53:233 [7] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-5:48:234 [2] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-31:48:230 [2] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-3:51:234 [5] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-31:48:230 [2] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-14:48:237 [2] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-14:48:237 [2] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-9:49:232 [3] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-14:50:236 [4] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-3:46:233 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-2:53:230 [7] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-9:49:232 [3] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-12:50:232 [4] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-14:50:236 [4] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-2:50:231 [4] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-6:46:236 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-3:46:233 [0] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-6:51:235 [5] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-19:51:235 [5] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-19:50:237 [4] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-6:50:237 [4] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-12:50:232 [4] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-6:52:233 [6] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-14:52:235 [6] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-2:53:230 [7] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-5:51:236 [5] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-19:49:236 [3] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-5:52:237 [6] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-2:50:231 [4] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-1:51:232 [5] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-3:53:235 [7] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-14:52:235 [6] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-6:49:234 [3] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-1:51:232 [5] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-3:53:235 [7] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-19:50:237 [4] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-19:51:235 [5] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-6:46:236 [0] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-19:49:236 [3] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-5:51:236 [5] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-5:52:237 [6] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-6:47:232 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-6:51:235 [5] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-6:50:237 [4] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-5:47:235 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-6:52:233 [6] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-2:46:232 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-6:49:234 [3] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-6:47:232 [1] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-5:47:235 [1] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-2:46:232 [0] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-2:51:234 [5] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-31:50:237 [4] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-15:48:231 [2] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-2:51:234 [5] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-31:50:237 [4] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-15:48:231 [2] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-1:52:235 [6] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-2:52:233 [6] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-1:52:235 [6] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-3:50:236 [4] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-1:54:234 [7] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-3:52:237 [6] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-2:47:235 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-2:52:233 [6] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-1:54:234 [7] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-1:49:236 [3] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-2:47:235 [1] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-3:50:236 [4] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-3:52:237 [6] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-1:49:236 [3] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-1:47:237 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-15:46:230 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-2:48:237 [2] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-1:47:237 [1] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-15:46:230 [0] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-11:46:230 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-2:48:237 [2] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-2:49:236 [3] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-15:51:233 [5] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-11:46:230 [0] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-15:50:234 [4] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-15:47:232 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-2:49:236 [3] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-11:48:231 [2] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-15:51:233 [5] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-15:53:236 [7] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-9:48:233 [2] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-11:51:234 [5] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-11:48:231 [2] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-15:50:234 [4] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-1:50:233 [4] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-15:47:232 [1] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-15:53:236 [7] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-9:48:233 [2] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-11:51:234 [5] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-15:49:235 [3] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-1:50:233 [4] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-12:54:233 [7] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-12:48:236 [2] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-12:51:234 [5] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-15:49:235 [3] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-11:53:236 [7] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-15:52:237 [6] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-11:49:232 [3] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-11:52:233 [6] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-11:47:235 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-15:52:237 [6] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-12:54:233 [7] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-12:51:234 [5] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-12:48:236 [2] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-12:52:235 [6] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-12:46:237 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-11:53:236 [7] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-9:53:235 [7] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-11:50:237 [4] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-12:52:235 [6] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-12:46:237 [0] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-9:53:235 [7] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-11:49:232 [3] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-11:52:233 [6] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-11:47:235 [1] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-11:50:237 [4] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-7:54:236 [7] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.115.133<0> | |
| nccl-alltoall-job-oguz-mpiworker-9:51:236 [5] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-7:54:236 [7] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-7:54:236 [7] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-9:51:236 [5] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-29:47:233 [1] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.90.183<0> | |
| nccl-alltoall-job-oguz-mpiworker-29:47:233 [1] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-29:47:233 [1] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-9:50:237 [4] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-17:48:230 [2] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-9:50:237 [4] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-17:48:230 [2] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-9:52:234 [6] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-9:52:234 [6] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-7:47:233 [1] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.115.133<0> | |
| nccl-alltoall-job-oguz-mpiworker-7:49:235 [3] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.115.133<0> | |
| nccl-alltoall-job-oguz-mpiworker-7:47:233 [1] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-7:47:233 [1] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-7:49:235 [3] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-7:49:235 [3] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-17:46:234 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-17:46:234 [0] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-10:49:237 [3] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.150.6<0> | |
| nccl-alltoall-job-oguz-mpiworker-10:49:237 [3] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-10:49:237 [3] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-17:51:232 [5] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-17:51:232 [5] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-17:50:235 [4] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-0:51:232 [5] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.72.84<0> | |
| nccl-alltoall-job-oguz-mpiworker-0:51:232 [5] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-0:51:232 [5] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-17:50:235 [4] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-10:53:235 [7] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.150.6<0> | |
| nccl-alltoall-job-oguz-mpiworker-17:53:236 [7] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-10:53:235 [7] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-10:53:235 [7] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-17:47:231 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-17:53:236 [7] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-17:47:231 [1] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-17:52:237 [6] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-17:52:237 [6] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-17:49:233 [3] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| nccl-alltoall-job-oguz-mpiworker-17:49:233 [3] NCCL INFO NCCL_IB_HCA set to mlx5 | |
| nccl-alltoall-job-oguz-mpiworker-24:48:230 [2] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.172.120<0> | |
| nccl-alltoall-job-oguz-mpiworker-24:48:230 [2] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-24:48:230 [2] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-21:46:230 [0] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.149.199<0> | |
| nccl-alltoall-job-oguz-mpiworker-21:46:230 [0] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-21:46:230 [0] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-27:47:236 [1] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.144.145<0> | |
| nccl-alltoall-job-oguz-mpiworker-27:47:236 [1] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-27:47:236 [1] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-10:50:234 [4] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.150.6<0> | |
| nccl-alltoall-job-oguz-mpiworker-10:50:234 [4] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-10:50:234 [4] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-7:51:232 [5] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.115.133<0> | |
| nccl-alltoall-job-oguz-mpiworker-7:51:232 [5] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-7:51:232 [5] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-27:49:233 [3] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.144.145<0> | |
| nccl-alltoall-job-oguz-mpiworker-27:49:233 [3] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-27:49:233 [3] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-10:51:232 [5] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.150.6<0> | |
| nccl-alltoall-job-oguz-mpiworker-10:51:232 [5] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-10:51:232 [5] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-13:52:233 [6] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.125.171<0> | |
| nccl-alltoall-job-oguz-mpiworker-13:52:233 [6] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-13:52:233 [6] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-26:53:232 [7] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.157.211<0> | |
| nccl-alltoall-job-oguz-mpiworker-26:53:232 [7] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-26:53:232 [7] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-7:48:237 [2] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.115.133<0> | |
| nccl-alltoall-job-oguz-mpiworker-7:48:237 [2] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-7:48:237 [2] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-27:51:234 [5] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.144.145<0> | |
| nccl-alltoall-job-oguz-mpiworker-27:51:234 [5] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-27:51:234 [5] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-13:50:236 [4] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.125.171<0> | |
| nccl-alltoall-job-oguz-mpiworker-13:50:236 [4] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-13:50:236 [4] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-29:49:236 [3] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.90.183<0> | |
| nccl-alltoall-job-oguz-mpiworker-29:49:236 [3] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-29:49:236 [3] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-10:47:236 [1] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.150.6<0> | |
| nccl-alltoall-job-oguz-mpiworker-10:47:236 [1] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-10:47:236 [1] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-24:47:231 [1] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.172.120<0> | |
| nccl-alltoall-job-oguz-mpiworker-24:47:231 [1] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-24:47:231 [1] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-24:49:233 [3] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.172.120<0> | |
| nccl-alltoall-job-oguz-mpiworker-24:49:233 [3] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-24:49:233 [3] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-7:46:230 [0] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.115.133<0> | |
| nccl-alltoall-job-oguz-mpiworker-7:46:230 [0] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-7:46:230 [0] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-27:46:231 [0] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.144.145<0> | |
| nccl-alltoall-job-oguz-mpiworker-27:46:231 [0] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-27:46:231 [0] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-0:50:238 [4] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.72.84<0> | |
| nccl-alltoall-job-oguz-mpiworker-27:52:230 [6] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.144.145<0> | |
| nccl-alltoall-job-oguz-mpiworker-0:50:238 [4] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-0:50:238 [4] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-27:52:230 [6] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-27:52:230 [6] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-24:53:235 [7] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.172.120<0> | |
| nccl-alltoall-job-oguz-mpiworker-24:53:235 [7] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-24:53:235 [7] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-13:51:237 [5] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.125.171<0> | |
| nccl-alltoall-job-oguz-mpiworker-13:51:237 [5] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-13:51:237 [5] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-25:49:233 [3] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.121.49<0> | |
| nccl-alltoall-job-oguz-mpiworker-25:49:233 [3] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-25:49:233 [3] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-16:53:236 [7] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.16.145<0> | |
| nccl-alltoall-job-oguz-mpiworker-16:53:236 [7] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-16:53:236 [7] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-10:48:231 [2] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.150.6<0> | |
| nccl-alltoall-job-oguz-mpiworker-10:48:231 [2] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-10:48:231 [2] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-9:47:230 [1] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.107.96<0> | |
| nccl-alltoall-job-oguz-mpiworker-9:47:230 [1] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-9:47:230 [1] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-29:48:231 [2] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.90.183<0> | |
| nccl-alltoall-job-oguz-mpiworker-29:48:231 [2] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-29:48:231 [2] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-4:49:234 [3] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.80.162<0> | |
| nccl-alltoall-job-oguz-mpiworker-23:47:233 [1] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.32.41<0> | |
| nccl-alltoall-job-oguz-mpiworker-4:49:234 [3] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-4:49:234 [3] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-23:47:233 [1] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-23:47:233 [1] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-29:46:232 [0] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.90.183<0> | |
| nccl-alltoall-job-oguz-mpiworker-29:52:237 [6] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.90.183<0> | |
| nccl-alltoall-job-oguz-mpiworker-29:46:232 [0] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-29:46:232 [0] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-29:52:237 [6] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-29:52:237 [6] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-20:51:230 [5] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.124.65<0> | |
| nccl-alltoall-job-oguz-mpiworker-13:46:235 [0] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.125.171<0> | |
| nccl-alltoall-job-oguz-mpiworker-20:49:235 [3] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.124.65<0> | |
| nccl-alltoall-job-oguz-mpiworker-29:51:230 [5] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.90.183<0> | |
| nccl-alltoall-job-oguz-mpiworker-13:48:230 [2] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.125.171<0> | |
| nccl-alltoall-job-oguz-mpiworker-20:51:230 [5] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-20:51:230 [5] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-13:46:235 [0] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-13:46:235 [0] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-20:49:235 [3] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-20:49:235 [3] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-29:51:230 [5] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-29:51:230 [5] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-13:53:232 [7] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.125.171<0> | |
| nccl-alltoall-job-oguz-mpiworker-13:48:230 [2] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-13:48:230 [2] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-13:53:232 [7] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-13:53:232 [7] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-27:48:235 [2] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.144.145<0> | |
| nccl-alltoall-job-oguz-mpiworker-27:48:235 [2] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-27:48:235 [2] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-26:51:236 [5] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.157.211<0> | |
| nccl-alltoall-job-oguz-mpiworker-26:51:236 [5] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-26:51:236 [5] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-26:49:233 [3] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.157.211<0> | |
| nccl-alltoall-job-oguz-mpiworker-26:49:233 [3] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-26:49:233 [3] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-16:47:234 [1] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.16.145<0> | |
| nccl-alltoall-job-oguz-mpiworker-16:47:234 [1] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-16:47:234 [1] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-23:49:236 [3] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.32.41<0> | |
| nccl-alltoall-job-oguz-mpiworker-23:49:236 [3] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-23:49:236 [3] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-7:50:234 [4] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.115.133<0> | |
| nccl-alltoall-job-oguz-mpiworker-3:46:233 [0] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.129.187<0> | |
| nccl-alltoall-job-oguz-mpiworker-7:50:234 [4] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-7:50:234 [4] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-3:46:233 [0] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-3:46:233 [0] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-29:50:234 [4] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.90.183<0> | |
| nccl-alltoall-job-oguz-mpiworker-29:50:234 [4] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-29:50:234 [4] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-5:53:233 [7] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.169.8<0> | |
| nccl-alltoall-job-oguz-mpiworker-5:53:233 [7] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-5:53:233 [7] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-16:48:231 [2] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.16.145<0> | |
| nccl-alltoall-job-oguz-mpiworker-16:48:231 [2] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-16:48:231 [2] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-24:51:234 [5] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.172.120<0> | |
| nccl-alltoall-job-oguz-mpiworker-26:46:231 [0] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.157.211<0> | |
| nccl-alltoall-job-oguz-mpiworker-24:51:234 [5] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-24:51:234 [5] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-26:46:231 [0] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-26:46:231 [0] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-16:46:230 [0] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.16.145<0> | |
| nccl-alltoall-job-oguz-mpiworker-16:46:230 [0] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-16:46:230 [0] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-21:48:232 [2] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.149.199<0> | |
| nccl-alltoall-job-oguz-mpiworker-21:48:232 [2] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-21:48:232 [2] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-23:51:235 [5] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.32.41<0> | |
| nccl-alltoall-job-oguz-mpiworker-23:51:235 [5] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-23:51:235 [5] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-25:47:234 [1] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.121.49<0> | |
| nccl-alltoall-job-oguz-mpiworker-25:47:234 [1] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-25:47:234 [1] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-0:48:233 [2] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.72.84<0> | |
| nccl-alltoall-job-oguz-mpiworker-0:48:233 [2] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-0:48:233 [2] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-0:53:234 [7] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.72.84<0> | |
| nccl-alltoall-job-oguz-mpiworker-0:53:234 [7] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-0:53:234 [7] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-30:49:234 [3] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.155.168<0> | |
| nccl-alltoall-job-oguz-mpiworker-30:49:234 [3] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-30:49:234 [3] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-30:51:233 [5] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.155.168<0> | |
| nccl-alltoall-job-oguz-mpiworker-30:51:233 [5] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-30:51:233 [5] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-25:52:236 [6] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.121.49<0> | |
| nccl-alltoall-job-oguz-mpiworker-25:52:236 [6] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-25:52:236 [6] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-27:53:237 [7] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.144.145<0> | |
| nccl-alltoall-job-oguz-mpiworker-18:47:232 [1] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.148.215<0> | |
| nccl-alltoall-job-oguz-mpiworker-30:53:231 [7] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.155.168<0> | |
| nccl-alltoall-job-oguz-mpiworker-27:53:237 [7] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-27:53:237 [7] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-30:53:231 [7] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-30:53:231 [7] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-18:47:232 [1] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-18:47:232 [1] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-16:51:235 [5] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.16.145<0> | |
| nccl-alltoall-job-oguz-mpiworker-30:46:230 [0] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.155.168<0> | |
| nccl-alltoall-job-oguz-mpiworker-16:51:235 [5] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-16:51:235 [5] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-8:51:234 [5] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.147.49<0> | |
| nccl-alltoall-job-oguz-mpiworker-30:46:230 [0] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-30:46:230 [0] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-8:51:234 [5] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-8:51:234 [5] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-26:48:235 [2] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.157.211<0> | |
| nccl-alltoall-job-oguz-mpiworker-26:48:235 [2] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-26:48:235 [2] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-4:47:232 [1] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.80.162<0> | |
| nccl-alltoall-job-oguz-mpiworker-31:52:233 [6] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.146.72<0> | |
| nccl-alltoall-job-oguz-mpiworker-4:53:235 [7] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.80.162<0> | |
| nccl-alltoall-job-oguz-mpiworker-4:47:232 [1] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-4:47:232 [1] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-14:47:233 [1] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.105.20<0> | |
| nccl-alltoall-job-oguz-mpiworker-31:52:233 [6] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-31:52:233 [6] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-4:53:235 [7] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-4:53:235 [7] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-14:47:233 [1] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-14:47:233 [1] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-26:47:230 [1] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.157.211<0> | |
| nccl-alltoall-job-oguz-mpiworker-26:47:230 [1] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-26:47:230 [1] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-27:50:232 [4] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.144.145<0> | |
| nccl-alltoall-job-oguz-mpiworker-27:50:232 [4] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-27:50:232 [4] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-0:46:231 [0] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.72.84<0> | |
| nccl-alltoall-job-oguz-mpiworker-0:46:231 [0] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-0:46:231 [0] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-21:49:236 [3] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.149.199<0> | |
| nccl-alltoall-job-oguz-mpiworker-30:48:237 [2] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.155.168<0> | |
| nccl-alltoall-job-oguz-mpiworker-21:49:236 [3] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-21:49:236 [3] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-30:48:237 [2] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-30:48:237 [2] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-3:48:231 [2] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.129.187<0> | |
| nccl-alltoall-job-oguz-mpiworker-3:48:231 [2] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-3:48:231 [2] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-23:50:234 [4] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.32.41<0> | |
| nccl-alltoall-job-oguz-mpiworker-23:50:234 [4] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-23:50:234 [4] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-30:52:235 [6] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.155.168<0> | |
| nccl-alltoall-job-oguz-mpiworker-30:52:235 [6] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-30:52:235 [6] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-19:53:231 [7] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.117.132<0> | |
| nccl-alltoall-job-oguz-mpiworker-19:53:231 [7] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-19:53:231 [7] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-21:52:231 [6] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.149.199<0> | |
| nccl-alltoall-job-oguz-mpiworker-21:52:231 [6] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-21:52:231 [6] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-24:52:232 [6] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.172.120<0> | |
| nccl-alltoall-job-oguz-mpiworker-24:52:232 [6] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-24:52:232 [6] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-21:53:233 [7] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.149.199<0> | |
| nccl-alltoall-job-oguz-mpiworker-21:53:233 [7] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-21:53:233 [7] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-21:50:237 [4] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.149.199<0> | |
| nccl-alltoall-job-oguz-mpiworker-21:50:237 [4] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-21:50:237 [4] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-1:46:230 [0] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.130.46<0> | |
| nccl-alltoall-job-oguz-mpiworker-25:53:231 [7] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.121.49<0> | |
| nccl-alltoall-job-oguz-mpiworker-1:46:230 [0] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-1:46:230 [0] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-25:53:231 [7] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-25:53:231 [7] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-20:47:234 [1] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.124.65<0> | |
| nccl-alltoall-job-oguz-mpiworker-20:47:234 [1] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-20:47:234 [1] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-12:49:231 [3] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.95.47<0> | |
| nccl-alltoall-job-oguz-mpiworker-12:49:231 [3] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-12:49:231 [3] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-22:47:230 [1] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.168.242<0> | |
| nccl-alltoall-job-oguz-mpiworker-22:47:230 [1] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-22:47:230 [1] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-6:51:235 [5] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.154.237<0> | |
| nccl-alltoall-job-oguz-mpiworker-6:51:235 [5] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-6:51:235 [5] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-18:48:233 [2] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.148.215<0> | |
| nccl-alltoall-job-oguz-mpiworker-18:48:233 [2] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-18:48:233 [2] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-20:50:237 [4] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.124.65<0> | |
| nccl-alltoall-job-oguz-mpiworker-21:47:234 [1] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.149.199<0> | |
| nccl-alltoall-job-oguz-mpiworker-20:50:237 [4] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-20:50:237 [4] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-21:47:234 [1] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-21:47:234 [1] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-4:52:237 [6] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.80.162<0> | |
| nccl-alltoall-job-oguz-mpiworker-4:52:237 [6] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-4:52:237 [6] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-14:53:231 [7] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.105.20<0> | |
| nccl-alltoall-job-oguz-mpiworker-14:49:232 [3] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.105.20<0> | |
| nccl-alltoall-job-oguz-mpiworker-14:53:231 [7] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-14:53:231 [7] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-14:49:232 [3] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-14:49:232 [3] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-4:48:233 [2] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.80.162<0> | |
| nccl-alltoall-job-oguz-mpiworker-19:49:236 [3] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.117.132<0> | |
| nccl-alltoall-job-oguz-mpiworker-4:48:233 [2] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-4:48:233 [2] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-19:51:235 [5] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.117.132<0> | |
| nccl-alltoall-job-oguz-mpiworker-19:49:236 [3] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-19:49:236 [3] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-19:51:235 [5] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-19:51:235 [5] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-16:49:233 [3] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.16.145<0> | |
| nccl-alltoall-job-oguz-mpiworker-16:49:233 [3] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-16:49:233 [3] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-16:52:237 [6] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.16.145<0> | |
| nccl-alltoall-job-oguz-mpiworker-14:52:235 [6] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.105.20<0> | |
| nccl-alltoall-job-oguz-mpiworker-16:52:237 [6] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-16:52:237 [6] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-14:52:235 [6] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-14:52:235 [6] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-23:48:232 [2] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.32.41<0> | |
| nccl-alltoall-job-oguz-mpiworker-23:48:232 [2] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-23:48:232 [2] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-1:48:231 [2] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.130.46<0> | |
| nccl-alltoall-job-oguz-mpiworker-1:48:231 [2] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-1:48:231 [2] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-30:47:232 [1] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.155.168<0> | |
| nccl-alltoall-job-oguz-mpiworker-1:50:233 [4] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.130.46<0> | |
| nccl-alltoall-job-oguz-mpiworker-30:47:232 [1] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-30:47:232 [1] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-1:50:233 [4] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-1:50:233 [4] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-3:47:230 [1] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.129.187<0> | |
| nccl-alltoall-job-oguz-mpiworker-3:47:230 [1] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-3:47:230 [1] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-7:52:231 [6] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.115.133<0> | |
| nccl-alltoall-job-oguz-mpiworker-7:52:231 [6] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-7:52:231 [6] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-28:50:234 [4] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.168.119<0> | |
| nccl-alltoall-job-oguz-mpiworker-28:50:234 [4] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-28:50:234 [4] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-28:53:235 [7] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.168.119<0> | |
| nccl-alltoall-job-oguz-mpiworker-28:53:235 [7] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-28:53:235 [7] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-25:50:235 [4] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.121.49<0> | |
| nccl-alltoall-job-oguz-mpiworker-25:50:235 [4] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-25:50:235 [4] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-26:52:237 [6] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.157.211<0> | |
| nccl-alltoall-job-oguz-mpiworker-26:52:237 [6] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-26:52:237 [6] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-2:49:236 [3] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.96.180<0> | |
| nccl-alltoall-job-oguz-mpiworker-2:49:236 [3] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-2:49:236 [3] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-18:46:230 [0] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.148.215<0> | |
| nccl-alltoall-job-oguz-mpiworker-18:46:230 [0] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-18:46:230 [0] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-12:46:237 [0] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.95.47<0> | |
| nccl-alltoall-job-oguz-mpiworker-12:46:237 [0] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-12:46:237 [0] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-12:51:234 [5] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.95.47<0> | |
| nccl-alltoall-job-oguz-mpiworker-12:51:234 [5] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-12:51:234 [5] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-29:53:235 [7] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.90.183<0> | |
| nccl-alltoall-job-oguz-mpiworker-29:53:235 [7] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-29:53:235 [7] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-0:52:235 [6] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.72.84<0> | |
| nccl-alltoall-job-oguz-mpiworker-0:52:235 [6] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-0:52:235 [6] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-1:54:234 [7] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.130.46<0> | |
| nccl-alltoall-job-oguz-mpiworker-1:54:234 [7] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-1:54:234 [7] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-28:49:237 [3] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.168.119<0> | |
| nccl-alltoall-job-oguz-mpiworker-18:52:237 [6] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.148.215<0> | |
| nccl-alltoall-job-oguz-mpiworker-28:49:237 [3] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-28:49:237 [3] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-18:52:237 [6] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-18:52:237 [6] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-26:50:234 [4] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.157.211<0> | |
| nccl-alltoall-job-oguz-mpiworker-26:50:234 [4] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-26:50:234 [4] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-17:52:237 [6] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.133.20<0> | |
| nccl-alltoall-job-oguz-mpiworker-17:52:237 [6] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-17:52:237 [6] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-23:52:237 [6] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.32.41<0> | |
| nccl-alltoall-job-oguz-mpiworker-23:52:237 [6] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-23:52:237 [6] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-28:51:236 [5] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.168.119<0> | |
| nccl-alltoall-job-oguz-mpiworker-28:51:236 [5] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-28:51:236 [5] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-9:49:232 [3] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.107.96<0> | |
| nccl-alltoall-job-oguz-mpiworker-9:49:232 [3] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-9:49:232 [3] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-25:46:230 [0] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.121.49<0> | |
| nccl-alltoall-job-oguz-mpiworker-25:46:230 [0] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-25:46:230 [0] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-10:52:233 [6] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.150.6<0> | |
| nccl-alltoall-job-oguz-mpiworker-6:50:237 [4] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.154.237<0> | |
| nccl-alltoall-job-oguz-mpiworker-6:52:233 [6] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.154.237<0> | |
| nccl-alltoall-job-oguz-mpiworker-10:52:233 [6] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-10:52:233 [6] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-6:50:237 [4] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-6:50:237 [4] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-18:49:234 [3] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.148.215<0> | |
| nccl-alltoall-job-oguz-mpiworker-6:52:233 [6] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-6:52:233 [6] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-18:49:234 [3] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-18:49:234 [3] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-5:47:235 [1] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.169.8<0> | |
| nccl-alltoall-job-oguz-mpiworker-5:47:235 [1] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-5:47:235 [1] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-15:49:235 [3] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.25.48<0> | |
| nccl-alltoall-job-oguz-mpiworker-15:49:235 [3] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-15:49:235 [3] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-20:48:236 [2] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.124.65<0> | |
| nccl-alltoall-job-oguz-mpiworker-20:48:236 [2] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-20:48:236 [2] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-19:52:234 [6] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.117.132<0> | |
| nccl-alltoall-job-oguz-mpiworker-2:51:234 [5] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.96.180<0> | |
| nccl-alltoall-job-oguz-mpiworker-19:52:234 [6] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-19:52:234 [6] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-23:46:230 [0] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.32.41<0> | |
| nccl-alltoall-job-oguz-mpiworker-2:51:234 [5] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-2:51:234 [5] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-23:46:230 [0] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-23:46:230 [0] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-6:48:231 [2] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.154.237<0> | |
| nccl-alltoall-job-oguz-mpiworker-6:48:231 [2] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-6:48:231 [2] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-6:49:234 [3] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.154.237<0> | |
| nccl-alltoall-job-oguz-mpiworker-6:49:234 [3] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-6:49:234 [3] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-21:51:235 [5] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.149.199<0> | |
| nccl-alltoall-job-oguz-mpiworker-21:51:235 [5] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-21:51:235 [5] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-12:50:232 [4] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.95.47<0> | |
| nccl-alltoall-job-oguz-mpiworker-12:50:232 [4] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-12:50:232 [4] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-12:52:235 [6] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.95.47<0> | |
| nccl-alltoall-job-oguz-mpiworker-12:52:235 [6] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-12:52:235 [6] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-28:48:231 [2] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.168.119<0> | |
| nccl-alltoall-job-oguz-mpiworker-28:48:231 [2] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-28:48:231 [2] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-8:52:236 [6] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.147.49<0> | |
| nccl-alltoall-job-oguz-mpiworker-8:52:236 [6] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-8:52:236 [6] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-22:53:234 [7] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.168.242<0> | |
| nccl-alltoall-job-oguz-mpiworker-22:53:234 [7] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-22:53:234 [7] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-1:51:232 [5] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.130.46<0> | |
| nccl-alltoall-job-oguz-mpiworker-28:47:230 [1] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.168.119<0> | |
| nccl-alltoall-job-oguz-mpiworker-1:51:232 [5] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-1:51:232 [5] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-28:47:230 [1] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-28:47:230 [1] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-18:50:236 [4] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.148.215<0> | |
| nccl-alltoall-job-oguz-mpiworker-19:48:233 [2] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.117.132<0> | |
| nccl-alltoall-job-oguz-mpiworker-18:50:236 [4] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-18:50:236 [4] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-19:48:233 [2] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-19:48:233 [2] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-8:46:231 [0] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.147.49<0> | |
| nccl-alltoall-job-oguz-mpiworker-8:46:231 [0] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-8:46:231 [0] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-5:49:232 [3] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.169.8<0> | |
| nccl-alltoall-job-oguz-mpiworker-8:47:232 [1] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.147.49<0> | |
| nccl-alltoall-job-oguz-mpiworker-5:49:232 [3] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-5:49:232 [3] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-8:47:232 [1] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-8:47:232 [1] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-5:46:230 [0] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.169.8<0> | |
| nccl-alltoall-job-oguz-mpiworker-5:46:230 [0] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-5:46:230 [0] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-11:47:235 [1] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.113.218<0> | |
| nccl-alltoall-job-oguz-mpiworker-11:47:235 [1] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-11:47:235 [1] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-3:53:235 [7] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.129.187<0> | |
| nccl-alltoall-job-oguz-mpiworker-3:53:235 [7] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-3:53:235 [7] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-18:53:231 [7] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.148.215<0> | |
| nccl-alltoall-job-oguz-mpiworker-18:53:231 [7] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-18:53:231 [7] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-8:50:233 [4] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.147.49<0> | |
| nccl-alltoall-job-oguz-mpiworker-22:48:236 [2] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.168.242<0> | |
| nccl-alltoall-job-oguz-mpiworker-8:50:233 [4] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-8:50:233 [4] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-22:48:236 [2] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-22:48:236 [2] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-12:48:236 [2] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.95.47<0> | |
| nccl-alltoall-job-oguz-mpiworker-12:48:236 [2] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-12:48:236 [2] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-24:46:237 [0] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.172.120<0> | |
| nccl-alltoall-job-oguz-mpiworker-24:46:237 [0] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-24:46:237 [0] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-20:46:233 [0] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.124.65<0> | |
| nccl-alltoall-job-oguz-mpiworker-20:46:233 [0] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-20:46:233 [0] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-23:53:231 [7] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.32.41<0> | |
| nccl-alltoall-job-oguz-mpiworker-23:53:231 [7] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-23:53:231 [7] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-11:53:236 [7] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.113.218<0> | |
| nccl-alltoall-job-oguz-mpiworker-11:53:236 [7] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-11:53:236 [7] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-11:50:237 [4] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.113.218<0> | |
| nccl-alltoall-job-oguz-mpiworker-11:50:237 [4] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-11:50:237 [4] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-6:47:232 [1] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.154.237<0> | |
| nccl-alltoall-job-oguz-mpiworker-2:46:232 [0] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.96.180<0> | |
| nccl-alltoall-job-oguz-mpiworker-6:47:232 [1] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-6:47:232 [1] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-2:46:232 [0] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-2:46:232 [0] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-25:48:237 [2] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.121.49<0> | |
| nccl-alltoall-job-oguz-mpiworker-25:48:237 [2] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-25:48:237 [2] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-11:48:231 [2] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.113.218<0> | |
| nccl-alltoall-job-oguz-mpiworker-11:48:231 [2] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-11:48:231 [2] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-19:47:232 [1] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.117.132<0> | |
| nccl-alltoall-job-oguz-mpiworker-19:47:232 [1] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-19:47:232 [1] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-22:52:235 [6] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.168.242<0> | |
| nccl-alltoall-job-oguz-mpiworker-22:52:235 [6] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-22:52:235 [6] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-1:49:236 [3] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.130.46<0> | |
| nccl-alltoall-job-oguz-mpiworker-1:49:236 [3] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-1:49:236 [3] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-31:49:235 [3] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.146.72<0> | |
| nccl-alltoall-job-oguz-mpiworker-31:46:234 [0] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.146.72<0> | |
| nccl-alltoall-job-oguz-mpiworker-31:49:235 [3] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-31:49:235 [3] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-16:50:232 [4] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.16.145<0> | |
| nccl-alltoall-job-oguz-mpiworker-31:46:234 [0] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-31:46:234 [0] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-16:50:232 [4] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-16:50:232 [4] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-17:49:233 [3] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.133.20<0> | |
| nccl-alltoall-job-oguz-mpiworker-17:49:233 [3] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-17:49:233 [3] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-9:48:233 [2] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.107.96<0> | |
| nccl-alltoall-job-oguz-mpiworker-9:48:233 [2] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-9:48:233 [2] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-14:50:236 [4] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.105.20<0> | |
| nccl-alltoall-job-oguz-mpiworker-14:50:236 [4] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-14:50:236 [4] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-2:53:230 [7] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.96.180<0> | |
| nccl-alltoall-job-oguz-mpiworker-20:53:231 [7] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.124.65<0> | |
| nccl-alltoall-job-oguz-mpiworker-2:53:230 [7] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-2:53:230 [7] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-20:53:231 [7] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-20:53:231 [7] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-4:50:230 [4] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.80.162<0> | |
| nccl-alltoall-job-oguz-mpiworker-3:49:232 [3] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.129.187<0> | |
| nccl-alltoall-job-oguz-mpiworker-11:52:233 [6] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.113.218<0> | |
| nccl-alltoall-job-oguz-mpiworker-4:50:230 [4] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-4:50:230 [4] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-3:49:232 [3] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-3:49:232 [3] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-11:52:233 [6] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-11:52:233 [6] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-31:51:236 [5] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.146.72<0> | |
| nccl-alltoall-job-oguz-mpiworker-31:53:231 [7] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.146.72<0> | |
| nccl-alltoall-job-oguz-mpiworker-31:51:236 [5] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-31:51:236 [5] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-31:53:231 [7] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-31:53:231 [7] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-28:52:232 [6] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.168.119<0> | |
| nccl-alltoall-job-oguz-mpiworker-28:52:232 [6] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-28:52:232 [6] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-9:46:231 [0] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.107.96<0> | |
| nccl-alltoall-job-oguz-mpiworker-9:46:231 [0] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-9:46:231 [0] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-28:46:233 [0] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.168.119<0> | |
| nccl-alltoall-job-oguz-mpiworker-28:46:233 [0] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-28:46:233 [0] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-17:46:234 [0] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.133.20<0> | |
| nccl-alltoall-job-oguz-mpiworker-14:46:234 [0] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.105.20<0> | |
| nccl-alltoall-job-oguz-mpiworker-17:46:234 [0] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-17:46:234 [0] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-13:47:231 [1] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.125.171<0> | |
| nccl-alltoall-job-oguz-mpiworker-14:46:234 [0] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-14:46:234 [0] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-13:49:234 [3] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.125.171<0> | |
| nccl-alltoall-job-oguz-mpiworker-13:47:231 [1] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-13:47:231 [1] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-13:49:234 [3] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-13:49:234 [3] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-8:53:230 [7] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.147.49<0> | |
| nccl-alltoall-job-oguz-mpiworker-8:53:230 [7] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-8:53:230 [7] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-15:47:232 [1] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.25.48<0> | |
| nccl-alltoall-job-oguz-mpiworker-15:53:236 [7] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.25.48<0> | |
| nccl-alltoall-job-oguz-mpiworker-15:47:232 [1] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-15:47:232 [1] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-15:53:236 [7] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-15:53:236 [7] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-12:47:230 [1] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.95.47<0> | |
| nccl-alltoall-job-oguz-mpiworker-12:47:230 [1] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-12:47:230 [1] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-22:46:231 [0] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.168.242<0> | |
| nccl-alltoall-job-oguz-mpiworker-22:46:231 [0] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-22:46:231 [0] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-9:50:237 [4] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.107.96<0> | |
| nccl-alltoall-job-oguz-mpiworker-9:50:237 [4] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-9:50:237 [4] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-31:48:230 [2] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.146.72<0> | |
| nccl-alltoall-job-oguz-mpiworker-31:48:230 [2] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-31:48:230 [2] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-6:53:230 [7] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.154.237<0> | |
| nccl-alltoall-job-oguz-mpiworker-6:53:230 [7] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-6:53:230 [7] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-31:50:237 [4] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.146.72<0> | |
| nccl-alltoall-job-oguz-mpiworker-31:50:237 [4] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-31:50:237 [4] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-14:51:230 [5] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.105.20<0> | |
| nccl-alltoall-job-oguz-mpiworker-14:51:230 [5] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-14:51:230 [5] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-3:52:237 [6] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.129.187<0> | |
| nccl-alltoall-job-oguz-mpiworker-3:52:237 [6] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-3:52:237 [6] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-0:49:236 [3] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.72.84<0> | |
| nccl-alltoall-job-oguz-mpiworker-1:47:237 [1] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.130.46<0> | |
| nccl-alltoall-job-oguz-mpiworker-0:49:236 [3] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-0:49:236 [3] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-1:47:237 [1] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-1:47:237 [1] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-20:52:232 [6] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.124.65<0> | |
| nccl-alltoall-job-oguz-mpiworker-20:52:232 [6] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-20:52:232 [6] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-11:46:230 [0] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.113.218<0> | |
| nccl-alltoall-job-oguz-mpiworker-11:46:230 [0] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-11:46:230 [0] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-17:53:236 [7] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.133.20<0> | |
| nccl-alltoall-job-oguz-mpiworker-17:53:236 [7] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-17:53:236 [7] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-3:51:234 [5] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.129.187<0> | |
| nccl-alltoall-job-oguz-mpiworker-3:51:234 [5] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-3:51:234 [5] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-22:51:233 [5] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.168.242<0> | |
| nccl-alltoall-job-oguz-mpiworker-22:51:233 [5] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-22:51:233 [5] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-5:50:231 [4] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.169.8<0> | |
| nccl-alltoall-job-oguz-mpiworker-5:50:231 [4] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-5:50:231 [4] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-0:47:237 [1] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.72.84<0> | |
| nccl-alltoall-job-oguz-mpiworker-0:47:237 [1] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-0:47:237 [1] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-11:51:234 [5] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.113.218<0> | |
| nccl-alltoall-job-oguz-mpiworker-11:51:234 [5] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-11:51:234 [5] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-8:48:235 [2] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.147.49<0> | |
| nccl-alltoall-job-oguz-mpiworker-19:46:230 [0] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.117.132<0> | |
| nccl-alltoall-job-oguz-mpiworker-8:48:235 [2] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-8:48:235 [2] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-19:46:230 [0] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-19:46:230 [0] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-4:46:231 [0] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.80.162<0> | |
| nccl-alltoall-job-oguz-mpiworker-4:46:231 [0] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-4:46:231 [0] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-5:51:236 [5] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.169.8<0> | |
| nccl-alltoall-job-oguz-mpiworker-5:51:236 [5] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-5:51:236 [5] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-5:52:237 [6] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.169.8<0> | |
| nccl-alltoall-job-oguz-mpiworker-5:52:237 [6] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-5:52:237 [6] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-15:48:231 [2] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.25.48<0> | |
| nccl-alltoall-job-oguz-mpiworker-15:48:231 [2] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-15:48:231 [2] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-30:50:236 [4] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.155.168<0> | |
| nccl-alltoall-job-oguz-mpiworker-30:50:236 [4] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-30:50:236 [4] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-4:51:236 [5] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.80.162<0> | |
| nccl-alltoall-job-oguz-mpiworker-4:51:236 [5] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-4:51:236 [5] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-3:50:236 [4] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.129.187<0> | |
| nccl-alltoall-job-oguz-mpiworker-3:50:236 [4] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-3:50:236 [4] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-2:50:231 [4] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.96.180<0> | |
| nccl-alltoall-job-oguz-mpiworker-2:50:231 [4] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-2:50:231 [4] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-5:48:234 [2] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.169.8<0> | |
| nccl-alltoall-job-oguz-mpiworker-5:48:234 [2] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-5:48:234 [2] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-2:47:235 [1] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.96.180<0> | |
| nccl-alltoall-job-oguz-mpiworker-2:47:235 [1] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-2:47:235 [1] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-15:46:230 [0] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.25.48<0> | |
| nccl-alltoall-job-oguz-mpiworker-15:46:230 [0] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-15:46:230 [0] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-9:52:234 [6] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.107.96<0> | |
| nccl-alltoall-job-oguz-mpiworker-9:52:234 [6] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-9:52:234 [6] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-31:47:232 [1] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.146.72<0> | |
| nccl-alltoall-job-oguz-mpiworker-31:47:232 [1] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-31:47:232 [1] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-8:49:237 [3] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.147.49<0> | |
| nccl-alltoall-job-oguz-mpiworker-8:49:237 [3] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-8:49:237 [3] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-9:53:235 [7] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.107.96<0> | |
| nccl-alltoall-job-oguz-mpiworker-9:53:235 [7] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-9:53:235 [7] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-25:51:232 [5] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.121.49<0> | |
| nccl-alltoall-job-oguz-mpiworker-25:51:232 [5] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-25:51:232 [5] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-15:52:237 [6] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.25.48<0> | |
| nccl-alltoall-job-oguz-mpiworker-24:50:236 [4] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.172.120<0> | |
| nccl-alltoall-job-oguz-mpiworker-15:52:237 [6] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-15:52:237 [6] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-24:50:236 [4] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-24:50:236 [4] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-2:52:233 [6] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.96.180<0> | |
| nccl-alltoall-job-oguz-mpiworker-2:52:233 [6] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-2:52:233 [6] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-17:47:231 [1] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.133.20<0> | |
| nccl-alltoall-job-oguz-mpiworker-17:47:231 [1] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-17:47:231 [1] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-1:52:235 [6] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.130.46<0> | |
| nccl-alltoall-job-oguz-mpiworker-1:52:235 [6] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-1:52:235 [6] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-2:48:237 [2] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.96.180<0> | |
| nccl-alltoall-job-oguz-mpiworker-2:48:237 [2] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-2:48:237 [2] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-14:48:237 [2] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.105.20<0> | |
| nccl-alltoall-job-oguz-mpiworker-14:48:237 [2] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-14:48:237 [2] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-11:49:232 [3] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.113.218<0> | |
| nccl-alltoall-job-oguz-mpiworker-11:49:232 [3] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-11:49:232 [3] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-17:50:235 [4] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.133.20<0> | |
| nccl-alltoall-job-oguz-mpiworker-17:50:235 [4] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-17:50:235 [4] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-12:54:233 [7] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.95.47<0> | |
| nccl-alltoall-job-oguz-mpiworker-12:54:233 [7] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-12:54:233 [7] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-17:51:232 [5] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.133.20<0> | |
| nccl-alltoall-job-oguz-mpiworker-17:51:232 [5] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-17:51:232 [5] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-22:49:237 [3] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.168.242<0> | |
| nccl-alltoall-job-oguz-mpiworker-22:49:237 [3] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-22:49:237 [3] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-17:48:230 [2] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.133.20<0> | |
| nccl-alltoall-job-oguz-mpiworker-17:48:230 [2] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-17:48:230 [2] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-6:46:236 [0] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.154.237<0> | |
| nccl-alltoall-job-oguz-mpiworker-6:46:236 [0] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-6:46:236 [0] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-18:51:235 [5] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.148.215<0> | |
| nccl-alltoall-job-oguz-mpiworker-18:51:235 [5] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-18:51:235 [5] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-15:51:233 [5] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.25.48<0> | |
| nccl-alltoall-job-oguz-mpiworker-15:51:233 [5] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-15:51:233 [5] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-19:50:237 [4] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.117.132<0> | |
| nccl-alltoall-job-oguz-mpiworker-19:50:237 [4] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-19:50:237 [4] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-22:50:232 [4] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.168.242<0> | |
| nccl-alltoall-job-oguz-mpiworker-22:50:232 [4] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-22:50:232 [4] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-15:50:234 [4] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.25.48<0> | |
| nccl-alltoall-job-oguz-mpiworker-15:50:234 [4] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-15:50:234 [4] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-9:51:236 [5] NCCL INFO NET/IB : Using [0]mlx5_18:1/RoCE [1]mlx5_19:1/RoCE [2]mlx5_20:1/RoCE [3]mlx5_21:1/RoCE [4]mlx5_22:1/RoCE [5]mlx5_23:1/RoCE [6]mlx5_24:1/RoCE [7]mlx5_25:1/RoCE [8]mlx5_26:1/RoCE [9]mlx5_27:1/RoCE [10]mlx5_28:1/RoCE [11]mlx5_29:1/RoCE [12]mlx5_30:1/RoCE [13]mlx5_31:1/RoCE [14]mlx5_32:1/RoCE [15]mlx5_33:1/RoCE [RO]; OOB eth0:10.244.107.96<0> | |
| nccl-alltoall-job-oguz-mpiworker-9:51:236 [5] NCCL INFO Using non-device net plugin version 0 | |
| nccl-alltoall-job-oguz-mpiworker-9:51:236 [5] NCCL INFO Using network IB | |
| nccl-alltoall-job-oguz-mpiworker-13:52:233 [6] NCCL INFO comm 0x55b022594aa0 rank 110 nranks 256 cudaDev 6 nvmlDev 6 busId c0000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-13:53:232 [7] NCCL INFO comm 0x55e86793a140 rank 111 nranks 256 cudaDev 7 nvmlDev 7 busId d8000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-14:49:232 [3] NCCL INFO comm 0x55b930c63670 rank 115 nranks 256 cudaDev 3 nvmlDev 3 busId 5b000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-21:51:235 [5] NCCL INFO comm 0x56303c287140 rank 173 nranks 256 cudaDev 5 nvmlDev 5 busId a8000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-14:48:237 [2] NCCL INFO comm 0x55d1152933f0 rank 114 nranks 256 cudaDev 2 nvmlDev 2 busId 44000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-21:53:233 [7] NCCL INFO comm 0x55dba5f8dc80 rank 175 nranks 256 cudaDev 7 nvmlDev 7 busId d8000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-14:46:234 [0] NCCL INFO comm 0x557c81633520 rank 112 nranks 256 cudaDev 0 nvmlDev 0 busId f000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-21:52:231 [6] NCCL INFO comm 0x562c240c3550 rank 174 nranks 256 cudaDev 6 nvmlDev 6 busId c0000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-14:47:233 [1] NCCL INFO comm 0x55c610388f20 rank 113 nranks 256 cudaDev 1 nvmlDev 1 busId 2d000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-14:53:231 [7] NCCL INFO comm 0x55dc5aee9420 rank 119 nranks 256 cudaDev 7 nvmlDev 7 busId d8000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-15:46:230 [0] NCCL INFO comm 0x55805397fc00 rank 120 nranks 256 cudaDev 0 nvmlDev 0 busId f000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-14:52:235 [6] NCCL INFO comm 0x564df9ba1460 rank 118 nranks 256 cudaDev 6 nvmlDev 6 busId c0000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-15:47:232 [1] NCCL INFO comm 0x555658ded0e0 rank 121 nranks 256 cudaDev 1 nvmlDev 1 busId 2d000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-21:50:237 [4] NCCL INFO comm 0x55f3b8c08770 rank 172 nranks 256 cudaDev 4 nvmlDev 4 busId 89000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-14:51:230 [5] NCCL INFO comm 0x555d81d73650 rank 117 nranks 256 cudaDev 5 nvmlDev 5 busId a8000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-21:49:236 [3] NCCL INFO comm 0x55ee1aed0a40 rank 171 nranks 256 cudaDev 3 nvmlDev 3 busId 5b000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-14:50:236 [4] NCCL INFO comm 0x5613f8c8a380 rank 116 nranks 256 cudaDev 4 nvmlDev 4 busId 89000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-21:48:232 [2] NCCL INFO comm 0x55d8ec0dccd0 rank 170 nranks 256 cudaDev 2 nvmlDev 2 busId 44000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-15:48:231 [2] NCCL INFO comm 0x56008cf26d60 rank 122 nranks 256 cudaDev 2 nvmlDev 2 busId 44000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-21:47:234 [1] NCCL INFO comm 0x56463b96eda0 rank 169 nranks 256 cudaDev 1 nvmlDev 1 busId 2d000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-15:52:237 [6] NCCL INFO comm 0x55e02e15c800 rank 126 nranks 256 cudaDev 6 nvmlDev 6 busId c0000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-21:46:230 [0] NCCL INFO comm 0x563504106bd0 rank 168 nranks 256 cudaDev 0 nvmlDev 0 busId f000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-15:51:233 [5] NCCL INFO comm 0x55b3173688d0 rank 125 nranks 256 cudaDev 5 nvmlDev 5 busId a8000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-16:47:234 [1] NCCL INFO comm 0x55a0b1af48d0 rank 129 nranks 256 cudaDev 1 nvmlDev 1 busId 2d000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-15:49:235 [3] NCCL INFO comm 0x559b66f87010 rank 123 nranks 256 cudaDev 3 nvmlDev 3 busId 5b000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-16:46:230 [0] NCCL INFO comm 0x55560ff8fdd0 rank 128 nranks 256 cudaDev 0 nvmlDev 0 busId f000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-22:46:231 [0] NCCL INFO comm 0x55b3f03a29a0 rank 176 nranks 256 cudaDev 0 nvmlDev 0 busId f000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-20:53:231 [7] NCCL INFO comm 0x563d8907f080 rank 167 nranks 256 cudaDev 7 nvmlDev 7 busId d8000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-15:53:236 [7] NCCL INFO comm 0x55879b9504f0 rank 127 nranks 256 cudaDev 7 nvmlDev 7 busId d8000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-16:48:231 [2] NCCL INFO comm 0x5622cb2a37f0 rank 130 nranks 256 cudaDev 2 nvmlDev 2 busId 44000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-20:52:232 [6] NCCL INFO comm 0x560cb79275b0 rank 166 nranks 256 cudaDev 6 nvmlDev 6 busId c0000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-15:50:234 [4] NCCL INFO comm 0x55753532c460 rank 124 nranks 256 cudaDev 4 nvmlDev 4 busId 89000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-16:49:233 [3] NCCL INFO comm 0x561666961030 rank 131 nranks 256 cudaDev 3 nvmlDev 3 busId 5b000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-22:48:236 [2] NCCL INFO comm 0x55963e754b20 rank 178 nranks 256 cudaDev 2 nvmlDev 2 busId 44000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-16:50:232 [4] NCCL INFO comm 0x55c59e6904b0 rank 132 nranks 256 cudaDev 4 nvmlDev 4 busId 89000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-22:47:230 [1] NCCL INFO comm 0x55ee87cace70 rank 177 nranks 256 cudaDev 1 nvmlDev 1 busId 2d000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-20:51:230 [5] NCCL INFO comm 0x55f3a5fbb240 rank 165 nranks 256 cudaDev 5 nvmlDev 5 busId a8000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-16:51:235 [5] NCCL INFO comm 0x5613bb097110 rank 133 nranks 256 cudaDev 5 nvmlDev 5 busId a8000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-22:49:237 [3] NCCL INFO comm 0x55fb20924de0 rank 179 nranks 256 cudaDev 3 nvmlDev 3 busId 5b000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-20:49:235 [3] NCCL INFO comm 0x5596deb175e0 rank 163 nranks 256 cudaDev 3 nvmlDev 3 busId 5b000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-16:52:237 [6] NCCL INFO comm 0x556daf468d50 rank 134 nranks 256 cudaDev 6 nvmlDev 6 busId c0000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-22:50:232 [4] NCCL INFO comm 0x561ad2dc44c0 rank 180 nranks 256 cudaDev 4 nvmlDev 4 busId 89000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-20:50:237 [4] NCCL INFO comm 0x55fea580ad00 rank 164 nranks 256 cudaDev 4 nvmlDev 4 busId 89000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-16:53:236 [7] NCCL INFO comm 0x56384c6bad50 rank 135 nranks 256 cudaDev 7 nvmlDev 7 busId d8000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-22:51:233 [5] NCCL INFO comm 0x55ead64a5e10 rank 181 nranks 256 cudaDev 5 nvmlDev 5 busId a8000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-20:48:236 [2] NCCL INFO comm 0x5586a07a5280 rank 162 nranks 256 cudaDev 2 nvmlDev 2 busId 44000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-13:51:237 [5] NCCL INFO comm 0x556e95b55770 rank 109 nranks 256 cudaDev 5 nvmlDev 5 busId a8000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-22:52:235 [6] NCCL INFO comm 0x55e3e6564950 rank 182 nranks 256 cudaDev 6 nvmlDev 6 busId c0000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-20:47:234 [1] NCCL INFO comm 0x558f7f569330 rank 161 nranks 256 cudaDev 1 nvmlDev 1 busId 2d000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-22:53:234 [7] NCCL INFO comm 0x560f4dca4040 rank 183 nranks 256 cudaDev 7 nvmlDev 7 busId d8000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-20:46:233 [0] NCCL INFO comm 0x55e6a6af0d00 rank 160 nranks 256 cudaDev 0 nvmlDev 0 busId f000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-23:46:230 [0] NCCL INFO comm 0x563bc82d2ac0 rank 184 nranks 256 cudaDev 0 nvmlDev 0 busId f000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-19:52:234 [6] NCCL INFO comm 0x55a620502910 rank 158 nranks 256 cudaDev 6 nvmlDev 6 busId c0000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-23:47:233 [1] NCCL INFO comm 0x558e7e0563e0 rank 185 nranks 256 cudaDev 1 nvmlDev 1 busId 2d000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-23:48:232 [2] NCCL INFO comm 0x55af91b73910 rank 186 nranks 256 cudaDev 2 nvmlDev 2 busId 44000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-17:46:234 [0] NCCL INFO comm 0x55ce81afae50 rank 136 nranks 256 cudaDev 0 nvmlDev 0 busId f000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-13:50:236 [4] NCCL INFO comm 0x55ddf2e57ba0 rank 108 nranks 256 cudaDev 4 nvmlDev 4 busId 89000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-19:50:237 [4] NCCL INFO comm 0x558068692c10 rank 156 nranks 256 cudaDev 4 nvmlDev 4 busId 89000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-17:47:231 [1] NCCL INFO comm 0x56079611b3a0 rank 137 nranks 256 cudaDev 1 nvmlDev 1 busId 2d000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-19:53:231 [7] NCCL INFO comm 0x55f1c8bf46f0 rank 159 nranks 256 cudaDev 7 nvmlDev 7 busId d8000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-19:48:233 [2] NCCL INFO comm 0x55d262db0710 rank 154 nranks 256 cudaDev 2 nvmlDev 2 busId 44000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-13:49:234 [3] NCCL INFO comm 0x55e401f92280 rank 107 nranks 256 cudaDev 3 nvmlDev 3 busId 5b000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-19:49:236 [3] NCCL INFO comm 0x55df6389b8a0 rank 155 nranks 256 cudaDev 3 nvmlDev 3 busId 5b000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-23:49:236 [3] NCCL INFO comm 0x5608b08f5390 rank 187 nranks 256 cudaDev 3 nvmlDev 3 busId 5b000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-17:48:230 [2] NCCL INFO comm 0x55f9265aee00 rank 138 nranks 256 cudaDev 2 nvmlDev 2 busId 44000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-19:51:235 [5] NCCL INFO comm 0x555bd7e1d260 rank 157 nranks 256 cudaDev 5 nvmlDev 5 busId a8000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-23:50:234 [4] NCCL INFO comm 0x557d98eefcb0 rank 188 nranks 256 cudaDev 4 nvmlDev 4 busId 89000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-17:49:233 [3] NCCL INFO comm 0x55e4212a5910 rank 139 nranks 256 cudaDev 3 nvmlDev 3 busId 5b000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-23:51:235 [5] NCCL INFO comm 0x5599e7802190 rank 189 nranks 256 cudaDev 5 nvmlDev 5 busId a8000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-17:50:235 [4] NCCL INFO comm 0x5577e5c86180 rank 140 nranks 256 cudaDev 4 nvmlDev 4 busId 89000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-23:53:231 [7] NCCL INFO comm 0x56410de19130 rank 191 nranks 256 cudaDev 7 nvmlDev 7 busId d8000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-17:51:232 [5] NCCL INFO comm 0x55ad38d2db90 rank 141 nranks 256 cudaDev 5 nvmlDev 5 busId a8000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-23:52:237 [6] NCCL INFO comm 0x5630c88ceec0 rank 190 nranks 256 cudaDev 6 nvmlDev 6 busId c0000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-17:52:237 [6] NCCL INFO comm 0x5625cda3a120 rank 142 nranks 256 cudaDev 6 nvmlDev 6 busId c0000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-17:53:236 [7] NCCL INFO comm 0x561ab3fc4640 rank 143 nranks 256 cudaDev 7 nvmlDev 7 busId d8000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-19:47:232 [1] NCCL INFO comm 0x5572cb0b82b0 rank 153 nranks 256 cudaDev 1 nvmlDev 1 busId 2d000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-19:46:230 [0] NCCL INFO comm 0x563ebe6f5750 rank 152 nranks 256 cudaDev 0 nvmlDev 0 busId f000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-24:46:237 [0] NCCL INFO comm 0x56544ca1bc10 rank 192 nranks 256 cudaDev 0 nvmlDev 0 busId f000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-18:52:237 [6] NCCL INFO comm 0x56333edf1ae0 rank 150 nranks 256 cudaDev 6 nvmlDev 6 busId c0000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-24:47:231 [1] NCCL INFO comm 0x560ebddd3df0 rank 193 nranks 256 cudaDev 1 nvmlDev 1 busId 2d000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-24:48:230 [2] NCCL INFO comm 0x55c899039f60 rank 194 nranks 256 cudaDev 2 nvmlDev 2 busId 44000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-24:49:233 [3] NCCL INFO comm 0x55943c4f4fa0 rank 195 nranks 256 cudaDev 3 nvmlDev 3 busId 5b000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-18:53:231 [7] NCCL INFO comm 0x55e6a5c32650 rank 151 nranks 256 cudaDev 7 nvmlDev 7 busId d8000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-18:48:233 [2] NCCL INFO comm 0x55ace68c86f0 rank 146 nranks 256 cudaDev 2 nvmlDev 2 busId 44000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-18:50:236 [4] NCCL INFO comm 0x560572e6bcb0 rank 148 nranks 256 cudaDev 4 nvmlDev 4 busId 89000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-18:51:235 [5] NCCL INFO comm 0x55aab43a2de0 rank 149 nranks 256 cudaDev 5 nvmlDev 5 busId a8000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-18:49:234 [3] NCCL INFO comm 0x56309d5c3bb0 rank 147 nranks 256 cudaDev 3 nvmlDev 3 busId 5b000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-24:50:236 [4] NCCL INFO comm 0x55c0dbcc02d0 rank 196 nranks 256 cudaDev 4 nvmlDev 4 busId 89000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-18:47:232 [1] NCCL INFO comm 0x55ad2cd0dd90 rank 145 nranks 256 cudaDev 1 nvmlDev 1 busId 2d000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-18:46:230 [0] NCCL INFO comm 0x55f1566edda0 rank 144 nranks 256 cudaDev 0 nvmlDev 0 busId f000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-24:51:234 [5] NCCL INFO comm 0x557ad011a400 rank 197 nranks 256 cudaDev 5 nvmlDev 5 busId a8000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-24:53:235 [7] NCCL INFO comm 0x55585d485a90 rank 199 nranks 256 cudaDev 7 nvmlDev 7 busId d8000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-24:52:232 [6] NCCL INFO comm 0x5637861ca530 rank 198 nranks 256 cudaDev 6 nvmlDev 6 busId c0000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-25:46:230 [0] NCCL INFO comm 0x5637f7cc0800 rank 200 nranks 256 cudaDev 0 nvmlDev 0 busId f000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-25:47:234 [1] NCCL INFO comm 0x5621bfadd110 rank 201 nranks 256 cudaDev 1 nvmlDev 1 busId 2d000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-7:49:235 [3] NCCL INFO comm 0x55ea31d875e0 rank 59 nranks 256 cudaDev 3 nvmlDev 3 busId 5b000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-7:50:234 [4] NCCL INFO comm 0x55830ac54e10 rank 60 nranks 256 cudaDev 4 nvmlDev 4 busId 89000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-25:48:237 [2] NCCL INFO comm 0x55fb9200f120 rank 202 nranks 256 cudaDev 2 nvmlDev 2 busId 44000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-7:48:237 [2] NCCL INFO comm 0x564fe5ca64b0 rank 58 nranks 256 cudaDev 2 nvmlDev 2 busId 44000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-7:52:231 [6] NCCL INFO comm 0x55edd865fa50 rank 62 nranks 256 cudaDev 6 nvmlDev 6 busId c0000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-7:51:232 [5] NCCL INFO comm 0x55ce71821a00 rank 61 nranks 256 cudaDev 5 nvmlDev 5 busId a8000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-27:53:237 [7] NCCL INFO comm 0x55cd248d6650 rank 223 nranks 256 cudaDev 7 nvmlDev 7 busId d8000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-7:54:236 [7] NCCL INFO comm 0x56318c471650 rank 63 nranks 256 cudaDev 7 nvmlDev 7 busId d8000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-2:46:232 [0] NCCL INFO comm 0x55feca5ebaa0 rank 16 nranks 256 cudaDev 0 nvmlDev 0 busId f000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-2:47:235 [1] NCCL INFO comm 0x555e19068ee0 rank 17 nranks 256 cudaDev 1 nvmlDev 1 busId 2d000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-25:49:233 [3] NCCL INFO comm 0x55a95fc14890 rank 203 nranks 256 cudaDev 3 nvmlDev 3 busId 5b000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-2:48:237 [2] NCCL INFO comm 0x5628d5188b50 rank 18 nranks 256 cudaDev 2 nvmlDev 2 busId 44000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-25:50:235 [4] NCCL INFO comm 0x55a4dddb5110 rank 204 nranks 256 cudaDev 4 nvmlDev 4 busId 89000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-28:46:233 [0] NCCL INFO comm 0x560045343880 rank 224 nranks 256 cudaDev 0 nvmlDev 0 busId f000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-25:51:232 [5] NCCL INFO comm 0x55ffa214a150 rank 205 nranks 256 cudaDev 5 nvmlDev 5 busId a8000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-28:47:230 [1] NCCL INFO comm 0x560b5e095660 rank 225 nranks 256 cudaDev 1 nvmlDev 1 busId 2d000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-25:53:231 [7] NCCL INFO comm 0x5619568e0740 rank 207 nranks 256 cudaDev 7 nvmlDev 7 busId d8000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-25:52:236 [6] NCCL INFO comm 0x55c5ef95a470 rank 206 nranks 256 cudaDev 6 nvmlDev 6 busId c0000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-2:49:236 [3] NCCL INFO comm 0x559b13133480 rank 19 nranks 256 cudaDev 3 nvmlDev 3 busId 5b000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-2:50:231 [4] NCCL INFO comm 0x55e527678d20 rank 20 nranks 256 cudaDev 4 nvmlDev 4 busId 89000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-2:52:233 [6] NCCL INFO comm 0x5617c264c4a0 rank 22 nranks 256 cudaDev 6 nvmlDev 6 busId c0000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-2:51:234 [5] NCCL INFO comm 0x55f1799c9cb0 rank 21 nranks 256 cudaDev 5 nvmlDev 5 busId a8000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-2:53:230 [7] NCCL INFO comm 0x5644b56207f0 rank 23 nranks 256 cudaDev 7 nvmlDev 7 busId d8000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-3:46:233 [0] NCCL INFO comm 0x557fc7887850 rank 24 nranks 256 cudaDev 0 nvmlDev 0 busId f000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-3:47:230 [1] NCCL INFO comm 0x558020b24a70 rank 25 nranks 256 cudaDev 1 nvmlDev 1 busId 2d000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-3:48:231 [2] NCCL INFO comm 0x560b12076170 rank 26 nranks 256 cudaDev 2 nvmlDev 2 busId 44000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-3:49:232 [3] NCCL INFO comm 0x55bd8d413ed0 rank 27 nranks 256 cudaDev 3 nvmlDev 3 busId 5b000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-28:48:231 [2] NCCL INFO comm 0x564d14707700 rank 226 nranks 256 cudaDev 2 nvmlDev 2 busId 44000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-3:50:236 [4] NCCL INFO comm 0x562d0c08a9e0 rank 28 nranks 256 cudaDev 4 nvmlDev 4 busId 89000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-3:51:234 [5] NCCL INFO comm 0x55a72d54b180 rank 29 nranks 256 cudaDev 5 nvmlDev 5 busId a8000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-3:52:237 [6] NCCL INFO comm 0x555c28322f90 rank 30 nranks 256 cudaDev 6 nvmlDev 6 busId c0000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-3:53:235 [7] NCCL INFO comm 0x564fe8e6de60 rank 31 nranks 256 cudaDev 7 nvmlDev 7 busId d8000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-7:47:233 [1] NCCL INFO comm 0x5565205c6dd0 rank 57 nranks 256 cudaDev 1 nvmlDev 1 busId 2d000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-7:46:230 [0] NCCL INFO comm 0x55a9239b4ef0 rank 56 nranks 256 cudaDev 0 nvmlDev 0 busId f000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-4:46:231 [0] NCCL INFO comm 0x5560f5b39750 rank 32 nranks 256 cudaDev 0 nvmlDev 0 busId f000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-6:53:230 [7] NCCL INFO comm 0x55ed2b66dfc0 rank 55 nranks 256 cudaDev 7 nvmlDev 7 busId d8000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-6:51:235 [5] NCCL INFO comm 0x55d09e0f6b80 rank 53 nranks 256 cudaDev 5 nvmlDev 5 busId a8000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-6:50:237 [4] NCCL INFO comm 0x5591045af0f0 rank 52 nranks 256 cudaDev 4 nvmlDev 4 busId 89000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-6:52:233 [6] NCCL INFO comm 0x5654fea4db60 rank 54 nranks 256 cudaDev 6 nvmlDev 6 busId c0000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-5:52:237 [6] NCCL INFO comm 0x55bf6d8aadf0 rank 46 nranks 256 cudaDev 6 nvmlDev 6 busId c0000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-4:48:233 [2] NCCL INFO comm 0x55b79b939830 rank 34 nranks 256 cudaDev 2 nvmlDev 2 busId 44000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-4:47:232 [1] NCCL INFO comm 0x55c5aa2cf330 rank 33 nranks 256 cudaDev 1 nvmlDev 1 busId 2d000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-6:49:234 [3] NCCL INFO comm 0x56426b333430 rank 51 nranks 256 cudaDev 3 nvmlDev 3 busId 5b000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-4:49:234 [3] NCCL INFO comm 0x560d800d09a0 rank 35 nranks 256 cudaDev 3 nvmlDev 3 busId 5b000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-6:48:231 [2] NCCL INFO comm 0x55e4269a0140 rank 50 nranks 256 cudaDev 2 nvmlDev 2 busId 44000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-5:53:233 [7] NCCL INFO comm 0x555f8ba8c280 rank 47 nranks 256 cudaDev 7 nvmlDev 7 busId d8000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-4:50:230 [4] NCCL INFO comm 0x55c59b579260 rank 36 nranks 256 cudaDev 4 nvmlDev 4 busId 89000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-6:47:232 [1] NCCL INFO comm 0x556cd35478f0 rank 49 nranks 256 cudaDev 1 nvmlDev 1 busId 2d000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-5:46:230 [0] NCCL INFO comm 0x55fed8ed7890 rank 40 nranks 256 cudaDev 0 nvmlDev 0 busId f000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-4:51:236 [5] NCCL INFO comm 0x560bf24f6980 rank 37 nranks 256 cudaDev 5 nvmlDev 5 busId a8000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-6:46:236 [0] NCCL INFO comm 0x560e3f3a8fb0 rank 48 nranks 256 cudaDev 0 nvmlDev 0 busId f000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-5:47:235 [1] NCCL INFO comm 0x56528fd0d240 rank 41 nranks 256 cudaDev 1 nvmlDev 1 busId 2d000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-4:53:235 [7] NCCL INFO comm 0x55ea4da05a80 rank 39 nranks 256 cudaDev 7 nvmlDev 7 busId d8000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-5:51:236 [5] NCCL INFO comm 0x55b9d2571f50 rank 45 nranks 256 cudaDev 5 nvmlDev 5 busId a8000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-4:52:237 [6] NCCL INFO comm 0x56199feec1b0 rank 38 nranks 256 cudaDev 6 nvmlDev 6 busId c0000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-5:50:231 [4] NCCL INFO comm 0x55b8fef0f470 rank 44 nranks 256 cudaDev 4 nvmlDev 4 busId 89000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-5:49:232 [3] NCCL INFO comm 0x5596c7e31280 rank 43 nranks 256 cudaDev 3 nvmlDev 3 busId 5b000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-5:48:234 [2] NCCL INFO comm 0x560bf6caaee0 rank 42 nranks 256 cudaDev 2 nvmlDev 2 busId 44000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-28:49:237 [3] NCCL INFO comm 0x55d65e187570 rank 227 nranks 256 cudaDev 3 nvmlDev 3 busId 5b000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-28:50:234 [4] NCCL INFO comm 0x55599854c4b0 rank 228 nranks 256 cudaDev 4 nvmlDev 4 busId 89000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-28:51:236 [5] NCCL INFO comm 0x55a7eb944550 rank 229 nranks 256 cudaDev 5 nvmlDev 5 busId a8000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-28:52:232 [6] NCCL INFO comm 0x562de2e684e0 rank 230 nranks 256 cudaDev 6 nvmlDev 6 busId c0000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-28:53:235 [7] NCCL INFO comm 0x557ed86c98c0 rank 231 nranks 256 cudaDev 7 nvmlDev 7 busId d8000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-8:46:231 [0] NCCL INFO comm 0x55b2a8913480 rank 64 nranks 256 cudaDev 0 nvmlDev 0 busId f000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-29:46:232 [0] NCCL INFO comm 0x556b3e8cf1d0 rank 232 nranks 256 cudaDev 0 nvmlDev 0 busId f000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-26:48:235 [2] NCCL INFO comm 0x556249f3f070 rank 210 nranks 256 cudaDev 2 nvmlDev 2 busId 44000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-29:48:231 [2] NCCL INFO comm 0x55e2203a2eb0 rank 234 nranks 256 cudaDev 2 nvmlDev 2 busId 44000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-29:47:233 [1] NCCL INFO comm 0x5556e210d5c0 rank 233 nranks 256 cudaDev 1 nvmlDev 1 busId 2d000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-26:46:231 [0] NCCL INFO comm 0x560c67cc78c0 rank 208 nranks 256 cudaDev 0 nvmlDev 0 busId f000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-29:52:237 [6] NCCL INFO comm 0x55873b7fa7c0 rank 238 nranks 256 cudaDev 6 nvmlDev 6 busId c0000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-29:51:230 [5] NCCL INFO comm 0x556e9ae44510 rank 237 nranks 256 cudaDev 5 nvmlDev 5 busId a8000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-29:50:234 [4] NCCL INFO comm 0x555e39736880 rank 236 nranks 256 cudaDev 4 nvmlDev 4 busId 89000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-29:49:236 [3] NCCL INFO comm 0x563ce9e49c20 rank 235 nranks 256 cudaDev 3 nvmlDev 3 busId 5b000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-29:53:235 [7] NCCL INFO comm 0x561d071717b0 rank 239 nranks 256 cudaDev 7 nvmlDev 7 busId d8000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-30:46:230 [0] NCCL INFO comm 0x55c0f92a3350 rank 240 nranks 256 cudaDev 0 nvmlDev 0 busId f000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-30:47:232 [1] NCCL INFO comm 0x55cc4995d970 rank 241 nranks 256 cudaDev 1 nvmlDev 1 busId 2d000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-30:48:237 [2] NCCL INFO comm 0x55dba29f72c0 rank 242 nranks 256 cudaDev 2 nvmlDev 2 busId 44000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-8:47:232 [1] NCCL INFO comm 0x55b171c1b7c0 rank 65 nranks 256 cudaDev 1 nvmlDev 1 busId 2d000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-30:49:234 [3] NCCL INFO comm 0x559ad7c41670 rank 243 nranks 256 cudaDev 3 nvmlDev 3 busId 5b000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-8:48:235 [2] NCCL INFO comm 0x563d811531d0 rank 66 nranks 256 cudaDev 2 nvmlDev 2 busId 44000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-30:51:233 [5] NCCL INFO comm 0x5652df15b730 rank 245 nranks 256 cudaDev 5 nvmlDev 5 busId a8000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-8:50:233 [4] NCCL INFO comm 0x558ea465e5e0 rank 68 nranks 256 cudaDev 4 nvmlDev 4 busId 89000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-31:47:232 [1] NCCL INFO comm 0x55ba41b86a80 rank 249 nranks 256 cudaDev 1 nvmlDev 1 busId 2d000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-30:50:236 [4] NCCL INFO comm 0x5609c40963c0 rank 244 nranks 256 cudaDev 4 nvmlDev 4 busId 89000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-8:49:237 [3] NCCL INFO comm 0x55f6b6febb40 rank 67 nranks 256 cudaDev 3 nvmlDev 3 busId 5b000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-31:46:234 [0] NCCL INFO comm 0x55adc6beb480 rank 248 nranks 256 cudaDev 0 nvmlDev 0 busId f000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-1:54:234 [7] NCCL INFO comm 0x55b1c3feed80 rank 15 nranks 256 cudaDev 7 nvmlDev 7 busId d8000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-30:52:235 [6] NCCL INFO comm 0x557a0d989be0 rank 246 nranks 256 cudaDev 6 nvmlDev 6 busId c0000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-8:51:234 [5] NCCL INFO comm 0x557b93322170 rank 69 nranks 256 cudaDev 5 nvmlDev 5 busId a8000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-31:49:235 [3] NCCL INFO comm 0x559300a3f9e0 rank 251 nranks 256 cudaDev 3 nvmlDev 3 busId 5b000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-26:47:230 [1] NCCL INFO comm 0x561f9201f610 rank 209 nranks 256 cudaDev 1 nvmlDev 1 busId 2d000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-9:46:231 [0] NCCL INFO comm 0x560b17c793c0 rank 72 nranks 256 cudaDev 0 nvmlDev 0 busId f000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-0:47:237 [1] NCCL INFO comm 0x559d0e65a0b0 rank 1 nranks 256 cudaDev 1 nvmlDev 1 busId 2d000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-30:53:231 [7] NCCL INFO comm 0x55d1720784b0 rank 247 nranks 256 cudaDev 7 nvmlDev 7 busId d8000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-8:52:236 [6] NCCL INFO comm 0x55a10c0036e0 rank 70 nranks 256 cudaDev 6 nvmlDev 6 busId c0000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-31:48:230 [2] NCCL INFO comm 0x55f9be55def0 rank 250 nranks 256 cudaDev 2 nvmlDev 2 busId 44000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-26:49:233 [3] NCCL INFO comm 0x55d613852270 rank 211 nranks 256 cudaDev 3 nvmlDev 3 busId 5b000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-9:48:233 [2] NCCL INFO comm 0x55e0df040210 rank 74 nranks 256 cudaDev 2 nvmlDev 2 busId 44000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-0:46:231 [0] NCCL INFO comm 0x561c845ae110 rank 0 nranks 256 cudaDev 0 nvmlDev 0 busId f000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-8:53:230 [7] NCCL INFO comm 0x562d74cc9460 rank 71 nranks 256 cudaDev 7 nvmlDev 7 busId d8000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-31:51:236 [5] NCCL INFO comm 0x5567fc685450 rank 253 nranks 256 cudaDev 5 nvmlDev 5 busId a8000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-26:50:234 [4] NCCL INFO comm 0x55aefa230a50 rank 212 nranks 256 cudaDev 4 nvmlDev 4 busId 89000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-9:47:230 [1] NCCL INFO comm 0x559248dd2160 rank 73 nranks 256 cudaDev 1 nvmlDev 1 busId 2d000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-0:48:233 [2] NCCL INFO comm 0x55cbde638920 rank 2 nranks 256 cudaDev 2 nvmlDev 2 busId 44000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-1:46:230 [0] NCCL INFO comm 0x55ab4b261490 rank 8 nranks 256 cudaDev 0 nvmlDev 0 busId f000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-31:50:237 [4] NCCL INFO comm 0x5614fb288050 rank 252 nranks 256 cudaDev 4 nvmlDev 4 busId 89000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-26:51:236 [5] NCCL INFO comm 0x55b2ffc8c3b0 rank 213 nranks 256 cudaDev 5 nvmlDev 5 busId a8000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-9:50:237 [4] NCCL INFO comm 0x55a7c63fca40 rank 76 nranks 256 cudaDev 4 nvmlDev 4 busId 89000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-0:49:236 [3] NCCL INFO comm 0x55fdf328bc50 rank 3 nranks 256 cudaDev 3 nvmlDev 3 busId 5b000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-1:52:235 [6] NCCL INFO comm 0x5627e9808f30 rank 14 nranks 256 cudaDev 6 nvmlDev 6 busId c0000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-31:52:233 [6] NCCL INFO comm 0x559af663bab0 rank 254 nranks 256 cudaDev 6 nvmlDev 6 busId c0000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-26:52:237 [6] NCCL INFO comm 0x555897c2d250 rank 214 nranks 256 cudaDev 6 nvmlDev 6 busId c0000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-9:49:232 [3] NCCL INFO comm 0x56538bed1b40 rank 75 nranks 256 cudaDev 3 nvmlDev 3 busId 5b000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-0:51:232 [5] NCCL INFO comm 0x555d591435c0 rank 5 nranks 256 cudaDev 5 nvmlDev 5 busId a8000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-1:47:237 [1] NCCL INFO comm 0x55a84c1f9de0 rank 9 nranks 256 cudaDev 1 nvmlDev 1 busId 2d000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-27:46:231 [0] NCCL INFO comm 0x556d5a1ad700 rank 216 nranks 256 cudaDev 0 nvmlDev 0 busId f000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-31:53:231 [7] NCCL INFO comm 0x562465bf4950 rank 255 nranks 256 cudaDev 7 nvmlDev 7 busId d8000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-26:53:232 [7] NCCL INFO comm 0x55e0e9f9f290 rank 215 nranks 256 cudaDev 7 nvmlDev 7 busId d8000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-9:51:236 [5] NCCL INFO comm 0x559841365870 rank 77 nranks 256 cudaDev 5 nvmlDev 5 busId a8000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-0:50:238 [4] NCCL INFO comm 0x55f213888110 rank 4 nranks 256 cudaDev 4 nvmlDev 4 busId 89000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-1:51:232 [5] NCCL INFO comm 0x561c42a35460 rank 13 nranks 256 cudaDev 5 nvmlDev 5 busId a8000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-27:47:236 [1] NCCL INFO comm 0x5609f27c5420 rank 217 nranks 256 cudaDev 1 nvmlDev 1 busId 2d000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-9:52:234 [6] NCCL INFO comm 0x561bf96260c0 rank 78 nranks 256 cudaDev 6 nvmlDev 6 busId c0000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-0:52:235 [6] NCCL INFO comm 0x55fab745e080 rank 6 nranks 256 cudaDev 6 nvmlDev 6 busId c0000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-1:49:236 [3] NCCL INFO comm 0x563a86390430 rank 11 nranks 256 cudaDev 3 nvmlDev 3 busId 5b000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-27:52:230 [6] NCCL INFO comm 0x556d19d74a70 rank 222 nranks 256 cudaDev 6 nvmlDev 6 busId c0000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-10:46:230 [0] NCCL INFO comm 0x559141a2ded0 rank 80 nranks 256 cudaDev 0 nvmlDev 0 busId f000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-9:53:235 [7] NCCL INFO comm 0x56477c32d170 rank 79 nranks 256 cudaDev 7 nvmlDev 7 busId d8000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-0:53:234 [7] NCCL INFO comm 0x557642567830 rank 7 nranks 256 cudaDev 7 nvmlDev 7 busId d8000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-1:48:231 [2] NCCL INFO comm 0x55eb323d7360 rank 10 nranks 256 cudaDev 2 nvmlDev 2 busId 44000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-27:48:235 [2] NCCL INFO comm 0x5607dbe55140 rank 218 nranks 256 cudaDev 2 nvmlDev 2 busId 44000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-10:47:236 [1] NCCL INFO comm 0x5618ca9e6bd0 rank 81 nranks 256 cudaDev 1 nvmlDev 1 busId 2d000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-1:50:233 [4] NCCL INFO comm 0x562b34b663f0 rank 12 nranks 256 cudaDev 4 nvmlDev 4 busId 89000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-27:50:232 [4] NCCL INFO comm 0x55afb5b98430 rank 220 nranks 256 cudaDev 4 nvmlDev 4 busId 89000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-10:48:231 [2] NCCL INFO comm 0x56177ecb1390 rank 82 nranks 256 cudaDev 2 nvmlDev 2 busId 44000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-27:51:234 [5] NCCL INFO comm 0x5630a7dc6c30 rank 221 nranks 256 cudaDev 5 nvmlDev 5 busId a8000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-27:49:233 [3] NCCL INFO comm 0x55c4690f0070 rank 219 nranks 256 cudaDev 3 nvmlDev 3 busId 5b000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-10:50:234 [4] NCCL INFO comm 0x559c8031d920 rank 84 nranks 256 cudaDev 4 nvmlDev 4 busId 89000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-10:49:237 [3] NCCL INFO comm 0x55bdf17a0ae0 rank 83 nranks 256 cudaDev 3 nvmlDev 3 busId 5b000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-10:51:232 [5] NCCL INFO comm 0x561a204987c0 rank 85 nranks 256 cudaDev 5 nvmlDev 5 busId a8000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-10:52:233 [6] NCCL INFO comm 0x55627aa105e0 rank 86 nranks 256 cudaDev 6 nvmlDev 6 busId c0000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-10:53:235 [7] NCCL INFO comm 0x55db0440ad00 rank 87 nranks 256 cudaDev 7 nvmlDev 7 busId d8000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-11:50:237 [4] NCCL INFO comm 0x5609e5645100 rank 92 nranks 256 cudaDev 4 nvmlDev 4 busId 89000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-13:48:230 [2] NCCL INFO comm 0x558c86a65ff0 rank 106 nranks 256 cudaDev 2 nvmlDev 2 busId 44000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-11:46:230 [0] NCCL INFO comm 0x55d2f2912980 rank 88 nranks 256 cudaDev 0 nvmlDev 0 busId f000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-11:48:231 [2] NCCL INFO comm 0x558b4f378ef0 rank 90 nranks 256 cudaDev 2 nvmlDev 2 busId 44000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-11:52:233 [6] NCCL INFO comm 0x55a6170fa8b0 rank 94 nranks 256 cudaDev 6 nvmlDev 6 busId c0000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-11:49:232 [3] NCCL INFO comm 0x5581516c6410 rank 91 nranks 256 cudaDev 3 nvmlDev 3 busId 5b000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-11:47:235 [1] NCCL INFO comm 0x55757986be20 rank 89 nranks 256 cudaDev 1 nvmlDev 1 busId 2d000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-11:51:234 [5] NCCL INFO comm 0x55c4ea72dc00 rank 93 nranks 256 cudaDev 5 nvmlDev 5 busId a8000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-11:53:236 [7] NCCL INFO comm 0x55a2727ad640 rank 95 nranks 256 cudaDev 7 nvmlDev 7 busId d8000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-12:46:237 [0] NCCL INFO comm 0x55a37230d110 rank 96 nranks 256 cudaDev 0 nvmlDev 0 busId f000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-12:47:230 [1] NCCL INFO comm 0x560cc685f7b0 rank 97 nranks 256 cudaDev 1 nvmlDev 1 busId 2d000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-12:49:231 [3] NCCL INFO comm 0x5655142fe2e0 rank 99 nranks 256 cudaDev 3 nvmlDev 3 busId 5b000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-13:47:231 [1] NCCL INFO comm 0x562f6743f640 rank 105 nranks 256 cudaDev 1 nvmlDev 1 busId 2d000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-12:48:236 [2] NCCL INFO comm 0x55a43bb83810 rank 98 nranks 256 cudaDev 2 nvmlDev 2 busId 44000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-13:46:235 [0] NCCL INFO comm 0x56406130ef30 rank 104 nranks 256 cudaDev 0 nvmlDev 0 busId f000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-12:51:234 [5] NCCL INFO comm 0x5627d0c94650 rank 101 nranks 256 cudaDev 5 nvmlDev 5 busId a8000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-12:50:232 [4] NCCL INFO comm 0x55a2d0ae4c50 rank 100 nranks 256 cudaDev 4 nvmlDev 4 busId 89000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-12:54:233 [7] NCCL INFO comm 0x55ac2e0d0d90 rank 103 nranks 256 cudaDev 7 nvmlDev 7 busId d8000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-12:52:235 [6] NCCL INFO comm 0x5594deee9d10 rank 102 nranks 256 cudaDev 6 nvmlDev 6 busId c0000 commId 0xc06f6f6fa494e154 - Init START | |
| nccl-alltoall-job-oguz-mpiworker-9:52:234 [6] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-9:51:236 [5] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-9:53:235 [7] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-10:46:230 [0] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-9:50:237 [4] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-9:49:232 [3] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-10:47:236 [1] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-9:48:233 [2] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-10:49:237 [3] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-10:48:231 [2] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-10:50:234 [4] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-10:51:232 [5] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-10:52:233 [6] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-10:53:235 [7] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-8:52:236 [6] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-9:47:230 [1] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-9:46:231 [0] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-11:46:230 [0] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-8:53:230 [7] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-8:51:234 [5] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-11:47:235 [1] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-11:49:232 [3] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-11:48:231 [2] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-11:50:237 [4] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-11:51:234 [5] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-11:52:233 [6] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-11:53:236 [7] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-12:47:230 [1] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-12:51:234 [5] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-13:47:231 [1] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-6:53:230 [7] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-7:50:234 [4] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-7:51:232 [5] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-6:52:233 [6] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-7:52:231 [6] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-6:51:235 [5] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-12:54:233 [7] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-7:54:236 [7] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-12:49:231 [3] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-12:46:237 [0] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-12:48:236 [2] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-8:46:231 [0] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-12:50:232 [4] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-14:49:232 [3] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-13:49:234 [3] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-12:52:235 [6] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-14:46:234 [0] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-13:46:235 [0] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-8:47:232 [1] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-7:47:233 [1] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-14:48:237 [2] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-13:48:230 [2] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-8:50:233 [4] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-7:48:237 [2] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-6:50:237 [4] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-14:47:233 [1] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-13:50:236 [4] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-8:49:237 [3] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-7:49:235 [3] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-22:52:235 [6] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-14:50:236 [4] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-13:52:233 [6] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-8:48:235 [2] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-7:46:230 [0] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-22:53:234 [7] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-15:53:236 [7] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-14:51:230 [5] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-13:51:237 [5] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-22:51:233 [5] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-15:51:233 [5] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-16:53:236 [7] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-18:49:234 [3] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-17:46:234 [0] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-14:52:235 [6] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-13:53:232 [7] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-22:50:232 [4] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-15:52:237 [6] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-16:52:237 [6] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-18:52:237 [6] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-17:47:231 [1] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-20:49:235 [3] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-19:52:234 [6] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-21:46:230 [0] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-14:53:231 [7] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-22:49:237 [3] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-15:46:230 [0] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-16:51:235 [5] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-18:51:235 [5] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-17:48:230 [2] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-20:48:236 [2] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-19:53:231 [7] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-21:47:234 [1] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-6:49:234 [3] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-5:53:233 [7] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-22:46:231 [0] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-15:50:234 [4] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-16:47:234 [1] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-18:48:233 [2] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-17:49:233 [3] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-20:52:232 [6] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-19:51:235 [5] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-21:48:232 [2] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-6:48:231 [2] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-23:46:230 [0] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-22:48:236 [2] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-15:47:232 [1] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-16:49:233 [3] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-18:50:236 [4] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-17:50:235 [4] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-20:51:230 [5] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-19:46:230 [0] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-21:49:236 [3] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-6:47:232 [1] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-23:47:233 [1] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-22:47:230 [1] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-15:49:235 [3] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-16:50:232 [4] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-18:53:231 [7] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-17:52:237 [6] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-20:53:231 [7] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-19:50:237 [4] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-21:52:231 [6] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-6:46:236 [0] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-23:48:232 [2] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-15:48:231 [2] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-16:46:230 [0] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-18:47:232 [1] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-17:53:236 [7] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-20:50:237 [4] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-19:49:236 [3] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-21:50:237 [4] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-23:49:236 [3] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-5:52:237 [6] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-25:53:231 [7] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-18:46:230 [0] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-17:51:232 [5] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-20:47:234 [1] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-19:47:232 [1] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-21:51:235 [5] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-23:50:234 [4] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-5:51:236 [5] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-26:46:231 [0] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-24:46:237 [0] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-20:46:233 [0] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-19:48:233 [2] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-21:53:233 [7] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-23:51:235 [5] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-5:50:231 [4] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-26:47:230 [1] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-24:47:231 [1] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-23:52:237 [6] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-5:49:232 [3] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-26:48:235 [2] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-24:48:230 [2] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-23:53:231 [7] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-5:47:235 [1] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-26:49:233 [3] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-24:52:232 [6] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-5:48:234 [2] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-26:51:236 [5] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-24:50:236 [4] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-25:46:230 [0] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-26:52:237 [6] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-24:51:234 [5] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-26:50:234 [4] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-24:53:235 [7] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-25:47:234 [1] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-26:53:232 [7] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-24:49:233 [3] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-25:51:232 [5] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-16:48:231 [2] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-25:48:237 [2] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-25:52:236 [6] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-27:46:231 [0] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-25:50:235 [4] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-25:49:233 [3] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-5:46:230 [0] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-27:47:236 [1] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-27:48:235 [2] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-27:49:233 [3] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-27:50:232 [4] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-27:51:234 [5] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-27:53:237 [7] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-27:52:230 [6] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-28:46:233 [0] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-28:47:230 [1] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-28:48:231 [2] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-28:49:237 [3] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-28:52:232 [6] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-28:51:236 [5] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-28:50:234 [4] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-28:53:235 [7] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-29:52:237 [6] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-29:46:232 [0] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-29:51:230 [5] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-29:53:235 [7] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-29:48:231 [2] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-29:50:234 [4] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-29:47:233 [1] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-29:49:236 [3] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-30:46:230 [0] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-31:53:231 [7] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-30:47:232 [1] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-31:52:233 [6] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-30:48:237 [2] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-31:51:236 [5] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-30:49:234 [3] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-31:49:235 [3] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-30:50:236 [4] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-30:51:233 [5] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-30:52:235 [6] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-0:53:234 [7] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-31:46:234 [0] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-30:53:231 [7] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-0:47:237 [1] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-31:50:237 [4] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-0:49:236 [3] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-31:48:230 [2] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-0:51:232 [5] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-31:47:232 [1] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-0:52:235 [6] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-0:46:231 [0] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-0:48:233 [2] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-0:50:238 [4] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-1:46:230 [0] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-1:47:237 [1] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-1:49:236 [3] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-1:48:231 [2] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-4:53:235 [7] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-1:50:233 [4] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-1:52:235 [6] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-1:51:232 [5] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-2:53:230 [7] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-1:54:234 [7] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-3:52:237 [6] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-3:51:234 [5] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-2:51:234 [5] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-3:50:236 [4] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-3:53:235 [7] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-3:48:231 [2] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-3:49:232 [3] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-3:47:230 [1] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-2:47:235 [1] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-3:46:233 [0] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-2:52:233 [6] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-2:46:232 [0] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-4:52:237 [6] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-2:48:237 [2] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-4:50:230 [4] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-2:49:236 [3] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-4:49:234 [3] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-2:50:231 [4] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-4:51:236 [5] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-4:48:233 [2] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-4:46:231 [0] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-4:47:232 [1] NCCL INFO NCCL_TOPO_FILE set by environment to /h100/topo.xml | |
| nccl-alltoall-job-oguz-mpiworker-16:53:236 [7] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-9:51:236 [5] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-9:53:235 [7] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-8:51:234 [5] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-8:53:230 [7] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-8:52:236 [6] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-9:49:232 [3] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-9:47:230 [1] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-9:52:234 [6] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-10:46:230 [0] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-9:50:237 [4] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-9:48:233 [2] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-9:46:231 [0] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-8:46:231 [0] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-10:50:234 [4] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-11:46:230 [0] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-10:52:233 [6] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-8:48:235 [2] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-10:47:236 [1] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-10:51:232 [5] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-10:49:237 [3] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-11:48:231 [2] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-8:49:237 [3] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-10:53:235 [7] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-8:50:233 [4] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-8:47:232 [1] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-12:46:237 [0] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-11:49:232 [3] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-11:53:236 [7] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-13:48:230 [2] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-12:50:232 [4] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-13:46:235 [0] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-12:52:235 [6] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-22:53:234 [7] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-22:49:237 [3] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-17:52:237 [6] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-10:48:231 [2] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-12:54:233 [7] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-12:47:230 [1] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-12:51:234 [5] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-22:51:233 [5] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-13:52:233 [6] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-17:50:235 [4] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-17:46:234 [0] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-6:50:237 [4] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-7:50:234 [4] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-7:52:231 [6] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-6:52:233 [6] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-12:49:231 [3] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-23:48:232 [2] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-13:49:234 [3] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-16:46:230 [0] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-14:49:232 [3] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-23:46:230 [0] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-6:51:235 [5] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-25:53:231 [7] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-7:48:237 [2] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-11:52:233 [6] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-14:47:233 [1] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-11:47:235 [1] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-23:50:234 [4] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-13:47:231 [1] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-11:50:237 [4] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-13:51:237 [5] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-22:52:235 [6] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-17:47:231 [1] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-13:53:232 [7] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-22:47:230 [1] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-16:48:231 [2] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-17:53:236 [7] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-23:52:237 [6] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-17:49:233 [3] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-7:47:233 [1] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-6:48:231 [2] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-22:50:232 [4] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-7:51:232 [5] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-6:46:236 [0] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-7:54:236 [7] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-6:53:230 [7] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-22:46:231 [0] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-22:48:236 [2] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-14:53:231 [7] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-13:50:236 [4] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-15:50:234 [4] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-12:48:236 [2] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-14:46:234 [0] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-15:46:230 [0] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-17:51:232 [5] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-23:47:233 [1] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-15:47:232 [1] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-17:48:230 [2] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-16:52:237 [6] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-20:52:232 [6] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-15:52:237 [6] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-16:50:232 [4] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-7:49:235 [3] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-19:53:231 [7] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-6:49:234 [3] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-14:50:236 [4] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-15:53:236 [7] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-6:47:232 [1] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-27:46:231 [0] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-23:53:231 [7] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-25:47:234 [1] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-14:48:237 [2] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-15:51:233 [5] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-23:51:235 [5] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-21:50:237 [4] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-18:49:234 [3] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-14:52:235 [6] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-15:49:235 [3] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-23:49:236 [3] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-21:52:231 [6] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-18:51:235 [5] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-16:49:233 [3] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-20:46:233 [0] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-21:48:232 [2] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-18:47:232 [1] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-20:49:235 [3] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-21:46:230 [0] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-18:53:231 [7] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-20:51:230 [5] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-16:47:234 [1] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-26:53:232 [7] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-21:49:236 [3] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-16:51:235 [5] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-26:47:230 [1] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-14:51:230 [5] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-26:49:233 [3] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-15:48:231 [2] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-5:52:237 [6] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-27:50:232 [4] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-25:51:232 [5] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-27:48:235 [2] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-7:46:230 [0] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-21:47:234 [1] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-26:51:236 [5] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-21:53:233 [7] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-20:53:231 [7] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-20:47:234 [1] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-18:46:230 [0] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-19:51:235 [5] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-25:49:233 [3] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-25:50:235 [4] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-25:46:230 [0] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-26:46:231 [0] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-25:48:237 [2] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-27:52:230 [6] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-11:51:234 [5] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-26:50:234 [4] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-21:51:235 [5] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-26:48:235 [2] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-19:49:236 [3] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-19:50:237 [4] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-26:52:237 [6] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-19:46:230 [0] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-19:47:232 [1] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-19:52:234 [6] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-20:48:236 [2] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-28:52:232 [6] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-20:50:237 [4] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-5:50:231 [4] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-28:46:233 [0] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-28:48:231 [2] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-28:50:234 [4] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-5:46:230 [0] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-27:53:237 [7] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-25:52:236 [6] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-30:47:232 [1] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-27:47:236 [1] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-5:49:232 [3] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-18:50:236 [4] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-28:51:236 [5] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-28:53:235 [7] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-28:47:230 [1] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-28:49:237 [3] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-30:51:233 [5] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-30:49:234 [3] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-19:48:233 [2] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-0:52:235 [6] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-0:48:233 [2] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-5:47:235 [1] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-27:51:234 [5] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-27:49:233 [3] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-30:53:231 [7] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-0:50:238 [4] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-5:51:236 [5] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-5:53:233 [7] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-5:48:234 [2] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-30:46:230 [0] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-0:46:231 [0] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-30:50:236 [4] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-30:52:235 [6] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-30:48:237 [2] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-29:53:235 [7] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-29:51:230 [5] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-0:51:232 [5] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-18:52:237 [6] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-0:47:237 [1] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-0:53:234 [7] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-16:53:236 [7] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-16:53:236 [7] NCCL INFO Setting affinity for GPU 7 to ffffffff,ffffff00,00000000,0000ffff,ffffffff,ff000000,00000000 | |
| nccl-alltoall-job-oguz-mpiworker-0:49:236 [3] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-16:53:236 [7] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-9:51:236 [5] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-9:51:236 [5] NCCL INFO Setting affinity for GPU 5 to ffffffff,ffffff00,00000000,0000ffff,ffffffff,ff000000,00000000 | |
| nccl-alltoall-job-oguz-mpiworker-9:51:236 [5] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-29:48:231 [2] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-29:46:232 [0] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-29:52:237 [6] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-9:53:235 [7] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-9:53:235 [7] NCCL INFO Setting affinity for GPU 7 to ffffffff,ffffff00,00000000,0000ffff,ffffffff,ff000000,00000000 | |
| nccl-alltoall-job-oguz-mpiworker-3:49:232 [3] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-9:53:235 [7] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-4:53:235 [7] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-3:51:234 [5] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-3:53:235 [7] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-4:49:234 [3] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-3:47:230 [1] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-8:51:234 [5] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-8:51:234 [5] NCCL INFO Setting affinity for GPU 5 to ffffffff,ffffff00,00000000,0000ffff,ffffffff,ff000000,00000000 | |
| nccl-alltoall-job-oguz-mpiworker-8:51:234 [5] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-4:47:232 [1] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-29:47:233 [1] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-8:53:230 [7] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-8:53:230 [7] NCCL INFO Setting affinity for GPU 7 to ffffffff,ffffff00,00000000,0000ffff,ffffffff,ff000000,00000000 | |
| nccl-alltoall-job-oguz-mpiworker-8:53:230 [7] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-8:52:236 [6] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-8:52:236 [6] NCCL INFO Setting affinity for GPU 6 to ffffffff,ffffff00,00000000,0000ffff,ffffffff,ff000000,00000000 | |
| nccl-alltoall-job-oguz-mpiworker-24:53:235 [7] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-8:52:236 [6] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-29:49:236 [3] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-9:49:232 [3] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-9:49:232 [3] NCCL INFO Setting affinity for GPU 3 to ff,ffffffff,ffff0000,00000000,00ffffff,ffffffff | |
| nccl-alltoall-job-oguz-mpiworker-29:50:234 [4] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-24:47:231 [1] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-3:48:231 [2] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-9:49:232 [3] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-3:46:233 [0] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-24:51:234 [5] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-18:48:233 [2] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-3:50:236 [4] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-3:52:237 [6] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-4:50:230 [4] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-9:47:230 [1] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-9:47:230 [1] NCCL INFO Setting affinity for GPU 1 to ff,ffffffff,ffff0000,00000000,00ffffff,ffffffff | |
| nccl-alltoall-job-oguz-mpiworker-9:52:234 [6] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-9:52:234 [6] NCCL INFO Setting affinity for GPU 6 to ffffffff,ffffff00,00000000,0000ffff,ffffffff,ff000000,00000000 | |
| nccl-alltoall-job-oguz-mpiworker-10:46:230 [0] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-10:46:230 [0] NCCL INFO Setting affinity for GPU 0 to ff,ffffffff,ffff0000,00000000,00ffffff,ffffffff | |
| nccl-alltoall-job-oguz-mpiworker-10:46:230 [0] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-9:47:230 [1] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-9:52:234 [6] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-1:52:235 [6] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-4:52:237 [6] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-4:51:236 [5] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-31:49:235 [3] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-9:50:237 [4] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-9:50:237 [4] NCCL INFO Setting affinity for GPU 4 to ffffffff,ffffff00,00000000,0000ffff,ffffffff,ff000000,00000000 | |
| nccl-alltoall-job-oguz-mpiworker-4:46:231 [0] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-9:48:233 [2] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-4:48:233 [2] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-31:53:231 [7] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-9:48:233 [2] NCCL INFO Setting affinity for GPU 2 to ff,ffffffff,ffff0000,00000000,00ffffff,ffffffff | |
| nccl-alltoall-job-oguz-mpiworker-9:48:233 [2] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-24:48:230 [2] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-9:46:231 [0] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-9:46:231 [0] NCCL INFO Setting affinity for GPU 0 to ff,ffffffff,ffff0000,00000000,00ffffff,ffffffff | |
| nccl-alltoall-job-oguz-mpiworker-9:50:237 [4] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-9:46:231 [0] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-24:46:237 [0] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-8:46:231 [0] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-8:46:231 [0] NCCL INFO Setting affinity for GPU 0 to ff,ffffffff,ffff0000,00000000,00ffffff,ffffffff | |
| nccl-alltoall-job-oguz-mpiworker-8:46:231 [0] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-24:50:236 [4] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-8:48:235 [2] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-8:48:235 [2] NCCL INFO Setting affinity for GPU 2 to ff,ffffffff,ffff0000,00000000,00ffffff,ffffffff | |
| nccl-alltoall-job-oguz-mpiworker-10:50:234 [4] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-10:50:234 [4] NCCL INFO Setting affinity for GPU 4 to ffffffff,ffffff00,00000000,0000ffff,ffffffff,ff000000,00000000 | |
| nccl-alltoall-job-oguz-mpiworker-8:48:235 [2] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-11:46:230 [0] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-11:46:230 [0] NCCL INFO Setting affinity for GPU 0 to ff,ffffffff,ffff0000,00000000,00ffffff,ffffffff | |
| nccl-alltoall-job-oguz-mpiworker-31:47:232 [1] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-11:46:230 [0] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-1:50:233 [4] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-10:52:233 [6] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-10:52:233 [6] NCCL INFO Setting affinity for GPU 6 to ffffffff,ffffff00,00000000,0000ffff,ffffffff,ff000000,00000000 | |
| nccl-alltoall-job-oguz-mpiworker-24:49:233 [3] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-1:48:231 [2] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-10:50:234 [4] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-31:51:236 [5] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-1:49:236 [3] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-31:46:234 [0] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-24:52:232 [6] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-1:47:237 [1] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-10:52:233 [6] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-1:54:234 [7] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-1:46:230 [0] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-10:47:236 [1] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-10:47:236 [1] NCCL INFO Setting affinity for GPU 1 to ff,ffffffff,ffff0000,00000000,00ffffff,ffffffff | |
| nccl-alltoall-job-oguz-mpiworker-10:51:232 [5] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-10:51:232 [5] NCCL INFO Setting affinity for GPU 5 to ffffffff,ffffff00,00000000,0000ffff,ffffffff,ff000000,00000000 | |
| nccl-alltoall-job-oguz-mpiworker-11:48:231 [2] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-11:48:231 [2] NCCL INFO Setting affinity for GPU 2 to ff,ffffffff,ffff0000,00000000,00ffffff,ffffffff | |
| nccl-alltoall-job-oguz-mpiworker-10:47:236 [1] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-10:51:232 [5] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-10:53:235 [7] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-10:53:235 [7] NCCL INFO Setting affinity for GPU 7 to ffffffff,ffffff00,00000000,0000ffff,ffffffff,ff000000,00000000 | |
| nccl-alltoall-job-oguz-mpiworker-11:48:231 [2] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-8:49:237 [3] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-8:49:237 [3] NCCL INFO Setting affinity for GPU 3 to ff,ffffffff,ffff0000,00000000,00ffffff,ffffffff | |
| nccl-alltoall-job-oguz-mpiworker-8:50:233 [4] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-8:50:233 [4] NCCL INFO Setting affinity for GPU 4 to ffffffff,ffffff00,00000000,0000ffff,ffffffff,ff000000,00000000 | |
| nccl-alltoall-job-oguz-mpiworker-10:49:237 [3] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-10:49:237 [3] NCCL INFO Setting affinity for GPU 3 to ff,ffffffff,ffff0000,00000000,00ffffff,ffffffff | |
| nccl-alltoall-job-oguz-mpiworker-10:53:235 [7] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-8:49:237 [3] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-10:49:237 [3] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-8:50:233 [4] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-8:47:232 [1] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-8:47:232 [1] NCCL INFO Setting affinity for GPU 1 to ff,ffffffff,ffff0000,00000000,00ffffff,ffffffff | |
| nccl-alltoall-job-oguz-mpiworker-12:46:237 [0] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-12:46:237 [0] NCCL INFO Setting affinity for GPU 0 to ff,ffffffff,ffff0000,00000000,00ffffff,ffffffff | |
| nccl-alltoall-job-oguz-mpiworker-12:46:237 [0] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-13:48:230 [2] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-13:48:230 [2] NCCL INFO Setting affinity for GPU 2 to ff,ffffffff,ffff0000,00000000,00ffffff,ffffffff | |
| nccl-alltoall-job-oguz-mpiworker-8:47:232 [1] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-1:51:232 [5] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-12:50:232 [4] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-12:50:232 [4] NCCL INFO Setting affinity for GPU 4 to ffffffff,ffffff00,00000000,0000ffff,ffffffff,ff000000,00000000 | |
| nccl-alltoall-job-oguz-mpiworker-12:52:235 [6] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-12:52:235 [6] NCCL INFO Setting affinity for GPU 6 to ffffffff,ffffff00,00000000,0000ffff,ffffffff,ff000000,00000000 | |
| nccl-alltoall-job-oguz-mpiworker-13:48:230 [2] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-13:46:235 [0] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-13:46:235 [0] NCCL INFO Setting affinity for GPU 0 to ff,ffffffff,ffff0000,00000000,00ffffff,ffffffff | |
| nccl-alltoall-job-oguz-mpiworker-13:46:235 [0] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-12:50:232 [4] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-17:52:237 [6] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-17:52:237 [6] NCCL INFO Setting affinity for GPU 6 to ffffffff,ffffff00,00000000,0000ffff,ffffffff,ff000000,00000000 | |
| nccl-alltoall-job-oguz-mpiworker-12:52:235 [6] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-11:53:236 [7] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-11:53:236 [7] NCCL INFO Setting affinity for GPU 7 to ffffffff,ffffff00,00000000,0000ffff,ffffffff,ff000000,00000000 | |
| nccl-alltoall-job-oguz-mpiworker-22:53:234 [7] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-22:53:234 [7] NCCL INFO Setting affinity for GPU 7 to ffffffff,ffffff00,00000000,0000ffff,ffffffff,ff000000,00000000 | |
| nccl-alltoall-job-oguz-mpiworker-12:54:233 [7] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-12:54:233 [7] NCCL INFO Setting affinity for GPU 7 to ffffffff,ffffff00,00000000,0000ffff,ffffffff,ff000000,00000000 | |
| nccl-alltoall-job-oguz-mpiworker-11:53:236 [7] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-22:53:234 [7] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-22:49:237 [3] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-22:49:237 [3] NCCL INFO Setting affinity for GPU 3 to ff,ffffffff,ffff0000,00000000,00ffffff,ffffffff | |
| nccl-alltoall-job-oguz-mpiworker-17:52:237 [6] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-17:46:234 [0] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-17:46:234 [0] NCCL INFO Setting affinity for GPU 0 to ff,ffffffff,ffff0000,00000000,00ffffff,ffffffff | |
| nccl-alltoall-job-oguz-mpiworker-12:47:230 [1] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-17:50:235 [4] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-17:50:235 [4] NCCL INFO Setting affinity for GPU 4 to ffffffff,ffffff00,00000000,0000ffff,ffffffff,ff000000,00000000 | |
| nccl-alltoall-job-oguz-mpiworker-12:47:230 [1] NCCL INFO Setting affinity for GPU 1 to ff,ffffffff,ffff0000,00000000,00ffffff,ffffffff | |
| nccl-alltoall-job-oguz-mpiworker-22:51:233 [5] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-22:51:233 [5] NCCL INFO Setting affinity for GPU 5 to ffffffff,ffffff00,00000000,0000ffff,ffffffff,ff000000,00000000 | |
| nccl-alltoall-job-oguz-mpiworker-6:50:237 [4] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-6:50:237 [4] NCCL INFO Setting affinity for GPU 4 to ffffffff,ffffff00,00000000,0000ffff,ffffffff,ff000000,00000000 | |
| nccl-alltoall-job-oguz-mpiworker-17:46:234 [0] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-12:54:233 [7] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-6:52:233 [6] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-6:52:233 [6] NCCL INFO Setting affinity for GPU 6 to ffffffff,ffffff00,00000000,0000ffff,ffffffff,ff000000,00000000 | |
| nccl-alltoall-job-oguz-mpiworker-11:49:232 [3] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-11:49:232 [3] NCCL INFO Setting affinity for GPU 3 to ff,ffffffff,ffff0000,00000000,00ffffff,ffffffff | |
| nccl-alltoall-job-oguz-mpiworker-22:49:237 [3] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-17:50:235 [4] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-12:51:234 [5] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-12:51:234 [5] NCCL INFO Setting affinity for GPU 5 to ffffffff,ffffff00,00000000,0000ffff,ffffffff,ff000000,00000000 | |
| nccl-alltoall-job-oguz-mpiworker-6:50:237 [4] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-13:52:233 [6] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-13:52:233 [6] NCCL INFO Setting affinity for GPU 6 to ffffffff,ffffff00,00000000,0000ffff,ffffffff,ff000000,00000000 | |
| nccl-alltoall-job-oguz-mpiworker-7:50:234 [4] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-7:50:234 [4] NCCL INFO Setting affinity for GPU 4 to ffffffff,ffffff00,00000000,0000ffff,ffffffff,ff000000,00000000 | |
| nccl-alltoall-job-oguz-mpiworker-22:51:233 [5] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-12:51:234 [5] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-6:52:233 [6] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-13:52:233 [6] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-7:52:231 [6] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-11:49:232 [3] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-16:46:230 [0] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-16:46:230 [0] NCCL INFO Setting affinity for GPU 0 to ff,ffffffff,ffff0000,00000000,00ffffff,ffffffff | |
| nccl-alltoall-job-oguz-mpiworker-31:52:233 [6] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-12:47:230 [1] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-7:52:231 [6] NCCL INFO Setting affinity for GPU 6 to ffffffff,ffffff00,00000000,0000ffff,ffffffff,ff000000,00000000 | |
| nccl-alltoall-job-oguz-mpiworker-16:46:230 [0] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-23:48:232 [2] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-23:48:232 [2] NCCL INFO Setting affinity for GPU 2 to ff,ffffffff,ffff0000,00000000,00ffffff,ffffffff | |
| nccl-alltoall-job-oguz-mpiworker-14:49:232 [3] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-14:49:232 [3] NCCL INFO Setting affinity for GPU 3 to ff,ffffffff,ffff0000,00000000,00ffffff,ffffffff | |
| nccl-alltoall-job-oguz-mpiworker-12:49:231 [3] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-12:49:231 [3] NCCL INFO Setting affinity for GPU 3 to ff,ffffffff,ffff0000,00000000,00ffffff,ffffffff | |
| nccl-alltoall-job-oguz-mpiworker-7:50:234 [4] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-23:48:232 [2] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-25:53:231 [7] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-25:53:231 [7] NCCL INFO Setting affinity for GPU 7 to ffffffff,ffffff00,00000000,0000ffff,ffffffff,ff000000,00000000 | |
| nccl-alltoall-job-oguz-mpiworker-13:49:234 [3] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-13:49:234 [3] NCCL INFO Setting affinity for GPU 3 to ff,ffffffff,ffff0000,00000000,00ffffff,ffffffff | |
| nccl-alltoall-job-oguz-mpiworker-11:47:235 [1] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-11:47:235 [1] NCCL INFO Setting affinity for GPU 1 to ff,ffffffff,ffff0000,00000000,00ffffff,ffffffff | |
| nccl-alltoall-job-oguz-mpiworker-12:49:231 [3] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-7:52:231 [6] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-23:46:230 [0] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-23:46:230 [0] NCCL INFO Setting affinity for GPU 0 to ff,ffffffff,ffff0000,00000000,00ffffff,ffffffff | |
| nccl-alltoall-job-oguz-mpiworker-11:52:233 [6] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-11:52:233 [6] NCCL INFO Setting affinity for GPU 6 to ffffffff,ffffff00,00000000,0000ffff,ffffffff,ff000000,00000000 | |
| nccl-alltoall-job-oguz-mpiworker-14:49:232 [3] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-6:51:235 [5] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-6:51:235 [5] NCCL INFO Setting affinity for GPU 5 to ffffffff,ffffff00,00000000,0000ffff,ffffffff,ff000000,00000000 | |
| nccl-alltoall-job-oguz-mpiworker-16:48:231 [2] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-16:48:231 [2] NCCL INFO Setting affinity for GPU 2 to ff,ffffffff,ffff0000,00000000,00ffffff,ffffffff | |
| nccl-alltoall-job-oguz-mpiworker-22:52:235 [6] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-22:52:235 [6] NCCL INFO Setting affinity for GPU 6 to ffffffff,ffffff00,00000000,0000ffff,ffffffff,ff000000,00000000 | |
| nccl-alltoall-job-oguz-mpiworker-7:48:237 [2] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-7:48:237 [2] NCCL INFO Setting affinity for GPU 2 to ff,ffffffff,ffff0000,00000000,00ffffff,ffffffff | |
| nccl-alltoall-job-oguz-mpiworker-23:50:234 [4] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-23:50:234 [4] NCCL INFO Setting affinity for GPU 4 to ffffffff,ffffff00,00000000,0000ffff,ffffffff,ff000000,00000000 | |
| nccl-alltoall-job-oguz-mpiworker-11:50:237 [4] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-11:50:237 [4] NCCL INFO Setting affinity for GPU 4 to ffffffff,ffffff00,00000000,0000ffff,ffffffff,ff000000,00000000 | |
| nccl-alltoall-job-oguz-mpiworker-14:47:233 [1] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-14:47:233 [1] NCCL INFO Setting affinity for GPU 1 to ff,ffffffff,ffff0000,00000000,00ffffff,ffffffff | |
| nccl-alltoall-job-oguz-mpiworker-16:48:231 [2] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-22:47:230 [1] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-22:47:230 [1] NCCL INFO Setting affinity for GPU 1 to ff,ffffffff,ffff0000,00000000,00ffffff,ffffffff | |
| nccl-alltoall-job-oguz-mpiworker-25:53:231 [7] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-17:47:231 [1] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-17:47:231 [1] NCCL INFO Setting affinity for GPU 1 to ff,ffffffff,ffff0000,00000000,00ffffff,ffffffff | |
| nccl-alltoall-job-oguz-mpiworker-7:48:237 [2] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-23:46:230 [0] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-11:52:233 [6] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-14:47:233 [1] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-22:50:232 [4] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-22:50:232 [4] NCCL INFO Setting affinity for GPU 4 to ffffffff,ffffff00,00000000,0000ffff,ffffffff,ff000000,00000000 | |
| nccl-alltoall-job-oguz-mpiworker-17:53:236 [7] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-17:53:236 [7] NCCL INFO Setting affinity for GPU 7 to ffffffff,ffffff00,00000000,0000ffff,ffffffff,ff000000,00000000 | |
| nccl-alltoall-job-oguz-mpiworker-13:47:231 [1] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-13:47:231 [1] NCCL INFO Setting affinity for GPU 1 to ff,ffffffff,ffff0000,00000000,00ffffff,ffffffff | |
| nccl-alltoall-job-oguz-mpiworker-10:48:231 [2] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-10:48:231 [2] NCCL INFO Setting affinity for GPU 2 to ff,ffffffff,ffff0000,00000000,00ffffff,ffffffff | |
| nccl-alltoall-job-oguz-mpiworker-6:51:235 [5] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-7:51:232 [5] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-7:51:232 [5] NCCL INFO Setting affinity for GPU 5 to ffffffff,ffffff00,00000000,0000ffff,ffffffff,ff000000,00000000 | |
| nccl-alltoall-job-oguz-mpiworker-23:50:234 [4] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-11:47:235 [1] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-14:53:231 [7] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-14:53:231 [7] NCCL INFO Setting affinity for GPU 7 to ffffffff,ffffff00,00000000,0000ffff,ffffffff,ff000000,00000000 | |
| nccl-alltoall-job-oguz-mpiworker-22:52:235 [6] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-17:53:236 [7] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-13:53:232 [7] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-13:53:232 [7] NCCL INFO Setting affinity for GPU 7 to ffffffff,ffffff00,00000000,0000ffff,ffffffff,ff000000,00000000 | |
| nccl-alltoall-job-oguz-mpiworker-10:48:231 [2] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-6:46:236 [0] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-6:46:236 [0] NCCL INFO Setting affinity for GPU 0 to ff,ffffffff,ffff0000,00000000,00ffffff,ffffffff | |
| nccl-alltoall-job-oguz-mpiworker-16:50:232 [4] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-16:50:232 [4] NCCL INFO Setting affinity for GPU 4 to ffffffff,ffffff00,00000000,0000ffff,ffffffff,ff000000,00000000 | |
| nccl-alltoall-job-oguz-mpiworker-15:50:234 [4] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-15:50:234 [4] NCCL INFO Setting affinity for GPU 4 to ffffffff,ffffff00,00000000,0000ffff,ffffffff,ff000000,00000000 | |
| nccl-alltoall-job-oguz-mpiworker-7:47:233 [1] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-7:47:233 [1] NCCL INFO Setting affinity for GPU 1 to ff,ffffffff,ffff0000,00000000,00ffffff,ffffffff | |
| nccl-alltoall-job-oguz-mpiworker-23:52:237 [6] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-23:52:237 [6] NCCL INFO Setting affinity for GPU 6 to ffffffff,ffffff00,00000000,0000ffff,ffffffff,ff000000,00000000 | |
| nccl-alltoall-job-oguz-mpiworker-11:50:237 [4] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-14:53:231 [7] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-22:47:230 [1] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-17:49:233 [3] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-17:49:233 [3] NCCL INFO Setting affinity for GPU 3 to ff,ffffffff,ffff0000,00000000,00ffffff,ffffffff | |
| nccl-alltoall-job-oguz-mpiworker-13:51:237 [5] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-13:51:237 [5] NCCL INFO Setting affinity for GPU 5 to ffffffff,ffffff00,00000000,0000ffff,ffffffff,ff000000,00000000 | |
| nccl-alltoall-job-oguz-mpiworker-6:48:231 [2] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-6:48:231 [2] NCCL INFO Setting affinity for GPU 2 to ff,ffffffff,ffff0000,00000000,00ffffff,ffffffff | |
| nccl-alltoall-job-oguz-mpiworker-16:52:237 [6] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-16:52:237 [6] NCCL INFO Setting affinity for GPU 6 to ffffffff,ffffff00,00000000,0000ffff,ffffffff,ff000000,00000000 | |
| nccl-alltoall-job-oguz-mpiworker-15:46:230 [0] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-15:46:230 [0] NCCL INFO Setting affinity for GPU 0 to ff,ffffffff,ffff0000,00000000,00ffffff,ffffffff | |
| nccl-alltoall-job-oguz-mpiworker-20:52:232 [6] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-20:52:232 [6] NCCL INFO Setting affinity for GPU 6 to ffffffff,ffffff00,00000000,0000ffff,ffffffff,ff000000,00000000 | |
| nccl-alltoall-job-oguz-mpiworker-19:53:231 [7] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-19:53:231 [7] NCCL INFO Setting affinity for GPU 7 to ffffffff,ffffff00,00000000,0000ffff,ffffffff,ff000000,00000000 | |
| nccl-alltoall-job-oguz-mpiworker-7:51:232 [5] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-23:52:237 [6] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-14:46:234 [0] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-14:46:234 [0] NCCL INFO Setting affinity for GPU 0 to ff,ffffffff,ffff0000,00000000,00ffffff,ffffffff | |
| nccl-alltoall-job-oguz-mpiworker-22:46:231 [0] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-22:46:231 [0] NCCL INFO Setting affinity for GPU 0 to ff,ffffffff,ffff0000,00000000,00ffffff,ffffffff | |
| nccl-alltoall-job-oguz-mpiworker-17:47:231 [1] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-13:53:232 [7] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-6:46:236 [0] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-16:50:232 [4] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-15:46:230 [0] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-27:46:231 [0] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-27:46:231 [0] NCCL INFO Setting affinity for GPU 0 to ff,ffffffff,ffff0000,00000000,00ffffff,ffffffff | |
| nccl-alltoall-job-oguz-mpiworker-19:53:231 [7] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-7:47:233 [1] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-23:47:233 [1] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-23:47:233 [1] NCCL INFO Setting affinity for GPU 1 to ff,ffffffff,ffff0000,00000000,00ffffff,ffffffff | |
| nccl-alltoall-job-oguz-mpiworker-14:46:234 [0] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-22:48:236 [2] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-22:48:236 [2] NCCL INFO Setting affinity for GPU 2 to ff,ffffffff,ffff0000,00000000,00ffffff,ffffffff | |
| nccl-alltoall-job-oguz-mpiworker-17:49:233 [3] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-13:51:237 [5] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-6:48:231 [2] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-16:52:237 [6] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-15:47:232 [1] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-15:47:232 [1] NCCL INFO Setting affinity for GPU 1 to ff,ffffffff,ffff0000,00000000,00ffffff,ffffffff | |
| nccl-alltoall-job-oguz-mpiworker-27:46:231 [0] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-20:52:232 [6] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-31:48:230 [2] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-21:48:232 [2] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-21:48:232 [2] NCCL INFO Setting affinity for GPU 2 to ff,ffffffff,ffff0000,00000000,00ffffff,ffffffff | |
| nccl-alltoall-job-oguz-mpiworker-25:47:234 [1] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-25:47:234 [1] NCCL INFO Setting affinity for GPU 1 to ff,ffffffff,ffff0000,00000000,00ffffff,ffffffff | |
| nccl-alltoall-job-oguz-mpiworker-18:49:234 [3] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-18:49:234 [3] NCCL INFO Setting affinity for GPU 3 to ff,ffffffff,ffff0000,00000000,00ffffff,ffffffff | |
| nccl-alltoall-job-oguz-mpiworker-7:54:236 [7] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-7:54:236 [7] NCCL INFO Setting affinity for GPU 7 to ffffffff,ffffff00,00000000,0000ffff,ffffffff,ff000000,00000000 | |
| nccl-alltoall-job-oguz-mpiworker-23:47:233 [1] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-14:50:236 [4] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-14:50:236 [4] NCCL INFO Setting affinity for GPU 4 to ffffffff,ffffff00,00000000,0000ffff,ffffffff,ff000000,00000000 | |
| nccl-alltoall-job-oguz-mpiworker-22:50:232 [4] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-17:51:232 [5] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-17:51:232 [5] NCCL INFO Setting affinity for GPU 5 to ffffffff,ffffff00,00000000,0000ffff,ffffffff,ff000000,00000000 | |
| nccl-alltoall-job-oguz-mpiworker-13:49:234 [3] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-6:53:230 [7] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-6:53:230 [7] NCCL INFO Setting affinity for GPU 7 to ffffffff,ffffff00,00000000,0000ffff,ffffffff,ff000000,00000000 | |
| nccl-alltoall-job-oguz-mpiworker-16:49:233 [3] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-16:49:233 [3] NCCL INFO Setting affinity for GPU 3 to ff,ffffffff,ffff0000,00000000,00ffffff,ffffffff | |
| nccl-alltoall-job-oguz-mpiworker-15:52:237 [6] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-15:52:237 [6] NCCL INFO Setting affinity for GPU 6 to ffffffff,ffffff00,00000000,0000ffff,ffffffff,ff000000,00000000 | |
| nccl-alltoall-job-oguz-mpiworker-31:50:237 [4] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-21:48:232 [2] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-25:47:234 [1] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-18:51:235 [5] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-18:51:235 [5] NCCL INFO Setting affinity for GPU 5 to ffffffff,ffffff00,00000000,0000ffff,ffffffff,ff000000,00000000 | |
| nccl-alltoall-job-oguz-mpiworker-2:46:232 [0] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-7:49:235 [3] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-7:49:235 [3] NCCL INFO Setting affinity for GPU 3 to ff,ffffffff,ffff0000,00000000,00ffffff,ffffffff | |
| nccl-alltoall-job-oguz-mpiworker-23:49:236 [3] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-23:49:236 [3] NCCL INFO Setting affinity for GPU 3 to ff,ffffffff,ffff0000,00000000,00ffffff,ffffffff | |
| nccl-alltoall-job-oguz-mpiworker-14:50:236 [4] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-22:46:231 [0] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-17:51:232 [5] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-13:47:231 [1] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-6:53:230 [7] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-16:49:233 [3] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-15:50:234 [4] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-21:52:231 [6] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-21:52:231 [6] NCCL INFO Setting affinity for GPU 6 to ffffffff,ffffff00,00000000,0000ffff,ffffffff,ff000000,00000000 | |
| nccl-alltoall-job-oguz-mpiworker-18:47:232 [1] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-18:47:232 [1] NCCL INFO Setting affinity for GPU 1 to ff,ffffffff,ffff0000,00000000,00ffffff,ffffffff | |
| nccl-alltoall-job-oguz-mpiworker-2:53:230 [7] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-20:46:233 [0] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-20:46:233 [0] NCCL INFO Setting affinity for GPU 0 to ff,ffffffff,ffff0000,00000000,00ffffff,ffffffff | |
| nccl-alltoall-job-oguz-mpiworker-26:47:230 [1] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-26:47:230 [1] NCCL INFO Setting affinity for GPU 1 to ff,ffffffff,ffff0000,00000000,00ffffff,ffffffff | |
| nccl-alltoall-job-oguz-mpiworker-12:48:236 [2] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-12:48:236 [2] NCCL INFO Setting affinity for GPU 2 to ff,ffffffff,ffff0000,00000000,00ffffff,ffffffff | |
| nccl-alltoall-job-oguz-mpiworker-5:52:237 [6] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-5:52:237 [6] NCCL INFO Setting affinity for GPU 6 to ffffffff,ffffff00,00000000,0000ffff,ffffffff,ff000000,00000000 | |
| nccl-alltoall-job-oguz-mpiworker-7:54:236 [7] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-23:53:231 [7] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-23:53:231 [7] NCCL INFO Setting affinity for GPU 7 to ffffffff,ffffff00,00000000,0000ffff,ffffffff,ff000000,00000000 | |
| nccl-alltoall-job-oguz-mpiworker-14:52:235 [6] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-14:52:235 [6] NCCL INFO Setting affinity for GPU 6 to ffffffff,ffffff00,00000000,0000ffff,ffffffff,ff000000,00000000 | |
| nccl-alltoall-job-oguz-mpiworker-22:48:236 [2] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-17:48:230 [2] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-17:48:230 [2] NCCL INFO Setting affinity for GPU 2 to ff,ffffffff,ffff0000,00000000,00ffffff,ffffffff | |
| nccl-alltoall-job-oguz-mpiworker-13:50:236 [4] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-13:50:236 [4] NCCL INFO Setting affinity for GPU 4 to ffffffff,ffffff00,00000000,0000ffff,ffffffff,ff000000,00000000 | |
| nccl-alltoall-job-oguz-mpiworker-6:49:234 [3] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-6:49:234 [3] NCCL INFO Setting affinity for GPU 3 to ff,ffffffff,ffff0000,00000000,00ffffff,ffffffff | |
| nccl-alltoall-job-oguz-mpiworker-16:47:234 [1] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-16:47:234 [1] NCCL INFO Setting affinity for GPU 1 to ff,ffffffff,ffff0000,00000000,00ffffff,ffffffff | |
| nccl-alltoall-job-oguz-mpiworker-15:53:236 [7] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-15:53:236 [7] NCCL INFO Setting affinity for GPU 7 to ffffffff,ffffff00,00000000,0000ffff,ffffffff,ff000000,00000000 | |
| nccl-alltoall-job-oguz-mpiworker-21:46:230 [0] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-18:51:235 [5] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-2:51:234 [5] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-20:49:235 [3] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-20:49:235 [3] NCCL INFO Setting affinity for GPU 3 to ff,ffffffff,ffff0000,00000000,00ffffff,ffffffff | |
| nccl-alltoall-job-oguz-mpiworker-26:53:232 [7] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-26:53:232 [7] NCCL INFO Setting affinity for GPU 7 to ffffffff,ffffff00,00000000,0000ffff,ffffffff,ff000000,00000000 | |
| nccl-alltoall-job-oguz-mpiworker-12:48:236 [2] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-5:52:237 [6] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-27:50:232 [4] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-27:50:232 [4] NCCL INFO Setting affinity for GPU 4 to ffffffff,ffffff00,00000000,0000ffff,ffffffff,ff000000,00000000 | |
| nccl-alltoall-job-oguz-mpiworker-19:51:235 [5] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-19:51:235 [5] NCCL INFO Setting affinity for GPU 5 to ffffffff,ffffff00,00000000,0000ffff,ffffffff,ff000000,00000000 | |
| nccl-alltoall-job-oguz-mpiworker-7:49:235 [3] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-23:51:235 [5] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-23:51:235 [5] NCCL INFO Setting affinity for GPU 5 to ffffffff,ffffff00,00000000,0000ffff,ffffffff,ff000000,00000000 | |
| nccl-alltoall-job-oguz-mpiworker-14:52:235 [6] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-17:48:230 [2] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-13:50:236 [4] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-6:47:232 [1] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-6:47:232 [1] NCCL INFO Setting affinity for GPU 1 to ff,ffffffff,ffff0000,00000000,00ffffff,ffffffff | |
| nccl-alltoall-job-oguz-mpiworker-16:51:235 [5] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-16:51:235 [5] NCCL INFO Setting affinity for GPU 5 to ffffffff,ffffff00,00000000,0000ffff,ffffffff,ff000000,00000000 | |
| nccl-alltoall-job-oguz-mpiworker-15:47:232 [1] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-21:46:230 [0] NCCL INFO Setting affinity for GPU 0 to ff,ffffffff,ffff0000,00000000,00ffffff,ffffffff | |
| nccl-alltoall-job-oguz-mpiworker-18:49:234 [3] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-2:50:231 [4] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-20:51:230 [5] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-20:51:230 [5] NCCL INFO Setting affinity for GPU 5 to ffffffff,ffffff00,00000000,0000ffff,ffffffff,ff000000,00000000 | |
| nccl-alltoall-job-oguz-mpiworker-26:53:232 [7] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-27:48:235 [2] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-27:48:235 [2] NCCL INFO Setting affinity for GPU 2 to ff,ffffffff,ffff0000,00000000,00ffffff,ffffffff | |
| nccl-alltoall-job-oguz-mpiworker-23:53:231 [7] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-14:51:230 [5] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-14:51:230 [5] NCCL INFO Setting affinity for GPU 5 to ffffffff,ffffff00,00000000,0000ffff,ffffffff,ff000000,00000000 | |
| nccl-alltoall-job-oguz-mpiworker-6:49:234 [3] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-16:51:235 [5] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-15:51:233 [5] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-15:51:233 [5] NCCL INFO Setting affinity for GPU 5 to ffffffff,ffffff00,00000000,0000ffff,ffffffff,ff000000,00000000 | |
| nccl-alltoall-job-oguz-mpiworker-21:46:230 [0] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-18:53:231 [7] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-18:53:231 [7] NCCL INFO Setting affinity for GPU 7 to ffffffff,ffffff00,00000000,0000ffff,ffffffff,ff000000,00000000 | |
| nccl-alltoall-job-oguz-mpiworker-2:49:236 [3] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-20:51:230 [5] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-26:47:230 [1] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-27:48:235 [2] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-23:51:235 [5] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-14:48:237 [2] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-14:48:237 [2] NCCL INFO Setting affinity for GPU 2 to ff,ffffffff,ffff0000,00000000,00ffffff,ffffffff | |
| nccl-alltoall-job-oguz-mpiworker-6:47:232 [1] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-16:47:234 [1] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-15:53:236 [7] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-21:52:231 [6] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-18:47:232 [1] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-2:48:237 [2] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-20:46:233 [0] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-26:49:233 [3] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-26:49:233 [3] NCCL INFO Setting affinity for GPU 3 to ff,ffffffff,ffff0000,00000000,00ffffff,ffffffff | |
| nccl-alltoall-job-oguz-mpiworker-27:50:232 [4] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-23:49:236 [3] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-14:51:230 [5] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-15:51:233 [5] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-21:49:236 [3] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-21:49:236 [3] NCCL INFO Setting affinity for GPU 3 to ff,ffffffff,ffff0000,00000000,00ffffff,ffffffff | |
| nccl-alltoall-job-oguz-mpiworker-18:53:231 [7] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-2:47:235 [1] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-20:49:235 [3] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-26:51:236 [5] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-26:51:236 [5] NCCL INFO Setting affinity for GPU 5 to ffffffff,ffffff00,00000000,0000ffff,ffffffff,ff000000,00000000 | |
| nccl-alltoall-job-oguz-mpiworker-27:52:230 [6] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-27:52:230 [6] NCCL INFO Setting affinity for GPU 6 to ffffffff,ffffff00,00000000,0000ffff,ffffffff,ff000000,00000000 | |
| nccl-alltoall-job-oguz-mpiworker-28:52:232 [6] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-28:52:232 [6] NCCL INFO Setting affinity for GPU 6 to ffffffff,ffffff00,00000000,0000ffff,ffffffff,ff000000,00000000 | |
| nccl-alltoall-job-oguz-mpiworker-14:48:237 [2] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-15:52:237 [6] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-21:49:236 [3] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-18:46:230 [0] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-18:46:230 [0] NCCL INFO Setting affinity for GPU 0 to ff,ffffffff,ffff0000,00000000,00ffffff,ffffffff | |
| nccl-alltoall-job-oguz-mpiworker-2:52:233 [6] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC | |
| nccl-alltoall-job-oguz-mpiworker-20:53:231 [7] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-20:53:231 [7] NCCL INFO Setting affinity for GPU 7 to ffffffff,ffffff00,00000000,0000ffff,ffffffff,ff000000,00000000 | |
| nccl-alltoall-job-oguz-mpiworker-26:51:236 [5] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-27:52:230 [6] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-28:50:234 [4] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-28:50:234 [4] NCCL INFO Setting affinity for GPU 4 to ffffffff,ffffff00,00000000,0000ffff,ffffffff,ff000000,00000000 | |
| nccl-alltoall-job-oguz-mpiworker-19:51:235 [5] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-25:51:232 [5] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-25:51:232 [5] NCCL INFO Setting affinity for GPU 5 to ffffffff,ffffff00,00000000,0000ffff,ffffffff,ff000000,00000000 | |
| nccl-alltoall-job-oguz-mpiworker-5:50:231 [4] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-5:50:231 [4] NCCL INFO Setting affinity for GPU 4 to ffffffff,ffffff00,00000000,0000ffff,ffffffff,ff000000,00000000 | |
| nccl-alltoall-job-oguz-mpiworker-7:46:230 [0] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-7:46:230 [0] NCCL INFO Setting affinity for GPU 0 to ff,ffffffff,ffff0000,00000000,00ffffff,ffffffff | |
| nccl-alltoall-job-oguz-mpiworker-15:49:235 [3] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-15:49:235 [3] NCCL INFO Setting affinity for GPU 3 to ff,ffffffff,ffff0000,00000000,00ffffff,ffffffff | |
| nccl-alltoall-job-oguz-mpiworker-21:47:234 [1] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-21:47:234 [1] NCCL INFO Setting affinity for GPU 1 to ff,ffffffff,ffff0000,00000000,00ffffff,ffffffff | |
| nccl-alltoall-job-oguz-mpiworker-18:46:230 [0] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-20:53:231 [7] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-26:49:233 [3] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-27:53:237 [7] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-27:53:237 [7] NCCL INFO Setting affinity for GPU 7 to ffffffff,ffffff00,00000000,0000ffff,ffffffff,ff000000,00000000 | |
| nccl-alltoall-job-oguz-mpiworker-28:48:231 [2] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-28:48:231 [2] NCCL INFO Setting affinity for GPU 2 to ff,ffffffff,ffff0000,00000000,00ffffff,ffffffff | |
| nccl-alltoall-job-oguz-mpiworker-19:49:236 [3] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-19:49:236 [3] NCCL INFO Setting affinity for GPU 3 to ff,ffffffff,ffff0000,00000000,00ffffff,ffffffff | |
| nccl-alltoall-job-oguz-mpiworker-25:49:233 [3] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-25:49:233 [3] NCCL INFO Setting affinity for GPU 3 to ff,ffffffff,ffff0000,00000000,00ffffff,ffffffff | |
| nccl-alltoall-job-oguz-mpiworker-5:46:230 [0] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-5:46:230 [0] NCCL INFO Setting affinity for GPU 0 to ff,ffffffff,ffff0000,00000000,00ffffff,ffffffff | |
| nccl-alltoall-job-oguz-mpiworker-7:46:230 [0] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-15:49:235 [3] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-21:53:233 [7] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-21:53:233 [7] NCCL INFO Setting affinity for GPU 7 to ffffffff,ffffff00,00000000,0000ffff,ffffffff,ff000000,00000000 | |
| nccl-alltoall-job-oguz-mpiworker-18:50:236 [4] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-18:50:236 [4] NCCL INFO Setting affinity for GPU 4 to ffffffff,ffffff00,00000000,0000ffff,ffffffff,ff000000,00000000 | |
| nccl-alltoall-job-oguz-mpiworker-20:47:234 [1] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-20:47:234 [1] NCCL INFO Setting affinity for GPU 1 to ff,ffffffff,ffff0000,00000000,00ffffff,ffffffff | |
| nccl-alltoall-job-oguz-mpiworker-26:50:234 [4] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-26:50:234 [4] NCCL INFO Setting affinity for GPU 4 to ffffffff,ffffff00,00000000,0000ffff,ffffffff,ff000000,00000000 | |
| nccl-alltoall-job-oguz-mpiworker-27:47:236 [1] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-27:47:236 [1] NCCL INFO Setting affinity for GPU 1 to ff,ffffffff,ffff0000,00000000,00ffffff,ffffffff | |
| nccl-alltoall-job-oguz-mpiworker-28:46:233 [0] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-28:46:233 [0] NCCL INFO Setting affinity for GPU 0 to ff,ffffffff,ffff0000,00000000,00ffffff,ffffffff | |
| nccl-alltoall-job-oguz-mpiworker-28:46:233 [0] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-19:46:230 [0] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-19:46:230 [0] NCCL INFO Setting affinity for GPU 0 to ff,ffffffff,ffff0000,00000000,00ffffff,ffffffff | |
| nccl-alltoall-job-oguz-mpiworker-25:50:235 [4] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-25:50:235 [4] NCCL INFO Setting affinity for GPU 4 to ffffffff,ffffff00,00000000,0000ffff,ffffffff,ff000000,00000000 | |
| nccl-alltoall-job-oguz-mpiworker-5:46:230 [0] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-30:47:232 [1] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-30:47:232 [1] NCCL INFO Setting affinity for GPU 1 to ff,ffffffff,ffff0000,00000000,00ffffff,ffffffff | |
| nccl-alltoall-job-oguz-mpiworker-11:51:234 [5] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-11:51:234 [5] NCCL INFO Setting affinity for GPU 5 to ffffffff,ffffff00,00000000,0000ffff,ffffffff,ff000000,00000000 | |
| nccl-alltoall-job-oguz-mpiworker-0:52:235 [6] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-0:52:235 [6] NCCL INFO Setting affinity for GPU 6 to ffffffff,ffffff00,00000000,0000ffff,ffffffff,ff000000,00000000 | |
| nccl-alltoall-job-oguz-mpiworker-15:48:231 [2] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-15:48:231 [2] NCCL INFO Setting affinity for GPU 2 to ff,ffffffff,ffff0000,00000000,00ffffff,ffffffff | |
| nccl-alltoall-job-oguz-mpiworker-21:53:233 [7] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-18:50:236 [4] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-20:47:234 [1] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-26:46:231 [0] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-26:46:231 [0] NCCL INFO Setting affinity for GPU 0 to ff,ffffffff,ffff0000,00000000,00ffffff,ffffffff | |
| nccl-alltoall-job-oguz-mpiworker-27:53:237 [7] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-28:48:231 [2] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-19:47:232 [1] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-19:47:232 [1] NCCL INFO Setting affinity for GPU 1 to ff,ffffffff,ffff0000,00000000,00ffffff,ffffffff | |
| nccl-alltoall-job-oguz-mpiworker-25:51:232 [5] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-5:50:231 [4] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-30:51:233 [5] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-30:51:233 [5] NCCL INFO Setting affinity for GPU 5 to ffffffff,ffffff00,00000000,0000ffff,ffffffff,ff000000,00000000 | |
| nccl-alltoall-job-oguz-mpiworker-11:51:234 [5] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-0:48:233 [2] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-0:48:233 [2] NCCL INFO Setting affinity for GPU 2 to ff,ffffffff,ffff0000,00000000,00ffffff,ffffffff | |
| nccl-alltoall-job-oguz-mpiworker-15:48:231 [2] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-21:47:234 [1] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-26:46:231 [0] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-27:47:236 [1] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-28:52:232 [6] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-19:50:237 [4] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-19:50:237 [4] NCCL INFO Setting affinity for GPU 4 to ffffffff,ffffff00,00000000,0000ffff,ffffffff,ff000000,00000000 | |
| nccl-alltoall-job-oguz-mpiworker-25:46:230 [0] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-25:46:230 [0] NCCL INFO Setting affinity for GPU 0 to ff,ffffffff,ffff0000,00000000,00ffffff,ffffffff | |
| nccl-alltoall-job-oguz-mpiworker-25:46:230 [0] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-5:49:232 [3] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-5:49:232 [3] NCCL INFO Setting affinity for GPU 3 to ff,ffffffff,ffff0000,00000000,00ffffff,ffffffff | |
| nccl-alltoall-job-oguz-mpiworker-30:51:233 [5] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-0:48:233 [2] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-21:50:237 [4] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-21:50:237 [4] NCCL INFO Setting affinity for GPU 4 to ffffffff,ffffff00,00000000,0000ffff,ffffffff,ff000000,00000000 | |
| nccl-alltoall-job-oguz-mpiworker-26:50:234 [4] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-27:51:234 [5] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-27:51:234 [5] NCCL INFO Setting affinity for GPU 5 to ffffffff,ffffff00,00000000,0000ffff,ffffffff,ff000000,00000000 | |
| nccl-alltoall-job-oguz-mpiworker-28:50:234 [4] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-19:46:230 [0] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-25:49:233 [3] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-5:49:232 [3] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-30:47:232 [1] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-0:52:235 [6] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-21:51:235 [5] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-21:51:235 [5] NCCL INFO Setting affinity for GPU 5 to ffffffff,ffffff00,00000000,0000ffff,ffffffff,ff000000,00000000 | |
| nccl-alltoall-job-oguz-mpiworker-26:52:237 [6] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-26:52:237 [6] NCCL INFO Setting affinity for GPU 6 to ffffffff,ffffff00,00000000,0000ffff,ffffffff,ff000000,00000000 | |
| nccl-alltoall-job-oguz-mpiworker-27:49:233 [3] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-27:49:233 [3] NCCL INFO Setting affinity for GPU 3 to ff,ffffffff,ffff0000,00000000,00ffffff,ffffffff | |
| nccl-alltoall-job-oguz-mpiworker-28:51:236 [5] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-28:51:236 [5] NCCL INFO Setting affinity for GPU 5 to ffffffff,ffffff00,00000000,0000ffff,ffffffff,ff000000,00000000 | |
| nccl-alltoall-job-oguz-mpiworker-19:52:234 [6] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-19:52:234 [6] NCCL INFO Setting affinity for GPU 6 to ffffffff,ffffff00,00000000,0000ffff,ffffffff,ff000000,00000000 | |
| nccl-alltoall-job-oguz-mpiworker-25:50:235 [4] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-5:51:236 [5] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-5:51:236 [5] NCCL INFO Setting affinity for GPU 5 to ffffffff,ffffff00,00000000,0000ffff,ffffffff,ff000000,00000000 | |
| nccl-alltoall-job-oguz-mpiworker-30:49:234 [3] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-30:49:234 [3] NCCL INFO Setting affinity for GPU 3 to ff,ffffffff,ffff0000,00000000,00ffffff,ffffffff | |
| nccl-alltoall-job-oguz-mpiworker-0:50:238 [4] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-0:50:238 [4] NCCL INFO Setting affinity for GPU 4 to ffffffff,ffffff00,00000000,0000ffff,ffffffff,ff000000,00000000 | |
| nccl-alltoall-job-oguz-mpiworker-20:50:237 [4] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-20:50:237 [4] NCCL INFO Setting affinity for GPU 4 to ffffffff,ffffff00,00000000,0000ffff,ffffffff,ff000000,00000000 | |
| nccl-alltoall-job-oguz-mpiworker-21:50:237 [4] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-26:52:237 [6] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-27:51:234 [5] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-28:51:236 [5] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-19:49:236 [3] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-25:48:237 [2] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-25:48:237 [2] NCCL INFO Setting affinity for GPU 2 to ff,ffffffff,ffff0000,00000000,00ffffff,ffffffff | |
| nccl-alltoall-job-oguz-mpiworker-5:51:236 [5] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-30:53:231 [7] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-30:53:231 [7] NCCL INFO Setting affinity for GPU 7 to ffffffff,ffffff00,00000000,0000ffff,ffffffff,ff000000,00000000 | |
| nccl-alltoall-job-oguz-mpiworker-0:50:238 [4] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-20:50:237 [4] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-29:53:235 [7] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-29:53:235 [7] NCCL INFO Setting affinity for GPU 7 to ffffffff,ffffff00,00000000,0000ffff,ffffffff,ff000000,00000000 | |
| nccl-alltoall-job-oguz-mpiworker-21:51:235 [5] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-26:48:235 [2] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-26:48:235 [2] NCCL INFO Setting affinity for GPU 2 to ff,ffffffff,ffff0000,00000000,00ffffff,ffffffff | |
| nccl-alltoall-job-oguz-mpiworker-27:49:233 [3] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-28:53:235 [7] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-28:53:235 [7] NCCL INFO Setting affinity for GPU 7 to ffffffff,ffffff00,00000000,0000ffff,ffffffff,ff000000,00000000 | |
| nccl-alltoall-job-oguz-mpiworker-19:47:232 [1] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-25:48:237 [2] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-5:47:235 [1] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-5:47:235 [1] NCCL INFO Setting affinity for GPU 1 to ff,ffffffff,ffff0000,00000000,00ffffff,ffffffff | |
| nccl-alltoall-job-oguz-mpiworker-30:53:231 [7] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-20:48:236 [2] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-20:48:236 [2] NCCL INFO Setting affinity for GPU 2 to ff,ffffffff,ffff0000,00000000,00ffffff,ffffffff | |
| nccl-alltoall-job-oguz-mpiworker-29:53:235 [7] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-26:48:235 [2] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-28:53:235 [7] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-19:50:237 [4] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-25:52:236 [6] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-25:52:236 [6] NCCL INFO Setting affinity for GPU 6 to ffffffff,ffffff00,00000000,0000ffff,ffffffff,ff000000,00000000 | |
| nccl-alltoall-job-oguz-mpiworker-5:47:235 [1] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-30:49:234 [3] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-20:48:236 [2] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-29:51:230 [5] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-0:46:231 [0] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-0:46:231 [0] NCCL INFO Setting affinity for GPU 0 to ff,ffffffff,ffff0000,00000000,00ffffff,ffffffff | |
| nccl-alltoall-job-oguz-mpiworker-0:46:231 [0] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-18:52:237 [6] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-18:52:237 [6] NCCL INFO Setting affinity for GPU 6 to ffffffff,ffffff00,00000000,0000ffff,ffffffff,ff000000,00000000 | |
| nccl-alltoall-job-oguz-mpiworker-28:49:237 [3] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-28:49:237 [3] NCCL INFO Setting affinity for GPU 3 to ff,ffffffff,ffff0000,00000000,00ffffff,ffffffff | |
| nccl-alltoall-job-oguz-mpiworker-19:52:234 [6] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-25:52:236 [6] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-5:53:233 [7] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-5:53:233 [7] NCCL INFO Setting affinity for GPU 7 to ffffffff,ffffff00,00000000,0000ffff,ffffffff,ff000000,00000000 | |
| nccl-alltoall-job-oguz-mpiworker-30:46:230 [0] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-30:46:230 [0] NCCL INFO Setting affinity for GPU 0 to ff,ffffffff,ffff0000,00000000,00ffffff,ffffffff | |
| nccl-alltoall-job-oguz-mpiworker-29:51:230 [5] NCCL INFO Setting affinity for GPU 5 to ffffffff,ffffff00,00000000,0000ffff,ffffffff,ff000000,00000000 | |
| nccl-alltoall-job-oguz-mpiworker-0:53:234 [7] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-0:53:234 [7] NCCL INFO Setting affinity for GPU 7 to ffffffff,ffffff00,00000000,0000ffff,ffffffff,ff000000,00000000 | |
| nccl-alltoall-job-oguz-mpiworker-19:48:233 [2] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-19:48:233 [2] NCCL INFO Setting affinity for GPU 2 to ff,ffffffff,ffff0000,00000000,00ffffff,ffffffff | |
| nccl-alltoall-job-oguz-mpiworker-5:53:233 [7] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-30:50:236 [4] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-30:50:236 [4] NCCL INFO Setting affinity for GPU 4 to ffffffff,ffffff00,00000000,0000ffff,ffffffff,ff000000,00000000 | |
| nccl-alltoall-job-oguz-mpiworker-29:51:230 [5] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-0:51:232 [5] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-0:51:232 [5] NCCL INFO Setting affinity for GPU 5 to ffffffff,ffffff00,00000000,0000ffff,ffffffff,ff000000,00000000 | |
| nccl-alltoall-job-oguz-mpiworker-18:52:237 [6] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-19:48:233 [2] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-5:48:234 [2] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-5:48:234 [2] NCCL INFO Setting affinity for GPU 2 to ff,ffffffff,ffff0000,00000000,00ffffff,ffffffff | |
| nccl-alltoall-job-oguz-mpiworker-30:46:230 [0] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-0:53:234 [7] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-28:47:230 [1] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-28:47:230 [1] NCCL INFO Setting affinity for GPU 1 to ff,ffffffff,ffff0000,00000000,00ffffff,ffffffff | |
| nccl-alltoall-job-oguz-mpiworker-5:48:234 [2] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-30:48:237 [2] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-30:48:237 [2] NCCL INFO Setting affinity for GPU 2 to ff,ffffffff,ffff0000,00000000,00ffffff,ffffffff | |
| nccl-alltoall-job-oguz-mpiworker-0:51:232 [5] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-28:47:230 [1] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-0:49:236 [3] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-0:49:236 [3] NCCL INFO Setting affinity for GPU 3 to ff,ffffffff,ffff0000,00000000,00ffffff,ffffffff | |
| nccl-alltoall-job-oguz-mpiworker-28:49:237 [3] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-0:49:236 [3] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-0:47:237 [1] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-0:47:237 [1] NCCL INFO Setting affinity for GPU 1 to ff,ffffffff,ffff0000,00000000,00ffffff,ffffffff | |
| nccl-alltoall-job-oguz-mpiworker-30:52:235 [6] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-30:52:235 [6] NCCL INFO Setting affinity for GPU 6 to ffffffff,ffffff00,00000000,0000ffff,ffffffff,ff000000,00000000 | |
| nccl-alltoall-job-oguz-mpiworker-30:48:237 [2] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-0:47:237 [1] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-30:50:236 [4] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-30:52:235 [6] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-29:48:231 [2] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-29:48:231 [2] NCCL INFO Setting affinity for GPU 2 to ff,ffffffff,ffff0000,00000000,00ffffff,ffffffff | |
| nccl-alltoall-job-oguz-mpiworker-29:48:231 [2] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-3:49:232 [3] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-3:49:232 [3] NCCL INFO Setting affinity for GPU 3 to ff,ffffffff,ffff0000,00000000,00ffffff,ffffffff | |
| nccl-alltoall-job-oguz-mpiworker-4:53:235 [7] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-4:53:235 [7] NCCL INFO Setting affinity for GPU 7 to ffffffff,ffffff00,00000000,0000ffff,ffffffff,ff000000,00000000 | |
| nccl-alltoall-job-oguz-mpiworker-3:51:234 [5] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-3:51:234 [5] NCCL INFO Setting affinity for GPU 5 to ffffffff,ffffff00,00000000,0000ffff,ffffffff,ff000000,00000000 | |
| nccl-alltoall-job-oguz-mpiworker-3:49:232 [3] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-3:51:234 [5] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-4:53:235 [7] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-4:49:234 [3] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-4:49:234 [3] NCCL INFO Setting affinity for GPU 3 to ff,ffffffff,ffff0000,00000000,00ffffff,ffffffff | |
| nccl-alltoall-job-oguz-mpiworker-3:47:230 [1] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-3:47:230 [1] NCCL INFO Setting affinity for GPU 1 to ff,ffffffff,ffff0000,00000000,00ffffff,ffffffff | |
| nccl-alltoall-job-oguz-mpiworker-3:53:235 [7] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-3:53:235 [7] NCCL INFO Setting affinity for GPU 7 to ffffffff,ffffff00,00000000,0000ffff,ffffffff,ff000000,00000000 | |
| nccl-alltoall-job-oguz-mpiworker-3:53:235 [7] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-29:46:232 [0] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-29:46:232 [0] NCCL INFO Setting affinity for GPU 0 to ff,ffffffff,ffff0000,00000000,00ffffff,ffffffff | |
| nccl-alltoall-job-oguz-mpiworker-3:47:230 [1] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-4:49:234 [3] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-29:46:232 [0] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-24:53:235 [7] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-24:53:235 [7] NCCL INFO Setting affinity for GPU 7 to ffffffff,ffffff00,00000000,0000ffff,ffffffff,ff000000,00000000 | |
| nccl-alltoall-job-oguz-mpiworker-4:47:232 [1] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-4:47:232 [1] NCCL INFO Setting affinity for GPU 1 to ff,ffffffff,ffff0000,00000000,00ffffff,ffffffff | |
| nccl-alltoall-job-oguz-mpiworker-4:47:232 [1] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-29:52:237 [6] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-29:52:237 [6] NCCL INFO Setting affinity for GPU 6 to ffffffff,ffffff00,00000000,0000ffff,ffffffff,ff000000,00000000 | |
| nccl-alltoall-job-oguz-mpiworker-24:53:235 [7] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-29:52:237 [6] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-24:47:231 [1] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-24:47:231 [1] NCCL INFO Setting affinity for GPU 1 to ff,ffffffff,ffff0000,00000000,00ffffff,ffffffff | |
| nccl-alltoall-job-oguz-mpiworker-3:48:231 [2] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-3:48:231 [2] NCCL INFO Setting affinity for GPU 2 to ff,ffffffff,ffff0000,00000000,00ffffff,ffffffff | |
| nccl-alltoall-job-oguz-mpiworker-3:48:231 [2] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-24:47:231 [1] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-3:46:233 [0] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-3:46:233 [0] NCCL INFO Setting affinity for GPU 0 to ff,ffffffff,ffff0000,00000000,00ffffff,ffffffff | |
| nccl-alltoall-job-oguz-mpiworker-3:50:236 [4] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-3:50:236 [4] NCCL INFO Setting affinity for GPU 4 to ffffffff,ffffff00,00000000,0000ffff,ffffffff,ff000000,00000000 | |
| nccl-alltoall-job-oguz-mpiworker-3:46:233 [0] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-3:52:237 [6] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-3:52:237 [6] NCCL INFO Setting affinity for GPU 6 to ffffffff,ffffff00,00000000,0000ffff,ffffffff,ff000000,00000000 | |
| nccl-alltoall-job-oguz-mpiworker-3:50:236 [4] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-3:52:237 [6] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-24:51:234 [5] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-24:51:234 [5] NCCL INFO Setting affinity for GPU 5 to ffffffff,ffffff00,00000000,0000ffff,ffffffff,ff000000,00000000 | |
| nccl-alltoall-job-oguz-mpiworker-24:51:234 [5] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-4:52:237 [6] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-4:52:237 [6] NCCL INFO Setting affinity for GPU 6 to ffffffff,ffffff00,00000000,0000ffff,ffffffff,ff000000,00000000 | |
| nccl-alltoall-job-oguz-mpiworker-29:47:233 [1] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-29:47:233 [1] NCCL INFO Setting affinity for GPU 1 to ff,ffffffff,ffff0000,00000000,00ffffff,ffffffff | |
| nccl-alltoall-job-oguz-mpiworker-1:52:235 [6] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-1:52:235 [6] NCCL INFO Setting affinity for GPU 6 to ffffffff,ffffff00,00000000,0000ffff,ffffffff,ff000000,00000000 | |
| nccl-alltoall-job-oguz-mpiworker-29:47:233 [1] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-31:49:235 [3] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-31:49:235 [3] NCCL INFO Setting affinity for GPU 3 to ff,ffffffff,ffff0000,00000000,00ffffff,ffffffff | |
| nccl-alltoall-job-oguz-mpiworker-31:53:231 [7] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-31:53:231 [7] NCCL INFO Setting affinity for GPU 7 to ffffffff,ffffff00,00000000,0000ffff,ffffffff,ff000000,00000000 | |
| nccl-alltoall-job-oguz-mpiworker-1:52:235 [6] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-31:53:231 [7] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-24:46:237 [0] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-24:46:237 [0] NCCL INFO Setting affinity for GPU 0 to ff,ffffffff,ffff0000,00000000,00ffffff,ffffffff | |
| nccl-alltoall-job-oguz-mpiworker-31:49:235 [3] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-4:50:230 [4] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-4:50:230 [4] NCCL INFO Setting affinity for GPU 4 to ffffffff,ffffff00,00000000,0000ffff,ffffffff,ff000000,00000000 | |
| nccl-alltoall-job-oguz-mpiworker-29:49:236 [3] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-29:49:236 [3] NCCL INFO Setting affinity for GPU 3 to ff,ffffffff,ffff0000,00000000,00ffffff,ffffffff | |
| nccl-alltoall-job-oguz-mpiworker-4:48:233 [2] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-4:48:233 [2] NCCL INFO Setting affinity for GPU 2 to ff,ffffffff,ffff0000,00000000,00ffffff,ffffffff | |
| nccl-alltoall-job-oguz-mpiworker-24:48:230 [2] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-24:48:230 [2] NCCL INFO Setting affinity for GPU 2 to ff,ffffffff,ffff0000,00000000,00ffffff,ffffffff | |
| nccl-alltoall-job-oguz-mpiworker-4:48:233 [2] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-29:50:234 [4] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-29:50:234 [4] NCCL INFO Setting affinity for GPU 4 to ffffffff,ffffff00,00000000,0000ffff,ffffffff,ff000000,00000000 | |
| nccl-alltoall-job-oguz-mpiworker-4:46:231 [0] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-4:46:231 [0] NCCL INFO Setting affinity for GPU 0 to ff,ffffffff,ffff0000,00000000,00ffffff,ffffffff | |
| nccl-alltoall-job-oguz-mpiworker-18:48:233 [2] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-18:48:233 [2] NCCL INFO Setting affinity for GPU 2 to ff,ffffffff,ffff0000,00000000,00ffffff,ffffffff | |
| nccl-alltoall-job-oguz-mpiworker-4:52:237 [6] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-18:48:233 [2] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-24:46:237 [0] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-4:46:231 [0] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-24:48:230 [2] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-29:49:236 [3] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-4:50:230 [4] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-24:50:236 [4] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-24:50:236 [4] NCCL INFO Setting affinity for GPU 4 to ffffffff,ffffff00,00000000,0000ffff,ffffffff,ff000000,00000000 | |
| nccl-alltoall-job-oguz-mpiworker-29:50:234 [4] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-24:52:232 [6] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-24:52:232 [6] NCCL INFO Setting affinity for GPU 6 to ffffffff,ffffff00,00000000,0000ffff,ffffffff,ff000000,00000000 | |
| nccl-alltoall-job-oguz-mpiworker-1:48:231 [2] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-1:48:231 [2] NCCL INFO Setting affinity for GPU 2 to ff,ffffffff,ffff0000,00000000,00ffffff,ffffffff | |
| nccl-alltoall-job-oguz-mpiworker-31:47:232 [1] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-31:47:232 [1] NCCL INFO Setting affinity for GPU 1 to ff,ffffffff,ffff0000,00000000,00ffffff,ffffffff | |
| nccl-alltoall-job-oguz-mpiworker-24:50:236 [4] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-1:50:233 [4] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-1:50:233 [4] NCCL INFO Setting affinity for GPU 4 to ffffffff,ffffff00,00000000,0000ffff,ffffffff,ff000000,00000000 | |
| nccl-alltoall-job-oguz-mpiworker-1:48:231 [2] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-31:51:236 [5] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-1:49:236 [3] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-1:49:236 [3] NCCL INFO Setting affinity for GPU 3 to ff,ffffffff,ffff0000,00000000,00ffffff,ffffffff | |
| nccl-alltoall-job-oguz-mpiworker-31:51:236 [5] NCCL INFO Setting affinity for GPU 5 to ffffffff,ffffff00,00000000,0000ffff,ffffffff,ff000000,00000000 | |
| nccl-alltoall-job-oguz-mpiworker-24:52:232 [6] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-1:50:233 [4] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-31:51:236 [5] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-1:47:237 [1] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-1:47:237 [1] NCCL INFO Setting affinity for GPU 1 to ff,ffffffff,ffff0000,00000000,00ffffff,ffffffff | |
| nccl-alltoall-job-oguz-mpiworker-31:46:234 [0] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-31:46:234 [0] NCCL INFO Setting affinity for GPU 0 to ff,ffffffff,ffff0000,00000000,00ffffff,ffffffff | |
| nccl-alltoall-job-oguz-mpiworker-1:49:236 [3] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-31:46:234 [0] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-4:51:236 [5] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-4:51:236 [5] NCCL INFO Setting affinity for GPU 5 to ffffffff,ffffff00,00000000,0000ffff,ffffffff,ff000000,00000000 | |
| nccl-alltoall-job-oguz-mpiworker-1:54:234 [7] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-1:54:234 [7] NCCL INFO Setting affinity for GPU 7 to ffffffff,ffffff00,00000000,0000ffff,ffffffff,ff000000,00000000 | |
| nccl-alltoall-job-oguz-mpiworker-31:47:232 [1] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-1:54:234 [7] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-1:47:237 [1] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-4:51:236 [5] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-24:49:233 [3] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-24:49:233 [3] NCCL INFO Setting affinity for GPU 3 to ff,ffffffff,ffff0000,00000000,00ffffff,ffffffff | |
| nccl-alltoall-job-oguz-mpiworker-24:49:233 [3] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-1:46:230 [0] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-1:46:230 [0] NCCL INFO Setting affinity for GPU 0 to ff,ffffffff,ffff0000,00000000,00ffffff,ffffffff | |
| nccl-alltoall-job-oguz-mpiworker-1:46:230 [0] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-1:51:232 [5] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-1:51:232 [5] NCCL INFO Setting affinity for GPU 5 to ffffffff,ffffff00,00000000,0000ffff,ffffffff,ff000000,00000000 | |
| nccl-alltoall-job-oguz-mpiworker-1:51:232 [5] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-31:52:233 [6] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-31:52:233 [6] NCCL INFO Setting affinity for GPU 6 to ffffffff,ffffff00,00000000,0000ffff,ffffffff,ff000000,00000000 | |
| nccl-alltoall-job-oguz-mpiworker-31:52:233 [6] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-2:46:232 [0] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-2:46:232 [0] NCCL INFO Setting affinity for GPU 0 to ff,ffffffff,ffff0000,00000000,00ffffff,ffffffff | |
| nccl-alltoall-job-oguz-mpiworker-2:46:232 [0] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-2:53:230 [7] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-2:53:230 [7] NCCL INFO Setting affinity for GPU 7 to ffffffff,ffffff00,00000000,0000ffff,ffffffff,ff000000,00000000 | |
| nccl-alltoall-job-oguz-mpiworker-2:50:231 [4] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-2:50:231 [4] NCCL INFO Setting affinity for GPU 4 to ffffffff,ffffff00,00000000,0000ffff,ffffffff,ff000000,00000000 | |
| nccl-alltoall-job-oguz-mpiworker-2:51:234 [5] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-2:51:234 [5] NCCL INFO Setting affinity for GPU 5 to ffffffff,ffffff00,00000000,0000ffff,ffffffff,ff000000,00000000 | |
| nccl-alltoall-job-oguz-mpiworker-2:53:230 [7] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-2:48:237 [2] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-2:48:237 [2] NCCL INFO Setting affinity for GPU 2 to ff,ffffffff,ffff0000,00000000,00ffffff,ffffffff | |
| nccl-alltoall-job-oguz-mpiworker-2:48:237 [2] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-2:51:234 [5] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-2:49:236 [3] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-2:49:236 [3] NCCL INFO Setting affinity for GPU 3 to ff,ffffffff,ffff0000,00000000,00ffffff,ffffffff | |
| nccl-alltoall-job-oguz-mpiworker-2:47:235 [1] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-2:47:235 [1] NCCL INFO Setting affinity for GPU 1 to ff,ffffffff,ffff0000,00000000,00ffffff,ffffffff | |
| nccl-alltoall-job-oguz-mpiworker-2:50:231 [4] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-2:52:233 [6] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-2:52:233 [6] NCCL INFO Setting affinity for GPU 6 to ffffffff,ffffff00,00000000,0000ffff,ffffffff,ff000000,00000000 | |
| nccl-alltoall-job-oguz-mpiworker-2:47:235 [1] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-2:49:236 [3] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-2:52:233 [6] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-31:50:237 [4] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-31:50:237 [4] NCCL INFO Setting affinity for GPU 4 to ffffffff,ffffff00,00000000,0000ffff,ffffffff,ff000000,00000000 | |
| nccl-alltoall-job-oguz-mpiworker-31:48:230 [2] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1. | |
| nccl-alltoall-job-oguz-mpiworker-31:48:230 [2] NCCL INFO Setting affinity for GPU 2 to ff,ffffffff,ffff0000,00000000,00ffffff,ffffffff | |
| nccl-alltoall-job-oguz-mpiworker-31:48:230 [2] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-31:50:237 [4] NCCL INFO NCCL_CROSS_NIC set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-0:51:232 [5] NCCL INFO Trees [0] -1/-1/-1->5->6 [1] -1/-1/-1->5->6 [2] 6/4/-1->5->3 [3] 6/4/-1->5->3 | |
| nccl-alltoall-job-oguz-mpiworker-0:51:232 [5] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-0:53:234 [7] NCCL INFO Trees [0] -1/-1/-1->7->6 [1] -1/-1/-1->7->6 [2] 11/3/-1->7->15 [3] 11/3/-1->7->15 | |
| nccl-alltoall-job-oguz-mpiworker-0:53:234 [7] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-0:50:238 [4] NCCL INFO Trees [0] 2/6/-1->4->8 [1] 2/6/-1->4->8 [2] -1/-1/-1->4->5 [3] -1/-1/-1->4->5 | |
| nccl-alltoall-job-oguz-mpiworker-0:50:238 [4] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-0:52:235 [6] NCCL INFO Trees [0] 5/7/-1->6->4 [1] 5/7/-1->6->4 [2] -1/-1/-1->6->5 [3] -1/-1/-1->6->5 | |
| nccl-alltoall-job-oguz-mpiworker-0:52:235 [6] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-0:49:236 [3] NCCL INFO Trees [0] -1/-1/-1->3->2 [1] -1/-1/-1->3->2 [2] 5/1/-1->3->7 [3] 5/1/-1->3->7 | |
| nccl-alltoall-job-oguz-mpiworker-0:49:236 [3] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-0:51:232 [5] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-0:53:234 [7] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-0:49:236 [3] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-0:50:238 [4] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-0:52:235 [6] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-1:46:230 [0] NCCL INFO Trees [0] 4/12/-1->8->16 [1] 4/12/-1->8->16 [2] -1/-1/-1->8->9 [3] -1/-1/-1->8->9 | |
| nccl-alltoall-job-oguz-mpiworker-1:46:230 [0] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-1:47:237 [1] NCCL INFO Trees [0] -1/-1/-1->9->10 [1] -1/-1/-1->9->10 [2] 10/8/-1->9->11 [3] 10/8/-1->9->11 | |
| nccl-alltoall-job-oguz-mpiworker-1:47:237 [1] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-1:48:231 [2] NCCL INFO Trees [0] 9/11/-1->10->12 [1] 9/11/-1->10->12 [2] -1/-1/-1->10->9 [3] -1/-1/-1->10->9 | |
| nccl-alltoall-job-oguz-mpiworker-1:48:231 [2] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-1:46:230 [0] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-31:52:233 [6] NCCL INFO Trees [0] 253/255/-1->254->252 [1] 253/255/-1->254->252 [2] -1/-1/-1->254->253 [3] -1/-1/-1->254->253 | |
| nccl-alltoall-job-oguz-mpiworker-31:52:233 [6] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-1:47:237 [1] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-31:53:231 [7] NCCL INFO Trees [0] -1/-1/-1->255->254 [1] -1/-1/-1->255->254 [2] 127/-1/-1->255->-1 [3] 127/-1/-1->255->-1 | |
| nccl-alltoall-job-oguz-mpiworker-31:53:231 [7] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-0:46:231 [0] NCCL INFO Channel 00/04 : 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 | |
| nccl-alltoall-job-oguz-mpiworker-1:48:231 [2] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-31:51:236 [5] NCCL INFO Trees [0] -1/-1/-1->253->254 [1] -1/-1/-1->253->254 [2] 254/252/-1->253->251 [3] 254/252/-1->253->251 | |
| nccl-alltoall-job-oguz-mpiworker-31:51:236 [5] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-0:46:231 [0] NCCL INFO Channel 01/04 : 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 | |
| nccl-alltoall-job-oguz-mpiworker-31:49:235 [3] NCCL INFO Trees [0] -1/-1/-1->251->250 [1] -1/-1/-1->251->250 [2] 253/249/-1->251->247 [3] 253/249/-1->251->247 | |
| nccl-alltoall-job-oguz-mpiworker-31:49:235 [3] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-0:46:231 [0] NCCL INFO Channel 02/04 : 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 | |
| nccl-alltoall-job-oguz-mpiworker-31:50:237 [4] NCCL INFO Trees [0] 250/254/-1->252->248 [1] 250/254/-1->252->248 [2] -1/-1/-1->252->253 [3] -1/-1/-1->252->253 | |
| nccl-alltoall-job-oguz-mpiworker-31:50:237 [4] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-1:54:234 [7] NCCL INFO Trees [0] -1/-1/-1->15->14 [1] -1/-1/-1->15->14 [2] 23/7/-1->15->31 [3] 23/7/-1->15->31 | |
| nccl-alltoall-job-oguz-mpiworker-1:54:234 [7] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-1:49:236 [3] NCCL INFO Trees [0] -1/-1/-1->11->10 [1] -1/-1/-1->11->10 [2] 13/9/-1->11->7 [3] 13/9/-1->11->7 | |
| nccl-alltoall-job-oguz-mpiworker-1:49:236 [3] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-0:48:233 [2] NCCL INFO Trees [0] 1/3/-1->2->4 [1] 1/3/-1->2->4 [2] -1/-1/-1->2->1 [3] -1/-1/-1->2->1 | |
| nccl-alltoall-job-oguz-mpiworker-0:48:233 [2] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-1:50:233 [4] NCCL INFO Trees [0] 10/14/-1->12->8 [1] 10/14/-1->12->8 [2] -1/-1/-1->12->13 [3] -1/-1/-1->12->13 | |
| nccl-alltoall-job-oguz-mpiworker-1:50:233 [4] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-0:46:231 [0] NCCL INFO Channel 03/04 : 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 | |
| nccl-alltoall-job-oguz-mpiworker-31:53:231 [7] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-1:52:235 [6] NCCL INFO Trees [0] 13/15/-1->14->12 [1] 13/15/-1->14->12 [2] -1/-1/-1->14->13 [3] -1/-1/-1->14->13 | |
| nccl-alltoall-job-oguz-mpiworker-1:52:235 [6] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-0:47:237 [1] NCCL INFO Trees [0] -1/-1/-1->1->2 [1] -1/-1/-1->1->2 [2] 2/0/-1->1->3 [3] 2/0/-1->1->3 | |
| nccl-alltoall-job-oguz-mpiworker-0:47:237 [1] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-31:52:233 [6] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-1:51:232 [5] NCCL INFO Trees [0] -1/-1/-1->13->14 [1] -1/-1/-1->13->14 [2] 14/12/-1->13->11 [3] 14/12/-1->13->11 | |
| nccl-alltoall-job-oguz-mpiworker-1:51:232 [5] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-0:46:231 [0] NCCL INFO Trees [0] 128/-1/-1->0->-1 [1] 128/-1/-1->0->-1 [2] -1/-1/-1->0->1 [3] -1/-1/-1->0->1 | |
| nccl-alltoall-job-oguz-mpiworker-0:46:231 [0] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-31:51:236 [5] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-1:49:236 [3] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-0:48:233 [2] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-31:50:237 [4] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-1:54:234 [7] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-0:46:231 [0] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-31:49:235 [3] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-1:50:233 [4] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-0:47:237 [1] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-1:52:235 [6] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-1:51:232 [5] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-2:46:232 [0] NCCL INFO Trees [0] 8/24/-1->16->32 [1] 8/24/-1->16->32 [2] -1/-1/-1->16->17 [3] -1/-1/-1->16->17 | |
| nccl-alltoall-job-oguz-mpiworker-2:46:232 [0] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-31:48:230 [2] NCCL INFO Trees [0] 249/251/-1->250->252 [1] 249/251/-1->250->252 [2] -1/-1/-1->250->249 [3] -1/-1/-1->250->249 | |
| nccl-alltoall-job-oguz-mpiworker-31:48:230 [2] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-2:46:232 [0] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-31:46:234 [0] NCCL INFO Trees [0] 244/252/-1->248->240 [1] 244/252/-1->248->240 [2] -1/-1/-1->248->249 [3] -1/-1/-1->248->249 | |
| nccl-alltoall-job-oguz-mpiworker-31:46:234 [0] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-31:47:232 [1] NCCL INFO Trees [0] -1/-1/-1->249->250 [1] -1/-1/-1->249->250 [2] 250/248/-1->249->251 [3] 250/248/-1->249->251 | |
| nccl-alltoall-job-oguz-mpiworker-31:47:232 [1] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-2:48:237 [2] NCCL INFO Trees [0] 17/19/-1->18->20 [1] 17/19/-1->18->20 [2] -1/-1/-1->18->17 [3] -1/-1/-1->18->17 | |
| nccl-alltoall-job-oguz-mpiworker-2:48:237 [2] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-31:48:230 [2] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-31:46:234 [0] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-31:47:232 [1] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-2:47:235 [1] NCCL INFO Trees [0] -1/-1/-1->17->18 [1] -1/-1/-1->17->18 [2] 18/16/-1->17->19 [3] 18/16/-1->17->19 | |
| nccl-alltoall-job-oguz-mpiworker-2:47:235 [1] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-2:48:237 [2] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-2:53:230 [7] NCCL INFO Trees [0] -1/-1/-1->23->22 [1] -1/-1/-1->23->22 [2] 27/19/-1->23->15 [3] 27/19/-1->23->15 | |
| nccl-alltoall-job-oguz-mpiworker-2:53:230 [7] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-2:52:233 [6] NCCL INFO Trees [0] 21/23/-1->22->20 [1] 21/23/-1->22->20 [2] -1/-1/-1->22->21 [3] -1/-1/-1->22->21 | |
| nccl-alltoall-job-oguz-mpiworker-2:52:233 [6] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-2:51:234 [5] NCCL INFO Trees [0] -1/-1/-1->21->22 [1] -1/-1/-1->21->22 [2] 22/20/-1->21->19 [3] 22/20/-1->21->19 | |
| nccl-alltoall-job-oguz-mpiworker-2:51:234 [5] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-2:50:231 [4] NCCL INFO Trees [0] 18/22/-1->20->24 [1] 18/22/-1->20->24 [2] -1/-1/-1->20->21 [3] -1/-1/-1->20->21 | |
| nccl-alltoall-job-oguz-mpiworker-2:50:231 [4] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-2:47:235 [1] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-2:49:236 [3] NCCL INFO Trees [0] -1/-1/-1->19->18 [1] -1/-1/-1->19->18 [2] 21/17/-1->19->23 [3] 21/17/-1->19->23 | |
| nccl-alltoall-job-oguz-mpiworker-2:49:236 [3] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-2:51:234 [5] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-30:53:231 [7] NCCL INFO Trees [0] -1/-1/-1->247->246 [1] -1/-1/-1->247->246 [2] 251/243/-1->247->239 [3] 251/243/-1->247->239 | |
| nccl-alltoall-job-oguz-mpiworker-30:53:231 [7] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-2:53:230 [7] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-30:51:233 [5] NCCL INFO Trees [0] -1/-1/-1->245->246 [1] -1/-1/-1->245->246 [2] 246/244/-1->245->243 [3] 246/244/-1->245->243 | |
| nccl-alltoall-job-oguz-mpiworker-30:51:233 [5] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-2:52:233 [6] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-2:50:231 [4] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-2:49:236 [3] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-3:46:233 [0] NCCL INFO Trees [0] 20/28/-1->24->16 [1] 20/28/-1->24->16 [2] -1/-1/-1->24->25 [3] -1/-1/-1->24->25 | |
| nccl-alltoall-job-oguz-mpiworker-3:46:233 [0] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-3:47:230 [1] NCCL INFO Trees [0] -1/-1/-1->25->26 [1] -1/-1/-1->25->26 [2] 26/24/-1->25->27 [3] 26/24/-1->25->27 | |
| nccl-alltoall-job-oguz-mpiworker-3:47:230 [1] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-3:48:231 [2] NCCL INFO Trees [0] 25/27/-1->26->28 [1] 25/27/-1->26->28 [2] -1/-1/-1->26->25 [3] -1/-1/-1->26->25 | |
| nccl-alltoall-job-oguz-mpiworker-3:48:231 [2] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-3:52:237 [6] NCCL INFO Trees [0] 29/31/-1->30->28 [1] 29/31/-1->30->28 [2] -1/-1/-1->30->29 [3] -1/-1/-1->30->29 | |
| nccl-alltoall-job-oguz-mpiworker-3:52:237 [6] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-30:52:235 [6] NCCL INFO Trees [0] 245/247/-1->246->244 [1] 245/247/-1->246->244 [2] -1/-1/-1->246->245 [3] -1/-1/-1->246->245 | |
| nccl-alltoall-job-oguz-mpiworker-30:52:235 [6] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-30:51:233 [5] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-30:53:231 [7] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-3:50:236 [4] NCCL INFO Trees [0] 26/30/-1->28->24 [1] 26/30/-1->28->24 [2] -1/-1/-1->28->29 [3] -1/-1/-1->28->29 | |
| nccl-alltoall-job-oguz-mpiworker-3:50:236 [4] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-30:50:236 [4] NCCL INFO Trees [0] 242/246/-1->244->248 [1] 242/246/-1->244->248 [2] -1/-1/-1->244->245 [3] -1/-1/-1->244->245 | |
| nccl-alltoall-job-oguz-mpiworker-30:50:236 [4] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-3:49:232 [3] NCCL INFO Trees [0] -1/-1/-1->27->26 [1] -1/-1/-1->27->26 [2] 29/25/-1->27->23 [3] 29/25/-1->27->23 | |
| nccl-alltoall-job-oguz-mpiworker-3:49:232 [3] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-3:51:234 [5] NCCL INFO Trees [0] -1/-1/-1->29->30 [1] -1/-1/-1->29->30 [2] 30/28/-1->29->27 [3] 30/28/-1->29->27 | |
| nccl-alltoall-job-oguz-mpiworker-3:51:234 [5] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-3:46:233 [0] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-30:52:235 [6] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-30:50:236 [4] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-3:47:230 [1] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-4:48:233 [2] NCCL INFO Trees [0] 33/35/-1->34->36 [1] 33/35/-1->34->36 [2] -1/-1/-1->34->33 [3] -1/-1/-1->34->33 | |
| nccl-alltoall-job-oguz-mpiworker-4:48:233 [2] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-3:48:231 [2] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-29:53:235 [7] NCCL INFO Trees [0] -1/-1/-1->239->238 [1] -1/-1/-1->239->238 [2] 247/231/-1->239->223 [3] 247/231/-1->239->223 | |
| nccl-alltoall-job-oguz-mpiworker-29:53:235 [7] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-3:52:237 [6] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-29:52:237 [6] NCCL INFO Trees [0] 237/239/-1->238->236 [1] 237/239/-1->238->236 [2] -1/-1/-1->238->237 [3] -1/-1/-1->238->237 | |
| nccl-alltoall-job-oguz-mpiworker-29:52:237 [6] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-3:51:234 [5] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-29:51:230 [5] NCCL INFO Trees [0] -1/-1/-1->237->238 [1] -1/-1/-1->237->238 [2] 238/236/-1->237->235 [3] 238/236/-1->237->235 | |
| nccl-alltoall-job-oguz-mpiworker-29:51:230 [5] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-30:49:234 [3] NCCL INFO Trees [0] -1/-1/-1->243->242 [1] -1/-1/-1->243->242 [2] 245/241/-1->243->247 [3] 245/241/-1->243->247 | |
| nccl-alltoall-job-oguz-mpiworker-30:49:234 [3] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-3:53:235 [7] NCCL INFO Trees [0] -1/-1/-1->31->30 [1] -1/-1/-1->31->30 [2] 47/15/-1->31->63 [3] 47/15/-1->31->63 | |
| nccl-alltoall-job-oguz-mpiworker-3:53:235 [7] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-29:50:234 [4] NCCL INFO Trees [0] 234/238/-1->236->232 [1] 234/238/-1->236->232 [2] -1/-1/-1->236->237 [3] -1/-1/-1->236->237 | |
| nccl-alltoall-job-oguz-mpiworker-29:50:234 [4] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-3:50:236 [4] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-5:46:230 [0] NCCL INFO Trees [0] 36/44/-1->40->48 [1] 36/44/-1->40->48 [2] -1/-1/-1->40->41 [3] -1/-1/-1->40->41 | |
| nccl-alltoall-job-oguz-mpiworker-5:46:230 [0] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-3:49:232 [3] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-5:47:235 [1] NCCL INFO Trees [0] -1/-1/-1->41->42 [1] -1/-1/-1->41->42 [2] 42/40/-1->41->43 [3] 42/40/-1->41->43 | |
| nccl-alltoall-job-oguz-mpiworker-5:47:235 [1] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-3:53:235 [7] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-29:53:235 [7] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-29:51:230 [5] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-5:48:234 [2] NCCL INFO Trees [0] 41/43/-1->42->44 [1] 41/43/-1->42->44 [2] -1/-1/-1->42->41 [3] -1/-1/-1->42->41 | |
| nccl-alltoall-job-oguz-mpiworker-5:48:234 [2] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-29:52:237 [6] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-30:48:237 [2] NCCL INFO Trees [0] 241/243/-1->242->244 [1] 241/243/-1->242->244 [2] -1/-1/-1->242->241 [3] -1/-1/-1->242->241 | |
| nccl-alltoall-job-oguz-mpiworker-30:48:237 [2] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-29:50:234 [4] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-6:53:230 [7] NCCL INFO Trees [0] -1/-1/-1->55->54 [1] -1/-1/-1->55->54 [2] 59/51/-1->55->47 [3] 59/51/-1->55->47 | |
| nccl-alltoall-job-oguz-mpiworker-6:53:230 [7] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-5:49:232 [3] NCCL INFO Trees [0] -1/-1/-1->43->42 [1] -1/-1/-1->43->42 [2] 45/41/-1->43->39 [3] 45/41/-1->43->39 | |
| nccl-alltoall-job-oguz-mpiworker-5:49:232 [3] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-5:51:236 [5] NCCL INFO Trees [0] -1/-1/-1->45->46 [1] -1/-1/-1->45->46 [2] 46/44/-1->45->43 [3] 46/44/-1->45->43 | |
| nccl-alltoall-job-oguz-mpiworker-5:51:236 [5] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-30:46:230 [0] NCCL INFO Trees [0] 232/248/-1->240->224 [1] 232/248/-1->240->224 [2] -1/-1/-1->240->241 [3] -1/-1/-1->240->241 | |
| nccl-alltoall-job-oguz-mpiworker-30:46:230 [0] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-5:46:230 [0] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-30:49:234 [3] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-5:47:235 [1] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-30:47:232 [1] NCCL INFO Trees [0] -1/-1/-1->241->242 [1] -1/-1/-1->241->242 [2] 242/240/-1->241->243 [3] 242/240/-1->241->243 | |
| nccl-alltoall-job-oguz-mpiworker-30:47:232 [1] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-30:48:237 [2] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-4:53:235 [7] NCCL INFO Trees [0] -1/-1/-1->39->38 [1] -1/-1/-1->39->38 [2] 43/35/-1->39->47 [3] 43/35/-1->39->47 | |
| nccl-alltoall-job-oguz-mpiworker-4:53:235 [7] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-30:46:230 [0] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-4:49:234 [3] NCCL INFO Trees [0] -1/-1/-1->35->34 [1] -1/-1/-1->35->34 [2] 37/33/-1->35->39 [3] 37/33/-1->35->39 | |
| nccl-alltoall-job-oguz-mpiworker-4:49:234 [3] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-5:48:234 [2] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-4:52:237 [6] NCCL INFO Trees [0] 37/39/-1->38->36 [1] 37/39/-1->38->36 [2] -1/-1/-1->38->37 [3] -1/-1/-1->38->37 | |
| nccl-alltoall-job-oguz-mpiworker-4:52:237 [6] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-6:52:233 [6] NCCL INFO Trees [0] 53/55/-1->54->52 [1] 53/55/-1->54->52 [2] -1/-1/-1->54->53 [3] -1/-1/-1->54->53 | |
| nccl-alltoall-job-oguz-mpiworker-6:52:233 [6] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-4:50:230 [4] NCCL INFO Trees [0] 34/38/-1->36->40 [1] 34/38/-1->36->40 [2] -1/-1/-1->36->37 [3] -1/-1/-1->36->37 | |
| nccl-alltoall-job-oguz-mpiworker-4:50:230 [4] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-6:51:235 [5] NCCL INFO Trees [0] -1/-1/-1->53->54 [1] -1/-1/-1->53->54 [2] 54/52/-1->53->51 [3] 54/52/-1->53->51 | |
| nccl-alltoall-job-oguz-mpiworker-6:51:235 [5] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-30:47:232 [1] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-4:47:232 [1] NCCL INFO Trees [0] -1/-1/-1->33->34 [1] -1/-1/-1->33->34 [2] 34/32/-1->33->35 [3] 34/32/-1->33->35 | |
| nccl-alltoall-job-oguz-mpiworker-4:47:232 [1] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-6:50:237 [4] NCCL INFO Trees [0] 50/54/-1->52->56 [1] 50/54/-1->52->56 [2] -1/-1/-1->52->53 [3] -1/-1/-1->52->53 | |
| nccl-alltoall-job-oguz-mpiworker-6:50:237 [4] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-4:51:236 [5] NCCL INFO Trees [0] -1/-1/-1->37->38 [1] -1/-1/-1->37->38 [2] 38/36/-1->37->35 [3] 38/36/-1->37->35 | |
| nccl-alltoall-job-oguz-mpiworker-4:51:236 [5] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-6:49:234 [3] NCCL INFO Trees [0] -1/-1/-1->51->50 [1] -1/-1/-1->51->50 [2] 53/49/-1->51->55 [3] 53/49/-1->51->55 | |
| nccl-alltoall-job-oguz-mpiworker-6:49:234 [3] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-4:46:231 [0] NCCL INFO Trees [0] 16/48/-1->32->64 [1] 16/48/-1->32->64 [2] -1/-1/-1->32->33 [3] -1/-1/-1->32->33 | |
| nccl-alltoall-job-oguz-mpiworker-4:46:231 [0] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-6:48:231 [2] NCCL INFO Trees [0] 49/51/-1->50->52 [1] 49/51/-1->50->52 [2] -1/-1/-1->50->49 [3] -1/-1/-1->50->49 | |
| nccl-alltoall-job-oguz-mpiworker-6:48:231 [2] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-4:53:235 [7] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-6:46:236 [0] NCCL INFO Trees [0] 40/56/-1->48->32 [1] 40/56/-1->48->32 [2] -1/-1/-1->48->49 [3] -1/-1/-1->48->49 | |
| nccl-alltoall-job-oguz-mpiworker-6:46:236 [0] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-5:53:233 [7] NCCL INFO Trees [0] -1/-1/-1->47->46 [1] -1/-1/-1->47->46 [2] 55/39/-1->47->31 [3] 55/39/-1->47->31 | |
| nccl-alltoall-job-oguz-mpiworker-5:53:233 [7] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-4:49:234 [3] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-5:49:232 [3] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-4:50:230 [4] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-5:51:236 [5] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-4:48:233 [2] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-5:50:231 [4] NCCL INFO Trees [0] 42/46/-1->44->40 [1] 42/46/-1->44->40 [2] -1/-1/-1->44->45 [3] -1/-1/-1->44->45 | |
| nccl-alltoall-job-oguz-mpiworker-5:50:231 [4] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-4:46:231 [0] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-5:52:237 [6] NCCL INFO Trees [0] 45/47/-1->46->44 [1] 45/47/-1->46->44 [2] -1/-1/-1->46->45 [3] -1/-1/-1->46->45 | |
| nccl-alltoall-job-oguz-mpiworker-5:52:237 [6] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-6:47:232 [1] NCCL INFO Trees [0] -1/-1/-1->49->50 [1] -1/-1/-1->49->50 [2] 50/48/-1->49->51 [3] 50/48/-1->49->51 | |
| nccl-alltoall-job-oguz-mpiworker-6:47:232 [1] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-4:52:237 [6] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-5:53:233 [7] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-6:51:235 [5] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-4:47:232 [1] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-5:50:231 [4] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-6:53:230 [7] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-4:51:236 [5] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-5:52:237 [6] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-6:52:233 [6] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-6:50:237 [4] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-6:48:231 [2] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-6:46:236 [0] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-7:46:230 [0] NCCL INFO Trees [0] 52/60/-1->56->48 [1] 52/60/-1->56->48 [2] -1/-1/-1->56->57 [3] -1/-1/-1->56->57 | |
| nccl-alltoall-job-oguz-mpiworker-7:46:230 [0] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-6:49:234 [3] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-6:47:232 [1] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-7:48:237 [2] NCCL INFO Trees [0] 57/59/-1->58->60 [1] 57/59/-1->58->60 [2] -1/-1/-1->58->57 [3] -1/-1/-1->58->57 | |
| nccl-alltoall-job-oguz-mpiworker-7:48:237 [2] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-7:47:233 [1] NCCL INFO Trees [0] -1/-1/-1->57->58 [1] -1/-1/-1->57->58 [2] 58/56/-1->57->59 [3] 58/56/-1->57->59 | |
| nccl-alltoall-job-oguz-mpiworker-7:47:233 [1] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-7:46:230 [0] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-7:51:232 [5] NCCL INFO Trees [0] -1/-1/-1->61->62 [1] -1/-1/-1->61->62 [2] 62/60/-1->61->59 [3] 62/60/-1->61->59 | |
| nccl-alltoall-job-oguz-mpiworker-7:51:232 [5] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-7:52:231 [6] NCCL INFO Trees [0] 61/63/-1->62->60 [1] 61/63/-1->62->60 [2] -1/-1/-1->62->61 [3] -1/-1/-1->62->61 | |
| nccl-alltoall-job-oguz-mpiworker-7:52:231 [6] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-7:50:234 [4] NCCL INFO Trees [0] 58/62/-1->60->56 [1] 58/62/-1->60->56 [2] -1/-1/-1->60->61 [3] -1/-1/-1->60->61 | |
| nccl-alltoall-job-oguz-mpiworker-7:50:234 [4] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-7:54:236 [7] NCCL INFO Trees [0] -1/-1/-1->63->62 [1] -1/-1/-1->63->62 [2] 95/31/-1->63->127 [3] 95/31/-1->63->127 | |
| nccl-alltoall-job-oguz-mpiworker-7:54:236 [7] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-7:49:235 [3] NCCL INFO Trees [0] -1/-1/-1->59->58 [1] -1/-1/-1->59->58 [2] 61/57/-1->59->55 [3] 61/57/-1->59->55 | |
| nccl-alltoall-job-oguz-mpiworker-7:49:235 [3] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-7:48:237 [2] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-7:47:233 [1] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-7:51:232 [5] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-7:50:234 [4] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-8:46:231 [0] NCCL INFO Trees [0] 32/96/-1->64->128 [1] 32/96/-1->64->128 [2] -1/-1/-1->64->65 [3] -1/-1/-1->64->65 | |
| nccl-alltoall-job-oguz-mpiworker-8:46:231 [0] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-7:52:231 [6] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-7:54:236 [7] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-7:49:235 [3] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-8:48:235 [2] NCCL INFO Trees [0] 65/67/-1->66->68 [1] 65/67/-1->66->68 [2] -1/-1/-1->66->65 [3] -1/-1/-1->66->65 | |
| nccl-alltoall-job-oguz-mpiworker-8:48:235 [2] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-8:46:231 [0] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-8:53:230 [7] NCCL INFO Trees [0] -1/-1/-1->71->70 [1] -1/-1/-1->71->70 [2] 75/67/-1->71->79 [3] 75/67/-1->71->79 | |
| nccl-alltoall-job-oguz-mpiworker-8:53:230 [7] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-8:51:234 [5] NCCL INFO Trees [0] -1/-1/-1->69->70 [1] -1/-1/-1->69->70 [2] 70/68/-1->69->67 [3] 70/68/-1->69->67 | |
| nccl-alltoall-job-oguz-mpiworker-8:51:234 [5] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-8:48:235 [2] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-8:47:232 [1] NCCL INFO Trees [0] -1/-1/-1->65->66 [1] -1/-1/-1->65->66 [2] 66/64/-1->65->67 [3] 66/64/-1->65->67 | |
| nccl-alltoall-job-oguz-mpiworker-8:47:232 [1] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-8:53:230 [7] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-8:51:234 [5] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-8:49:237 [3] NCCL INFO Trees [0] -1/-1/-1->67->66 [1] -1/-1/-1->67->66 [2] 69/65/-1->67->71 [3] 69/65/-1->67->71 | |
| nccl-alltoall-job-oguz-mpiworker-8:49:237 [3] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-8:52:236 [6] NCCL INFO Trees [0] 69/71/-1->70->68 [1] 69/71/-1->70->68 [2] -1/-1/-1->70->69 [3] -1/-1/-1->70->69 | |
| nccl-alltoall-job-oguz-mpiworker-8:52:236 [6] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-8:50:233 [4] NCCL INFO Trees [0] 66/70/-1->68->72 [1] 66/70/-1->68->72 [2] -1/-1/-1->68->69 [3] -1/-1/-1->68->69 | |
| nccl-alltoall-job-oguz-mpiworker-8:50:233 [4] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-8:47:232 [1] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-8:49:237 [3] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-8:50:233 [4] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-8:52:236 [6] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-9:46:231 [0] NCCL INFO Trees [0] 68/76/-1->72->80 [1] 68/76/-1->72->80 [2] -1/-1/-1->72->73 [3] -1/-1/-1->72->73 | |
| nccl-alltoall-job-oguz-mpiworker-9:46:231 [0] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-9:47:230 [1] NCCL INFO Trees [0] -1/-1/-1->73->74 [1] -1/-1/-1->73->74 [2] 74/72/-1->73->75 [3] 74/72/-1->73->75 | |
| nccl-alltoall-job-oguz-mpiworker-9:47:230 [1] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-9:48:233 [2] NCCL INFO Trees [0] 73/75/-1->74->76 [1] 73/75/-1->74->76 [2] -1/-1/-1->74->73 [3] -1/-1/-1->74->73 | |
| nccl-alltoall-job-oguz-mpiworker-9:48:233 [2] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-9:49:232 [3] NCCL INFO Trees [0] -1/-1/-1->75->74 [1] -1/-1/-1->75->74 [2] 77/73/-1->75->71 [3] 77/73/-1->75->71 | |
| nccl-alltoall-job-oguz-mpiworker-9:49:232 [3] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-9:46:231 [0] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-9:47:230 [1] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-9:48:233 [2] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-9:51:236 [5] NCCL INFO Trees [0] -1/-1/-1->77->78 [1] -1/-1/-1->77->78 [2] 78/76/-1->77->75 [3] 78/76/-1->77->75 | |
| nccl-alltoall-job-oguz-mpiworker-9:51:236 [5] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-9:53:235 [7] NCCL INFO Trees [0] -1/-1/-1->79->78 [1] -1/-1/-1->79->78 [2] 87/71/-1->79->95 [3] 87/71/-1->79->95 | |
| nccl-alltoall-job-oguz-mpiworker-9:53:235 [7] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-9:49:232 [3] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-9:51:236 [5] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-9:53:235 [7] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-10:46:230 [0] NCCL INFO Trees [0] 72/88/-1->80->96 [1] 72/88/-1->80->96 [2] -1/-1/-1->80->81 [3] -1/-1/-1->80->81 | |
| nccl-alltoall-job-oguz-mpiworker-10:46:230 [0] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-9:52:234 [6] NCCL INFO Trees [0] 77/79/-1->78->76 [1] 77/79/-1->78->76 [2] -1/-1/-1->78->77 [3] -1/-1/-1->78->77 | |
| nccl-alltoall-job-oguz-mpiworker-9:52:234 [6] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-9:50:237 [4] NCCL INFO Trees [0] 74/78/-1->76->72 [1] 74/78/-1->76->72 [2] -1/-1/-1->76->77 [3] -1/-1/-1->76->77 | |
| nccl-alltoall-job-oguz-mpiworker-9:50:237 [4] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-10:47:236 [1] NCCL INFO Trees [0] -1/-1/-1->81->82 [1] -1/-1/-1->81->82 [2] 82/80/-1->81->83 [3] 82/80/-1->81->83 | |
| nccl-alltoall-job-oguz-mpiworker-10:47:236 [1] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-25:53:231 [7] NCCL INFO Trees [0] -1/-1/-1->207->206 [1] -1/-1/-1->207->206 [2] 215/199/-1->207->223 [3] 215/199/-1->207->223 | |
| nccl-alltoall-job-oguz-mpiworker-25:53:231 [7] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-10:46:230 [0] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-9:52:234 [6] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-9:50:237 [4] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-28:48:231 [2] NCCL INFO Trees [0] 225/227/-1->226->228 [1] 225/227/-1->226->228 [2] -1/-1/-1->226->225 [3] -1/-1/-1->226->225 | |
| nccl-alltoall-job-oguz-mpiworker-28:48:231 [2] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-10:48:231 [2] NCCL INFO Trees [0] 81/83/-1->82->84 [1] 81/83/-1->82->84 [2] -1/-1/-1->82->81 [3] -1/-1/-1->82->81 | |
| nccl-alltoall-job-oguz-mpiworker-10:48:231 [2] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-10:47:236 [1] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-28:53:235 [7] NCCL INFO Trees [0] -1/-1/-1->231->230 [1] -1/-1/-1->231->230 [2] 235/227/-1->231->239 [3] 235/227/-1->231->239 | |
| nccl-alltoall-job-oguz-mpiworker-28:53:235 [7] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-26:46:231 [0] NCCL INFO Trees [0] 200/216/-1->208->224 [1] 200/216/-1->208->224 [2] -1/-1/-1->208->209 [3] -1/-1/-1->208->209 | |
| nccl-alltoall-job-oguz-mpiworker-26:46:231 [0] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-28:51:236 [5] NCCL INFO Trees [0] -1/-1/-1->229->230 [1] -1/-1/-1->229->230 [2] 230/228/-1->229->227 [3] 230/228/-1->229->227 | |
| nccl-alltoall-job-oguz-mpiworker-28:51:236 [5] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-28:52:232 [6] NCCL INFO Trees [0] 229/231/-1->230->228 [1] 229/231/-1->230->228 [2] -1/-1/-1->230->229 [3] -1/-1/-1->230->229 | |
| nccl-alltoall-job-oguz-mpiworker-28:52:232 [6] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-21:53:233 [7] NCCL INFO Trees [0] -1/-1/-1->175->174 [1] -1/-1/-1->175->174 [2] 183/167/-1->175->159 [3] 183/167/-1->175->159 | |
| nccl-alltoall-job-oguz-mpiworker-21:53:233 [7] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-10:48:231 [2] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-28:50:234 [4] NCCL INFO Trees [0] 226/230/-1->228->232 [1] 226/230/-1->228->232 [2] -1/-1/-1->228->229 [3] -1/-1/-1->228->229 | |
| nccl-alltoall-job-oguz-mpiworker-28:50:234 [4] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-29:46:232 [0] NCCL INFO Trees [0] 228/236/-1->232->240 [1] 228/236/-1->232->240 [2] -1/-1/-1->232->233 [3] -1/-1/-1->232->233 | |
| nccl-alltoall-job-oguz-mpiworker-29:46:232 [0] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-25:51:232 [5] NCCL INFO Trees [0] -1/-1/-1->205->206 [1] -1/-1/-1->205->206 [2] 206/204/-1->205->203 [3] 206/204/-1->205->203 | |
| nccl-alltoall-job-oguz-mpiworker-25:51:232 [5] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-28:48:231 [2] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-29:48:231 [2] NCCL INFO Trees [0] 233/235/-1->234->236 [1] 233/235/-1->234->236 [2] -1/-1/-1->234->233 [3] -1/-1/-1->234->233 | |
| nccl-alltoall-job-oguz-mpiworker-29:48:231 [2] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-25:52:236 [6] NCCL INFO Trees [0] 205/207/-1->206->204 [1] 205/207/-1->206->204 [2] -1/-1/-1->206->205 [3] -1/-1/-1->206->205 | |
| nccl-alltoall-job-oguz-mpiworker-25:52:236 [6] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-22:46:231 [0] NCCL INFO Trees [0] 168/184/-1->176->160 [1] 168/184/-1->176->160 [2] -1/-1/-1->176->177 [3] -1/-1/-1->176->177 | |
| nccl-alltoall-job-oguz-mpiworker-22:46:231 [0] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-28:53:235 [7] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-29:48:231 [2] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-25:50:235 [4] NCCL INFO Trees [0] 202/206/-1->204->200 [1] 202/206/-1->204->200 [2] -1/-1/-1->204->205 [3] -1/-1/-1->204->205 | |
| nccl-alltoall-job-oguz-mpiworker-25:50:235 [4] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-22:48:236 [2] NCCL INFO Trees [0] 177/179/-1->178->180 [1] 177/179/-1->178->180 [2] -1/-1/-1->178->177 [3] -1/-1/-1->178->177 | |
| nccl-alltoall-job-oguz-mpiworker-22:48:236 [2] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-26:52:237 [6] NCCL INFO Trees [0] 213/215/-1->214->212 [1] 213/215/-1->214->212 [2] -1/-1/-1->214->213 [3] -1/-1/-1->214->213 | |
| nccl-alltoall-job-oguz-mpiworker-26:52:237 [6] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-21:53:233 [7] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-10:49:237 [3] NCCL INFO Trees [0] -1/-1/-1->83->82 [1] -1/-1/-1->83->82 [2] 85/81/-1->83->87 [3] 85/81/-1->83->87 | |
| nccl-alltoall-job-oguz-mpiworker-10:49:237 [3] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-28:51:236 [5] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-29:46:232 [0] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-25:49:233 [3] NCCL INFO Trees [0] -1/-1/-1->203->202 [1] -1/-1/-1->203->202 [2] 205/201/-1->203->199 [3] 205/201/-1->203->199 | |
| nccl-alltoall-job-oguz-mpiworker-25:49:233 [3] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-22:53:234 [7] NCCL INFO Trees [0] -1/-1/-1->183->182 [1] -1/-1/-1->183->182 [2] 187/179/-1->183->175 [3] 187/179/-1->183->175 | |
| nccl-alltoall-job-oguz-mpiworker-22:53:234 [7] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-28:49:237 [3] NCCL INFO Trees [0] -1/-1/-1->227->226 [1] -1/-1/-1->227->226 [2] 229/225/-1->227->231 [3] 229/225/-1->227->231 | |
| nccl-alltoall-job-oguz-mpiworker-28:49:237 [3] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-29:47:233 [1] NCCL INFO Trees [0] -1/-1/-1->233->234 [1] -1/-1/-1->233->234 [2] 234/232/-1->233->235 [3] 234/232/-1->233->235 | |
| nccl-alltoall-job-oguz-mpiworker-29:47:233 [1] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-25:51:232 [5] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-22:51:233 [5] NCCL INFO Trees [0] -1/-1/-1->181->182 [1] -1/-1/-1->181->182 [2] 182/180/-1->181->179 [3] 182/180/-1->181->179 | |
| nccl-alltoall-job-oguz-mpiworker-22:51:233 [5] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-10:53:235 [7] NCCL INFO Trees [0] -1/-1/-1->87->86 [1] -1/-1/-1->87->86 [2] 91/83/-1->87->79 [3] 91/83/-1->87->79 | |
| nccl-alltoall-job-oguz-mpiworker-10:53:235 [7] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-28:52:232 [6] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-29:49:236 [3] NCCL INFO Trees [0] -1/-1/-1->235->234 [1] -1/-1/-1->235->234 [2] 237/233/-1->235->231 [3] 237/233/-1->235->231 | |
| nccl-alltoall-job-oguz-mpiworker-29:49:236 [3] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-25:50:235 [4] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-22:52:235 [6] NCCL INFO Trees [0] 181/183/-1->182->180 [1] 181/183/-1->182->180 [2] -1/-1/-1->182->181 [3] -1/-1/-1->182->181 | |
| nccl-alltoall-job-oguz-mpiworker-22:52:235 [6] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-10:50:234 [4] NCCL INFO Trees [0] 82/86/-1->84->88 [1] 82/86/-1->84->88 [2] -1/-1/-1->84->85 [3] -1/-1/-1->84->85 | |
| nccl-alltoall-job-oguz-mpiworker-10:50:234 [4] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-28:50:234 [4] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-29:49:236 [3] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-25:53:231 [7] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-22:50:232 [4] NCCL INFO Trees [0] 178/182/-1->180->184 [1] 178/182/-1->180->184 [2] -1/-1/-1->180->181 [3] -1/-1/-1->180->181 | |
| nccl-alltoall-job-oguz-mpiworker-22:50:232 [4] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-10:51:232 [5] NCCL INFO Trees [0] -1/-1/-1->85->86 [1] -1/-1/-1->85->86 [2] 86/84/-1->85->83 [3] 86/84/-1->85->83 | |
| nccl-alltoall-job-oguz-mpiworker-10:51:232 [5] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-27:48:235 [2] NCCL INFO Trees [0] 217/219/-1->218->220 [1] 217/219/-1->218->220 [2] -1/-1/-1->218->217 [3] -1/-1/-1->218->217 | |
| nccl-alltoall-job-oguz-mpiworker-27:48:235 [2] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-28:49:237 [3] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-29:47:233 [1] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-25:52:236 [6] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-22:48:236 [2] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-10:52:233 [6] NCCL INFO Trees [0] 85/87/-1->86->84 [1] 85/87/-1->86->84 [2] -1/-1/-1->86->85 [3] -1/-1/-1->86->85 | |
| nccl-alltoall-job-oguz-mpiworker-10:52:233 [6] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-27:53:237 [7] NCCL INFO Trees [0] -1/-1/-1->223->222 [1] -1/-1/-1->223->222 [2] 239/207/-1->223->191 [3] 239/207/-1->223->191 | |
| nccl-alltoall-job-oguz-mpiworker-27:53:237 [7] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-23:48:232 [2] NCCL INFO Trees [0] 185/187/-1->186->188 [1] 185/187/-1->186->188 [2] -1/-1/-1->186->185 [3] -1/-1/-1->186->185 | |
| nccl-alltoall-job-oguz-mpiworker-23:48:232 [2] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-28:47:230 [1] NCCL INFO Trees [0] -1/-1/-1->225->226 [1] -1/-1/-1->225->226 [2] 226/224/-1->225->227 [3] 226/224/-1->225->227 | |
| nccl-alltoall-job-oguz-mpiworker-28:47:230 [1] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-25:49:233 [3] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-22:46:231 [0] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-10:49:237 [3] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-27:49:233 [3] NCCL INFO Trees [0] -1/-1/-1->219->218 [1] -1/-1/-1->219->218 [2] 221/217/-1->219->215 [3] 221/217/-1->219->215 | |
| nccl-alltoall-job-oguz-mpiworker-27:49:233 [3] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-23:47:233 [1] NCCL INFO Trees [0] -1/-1/-1->185->186 [1] -1/-1/-1->185->186 [2] 186/184/-1->185->187 [3] 186/184/-1->185->187 | |
| nccl-alltoall-job-oguz-mpiworker-23:47:233 [1] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-26:51:236 [5] NCCL INFO Trees [0] -1/-1/-1->213->214 [1] -1/-1/-1->213->214 [2] 214/212/-1->213->211 [3] 214/212/-1->213->211 | |
| nccl-alltoall-job-oguz-mpiworker-26:51:236 [5] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-24:46:237 [0] NCCL INFO Trees [0] 160/224/-1->192->128 [1] 160/224/-1->192->128 [2] -1/-1/-1->192->193 [3] -1/-1/-1->192->193 | |
| nccl-alltoall-job-oguz-mpiworker-24:46:237 [0] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-28:46:233 [0] NCCL INFO Trees [0] 208/240/-1->224->192 [1] 208/240/-1->224->192 [2] -1/-1/-1->224->225 [3] -1/-1/-1->224->225 | |
| nccl-alltoall-job-oguz-mpiworker-28:46:233 [0] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-22:53:234 [7] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-10:50:234 [4] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-27:51:234 [5] NCCL INFO Trees [0] -1/-1/-1->221->222 [1] -1/-1/-1->221->222 [2] 222/220/-1->221->219 [3] 222/220/-1->221->219 | |
| nccl-alltoall-job-oguz-mpiworker-27:51:234 [5] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-23:53:231 [7] NCCL INFO Trees [0] -1/-1/-1->191->190 [1] -1/-1/-1->191->190 [2] 223/159/-1->191->127 [3] 223/159/-1->191->127 | |
| nccl-alltoall-job-oguz-mpiworker-23:53:231 [7] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-26:50:234 [4] NCCL INFO Trees [0] 210/214/-1->212->216 [1] 210/214/-1->212->216 [2] -1/-1/-1->212->213 [3] -1/-1/-1->212->213 | |
| nccl-alltoall-job-oguz-mpiworker-26:50:234 [4] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-28:47:230 [1] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-22:47:230 [1] NCCL INFO Trees [0] -1/-1/-1->177->178 [1] -1/-1/-1->177->178 [2] 178/176/-1->177->179 [3] 178/176/-1->177->179 | |
| nccl-alltoall-job-oguz-mpiworker-22:47:230 [1] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-10:51:232 [5] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-27:53:237 [7] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-23:51:235 [5] NCCL INFO Trees [0] -1/-1/-1->189->190 [1] -1/-1/-1->189->190 [2] 190/188/-1->189->187 [3] 190/188/-1->189->187 | |
| nccl-alltoall-job-oguz-mpiworker-23:51:235 [5] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-26:53:232 [7] NCCL INFO Trees [0] -1/-1/-1->215->214 [1] -1/-1/-1->215->214 [2] 219/211/-1->215->207 [3] 219/211/-1->215->207 | |
| nccl-alltoall-job-oguz-mpiworker-26:53:232 [7] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-24:48:230 [2] NCCL INFO Trees [0] 193/195/-1->194->196 [1] 193/195/-1->194->196 [2] -1/-1/-1->194->193 [3] -1/-1/-1->194->193 | |
| nccl-alltoall-job-oguz-mpiworker-24:48:230 [2] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-25:48:237 [2] NCCL INFO Trees [0] 201/203/-1->202->204 [1] 201/203/-1->202->204 [2] -1/-1/-1->202->201 [3] -1/-1/-1->202->201 | |
| nccl-alltoall-job-oguz-mpiworker-25:48:237 [2] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-11:46:230 [0] NCCL INFO Trees [0] 84/92/-1->88->80 [1] 84/92/-1->88->80 [2] -1/-1/-1->88->89 [3] -1/-1/-1->88->89 | |
| nccl-alltoall-job-oguz-mpiworker-11:46:230 [0] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-28:46:233 [0] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-22:51:233 [5] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-10:52:233 [6] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-27:48:235 [2] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-23:52:237 [6] NCCL INFO Trees [0] 189/191/-1->190->188 [1] 189/191/-1->190->188 [2] -1/-1/-1->190->189 [3] -1/-1/-1->190->189 | |
| nccl-alltoall-job-oguz-mpiworker-23:52:237 [6] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-26:46:231 [0] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-24:47:231 [1] NCCL INFO Trees [0] -1/-1/-1->193->194 [1] -1/-1/-1->193->194 [2] 194/192/-1->193->195 [3] 194/192/-1->193->195 | |
| nccl-alltoall-job-oguz-mpiworker-24:47:231 [1] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-22:52:235 [6] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-10:53:235 [7] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-27:46:231 [0] NCCL INFO Trees [0] 212/220/-1->216->208 [1] 212/220/-1->216->208 [2] -1/-1/-1->216->217 [3] -1/-1/-1->216->217 | |
| nccl-alltoall-job-oguz-mpiworker-27:46:231 [0] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-23:50:234 [4] NCCL INFO Trees [0] 186/190/-1->188->184 [1] 186/190/-1->188->184 [2] -1/-1/-1->188->189 [3] -1/-1/-1->188->189 | |
| nccl-alltoall-job-oguz-mpiworker-23:50:234 [4] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-26:49:233 [3] NCCL INFO Trees [0] -1/-1/-1->211->210 [1] -1/-1/-1->211->210 [2] 213/209/-1->211->215 [3] 213/209/-1->211->215 | |
| nccl-alltoall-job-oguz-mpiworker-26:49:233 [3] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-24:49:233 [3] NCCL INFO Trees [0] -1/-1/-1->195->194 [1] -1/-1/-1->195->194 [2] 197/193/-1->195->199 [3] 197/193/-1->195->199 | |
| nccl-alltoall-job-oguz-mpiworker-24:49:233 [3] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-25:47:234 [1] NCCL INFO Trees [0] -1/-1/-1->201->202 [1] -1/-1/-1->201->202 [2] 202/200/-1->201->203 [3] 202/200/-1->201->203 | |
| nccl-alltoall-job-oguz-mpiworker-25:47:234 [1] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-11:47:235 [1] NCCL INFO Trees [0] -1/-1/-1->89->90 [1] -1/-1/-1->89->90 [2] 90/88/-1->89->91 [3] 90/88/-1->89->91 | |
| nccl-alltoall-job-oguz-mpiworker-11:47:235 [1] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-22:50:232 [4] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-27:47:236 [1] NCCL INFO Trees [0] -1/-1/-1->217->218 [1] -1/-1/-1->217->218 [2] 218/216/-1->217->219 [3] 218/216/-1->217->219 | |
| nccl-alltoall-job-oguz-mpiworker-27:47:236 [1] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-23:46:230 [0] NCCL INFO Trees [0] 180/188/-1->184->176 [1] 180/188/-1->184->176 [2] -1/-1/-1->184->185 [3] -1/-1/-1->184->185 | |
| nccl-alltoall-job-oguz-mpiworker-23:46:230 [0] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-26:47:230 [1] NCCL INFO Trees [0] -1/-1/-1->209->210 [1] -1/-1/-1->209->210 [2] 210/208/-1->209->211 [3] 210/208/-1->209->211 | |
| nccl-alltoall-job-oguz-mpiworker-26:47:230 [1] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-24:53:235 [7] NCCL INFO Trees [0] -1/-1/-1->199->198 [1] -1/-1/-1->199->198 [2] 203/195/-1->199->207 [3] 203/195/-1->199->207 | |
| nccl-alltoall-job-oguz-mpiworker-24:53:235 [7] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-25:46:230 [0] NCCL INFO Trees [0] 196/204/-1->200->208 [1] 196/204/-1->200->208 [2] -1/-1/-1->200->201 [3] -1/-1/-1->200->201 | |
| nccl-alltoall-job-oguz-mpiworker-25:46:230 [0] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-11:46:230 [0] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-22:49:237 [3] NCCL INFO Trees [0] -1/-1/-1->179->178 [1] -1/-1/-1->179->178 [2] 181/177/-1->179->183 [3] 181/177/-1->179->183 | |
| nccl-alltoall-job-oguz-mpiworker-22:49:237 [3] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-27:52:230 [6] NCCL INFO Trees [0] 221/223/-1->222->220 [1] 221/223/-1->222->220 [2] -1/-1/-1->222->221 [3] -1/-1/-1->222->221 | |
| nccl-alltoall-job-oguz-mpiworker-27:52:230 [6] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-23:48:232 [2] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-26:48:235 [2] NCCL INFO Trees [0] 209/211/-1->210->212 [1] 209/211/-1->210->212 [2] -1/-1/-1->210->209 [3] -1/-1/-1->210->209 | |
| nccl-alltoall-job-oguz-mpiworker-26:48:235 [2] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-24:51:234 [5] NCCL INFO Trees [0] -1/-1/-1->197->198 [1] -1/-1/-1->197->198 [2] 198/196/-1->197->195 [3] 198/196/-1->197->195 | |
| nccl-alltoall-job-oguz-mpiworker-24:51:234 [5] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-25:48:237 [2] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-11:48:231 [2] NCCL INFO Trees [0] 89/91/-1->90->92 [1] 89/91/-1->90->92 [2] -1/-1/-1->90->89 [3] -1/-1/-1->90->89 | |
| nccl-alltoall-job-oguz-mpiworker-11:48:231 [2] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-22:49:237 [3] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-27:50:232 [4] NCCL INFO Trees [0] 218/222/-1->220->216 [1] 218/222/-1->220->216 [2] -1/-1/-1->220->221 [3] -1/-1/-1->220->221 | |
| nccl-alltoall-job-oguz-mpiworker-27:50:232 [4] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-23:47:233 [1] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-26:51:236 [5] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-24:48:230 [2] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-25:47:234 [1] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-11:47:235 [1] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-21:52:231 [6] NCCL INFO Trees [0] 173/175/-1->174->172 [1] 173/175/-1->174->172 [2] -1/-1/-1->174->173 [3] -1/-1/-1->174->173 | |
| nccl-alltoall-job-oguz-mpiworker-21:52:231 [6] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-22:47:230 [1] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-27:51:234 [5] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-23:51:235 [5] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-26:52:237 [6] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-24:46:237 [0] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-25:46:230 [0] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-11:52:233 [6] NCCL INFO Trees [0] 93/95/-1->94->92 [1] 93/95/-1->94->92 [2] -1/-1/-1->94->93 [3] -1/-1/-1->94->93 | |
| nccl-alltoall-job-oguz-mpiworker-11:52:233 [6] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-27:49:233 [3] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-23:53:231 [7] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-26:53:232 [7] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-24:47:231 [1] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-11:49:232 [3] NCCL INFO Trees [0] -1/-1/-1->91->90 [1] -1/-1/-1->91->90 [2] 93/89/-1->91->87 [3] 93/89/-1->91->87 | |
| nccl-alltoall-job-oguz-mpiworker-11:49:232 [3] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-21:52:231 [6] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-27:46:231 [0] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-23:52:237 [6] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-26:50:234 [4] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-24:49:233 [3] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-11:50:237 [4] NCCL INFO Trees [0] 90/94/-1->92->88 [1] 90/94/-1->92->88 [2] -1/-1/-1->92->93 [3] -1/-1/-1->92->93 | |
| nccl-alltoall-job-oguz-mpiworker-11:50:237 [4] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-27:47:236 [1] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-23:50:234 [4] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-26:49:233 [3] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-24:53:235 [7] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-11:48:231 [2] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-27:50:232 [4] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-23:49:236 [3] NCCL INFO Trees [0] -1/-1/-1->187->186 [1] -1/-1/-1->187->186 [2] 189/185/-1->187->183 [3] 189/185/-1->187->183 | |
| nccl-alltoall-job-oguz-mpiworker-23:49:236 [3] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-26:47:230 [1] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-24:51:234 [5] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-11:51:234 [5] NCCL INFO Trees [0] -1/-1/-1->93->94 [1] -1/-1/-1->93->94 [2] 94/92/-1->93->91 [3] 94/92/-1->93->91 | |
| nccl-alltoall-job-oguz-mpiworker-11:51:234 [5] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-27:52:230 [6] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-23:46:230 [0] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-26:48:235 [2] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-24:52:232 [6] NCCL INFO Trees [0] 197/199/-1->198->196 [1] 197/199/-1->198->196 [2] -1/-1/-1->198->197 [3] -1/-1/-1->198->197 | |
| nccl-alltoall-job-oguz-mpiworker-24:52:232 [6] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-11:53:236 [7] NCCL INFO Trees [0] -1/-1/-1->95->94 [1] -1/-1/-1->95->94 [2] 111/79/-1->95->63 [3] 111/79/-1->95->63 | |
| nccl-alltoall-job-oguz-mpiworker-11:53:236 [7] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-12:46:237 [0] NCCL INFO Trees [0] 80/112/-1->96->64 [1] 80/112/-1->96->64 [2] -1/-1/-1->96->97 [3] -1/-1/-1->96->97 | |
| nccl-alltoall-job-oguz-mpiworker-12:46:237 [0] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-23:49:236 [3] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-24:50:236 [4] NCCL INFO Trees [0] 194/198/-1->196->200 [1] 194/198/-1->196->200 [2] -1/-1/-1->196->197 [3] -1/-1/-1->196->197 | |
| nccl-alltoall-job-oguz-mpiworker-24:50:236 [4] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-11:53:236 [7] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-12:47:230 [1] NCCL INFO Trees [0] -1/-1/-1->97->98 [1] -1/-1/-1->97->98 [2] 98/96/-1->97->99 [3] 98/96/-1->97->99 | |
| nccl-alltoall-job-oguz-mpiworker-12:47:230 [1] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-21:51:235 [5] NCCL INFO Trees [0] -1/-1/-1->173->174 [1] -1/-1/-1->173->174 [2] 174/172/-1->173->171 [3] 174/172/-1->173->171 | |
| nccl-alltoall-job-oguz-mpiworker-21:51:235 [5] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-24:52:232 [6] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-11:49:232 [3] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-12:48:236 [2] NCCL INFO Trees [0] 97/99/-1->98->100 [1] 97/99/-1->98->100 [2] -1/-1/-1->98->97 [3] -1/-1/-1->98->97 | |
| nccl-alltoall-job-oguz-mpiworker-12:48:236 [2] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-24:50:236 [4] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-11:50:237 [4] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-12:46:237 [0] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-21:51:235 [5] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-11:52:233 [6] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-12:49:231 [3] NCCL INFO Trees [0] -1/-1/-1->99->98 [1] -1/-1/-1->99->98 [2] 101/97/-1->99->103 [3] 101/97/-1->99->103 | |
| nccl-alltoall-job-oguz-mpiworker-12:49:231 [3] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-11:51:234 [5] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-12:47:230 [1] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-12:48:236 [2] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-21:48:232 [2] NCCL INFO Trees [0] 169/171/-1->170->172 [1] 169/171/-1->170->172 [2] -1/-1/-1->170->169 [3] -1/-1/-1->170->169 | |
| nccl-alltoall-job-oguz-mpiworker-21:48:232 [2] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-12:51:234 [5] NCCL INFO Trees [0] -1/-1/-1->101->102 [1] -1/-1/-1->101->102 [2] 102/100/-1->101->99 [3] 102/100/-1->101->99 | |
| nccl-alltoall-job-oguz-mpiworker-12:51:234 [5] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-21:50:237 [4] NCCL INFO Trees [0] 170/174/-1->172->168 [1] 170/174/-1->172->168 [2] -1/-1/-1->172->173 [3] -1/-1/-1->172->173 | |
| nccl-alltoall-job-oguz-mpiworker-21:50:237 [4] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-21:47:234 [1] NCCL INFO Trees [0] -1/-1/-1->169->170 [1] -1/-1/-1->169->170 [2] 170/168/-1->169->171 [3] 170/168/-1->169->171 | |
| nccl-alltoall-job-oguz-mpiworker-21:47:234 [1] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-21:50:237 [4] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-12:49:231 [3] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-12:54:233 [7] NCCL INFO Trees [0] -1/-1/-1->103->102 [1] -1/-1/-1->103->102 [2] 107/99/-1->103->111 [3] 107/99/-1->103->111 | |
| nccl-alltoall-job-oguz-mpiworker-12:54:233 [7] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-21:46:230 [0] NCCL INFO Trees [0] 164/172/-1->168->176 [1] 164/172/-1->168->176 [2] -1/-1/-1->168->169 [3] -1/-1/-1->168->169 | |
| nccl-alltoall-job-oguz-mpiworker-21:46:230 [0] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-13:48:230 [2] NCCL INFO Trees [0] 105/107/-1->106->108 [1] 105/107/-1->106->108 [2] -1/-1/-1->106->105 [3] -1/-1/-1->106->105 | |
| nccl-alltoall-job-oguz-mpiworker-13:48:230 [2] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-21:48:232 [2] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-21:49:236 [3] NCCL INFO Trees [0] -1/-1/-1->171->170 [1] -1/-1/-1->171->170 [2] 173/169/-1->171->167 [3] 173/169/-1->171->167 | |
| nccl-alltoall-job-oguz-mpiworker-21:49:236 [3] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-12:51:234 [5] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-21:47:234 [1] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-13:46:235 [0] NCCL INFO Trees [0] 100/108/-1->104->112 [1] 100/108/-1->104->112 [2] -1/-1/-1->104->105 [3] -1/-1/-1->104->105 | |
| nccl-alltoall-job-oguz-mpiworker-13:46:235 [0] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-13:51:237 [5] NCCL INFO Trees [0] -1/-1/-1->109->110 [1] -1/-1/-1->109->110 [2] 110/108/-1->109->107 [3] 110/108/-1->109->107 | |
| nccl-alltoall-job-oguz-mpiworker-13:51:237 [5] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-21:46:230 [0] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-12:50:232 [4] NCCL INFO Trees [0] 98/102/-1->100->104 [1] 98/102/-1->100->104 [2] -1/-1/-1->100->101 [3] -1/-1/-1->100->101 | |
| nccl-alltoall-job-oguz-mpiworker-12:50:232 [4] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-21:49:236 [3] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-12:52:235 [6] NCCL INFO Trees [0] 101/103/-1->102->100 [1] 101/103/-1->102->100 [2] -1/-1/-1->102->101 [3] -1/-1/-1->102->101 | |
| nccl-alltoall-job-oguz-mpiworker-12:52:235 [6] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-12:54:233 [7] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-12:50:232 [4] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-12:52:235 [6] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-14:46:234 [0] NCCL INFO Trees [0] 104/120/-1->112->96 [1] 104/120/-1->112->96 [2] -1/-1/-1->112->113 [3] -1/-1/-1->112->113 | |
| nccl-alltoall-job-oguz-mpiworker-14:46:234 [0] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-13:52:233 [6] NCCL INFO Trees [0] 109/111/-1->110->108 [1] 109/111/-1->110->108 [2] -1/-1/-1->110->109 [3] -1/-1/-1->110->109 | |
| nccl-alltoall-job-oguz-mpiworker-13:52:233 [6] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-13:50:236 [4] NCCL INFO Trees [0] 106/110/-1->108->104 [1] 106/110/-1->108->104 [2] -1/-1/-1->108->109 [3] -1/-1/-1->108->109 | |
| nccl-alltoall-job-oguz-mpiworker-13:50:236 [4] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-13:53:232 [7] NCCL INFO Trees [0] -1/-1/-1->111->110 [1] -1/-1/-1->111->110 [2] 119/103/-1->111->95 [3] 119/103/-1->111->95 | |
| nccl-alltoall-job-oguz-mpiworker-13:53:232 [7] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-13:47:231 [1] NCCL INFO Trees [0] -1/-1/-1->105->106 [1] -1/-1/-1->105->106 [2] 106/104/-1->105->107 [3] 106/104/-1->105->107 | |
| nccl-alltoall-job-oguz-mpiworker-13:47:231 [1] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-14:47:233 [1] NCCL INFO Trees [0] -1/-1/-1->113->114 [1] -1/-1/-1->113->114 [2] 114/112/-1->113->115 [3] 114/112/-1->113->115 | |
| nccl-alltoall-job-oguz-mpiworker-14:47:233 [1] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-13:48:230 [2] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-14:46:234 [0] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-14:48:237 [2] NCCL INFO Trees [0] 113/115/-1->114->116 [1] 113/115/-1->114->116 [2] -1/-1/-1->114->113 [3] -1/-1/-1->114->113 | |
| nccl-alltoall-job-oguz-mpiworker-14:48:237 [2] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-13:46:235 [0] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-14:47:233 [1] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-13:51:237 [5] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-13:49:234 [3] NCCL INFO Trees [0] -1/-1/-1->107->106 [1] -1/-1/-1->107->106 [2] 109/105/-1->107->103 [3] 109/105/-1->107->103 | |
| nccl-alltoall-job-oguz-mpiworker-13:49:234 [3] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-13:52:233 [6] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-13:50:236 [4] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-13:53:232 [7] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-13:47:231 [1] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-13:49:234 [3] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-14:51:230 [5] NCCL INFO Trees [0] -1/-1/-1->117->118 [1] -1/-1/-1->117->118 [2] 118/116/-1->117->115 [3] 118/116/-1->117->115 | |
| nccl-alltoall-job-oguz-mpiworker-14:51:230 [5] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-14:53:231 [7] NCCL INFO Trees [0] -1/-1/-1->119->118 [1] -1/-1/-1->119->118 [2] 123/115/-1->119->111 [3] 123/115/-1->119->111 | |
| nccl-alltoall-job-oguz-mpiworker-14:53:231 [7] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-14:52:235 [6] NCCL INFO Trees [0] 117/119/-1->118->116 [1] 117/119/-1->118->116 [2] -1/-1/-1->118->117 [3] -1/-1/-1->118->117 | |
| nccl-alltoall-job-oguz-mpiworker-14:52:235 [6] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-14:50:236 [4] NCCL INFO Trees [0] 114/118/-1->116->120 [1] 114/118/-1->116->120 [2] -1/-1/-1->116->117 [3] -1/-1/-1->116->117 | |
| nccl-alltoall-job-oguz-mpiworker-14:50:236 [4] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-14:48:237 [2] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-14:49:232 [3] NCCL INFO Trees [0] -1/-1/-1->115->114 [1] -1/-1/-1->115->114 [2] 117/113/-1->115->119 [3] 117/113/-1->115->119 | |
| nccl-alltoall-job-oguz-mpiworker-14:49:232 [3] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-14:53:231 [7] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-14:51:230 [5] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-14:50:236 [4] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-14:52:235 [6] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-14:49:232 [3] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-15:46:230 [0] NCCL INFO Trees [0] 116/124/-1->120->112 [1] 116/124/-1->120->112 [2] -1/-1/-1->120->121 [3] -1/-1/-1->120->121 | |
| nccl-alltoall-job-oguz-mpiworker-15:46:230 [0] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-15:47:232 [1] NCCL INFO Trees [0] -1/-1/-1->121->122 [1] -1/-1/-1->121->122 [2] 122/120/-1->121->123 [3] 122/120/-1->121->123 | |
| nccl-alltoall-job-oguz-mpiworker-15:47:232 [1] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-15:49:235 [3] NCCL INFO Trees [0] -1/-1/-1->123->122 [1] -1/-1/-1->123->122 [2] 125/121/-1->123->119 [3] 125/121/-1->123->119 | |
| nccl-alltoall-job-oguz-mpiworker-15:49:235 [3] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-15:48:231 [2] NCCL INFO Trees [0] 121/123/-1->122->124 [1] 121/123/-1->122->124 [2] -1/-1/-1->122->121 [3] -1/-1/-1->122->121 | |
| nccl-alltoall-job-oguz-mpiworker-15:48:231 [2] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-15:51:233 [5] NCCL INFO Trees [0] -1/-1/-1->125->126 [1] -1/-1/-1->125->126 [2] 126/124/-1->125->123 [3] 126/124/-1->125->123 | |
| nccl-alltoall-job-oguz-mpiworker-15:51:233 [5] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-15:53:236 [7] NCCL INFO Trees [0] -1/-1/-1->127->126 [1] -1/-1/-1->127->126 [2] 191/63/-1->127->255 [3] 191/63/-1->127->255 | |
| nccl-alltoall-job-oguz-mpiworker-15:53:236 [7] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-15:46:230 [0] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-15:49:235 [3] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-15:48:231 [2] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-15:47:232 [1] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-16:46:230 [0] NCCL INFO Trees [0] 64/192/-1->128->0 [1] 64/192/-1->128->0 [2] -1/-1/-1->128->129 [3] -1/-1/-1->128->129 | |
| nccl-alltoall-job-oguz-mpiworker-16:46:230 [0] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-15:51:233 [5] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-15:53:236 [7] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-15:52:237 [6] NCCL INFO Trees [0] 125/127/-1->126->124 [1] 125/127/-1->126->124 [2] -1/-1/-1->126->125 [3] -1/-1/-1->126->125 | |
| nccl-alltoall-job-oguz-mpiworker-15:52:237 [6] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-15:50:234 [4] NCCL INFO Trees [0] 122/126/-1->124->120 [1] 122/126/-1->124->120 [2] -1/-1/-1->124->125 [3] -1/-1/-1->124->125 | |
| nccl-alltoall-job-oguz-mpiworker-15:50:234 [4] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-16:53:236 [7] NCCL INFO Trees [0] -1/-1/-1->135->134 [1] -1/-1/-1->135->134 [2] 139/131/-1->135->143 [3] 139/131/-1->135->143 | |
| nccl-alltoall-job-oguz-mpiworker-16:53:236 [7] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-15:50:234 [4] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-15:52:237 [6] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-16:48:231 [2] NCCL INFO Trees [0] 129/131/-1->130->132 [1] 129/131/-1->130->132 [2] -1/-1/-1->130->129 [3] -1/-1/-1->130->129 | |
| nccl-alltoall-job-oguz-mpiworker-16:48:231 [2] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-16:52:237 [6] NCCL INFO Trees [0] 133/135/-1->134->132 [1] 133/135/-1->134->132 [2] -1/-1/-1->134->133 [3] -1/-1/-1->134->133 | |
| nccl-alltoall-job-oguz-mpiworker-16:52:237 [6] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-16:51:235 [5] NCCL INFO Trees [0] -1/-1/-1->133->134 [1] -1/-1/-1->133->134 [2] 134/132/-1->133->131 [3] 134/132/-1->133->131 | |
| nccl-alltoall-job-oguz-mpiworker-16:51:235 [5] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-16:50:232 [4] NCCL INFO Trees [0] 130/134/-1->132->136 [1] 130/134/-1->132->136 [2] -1/-1/-1->132->133 [3] -1/-1/-1->132->133 | |
| nccl-alltoall-job-oguz-mpiworker-16:50:232 [4] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-17:46:234 [0] NCCL INFO Trees [0] 132/140/-1->136->144 [1] 132/140/-1->136->144 [2] -1/-1/-1->136->137 [3] -1/-1/-1->136->137 | |
| nccl-alltoall-job-oguz-mpiworker-17:46:234 [0] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-16:46:230 [0] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-16:48:231 [2] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-17:47:231 [1] NCCL INFO Trees [0] -1/-1/-1->137->138 [1] -1/-1/-1->137->138 [2] 138/136/-1->137->139 [3] 138/136/-1->137->139 | |
| nccl-alltoall-job-oguz-mpiworker-17:47:231 [1] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-17:48:230 [2] NCCL INFO Trees [0] 137/139/-1->138->140 [1] 137/139/-1->138->140 [2] -1/-1/-1->138->137 [3] -1/-1/-1->138->137 | |
| nccl-alltoall-job-oguz-mpiworker-17:48:230 [2] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-17:49:233 [3] NCCL INFO Trees [0] -1/-1/-1->139->138 [1] -1/-1/-1->139->138 [2] 141/137/-1->139->135 [3] 141/137/-1->139->135 | |
| nccl-alltoall-job-oguz-mpiworker-17:49:233 [3] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-17:46:234 [0] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-16:47:234 [1] NCCL INFO Trees [0] -1/-1/-1->129->130 [1] -1/-1/-1->129->130 [2] 130/128/-1->129->131 [3] 130/128/-1->129->131 | |
| nccl-alltoall-job-oguz-mpiworker-16:47:234 [1] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-16:53:236 [7] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-16:52:237 [6] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-16:51:235 [5] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-16:49:233 [3] NCCL INFO Trees [0] -1/-1/-1->131->130 [1] -1/-1/-1->131->130 [2] 133/129/-1->131->135 [3] 133/129/-1->131->135 | |
| nccl-alltoall-job-oguz-mpiworker-16:49:233 [3] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-16:50:232 [4] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-17:48:230 [2] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-16:47:234 [1] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-17:47:231 [1] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-16:49:233 [3] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-17:51:232 [5] NCCL INFO Trees [0] -1/-1/-1->141->142 [1] -1/-1/-1->141->142 [2] 142/140/-1->141->139 [3] 142/140/-1->141->139 | |
| nccl-alltoall-job-oguz-mpiworker-17:51:232 [5] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-17:49:233 [3] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-20:53:231 [7] NCCL INFO Trees [0] -1/-1/-1->167->166 [1] -1/-1/-1->167->166 [2] 171/163/-1->167->175 [3] 171/163/-1->167->175 | |
| nccl-alltoall-job-oguz-mpiworker-20:53:231 [7] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-17:53:236 [7] NCCL INFO Trees [0] -1/-1/-1->143->142 [1] -1/-1/-1->143->142 [2] 151/135/-1->143->159 [3] 151/135/-1->143->159 | |
| nccl-alltoall-job-oguz-mpiworker-17:53:236 [7] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-17:51:232 [5] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-20:51:230 [5] NCCL INFO Trees [0] -1/-1/-1->165->166 [1] -1/-1/-1->165->166 [2] 166/164/-1->165->163 [3] 166/164/-1->165->163 | |
| nccl-alltoall-job-oguz-mpiworker-20:51:230 [5] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-19:51:235 [5] NCCL INFO Trees [0] -1/-1/-1->157->158 [1] -1/-1/-1->157->158 [2] 158/156/-1->157->155 [3] 158/156/-1->157->155 | |
| nccl-alltoall-job-oguz-mpiworker-19:51:235 [5] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-20:49:235 [3] NCCL INFO Trees [0] -1/-1/-1->163->162 [1] -1/-1/-1->163->162 [2] 165/161/-1->163->167 [3] 165/161/-1->163->167 | |
| nccl-alltoall-job-oguz-mpiworker-20:49:235 [3] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-17:53:236 [7] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-17:50:235 [4] NCCL INFO Trees [0] 138/142/-1->140->136 [1] 138/142/-1->140->136 [2] -1/-1/-1->140->141 [3] -1/-1/-1->140->141 | |
| nccl-alltoall-job-oguz-mpiworker-17:50:235 [4] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-17:52:237 [6] NCCL INFO Trees [0] 141/143/-1->142->140 [1] 141/143/-1->142->140 [2] -1/-1/-1->142->141 [3] -1/-1/-1->142->141 | |
| nccl-alltoall-job-oguz-mpiworker-17:52:237 [6] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-18:46:230 [0] NCCL INFO Trees [0] 136/152/-1->144->160 [1] 136/152/-1->144->160 [2] -1/-1/-1->144->145 [3] -1/-1/-1->144->145 | |
| nccl-alltoall-job-oguz-mpiworker-18:46:230 [0] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-20:52:232 [6] NCCL INFO Trees [0] 165/167/-1->166->164 [1] 165/167/-1->166->164 [2] -1/-1/-1->166->165 [3] -1/-1/-1->166->165 | |
| nccl-alltoall-job-oguz-mpiworker-20:52:232 [6] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-20:53:231 [7] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-20:51:230 [5] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-20:46:233 [0] NCCL INFO Trees [0] 144/176/-1->160->192 [1] 144/176/-1->160->192 [2] -1/-1/-1->160->161 [3] -1/-1/-1->160->161 | |
| nccl-alltoall-job-oguz-mpiworker-20:46:233 [0] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-19:52:234 [6] NCCL INFO Trees [0] 157/159/-1->158->156 [1] 157/159/-1->158->156 [2] -1/-1/-1->158->157 [3] -1/-1/-1->158->157 | |
| nccl-alltoall-job-oguz-mpiworker-19:52:234 [6] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-17:50:235 [4] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-20:50:237 [4] NCCL INFO Trees [0] 162/166/-1->164->168 [1] 162/166/-1->164->168 [2] -1/-1/-1->164->165 [3] -1/-1/-1->164->165 | |
| nccl-alltoall-job-oguz-mpiworker-20:50:237 [4] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-19:53:231 [7] NCCL INFO Trees [0] -1/-1/-1->159->158 [1] -1/-1/-1->159->158 [2] 175/143/-1->159->191 [3] 175/143/-1->159->191 | |
| nccl-alltoall-job-oguz-mpiworker-19:53:231 [7] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-17:52:237 [6] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-20:48:236 [2] NCCL INFO Trees [0] 161/163/-1->162->164 [1] 161/163/-1->162->164 [2] -1/-1/-1->162->161 [3] -1/-1/-1->162->161 | |
| nccl-alltoall-job-oguz-mpiworker-20:48:236 [2] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-19:53:231 [7] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-18:48:233 [2] NCCL INFO Trees [0] 145/147/-1->146->148 [1] 145/147/-1->146->148 [2] -1/-1/-1->146->145 [3] -1/-1/-1->146->145 | |
| nccl-alltoall-job-oguz-mpiworker-18:48:233 [2] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-20:49:235 [3] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-18:51:235 [5] NCCL INFO Trees [0] -1/-1/-1->149->150 [1] -1/-1/-1->149->150 [2] 150/148/-1->149->147 [3] 150/148/-1->149->147 | |
| nccl-alltoall-job-oguz-mpiworker-18:51:235 [5] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-20:47:234 [1] NCCL INFO Trees [0] -1/-1/-1->161->162 [1] -1/-1/-1->161->162 [2] 162/160/-1->161->163 [3] 162/160/-1->161->163 | |
| nccl-alltoall-job-oguz-mpiworker-20:47:234 [1] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-18:53:231 [7] NCCL INFO Trees [0] -1/-1/-1->151->150 [1] -1/-1/-1->151->150 [2] 155/147/-1->151->143 [3] 155/147/-1->151->143 | |
| nccl-alltoall-job-oguz-mpiworker-18:53:231 [7] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-20:52:232 [6] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-18:46:230 [0] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-20:46:233 [0] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-18:47:232 [1] NCCL INFO Trees [0] -1/-1/-1->145->146 [1] -1/-1/-1->145->146 [2] 146/144/-1->145->147 [3] 146/144/-1->145->147 | |
| nccl-alltoall-job-oguz-mpiworker-18:47:232 [1] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-20:48:236 [2] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-18:53:231 [7] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-20:50:237 [4] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-20:47:234 [1] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-18:49:234 [3] NCCL INFO Trees [0] -1/-1/-1->147->146 [1] -1/-1/-1->147->146 [2] 149/145/-1->147->151 [3] 149/145/-1->147->151 | |
| nccl-alltoall-job-oguz-mpiworker-18:49:234 [3] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-19:51:235 [5] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-19:52:234 [6] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-19:50:237 [4] NCCL INFO Trees [0] 154/158/-1->156->152 [1] 154/158/-1->156->152 [2] -1/-1/-1->156->157 [3] -1/-1/-1->156->157 | |
| nccl-alltoall-job-oguz-mpiworker-19:50:237 [4] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-18:52:237 [6] NCCL INFO Trees [0] 149/151/-1->150->148 [1] 149/151/-1->150->148 [2] -1/-1/-1->150->149 [3] -1/-1/-1->150->149 | |
| nccl-alltoall-job-oguz-mpiworker-18:52:237 [6] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-19:49:236 [3] NCCL INFO Trees [0] -1/-1/-1->155->154 [1] -1/-1/-1->155->154 [2] 157/153/-1->155->151 [3] 157/153/-1->155->151 | |
| nccl-alltoall-job-oguz-mpiworker-19:49:236 [3] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-18:51:235 [5] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-19:46:230 [0] NCCL INFO Trees [0] 148/156/-1->152->144 [1] 148/156/-1->152->144 [2] -1/-1/-1->152->153 [3] -1/-1/-1->152->153 | |
| nccl-alltoall-job-oguz-mpiworker-19:46:230 [0] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-18:48:233 [2] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-19:47:232 [1] NCCL INFO Trees [0] -1/-1/-1->153->154 [1] -1/-1/-1->153->154 [2] 154/152/-1->153->155 [3] 154/152/-1->153->155 | |
| nccl-alltoall-job-oguz-mpiworker-19:47:232 [1] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-18:50:236 [4] NCCL INFO Trees [0] 146/150/-1->148->152 [1] 146/150/-1->148->152 [2] -1/-1/-1->148->149 [3] -1/-1/-1->148->149 | |
| nccl-alltoall-job-oguz-mpiworker-18:50:236 [4] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-19:48:233 [2] NCCL INFO Trees [0] 153/155/-1->154->156 [1] 153/155/-1->154->156 [2] -1/-1/-1->154->153 [3] -1/-1/-1->154->153 | |
| nccl-alltoall-job-oguz-mpiworker-19:48:233 [2] NCCL INFO P2P Chunksize set to 131072 | |
| nccl-alltoall-job-oguz-mpiworker-18:47:232 [1] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-19:50:237 [4] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-18:49:234 [3] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-19:47:232 [1] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-18:52:237 [6] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-19:46:230 [0] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-18:50:236 [4] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-19:49:236 [3] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-19:48:233 [2] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. | |
| nccl-alltoall-job-oguz-mpiworker-0:53:234 [7] NCCL INFO Channel 00/0 : 6[6] -> 7[7] [receive] via NET/IB/7/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-0:51:232 [5] NCCL INFO Channel 00/0 : 4[4] -> 5[5] [receive] via NET/IB/3/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-0:53:234 [7] NCCL INFO Channel 01/0 : 6[6] -> 7[7] [receive] via NET/IB/6/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-0:51:232 [5] NCCL INFO Channel 01/0 : 4[4] -> 5[5] [receive] via NET/IB/2/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-0:53:234 [7] NCCL INFO Channel 02/0 : 6[6] -> 7[7] [receive] via NET/IB/7/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-0:51:232 [5] NCCL INFO Channel 02/0 : 4[4] -> 5[5] [receive] via NET/IB/3/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-0:53:234 [7] NCCL INFO Channel 03/0 : 6[6] -> 7[7] [receive] via NET/IB/6/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-0:51:232 [5] NCCL INFO Channel 03/0 : 4[4] -> 5[5] [receive] via NET/IB/2/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-0:53:234 [7] NCCL INFO Channel 00/0 : 7[7] -> 8[0] [send] via NET/IB/7/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-0:48:233 [2] NCCL INFO Channel 00/0 : 1[1] -> 2[2] [receive] via NET/IB/10/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-0:46:231 [0] NCCL INFO Channel 00/0 : 255[7] -> 0[0] [receive] via NET/IB/0/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-0:51:232 [5] NCCL INFO Channel 00/0 : 5[5] -> 6[6] [send] via NET/IB/3/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-0:53:234 [7] NCCL INFO Channel 01/0 : 7[7] -> 8[0] [send] via NET/IB/6/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-0:51:232 [5] NCCL INFO Channel 01/0 : 5[5] -> 6[6] [send] via NET/IB/2/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-0:48:233 [2] NCCL INFO Channel 01/0 : 1[1] -> 2[2] [receive] via NET/IB/11/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-0:46:231 [0] NCCL INFO Channel 01/0 : 255[7] -> 0[0] [receive] via NET/IB/1/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-0:53:234 [7] NCCL INFO Channel 02/0 : 7[7] -> 8[0] [send] via NET/IB/7/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-0:51:232 [5] NCCL INFO Channel 02/0 : 5[5] -> 6[6] [send] via NET/IB/3/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-0:48:233 [2] NCCL INFO Channel 02/0 : 1[1] -> 2[2] [receive] via NET/IB/10/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-0:46:231 [0] NCCL INFO Channel 02/0 : 255[7] -> 0[0] [receive] via NET/IB/0/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-0:53:234 [7] NCCL INFO Channel 03/0 : 7[7] -> 8[0] [send] via NET/IB/6/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-0:51:232 [5] NCCL INFO Channel 03/0 : 5[5] -> 6[6] [send] via NET/IB/2/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-0:48:233 [2] NCCL INFO Channel 03/0 : 1[1] -> 2[2] [receive] via NET/IB/11/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-0:50:238 [4] NCCL INFO Channel 00/0 : 3[3] -> 4[4] [receive] via NET/IB/14/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-0:48:233 [2] NCCL INFO Channel 00/0 : 2[2] -> 3[3] [send] via NET/IB/10/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-31:53:231 [7] NCCL INFO Channel 00/0 : 254[6] -> 255[7] [receive] via NET/IB/7/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-0:46:231 [0] NCCL INFO Channel 03/0 : 255[7] -> 0[0] [receive] via NET/IB/1/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-31:51:236 [5] NCCL INFO Channel 00/0 : 252[4] -> 253[5] [receive] via NET/IB/3/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-31:53:231 [7] NCCL INFO Channel 01/0 : 254[6] -> 255[7] [receive] via NET/IB/6/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-0:48:233 [2] NCCL INFO Channel 01/0 : 2[2] -> 3[3] [send] via NET/IB/11/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-31:51:236 [5] NCCL INFO Channel 01/0 : 252[4] -> 253[5] [receive] via NET/IB/2/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-1:48:231 [2] NCCL INFO Channel 00/0 : 9[1] -> 10[2] [receive] via NET/IB/10/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-0:50:238 [4] NCCL INFO Channel 01/0 : 3[3] -> 4[4] [receive] via NET/IB/15/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-0:46:231 [0] NCCL INFO Channel 00/0 : 0[0] -> 1[1] [send] via NET/IB/0/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-0:48:233 [2] NCCL INFO Channel 02/0 : 2[2] -> 3[3] [send] via NET/IB/10/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-31:53:231 [7] NCCL INFO Channel 02/0 : 254[6] -> 255[7] [receive] via NET/IB/7/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-0:50:238 [4] NCCL INFO Channel 02/0 : 3[3] -> 4[4] [receive] via NET/IB/14/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-31:51:236 [5] NCCL INFO Channel 02/0 : 252[4] -> 253[5] [receive] via NET/IB/3/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-1:48:231 [2] NCCL INFO Channel 01/0 : 9[1] -> 10[2] [receive] via NET/IB/11/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-0:46:231 [0] NCCL INFO Channel 01/0 : 0[0] -> 1[1] [send] via NET/IB/1/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-0:48:233 [2] NCCL INFO Channel 03/0 : 2[2] -> 3[3] [send] via NET/IB/11/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-0:46:231 [0] NCCL INFO Channel 02/0 : 0[0] -> 1[1] [send] via NET/IB/0/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-31:53:231 [7] NCCL INFO Channel 03/0 : 254[6] -> 255[7] [receive] via NET/IB/6/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-0:50:238 [4] NCCL INFO Channel 03/0 : 3[3] -> 4[4] [receive] via NET/IB/15/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-31:51:236 [5] NCCL INFO Channel 03/0 : 252[4] -> 253[5] [receive] via NET/IB/2/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-0:46:231 [0] NCCL INFO Channel 03/0 : 0[0] -> 1[1] [send] via NET/IB/1/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-31:48:230 [2] NCCL INFO Channel 00/0 : 249[1] -> 250[2] [receive] via NET/IB/10/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-1:48:231 [2] NCCL INFO Channel 02/0 : 9[1] -> 10[2] [receive] via NET/IB/10/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-0:50:238 [4] NCCL INFO Channel 00/0 : 4[4] -> 5[5] [send] via NET/IB/14/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-31:53:231 [7] NCCL INFO Channel 00/0 : 255[7] -> 0[0] [send] via NET/IB/7/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-31:51:236 [5] NCCL INFO Channel 00/0 : 253[5] -> 254[6] [send] via NET/IB/3/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-2:46:232 [0] NCCL INFO Channel 00/0 : 15[7] -> 16[0] [receive] via NET/IB/0/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-31:46:234 [0] NCCL INFO Channel 00/0 : 247[7] -> 248[0] [receive] via NET/IB/0/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-0:50:238 [4] NCCL INFO Channel 01/0 : 4[4] -> 5[5] [send] via NET/IB/15/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-1:46:230 [0] NCCL INFO Channel 00/0 : 7[7] -> 8[0] [receive] via NET/IB/0/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-31:48:230 [2] NCCL INFO Channel 01/0 : 249[1] -> 250[2] [receive] via NET/IB/11/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-0:47:237 [1] NCCL INFO Channel 00/0 : 0[0] -> 1[1] [receive] via NET/IB/9/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-1:51:232 [5] NCCL INFO Channel 00/0 : 12[4] -> 13[5] [receive] via NET/IB/3/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-31:53:231 [7] NCCL INFO Channel 01/0 : 255[7] -> 0[0] [send] via NET/IB/6/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-0:50:238 [4] NCCL INFO Channel 02/0 : 4[4] -> 5[5] [send] via NET/IB/14/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-1:48:231 [2] NCCL INFO Channel 03/0 : 9[1] -> 10[2] [receive] via NET/IB/11/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-2:46:232 [0] NCCL INFO Channel 01/0 : 15[7] -> 16[0] [receive] via NET/IB/1/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-31:51:236 [5] NCCL INFO Channel 01/0 : 253[5] -> 254[6] [send] via NET/IB/2/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-0:47:237 [1] NCCL INFO Channel 01/0 : 0[0] -> 1[1] [receive] via NET/IB/8/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-1:48:231 [2] NCCL INFO Channel 00/0 : 10[2] -> 11[3] [send] via NET/IB/10/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-31:53:231 [7] NCCL INFO Channel 02/0 : 255[7] -> 0[0] [send] via NET/IB/7/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-0:50:238 [4] NCCL INFO Channel 03/0 : 4[4] -> 5[5] [send] via NET/IB/15/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-1:46:230 [0] NCCL INFO Channel 01/0 : 7[7] -> 8[0] [receive] via NET/IB/1/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-2:46:232 [0] NCCL INFO Channel 02/0 : 15[7] -> 16[0] [receive] via NET/IB/0/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-31:46:234 [0] NCCL INFO Channel 01/0 : 247[7] -> 248[0] [receive] via NET/IB/1/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-0:47:237 [1] NCCL INFO Channel 02/0 : 0[0] -> 1[1] [receive] via NET/IB/9/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-1:51:232 [5] NCCL INFO Channel 01/0 : 12[4] -> 13[5] [receive] via NET/IB/2/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-31:48:230 [2] NCCL INFO Channel 02/0 : 249[1] -> 250[2] [receive] via NET/IB/10/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-1:48:231 [2] NCCL INFO Channel 01/0 : 10[2] -> 11[3] [send] via NET/IB/11/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-31:51:236 [5] NCCL INFO Channel 02/0 : 253[5] -> 254[6] [send] via NET/IB/3/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-1:46:230 [0] NCCL INFO Channel 02/0 : 7[7] -> 8[0] [receive] via NET/IB/0/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-2:46:232 [0] NCCL INFO Channel 03/0 : 15[7] -> 16[0] [receive] via NET/IB/1/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-3:46:233 [0] NCCL INFO Channel 00/0 : 23[7] -> 24[0] [receive] via NET/IB/0/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-0:52:235 [6] NCCL INFO Channel 00/0 : 5[5] -> 6[6] [receive] via NET/IB/4/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-31:53:231 [7] NCCL INFO Channel 03/0 : 255[7] -> 0[0] [send] via NET/IB/6/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-1:51:232 [5] NCCL INFO Channel 02/0 : 12[4] -> 13[5] [receive] via NET/IB/3/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-2:48:237 [2] NCCL INFO Channel 00/0 : 17[1] -> 18[2] [receive] via NET/IB/10/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-3:48:231 [2] NCCL INFO Channel 00/0 : 25[1] -> 26[2] [receive] via NET/IB/10/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-0:47:237 [1] NCCL INFO Channel 03/0 : 0[0] -> 1[1] [receive] via NET/IB/8/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-29:53:235 [7] NCCL INFO Channel 00/0 : 238[6] -> 239[7] [receive] via NET/IB/7/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-31:46:234 [0] NCCL INFO Channel 02/0 : 247[7] -> 248[0] [receive] via NET/IB/0/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-1:48:231 [2] NCCL INFO Channel 02/0 : 10[2] -> 11[3] [send] via NET/IB/10/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-2:46:232 [0] NCCL INFO Channel 00/0 : 16[0] -> 17[1] [send] via NET/IB/0/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-3:46:233 [0] NCCL INFO Channel 01/0 : 23[7] -> 24[0] [receive] via NET/IB/1/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-0:49:236 [3] NCCL INFO Channel 00/0 : 2[2] -> 3[3] [receive] via NET/IB/13/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-29:51:230 [5] NCCL INFO Channel 00/0 : 236[4] -> 237[5] [receive] via NET/IB/3/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-31:48:230 [2] NCCL INFO Channel 03/0 : 249[1] -> 250[2] [receive] via NET/IB/11/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-1:47:237 [1] NCCL INFO Channel 00/0 : 8[0] -> 9[1] [receive] via NET/IB/9/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-2:48:237 [2] NCCL INFO Channel 01/0 : 17[1] -> 18[2] [receive] via NET/IB/11/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-3:48:231 [2] NCCL INFO Channel 01/0 : 25[1] -> 26[2] [receive] via NET/IB/11/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-0:47:237 [1] NCCL INFO Channel 00/0 : 1[1] -> 2[2] [send] via NET/IB/9/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-29:53:235 [7] NCCL INFO Channel 01/0 : 238[6] -> 239[7] [receive] via NET/IB/6/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-31:51:236 [5] NCCL INFO Channel 03/0 : 253[5] -> 254[6] [send] via NET/IB/2/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-1:46:230 [0] NCCL INFO Channel 03/0 : 7[7] -> 8[0] [receive] via NET/IB/1/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-2:46:232 [0] NCCL INFO Channel 01/0 : 16[0] -> 17[1] [send] via NET/IB/1/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-3:46:233 [0] NCCL INFO Channel 02/0 : 23[7] -> 24[0] [receive] via NET/IB/0/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-0:52:235 [6] NCCL INFO Channel 01/0 : 5[5] -> 6[6] [receive] via NET/IB/5/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-29:51:230 [5] NCCL INFO Channel 01/0 : 236[4] -> 237[5] [receive] via NET/IB/2/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-31:52:233 [6] NCCL INFO Channel 00/0 : 253[5] -> 254[6] [receive] via NET/IB/4/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-1:48:231 [2] NCCL INFO Channel 03/0 : 10[2] -> 11[3] [send] via NET/IB/11/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-2:48:237 [2] NCCL INFO Channel 02/0 : 17[1] -> 18[2] [receive] via NET/IB/10/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-3:48:231 [2] NCCL INFO Channel 02/0 : 25[1] -> 26[2] [receive] via NET/IB/10/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-0:47:237 [1] NCCL INFO Channel 01/0 : 1[1] -> 2[2] [send] via NET/IB/8/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-29:53:235 [7] NCCL INFO Channel 02/0 : 238[6] -> 239[7] [receive] via NET/IB/7/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-31:46:234 [0] NCCL INFO Channel 03/0 : 247[7] -> 248[0] [receive] via NET/IB/1/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-1:51:232 [5] NCCL INFO Channel 03/0 : 12[4] -> 13[5] [receive] via NET/IB/2/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-2:51:234 [5] NCCL INFO Channel 00/0 : 20[4] -> 21[5] [receive] via NET/IB/3/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-3:46:233 [0] NCCL INFO Channel 03/0 : 23[7] -> 24[0] [receive] via NET/IB/1/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-0:49:236 [3] NCCL INFO Channel 01/0 : 2[2] -> 3[3] [receive] via NET/IB/12/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-29:51:230 [5] NCCL INFO Channel 02/0 : 236[4] -> 237[5] [receive] via NET/IB/3/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-31:48:230 [2] NCCL INFO Channel 00/0 : 250[2] -> 251[3] [send] via NET/IB/10/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-1:47:237 [1] NCCL INFO Channel 01/0 : 8[0] -> 9[1] [receive] via NET/IB/8/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-2:46:232 [0] NCCL INFO Channel 02/0 : 16[0] -> 17[1] [send] via NET/IB/0/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-3:48:231 [2] NCCL INFO Channel 03/0 : 25[1] -> 26[2] [receive] via NET/IB/11/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-0:52:235 [6] NCCL INFO Channel 02/0 : 5[5] -> 6[6] [receive] via NET/IB/4/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-29:53:235 [7] NCCL INFO Channel 03/0 : 238[6] -> 239[7] [receive] via NET/IB/6/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-31:48:230 [2] NCCL INFO Channel 01/0 : 250[2] -> 251[3] [send] via NET/IB/11/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-1:46:230 [0] NCCL INFO Channel 00/0 : 8[0] -> 9[1] [send] via NET/IB/0/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-2:48:237 [2] NCCL INFO Channel 03/0 : 17[1] -> 18[2] [receive] via NET/IB/11/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-3:46:233 [0] NCCL INFO Channel 00/0 : 24[0] -> 25[1] [send] via NET/IB/0/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-0:47:237 [1] NCCL INFO Channel 02/0 : 1[1] -> 2[2] [send] via NET/IB/9/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-29:51:230 [5] NCCL INFO Channel 03/0 : 236[4] -> 237[5] [receive] via NET/IB/2/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-31:46:234 [0] NCCL INFO Channel 00/0 : 248[0] -> 249[1] [send] via NET/IB/0/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-1:51:232 [5] NCCL INFO Channel 00/0 : 13[5] -> 14[6] [send] via NET/IB/3/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-2:46:232 [0] NCCL INFO Channel 03/0 : 16[0] -> 17[1] [send] via NET/IB/1/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-3:48:231 [2] NCCL INFO Channel 00/0 : 26[2] -> 27[3] [send] via NET/IB/10/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-0:49:236 [3] NCCL INFO Channel 02/0 : 2[2] -> 3[3] [receive] via NET/IB/13/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-29:53:235 [7] NCCL INFO Channel 00/0 : 239[7] -> 240[0] [send] via NET/IB/7/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-30:51:233 [5] NCCL INFO Channel 00/0 : 244[4] -> 245[5] [receive] via NET/IB/3/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-31:52:233 [6] NCCL INFO Channel 01/0 : 253[5] -> 254[6] [receive] via NET/IB/5/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-2:51:234 [5] NCCL INFO Channel 01/0 : 20[4] -> 21[5] [receive] via NET/IB/2/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-3:46:233 [0] NCCL INFO Channel 01/0 : 24[0] -> 25[1] [send] via NET/IB/1/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-0:52:235 [6] NCCL INFO Channel 03/0 : 5[5] -> 6[6] [receive] via NET/IB/5/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-29:51:230 [5] NCCL INFO Channel 00/0 : 237[5] -> 238[6] [send] via NET/IB/3/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-30:53:231 [7] NCCL INFO Channel 00/0 : 246[6] -> 247[7] [receive] via NET/IB/7/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-31:50:237 [4] NCCL INFO Channel 00/0 : 251[3] -> 252[4] [receive] via NET/IB/14/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-2:48:237 [2] NCCL INFO Channel 00/0 : 18[2] -> 19[3] [send] via NET/IB/10/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-3:48:231 [2] NCCL INFO Channel 01/0 : 26[2] -> 27[3] [send] via NET/IB/11/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-0:47:237 [1] NCCL INFO Channel 03/0 : 1[1] -> 2[2] [send] via NET/IB/8/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-29:53:235 [7] NCCL INFO Channel 01/0 : 239[7] -> 240[0] [send] via NET/IB/6/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-30:51:233 [5] NCCL INFO Channel 01/0 : 244[4] -> 245[5] [receive] via NET/IB/2/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-31:48:230 [2] NCCL INFO Channel 02/0 : 250[2] -> 251[3] [send] via NET/IB/10/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-2:48:237 [2] NCCL INFO Channel 01/0 : 18[2] -> 19[3] [send] via NET/IB/11/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-3:46:233 [0] NCCL INFO Channel 02/0 : 24[0] -> 25[1] [send] via NET/IB/0/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-0:49:236 [3] NCCL INFO Channel 03/0 : 2[2] -> 3[3] [receive] via NET/IB/12/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-29:51:230 [5] NCCL INFO Channel 01/0 : 237[5] -> 238[6] [send] via NET/IB/2/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-30:53:231 [7] NCCL INFO Channel 01/0 : 246[6] -> 247[7] [receive] via NET/IB/6/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-1:47:237 [1] NCCL INFO Channel 02/0 : 8[0] -> 9[1] [receive] via NET/IB/9/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-31:46:234 [0] NCCL INFO Channel 01/0 : 248[0] -> 249[1] [send] via NET/IB/1/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-2:51:234 [5] NCCL INFO Channel 02/0 : 20[4] -> 21[5] [receive] via NET/IB/3/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-3:48:231 [2] NCCL INFO Channel 02/0 : 26[2] -> 27[3] [send] via NET/IB/10/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-0:52:235 [6] NCCL INFO Channel 00/0 : 6[6] -> 7[7] [send] via NET/IB/4/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-29:50:234 [4] NCCL INFO Channel 00/0 : 235[3] -> 236[4] [receive] via NET/IB/14/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-30:51:233 [5] NCCL INFO Channel 02/0 : 244[4] -> 245[5] [receive] via NET/IB/3/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-1:51:232 [5] NCCL INFO Channel 01/0 : 13[5] -> 14[6] [send] via NET/IB/2/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-31:52:233 [6] NCCL INFO Channel 02/0 : 253[5] -> 254[6] [receive] via NET/IB/4/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-2:53:230 [7] NCCL INFO Channel 00/0 : 22[6] -> 23[7] [receive] via NET/IB/7/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-3:46:233 [0] NCCL INFO Channel 03/0 : 24[0] -> 25[1] [send] via NET/IB/1/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-0:52:235 [6] NCCL INFO Channel 01/0 : 6[6] -> 7[7] [send] via NET/IB/5/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-29:53:235 [7] NCCL INFO Channel 02/0 : 239[7] -> 240[0] [send] via NET/IB/7/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-30:53:231 [7] NCCL INFO Channel 02/0 : 246[6] -> 247[7] [receive] via NET/IB/7/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-1:46:230 [0] NCCL INFO Channel 01/0 : 8[0] -> 9[1] [send] via NET/IB/1/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-31:46:234 [0] NCCL INFO Channel 02/0 : 248[0] -> 249[1] [send] via NET/IB/0/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-2:48:237 [2] NCCL INFO Channel 02/0 : 18[2] -> 19[3] [send] via NET/IB/10/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-3:48:231 [2] NCCL INFO Channel 03/0 : 26[2] -> 27[3] [send] via NET/IB/11/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-0:49:236 [3] NCCL INFO Channel 00/0 : 3[3] -> 4[4] [send] via NET/IB/13/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-29:51:230 [5] NCCL INFO Channel 02/0 : 237[5] -> 238[6] [send] via NET/IB/3/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-30:51:233 [5] NCCL INFO Channel 03/0 : 244[4] -> 245[5] [receive] via NET/IB/2/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-1:49:236 [3] NCCL INFO Channel 00/0 : 10[2] -> 11[3] [receive] via NET/IB/13/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-31:48:230 [2] NCCL INFO Channel 03/0 : 250[2] -> 251[3] [send] via NET/IB/11/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-2:51:234 [5] NCCL INFO Channel 03/0 : 20[4] -> 21[5] [receive] via NET/IB/2/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-0:52:235 [6] NCCL INFO Channel 02/0 : 6[6] -> 7[7] [send] via NET/IB/4/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-29:52:237 [6] NCCL INFO Channel 00/0 : 237[5] -> 238[6] [receive] via NET/IB/4/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-30:53:231 [7] NCCL INFO Channel 03/0 : 246[6] -> 247[7] [receive] via NET/IB/6/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-1:47:237 [1] NCCL INFO Channel 03/0 : 8[0] -> 9[1] [receive] via NET/IB/8/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-31:50:237 [4] NCCL INFO Channel 01/0 : 251[3] -> 252[4] [receive] via NET/IB/15/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-2:53:230 [7] NCCL INFO Channel 01/0 : 22[6] -> 23[7] [receive] via NET/IB/6/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-0:49:236 [3] NCCL INFO Channel 01/0 : 3[3] -> 4[4] [send] via NET/IB/12/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-29:50:234 [4] NCCL INFO Channel 01/0 : 235[3] -> 236[4] [receive] via NET/IB/15/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-30:51:233 [5] NCCL INFO Channel 00/0 : 245[5] -> 246[6] [send] via NET/IB/3/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-1:52:235 [6] NCCL INFO Channel 00/0 : 13[5] -> 14[6] [receive] via NET/IB/4/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-3:47:230 [1] NCCL INFO Channel 00/0 : 24[0] -> 25[1] [receive] via NET/IB/9/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-31:52:233 [6] NCCL INFO Channel 03/0 : 253[5] -> 254[6] [receive] via NET/IB/5/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-2:48:237 [2] NCCL INFO Channel 03/0 : 18[2] -> 19[3] [send] via NET/IB/11/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-29:53:235 [7] NCCL INFO Channel 03/0 : 239[7] -> 240[0] [send] via NET/IB/6/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-30:53:231 [7] NCCL INFO Channel 00/0 : 247[7] -> 248[0] [send] via NET/IB/7/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-1:51:232 [5] NCCL INFO Channel 02/0 : 13[5] -> 14[6] [send] via NET/IB/3/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-3:47:230 [1] NCCL INFO Channel 01/0 : 24[0] -> 25[1] [receive] via NET/IB/8/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-31:46:234 [0] NCCL INFO Channel 03/0 : 248[0] -> 249[1] [send] via NET/IB/1/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-2:51:234 [5] NCCL INFO Channel 00/0 : 21[5] -> 22[6] [send] via NET/IB/3/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-30:51:233 [5] NCCL INFO Channel 01/0 : 245[5] -> 246[6] [send] via NET/IB/2/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-1:46:230 [0] NCCL INFO Channel 02/0 : 8[0] -> 9[1] [send] via NET/IB/0/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-3:47:230 [1] NCCL INFO Channel 02/0 : 24[0] -> 25[1] [receive] via NET/IB/9/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-4:53:235 [7] NCCL INFO Channel 00/0 : 38[6] -> 39[7] [receive] via NET/IB/7/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-31:49:235 [3] NCCL INFO Channel 00/0 : 250[2] -> 251[3] [receive] via NET/IB/13/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-2:53:230 [7] NCCL INFO Channel 02/0 : 22[6] -> 23[7] [receive] via NET/IB/7/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-30:53:231 [7] NCCL INFO Channel 01/0 : 247[7] -> 248[0] [send] via NET/IB/6/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-1:49:236 [3] NCCL INFO Channel 01/0 : 10[2] -> 11[3] [receive] via NET/IB/12/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-0:52:235 [6] NCCL INFO Channel 03/0 : 6[6] -> 7[7] [send] via NET/IB/5/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-31:50:237 [4] NCCL INFO Channel 02/0 : 251[3] -> 252[4] [receive] via NET/IB/14/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-2:51:234 [5] NCCL INFO Channel 01/0 : 21[5] -> 22[6] [send] via NET/IB/2/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-30:51:233 [5] NCCL INFO Channel 02/0 : 245[5] -> 246[6] [send] via NET/IB/3/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-1:47:237 [1] NCCL INFO Channel 00/0 : 9[1] -> 10[2] [send] via NET/IB/9/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-0:49:236 [3] NCCL INFO Channel 02/0 : 3[3] -> 4[4] [send] via NET/IB/13/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-3:47:230 [1] NCCL INFO Channel 03/0 : 24[0] -> 25[1] [receive] via NET/IB/8/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-31:52:233 [6] NCCL INFO Channel 00/0 : 254[6] -> 255[7] [send] via NET/IB/4/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-2:53:230 [7] NCCL INFO Channel 03/0 : 22[6] -> 23[7] [receive] via NET/IB/6/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-30:53:231 [7] NCCL INFO Channel 02/0 : 247[7] -> 248[0] [send] via NET/IB/7/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-1:51:232 [5] NCCL INFO Channel 03/0 : 13[5] -> 14[6] [send] via NET/IB/2/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-0:49:236 [3] NCCL INFO Channel 03/0 : 3[3] -> 4[4] [send] via NET/IB/12/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-3:51:234 [5] NCCL INFO Channel 00/0 : 28[4] -> 29[5] [receive] via NET/IB/3/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-29:51:230 [5] NCCL INFO Channel 03/0 : 237[5] -> 238[6] [send] via NET/IB/2/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-31:47:232 [1] NCCL INFO Channel 00/0 : 248[0] -> 249[1] [receive] via NET/IB/9/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-2:51:234 [5] NCCL INFO Channel 02/0 : 21[5] -> 22[6] [send] via NET/IB/3/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-30:51:233 [5] NCCL INFO Channel 03/0 : 245[5] -> 246[6] [send] via NET/IB/2/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-1:52:235 [6] NCCL INFO Channel 01/0 : 13[5] -> 14[6] [receive] via NET/IB/5/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-0:46:373 [0] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-3:47:230 [1] NCCL INFO Channel 00/0 : 25[1] -> 26[2] [send] via NET/IB/9/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-29:52:237 [6] NCCL INFO Channel 01/0 : 237[5] -> 238[6] [receive] via NET/IB/5/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-6:53:230 [7] NCCL INFO Channel 00/0 : 54[6] -> 55[7] [receive] via NET/IB/7/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-31:49:235 [3] NCCL INFO Channel 01/0 : 250[2] -> 251[3] [receive] via NET/IB/12/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-2:53:230 [7] NCCL INFO Channel 00/0 : 23[7] -> 24[0] [send] via NET/IB/7/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-30:48:237 [2] NCCL INFO Channel 00/0 : 241[1] -> 242[2] [receive] via NET/IB/10/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-1:46:230 [0] NCCL INFO Channel 03/0 : 8[0] -> 9[1] [send] via NET/IB/1/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-0:51:367 [5] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-3:51:234 [5] NCCL INFO Channel 01/0 : 28[4] -> 29[5] [receive] via NET/IB/2/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-29:50:234 [4] NCCL INFO Channel 02/0 : 235[3] -> 236[4] [receive] via NET/IB/14/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-6:51:235 [5] NCCL INFO Channel 00/0 : 52[4] -> 53[5] [receive] via NET/IB/3/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-31:52:233 [6] NCCL INFO Channel 01/0 : 254[6] -> 255[7] [send] via NET/IB/5/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-2:51:234 [5] NCCL INFO Channel 03/0 : 21[5] -> 22[6] [send] via NET/IB/2/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-30:46:230 [0] NCCL INFO Channel 00/0 : 239[7] -> 240[0] [receive] via NET/IB/0/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-1:54:234 [7] NCCL INFO Channel 00/0 : 14[6] -> 15[7] [receive] via NET/IB/7/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-0:48:372 [2] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-3:47:230 [1] NCCL INFO Channel 01/0 : 25[1] -> 26[2] [send] via NET/IB/8/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-29:52:237 [6] NCCL INFO Channel 02/0 : 237[5] -> 238[6] [receive] via NET/IB/4/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-6:53:230 [7] NCCL INFO Channel 01/0 : 54[6] -> 55[7] [receive] via NET/IB/6/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-21:53:233 [7] NCCL INFO Channel 00/0 : 174[6] -> 175[7] [receive] via NET/IB/7/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-5:46:230 [0] NCCL INFO Channel 00/0 : 39[7] -> 40[0] [receive] via NET/IB/0/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-31:50:237 [4] NCCL INFO Channel 03/0 : 251[3] -> 252[4] [receive] via NET/IB/15/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-2:53:230 [7] NCCL INFO Channel 01/0 : 23[7] -> 24[0] [send] via NET/IB/6/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-30:53:231 [7] NCCL INFO Channel 03/0 : 247[7] -> 248[0] [send] via NET/IB/6/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-1:49:236 [3] NCCL INFO Channel 02/0 : 10[2] -> 11[3] [receive] via NET/IB/13/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-0:52:370 [6] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-3:51:234 [5] NCCL INFO Channel 02/0 : 28[4] -> 29[5] [receive] via NET/IB/3/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-29:50:234 [4] NCCL INFO Channel 03/0 : 235[3] -> 236[4] [receive] via NET/IB/15/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-6:51:235 [5] NCCL INFO Channel 01/0 : 52[4] -> 53[5] [receive] via NET/IB/2/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-21:53:233 [7] NCCL INFO Channel 01/0 : 174[6] -> 175[7] [receive] via NET/IB/6/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-31:47:232 [1] NCCL INFO Channel 01/0 : 248[0] -> 249[1] [receive] via NET/IB/8/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-2:52:233 [6] NCCL INFO Channel 00/0 : 21[5] -> 22[6] [receive] via NET/IB/4/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-30:48:237 [2] NCCL INFO Channel 01/0 : 241[1] -> 242[2] [receive] via NET/IB/11/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-1:47:237 [1] NCCL INFO Channel 01/0 : 9[1] -> 10[2] [send] via NET/IB/8/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-0:50:369 [4] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-3:52:237 [6] NCCL INFO Channel 00/0 : 29[5] -> 30[6] [receive] via NET/IB/4/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-29:52:237 [6] NCCL INFO Channel 03/0 : 237[5] -> 238[6] [receive] via NET/IB/5/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-6:53:230 [7] NCCL INFO Channel 02/0 : 54[6] -> 55[7] [receive] via NET/IB/7/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-21:53:233 [7] NCCL INFO Channel 02/0 : 174[6] -> 175[7] [receive] via NET/IB/7/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-5:46:230 [0] NCCL INFO Channel 01/0 : 39[7] -> 40[0] [receive] via NET/IB/1/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-31:52:233 [6] NCCL INFO Channel 02/0 : 254[6] -> 255[7] [send] via NET/IB/4/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-2:50:231 [4] NCCL INFO Channel 00/0 : 19[3] -> 20[4] [receive] via NET/IB/14/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-30:46:230 [0] NCCL INFO Channel 01/0 : 239[7] -> 240[0] [receive] via NET/IB/1/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-1:52:235 [6] NCCL INFO Channel 02/0 : 13[5] -> 14[6] [receive] via NET/IB/4/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-0:53:368 [7] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-3:47:230 [1] NCCL INFO Channel 02/0 : 25[1] -> 26[2] [send] via NET/IB/9/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-29:50:234 [4] NCCL INFO Channel 00/0 : 236[4] -> 237[5] [send] via NET/IB/14/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-6:53:230 [7] NCCL INFO Channel 03/0 : 54[6] -> 55[7] [receive] via NET/IB/6/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-21:53:233 [7] NCCL INFO Channel 03/0 : 174[6] -> 175[7] [receive] via NET/IB/6/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-5:46:230 [0] NCCL INFO Channel 02/0 : 39[7] -> 40[0] [receive] via NET/IB/0/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-31:49:235 [3] NCCL INFO Channel 02/0 : 250[2] -> 251[3] [receive] via NET/IB/13/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-2:53:230 [7] NCCL INFO Channel 02/0 : 23[7] -> 24[0] [send] via NET/IB/7/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-30:48:237 [2] NCCL INFO Channel 02/0 : 241[1] -> 242[2] [receive] via NET/IB/10/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-1:47:237 [1] NCCL INFO Channel 02/0 : 9[1] -> 10[2] [send] via NET/IB/9/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-0:49:371 [3] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-3:51:234 [5] NCCL INFO Channel 03/0 : 28[4] -> 29[5] [receive] via NET/IB/2/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-29:50:234 [4] NCCL INFO Channel 01/0 : 236[4] -> 237[5] [send] via NET/IB/15/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-6:51:235 [5] NCCL INFO Channel 02/0 : 52[4] -> 53[5] [receive] via NET/IB/3/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-21:53:233 [7] NCCL INFO Channel 00/0 : 175[7] -> 176[0] [send] via NET/IB/7/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-5:51:236 [5] NCCL INFO Channel 00/0 : 44[4] -> 45[5] [receive] via NET/IB/3/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-31:50:237 [4] NCCL INFO Channel 00/0 : 252[4] -> 253[5] [send] via NET/IB/14/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-2:52:233 [6] NCCL INFO Channel 01/0 : 21[5] -> 22[6] [receive] via NET/IB/5/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-30:46:230 [0] NCCL INFO Channel 02/0 : 239[7] -> 240[0] [receive] via NET/IB/0/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-1:54:234 [7] NCCL INFO Channel 01/0 : 14[6] -> 15[7] [receive] via NET/IB/6/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-3:52:237 [6] NCCL INFO Channel 01/0 : 29[5] -> 30[6] [receive] via NET/IB/5/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-29:52:237 [6] NCCL INFO Channel 00/0 : 238[6] -> 239[7] [send] via NET/IB/4/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-6:53:230 [7] NCCL INFO Channel 00/0 : 55[7] -> 56[0] [send] via NET/IB/7/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-21:53:233 [7] NCCL INFO Channel 01/0 : 175[7] -> 176[0] [send] via NET/IB/6/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-5:46:230 [0] NCCL INFO Channel 03/0 : 39[7] -> 40[0] [receive] via NET/IB/1/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-4:48:233 [2] NCCL INFO Channel 00/0 : 33[1] -> 34[2] [receive] via NET/IB/10/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-31:47:232 [1] NCCL INFO Channel 02/0 : 248[0] -> 249[1] [receive] via NET/IB/9/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-2:50:231 [4] NCCL INFO Channel 01/0 : 19[3] -> 20[4] [receive] via NET/IB/15/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-30:48:237 [2] NCCL INFO Channel 03/0 : 241[1] -> 242[2] [receive] via NET/IB/11/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-1:52:235 [6] NCCL INFO Channel 03/0 : 13[5] -> 14[6] [receive] via NET/IB/5/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-3:47:230 [1] NCCL INFO Channel 03/0 : 25[1] -> 26[2] [send] via NET/IB/8/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-29:50:234 [4] NCCL INFO Channel 02/0 : 236[4] -> 237[5] [send] via NET/IB/14/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-6:51:235 [5] NCCL INFO Channel 03/0 : 52[4] -> 53[5] [receive] via NET/IB/2/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-21:53:233 [7] NCCL INFO Channel 02/0 : 175[7] -> 176[0] [send] via NET/IB/7/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-5:51:236 [5] NCCL INFO Channel 01/0 : 44[4] -> 45[5] [receive] via NET/IB/2/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-4:46:231 [0] NCCL INFO Channel 00/0 : 31[7] -> 32[0] [receive] via NET/IB/0/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-31:52:233 [6] NCCL INFO Channel 03/0 : 254[6] -> 255[7] [send] via NET/IB/5/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-2:53:230 [7] NCCL INFO Channel 03/0 : 23[7] -> 24[0] [send] via NET/IB/6/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-30:46:230 [0] NCCL INFO Channel 03/0 : 239[7] -> 240[0] [receive] via NET/IB/1/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-1:47:237 [1] NCCL INFO Channel 03/0 : 9[1] -> 10[2] [send] via NET/IB/8/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-3:51:234 [5] NCCL INFO Channel 00/0 : 29[5] -> 30[6] [send] via NET/IB/3/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-29:52:237 [6] NCCL INFO Channel 01/0 : 238[6] -> 239[7] [send] via NET/IB/5/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-6:53:230 [7] NCCL INFO Channel 01/0 : 55[7] -> 56[0] [send] via NET/IB/6/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-21:53:233 [7] NCCL INFO Channel 03/0 : 175[7] -> 176[0] [send] via NET/IB/6/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-5:53:233 [7] NCCL INFO Channel 00/0 : 46[6] -> 47[7] [receive] via NET/IB/7/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-4:53:235 [7] NCCL INFO Channel 01/0 : 38[6] -> 39[7] [receive] via NET/IB/6/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-31:50:237 [4] NCCL INFO Channel 01/0 : 252[4] -> 253[5] [send] via NET/IB/15/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-2:52:233 [6] NCCL INFO Channel 02/0 : 21[5] -> 22[6] [receive] via NET/IB/4/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-30:48:237 [2] NCCL INFO Channel 00/0 : 242[2] -> 243[3] [send] via NET/IB/10/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-1:49:236 [3] NCCL INFO Channel 03/0 : 10[2] -> 11[3] [receive] via NET/IB/12/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-3:52:237 [6] NCCL INFO Channel 02/0 : 29[5] -> 30[6] [receive] via NET/IB/4/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-29:50:234 [4] NCCL INFO Channel 03/0 : 236[4] -> 237[5] [send] via NET/IB/15/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-6:51:235 [5] NCCL INFO Channel 00/0 : 53[5] -> 54[6] [send] via NET/IB/3/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-5:46:230 [0] NCCL INFO Channel 00/0 : 40[0] -> 41[1] [send] via NET/IB/0/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-4:48:233 [2] NCCL INFO Channel 01/0 : 33[1] -> 34[2] [receive] via NET/IB/11/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-8:46:231 [0] NCCL INFO Channel 00/0 : 63[7] -> 64[0] [receive] via NET/IB/0/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-31:49:235 [3] NCCL INFO Channel 03/0 : 250[2] -> 251[3] [receive] via NET/IB/12/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-2:50:231 [4] NCCL INFO Channel 02/0 : 19[3] -> 20[4] [receive] via NET/IB/14/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-30:46:230 [0] NCCL INFO Channel 00/0 : 240[0] -> 241[1] [send] via NET/IB/0/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-1:52:235 [6] NCCL INFO Channel 00/0 : 14[6] -> 15[7] [send] via NET/IB/4/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-3:51:234 [5] NCCL INFO Channel 01/0 : 29[5] -> 30[6] [send] via NET/IB/2/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-29:52:237 [6] NCCL INFO Channel 02/0 : 238[6] -> 239[7] [send] via NET/IB/4/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-6:53:230 [7] NCCL INFO Channel 02/0 : 55[7] -> 56[0] [send] via NET/IB/7/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-5:51:236 [5] NCCL INFO Channel 02/0 : 44[4] -> 45[5] [receive] via NET/IB/3/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-4:53:235 [7] NCCL INFO Channel 02/0 : 38[6] -> 39[7] [receive] via NET/IB/7/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-8:46:231 [0] NCCL INFO Channel 01/0 : 63[7] -> 64[0] [receive] via NET/IB/1/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-31:47:232 [1] NCCL INFO Channel 03/0 : 248[0] -> 249[1] [receive] via NET/IB/8/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-2:52:233 [6] NCCL INFO Channel 03/0 : 21[5] -> 22[6] [receive] via NET/IB/5/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-30:48:237 [2] NCCL INFO Channel 01/0 : 242[2] -> 243[3] [send] via NET/IB/11/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-1:54:234 [7] NCCL INFO Channel 02/0 : 14[6] -> 15[7] [receive] via NET/IB/7/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-3:52:237 [6] NCCL INFO Channel 03/0 : 29[5] -> 30[6] [receive] via NET/IB/5/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-29:52:237 [6] NCCL INFO Channel 03/0 : 238[6] -> 239[7] [send] via NET/IB/5/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-6:51:235 [5] NCCL INFO Channel 01/0 : 53[5] -> 54[6] [send] via NET/IB/2/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-5:46:230 [0] NCCL INFO Channel 01/0 : 40[0] -> 41[1] [send] via NET/IB/1/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-4:46:231 [0] NCCL INFO Channel 01/0 : 31[7] -> 32[0] [receive] via NET/IB/1/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-8:46:231 [0] NCCL INFO Channel 02/0 : 63[7] -> 64[0] [receive] via NET/IB/0/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-7:51:232 [5] NCCL INFO Channel 00/0 : 60[4] -> 61[5] [receive] via NET/IB/3/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-31:50:237 [4] NCCL INFO Channel 02/0 : 252[4] -> 253[5] [send] via NET/IB/14/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-2:50:231 [4] NCCL INFO Channel 03/0 : 19[3] -> 20[4] [receive] via NET/IB/15/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-30:46:230 [0] NCCL INFO Channel 01/0 : 240[0] -> 241[1] [send] via NET/IB/1/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-1:49:236 [3] NCCL INFO Channel 00/0 : 11[3] -> 12[4] [send] via NET/IB/13/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-3:51:234 [5] NCCL INFO Channel 02/0 : 29[5] -> 30[6] [send] via NET/IB/3/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-29:51:368 [5] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-6:53:230 [7] NCCL INFO Channel 03/0 : 55[7] -> 56[0] [send] via NET/IB/6/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-5:53:233 [7] NCCL INFO Channel 01/0 : 46[6] -> 47[7] [receive] via NET/IB/6/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-4:48:233 [2] NCCL INFO Channel 02/0 : 33[1] -> 34[2] [receive] via NET/IB/10/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-8:48:235 [2] NCCL INFO Channel 00/0 : 65[1] -> 66[2] [receive] via NET/IB/10/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-7:54:236 [7] NCCL INFO Channel 00/0 : 62[6] -> 63[7] [receive] via NET/IB/7/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-31:49:235 [3] NCCL INFO Channel 00/0 : 251[3] -> 252[4] [send] via NET/IB/13/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-2:52:233 [6] NCCL INFO Channel 00/0 : 22[6] -> 23[7] [send] via NET/IB/4/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-30:48:237 [2] NCCL INFO Channel 02/0 : 242[2] -> 243[3] [send] via NET/IB/10/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-1:52:235 [6] NCCL INFO Channel 01/0 : 14[6] -> 15[7] [send] via NET/IB/5/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-3:52:237 [6] NCCL INFO Channel 00/0 : 30[6] -> 31[7] [send] via NET/IB/4/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-29:52:367 [6] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-6:51:235 [5] NCCL INFO Channel 02/0 : 53[5] -> 54[6] [send] via NET/IB/3/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-5:51:236 [5] NCCL INFO Channel 03/0 : 44[4] -> 45[5] [receive] via NET/IB/2/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-4:53:235 [7] NCCL INFO Channel 03/0 : 38[6] -> 39[7] [receive] via NET/IB/6/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-8:46:231 [0] NCCL INFO Channel 03/0 : 63[7] -> 64[0] [receive] via NET/IB/1/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-7:51:232 [5] NCCL INFO Channel 01/0 : 60[4] -> 61[5] [receive] via NET/IB/2/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-31:47:232 [1] NCCL INFO Channel 00/0 : 249[1] -> 250[2] [send] via NET/IB/9/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-2:50:231 [4] NCCL INFO Channel 00/0 : 20[4] -> 21[5] [send] via NET/IB/14/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-30:46:230 [0] NCCL INFO Channel 02/0 : 240[0] -> 241[1] [send] via NET/IB/0/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-1:54:234 [7] NCCL INFO Channel 03/0 : 14[6] -> 15[7] [receive] via NET/IB/6/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-3:51:234 [5] NCCL INFO Channel 03/0 : 29[5] -> 30[6] [send] via NET/IB/2/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-29:48:231 [2] NCCL INFO Channel 00/0 : 233[1] -> 234[2] [receive] via NET/IB/10/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-6:51:235 [5] NCCL INFO Channel 03/0 : 53[5] -> 54[6] [send] via NET/IB/2/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-5:46:230 [0] NCCL INFO Channel 02/0 : 40[0] -> 41[1] [send] via NET/IB/0/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-4:46:231 [0] NCCL INFO Channel 02/0 : 31[7] -> 32[0] [receive] via NET/IB/0/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-8:53:230 [7] NCCL INFO Channel 00/0 : 70[6] -> 71[7] [receive] via NET/IB/7/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-7:46:230 [0] NCCL INFO Channel 00/0 : 55[7] -> 56[0] [receive] via NET/IB/0/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-31:50:237 [4] NCCL INFO Channel 03/0 : 252[4] -> 253[5] [send] via NET/IB/15/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-2:52:233 [6] NCCL INFO Channel 01/0 : 22[6] -> 23[7] [send] via NET/IB/5/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-30:48:237 [2] NCCL INFO Channel 03/0 : 242[2] -> 243[3] [send] via NET/IB/11/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-1:52:235 [6] NCCL INFO Channel 02/0 : 14[6] -> 15[7] [send] via NET/IB/4/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-3:52:237 [6] NCCL INFO Channel 01/0 : 30[6] -> 31[7] [send] via NET/IB/5/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-29:48:231 [2] NCCL INFO Channel 01/0 : 233[1] -> 234[2] [receive] via NET/IB/11/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-6:49:234 [3] NCCL INFO Channel 00/0 : 50[2] -> 51[3] [receive] via NET/IB/13/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-5:53:233 [7] NCCL INFO Channel 02/0 : 46[6] -> 47[7] [receive] via NET/IB/7/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-4:48:233 [2] NCCL INFO Channel 03/0 : 33[1] -> 34[2] [receive] via NET/IB/11/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-8:46:231 [0] NCCL INFO Channel 00/0 : 64[0] -> 65[1] [send] via NET/IB/0/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-7:51:232 [5] NCCL INFO Channel 02/0 : 60[4] -> 61[5] [receive] via NET/IB/3/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-31:49:235 [3] NCCL INFO Channel 01/0 : 251[3] -> 252[4] [send] via NET/IB/12/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-2:50:231 [4] NCCL INFO Channel 01/0 : 20[4] -> 21[5] [send] via NET/IB/15/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-30:46:230 [0] NCCL INFO Channel 03/0 : 240[0] -> 241[1] [send] via NET/IB/1/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-1:54:234 [7] NCCL INFO Channel 00/0 : 15[7] -> 16[0] [send] via NET/IB/7/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-3:52:237 [6] NCCL INFO Channel 02/0 : 30[6] -> 31[7] [send] via NET/IB/4/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-29:46:232 [0] NCCL INFO Channel 00/0 : 231[7] -> 232[0] [receive] via NET/IB/0/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-6:49:234 [3] NCCL INFO Channel 01/0 : 50[2] -> 51[3] [receive] via NET/IB/12/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-5:51:236 [5] NCCL INFO Channel 00/0 : 45[5] -> 46[6] [send] via NET/IB/3/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-4:53:235 [7] NCCL INFO Channel 00/0 : 39[7] -> 40[0] [send] via NET/IB/7/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-8:48:235 [2] NCCL INFO Channel 01/0 : 65[1] -> 66[2] [receive] via NET/IB/11/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-7:54:236 [7] NCCL INFO Channel 01/0 : 62[6] -> 63[7] [receive] via NET/IB/6/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-9:46:231 [0] NCCL INFO Channel 00/0 : 71[7] -> 72[0] [receive] via NET/IB/0/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-31:47:232 [1] NCCL INFO Channel 01/0 : 249[1] -> 250[2] [send] via NET/IB/8/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-2:52:233 [6] NCCL INFO Channel 02/0 : 22[6] -> 23[7] [send] via NET/IB/4/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-1:49:236 [3] NCCL INFO Channel 01/0 : 11[3] -> 12[4] [send] via NET/IB/12/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-29:48:231 [2] NCCL INFO Channel 02/0 : 233[1] -> 234[2] [receive] via NET/IB/10/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-6:49:234 [3] NCCL INFO Channel 02/0 : 50[2] -> 51[3] [receive] via NET/IB/13/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-5:46:230 [0] NCCL INFO Channel 03/0 : 40[0] -> 41[1] [send] via NET/IB/1/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-4:46:231 [0] NCCL INFO Channel 03/0 : 31[7] -> 32[0] [receive] via NET/IB/1/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-8:53:230 [7] NCCL INFO Channel 01/0 : 70[6] -> 71[7] [receive] via NET/IB/6/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-7:51:232 [5] NCCL INFO Channel 03/0 : 60[4] -> 61[5] [receive] via NET/IB/2/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-9:46:231 [0] NCCL INFO Channel 01/0 : 71[7] -> 72[0] [receive] via NET/IB/1/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-10:46:230 [0] NCCL INFO Channel 00/0 : 79[7] -> 80[0] [receive] via NET/IB/0/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-31:49:235 [3] NCCL INFO Channel 02/0 : 251[3] -> 252[4] [send] via NET/IB/13/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-2:50:231 [4] NCCL INFO Channel 02/0 : 20[4] -> 21[5] [send] via NET/IB/14/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-1:52:235 [6] NCCL INFO Channel 03/0 : 14[6] -> 15[7] [send] via NET/IB/5/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-29:46:232 [0] NCCL INFO Channel 01/0 : 231[7] -> 232[0] [receive] via NET/IB/1/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-6:49:234 [3] NCCL INFO Channel 03/0 : 50[2] -> 51[3] [receive] via NET/IB/12/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-5:53:233 [7] NCCL INFO Channel 03/0 : 46[6] -> 47[7] [receive] via NET/IB/6/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-4:48:233 [2] NCCL INFO Channel 00/0 : 34[2] -> 35[3] [send] via NET/IB/10/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-8:51:234 [5] NCCL INFO Channel 00/0 : 68[4] -> 69[5] [receive] via NET/IB/3/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-7:46:230 [0] NCCL INFO Channel 01/0 : 55[7] -> 56[0] [receive] via NET/IB/1/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-9:48:233 [2] NCCL INFO Channel 00/0 : 73[1] -> 74[2] [receive] via NET/IB/10/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-3:53:235 [7] NCCL INFO Channel 00/0 : 30[6] -> 31[7] [receive] via NET/IB/7/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-10:46:230 [0] NCCL INFO Channel 01/0 : 79[7] -> 80[0] [receive] via NET/IB/1/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-31:47:232 [1] NCCL INFO Channel 02/0 : 249[1] -> 250[2] [send] via NET/IB/9/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-2:52:233 [6] NCCL INFO Channel 03/0 : 22[6] -> 23[7] [send] via NET/IB/5/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-1:49:236 [3] NCCL INFO Channel 02/0 : 11[3] -> 12[4] [send] via NET/IB/13/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-29:48:231 [2] NCCL INFO Channel 03/0 : 233[1] -> 234[2] [receive] via NET/IB/11/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-6:48:231 [2] NCCL INFO Channel 00/0 : 49[1] -> 50[2] [receive] via NET/IB/10/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-5:51:236 [5] NCCL INFO Channel 01/0 : 45[5] -> 46[6] [send] via NET/IB/2/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-4:53:235 [7] NCCL INFO Channel 01/0 : 39[7] -> 40[0] [send] via NET/IB/6/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-8:46:231 [0] NCCL INFO Channel 01/0 : 64[0] -> 65[1] [send] via NET/IB/1/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-7:54:236 [7] NCCL INFO Channel 02/0 : 62[6] -> 63[7] [receive] via NET/IB/7/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-9:46:231 [0] NCCL INFO Channel 02/0 : 71[7] -> 72[0] [receive] via NET/IB/0/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-3:52:237 [6] NCCL INFO Channel 03/0 : 30[6] -> 31[7] [send] via NET/IB/5/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-10:46:230 [0] NCCL INFO Channel 02/0 : 79[7] -> 80[0] [receive] via NET/IB/0/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-31:49:235 [3] NCCL INFO Channel 03/0 : 251[3] -> 252[4] [send] via NET/IB/12/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-2:50:231 [4] NCCL INFO Channel 03/0 : 20[4] -> 21[5] [send] via NET/IB/15/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-1:54:234 [7] NCCL INFO Channel 01/0 : 15[7] -> 16[0] [send] via NET/IB/6/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-29:46:232 [0] NCCL INFO Channel 02/0 : 231[7] -> 232[0] [receive] via NET/IB/0/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-6:47:232 [1] NCCL INFO Channel 00/0 : 48[0] -> 49[1] [receive] via NET/IB/9/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-5:51:236 [5] NCCL INFO Channel 02/0 : 45[5] -> 46[6] [send] via NET/IB/3/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-4:46:231 [0] NCCL INFO Channel 00/0 : 32[0] -> 33[1] [send] via NET/IB/0/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-8:48:235 [2] NCCL INFO Channel 02/0 : 65[1] -> 66[2] [receive] via NET/IB/10/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-7:51:232 [5] NCCL INFO Channel 00/0 : 61[5] -> 62[6] [send] via NET/IB/3/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-9:48:233 [2] NCCL INFO Channel 01/0 : 73[1] -> 74[2] [receive] via NET/IB/11/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-3:53:235 [7] NCCL INFO Channel 01/0 : 30[6] -> 31[7] [receive] via NET/IB/6/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-10:46:230 [0] NCCL INFO Channel 03/0 : 79[7] -> 80[0] [receive] via NET/IB/1/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-31:47:232 [1] NCCL INFO Channel 03/0 : 249[1] -> 250[2] [send] via NET/IB/8/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-2:51:371 [5] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-1:49:236 [3] NCCL INFO Channel 03/0 : 11[3] -> 12[4] [send] via NET/IB/12/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-29:48:231 [2] NCCL INFO Channel 00/0 : 234[2] -> 235[3] [send] via NET/IB/10/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-6:46:236 [0] NCCL INFO Channel 00/0 : 47[7] -> 48[0] [receive] via NET/IB/0/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-5:53:233 [7] NCCL INFO Channel 00/0 : 47[7] -> 48[0] [send] via NET/IB/7/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-4:48:233 [2] NCCL INFO Channel 01/0 : 34[2] -> 35[3] [send] via NET/IB/11/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-8:46:231 [0] NCCL INFO Channel 02/0 : 64[0] -> 65[1] [send] via NET/IB/0/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-7:46:230 [0] NCCL INFO Channel 02/0 : 55[7] -> 56[0] [receive] via NET/IB/0/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-9:46:231 [0] NCCL INFO Channel 03/0 : 71[7] -> 72[0] [receive] via NET/IB/1/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-3:53:235 [7] NCCL INFO Channel 02/0 : 30[6] -> 31[7] [receive] via NET/IB/7/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-30:47:232 [1] NCCL INFO Channel 00/0 : 240[0] -> 241[1] [receive] via NET/IB/9/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-10:46:230 [0] NCCL INFO Channel 00/0 : 80[0] -> 81[1] [send] via NET/IB/0/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-31:52:366 [6] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-2:52:369 [6] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-1:54:234 [7] NCCL INFO Channel 02/0 : 15[7] -> 16[0] [send] via NET/IB/7/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-29:48:231 [2] NCCL INFO Channel 01/0 : 234[2] -> 235[3] [send] via NET/IB/11/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-6:49:234 [3] NCCL INFO Channel 00/0 : 51[3] -> 52[4] [send] via NET/IB/13/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-5:51:236 [5] NCCL INFO Channel 03/0 : 45[5] -> 46[6] [send] via NET/IB/2/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-4:53:235 [7] NCCL INFO Channel 02/0 : 39[7] -> 40[0] [send] via NET/IB/7/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-8:51:234 [5] NCCL INFO Channel 01/0 : 68[4] -> 69[5] [receive] via NET/IB/2/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-7:54:236 [7] NCCL INFO Channel 03/0 : 62[6] -> 63[7] [receive] via NET/IB/6/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-9:48:233 [2] NCCL INFO Channel 02/0 : 73[1] -> 74[2] [receive] via NET/IB/10/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-3:50:236 [4] NCCL INFO Channel 00/0 : 27[3] -> 28[4] [receive] via NET/IB/14/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-30:47:232 [1] NCCL INFO Channel 01/0 : 240[0] -> 241[1] [receive] via NET/IB/8/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-21:52:231 [6] NCCL INFO Channel 00/0 : 173[5] -> 174[6] [receive] via NET/IB/4/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-28:48:231 [2] NCCL INFO Channel 00/0 : 225[1] -> 226[2] [receive] via NET/IB/10/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-10:46:230 [0] NCCL INFO Channel 01/0 : 80[0] -> 81[1] [send] via NET/IB/1/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-31:53:367 [7] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-2:53:370 [7] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-1:54:234 [7] NCCL INFO Channel 03/0 : 15[7] -> 16[0] [send] via NET/IB/6/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-29:46:232 [0] NCCL INFO Channel 03/0 : 231[7] -> 232[0] [receive] via NET/IB/1/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-6:48:231 [2] NCCL INFO Channel 01/0 : 49[1] -> 50[2] [receive] via NET/IB/11/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-5:53:233 [7] NCCL INFO Channel 01/0 : 47[7] -> 48[0] [send] via NET/IB/6/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-4:46:231 [0] NCCL INFO Channel 01/0 : 32[0] -> 33[1] [send] via NET/IB/1/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-8:53:230 [7] NCCL INFO Channel 02/0 : 70[6] -> 71[7] [receive] via NET/IB/7/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-7:51:232 [5] NCCL INFO Channel 01/0 : 61[5] -> 62[6] [send] via NET/IB/2/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-9:46:231 [0] NCCL INFO Channel 00/0 : 72[0] -> 73[1] [send] via NET/IB/0/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-3:53:235 [7] NCCL INFO Channel 03/0 : 30[6] -> 31[7] [receive] via NET/IB/6/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-30:49:234 [3] NCCL INFO Channel 00/0 : 242[2] -> 243[3] [receive] via NET/IB/13/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-21:52:231 [6] NCCL INFO Channel 01/0 : 173[5] -> 174[6] [receive] via NET/IB/5/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-28:48:231 [2] NCCL INFO Channel 01/0 : 225[1] -> 226[2] [receive] via NET/IB/11/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-26:53:232 [7] NCCL INFO Channel 00/0 : 214[6] -> 215[7] [receive] via NET/IB/7/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-0:47:374 [1] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-10:48:231 [2] NCCL INFO Channel 00/0 : 81[1] -> 82[2] [receive] via NET/IB/10/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-31:48:371 [2] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-2:47:235 [1] NCCL INFO Channel 00/0 : 16[0] -> 17[1] [receive] via NET/IB/9/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-1:50:233 [4] NCCL INFO Channel 00/0 : 11[3] -> 12[4] [receive] via NET/IB/14/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-29:48:231 [2] NCCL INFO Channel 02/0 : 234[2] -> 235[3] [send] via NET/IB/10/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-6:47:232 [1] NCCL INFO Channel 01/0 : 48[0] -> 49[1] [receive] via NET/IB/8/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-5:53:233 [7] NCCL INFO Channel 02/0 : 47[7] -> 48[0] [send] via NET/IB/7/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-4:53:235 [7] NCCL INFO Channel 03/0 : 39[7] -> 40[0] [send] via NET/IB/6/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-8:48:235 [2] NCCL INFO Channel 03/0 : 65[1] -> 66[2] [receive] via NET/IB/11/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-7:46:230 [0] NCCL INFO Channel 03/0 : 55[7] -> 56[0] [receive] via NET/IB/1/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-9:48:233 [2] NCCL INFO Channel 03/0 : 73[1] -> 74[2] [receive] via NET/IB/11/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-3:50:236 [4] NCCL INFO Channel 01/0 : 27[3] -> 28[4] [receive] via NET/IB/15/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-30:47:232 [1] NCCL INFO Channel 02/0 : 240[0] -> 241[1] [receive] via NET/IB/9/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-21:52:231 [6] NCCL INFO Channel 02/0 : 173[5] -> 174[6] [receive] via NET/IB/4/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-28:53:235 [7] NCCL INFO Channel 00/0 : 230[6] -> 231[7] [receive] via NET/IB/7/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-26:51:236 [5] NCCL INFO Channel 00/0 : 212[4] -> 213[5] [receive] via NET/IB/3/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-10:46:230 [0] NCCL INFO Channel 02/0 : 80[0] -> 81[1] [send] via NET/IB/0/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-31:50:369 [4] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-2:47:235 [1] NCCL INFO Channel 01/0 : 16[0] -> 17[1] [receive] via NET/IB/8/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-1:50:233 [4] NCCL INFO Channel 01/0 : 11[3] -> 12[4] [receive] via NET/IB/15/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-29:46:232 [0] NCCL INFO Channel 00/0 : 232[0] -> 233[1] [send] via NET/IB/0/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-6:46:236 [0] NCCL INFO Channel 01/0 : 47[7] -> 48[0] [receive] via NET/IB/1/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-5:49:232 [3] NCCL INFO Channel 00/0 : 42[2] -> 43[3] [receive] via NET/IB/13/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-4:48:233 [2] NCCL INFO Channel 02/0 : 34[2] -> 35[3] [send] via NET/IB/10/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-8:46:231 [0] NCCL INFO Channel 03/0 : 64[0] -> 65[1] [send] via NET/IB/1/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-7:51:232 [5] NCCL INFO Channel 02/0 : 61[5] -> 62[6] [send] via NET/IB/3/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-9:46:231 [0] NCCL INFO Channel 01/0 : 72[0] -> 73[1] [send] via NET/IB/1/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-3:53:235 [7] NCCL INFO Channel 00/0 : 31[7] -> 32[0] [send] via NET/IB/7/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-30:49:234 [3] NCCL INFO Channel 01/0 : 242[2] -> 243[3] [receive] via NET/IB/12/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-21:52:231 [6] NCCL INFO Channel 03/0 : 173[5] -> 174[6] [receive] via NET/IB/5/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-28:48:231 [2] NCCL INFO Channel 02/0 : 225[1] -> 226[2] [receive] via NET/IB/10/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-26:46:231 [0] NCCL INFO Channel 00/0 : 207[7] -> 208[0] [receive] via NET/IB/0/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-10:48:231 [2] NCCL INFO Channel 01/0 : 81[1] -> 82[2] [receive] via NET/IB/11/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-31:49:370 [3] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-2:47:235 [1] NCCL INFO Channel 02/0 : 16[0] -> 17[1] [receive] via NET/IB/9/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-1:50:233 [4] NCCL INFO Channel 02/0 : 11[3] -> 12[4] [receive] via NET/IB/14/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-29:48:231 [2] NCCL INFO Channel 03/0 : 234[2] -> 235[3] [send] via NET/IB/11/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-6:49:234 [3] NCCL INFO Channel 01/0 : 51[3] -> 52[4] [send] via NET/IB/12/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-5:53:233 [7] NCCL INFO Channel 03/0 : 47[7] -> 48[0] [send] via NET/IB/6/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-4:46:231 [0] NCCL INFO Channel 02/0 : 32[0] -> 33[1] [send] via NET/IB/0/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-8:53:230 [7] NCCL INFO Channel 03/0 : 70[6] -> 71[7] [receive] via NET/IB/6/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-7:54:236 [7] NCCL INFO Channel 00/0 : 63[7] -> 64[0] [send] via NET/IB/7/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-9:48:233 [2] NCCL INFO Channel 00/0 : 74[2] -> 75[3] [send] via NET/IB/10/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-3:46:366 [0] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-30:47:232 [1] NCCL INFO Channel 03/0 : 240[0] -> 241[1] [receive] via NET/IB/8/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-21:52:231 [6] NCCL INFO Channel 00/0 : 174[6] -> 175[7] [send] via NET/IB/4/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-28:53:235 [7] NCCL INFO Channel 01/0 : 230[6] -> 231[7] [receive] via NET/IB/6/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-26:53:232 [7] NCCL INFO Channel 01/0 : 214[6] -> 215[7] [receive] via NET/IB/6/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-27:48:235 [2] NCCL INFO Channel 00/0 : 217[1] -> 218[2] [receive] via NET/IB/10/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-11:46:230 [0] NCCL INFO Channel 00/0 : 87[7] -> 88[0] [receive] via NET/IB/0/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-23:53:231 [7] NCCL INFO Channel 00/0 : 190[6] -> 191[7] [receive] via NET/IB/7/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-24:46:237 [0] NCCL INFO Channel 00/0 : 191[7] -> 192[0] [receive] via NET/IB/0/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-10:46:230 [0] NCCL INFO Channel 03/0 : 80[0] -> 81[1] [send] via NET/IB/1/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-31:47:373 [1] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-2:49:236 [3] NCCL INFO Channel 00/0 : 18[2] -> 19[3] [receive] via NET/IB/13/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-1:50:233 [4] NCCL INFO Channel 03/0 : 11[3] -> 12[4] [receive] via NET/IB/15/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-29:46:232 [0] NCCL INFO Channel 01/0 : 232[0] -> 233[1] [send] via NET/IB/1/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-6:48:231 [2] NCCL INFO Channel 02/0 : 49[1] -> 50[2] [receive] via NET/IB/10/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-5:49:232 [3] NCCL INFO Channel 01/0 : 42[2] -> 43[3] [receive] via NET/IB/12/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-4:48:233 [2] NCCL INFO Channel 03/0 : 34[2] -> 35[3] [send] via NET/IB/11/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-8:51:234 [5] NCCL INFO Channel 02/0 : 68[4] -> 69[5] [receive] via NET/IB/3/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-7:51:232 [5] NCCL INFO Channel 03/0 : 61[5] -> 62[6] [send] via NET/IB/2/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-9:46:231 [0] NCCL INFO Channel 02/0 : 72[0] -> 73[1] [send] via NET/IB/0/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-3:50:236 [4] NCCL INFO Channel 02/0 : 27[3] -> 28[4] [receive] via NET/IB/14/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-30:49:234 [3] NCCL INFO Channel 02/0 : 242[2] -> 243[3] [receive] via NET/IB/13/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-21:52:231 [6] NCCL INFO Channel 01/0 : 174[6] -> 175[7] [send] via NET/IB/5/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-28:51:236 [5] NCCL INFO Channel 00/0 : 228[4] -> 229[5] [receive] via NET/IB/3/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-26:51:236 [5] NCCL INFO Channel 01/0 : 212[4] -> 213[5] [receive] via NET/IB/2/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-27:53:237 [7] NCCL INFO Channel 00/0 : 222[6] -> 223[7] [receive] via NET/IB/7/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-11:46:230 [0] NCCL INFO Channel 01/0 : 87[7] -> 88[0] [receive] via NET/IB/1/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-23:51:235 [5] NCCL INFO Channel 00/0 : 188[4] -> 189[5] [receive] via NET/IB/3/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-10:53:235 [7] NCCL INFO Channel 00/0 : 86[6] -> 87[7] [receive] via NET/IB/7/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-31:51:368 [5] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-2:47:235 [1] NCCL INFO Channel 03/0 : 16[0] -> 17[1] [receive] via NET/IB/8/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-1:50:233 [4] NCCL INFO Channel 00/0 : 12[4] -> 13[5] [send] via NET/IB/14/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-29:46:232 [0] NCCL INFO Channel 02/0 : 232[0] -> 233[1] [send] via NET/IB/0/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-6:47:232 [1] NCCL INFO Channel 02/0 : 48[0] -> 49[1] [receive] via NET/IB/9/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-5:48:234 [2] NCCL INFO Channel 00/0 : 41[1] -> 42[2] [receive] via NET/IB/10/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-4:46:231 [0] NCCL INFO Channel 03/0 : 32[0] -> 33[1] [send] via NET/IB/1/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-8:48:235 [2] NCCL INFO Channel 00/0 : 66[2] -> 67[3] [send] via NET/IB/10/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-7:46:230 [0] NCCL INFO Channel 00/0 : 56[0] -> 57[1] [send] via NET/IB/0/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-9:46:231 [0] NCCL INFO Channel 03/0 : 72[0] -> 73[1] [send] via NET/IB/1/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-3:53:235 [7] NCCL INFO Channel 01/0 : 31[7] -> 32[0] [send] via NET/IB/6/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-30:47:232 [1] NCCL INFO Channel 00/0 : 241[1] -> 242[2] [send] via NET/IB/9/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-21:52:231 [6] NCCL INFO Channel 02/0 : 174[6] -> 175[7] [send] via NET/IB/4/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-28:48:231 [2] NCCL INFO Channel 03/0 : 225[1] -> 226[2] [receive] via NET/IB/11/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-26:46:231 [0] NCCL INFO Channel 01/0 : 207[7] -> 208[0] [receive] via NET/IB/1/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-27:48:235 [2] NCCL INFO Channel 01/0 : 217[1] -> 218[2] [receive] via NET/IB/11/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-11:46:230 [0] NCCL INFO Channel 02/0 : 87[7] -> 88[0] [receive] via NET/IB/0/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-23:53:231 [7] NCCL INFO Channel 01/0 : 190[6] -> 191[7] [receive] via NET/IB/6/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-22:46:231 [0] NCCL INFO Channel 00/0 : 175[7] -> 176[0] [receive] via NET/IB/0/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-10:51:232 [5] NCCL INFO Channel 00/0 : 84[4] -> 85[5] [receive] via NET/IB/3/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-31:46:372 [0] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-2:49:236 [3] NCCL INFO Channel 01/0 : 18[2] -> 19[3] [receive] via NET/IB/12/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-1:46:366 [0] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-29:46:232 [0] NCCL INFO Channel 03/0 : 232[0] -> 233[1] [send] via NET/IB/1/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-6:46:236 [0] NCCL INFO Channel 02/0 : 47[7] -> 48[0] [receive] via NET/IB/0/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-5:49:232 [3] NCCL INFO Channel 02/0 : 42[2] -> 43[3] [receive] via NET/IB/13/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-4:49:234 [3] NCCL INFO Channel 00/0 : 34[2] -> 35[3] [receive] via NET/IB/13/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-8:53:230 [7] NCCL INFO Channel 00/0 : 71[7] -> 72[0] [send] via NET/IB/7/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-7:54:236 [7] NCCL INFO Channel 01/0 : 63[7] -> 64[0] [send] via NET/IB/6/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-9:48:233 [2] NCCL INFO Channel 01/0 : 74[2] -> 75[3] [send] via NET/IB/11/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-3:47:367 [1] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-30:49:234 [3] NCCL INFO Channel 03/0 : 242[2] -> 243[3] [receive] via NET/IB/12/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-21:52:231 [6] NCCL INFO Channel 03/0 : 174[6] -> 175[7] [send] via NET/IB/5/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-28:53:235 [7] NCCL INFO Channel 02/0 : 230[6] -> 231[7] [receive] via NET/IB/7/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-26:53:232 [7] NCCL INFO Channel 02/0 : 214[6] -> 215[7] [receive] via NET/IB/7/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-27:53:237 [7] NCCL INFO Channel 01/0 : 222[6] -> 223[7] [receive] via NET/IB/6/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-11:46:230 [0] NCCL INFO Channel 03/0 : 87[7] -> 88[0] [receive] via NET/IB/1/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-23:51:235 [5] NCCL INFO Channel 01/0 : 188[4] -> 189[5] [receive] via NET/IB/2/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-22:53:234 [7] NCCL INFO Channel 00/0 : 182[6] -> 183[7] [receive] via NET/IB/7/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-24:46:237 [0] NCCL INFO Channel 01/0 : 191[7] -> 192[0] [receive] via NET/IB/1/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-10:48:231 [2] NCCL INFO Channel 02/0 : 81[1] -> 82[2] [receive] via NET/IB/10/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-2:47:235 [1] NCCL INFO Channel 00/0 : 17[1] -> 18[2] [send] via NET/IB/9/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-1:50:233 [4] NCCL INFO Channel 01/0 : 12[4] -> 13[5] [send] via NET/IB/15/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-29:53:366 [7] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-6:48:231 [2] NCCL INFO Channel 03/0 : 49[1] -> 50[2] [receive] via NET/IB/11/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-5:48:234 [2] NCCL INFO Channel 01/0 : 41[1] -> 42[2] [receive] via NET/IB/11/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-4:50:230 [4] NCCL INFO Channel 00/0 : 35[3] -> 36[4] [receive] via NET/IB/14/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-7:46:230 [0] NCCL INFO Channel 01/0 : 56[0] -> 57[1] [send] via NET/IB/1/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-9:48:233 [2] NCCL INFO Channel 02/0 : 74[2] -> 75[3] [send] via NET/IB/10/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-3:53:235 [7] NCCL INFO Channel 02/0 : 31[7] -> 32[0] [send] via NET/IB/7/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-30:47:232 [1] NCCL INFO Channel 01/0 : 241[1] -> 242[2] [send] via NET/IB/8/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-21:51:235 [5] NCCL INFO Channel 00/0 : 172[4] -> 173[5] [receive] via NET/IB/3/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-28:51:236 [5] NCCL INFO Channel 01/0 : 228[4] -> 229[5] [receive] via NET/IB/2/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-26:51:236 [5] NCCL INFO Channel 02/0 : 212[4] -> 213[5] [receive] via NET/IB/3/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-27:51:234 [5] NCCL INFO Channel 00/0 : 220[4] -> 221[5] [receive] via NET/IB/3/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-11:46:230 [0] NCCL INFO Channel 00/0 : 88[0] -> 89[1] [send] via NET/IB/0/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-23:53:231 [7] NCCL INFO Channel 02/0 : 190[6] -> 191[7] [receive] via NET/IB/7/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-22:51:233 [5] NCCL INFO Channel 00/0 : 180[4] -> 181[5] [receive] via NET/IB/3/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-24:48:230 [2] NCCL INFO Channel 00/0 : 193[1] -> 194[2] [receive] via NET/IB/10/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-14:46:234 [0] NCCL INFO Channel 00/0 : 111[7] -> 112[0] [receive] via NET/IB/0/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-8:51:234 [5] NCCL INFO Channel 03/0 : 68[4] -> 69[5] [receive] via NET/IB/2/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-10:53:235 [7] NCCL INFO Channel 01/0 : 86[6] -> 87[7] [receive] via NET/IB/6/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-2:47:235 [1] NCCL INFO Channel 01/0 : 17[1] -> 18[2] [send] via NET/IB/8/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-29:47:233 [1] NCCL INFO Channel 00/0 : 232[0] -> 233[1] [receive] via NET/IB/9/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-6:47:232 [1] NCCL INFO Channel 03/0 : 48[0] -> 49[1] [receive] via NET/IB/8/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-5:49:232 [3] NCCL INFO Channel 03/0 : 42[2] -> 43[3] [receive] via NET/IB/12/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-4:49:234 [3] NCCL INFO Channel 01/0 : 34[2] -> 35[3] [receive] via NET/IB/12/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-7:50:234 [4] NCCL INFO Channel 00/0 : 59[3] -> 60[4] [receive] via NET/IB/14/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-9:48:233 [2] NCCL INFO Channel 03/0 : 74[2] -> 75[3] [send] via NET/IB/11/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-3:50:236 [4] NCCL INFO Channel 03/0 : 27[3] -> 28[4] [receive] via NET/IB/15/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-30:49:234 [3] NCCL INFO Channel 00/0 : 243[3] -> 244[4] [send] via NET/IB/13/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-21:51:235 [5] NCCL INFO Channel 01/0 : 172[4] -> 173[5] [receive] via NET/IB/2/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-28:48:231 [2] NCCL INFO Channel 00/0 : 226[2] -> 227[3] [send] via NET/IB/10/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-26:46:231 [0] NCCL INFO Channel 02/0 : 207[7] -> 208[0] [receive] via NET/IB/0/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-27:48:235 [2] NCCL INFO Channel 02/0 : 217[1] -> 218[2] [receive] via NET/IB/10/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-11:46:230 [0] NCCL INFO Channel 01/0 : 88[0] -> 89[1] [send] via NET/IB/1/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-23:51:235 [5] NCCL INFO Channel 02/0 : 188[4] -> 189[5] [receive] via NET/IB/3/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-22:48:236 [2] NCCL INFO Channel 00/0 : 177[1] -> 178[2] [receive] via NET/IB/10/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-24:51:234 [5] NCCL INFO Channel 00/0 : 196[4] -> 197[5] [receive] via NET/IB/3/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-14:46:234 [0] NCCL INFO Channel 01/0 : 111[7] -> 112[0] [receive] via NET/IB/1/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-13:48:230 [2] NCCL INFO Channel 00/0 : 105[1] -> 106[2] [receive] via NET/IB/10/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-8:48:235 [2] NCCL INFO Channel 01/0 : 66[2] -> 67[3] [send] via NET/IB/11/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-10:51:232 [5] NCCL INFO Channel 01/0 : 84[4] -> 85[5] [receive] via NET/IB/2/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-2:49:236 [3] NCCL INFO Channel 02/0 : 18[2] -> 19[3] [receive] via NET/IB/13/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-29:47:233 [1] NCCL INFO Channel 01/0 : 232[0] -> 233[1] [receive] via NET/IB/8/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-6:49:234 [3] NCCL INFO Channel 02/0 : 51[3] -> 52[4] [send] via NET/IB/13/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-5:49:232 [3] NCCL INFO Channel 00/0 : 43[3] -> 44[4] [send] via NET/IB/13/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-4:50:230 [4] NCCL INFO Channel 01/0 : 35[3] -> 36[4] [receive] via NET/IB/15/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-9:47:230 [1] NCCL INFO Channel 00/0 : 72[0] -> 73[1] [receive] via NET/IB/9/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-3:49:232 [3] NCCL INFO Channel 00/0 : 26[2] -> 27[3] [receive] via NET/IB/13/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-30:47:232 [1] NCCL INFO Channel 02/0 : 241[1] -> 242[2] [send] via NET/IB/9/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-21:51:235 [5] NCCL INFO Channel 02/0 : 172[4] -> 173[5] [receive] via NET/IB/3/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-28:53:235 [7] NCCL INFO Channel 03/0 : 230[6] -> 231[7] [receive] via NET/IB/6/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-26:53:232 [7] NCCL INFO Channel 03/0 : 214[6] -> 215[7] [receive] via NET/IB/6/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-27:53:237 [7] NCCL INFO Channel 02/0 : 222[6] -> 223[7] [receive] via NET/IB/7/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-11:46:230 [0] NCCL INFO Channel 02/0 : 88[0] -> 89[1] [send] via NET/IB/0/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-23:53:231 [7] NCCL INFO Channel 03/0 : 190[6] -> 191[7] [receive] via NET/IB/6/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-22:46:231 [0] NCCL INFO Channel 01/0 : 175[7] -> 176[0] [receive] via NET/IB/1/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-24:46:237 [0] NCCL INFO Channel 02/0 : 191[7] -> 192[0] [receive] via NET/IB/0/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-14:46:234 [0] NCCL INFO Channel 02/0 : 111[7] -> 112[0] [receive] via NET/IB/0/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-13:48:230 [2] NCCL INFO Channel 01/0 : 105[1] -> 106[2] [receive] via NET/IB/11/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-1:50:233 [4] NCCL INFO Channel 02/0 : 12[4] -> 13[5] [send] via NET/IB/14/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-8:53:230 [7] NCCL INFO Channel 01/0 : 71[7] -> 72[0] [send] via NET/IB/6/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-10:48:231 [2] NCCL INFO Channel 03/0 : 81[1] -> 82[2] [receive] via NET/IB/11/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-2:47:235 [1] NCCL INFO Channel 02/0 : 17[1] -> 18[2] [send] via NET/IB/9/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-29:47:233 [1] NCCL INFO Channel 02/0 : 232[0] -> 233[1] [receive] via NET/IB/9/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-6:48:231 [2] NCCL INFO Channel 00/0 : 50[2] -> 51[3] [send] via NET/IB/10/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-5:48:234 [2] NCCL INFO Channel 02/0 : 41[1] -> 42[2] [receive] via NET/IB/10/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-4:49:234 [3] NCCL INFO Channel 02/0 : 34[2] -> 35[3] [receive] via NET/IB/13/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-9:47:230 [1] NCCL INFO Channel 01/0 : 72[0] -> 73[1] [receive] via NET/IB/8/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-3:53:235 [7] NCCL INFO Channel 03/0 : 31[7] -> 32[0] [send] via NET/IB/6/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-30:49:234 [3] NCCL INFO Channel 01/0 : 243[3] -> 244[4] [send] via NET/IB/12/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-21:51:235 [5] NCCL INFO Channel 03/0 : 172[4] -> 173[5] [receive] via NET/IB/2/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-28:48:231 [2] NCCL INFO Channel 01/0 : 226[2] -> 227[3] [send] via NET/IB/11/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-26:51:236 [5] NCCL INFO Channel 03/0 : 212[4] -> 213[5] [receive] via NET/IB/2/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-27:51:234 [5] NCCL INFO Channel 01/0 : 220[4] -> 221[5] [receive] via NET/IB/2/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-11:48:231 [2] NCCL INFO Channel 00/0 : 89[1] -> 90[2] [receive] via NET/IB/10/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-23:51:235 [5] NCCL INFO Channel 03/0 : 188[4] -> 189[5] [receive] via NET/IB/2/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-22:53:234 [7] NCCL INFO Channel 01/0 : 182[6] -> 183[7] [receive] via NET/IB/6/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-24:48:230 [2] NCCL INFO Channel 01/0 : 193[1] -> 194[2] [receive] via NET/IB/11/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-14:46:234 [0] NCCL INFO Channel 03/0 : 111[7] -> 112[0] [receive] via NET/IB/1/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-13:51:237 [5] NCCL INFO Channel 00/0 : 108[4] -> 109[5] [receive] via NET/IB/3/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-1:48:368 [2] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-7:54:236 [7] NCCL INFO Channel 02/0 : 63[7] -> 64[0] [send] via NET/IB/7/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-8:51:234 [5] NCCL INFO Channel 00/0 : 69[5] -> 70[6] [send] via NET/IB/3/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-10:53:235 [7] NCCL INFO Channel 02/0 : 86[6] -> 87[7] [receive] via NET/IB/7/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-2:49:236 [3] NCCL INFO Channel 03/0 : 18[2] -> 19[3] [receive] via NET/IB/12/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-29:47:233 [1] NCCL INFO Channel 03/0 : 232[0] -> 233[1] [receive] via NET/IB/8/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-6:46:236 [0] NCCL INFO Channel 03/0 : 47[7] -> 48[0] [receive] via NET/IB/1/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-5:49:232 [3] NCCL INFO Channel 01/0 : 43[3] -> 44[4] [send] via NET/IB/12/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-4:50:230 [4] NCCL INFO Channel 02/0 : 35[3] -> 36[4] [receive] via NET/IB/14/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-9:49:232 [3] NCCL INFO Channel 00/0 : 74[2] -> 75[3] [receive] via NET/IB/13/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-3:50:236 [4] NCCL INFO Channel 00/0 : 28[4] -> 29[5] [send] via NET/IB/14/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-30:47:232 [1] NCCL INFO Channel 03/0 : 241[1] -> 242[2] [send] via NET/IB/8/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-21:51:235 [5] NCCL INFO Channel 00/0 : 173[5] -> 174[6] [send] via NET/IB/3/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-28:51:236 [5] NCCL INFO Channel 02/0 : 228[4] -> 229[5] [receive] via NET/IB/3/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-26:46:231 [0] NCCL INFO Channel 03/0 : 207[7] -> 208[0] [receive] via NET/IB/1/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-27:48:235 [2] NCCL INFO Channel 03/0 : 217[1] -> 218[2] [receive] via NET/IB/11/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-11:46:230 [0] NCCL INFO Channel 03/0 : 88[0] -> 89[1] [send] via NET/IB/1/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-23:53:231 [7] NCCL INFO Channel 00/0 : 191[7] -> 192[0] [send] via NET/IB/7/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-22:51:233 [5] NCCL INFO Channel 01/0 : 180[4] -> 181[5] [receive] via NET/IB/2/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-24:51:234 [5] NCCL INFO Channel 01/0 : 196[4] -> 197[5] [receive] via NET/IB/2/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-14:46:234 [0] NCCL INFO Channel 00/0 : 112[0] -> 113[1] [send] via NET/IB/0/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-13:48:230 [2] NCCL INFO Channel 02/0 : 105[1] -> 106[2] [receive] via NET/IB/10/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-1:50:233 [4] NCCL INFO Channel 03/0 : 12[4] -> 13[5] [send] via NET/IB/15/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-7:52:231 [6] NCCL INFO Channel 00/0 : 61[5] -> 62[6] [receive] via NET/IB/4/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-8:48:235 [2] NCCL INFO Channel 02/0 : 66[2] -> 67[3] [send] via NET/IB/10/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-10:51:232 [5] NCCL INFO Channel 02/0 : 84[4] -> 85[5] [receive] via NET/IB/3/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-2:47:235 [1] NCCL INFO Channel 03/0 : 17[1] -> 18[2] [send] via NET/IB/8/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-29:47:233 [1] NCCL INFO Channel 00/0 : 233[1] -> 234[2] [send] via NET/IB/9/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-6:47:232 [1] NCCL INFO Channel 00/0 : 49[1] -> 50[2] [send] via NET/IB/9/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-5:48:234 [2] NCCL INFO Channel 03/0 : 41[1] -> 42[2] [receive] via NET/IB/11/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-4:49:234 [3] NCCL INFO Channel 03/0 : 34[2] -> 35[3] [receive] via NET/IB/12/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-9:47:230 [1] NCCL INFO Channel 02/0 : 72[0] -> 73[1] [receive] via NET/IB/9/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-3:49:232 [3] NCCL INFO Channel 01/0 : 26[2] -> 27[3] [receive] via NET/IB/12/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-30:49:234 [3] NCCL INFO Channel 02/0 : 243[3] -> 244[4] [send] via NET/IB/13/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-21:48:232 [2] NCCL INFO Channel 00/0 : 169[1] -> 170[2] [receive] via NET/IB/10/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-28:53:235 [7] NCCL INFO Channel 00/0 : 231[7] -> 232[0] [send] via NET/IB/7/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-26:51:236 [5] NCCL INFO Channel 00/0 : 213[5] -> 214[6] [send] via NET/IB/3/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-27:53:237 [7] NCCL INFO Channel 03/0 : 222[6] -> 223[7] [receive] via NET/IB/6/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-11:48:231 [2] NCCL INFO Channel 01/0 : 89[1] -> 90[2] [receive] via NET/IB/11/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-23:51:235 [5] NCCL INFO Channel 00/0 : 189[5] -> 190[6] [send] via NET/IB/3/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-22:48:236 [2] NCCL INFO Channel 01/0 : 177[1] -> 178[2] [receive] via NET/IB/11/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-24:46:237 [0] NCCL INFO Channel 03/0 : 191[7] -> 192[0] [receive] via NET/IB/1/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-14:46:234 [0] NCCL INFO Channel 01/0 : 112[0] -> 113[1] [send] via NET/IB/1/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-13:46:235 [0] NCCL INFO Channel 00/0 : 103[7] -> 104[0] [receive] via NET/IB/0/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-1:52:372 [6] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-7:46:230 [0] NCCL INFO Channel 02/0 : 56[0] -> 57[1] [send] via NET/IB/0/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-8:53:230 [7] NCCL INFO Channel 02/0 : 71[7] -> 72[0] [send] via NET/IB/7/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-10:48:231 [2] NCCL INFO Channel 00/0 : 82[2] -> 83[3] [send] via NET/IB/10/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-2:49:236 [3] NCCL INFO Channel 00/0 : 19[3] -> 20[4] [send] via NET/IB/13/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-29:47:233 [1] NCCL INFO Channel 01/0 : 233[1] -> 234[2] [send] via NET/IB/8/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-6:49:234 [3] NCCL INFO Channel 03/0 : 51[3] -> 52[4] [send] via NET/IB/12/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-5:49:232 [3] NCCL INFO Channel 02/0 : 43[3] -> 44[4] [send] via NET/IB/13/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-4:50:230 [4] NCCL INFO Channel 03/0 : 35[3] -> 36[4] [receive] via NET/IB/15/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-9:49:232 [3] NCCL INFO Channel 01/0 : 74[2] -> 75[3] [receive] via NET/IB/12/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-3:50:236 [4] NCCL INFO Channel 01/0 : 28[4] -> 29[5] [send] via NET/IB/15/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-30:49:234 [3] NCCL INFO Channel 03/0 : 243[3] -> 244[4] [send] via NET/IB/12/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-21:51:235 [5] NCCL INFO Channel 01/0 : 173[5] -> 174[6] [send] via NET/IB/2/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-28:48:231 [2] NCCL INFO Channel 02/0 : 226[2] -> 227[3] [send] via NET/IB/10/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-26:53:232 [7] NCCL INFO Channel 00/0 : 215[7] -> 216[0] [send] via NET/IB/7/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-27:51:234 [5] NCCL INFO Channel 02/0 : 220[4] -> 221[5] [receive] via NET/IB/3/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-11:48:231 [2] NCCL INFO Channel 02/0 : 89[1] -> 90[2] [receive] via NET/IB/10/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-23:53:231 [7] NCCL INFO Channel 01/0 : 191[7] -> 192[0] [send] via NET/IB/6/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-22:46:231 [0] NCCL INFO Channel 02/0 : 175[7] -> 176[0] [receive] via NET/IB/0/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-24:53:235 [7] NCCL INFO Channel 00/0 : 198[6] -> 199[7] [receive] via NET/IB/7/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-14:46:234 [0] NCCL INFO Channel 02/0 : 112[0] -> 113[1] [send] via NET/IB/0/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-13:51:237 [5] NCCL INFO Channel 01/0 : 108[4] -> 109[5] [receive] via NET/IB/2/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-1:54:369 [7] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-7:50:234 [4] NCCL INFO Channel 01/0 : 59[3] -> 60[4] [receive] via NET/IB/15/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-15:46:230 [0] NCCL INFO Channel 00/0 : 119[7] -> 120[0] [receive] via NET/IB/0/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-16:46:230 [0] NCCL INFO Channel 00/0 : 127[7] -> 128[0] [receive] via NET/IB/0/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-8:51:234 [5] NCCL INFO Channel 01/0 : 69[5] -> 70[6] [send] via NET/IB/2/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-10:53:235 [7] NCCL INFO Channel 03/0 : 86[6] -> 87[7] [receive] via NET/IB/6/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-2:49:236 [3] NCCL INFO Channel 01/0 : 19[3] -> 20[4] [send] via NET/IB/12/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-29:47:233 [1] NCCL INFO Channel 02/0 : 233[1] -> 234[2] [send] via NET/IB/9/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-6:48:231 [2] NCCL INFO Channel 01/0 : 50[2] -> 51[3] [send] via NET/IB/11/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-5:48:234 [2] NCCL INFO Channel 00/0 : 42[2] -> 43[3] [send] via NET/IB/10/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-4:49:234 [3] NCCL INFO Channel 00/0 : 35[3] -> 36[4] [send] via NET/IB/13/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-9:47:230 [1] NCCL INFO Channel 03/0 : 72[0] -> 73[1] [receive] via NET/IB/8/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-3:49:232 [3] NCCL INFO Channel 02/0 : 26[2] -> 27[3] [receive] via NET/IB/13/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-30:50:236 [4] NCCL INFO Channel 00/0 : 243[3] -> 244[4] [receive] via NET/IB/14/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-21:48:232 [2] NCCL INFO Channel 01/0 : 169[1] -> 170[2] [receive] via NET/IB/11/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-28:51:236 [5] NCCL INFO Channel 03/0 : 228[4] -> 229[5] [receive] via NET/IB/2/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-26:46:231 [0] NCCL INFO Channel 00/0 : 208[0] -> 209[1] [send] via NET/IB/0/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-27:48:235 [2] NCCL INFO Channel 00/0 : 218[2] -> 219[3] [send] via NET/IB/10/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-11:48:231 [2] NCCL INFO Channel 03/0 : 89[1] -> 90[2] [receive] via NET/IB/11/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-23:51:235 [5] NCCL INFO Channel 01/0 : 189[5] -> 190[6] [send] via NET/IB/2/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-22:53:234 [7] NCCL INFO Channel 02/0 : 182[6] -> 183[7] [receive] via NET/IB/7/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-24:48:230 [2] NCCL INFO Channel 02/0 : 193[1] -> 194[2] [receive] via NET/IB/10/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-14:51:230 [5] NCCL INFO Channel 00/0 : 116[4] -> 117[5] [receive] via NET/IB/3/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-13:48:230 [2] NCCL INFO Channel 03/0 : 105[1] -> 106[2] [receive] via NET/IB/11/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-1:49:370 [3] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-7:54:236 [7] NCCL INFO Channel 03/0 : 63[7] -> 64[0] [send] via NET/IB/6/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-15:46:230 [0] NCCL INFO Channel 01/0 : 119[7] -> 120[0] [receive] via NET/IB/1/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-16:46:230 [0] NCCL INFO Channel 01/0 : 127[7] -> 128[0] [receive] via NET/IB/1/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-17:46:234 [0] NCCL INFO Channel 00/0 : 135[7] -> 136[0] [receive] via NET/IB/0/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-8:48:235 [2] NCCL INFO Channel 03/0 : 66[2] -> 67[3] [send] via NET/IB/11/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-10:51:232 [5] NCCL INFO Channel 03/0 : 84[4] -> 85[5] [receive] via NET/IB/2/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-2:49:236 [3] NCCL INFO Channel 02/0 : 19[3] -> 20[4] [send] via NET/IB/13/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-29:47:233 [1] NCCL INFO Channel 03/0 : 233[1] -> 234[2] [send] via NET/IB/8/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-6:46:236 [0] NCCL INFO Channel 00/0 : 48[0] -> 49[1] [send] via NET/IB/0/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-5:49:232 [3] NCCL INFO Channel 03/0 : 43[3] -> 44[4] [send] via NET/IB/12/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-4:50:230 [4] NCCL INFO Channel 00/0 : 36[4] -> 37[5] [send] via NET/IB/14/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-9:49:232 [3] NCCL INFO Channel 02/0 : 74[2] -> 75[3] [receive] via NET/IB/13/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-3:50:236 [4] NCCL INFO Channel 02/0 : 28[4] -> 29[5] [send] via NET/IB/14/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-30:52:235 [6] NCCL INFO Channel 00/0 : 245[5] -> 246[6] [receive] via NET/IB/4/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-21:51:235 [5] NCCL INFO Channel 02/0 : 173[5] -> 174[6] [send] via NET/IB/3/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-28:53:235 [7] NCCL INFO Channel 01/0 : 231[7] -> 232[0] [send] via NET/IB/6/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-26:53:232 [7] NCCL INFO Channel 01/0 : 215[7] -> 216[0] [send] via NET/IB/6/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-27:53:237 [7] NCCL INFO Channel 00/0 : 223[7] -> 224[0] [send] via NET/IB/7/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-11:48:231 [2] NCCL INFO Channel 00/0 : 90[2] -> 91[3] [send] via NET/IB/10/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-23:53:231 [7] NCCL INFO Channel 02/0 : 191[7] -> 192[0] [send] via NET/IB/7/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-22:51:233 [5] NCCL INFO Channel 02/0 : 180[4] -> 181[5] [receive] via NET/IB/3/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-24:51:234 [5] NCCL INFO Channel 02/0 : 196[4] -> 197[5] [receive] via NET/IB/3/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-14:53:231 [7] NCCL INFO Channel 00/0 : 118[6] -> 119[7] [receive] via NET/IB/7/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-13:46:235 [0] NCCL INFO Channel 01/0 : 103[7] -> 104[0] [receive] via NET/IB/1/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-1:51:373 [5] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-7:46:230 [0] NCCL INFO Channel 03/0 : 56[0] -> 57[1] [send] via NET/IB/1/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-15:46:230 [0] NCCL INFO Channel 02/0 : 119[7] -> 120[0] [receive] via NET/IB/0/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-16:48:231 [2] NCCL INFO Channel 00/0 : 129[1] -> 130[2] [receive] via NET/IB/10/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-17:46:234 [0] NCCL INFO Channel 01/0 : 135[7] -> 136[0] [receive] via NET/IB/1/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-8:53:230 [7] NCCL INFO Channel 03/0 : 71[7] -> 72[0] [send] via NET/IB/6/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-10:48:231 [2] NCCL INFO Channel 01/0 : 82[2] -> 83[3] [send] via NET/IB/11/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-2:49:236 [3] NCCL INFO Channel 03/0 : 19[3] -> 20[4] [send] via NET/IB/12/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-29:46:371 [0] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-6:47:232 [1] NCCL INFO Channel 01/0 : 49[1] -> 50[2] [send] via NET/IB/8/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-5:48:234 [2] NCCL INFO Channel 01/0 : 42[2] -> 43[3] [send] via NET/IB/11/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-4:47:232 [1] NCCL INFO Channel 00/0 : 32[0] -> 33[1] [receive] via NET/IB/9/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-9:47:230 [1] NCCL INFO Channel 00/0 : 73[1] -> 74[2] [send] via NET/IB/9/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-3:49:232 [3] NCCL INFO Channel 03/0 : 26[2] -> 27[3] [receive] via NET/IB/12/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-30:50:236 [4] NCCL INFO Channel 01/0 : 243[3] -> 244[4] [receive] via NET/IB/15/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-21:51:235 [5] NCCL INFO Channel 03/0 : 173[5] -> 174[6] [send] via NET/IB/2/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-28:48:231 [2] NCCL INFO Channel 03/0 : 226[2] -> 227[3] [send] via NET/IB/11/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-26:51:236 [5] NCCL INFO Channel 01/0 : 213[5] -> 214[6] [send] via NET/IB/2/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-27:48:235 [2] NCCL INFO Channel 01/0 : 218[2] -> 219[3] [send] via NET/IB/11/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-11:48:231 [2] NCCL INFO Channel 01/0 : 90[2] -> 91[3] [send] via NET/IB/11/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-23:52:237 [6] NCCL INFO Channel 00/0 : 189[5] -> 190[6] [receive] via NET/IB/4/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-22:48:236 [2] NCCL INFO Channel 02/0 : 177[1] -> 178[2] [receive] via NET/IB/10/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-24:46:237 [0] NCCL INFO Channel 00/0 : 192[0] -> 193[1] [send] via NET/IB/0/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-14:46:234 [0] NCCL INFO Channel 03/0 : 112[0] -> 113[1] [send] via NET/IB/1/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-13:53:232 [7] NCCL INFO Channel 00/0 : 110[6] -> 111[7] [receive] via NET/IB/7/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-1:50:371 [4] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-7:52:231 [6] NCCL INFO Channel 01/0 : 61[5] -> 62[6] [receive] via NET/IB/5/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-15:46:230 [0] NCCL INFO Channel 03/0 : 119[7] -> 120[0] [receive] via NET/IB/1/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-16:46:230 [0] NCCL INFO Channel 02/0 : 127[7] -> 128[0] [receive] via NET/IB/0/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-17:46:234 [0] NCCL INFO Channel 02/0 : 135[7] -> 136[0] [receive] via NET/IB/0/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-8:51:234 [5] NCCL INFO Channel 02/0 : 69[5] -> 70[6] [send] via NET/IB/3/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-10:51:232 [5] NCCL INFO Channel 00/0 : 85[5] -> 86[6] [send] via NET/IB/3/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-2:46:366 [0] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-29:47:372 [1] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-6:48:231 [2] NCCL INFO Channel 02/0 : 50[2] -> 51[3] [send] via NET/IB/10/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-5:48:234 [2] NCCL INFO Channel 02/0 : 42[2] -> 43[3] [send] via NET/IB/10/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-4:49:234 [3] NCCL INFO Channel 01/0 : 35[3] -> 36[4] [send] via NET/IB/12/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-9:49:232 [3] NCCL INFO Channel 03/0 : 74[2] -> 75[3] [receive] via NET/IB/12/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-3:50:236 [4] NCCL INFO Channel 03/0 : 28[4] -> 29[5] [send] via NET/IB/15/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-30:52:235 [6] NCCL INFO Channel 01/0 : 245[5] -> 246[6] [receive] via NET/IB/5/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-21:48:232 [2] NCCL INFO Channel 02/0 : 169[1] -> 170[2] [receive] via NET/IB/10/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-28:53:235 [7] NCCL INFO Channel 02/0 : 231[7] -> 232[0] [send] via NET/IB/7/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-26:46:231 [0] NCCL INFO Channel 01/0 : 208[0] -> 209[1] [send] via NET/IB/1/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-27:51:234 [5] NCCL INFO Channel 03/0 : 220[4] -> 221[5] [receive] via NET/IB/2/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-11:48:231 [2] NCCL INFO Channel 02/0 : 90[2] -> 91[3] [send] via NET/IB/10/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-23:51:235 [5] NCCL INFO Channel 02/0 : 189[5] -> 190[6] [send] via NET/IB/3/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-22:46:231 [0] NCCL INFO Channel 03/0 : 175[7] -> 176[0] [receive] via NET/IB/1/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-24:53:235 [7] NCCL INFO Channel 01/0 : 198[6] -> 199[7] [receive] via NET/IB/6/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-14:53:231 [7] NCCL INFO Channel 01/0 : 118[6] -> 119[7] [receive] via NET/IB/6/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-13:51:237 [5] NCCL INFO Channel 02/0 : 108[4] -> 109[5] [receive] via NET/IB/3/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-1:47:367 [1] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-7:50:234 [4] NCCL INFO Channel 02/0 : 59[3] -> 60[4] [receive] via NET/IB/14/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-15:48:231 [2] NCCL INFO Channel 00/0 : 121[1] -> 122[2] [receive] via NET/IB/10/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-16:48:231 [2] NCCL INFO Channel 01/0 : 129[1] -> 130[2] [receive] via NET/IB/11/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-17:46:234 [0] NCCL INFO Channel 03/0 : 135[7] -> 136[0] [receive] via NET/IB/1/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-18:46:230 [0] NCCL INFO Channel 00/0 : 143[7] -> 144[0] [receive] via NET/IB/0/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-19:53:231 [7] NCCL INFO Channel 00/0 : 158[6] -> 159[7] [receive] via NET/IB/7/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-20:51:230 [5] NCCL INFO Channel 00/0 : 164[4] -> 165[5] [receive] via NET/IB/3/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-8:51:234 [5] NCCL INFO Channel 03/0 : 69[5] -> 70[6] [send] via NET/IB/2/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-10:53:235 [7] NCCL INFO Channel 00/0 : 87[7] -> 88[0] [send] via NET/IB/7/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-2:47:368 [1] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-6:50:237 [4] NCCL INFO Channel 00/0 : 51[3] -> 52[4] [receive] via NET/IB/14/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-5:48:234 [2] NCCL INFO Channel 03/0 : 42[2] -> 43[3] [send] via NET/IB/11/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-4:50:230 [4] NCCL INFO Channel 01/0 : 36[4] -> 37[5] [send] via NET/IB/15/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-9:47:230 [1] NCCL INFO Channel 01/0 : 73[1] -> 74[2] [send] via NET/IB/8/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-3:49:232 [3] NCCL INFO Channel 00/0 : 27[3] -> 28[4] [send] via NET/IB/13/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-30:50:236 [4] NCCL INFO Channel 02/0 : 243[3] -> 244[4] [receive] via NET/IB/14/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-21:48:232 [2] NCCL INFO Channel 03/0 : 169[1] -> 170[2] [receive] via NET/IB/11/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-28:51:236 [5] NCCL INFO Channel 00/0 : 229[5] -> 230[6] [send] via NET/IB/3/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-26:51:236 [5] NCCL INFO Channel 02/0 : 213[5] -> 214[6] [send] via NET/IB/3/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-27:53:237 [7] NCCL INFO Channel 01/0 : 223[7] -> 224[0] [send] via NET/IB/6/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-11:51:234 [5] NCCL INFO Channel 00/0 : 92[4] -> 93[5] [receive] via NET/IB/3/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-23:50:234 [4] NCCL INFO Channel 00/0 : 187[3] -> 188[4] [receive] via NET/IB/14/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-22:53:234 [7] NCCL INFO Channel 03/0 : 182[6] -> 183[7] [receive] via NET/IB/6/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-24:48:230 [2] NCCL INFO Channel 03/0 : 193[1] -> 194[2] [receive] via NET/IB/11/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-14:51:230 [5] NCCL INFO Channel 01/0 : 116[4] -> 117[5] [receive] via NET/IB/2/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-13:48:230 [2] NCCL INFO Channel 00/0 : 106[2] -> 107[3] [send] via NET/IB/10/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-7:52:231 [6] NCCL INFO Channel 02/0 : 61[5] -> 62[6] [receive] via NET/IB/4/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-15:46:230 [0] NCCL INFO Channel 00/0 : 120[0] -> 121[1] [send] via NET/IB/0/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-16:46:230 [0] NCCL INFO Channel 03/0 : 127[7] -> 128[0] [receive] via NET/IB/1/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-17:46:234 [0] NCCL INFO Channel 00/0 : 136[0] -> 137[1] [send] via NET/IB/0/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-18:46:230 [0] NCCL INFO Channel 01/0 : 143[7] -> 144[0] [receive] via NET/IB/1/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-20:53:231 [7] NCCL INFO Channel 00/0 : 166[6] -> 167[7] [receive] via NET/IB/7/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-8:47:232 [1] NCCL INFO Channel 00/0 : 64[0] -> 65[1] [receive] via NET/IB/9/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-10:48:231 [2] NCCL INFO Channel 02/0 : 82[2] -> 83[3] [send] via NET/IB/10/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-2:48:367 [2] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-6:46:236 [0] NCCL INFO Channel 01/0 : 48[0] -> 49[1] [send] via NET/IB/1/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-5:47:235 [1] NCCL INFO Channel 00/0 : 40[0] -> 41[1] [receive] via NET/IB/9/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-4:49:234 [3] NCCL INFO Channel 02/0 : 35[3] -> 36[4] [send] via NET/IB/13/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-9:49:232 [3] NCCL INFO Channel 00/0 : 75[3] -> 76[4] [send] via NET/IB/13/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-3:49:232 [3] NCCL INFO Channel 01/0 : 27[3] -> 28[4] [send] via NET/IB/12/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-30:52:235 [6] NCCL INFO Channel 02/0 : 245[5] -> 246[6] [receive] via NET/IB/4/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-21:48:232 [2] NCCL INFO Channel 00/0 : 170[2] -> 171[3] [send] via NET/IB/10/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-28:53:235 [7] NCCL INFO Channel 03/0 : 231[7] -> 232[0] [send] via NET/IB/6/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-26:53:232 [7] NCCL INFO Channel 02/0 : 215[7] -> 216[0] [send] via NET/IB/7/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-27:48:235 [2] NCCL INFO Channel 02/0 : 218[2] -> 219[3] [send] via NET/IB/10/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-11:48:231 [2] NCCL INFO Channel 03/0 : 90[2] -> 91[3] [send] via NET/IB/11/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-23:53:231 [7] NCCL INFO Channel 03/0 : 191[7] -> 192[0] [send] via NET/IB/6/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-22:51:233 [5] NCCL INFO Channel 03/0 : 180[4] -> 181[5] [receive] via NET/IB/2/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-24:46:237 [0] NCCL INFO Channel 01/0 : 192[0] -> 193[1] [send] via NET/IB/1/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-14:51:230 [5] NCCL INFO Channel 02/0 : 116[4] -> 117[5] [receive] via NET/IB/3/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-13:46:235 [0] NCCL INFO Channel 02/0 : 103[7] -> 104[0] [receive] via NET/IB/0/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-7:50:234 [4] NCCL INFO Channel 03/0 : 59[3] -> 60[4] [receive] via NET/IB/15/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-15:48:231 [2] NCCL INFO Channel 01/0 : 121[1] -> 122[2] [receive] via NET/IB/11/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-16:51:235 [5] NCCL INFO Channel 00/0 : 132[4] -> 133[5] [receive] via NET/IB/3/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-17:53:236 [7] NCCL INFO Channel 00/0 : 142[6] -> 143[7] [receive] via NET/IB/7/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-18:46:230 [0] NCCL INFO Channel 02/0 : 143[7] -> 144[0] [receive] via NET/IB/0/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-20:46:233 [0] NCCL INFO Channel 00/0 : 159[7] -> 160[0] [receive] via NET/IB/0/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-19:53:231 [7] NCCL INFO Channel 01/0 : 158[6] -> 159[7] [receive] via NET/IB/6/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-8:49:237 [3] NCCL INFO Channel 00/0 : 66[2] -> 67[3] [receive] via NET/IB/13/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-10:53:235 [7] NCCL INFO Channel 01/0 : 87[7] -> 88[0] [send] via NET/IB/6/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-2:50:372 [4] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-6:52:233 [6] NCCL INFO Channel 00/0 : 53[5] -> 54[6] [receive] via NET/IB/4/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-5:47:235 [1] NCCL INFO Channel 01/0 : 40[0] -> 41[1] [receive] via NET/IB/8/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-4:47:232 [1] NCCL INFO Channel 01/0 : 32[0] -> 33[1] [receive] via NET/IB/8/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-9:47:230 [1] NCCL INFO Channel 02/0 : 73[1] -> 74[2] [send] via NET/IB/9/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-3:49:232 [3] NCCL INFO Channel 02/0 : 27[3] -> 28[4] [send] via NET/IB/13/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-30:50:236 [4] NCCL INFO Channel 03/0 : 243[3] -> 244[4] [receive] via NET/IB/15/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-21:48:232 [2] NCCL INFO Channel 01/0 : 170[2] -> 171[3] [send] via NET/IB/11/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-28:51:236 [5] NCCL INFO Channel 01/0 : 229[5] -> 230[6] [send] via NET/IB/2/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-26:46:231 [0] NCCL INFO Channel 02/0 : 208[0] -> 209[1] [send] via NET/IB/0/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-27:51:234 [5] NCCL INFO Channel 00/0 : 221[5] -> 222[6] [send] via NET/IB/3/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-11:51:234 [5] NCCL INFO Channel 01/0 : 92[4] -> 93[5] [receive] via NET/IB/2/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-23:52:237 [6] NCCL INFO Channel 01/0 : 189[5] -> 190[6] [receive] via NET/IB/5/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-22:46:231 [0] NCCL INFO Channel 00/0 : 176[0] -> 177[1] [send] via NET/IB/0/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-24:51:234 [5] NCCL INFO Channel 03/0 : 196[4] -> 197[5] [receive] via NET/IB/2/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-14:53:231 [7] NCCL INFO Channel 02/0 : 118[6] -> 119[7] [receive] via NET/IB/7/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-13:48:230 [2] NCCL INFO Channel 01/0 : 106[2] -> 107[3] [send] via NET/IB/11/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-7:48:237 [2] NCCL INFO Channel 00/0 : 57[1] -> 58[2] [receive] via NET/IB/10/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-15:46:230 [0] NCCL INFO Channel 01/0 : 120[0] -> 121[1] [send] via NET/IB/1/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-16:48:231 [2] NCCL INFO Channel 02/0 : 129[1] -> 130[2] [receive] via NET/IB/10/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-17:46:234 [0] NCCL INFO Channel 01/0 : 136[0] -> 137[1] [send] via NET/IB/1/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-18:53:231 [7] NCCL INFO Channel 00/0 : 150[6] -> 151[7] [receive] via NET/IB/7/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-20:51:230 [5] NCCL INFO Channel 01/0 : 164[4] -> 165[5] [receive] via NET/IB/2/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-19:53:231 [7] NCCL INFO Channel 02/0 : 158[6] -> 159[7] [receive] via NET/IB/7/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-12:47:230 [1] NCCL INFO Channel 00/0 : 96[0] -> 97[1] [receive] via NET/IB/9/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-8:52:236 [6] NCCL INFO Channel 00/0 : 69[5] -> 70[6] [receive] via NET/IB/4/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-10:51:232 [5] NCCL INFO Channel 01/0 : 85[5] -> 86[6] [send] via NET/IB/2/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-2:49:373 [3] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-6:47:232 [1] NCCL INFO Channel 02/0 : 49[1] -> 50[2] [send] via NET/IB/9/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-5:47:235 [1] NCCL INFO Channel 02/0 : 40[0] -> 41[1] [receive] via NET/IB/9/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-4:50:230 [4] NCCL INFO Channel 02/0 : 36[4] -> 37[5] [send] via NET/IB/14/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-9:51:236 [5] NCCL INFO Channel 00/0 : 76[4] -> 77[5] [receive] via NET/IB/3/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-3:49:232 [3] NCCL INFO Channel 03/0 : 27[3] -> 28[4] [send] via NET/IB/12/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-30:52:235 [6] NCCL INFO Channel 03/0 : 245[5] -> 246[6] [receive] via NET/IB/5/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-21:47:234 [1] NCCL INFO Channel 00/0 : 168[0] -> 169[1] [receive] via NET/IB/9/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-28:51:236 [5] NCCL INFO Channel 02/0 : 229[5] -> 230[6] [send] via NET/IB/3/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-26:51:236 [5] NCCL INFO Channel 03/0 : 213[5] -> 214[6] [send] via NET/IB/2/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-27:53:237 [7] NCCL INFO Channel 02/0 : 223[7] -> 224[0] [send] via NET/IB/7/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-11:53:236 [7] NCCL INFO Channel 00/0 : 94[6] -> 95[7] [receive] via NET/IB/7/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-23:51:235 [5] NCCL INFO Channel 03/0 : 189[5] -> 190[6] [send] via NET/IB/2/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-22:48:236 [2] NCCL INFO Channel 03/0 : 177[1] -> 178[2] [receive] via NET/IB/11/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-24:53:235 [7] NCCL INFO Channel 02/0 : 198[6] -> 199[7] [receive] via NET/IB/7/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-14:51:230 [5] NCCL INFO Channel 03/0 : 116[4] -> 117[5] [receive] via NET/IB/2/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-13:51:237 [5] NCCL INFO Channel 03/0 : 108[4] -> 109[5] [receive] via NET/IB/2/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-7:52:231 [6] NCCL INFO Channel 03/0 : 61[5] -> 62[6] [receive] via NET/IB/5/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-15:53:236 [7] NCCL INFO Channel 00/0 : 126[6] -> 127[7] [receive] via NET/IB/7/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-16:46:230 [0] NCCL INFO Channel 00/0 : 128[0] -> 129[1] [send] via NET/IB/0/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-17:51:232 [5] NCCL INFO Channel 00/0 : 140[4] -> 141[5] [receive] via NET/IB/3/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-18:46:230 [0] NCCL INFO Channel 03/0 : 143[7] -> 144[0] [receive] via NET/IB/1/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-20:53:231 [7] NCCL INFO Channel 01/0 : 166[6] -> 167[7] [receive] via NET/IB/6/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-19:53:231 [7] NCCL INFO Channel 03/0 : 158[6] -> 159[7] [receive] via NET/IB/6/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-12:47:230 [1] NCCL INFO Channel 01/0 : 96[0] -> 97[1] [receive] via NET/IB/8/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-8:50:233 [4] NCCL INFO Channel 00/0 : 67[3] -> 68[4] [receive] via NET/IB/14/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-10:48:231 [2] NCCL INFO Channel 03/0 : 82[2] -> 83[3] [send] via NET/IB/11/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-6:48:231 [2] NCCL INFO Channel 03/0 : 50[2] -> 51[3] [send] via NET/IB/11/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-5:47:235 [1] NCCL INFO Channel 03/0 : 40[0] -> 41[1] [receive] via NET/IB/8/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-4:49:234 [3] NCCL INFO Channel 03/0 : 35[3] -> 36[4] [send] via NET/IB/12/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-9:53:235 [7] NCCL INFO Channel 00/0 : 78[6] -> 79[7] [receive] via NET/IB/7/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-3:52:369 [6] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-30:50:236 [4] NCCL INFO Channel 00/0 : 244[4] -> 245[5] [send] via NET/IB/14/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-21:48:232 [2] NCCL INFO Channel 02/0 : 170[2] -> 171[3] [send] via NET/IB/10/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-28:51:236 [5] NCCL INFO Channel 03/0 : 229[5] -> 230[6] [send] via NET/IB/2/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-26:46:231 [0] NCCL INFO Channel 03/0 : 208[0] -> 209[1] [send] via NET/IB/1/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-27:48:235 [2] NCCL INFO Channel 03/0 : 218[2] -> 219[3] [send] via NET/IB/11/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-11:52:233 [6] NCCL INFO Channel 00/0 : 93[5] -> 94[6] [receive] via NET/IB/4/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-23:50:234 [4] NCCL INFO Channel 01/0 : 187[3] -> 188[4] [receive] via NET/IB/15/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-22:53:234 [7] NCCL INFO Channel 00/0 : 183[7] -> 184[0] [send] via NET/IB/7/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-24:46:237 [0] NCCL INFO Channel 02/0 : 192[0] -> 193[1] [send] via NET/IB/0/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-14:53:231 [7] NCCL INFO Channel 03/0 : 118[6] -> 119[7] [receive] via NET/IB/6/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-13:53:232 [7] NCCL INFO Channel 01/0 : 110[6] -> 111[7] [receive] via NET/IB/6/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-7:50:234 [4] NCCL INFO Channel 00/0 : 60[4] -> 61[5] [send] via NET/IB/14/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-15:51:233 [5] NCCL INFO Channel 00/0 : 124[4] -> 125[5] [receive] via NET/IB/3/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-16:51:235 [5] NCCL INFO Channel 01/0 : 132[4] -> 133[5] [receive] via NET/IB/2/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-17:48:230 [2] NCCL INFO Channel 00/0 : 137[1] -> 138[2] [receive] via NET/IB/10/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-18:53:231 [7] NCCL INFO Channel 01/0 : 150[6] -> 151[7] [receive] via NET/IB/6/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-20:46:233 [0] NCCL INFO Channel 01/0 : 159[7] -> 160[0] [receive] via NET/IB/1/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-19:46:230 [0] NCCL INFO Channel 00/0 : 151[7] -> 152[0] [receive] via NET/IB/0/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-12:47:230 [1] NCCL INFO Channel 02/0 : 96[0] -> 97[1] [receive] via NET/IB/9/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-8:52:236 [6] NCCL INFO Channel 01/0 : 69[5] -> 70[6] [receive] via NET/IB/5/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-10:50:234 [4] NCCL INFO Channel 00/0 : 83[3] -> 84[4] [receive] via NET/IB/14/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-6:46:236 [0] NCCL INFO Channel 02/0 : 48[0] -> 49[1] [send] via NET/IB/0/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-5:47:235 [1] NCCL INFO Channel 00/0 : 41[1] -> 42[2] [send] via NET/IB/9/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-4:50:230 [4] NCCL INFO Channel 03/0 : 36[4] -> 37[5] [send] via NET/IB/15/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-9:47:230 [1] NCCL INFO Channel 03/0 : 73[1] -> 74[2] [send] via NET/IB/8/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-3:51:370 [5] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-30:46:372 [0] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-21:50:237 [4] NCCL INFO Channel 00/0 : 171[3] -> 172[4] [receive] via NET/IB/14/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-28:52:232 [6] NCCL INFO Channel 00/0 : 229[5] -> 230[6] [receive] via NET/IB/4/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-26:53:232 [7] NCCL INFO Channel 03/0 : 215[7] -> 216[0] [send] via NET/IB/6/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-27:51:234 [5] NCCL INFO Channel 01/0 : 221[5] -> 222[6] [send] via NET/IB/2/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-11:51:234 [5] NCCL INFO Channel 02/0 : 92[4] -> 93[5] [receive] via NET/IB/3/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-23:48:232 [2] NCCL INFO Channel 00/0 : 185[1] -> 186[2] [receive] via NET/IB/10/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-24:48:230 [2] NCCL INFO Channel 00/0 : 194[2] -> 195[3] [send] via NET/IB/10/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-14:51:230 [5] NCCL INFO Channel 00/0 : 117[5] -> 118[6] [send] via NET/IB/3/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-13:48:230 [2] NCCL INFO Channel 02/0 : 106[2] -> 107[3] [send] via NET/IB/10/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-7:52:231 [6] NCCL INFO Channel 00/0 : 62[6] -> 63[7] [send] via NET/IB/4/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-15:46:230 [0] NCCL INFO Channel 02/0 : 120[0] -> 121[1] [send] via NET/IB/0/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-16:48:231 [2] NCCL INFO Channel 03/0 : 129[1] -> 130[2] [receive] via NET/IB/11/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-17:53:236 [7] NCCL INFO Channel 01/0 : 142[6] -> 143[7] [receive] via NET/IB/6/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-18:51:235 [5] NCCL INFO Channel 00/0 : 148[4] -> 149[5] [receive] via NET/IB/3/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-20:51:230 [5] NCCL INFO Channel 02/0 : 164[4] -> 165[5] [receive] via NET/IB/3/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-19:53:231 [7] NCCL INFO Channel 00/0 : 159[7] -> 160[0] [send] via NET/IB/7/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-12:47:230 [1] NCCL INFO Channel 03/0 : 96[0] -> 97[1] [receive] via NET/IB/8/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-29:49:236 [3] NCCL INFO Channel 00/0 : 234[2] -> 235[3] [receive] via NET/IB/13/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-8:50:233 [4] NCCL INFO Channel 01/0 : 67[3] -> 68[4] [receive] via NET/IB/15/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-10:51:232 [5] NCCL INFO Channel 02/0 : 85[5] -> 86[6] [send] via NET/IB/3/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-6:50:237 [4] NCCL INFO Channel 01/0 : 51[3] -> 52[4] [receive] via NET/IB/15/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-5:47:235 [1] NCCL INFO Channel 01/0 : 41[1] -> 42[2] [send] via NET/IB/8/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-4:47:232 [1] NCCL INFO Channel 02/0 : 32[0] -> 33[1] [receive] via NET/IB/9/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-9:49:232 [3] NCCL INFO Channel 01/0 : 75[3] -> 76[4] [send] via NET/IB/12/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-3:48:368 [2] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-30:52:235 [6] NCCL INFO Channel 00/0 : 246[6] -> 247[7] [send] via NET/IB/4/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-21:48:232 [2] NCCL INFO Channel 03/0 : 170[2] -> 171[3] [send] via NET/IB/11/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-28:50:234 [4] NCCL INFO Channel 00/0 : 227[3] -> 228[4] [receive] via NET/IB/14/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-26:52:237 [6] NCCL INFO Channel 00/0 : 213[5] -> 214[6] [receive] via NET/IB/4/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-27:53:237 [7] NCCL INFO Channel 03/0 : 223[7] -> 224[0] [send] via NET/IB/6/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-11:49:232 [3] NCCL INFO Channel 00/0 : 90[2] -> 91[3] [receive] via NET/IB/13/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-23:52:237 [6] NCCL INFO Channel 02/0 : 189[5] -> 190[6] [receive] via NET/IB/4/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-14:53:231 [7] NCCL INFO Channel 00/0 : 119[7] -> 120[0] [send] via NET/IB/7/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-13:46:235 [0] NCCL INFO Channel 03/0 : 103[7] -> 104[0] [receive] via NET/IB/1/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-7:50:234 [4] NCCL INFO Channel 01/0 : 60[4] -> 61[5] [send] via NET/IB/15/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-15:48:231 [2] NCCL INFO Channel 02/0 : 121[1] -> 122[2] [receive] via NET/IB/10/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-16:46:230 [0] NCCL INFO Channel 01/0 : 128[0] -> 129[1] [send] via NET/IB/1/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-17:46:234 [0] NCCL INFO Channel 02/0 : 136[0] -> 137[1] [send] via NET/IB/0/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-18:46:230 [0] NCCL INFO Channel 00/0 : 144[0] -> 145[1] [send] via NET/IB/0/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-20:53:231 [7] NCCL INFO Channel 02/0 : 166[6] -> 167[7] [receive] via NET/IB/7/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-19:53:231 [7] NCCL INFO Channel 01/0 : 159[7] -> 160[0] [send] via NET/IB/6/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-12:47:230 [1] NCCL INFO Channel 00/0 : 97[1] -> 98[2] [send] via NET/IB/9/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-29:49:236 [3] NCCL INFO Channel 01/0 : 234[2] -> 235[3] [receive] via NET/IB/12/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-22:51:233 [5] NCCL INFO Channel 00/0 : 181[5] -> 182[6] [send] via NET/IB/3/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-25:53:231 [7] NCCL INFO Channel 00/0 : 206[6] -> 207[7] [receive] via NET/IB/7/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-8:47:232 [1] NCCL INFO Channel 01/0 : 64[0] -> 65[1] [receive] via NET/IB/8/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-10:53:235 [7] NCCL INFO Channel 02/0 : 87[7] -> 88[0] [send] via NET/IB/7/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-6:47:232 [1] NCCL INFO Channel 03/0 : 49[1] -> 50[2] [send] via NET/IB/8/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-5:47:235 [1] NCCL INFO Channel 02/0 : 41[1] -> 42[2] [send] via NET/IB/9/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-4:51:236 [5] NCCL INFO Channel 00/0 : 36[4] -> 37[5] [receive] via NET/IB/3/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-9:49:232 [3] NCCL INFO Channel 02/0 : 75[3] -> 76[4] [send] via NET/IB/13/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-3:50:371 [4] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-30:50:236 [4] NCCL INFO Channel 01/0 : 244[4] -> 245[5] [send] via NET/IB/15/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-21:47:234 [1] NCCL INFO Channel 01/0 : 168[0] -> 169[1] [receive] via NET/IB/8/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-28:52:232 [6] NCCL INFO Channel 01/0 : 229[5] -> 230[6] [receive] via NET/IB/5/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-26:52:237 [6] NCCL INFO Channel 01/0 : 213[5] -> 214[6] [receive] via NET/IB/5/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-27:51:234 [5] NCCL INFO Channel 02/0 : 221[5] -> 222[6] [send] via NET/IB/3/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-11:50:237 [4] NCCL INFO Channel 00/0 : 91[3] -> 92[4] [receive] via NET/IB/14/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-23:50:234 [4] NCCL INFO Channel 02/0 : 187[3] -> 188[4] [receive] via NET/IB/14/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-13:51:237 [5] NCCL INFO Channel 00/0 : 109[5] -> 110[6] [send] via NET/IB/3/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-7:52:231 [6] NCCL INFO Channel 01/0 : 62[6] -> 63[7] [send] via NET/IB/5/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-15:46:230 [0] NCCL INFO Channel 03/0 : 120[0] -> 121[1] [send] via NET/IB/1/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-16:51:235 [5] NCCL INFO Channel 02/0 : 132[4] -> 133[5] [receive] via NET/IB/3/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-17:51:232 [5] NCCL INFO Channel 01/0 : 140[4] -> 141[5] [receive] via NET/IB/2/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-18:53:231 [7] NCCL INFO Channel 02/0 : 150[6] -> 151[7] [receive] via NET/IB/7/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-20:46:233 [0] NCCL INFO Channel 02/0 : 159[7] -> 160[0] [receive] via NET/IB/0/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-19:46:230 [0] NCCL INFO Channel 01/0 : 151[7] -> 152[0] [receive] via NET/IB/1/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-12:47:230 [1] NCCL INFO Channel 01/0 : 97[1] -> 98[2] [send] via NET/IB/8/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-29:49:236 [3] NCCL INFO Channel 02/0 : 234[2] -> 235[3] [receive] via NET/IB/13/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-22:46:231 [0] NCCL INFO Channel 01/0 : 176[0] -> 177[1] [send] via NET/IB/1/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-24:47:231 [1] NCCL INFO Channel 00/0 : 192[0] -> 193[1] [receive] via NET/IB/9/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-8:49:237 [3] NCCL INFO Channel 01/0 : 66[2] -> 67[3] [receive] via NET/IB/12/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-10:51:232 [5] NCCL INFO Channel 03/0 : 85[5] -> 86[6] [send] via NET/IB/2/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-6:52:233 [6] NCCL INFO Channel 01/0 : 53[5] -> 54[6] [receive] via NET/IB/5/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-5:47:235 [1] NCCL INFO Channel 03/0 : 41[1] -> 42[2] [send] via NET/IB/8/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-4:47:232 [1] NCCL INFO Channel 03/0 : 32[0] -> 33[1] [receive] via NET/IB/8/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-9:51:236 [5] NCCL INFO Channel 01/0 : 76[4] -> 77[5] [receive] via NET/IB/2/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-3:49:372 [3] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-30:50:236 [4] NCCL INFO Channel 02/0 : 244[4] -> 245[5] [send] via NET/IB/14/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-21:50:237 [4] NCCL INFO Channel 01/0 : 171[3] -> 172[4] [receive] via NET/IB/15/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-28:50:234 [4] NCCL INFO Channel 01/0 : 227[3] -> 228[4] [receive] via NET/IB/15/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-26:50:234 [4] NCCL INFO Channel 00/0 : 211[3] -> 212[4] [receive] via NET/IB/14/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-27:51:234 [5] NCCL INFO Channel 03/0 : 221[5] -> 222[6] [send] via NET/IB/2/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-11:53:236 [7] NCCL INFO Channel 01/0 : 94[6] -> 95[7] [receive] via NET/IB/6/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-23:48:232 [2] NCCL INFO Channel 01/0 : 185[1] -> 186[2] [receive] via NET/IB/11/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-7:50:234 [4] NCCL INFO Channel 02/0 : 60[4] -> 61[5] [send] via NET/IB/14/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-15:53:236 [7] NCCL INFO Channel 01/0 : 126[6] -> 127[7] [receive] via NET/IB/6/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-16:53:236 [7] NCCL INFO Channel 00/0 : 134[6] -> 135[7] [receive] via NET/IB/7/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-17:53:236 [7] NCCL INFO Channel 02/0 : 142[6] -> 143[7] [receive] via NET/IB/7/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-18:51:235 [5] NCCL INFO Channel 01/0 : 148[4] -> 149[5] [receive] via NET/IB/2/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-20:51:230 [5] NCCL INFO Channel 03/0 : 164[4] -> 165[5] [receive] via NET/IB/2/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-19:53:231 [7] NCCL INFO Channel 02/0 : 159[7] -> 160[0] [send] via NET/IB/7/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-12:47:230 [1] NCCL INFO Channel 02/0 : 97[1] -> 98[2] [send] via NET/IB/9/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-29:49:236 [3] NCCL INFO Channel 03/0 : 234[2] -> 235[3] [receive] via NET/IB/12/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-22:48:236 [2] NCCL INFO Channel 00/0 : 178[2] -> 179[3] [send] via NET/IB/10/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-24:51:234 [5] NCCL INFO Channel 00/0 : 197[5] -> 198[6] [send] via NET/IB/3/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-14:51:230 [5] NCCL INFO Channel 01/0 : 117[5] -> 118[6] [send] via NET/IB/2/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-8:52:236 [6] NCCL INFO Channel 02/0 : 69[5] -> 70[6] [receive] via NET/IB/4/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-10:53:235 [7] NCCL INFO Channel 03/0 : 87[7] -> 88[0] [send] via NET/IB/6/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-6:46:236 [0] NCCL INFO Channel 03/0 : 48[0] -> 49[1] [send] via NET/IB/1/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-5:50:231 [4] NCCL INFO Channel 00/0 : 43[3] -> 44[4] [receive] via NET/IB/14/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-4:47:232 [1] NCCL INFO Channel 00/0 : 33[1] -> 34[2] [send] via NET/IB/9/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-9:53:235 [7] NCCL INFO Channel 01/0 : 78[6] -> 79[7] [receive] via NET/IB/6/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-3:53:373 [7] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-30:52:235 [6] NCCL INFO Channel 01/0 : 246[6] -> 247[7] [send] via NET/IB/5/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-21:47:234 [1] NCCL INFO Channel 02/0 : 168[0] -> 169[1] [receive] via NET/IB/9/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-28:47:230 [1] NCCL INFO Channel 00/0 : 224[0] -> 225[1] [receive] via NET/IB/9/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-26:52:237 [6] NCCL INFO Channel 02/0 : 213[5] -> 214[6] [receive] via NET/IB/4/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-27:49:233 [3] NCCL INFO Channel 00/0 : 218[2] -> 219[3] [receive] via NET/IB/13/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-11:52:233 [6] NCCL INFO Channel 01/0 : 93[5] -> 94[6] [receive] via NET/IB/5/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-23:52:237 [6] NCCL INFO Channel 03/0 : 189[5] -> 190[6] [receive] via NET/IB/5/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-7:48:237 [2] NCCL INFO Channel 01/0 : 57[1] -> 58[2] [receive] via NET/IB/11/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-15:51:233 [5] NCCL INFO Channel 01/0 : 124[4] -> 125[5] [receive] via NET/IB/2/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-16:48:231 [2] NCCL INFO Channel 00/0 : 130[2] -> 131[3] [send] via NET/IB/10/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-17:46:234 [0] NCCL INFO Channel 03/0 : 136[0] -> 137[1] [send] via NET/IB/1/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-18:46:230 [0] NCCL INFO Channel 01/0 : 144[0] -> 145[1] [send] via NET/IB/1/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-20:53:231 [7] NCCL INFO Channel 03/0 : 166[6] -> 167[7] [receive] via NET/IB/6/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-19:46:230 [0] NCCL INFO Channel 02/0 : 151[7] -> 152[0] [receive] via NET/IB/0/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-12:47:230 [1] NCCL INFO Channel 03/0 : 97[1] -> 98[2] [send] via NET/IB/8/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-29:49:236 [3] NCCL INFO Channel 00/0 : 235[3] -> 236[4] [send] via NET/IB/13/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-22:53:234 [7] NCCL INFO Channel 01/0 : 183[7] -> 184[0] [send] via NET/IB/6/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-24:53:235 [7] NCCL INFO Channel 03/0 : 198[6] -> 199[7] [receive] via NET/IB/6/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-14:53:231 [7] NCCL INFO Channel 01/0 : 119[7] -> 120[0] [send] via NET/IB/6/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-13:53:232 [7] NCCL INFO Channel 02/0 : 110[6] -> 111[7] [receive] via NET/IB/7/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-25:51:232 [5] NCCL INFO Channel 00/0 : 204[4] -> 205[5] [receive] via NET/IB/3/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-8:47:232 [1] NCCL INFO Channel 02/0 : 64[0] -> 65[1] [receive] via NET/IB/9/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-10:50:234 [4] NCCL INFO Channel 01/0 : 83[3] -> 84[4] [receive] via NET/IB/15/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-6:50:237 [4] NCCL INFO Channel 02/0 : 51[3] -> 52[4] [receive] via NET/IB/14/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-5:50:231 [4] NCCL INFO Channel 01/0 : 43[3] -> 44[4] [receive] via NET/IB/15/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-4:51:236 [5] NCCL INFO Channel 01/0 : 36[4] -> 37[5] [receive] via NET/IB/2/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-9:49:232 [3] NCCL INFO Channel 03/0 : 75[3] -> 76[4] [send] via NET/IB/12/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-30:48:371 [2] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-21:50:237 [4] NCCL INFO Channel 02/0 : 171[3] -> 172[4] [receive] via NET/IB/14/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-28:52:232 [6] NCCL INFO Channel 02/0 : 229[5] -> 230[6] [receive] via NET/IB/4/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-26:50:234 [4] NCCL INFO Channel 01/0 : 211[3] -> 212[4] [receive] via NET/IB/15/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-27:49:233 [3] NCCL INFO Channel 01/0 : 218[2] -> 219[3] [receive] via NET/IB/12/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-11:51:234 [5] NCCL INFO Channel 03/0 : 92[4] -> 93[5] [receive] via NET/IB/2/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-23:50:234 [4] NCCL INFO Channel 03/0 : 187[3] -> 188[4] [receive] via NET/IB/15/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-7:49:235 [3] NCCL INFO Channel 00/0 : 58[2] -> 59[3] [receive] via NET/IB/13/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-15:48:231 [2] NCCL INFO Channel 03/0 : 121[1] -> 122[2] [receive] via NET/IB/11/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-16:46:230 [0] NCCL INFO Channel 02/0 : 128[0] -> 129[1] [send] via NET/IB/0/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-17:48:230 [2] NCCL INFO Channel 01/0 : 137[1] -> 138[2] [receive] via NET/IB/11/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-18:53:231 [7] NCCL INFO Channel 03/0 : 150[6] -> 151[7] [receive] via NET/IB/6/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-20:46:233 [0] NCCL INFO Channel 03/0 : 159[7] -> 160[0] [receive] via NET/IB/1/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-19:53:231 [7] NCCL INFO Channel 03/0 : 159[7] -> 160[0] [send] via NET/IB/6/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-12:46:237 [0] NCCL INFO Channel 00/0 : 95[7] -> 96[0] [receive] via NET/IB/0/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-29:49:236 [3] NCCL INFO Channel 01/0 : 235[3] -> 236[4] [send] via NET/IB/12/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-22:51:233 [5] NCCL INFO Channel 01/0 : 181[5] -> 182[6] [send] via NET/IB/2/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-24:46:237 [0] NCCL INFO Channel 03/0 : 192[0] -> 193[1] [send] via NET/IB/1/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-14:51:230 [5] NCCL INFO Channel 02/0 : 117[5] -> 118[6] [send] via NET/IB/3/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-13:48:230 [2] NCCL INFO Channel 03/0 : 106[2] -> 107[3] [send] via NET/IB/11/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-25:53:231 [7] NCCL INFO Channel 01/0 : 206[6] -> 207[7] [receive] via NET/IB/6/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-8:49:237 [3] NCCL INFO Channel 02/0 : 66[2] -> 67[3] [receive] via NET/IB/13/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-10:49:237 [3] NCCL INFO Channel 00/0 : 82[2] -> 83[3] [receive] via NET/IB/13/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-6:52:233 [6] NCCL INFO Channel 02/0 : 53[5] -> 54[6] [receive] via NET/IB/4/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-5:50:231 [4] NCCL INFO Channel 02/0 : 43[3] -> 44[4] [receive] via NET/IB/14/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-4:47:232 [1] NCCL INFO Channel 01/0 : 33[1] -> 34[2] [send] via NET/IB/8/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-9:51:236 [5] NCCL INFO Channel 02/0 : 76[4] -> 77[5] [receive] via NET/IB/3/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-30:50:236 [4] NCCL INFO Channel 03/0 : 244[4] -> 245[5] [send] via NET/IB/15/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-21:47:234 [1] NCCL INFO Channel 03/0 : 168[0] -> 169[1] [receive] via NET/IB/8/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-28:50:234 [4] NCCL INFO Channel 02/0 : 227[3] -> 228[4] [receive] via NET/IB/14/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-26:52:237 [6] NCCL INFO Channel 03/0 : 213[5] -> 214[6] [receive] via NET/IB/5/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-27:49:233 [3] NCCL INFO Channel 02/0 : 218[2] -> 219[3] [receive] via NET/IB/13/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-11:49:232 [3] NCCL INFO Channel 01/0 : 90[2] -> 91[3] [receive] via NET/IB/12/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-23:52:237 [6] NCCL INFO Channel 00/0 : 190[6] -> 191[7] [send] via NET/IB/4/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-7:52:231 [6] NCCL INFO Channel 02/0 : 62[6] -> 63[7] [send] via NET/IB/4/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-15:53:236 [7] NCCL INFO Channel 02/0 : 126[6] -> 127[7] [receive] via NET/IB/7/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-16:46:230 [0] NCCL INFO Channel 03/0 : 128[0] -> 129[1] [send] via NET/IB/1/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-17:53:236 [7] NCCL INFO Channel 03/0 : 142[6] -> 143[7] [receive] via NET/IB/6/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-18:51:235 [5] NCCL INFO Channel 02/0 : 148[4] -> 149[5] [receive] via NET/IB/3/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-20:51:230 [5] NCCL INFO Channel 00/0 : 165[5] -> 166[6] [send] via NET/IB/3/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-19:46:230 [0] NCCL INFO Channel 03/0 : 151[7] -> 152[0] [receive] via NET/IB/1/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-12:48:236 [2] NCCL INFO Channel 00/0 : 97[1] -> 98[2] [receive] via NET/IB/10/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-29:49:236 [3] NCCL INFO Channel 02/0 : 235[3] -> 236[4] [send] via NET/IB/13/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-22:46:231 [0] NCCL INFO Channel 02/0 : 176[0] -> 177[1] [send] via NET/IB/0/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-24:48:230 [2] NCCL INFO Channel 01/0 : 194[2] -> 195[3] [send] via NET/IB/11/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-14:53:231 [7] NCCL INFO Channel 02/0 : 119[7] -> 120[0] [send] via NET/IB/7/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-13:46:235 [0] NCCL INFO Channel 00/0 : 104[0] -> 105[1] [send] via NET/IB/0/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-25:51:232 [5] NCCL INFO Channel 01/0 : 204[4] -> 205[5] [receive] via NET/IB/2/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-8:50:233 [4] NCCL INFO Channel 02/0 : 67[3] -> 68[4] [receive] via NET/IB/14/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-10:50:234 [4] NCCL INFO Channel 02/0 : 83[3] -> 84[4] [receive] via NET/IB/14/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-6:50:237 [4] NCCL INFO Channel 03/0 : 51[3] -> 52[4] [receive] via NET/IB/15/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-5:46:366 [0] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-4:47:232 [1] NCCL INFO Channel 02/0 : 33[1] -> 34[2] [send] via NET/IB/9/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-9:53:235 [7] NCCL INFO Channel 02/0 : 78[6] -> 79[7] [receive] via NET/IB/7/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-30:52:235 [6] NCCL INFO Channel 02/0 : 246[6] -> 247[7] [send] via NET/IB/4/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-21:50:237 [4] NCCL INFO Channel 03/0 : 171[3] -> 172[4] [receive] via NET/IB/15/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-28:47:230 [1] NCCL INFO Channel 01/0 : 224[0] -> 225[1] [receive] via NET/IB/8/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-26:50:234 [4] NCCL INFO Channel 02/0 : 211[3] -> 212[4] [receive] via NET/IB/14/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-27:49:233 [3] NCCL INFO Channel 03/0 : 218[2] -> 219[3] [receive] via NET/IB/12/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-11:50:237 [4] NCCL INFO Channel 01/0 : 91[3] -> 92[4] [receive] via NET/IB/15/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-23:48:232 [2] NCCL INFO Channel 02/0 : 185[1] -> 186[2] [receive] via NET/IB/10/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-7:50:234 [4] NCCL INFO Channel 03/0 : 60[4] -> 61[5] [send] via NET/IB/15/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-15:51:233 [5] NCCL INFO Channel 02/0 : 124[4] -> 125[5] [receive] via NET/IB/3/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-16:51:235 [5] NCCL INFO Channel 03/0 : 132[4] -> 133[5] [receive] via NET/IB/2/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-17:51:232 [5] NCCL INFO Channel 02/0 : 140[4] -> 141[5] [receive] via NET/IB/3/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-18:46:230 [0] NCCL INFO Channel 02/0 : 144[0] -> 145[1] [send] via NET/IB/0/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-20:53:231 [7] NCCL INFO Channel 00/0 : 167[7] -> 168[0] [send] via NET/IB/7/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-19:46:230 [0] NCCL INFO Channel 00/0 : 152[0] -> 153[1] [send] via NET/IB/0/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-12:48:236 [2] NCCL INFO Channel 01/0 : 97[1] -> 98[2] [receive] via NET/IB/11/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-29:49:236 [3] NCCL INFO Channel 03/0 : 235[3] -> 236[4] [send] via NET/IB/12/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-22:48:236 [2] NCCL INFO Channel 01/0 : 178[2] -> 179[3] [send] via NET/IB/11/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-24:51:234 [5] NCCL INFO Channel 01/0 : 197[5] -> 198[6] [send] via NET/IB/2/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-14:51:230 [5] NCCL INFO Channel 03/0 : 117[5] -> 118[6] [send] via NET/IB/2/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-13:51:237 [5] NCCL INFO Channel 01/0 : 109[5] -> 110[6] [send] via NET/IB/2/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-25:53:231 [7] NCCL INFO Channel 02/0 : 206[6] -> 207[7] [receive] via NET/IB/7/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-8:52:236 [6] NCCL INFO Channel 03/0 : 69[5] -> 70[6] [receive] via NET/IB/5/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-10:49:237 [3] NCCL INFO Channel 01/0 : 82[2] -> 83[3] [receive] via NET/IB/12/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-6:52:233 [6] NCCL INFO Channel 03/0 : 53[5] -> 54[6] [receive] via NET/IB/5/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-5:50:231 [4] NCCL INFO Channel 03/0 : 43[3] -> 44[4] [receive] via NET/IB/15/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-4:51:236 [5] NCCL INFO Channel 02/0 : 36[4] -> 37[5] [receive] via NET/IB/3/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-9:51:236 [5] NCCL INFO Channel 03/0 : 76[4] -> 77[5] [receive] via NET/IB/2/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-30:52:235 [6] NCCL INFO Channel 03/0 : 246[6] -> 247[7] [send] via NET/IB/5/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-21:47:234 [1] NCCL INFO Channel 00/0 : 169[1] -> 170[2] [send] via NET/IB/9/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-28:50:234 [4] NCCL INFO Channel 03/0 : 227[3] -> 228[4] [receive] via NET/IB/15/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-26:52:237 [6] NCCL INFO Channel 00/0 : 214[6] -> 215[7] [send] via NET/IB/4/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-27:49:233 [3] NCCL INFO Channel 00/0 : 219[3] -> 220[4] [send] via NET/IB/13/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-11:53:236 [7] NCCL INFO Channel 02/0 : 94[6] -> 95[7] [receive] via NET/IB/7/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-23:50:234 [4] NCCL INFO Channel 00/0 : 188[4] -> 189[5] [send] via NET/IB/14/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-7:52:231 [6] NCCL INFO Channel 03/0 : 62[6] -> 63[7] [send] via NET/IB/5/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-15:48:231 [2] NCCL INFO Channel 00/0 : 122[2] -> 123[3] [send] via NET/IB/10/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-17:48:230 [2] NCCL INFO Channel 02/0 : 137[1] -> 138[2] [receive] via NET/IB/10/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-18:48:233 [2] NCCL INFO Channel 00/0 : 145[1] -> 146[2] [receive] via NET/IB/10/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-20:49:235 [3] NCCL INFO Channel 00/0 : 162[2] -> 163[3] [receive] via NET/IB/13/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-19:48:233 [2] NCCL INFO Channel 00/0 : 153[1] -> 154[2] [receive] via NET/IB/10/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-12:46:237 [0] NCCL INFO Channel 01/0 : 95[7] -> 96[0] [receive] via NET/IB/1/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-29:48:370 [2] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-22:51:233 [5] NCCL INFO Channel 02/0 : 181[5] -> 182[6] [send] via NET/IB/3/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-24:47:231 [1] NCCL INFO Channel 01/0 : 192[0] -> 193[1] [receive] via NET/IB/8/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-14:47:233 [1] NCCL INFO Channel 00/0 : 112[0] -> 113[1] [receive] via NET/IB/9/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-13:53:232 [7] NCCL INFO Channel 03/0 : 110[6] -> 111[7] [receive] via NET/IB/6/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-25:51:232 [5] NCCL INFO Channel 02/0 : 204[4] -> 205[5] [receive] via NET/IB/3/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-8:49:237 [3] NCCL INFO Channel 03/0 : 66[2] -> 67[3] [receive] via NET/IB/12/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-10:50:234 [4] NCCL INFO Channel 03/0 : 83[3] -> 84[4] [receive] via NET/IB/15/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-6:50:237 [4] NCCL INFO Channel 00/0 : 52[4] -> 53[5] [send] via NET/IB/14/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-5:48:368 [2] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-4:52:237 [6] NCCL INFO Channel 00/0 : 37[5] -> 38[6] [receive] via NET/IB/4/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-9:53:235 [7] NCCL INFO Channel 03/0 : 78[6] -> 79[7] [receive] via NET/IB/6/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-30:51:366 [5] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-21:47:234 [1] NCCL INFO Channel 01/0 : 169[1] -> 170[2] [send] via NET/IB/8/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-28:52:232 [6] NCCL INFO Channel 03/0 : 229[5] -> 230[6] [receive] via NET/IB/5/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-26:50:234 [4] NCCL INFO Channel 03/0 : 211[3] -> 212[4] [receive] via NET/IB/15/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-27:49:233 [3] NCCL INFO Channel 01/0 : 219[3] -> 220[4] [send] via NET/IB/12/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-11:47:235 [1] NCCL INFO Channel 00/0 : 88[0] -> 89[1] [receive] via NET/IB/9/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-23:52:237 [6] NCCL INFO Channel 01/0 : 190[6] -> 191[7] [send] via NET/IB/5/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-7:49:235 [3] NCCL INFO Channel 01/0 : 58[2] -> 59[3] [receive] via NET/IB/12/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-15:53:236 [7] NCCL INFO Channel 03/0 : 126[6] -> 127[7] [receive] via NET/IB/6/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-17:53:236 [7] NCCL INFO Channel 00/0 : 143[7] -> 144[0] [send] via NET/IB/7/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-18:53:231 [7] NCCL INFO Channel 00/0 : 151[7] -> 152[0] [send] via NET/IB/7/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-20:46:233 [0] NCCL INFO Channel 00/0 : 160[0] -> 161[1] [send] via NET/IB/0/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-19:46:230 [0] NCCL INFO Channel 01/0 : 152[0] -> 153[1] [send] via NET/IB/1/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-12:46:237 [0] NCCL INFO Channel 02/0 : 95[7] -> 96[0] [receive] via NET/IB/0/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-29:50:369 [4] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-22:53:234 [7] NCCL INFO Channel 02/0 : 183[7] -> 184[0] [send] via NET/IB/7/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-24:53:235 [7] NCCL INFO Channel 00/0 : 199[7] -> 200[0] [send] via NET/IB/7/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-14:53:231 [7] NCCL INFO Channel 03/0 : 119[7] -> 120[0] [send] via NET/IB/6/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-13:46:235 [0] NCCL INFO Channel 01/0 : 104[0] -> 105[1] [send] via NET/IB/1/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-25:53:231 [7] NCCL INFO Channel 03/0 : 206[6] -> 207[7] [receive] via NET/IB/6/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-16:48:231 [2] NCCL INFO Channel 01/0 : 130[2] -> 131[3] [send] via NET/IB/11/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-8:47:232 [1] NCCL INFO Channel 03/0 : 64[0] -> 65[1] [receive] via NET/IB/8/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-10:49:237 [3] NCCL INFO Channel 02/0 : 82[2] -> 83[3] [receive] via NET/IB/13/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-6:52:233 [6] NCCL INFO Channel 00/0 : 54[6] -> 55[7] [send] via NET/IB/4/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-5:47:367 [1] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-4:47:232 [1] NCCL INFO Channel 03/0 : 33[1] -> 34[2] [send] via NET/IB/8/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-9:51:236 [5] NCCL INFO Channel 00/0 : 77[5] -> 78[6] [send] via NET/IB/3/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-30:49:370 [3] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-21:50:237 [4] NCCL INFO Channel 00/0 : 172[4] -> 173[5] [send] via NET/IB/14/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-28:47:230 [1] NCCL INFO Channel 02/0 : 224[0] -> 225[1] [receive] via NET/IB/9/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-26:52:237 [6] NCCL INFO Channel 01/0 : 214[6] -> 215[7] [send] via NET/IB/5/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-27:49:233 [3] NCCL INFO Channel 02/0 : 219[3] -> 220[4] [send] via NET/IB/13/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-11:52:233 [6] NCCL INFO Channel 02/0 : 93[5] -> 94[6] [receive] via NET/IB/4/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-23:48:232 [2] NCCL INFO Channel 03/0 : 185[1] -> 186[2] [receive] via NET/IB/11/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-7:48:237 [2] NCCL INFO Channel 02/0 : 57[1] -> 58[2] [receive] via NET/IB/10/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-15:51:233 [5] NCCL INFO Channel 03/0 : 124[4] -> 125[5] [receive] via NET/IB/2/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-17:48:230 [2] NCCL INFO Channel 03/0 : 137[1] -> 138[2] [receive] via NET/IB/11/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-18:46:230 [0] NCCL INFO Channel 03/0 : 144[0] -> 145[1] [send] via NET/IB/1/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-20:51:230 [5] NCCL INFO Channel 01/0 : 165[5] -> 166[6] [send] via NET/IB/2/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-19:48:233 [2] NCCL INFO Channel 01/0 : 153[1] -> 154[2] [receive] via NET/IB/11/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-12:48:236 [2] NCCL INFO Channel 02/0 : 97[1] -> 98[2] [receive] via NET/IB/10/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-29:49:373 [3] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-22:46:231 [0] NCCL INFO Channel 03/0 : 176[0] -> 177[1] [send] via NET/IB/1/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-24:48:230 [2] NCCL INFO Channel 02/0 : 194[2] -> 195[3] [send] via NET/IB/10/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-14:47:233 [1] NCCL INFO Channel 01/0 : 112[0] -> 113[1] [receive] via NET/IB/8/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-13:51:237 [5] NCCL INFO Channel 02/0 : 109[5] -> 110[6] [send] via NET/IB/3/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-25:51:232 [5] NCCL INFO Channel 03/0 : 204[4] -> 205[5] [receive] via NET/IB/2/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-16:53:236 [7] NCCL INFO Channel 01/0 : 134[6] -> 135[7] [receive] via NET/IB/6/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-8:50:233 [4] NCCL INFO Channel 03/0 : 67[3] -> 68[4] [receive] via NET/IB/15/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-10:50:234 [4] NCCL INFO Channel 00/0 : 84[4] -> 85[5] [send] via NET/IB/14/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-6:50:237 [4] NCCL INFO Channel 01/0 : 52[4] -> 53[5] [send] via NET/IB/15/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-5:52:237 [6] NCCL INFO Channel 00/0 : 45[5] -> 46[6] [receive] via NET/IB/4/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-4:51:236 [5] NCCL INFO Channel 03/0 : 36[4] -> 37[5] [receive] via NET/IB/2/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-9:53:235 [7] NCCL INFO Channel 00/0 : 79[7] -> 80[0] [send] via NET/IB/7/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-30:50:368 [4] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-21:47:234 [1] NCCL INFO Channel 02/0 : 169[1] -> 170[2] [send] via NET/IB/9/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-28:50:234 [4] NCCL INFO Channel 00/0 : 228[4] -> 229[5] [send] via NET/IB/14/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-26:50:234 [4] NCCL INFO Channel 00/0 : 212[4] -> 213[5] [send] via NET/IB/14/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-27:49:233 [3] NCCL INFO Channel 03/0 : 219[3] -> 220[4] [send] via NET/IB/12/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-11:51:234 [5] NCCL INFO Channel 00/0 : 93[5] -> 94[6] [send] via NET/IB/3/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-23:50:234 [4] NCCL INFO Channel 01/0 : 188[4] -> 189[5] [send] via NET/IB/15/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-7:49:235 [3] NCCL INFO Channel 02/0 : 58[2] -> 59[3] [receive] via NET/IB/13/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-15:48:231 [2] NCCL INFO Channel 01/0 : 122[2] -> 123[3] [send] via NET/IB/11/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-17:51:232 [5] NCCL INFO Channel 03/0 : 140[4] -> 141[5] [receive] via NET/IB/2/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-18:51:235 [5] NCCL INFO Channel 03/0 : 148[4] -> 149[5] [receive] via NET/IB/2/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-20:53:231 [7] NCCL INFO Channel 01/0 : 167[7] -> 168[0] [send] via NET/IB/6/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-19:46:230 [0] NCCL INFO Channel 02/0 : 152[0] -> 153[1] [send] via NET/IB/0/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-12:46:237 [0] NCCL INFO Channel 03/0 : 95[7] -> 96[0] [receive] via NET/IB/1/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-22:48:236 [2] NCCL INFO Channel 02/0 : 178[2] -> 179[3] [send] via NET/IB/10/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-24:51:234 [5] NCCL INFO Channel 02/0 : 197[5] -> 198[6] [send] via NET/IB/3/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-14:52:235 [6] NCCL INFO Channel 00/0 : 117[5] -> 118[6] [receive] via NET/IB/4/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-13:53:232 [7] NCCL INFO Channel 00/0 : 111[7] -> 112[0] [send] via NET/IB/7/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-25:53:231 [7] NCCL INFO Channel 00/0 : 207[7] -> 208[0] [send] via NET/IB/7/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-16:48:231 [2] NCCL INFO Channel 02/0 : 130[2] -> 131[3] [send] via NET/IB/10/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-8:52:236 [6] NCCL INFO Channel 00/0 : 70[6] -> 71[7] [send] via NET/IB/4/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-10:49:237 [3] NCCL INFO Channel 03/0 : 82[2] -> 83[3] [receive] via NET/IB/12/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-6:52:233 [6] NCCL INFO Channel 01/0 : 54[6] -> 55[7] [send] via NET/IB/5/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-5:50:231 [4] NCCL INFO Channel 00/0 : 44[4] -> 45[5] [send] via NET/IB/14/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-4:52:237 [6] NCCL INFO Channel 01/0 : 37[5] -> 38[6] [receive] via NET/IB/5/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-9:51:236 [5] NCCL INFO Channel 01/0 : 77[5] -> 78[6] [send] via NET/IB/2/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-30:53:367 [7] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-21:50:237 [4] NCCL INFO Channel 01/0 : 172[4] -> 173[5] [send] via NET/IB/15/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-28:52:232 [6] NCCL INFO Channel 00/0 : 230[6] -> 231[7] [send] via NET/IB/4/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-26:52:237 [6] NCCL INFO Channel 02/0 : 214[6] -> 215[7] [send] via NET/IB/4/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-27:46:231 [0] NCCL INFO Channel 00/0 : 215[7] -> 216[0] [receive] via NET/IB/0/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-11:49:232 [3] NCCL INFO Channel 02/0 : 90[2] -> 91[3] [receive] via NET/IB/13/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-23:52:237 [6] NCCL INFO Channel 02/0 : 190[6] -> 191[7] [send] via NET/IB/4/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-7:48:237 [2] NCCL INFO Channel 03/0 : 57[1] -> 58[2] [receive] via NET/IB/11/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-15:51:233 [5] NCCL INFO Channel 00/0 : 125[5] -> 126[6] [send] via NET/IB/3/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-17:53:236 [7] NCCL INFO Channel 01/0 : 143[7] -> 144[0] [send] via NET/IB/6/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-18:48:233 [2] NCCL INFO Channel 01/0 : 145[1] -> 146[2] [receive] via NET/IB/11/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-19:48:233 [2] NCCL INFO Channel 02/0 : 153[1] -> 154[2] [receive] via NET/IB/10/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-12:48:236 [2] NCCL INFO Channel 03/0 : 97[1] -> 98[2] [receive] via NET/IB/11/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-22:50:232 [4] NCCL INFO Channel 00/0 : 179[3] -> 180[4] [receive] via NET/IB/14/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-24:47:231 [1] NCCL INFO Channel 02/0 : 192[0] -> 193[1] [receive] via NET/IB/9/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-14:50:236 [4] NCCL INFO Channel 00/0 : 115[3] -> 116[4] [receive] via NET/IB/14/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-13:46:235 [0] NCCL INFO Channel 02/0 : 104[0] -> 105[1] [send] via NET/IB/0/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-25:51:232 [5] NCCL INFO Channel 00/0 : 205[5] -> 206[6] [send] via NET/IB/3/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-16:51:235 [5] NCCL INFO Channel 00/0 : 133[5] -> 134[6] [send] via NET/IB/3/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-8:49:237 [3] NCCL INFO Channel 00/0 : 67[3] -> 68[4] [send] via NET/IB/13/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-10:47:236 [1] NCCL INFO Channel 00/0 : 80[0] -> 81[1] [receive] via NET/IB/9/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-6:50:237 [4] NCCL INFO Channel 02/0 : 52[4] -> 53[5] [send] via NET/IB/14/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-5:52:237 [6] NCCL INFO Channel 01/0 : 45[5] -> 46[6] [receive] via NET/IB/5/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-4:51:236 [5] NCCL INFO Channel 00/0 : 37[5] -> 38[6] [send] via NET/IB/3/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-9:53:235 [7] NCCL INFO Channel 01/0 : 79[7] -> 80[0] [send] via NET/IB/6/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-30:52:369 [6] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-21:47:234 [1] NCCL INFO Channel 03/0 : 169[1] -> 170[2] [send] via NET/IB/8/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-28:47:230 [1] NCCL INFO Channel 03/0 : 224[0] -> 225[1] [receive] via NET/IB/8/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-26:50:234 [4] NCCL INFO Channel 01/0 : 212[4] -> 213[5] [send] via NET/IB/15/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-27:47:236 [1] NCCL INFO Channel 00/0 : 216[0] -> 217[1] [receive] via NET/IB/9/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-11:50:237 [4] NCCL INFO Channel 02/0 : 91[3] -> 92[4] [receive] via NET/IB/14/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-23:50:234 [4] NCCL INFO Channel 02/0 : 188[4] -> 189[5] [send] via NET/IB/14/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-7:49:235 [3] NCCL INFO Channel 03/0 : 58[2] -> 59[3] [receive] via NET/IB/12/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-15:53:236 [7] NCCL INFO Channel 00/0 : 127[7] -> 128[0] [send] via NET/IB/7/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-17:48:230 [2] NCCL INFO Channel 00/0 : 138[2] -> 139[3] [send] via NET/IB/10/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-18:53:231 [7] NCCL INFO Channel 01/0 : 151[7] -> 152[0] [send] via NET/IB/6/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-19:46:230 [0] NCCL INFO Channel 03/0 : 152[0] -> 153[1] [send] via NET/IB/1/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-12:48:236 [2] NCCL INFO Channel 00/0 : 98[2] -> 99[3] [send] via NET/IB/10/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-22:51:233 [5] NCCL INFO Channel 03/0 : 181[5] -> 182[6] [send] via NET/IB/2/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-24:53:235 [7] NCCL INFO Channel 01/0 : 199[7] -> 200[0] [send] via NET/IB/6/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-14:47:233 [1] NCCL INFO Channel 02/0 : 112[0] -> 113[1] [receive] via NET/IB/9/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-13:51:237 [5] NCCL INFO Channel 03/0 : 109[5] -> 110[6] [send] via NET/IB/2/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-25:53:231 [7] NCCL INFO Channel 01/0 : 207[7] -> 208[0] [send] via NET/IB/6/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-16:53:236 [7] NCCL INFO Channel 02/0 : 134[6] -> 135[7] [receive] via NET/IB/7/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-20:49:235 [3] NCCL INFO Channel 01/0 : 162[2] -> 163[3] [receive] via NET/IB/12/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-8:47:232 [1] NCCL INFO Channel 00/0 : 65[1] -> 66[2] [send] via NET/IB/9/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-10:50:234 [4] NCCL INFO Channel 01/0 : 84[4] -> 85[5] [send] via NET/IB/15/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-6:52:233 [6] NCCL INFO Channel 02/0 : 54[6] -> 55[7] [send] via NET/IB/4/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-5:50:231 [4] NCCL INFO Channel 01/0 : 44[4] -> 45[5] [send] via NET/IB/15/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-4:52:237 [6] NCCL INFO Channel 02/0 : 37[5] -> 38[6] [receive] via NET/IB/4/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-9:51:236 [5] NCCL INFO Channel 02/0 : 77[5] -> 78[6] [send] via NET/IB/3/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-30:47:373 [1] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-21:50:237 [4] NCCL INFO Channel 02/0 : 172[4] -> 173[5] [send] via NET/IB/14/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-28:50:234 [4] NCCL INFO Channel 01/0 : 228[4] -> 229[5] [send] via NET/IB/15/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-26:49:233 [3] NCCL INFO Channel 00/0 : 210[2] -> 211[3] [receive] via NET/IB/13/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-27:46:231 [0] NCCL INFO Channel 01/0 : 215[7] -> 216[0] [receive] via NET/IB/1/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-11:53:236 [7] NCCL INFO Channel 03/0 : 94[6] -> 95[7] [receive] via NET/IB/6/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-23:52:237 [6] NCCL INFO Channel 03/0 : 190[6] -> 191[7] [send] via NET/IB/5/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-7:48:237 [2] NCCL INFO Channel 00/0 : 58[2] -> 59[3] [send] via NET/IB/10/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-15:49:235 [3] NCCL INFO Channel 00/0 : 122[2] -> 123[3] [receive] via NET/IB/13/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-17:53:236 [7] NCCL INFO Channel 02/0 : 143[7] -> 144[0] [send] via NET/IB/7/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-18:51:235 [5] NCCL INFO Channel 00/0 : 149[5] -> 150[6] [send] via NET/IB/3/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-19:50:237 [4] NCCL INFO Channel 00/0 : 155[3] -> 156[4] [receive] via NET/IB/14/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-12:51:234 [5] NCCL INFO Channel 00/0 : 100[4] -> 101[5] [receive] via NET/IB/3/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-22:53:234 [7] NCCL INFO Channel 03/0 : 183[7] -> 184[0] [send] via NET/IB/6/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-24:48:230 [2] NCCL INFO Channel 03/0 : 194[2] -> 195[3] [send] via NET/IB/11/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-14:52:235 [6] NCCL INFO Channel 01/0 : 117[5] -> 118[6] [receive] via NET/IB/5/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-13:53:232 [7] NCCL INFO Channel 01/0 : 111[7] -> 112[0] [send] via NET/IB/6/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-25:51:232 [5] NCCL INFO Channel 01/0 : 205[5] -> 206[6] [send] via NET/IB/2/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-16:48:231 [2] NCCL INFO Channel 03/0 : 130[2] -> 131[3] [send] via NET/IB/11/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-20:46:233 [0] NCCL INFO Channel 01/0 : 160[0] -> 161[1] [send] via NET/IB/1/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-8:50:233 [4] NCCL INFO Channel 00/0 : 68[4] -> 69[5] [send] via NET/IB/14/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-10:49:237 [3] NCCL INFO Channel 00/0 : 83[3] -> 84[4] [send] via NET/IB/13/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-6:50:237 [4] NCCL INFO Channel 03/0 : 52[4] -> 53[5] [send] via NET/IB/15/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-5:50:231 [4] NCCL INFO Channel 02/0 : 44[4] -> 45[5] [send] via NET/IB/14/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-4:51:236 [5] NCCL INFO Channel 01/0 : 37[5] -> 38[6] [send] via NET/IB/2/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-9:53:235 [7] NCCL INFO Channel 02/0 : 79[7] -> 80[0] [send] via NET/IB/7/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-21:52:367 [6] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-28:47:230 [1] NCCL INFO Channel 00/0 : 225[1] -> 226[2] [send] via NET/IB/9/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-26:52:237 [6] NCCL INFO Channel 03/0 : 214[6] -> 215[7] [send] via NET/IB/5/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-27:47:236 [1] NCCL INFO Channel 01/0 : 216[0] -> 217[1] [receive] via NET/IB/8/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-11:47:235 [1] NCCL INFO Channel 01/0 : 88[0] -> 89[1] [receive] via NET/IB/8/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-23:48:232 [2] NCCL INFO Channel 00/0 : 186[2] -> 187[3] [send] via NET/IB/10/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-7:49:235 [3] NCCL INFO Channel 00/0 : 59[3] -> 60[4] [send] via NET/IB/13/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-15:48:231 [2] NCCL INFO Channel 02/0 : 122[2] -> 123[3] [send] via NET/IB/10/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-17:51:232 [5] NCCL INFO Channel 00/0 : 141[5] -> 142[6] [send] via NET/IB/3/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-18:53:231 [7] NCCL INFO Channel 02/0 : 151[7] -> 152[0] [send] via NET/IB/7/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-19:48:233 [2] NCCL INFO Channel 03/0 : 153[1] -> 154[2] [receive] via NET/IB/11/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-12:46:237 [0] NCCL INFO Channel 00/0 : 96[0] -> 97[1] [send] via NET/IB/0/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-22:48:236 [2] NCCL INFO Channel 03/0 : 178[2] -> 179[3] [send] via NET/IB/11/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-24:51:234 [5] NCCL INFO Channel 03/0 : 197[5] -> 198[6] [send] via NET/IB/2/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-14:50:236 [4] NCCL INFO Channel 01/0 : 115[3] -> 116[4] [receive] via NET/IB/15/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-13:46:235 [0] NCCL INFO Channel 03/0 : 104[0] -> 105[1] [send] via NET/IB/1/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-25:53:231 [7] NCCL INFO Channel 02/0 : 207[7] -> 208[0] [send] via NET/IB/7/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-16:51:235 [5] NCCL INFO Channel 01/0 : 133[5] -> 134[6] [send] via NET/IB/2/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-20:51:230 [5] NCCL INFO Channel 02/0 : 165[5] -> 166[6] [send] via NET/IB/3/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-8:52:236 [6] NCCL INFO Channel 01/0 : 70[6] -> 71[7] [send] via NET/IB/5/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-10:52:233 [6] NCCL INFO Channel 00/0 : 85[5] -> 86[6] [receive] via NET/IB/4/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-6:52:233 [6] NCCL INFO Channel 03/0 : 54[6] -> 55[7] [send] via NET/IB/5/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-5:52:237 [6] NCCL INFO Channel 02/0 : 45[5] -> 46[6] [receive] via NET/IB/4/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-4:52:237 [6] NCCL INFO Channel 03/0 : 37[5] -> 38[6] [receive] via NET/IB/5/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-9:51:236 [5] NCCL INFO Channel 03/0 : 77[5] -> 78[6] [send] via NET/IB/2/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-21:50:237 [4] NCCL INFO Channel 03/0 : 172[4] -> 173[5] [send] via NET/IB/15/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-28:52:232 [6] NCCL INFO Channel 01/0 : 230[6] -> 231[7] [send] via NET/IB/5/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-26:50:234 [4] NCCL INFO Channel 02/0 : 212[4] -> 213[5] [send] via NET/IB/14/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-27:46:231 [0] NCCL INFO Channel 02/0 : 215[7] -> 216[0] [receive] via NET/IB/0/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-11:51:234 [5] NCCL INFO Channel 01/0 : 93[5] -> 94[6] [send] via NET/IB/2/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-23:50:234 [4] NCCL INFO Channel 03/0 : 188[4] -> 189[5] [send] via NET/IB/15/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-7:48:237 [2] NCCL INFO Channel 01/0 : 58[2] -> 59[3] [send] via NET/IB/11/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-15:51:233 [5] NCCL INFO Channel 01/0 : 125[5] -> 126[6] [send] via NET/IB/2/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-17:53:236 [7] NCCL INFO Channel 03/0 : 143[7] -> 144[0] [send] via NET/IB/6/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-18:48:233 [2] NCCL INFO Channel 02/0 : 145[1] -> 146[2] [receive] via NET/IB/10/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-12:54:233 [7] NCCL INFO Channel 00/0 : 102[6] -> 103[7] [receive] via NET/IB/7/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-22:50:232 [4] NCCL INFO Channel 01/0 : 179[3] -> 180[4] [receive] via NET/IB/15/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-24:53:235 [7] NCCL INFO Channel 02/0 : 199[7] -> 200[0] [send] via NET/IB/7/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-14:47:233 [1] NCCL INFO Channel 03/0 : 112[0] -> 113[1] [receive] via NET/IB/8/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-13:52:233 [6] NCCL INFO Channel 00/0 : 109[5] -> 110[6] [receive] via NET/IB/4/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-25:50:235 [4] NCCL INFO Channel 00/0 : 203[3] -> 204[4] [receive] via NET/IB/14/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-16:53:236 [7] NCCL INFO Channel 03/0 : 134[6] -> 135[7] [receive] via NET/IB/6/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-20:53:231 [7] NCCL INFO Channel 02/0 : 167[7] -> 168[0] [send] via NET/IB/7/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-8:49:237 [3] NCCL INFO Channel 01/0 : 67[3] -> 68[4] [send] via NET/IB/12/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-10:50:234 [4] NCCL INFO Channel 02/0 : 84[4] -> 85[5] [send] via NET/IB/14/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-6:48:370 [2] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-5:50:231 [4] NCCL INFO Channel 03/0 : 44[4] -> 45[5] [send] via NET/IB/15/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-4:51:236 [5] NCCL INFO Channel 02/0 : 37[5] -> 38[6] [send] via NET/IB/3/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-9:53:235 [7] NCCL INFO Channel 03/0 : 79[7] -> 80[0] [send] via NET/IB/6/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-21:46:230 [0] NCCL INFO Channel 00/0 : 167[7] -> 168[0] [receive] via NET/IB/0/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-28:50:234 [4] NCCL INFO Channel 02/0 : 228[4] -> 229[5] [send] via NET/IB/14/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-26:50:234 [4] NCCL INFO Channel 03/0 : 212[4] -> 213[5] [send] via NET/IB/15/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-27:47:236 [1] NCCL INFO Channel 02/0 : 216[0] -> 217[1] [receive] via NET/IB/9/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-11:52:233 [6] NCCL INFO Channel 03/0 : 93[5] -> 94[6] [receive] via NET/IB/5/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-23:48:232 [2] NCCL INFO Channel 01/0 : 186[2] -> 187[3] [send] via NET/IB/11/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-7:49:235 [3] NCCL INFO Channel 01/0 : 59[3] -> 60[4] [send] via NET/IB/12/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-15:53:236 [7] NCCL INFO Channel 01/0 : 127[7] -> 128[0] [send] via NET/IB/6/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-17:48:230 [2] NCCL INFO Channel 01/0 : 138[2] -> 139[3] [send] via NET/IB/11/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-18:51:235 [5] NCCL INFO Channel 01/0 : 149[5] -> 150[6] [send] via NET/IB/2/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-12:48:236 [2] NCCL INFO Channel 01/0 : 98[2] -> 99[3] [send] via NET/IB/11/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-22:52:235 [6] NCCL INFO Channel 00/0 : 181[5] -> 182[6] [receive] via NET/IB/4/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-24:47:231 [1] NCCL INFO Channel 03/0 : 192[0] -> 193[1] [receive] via NET/IB/8/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-14:52:235 [6] NCCL INFO Channel 02/0 : 117[5] -> 118[6] [receive] via NET/IB/4/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-13:53:232 [7] NCCL INFO Channel 02/0 : 111[7] -> 112[0] [send] via NET/IB/7/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-25:51:232 [5] NCCL INFO Channel 02/0 : 205[5] -> 206[6] [send] via NET/IB/3/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-16:51:235 [5] NCCL INFO Channel 02/0 : 133[5] -> 134[6] [send] via NET/IB/3/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-20:51:230 [5] NCCL INFO Channel 03/0 : 165[5] -> 166[6] [send] via NET/IB/2/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-19:50:237 [4] NCCL INFO Channel 01/0 : 155[3] -> 156[4] [receive] via NET/IB/15/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-8:47:232 [1] NCCL INFO Channel 01/0 : 65[1] -> 66[2] [send] via NET/IB/8/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-10:47:236 [1] NCCL INFO Channel 01/0 : 80[0] -> 81[1] [receive] via NET/IB/8/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-6:46:372 [0] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-5:52:237 [6] NCCL INFO Channel 03/0 : 45[5] -> 46[6] [receive] via NET/IB/5/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-4:51:236 [5] NCCL INFO Channel 03/0 : 37[5] -> 38[6] [send] via NET/IB/2/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-9:46:366 [0] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-21:46:230 [0] NCCL INFO Channel 01/0 : 167[7] -> 168[0] [receive] via NET/IB/1/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-28:47:230 [1] NCCL INFO Channel 01/0 : 225[1] -> 226[2] [send] via NET/IB/8/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-26:49:233 [3] NCCL INFO Channel 01/0 : 210[2] -> 211[3] [receive] via NET/IB/12/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-27:46:231 [0] NCCL INFO Channel 03/0 : 215[7] -> 216[0] [receive] via NET/IB/1/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-11:49:232 [3] NCCL INFO Channel 03/0 : 90[2] -> 91[3] [receive] via NET/IB/12/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-23:48:232 [2] NCCL INFO Channel 02/0 : 186[2] -> 187[3] [send] via NET/IB/10/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-7:48:237 [2] NCCL INFO Channel 02/0 : 58[2] -> 59[3] [send] via NET/IB/10/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-15:48:231 [2] NCCL INFO Channel 03/0 : 122[2] -> 123[3] [send] via NET/IB/11/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-17:51:232 [5] NCCL INFO Channel 01/0 : 141[5] -> 142[6] [send] via NET/IB/2/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-18:53:231 [7] NCCL INFO Channel 03/0 : 151[7] -> 152[0] [send] via NET/IB/6/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-12:46:237 [0] NCCL INFO Channel 01/0 : 96[0] -> 97[1] [send] via NET/IB/1/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-22:50:232 [4] NCCL INFO Channel 02/0 : 179[3] -> 180[4] [receive] via NET/IB/14/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-24:53:235 [7] NCCL INFO Channel 03/0 : 199[7] -> 200[0] [send] via NET/IB/6/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-14:50:236 [4] NCCL INFO Channel 02/0 : 115[3] -> 116[4] [receive] via NET/IB/14/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-13:50:236 [4] NCCL INFO Channel 00/0 : 107[3] -> 108[4] [receive] via NET/IB/14/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-25:53:231 [7] NCCL INFO Channel 03/0 : 207[7] -> 208[0] [send] via NET/IB/6/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-16:53:236 [7] NCCL INFO Channel 00/0 : 135[7] -> 136[0] [send] via NET/IB/7/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-20:46:233 [0] NCCL INFO Channel 02/0 : 160[0] -> 161[1] [send] via NET/IB/0/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-19:48:233 [2] NCCL INFO Channel 00/0 : 154[2] -> 155[3] [send] via NET/IB/10/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-8:50:233 [4] NCCL INFO Channel 01/0 : 68[4] -> 69[5] [send] via NET/IB/15/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-10:49:237 [3] NCCL INFO Channel 01/0 : 83[3] -> 84[4] [send] via NET/IB/12/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-6:47:373 [1] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-5:52:237 [6] NCCL INFO Channel 00/0 : 46[6] -> 47[7] [send] via NET/IB/4/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-4:52:237 [6] NCCL INFO Channel 00/0 : 38[6] -> 39[7] [send] via NET/IB/4/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-9:47:367 [1] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-21:46:230 [0] NCCL INFO Channel 02/0 : 167[7] -> 168[0] [receive] via NET/IB/0/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-28:50:234 [4] NCCL INFO Channel 03/0 : 228[4] -> 229[5] [send] via NET/IB/15/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-26:49:233 [3] NCCL INFO Channel 02/0 : 210[2] -> 211[3] [receive] via NET/IB/13/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-27:47:236 [1] NCCL INFO Channel 03/0 : 216[0] -> 217[1] [receive] via NET/IB/8/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-11:53:236 [7] NCCL INFO Channel 00/0 : 95[7] -> 96[0] [send] via NET/IB/7/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-23:48:232 [2] NCCL INFO Channel 03/0 : 186[2] -> 187[3] [send] via NET/IB/11/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-7:49:235 [3] NCCL INFO Channel 02/0 : 59[3] -> 60[4] [send] via NET/IB/13/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-15:51:233 [5] NCCL INFO Channel 02/0 : 125[5] -> 126[6] [send] via NET/IB/3/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-17:48:230 [2] NCCL INFO Channel 02/0 : 138[2] -> 139[3] [send] via NET/IB/10/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-18:48:233 [2] NCCL INFO Channel 03/0 : 145[1] -> 146[2] [receive] via NET/IB/11/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-12:51:234 [5] NCCL INFO Channel 01/0 : 100[4] -> 101[5] [receive] via NET/IB/2/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-22:52:235 [6] NCCL INFO Channel 01/0 : 181[5] -> 182[6] [receive] via NET/IB/5/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-24:49:233 [3] NCCL INFO Channel 00/0 : 194[2] -> 195[3] [receive] via NET/IB/13/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-14:47:233 [1] NCCL INFO Channel 00/0 : 113[1] -> 114[2] [send] via NET/IB/9/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-13:53:232 [7] NCCL INFO Channel 03/0 : 111[7] -> 112[0] [send] via NET/IB/6/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-25:50:235 [4] NCCL INFO Channel 01/0 : 203[3] -> 204[4] [receive] via NET/IB/15/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-16:51:235 [5] NCCL INFO Channel 03/0 : 133[5] -> 134[6] [send] via NET/IB/2/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-20:49:235 [3] NCCL INFO Channel 02/0 : 162[2] -> 163[3] [receive] via NET/IB/13/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-19:50:237 [4] NCCL INFO Channel 02/0 : 155[3] -> 156[4] [receive] via NET/IB/14/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-8:52:236 [6] NCCL INFO Channel 02/0 : 70[6] -> 71[7] [send] via NET/IB/4/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-10:50:234 [4] NCCL INFO Channel 03/0 : 84[4] -> 85[5] [send] via NET/IB/15/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-6:49:371 [3] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-5:52:237 [6] NCCL INFO Channel 01/0 : 46[6] -> 47[7] [send] via NET/IB/5/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-4:52:237 [6] NCCL INFO Channel 01/0 : 38[6] -> 39[7] [send] via NET/IB/5/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-9:48:368 [2] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-21:51:368 [5] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-28:52:232 [6] NCCL INFO Channel 02/0 : 230[6] -> 231[7] [send] via NET/IB/4/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-26:49:233 [3] NCCL INFO Channel 03/0 : 210[2] -> 211[3] [receive] via NET/IB/12/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-27:46:231 [0] NCCL INFO Channel 00/0 : 216[0] -> 217[1] [send] via NET/IB/0/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-11:50:237 [4] NCCL INFO Channel 03/0 : 91[3] -> 92[4] [receive] via NET/IB/15/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-23:47:233 [1] NCCL INFO Channel 00/0 : 184[0] -> 185[1] [receive] via NET/IB/9/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-7:48:237 [2] NCCL INFO Channel 03/0 : 58[2] -> 59[3] [send] via NET/IB/11/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-15:53:236 [7] NCCL INFO Channel 02/0 : 127[7] -> 128[0] [send] via NET/IB/7/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-17:51:232 [5] NCCL INFO Channel 02/0 : 141[5] -> 142[6] [send] via NET/IB/3/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-18:51:235 [5] NCCL INFO Channel 02/0 : 149[5] -> 150[6] [send] via NET/IB/3/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-12:54:233 [7] NCCL INFO Channel 01/0 : 102[6] -> 103[7] [receive] via NET/IB/6/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-22:50:232 [4] NCCL INFO Channel 03/0 : 179[3] -> 180[4] [receive] via NET/IB/15/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-24:47:231 [1] NCCL INFO Channel 00/0 : 193[1] -> 194[2] [send] via NET/IB/9/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-14:52:235 [6] NCCL INFO Channel 03/0 : 117[5] -> 118[6] [receive] via NET/IB/5/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-13:52:233 [6] NCCL INFO Channel 01/0 : 109[5] -> 110[6] [receive] via NET/IB/5/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-25:51:232 [5] NCCL INFO Channel 03/0 : 205[5] -> 206[6] [send] via NET/IB/2/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-16:53:236 [7] NCCL INFO Channel 01/0 : 135[7] -> 136[0] [send] via NET/IB/6/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-20:53:231 [7] NCCL INFO Channel 03/0 : 167[7] -> 168[0] [send] via NET/IB/6/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-19:47:232 [1] NCCL INFO Channel 00/0 : 152[0] -> 153[1] [receive] via NET/IB/9/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-8:49:237 [3] NCCL INFO Channel 02/0 : 67[3] -> 68[4] [send] via NET/IB/13/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-10:52:233 [6] NCCL INFO Channel 01/0 : 85[5] -> 86[6] [receive] via NET/IB/5/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-6:50:369 [4] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-5:52:237 [6] NCCL INFO Channel 02/0 : 46[6] -> 47[7] [send] via NET/IB/4/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-4:52:237 [6] NCCL INFO Channel 02/0 : 38[6] -> 39[7] [send] via NET/IB/4/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-9:52:234 [6] NCCL INFO Channel 00/0 : 77[5] -> 78[6] [receive] via NET/IB/4/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-21:46:230 [0] NCCL INFO Channel 03/0 : 167[7] -> 168[0] [receive] via NET/IB/1/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-28:47:230 [1] NCCL INFO Channel 02/0 : 225[1] -> 226[2] [send] via NET/IB/9/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-26:49:233 [3] NCCL INFO Channel 00/0 : 211[3] -> 212[4] [send] via NET/IB/13/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-27:47:236 [1] NCCL INFO Channel 00/0 : 217[1] -> 218[2] [send] via NET/IB/9/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-11:51:234 [5] NCCL INFO Channel 02/0 : 93[5] -> 94[6] [send] via NET/IB/3/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-23:47:233 [1] NCCL INFO Channel 01/0 : 184[0] -> 185[1] [receive] via NET/IB/8/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-7:49:235 [3] NCCL INFO Channel 03/0 : 59[3] -> 60[4] [send] via NET/IB/12/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-15:49:235 [3] NCCL INFO Channel 01/0 : 122[2] -> 123[3] [receive] via NET/IB/12/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-17:49:233 [3] NCCL INFO Channel 00/0 : 138[2] -> 139[3] [receive] via NET/IB/13/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-18:51:235 [5] NCCL INFO Channel 03/0 : 149[5] -> 150[6] [send] via NET/IB/2/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-12:48:236 [2] NCCL INFO Channel 02/0 : 98[2] -> 99[3] [send] via NET/IB/10/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-22:52:235 [6] NCCL INFO Channel 02/0 : 181[5] -> 182[6] [receive] via NET/IB/4/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-24:49:233 [3] NCCL INFO Channel 01/0 : 194[2] -> 195[3] [receive] via NET/IB/12/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-14:50:236 [4] NCCL INFO Channel 03/0 : 115[3] -> 116[4] [receive] via NET/IB/15/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-13:50:236 [4] NCCL INFO Channel 01/0 : 107[3] -> 108[4] [receive] via NET/IB/15/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-16:52:237 [6] NCCL INFO Channel 00/0 : 133[5] -> 134[6] [receive] via NET/IB/4/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-20:46:233 [0] NCCL INFO Channel 03/0 : 160[0] -> 161[1] [send] via NET/IB/1/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-19:48:233 [2] NCCL INFO Channel 01/0 : 154[2] -> 155[3] [send] via NET/IB/11/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-8:47:232 [1] NCCL INFO Channel 02/0 : 65[1] -> 66[2] [send] via NET/IB/9/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-10:47:236 [1] NCCL INFO Channel 02/0 : 80[0] -> 81[1] [receive] via NET/IB/9/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-6:52:367 [6] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-5:52:237 [6] NCCL INFO Channel 03/0 : 46[6] -> 47[7] [send] via NET/IB/5/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-4:52:237 [6] NCCL INFO Channel 03/0 : 38[6] -> 39[7] [send] via NET/IB/5/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-9:52:234 [6] NCCL INFO Channel 01/0 : 77[5] -> 78[6] [receive] via NET/IB/5/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-21:53:366 [7] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-28:52:232 [6] NCCL INFO Channel 03/0 : 230[6] -> 231[7] [send] via NET/IB/5/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-26:49:233 [3] NCCL INFO Channel 01/0 : 211[3] -> 212[4] [send] via NET/IB/12/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-27:46:231 [0] NCCL INFO Channel 01/0 : 216[0] -> 217[1] [send] via NET/IB/1/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-11:52:233 [6] NCCL INFO Channel 00/0 : 94[6] -> 95[7] [send] via NET/IB/4/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-23:47:233 [1] NCCL INFO Channel 02/0 : 184[0] -> 185[1] [receive] via NET/IB/9/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-7:47:233 [1] NCCL INFO Channel 00/0 : 56[0] -> 57[1] [receive] via NET/IB/9/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-15:51:233 [5] NCCL INFO Channel 03/0 : 125[5] -> 126[6] [send] via NET/IB/2/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-17:48:230 [2] NCCL INFO Channel 03/0 : 138[2] -> 139[3] [send] via NET/IB/11/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-18:48:233 [2] NCCL INFO Channel 00/0 : 146[2] -> 147[3] [send] via NET/IB/10/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-12:46:237 [0] NCCL INFO Channel 02/0 : 96[0] -> 97[1] [send] via NET/IB/0/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-22:50:232 [4] NCCL INFO Channel 00/0 : 180[4] -> 181[5] [send] via NET/IB/14/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-24:47:231 [1] NCCL INFO Channel 01/0 : 193[1] -> 194[2] [send] via NET/IB/8/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-14:52:235 [6] NCCL INFO Channel 00/0 : 118[6] -> 119[7] [send] via NET/IB/4/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-13:52:233 [6] NCCL INFO Channel 02/0 : 109[5] -> 110[6] [receive] via NET/IB/4/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-16:53:236 [7] NCCL INFO Channel 02/0 : 135[7] -> 136[0] [send] via NET/IB/7/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-20:49:235 [3] NCCL INFO Channel 03/0 : 162[2] -> 163[3] [receive] via NET/IB/12/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-19:50:237 [4] NCCL INFO Channel 03/0 : 155[3] -> 156[4] [receive] via NET/IB/15/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-25:49:233 [3] NCCL INFO Channel 00/0 : 202[2] -> 203[3] [receive] via NET/IB/13/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-8:50:233 [4] NCCL INFO Channel 02/0 : 68[4] -> 69[5] [send] via NET/IB/14/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-10:49:237 [3] NCCL INFO Channel 02/0 : 83[3] -> 84[4] [send] via NET/IB/13/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-6:53:366 [7] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-5:49:369 [3] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-4:48:369 [2] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-9:50:237 [4] NCCL INFO Channel 00/0 : 75[3] -> 76[4] [receive] via NET/IB/14/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-21:46:230 [0] NCCL INFO Channel 00/0 : 168[0] -> 169[1] [send] via NET/IB/0/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-28:47:230 [1] NCCL INFO Channel 03/0 : 225[1] -> 226[2] [send] via NET/IB/8/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-26:49:233 [3] NCCL INFO Channel 02/0 : 211[3] -> 212[4] [send] via NET/IB/13/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-27:47:236 [1] NCCL INFO Channel 01/0 : 217[1] -> 218[2] [send] via NET/IB/8/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-11:47:235 [1] NCCL INFO Channel 02/0 : 88[0] -> 89[1] [receive] via NET/IB/9/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-23:47:233 [1] NCCL INFO Channel 03/0 : 184[0] -> 185[1] [receive] via NET/IB/8/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-7:51:369 [5] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-15:53:236 [7] NCCL INFO Channel 03/0 : 127[7] -> 128[0] [send] via NET/IB/6/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-17:51:232 [5] NCCL INFO Channel 03/0 : 141[5] -> 142[6] [send] via NET/IB/2/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-18:48:233 [2] NCCL INFO Channel 01/0 : 146[2] -> 147[3] [send] via NET/IB/11/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-12:49:231 [3] NCCL INFO Channel 00/0 : 98[2] -> 99[3] [receive] via NET/IB/13/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-22:52:235 [6] NCCL INFO Channel 03/0 : 181[5] -> 182[6] [receive] via NET/IB/5/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-24:49:233 [3] NCCL INFO Channel 02/0 : 194[2] -> 195[3] [receive] via NET/IB/13/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-14:50:236 [4] NCCL INFO Channel 00/0 : 116[4] -> 117[5] [send] via NET/IB/14/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-13:50:236 [4] NCCL INFO Channel 02/0 : 107[3] -> 108[4] [receive] via NET/IB/14/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-16:50:232 [4] NCCL INFO Channel 00/0 : 131[3] -> 132[4] [receive] via NET/IB/14/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-20:49:235 [3] NCCL INFO Channel 00/0 : 163[3] -> 164[4] [send] via NET/IB/13/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-19:48:233 [2] NCCL INFO Channel 02/0 : 154[2] -> 155[3] [send] via NET/IB/10/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-25:50:235 [4] NCCL INFO Channel 02/0 : 203[3] -> 204[4] [receive] via NET/IB/14/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-8:52:236 [6] NCCL INFO Channel 03/0 : 70[6] -> 71[7] [send] via NET/IB/5/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-10:52:233 [6] NCCL INFO Channel 02/0 : 85[5] -> 86[6] [receive] via NET/IB/4/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-6:51:368 [5] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-5:50:372 [4] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-4:46:371 [0] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-9:52:234 [6] NCCL INFO Channel 02/0 : 77[5] -> 78[6] [receive] via NET/IB/4/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-21:46:230 [0] NCCL INFO Channel 01/0 : 168[0] -> 169[1] [send] via NET/IB/1/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-28:49:237 [3] NCCL INFO Channel 00/0 : 226[2] -> 227[3] [receive] via NET/IB/13/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-26:49:233 [3] NCCL INFO Channel 03/0 : 211[3] -> 212[4] [send] via NET/IB/12/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-27:46:231 [0] NCCL INFO Channel 02/0 : 216[0] -> 217[1] [send] via NET/IB/0/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-11:49:232 [3] NCCL INFO Channel 00/0 : 91[3] -> 92[4] [send] via NET/IB/13/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-23:46:230 [0] NCCL INFO Channel 00/0 : 183[7] -> 184[0] [receive] via NET/IB/0/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-7:47:233 [1] NCCL INFO Channel 01/0 : 56[0] -> 57[1] [receive] via NET/IB/8/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-15:49:235 [3] NCCL INFO Channel 02/0 : 122[2] -> 123[3] [receive] via NET/IB/13/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-17:49:233 [3] NCCL INFO Channel 01/0 : 138[2] -> 139[3] [receive] via NET/IB/12/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-18:48:233 [2] NCCL INFO Channel 02/0 : 146[2] -> 147[3] [send] via NET/IB/10/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-12:51:234 [5] NCCL INFO Channel 02/0 : 100[4] -> 101[5] [receive] via NET/IB/3/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-22:50:232 [4] NCCL INFO Channel 01/0 : 180[4] -> 181[5] [send] via NET/IB/15/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-24:47:231 [1] NCCL INFO Channel 02/0 : 193[1] -> 194[2] [send] via NET/IB/9/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-14:47:233 [1] NCCL INFO Channel 01/0 : 113[1] -> 114[2] [send] via NET/IB/8/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-13:52:233 [6] NCCL INFO Channel 03/0 : 109[5] -> 110[6] [receive] via NET/IB/5/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-16:52:237 [6] NCCL INFO Channel 01/0 : 133[5] -> 134[6] [receive] via NET/IB/5/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-20:49:235 [3] NCCL INFO Channel 01/0 : 163[3] -> 164[4] [send] via NET/IB/12/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-19:47:232 [1] NCCL INFO Channel 01/0 : 152[0] -> 153[1] [receive] via NET/IB/8/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-25:52:236 [6] NCCL INFO Channel 00/0 : 205[5] -> 206[6] [receive] via NET/IB/4/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-8:49:237 [3] NCCL INFO Channel 03/0 : 67[3] -> 68[4] [send] via NET/IB/12/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-10:47:236 [1] NCCL INFO Channel 03/0 : 80[0] -> 81[1] [receive] via NET/IB/8/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-5:52:373 [6] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-4:50:368 [4] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-9:50:237 [4] NCCL INFO Channel 01/0 : 75[3] -> 76[4] [receive] via NET/IB/15/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-21:46:230 [0] NCCL INFO Channel 02/0 : 168[0] -> 169[1] [send] via NET/IB/0/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-28:51:368 [5] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-26:48:235 [2] NCCL INFO Channel 00/0 : 209[1] -> 210[2] [receive] via NET/IB/10/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-27:47:236 [1] NCCL INFO Channel 02/0 : 217[1] -> 218[2] [send] via NET/IB/9/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-11:51:234 [5] NCCL INFO Channel 03/0 : 93[5] -> 94[6] [send] via NET/IB/2/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-23:47:233 [1] NCCL INFO Channel 00/0 : 185[1] -> 186[2] [send] via NET/IB/9/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-7:52:370 [6] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-15:49:235 [3] NCCL INFO Channel 03/0 : 122[2] -> 123[3] [receive] via NET/IB/12/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-17:49:233 [3] NCCL INFO Channel 02/0 : 138[2] -> 139[3] [receive] via NET/IB/13/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-18:48:233 [2] NCCL INFO Channel 03/0 : 146[2] -> 147[3] [send] via NET/IB/11/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-12:54:233 [7] NCCL INFO Channel 02/0 : 102[6] -> 103[7] [receive] via NET/IB/7/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-22:52:235 [6] NCCL INFO Channel 00/0 : 182[6] -> 183[7] [send] via NET/IB/4/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-24:49:233 [3] NCCL INFO Channel 03/0 : 194[2] -> 195[3] [receive] via NET/IB/12/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-14:52:235 [6] NCCL INFO Channel 01/0 : 118[6] -> 119[7] [send] via NET/IB/5/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-13:50:236 [4] NCCL INFO Channel 03/0 : 107[3] -> 108[4] [receive] via NET/IB/15/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-16:53:236 [7] NCCL INFO Channel 03/0 : 135[7] -> 136[0] [send] via NET/IB/6/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-20:49:235 [3] NCCL INFO Channel 02/0 : 163[3] -> 164[4] [send] via NET/IB/13/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-19:52:234 [6] NCCL INFO Channel 00/0 : 157[5] -> 158[6] [receive] via NET/IB/4/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-25:49:233 [3] NCCL INFO Channel 01/0 : 202[2] -> 203[3] [receive] via NET/IB/12/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-8:47:232 [1] NCCL INFO Channel 03/0 : 65[1] -> 66[2] [send] via NET/IB/8/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-10:49:237 [3] NCCL INFO Channel 03/0 : 83[3] -> 84[4] [send] via NET/IB/12/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-5:53:371 [7] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-4:51:373 [5] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-9:52:234 [6] NCCL INFO Channel 03/0 : 77[5] -> 78[6] [receive] via NET/IB/5/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-21:49:236 [3] NCCL INFO Channel 00/0 : 170[2] -> 171[3] [receive] via NET/IB/13/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-28:46:233 [0] NCCL INFO Channel 00/0 : 223[7] -> 224[0] [receive] via NET/IB/0/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-26:48:235 [2] NCCL INFO Channel 01/0 : 209[1] -> 210[2] [receive] via NET/IB/11/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-27:46:231 [0] NCCL INFO Channel 03/0 : 216[0] -> 217[1] [send] via NET/IB/1/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-11:53:236 [7] NCCL INFO Channel 01/0 : 95[7] -> 96[0] [send] via NET/IB/6/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-23:49:236 [3] NCCL INFO Channel 00/0 : 186[2] -> 187[3] [receive] via NET/IB/13/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-7:47:233 [1] NCCL INFO Channel 02/0 : 56[0] -> 57[1] [receive] via NET/IB/9/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-15:49:235 [3] NCCL INFO Channel 00/0 : 123[3] -> 124[4] [send] via NET/IB/13/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-17:49:233 [3] NCCL INFO Channel 03/0 : 138[2] -> 139[3] [receive] via NET/IB/12/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-18:47:232 [1] NCCL INFO Channel 00/0 : 144[0] -> 145[1] [receive] via NET/IB/9/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-12:48:236 [2] NCCL INFO Channel 03/0 : 98[2] -> 99[3] [send] via NET/IB/11/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-22:50:232 [4] NCCL INFO Channel 02/0 : 180[4] -> 181[5] [send] via NET/IB/14/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-24:47:231 [1] NCCL INFO Channel 03/0 : 193[1] -> 194[2] [send] via NET/IB/8/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-14:50:236 [4] NCCL INFO Channel 01/0 : 116[4] -> 117[5] [send] via NET/IB/15/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-13:52:233 [6] NCCL INFO Channel 00/0 : 110[6] -> 111[7] [send] via NET/IB/4/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-16:50:232 [4] NCCL INFO Channel 01/0 : 131[3] -> 132[4] [receive] via NET/IB/15/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-20:49:235 [3] NCCL INFO Channel 03/0 : 163[3] -> 164[4] [send] via NET/IB/12/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-19:50:237 [4] NCCL INFO Channel 00/0 : 156[4] -> 157[5] [send] via NET/IB/14/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-25:50:235 [4] NCCL INFO Channel 03/0 : 203[3] -> 204[4] [receive] via NET/IB/15/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-8:50:233 [4] NCCL INFO Channel 03/0 : 68[4] -> 69[5] [send] via NET/IB/15/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-10:47:236 [1] NCCL INFO Channel 00/0 : 81[1] -> 82[2] [send] via NET/IB/9/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-5:51:370 [5] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-4:52:370 [6] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-9:50:237 [4] NCCL INFO Channel 02/0 : 75[3] -> 76[4] [receive] via NET/IB/14/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-21:46:230 [0] NCCL INFO Channel 03/0 : 168[0] -> 169[1] [send] via NET/IB/1/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-28:49:237 [3] NCCL INFO Channel 01/0 : 226[2] -> 227[3] [receive] via NET/IB/12/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-26:52:367 [6] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-27:47:236 [1] NCCL INFO Channel 03/0 : 217[1] -> 218[2] [send] via NET/IB/8/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-11:50:237 [4] NCCL INFO Channel 00/0 : 92[4] -> 93[5] [send] via NET/IB/14/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-23:46:230 [0] NCCL INFO Channel 01/0 : 183[7] -> 184[0] [receive] via NET/IB/1/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-7:47:233 [1] NCCL INFO Channel 03/0 : 56[0] -> 57[1] [receive] via NET/IB/8/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-15:49:235 [3] NCCL INFO Channel 01/0 : 123[3] -> 124[4] [send] via NET/IB/12/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-17:49:233 [3] NCCL INFO Channel 00/0 : 139[3] -> 140[4] [send] via NET/IB/13/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-18:49:234 [3] NCCL INFO Channel 00/0 : 146[2] -> 147[3] [receive] via NET/IB/13/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-12:46:237 [0] NCCL INFO Channel 03/0 : 96[0] -> 97[1] [send] via NET/IB/1/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-22:50:232 [4] NCCL INFO Channel 03/0 : 180[4] -> 181[5] [send] via NET/IB/15/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-24:49:233 [3] NCCL INFO Channel 00/0 : 195[3] -> 196[4] [send] via NET/IB/13/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-14:47:233 [1] NCCL INFO Channel 02/0 : 113[1] -> 114[2] [send] via NET/IB/9/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-13:50:236 [4] NCCL INFO Channel 00/0 : 108[4] -> 109[5] [send] via NET/IB/14/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-16:52:237 [6] NCCL INFO Channel 02/0 : 133[5] -> 134[6] [receive] via NET/IB/4/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-20:48:236 [2] NCCL INFO Channel 00/0 : 161[1] -> 162[2] [receive] via NET/IB/10/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-19:48:233 [2] NCCL INFO Channel 03/0 : 154[2] -> 155[3] [send] via NET/IB/11/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-25:52:236 [6] NCCL INFO Channel 01/0 : 205[5] -> 206[6] [receive] via NET/IB/5/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-8:46:366 [0] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-10:52:233 [6] NCCL INFO Channel 03/0 : 85[5] -> 86[6] [receive] via NET/IB/5/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-4:53:366 [7] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-9:52:234 [6] NCCL INFO Channel 00/0 : 78[6] -> 79[7] [send] via NET/IB/4/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-21:49:236 [3] NCCL INFO Channel 01/0 : 170[2] -> 171[3] [receive] via NET/IB/12/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-28:46:233 [0] NCCL INFO Channel 01/0 : 223[7] -> 224[0] [receive] via NET/IB/1/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-26:51:369 [5] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-27:52:230 [6] NCCL INFO Channel 00/0 : 221[5] -> 222[6] [receive] via NET/IB/4/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-11:52:233 [6] NCCL INFO Channel 01/0 : 94[6] -> 95[7] [send] via NET/IB/5/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-23:47:233 [1] NCCL INFO Channel 01/0 : 185[1] -> 186[2] [send] via NET/IB/8/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-7:47:233 [1] NCCL INFO Channel 00/0 : 57[1] -> 58[2] [send] via NET/IB/9/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-15:47:232 [1] NCCL INFO Channel 00/0 : 120[0] -> 121[1] [receive] via NET/IB/9/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-17:47:231 [1] NCCL INFO Channel 00/0 : 136[0] -> 137[1] [receive] via NET/IB/9/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-18:47:232 [1] NCCL INFO Channel 01/0 : 144[0] -> 145[1] [receive] via NET/IB/8/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-12:49:231 [3] NCCL INFO Channel 01/0 : 98[2] -> 99[3] [receive] via NET/IB/12/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-22:52:235 [6] NCCL INFO Channel 01/0 : 182[6] -> 183[7] [send] via NET/IB/5/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-24:49:233 [3] NCCL INFO Channel 01/0 : 195[3] -> 196[4] [send] via NET/IB/12/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-14:52:235 [6] NCCL INFO Channel 02/0 : 118[6] -> 119[7] [send] via NET/IB/4/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-13:52:233 [6] NCCL INFO Channel 01/0 : 110[6] -> 111[7] [send] via NET/IB/5/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-16:50:232 [4] NCCL INFO Channel 02/0 : 131[3] -> 132[4] [receive] via NET/IB/14/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-20:47:234 [1] NCCL INFO Channel 00/0 : 160[0] -> 161[1] [receive] via NET/IB/9/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-19:50:237 [4] NCCL INFO Channel 01/0 : 156[4] -> 157[5] [send] via NET/IB/15/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-25:49:233 [3] NCCL INFO Channel 02/0 : 202[2] -> 203[3] [receive] via NET/IB/13/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-8:51:368 [5] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-10:47:236 [1] NCCL INFO Channel 01/0 : 81[1] -> 82[2] [send] via NET/IB/8/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-4:49:367 [3] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-9:50:237 [4] NCCL INFO Channel 03/0 : 75[3] -> 76[4] [receive] via NET/IB/15/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-21:49:236 [3] NCCL INFO Channel 02/0 : 170[2] -> 171[3] [receive] via NET/IB/13/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-28:52:369 [6] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-26:48:235 [2] NCCL INFO Channel 02/0 : 209[1] -> 210[2] [receive] via NET/IB/10/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-27:52:230 [6] NCCL INFO Channel 01/0 : 221[5] -> 222[6] [receive] via NET/IB/5/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-11:47:235 [1] NCCL INFO Channel 03/0 : 88[0] -> 89[1] [receive] via NET/IB/8/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-23:51:369 [5] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-7:47:233 [1] NCCL INFO Channel 01/0 : 57[1] -> 58[2] [send] via NET/IB/8/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-15:49:235 [3] NCCL INFO Channel 02/0 : 123[3] -> 124[4] [send] via NET/IB/13/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-17:49:233 [3] NCCL INFO Channel 01/0 : 139[3] -> 140[4] [send] via NET/IB/12/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-18:50:236 [4] NCCL INFO Channel 00/0 : 147[3] -> 148[4] [receive] via NET/IB/14/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-12:51:234 [5] NCCL INFO Channel 03/0 : 100[4] -> 101[5] [receive] via NET/IB/2/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-22:52:235 [6] NCCL INFO Channel 02/0 : 182[6] -> 183[7] [send] via NET/IB/4/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-24:49:233 [3] NCCL INFO Channel 02/0 : 195[3] -> 196[4] [send] via NET/IB/13/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-14:50:236 [4] NCCL INFO Channel 02/0 : 116[4] -> 117[5] [send] via NET/IB/14/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-13:52:233 [6] NCCL INFO Channel 02/0 : 110[6] -> 111[7] [send] via NET/IB/4/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-16:52:237 [6] NCCL INFO Channel 03/0 : 133[5] -> 134[6] [receive] via NET/IB/5/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-20:48:236 [2] NCCL INFO Channel 01/0 : 161[1] -> 162[2] [receive] via NET/IB/11/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-19:47:232 [1] NCCL INFO Channel 02/0 : 152[0] -> 153[1] [receive] via NET/IB/9/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-25:50:235 [4] NCCL INFO Channel 00/0 : 204[4] -> 205[5] [send] via NET/IB/14/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-8:48:367 [2] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-10:52:233 [6] NCCL INFO Channel 00/0 : 86[6] -> 87[7] [send] via NET/IB/4/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-4:47:372 [1] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-9:52:234 [6] NCCL INFO Channel 01/0 : 78[6] -> 79[7] [send] via NET/IB/5/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-21:49:236 [3] NCCL INFO Channel 03/0 : 170[2] -> 171[3] [receive] via NET/IB/12/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-28:49:237 [3] NCCL INFO Channel 02/0 : 226[2] -> 227[3] [receive] via NET/IB/13/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-26:48:235 [2] NCCL INFO Channel 03/0 : 209[1] -> 210[2] [receive] via NET/IB/11/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-27:52:230 [6] NCCL INFO Channel 02/0 : 221[5] -> 222[6] [receive] via NET/IB/4/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-11:49:232 [3] NCCL INFO Channel 01/0 : 91[3] -> 92[4] [send] via NET/IB/12/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-23:52:370 [6] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-7:54:372 [7] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-15:49:235 [3] NCCL INFO Channel 03/0 : 123[3] -> 124[4] [send] via NET/IB/12/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-17:47:231 [1] NCCL INFO Channel 01/0 : 136[0] -> 137[1] [receive] via NET/IB/8/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-18:49:234 [3] NCCL INFO Channel 01/0 : 146[2] -> 147[3] [receive] via NET/IB/12/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-12:54:233 [7] NCCL INFO Channel 03/0 : 102[6] -> 103[7] [receive] via NET/IB/6/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-22:52:235 [6] NCCL INFO Channel 03/0 : 182[6] -> 183[7] [send] via NET/IB/5/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-24:49:233 [3] NCCL INFO Channel 03/0 : 195[3] -> 196[4] [send] via NET/IB/12/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-14:47:233 [1] NCCL INFO Channel 03/0 : 113[1] -> 114[2] [send] via NET/IB/8/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-13:50:236 [4] NCCL INFO Channel 01/0 : 108[4] -> 109[5] [send] via NET/IB/15/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-16:50:232 [4] NCCL INFO Channel 03/0 : 131[3] -> 132[4] [receive] via NET/IB/15/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-20:48:236 [2] NCCL INFO Channel 02/0 : 161[1] -> 162[2] [receive] via NET/IB/10/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-19:52:234 [6] NCCL INFO Channel 01/0 : 157[5] -> 158[6] [receive] via NET/IB/5/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-25:52:236 [6] NCCL INFO Channel 02/0 : 205[5] -> 206[6] [receive] via NET/IB/4/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-8:52:373 [6] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-10:47:236 [1] NCCL INFO Channel 02/0 : 81[1] -> 82[2] [send] via NET/IB/9/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-9:50:237 [4] NCCL INFO Channel 00/0 : 76[4] -> 77[5] [send] via NET/IB/14/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-21:49:236 [3] NCCL INFO Channel 00/0 : 171[3] -> 172[4] [send] via NET/IB/13/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-28:49:237 [3] NCCL INFO Channel 03/0 : 226[2] -> 227[3] [receive] via NET/IB/12/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-26:48:235 [2] NCCL INFO Channel 00/0 : 210[2] -> 211[3] [send] via NET/IB/10/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-27:52:230 [6] NCCL INFO Channel 03/0 : 221[5] -> 222[6] [receive] via NET/IB/5/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-11:53:236 [7] NCCL INFO Channel 02/0 : 95[7] -> 96[0] [send] via NET/IB/7/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-23:49:236 [3] NCCL INFO Channel 01/0 : 186[2] -> 187[3] [receive] via NET/IB/12/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-7:47:233 [1] NCCL INFO Channel 02/0 : 57[1] -> 58[2] [send] via NET/IB/9/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-15:47:232 [1] NCCL INFO Channel 01/0 : 120[0] -> 121[1] [receive] via NET/IB/8/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-17:49:233 [3] NCCL INFO Channel 02/0 : 139[3] -> 140[4] [send] via NET/IB/13/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-18:47:232 [1] NCCL INFO Channel 02/0 : 144[0] -> 145[1] [receive] via NET/IB/9/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-12:54:233 [7] NCCL INFO Channel 00/0 : 103[7] -> 104[0] [send] via NET/IB/7/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-22:51:369 [5] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-24:46:367 [0] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-14:52:235 [6] NCCL INFO Channel 03/0 : 118[6] -> 119[7] [send] via NET/IB/5/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-13:52:233 [6] NCCL INFO Channel 03/0 : 110[6] -> 111[7] [send] via NET/IB/5/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-16:52:237 [6] NCCL INFO Channel 00/0 : 134[6] -> 135[7] [send] via NET/IB/4/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-20:47:234 [1] NCCL INFO Channel 01/0 : 160[0] -> 161[1] [receive] via NET/IB/8/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-19:50:237 [4] NCCL INFO Channel 02/0 : 156[4] -> 157[5] [send] via NET/IB/14/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-25:49:233 [3] NCCL INFO Channel 03/0 : 202[2] -> 203[3] [receive] via NET/IB/12/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-8:50:372 [4] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-10:52:233 [6] NCCL INFO Channel 01/0 : 86[6] -> 87[7] [send] via NET/IB/5/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-9:52:234 [6] NCCL INFO Channel 02/0 : 78[6] -> 79[7] [send] via NET/IB/4/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-21:49:236 [3] NCCL INFO Channel 01/0 : 171[3] -> 172[4] [send] via NET/IB/12/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-28:46:233 [0] NCCL INFO Channel 02/0 : 223[7] -> 224[0] [receive] via NET/IB/0/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-26:48:235 [2] NCCL INFO Channel 01/0 : 210[2] -> 211[3] [send] via NET/IB/11/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-27:52:230 [6] NCCL INFO Channel 00/0 : 222[6] -> 223[7] [send] via NET/IB/4/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-11:50:237 [4] NCCL INFO Channel 01/0 : 92[4] -> 93[5] [send] via NET/IB/15/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-23:47:233 [1] NCCL INFO Channel 02/0 : 185[1] -> 186[2] [send] via NET/IB/9/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-7:47:233 [1] NCCL INFO Channel 03/0 : 57[1] -> 58[2] [send] via NET/IB/8/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-15:47:232 [1] NCCL INFO Channel 02/0 : 120[0] -> 121[1] [receive] via NET/IB/9/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-17:49:233 [3] NCCL INFO Channel 03/0 : 139[3] -> 140[4] [send] via NET/IB/12/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-18:52:237 [6] NCCL INFO Channel 00/0 : 149[5] -> 150[6] [receive] via NET/IB/4/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-12:51:234 [5] NCCL INFO Channel 00/0 : 101[5] -> 102[6] [send] via NET/IB/3/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-22:52:370 [6] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-24:47:368 [1] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-14:50:236 [4] NCCL INFO Channel 03/0 : 116[4] -> 117[5] [send] via NET/IB/15/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-13:50:236 [4] NCCL INFO Channel 02/0 : 108[4] -> 109[5] [send] via NET/IB/14/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-16:52:237 [6] NCCL INFO Channel 01/0 : 134[6] -> 135[7] [send] via NET/IB/5/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-20:48:236 [2] NCCL INFO Channel 03/0 : 161[1] -> 162[2] [receive] via NET/IB/11/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-19:47:232 [1] NCCL INFO Channel 03/0 : 152[0] -> 153[1] [receive] via NET/IB/8/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-25:50:235 [4] NCCL INFO Channel 01/0 : 204[4] -> 205[5] [send] via NET/IB/15/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-8:49:371 [3] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-10:47:236 [1] NCCL INFO Channel 03/0 : 81[1] -> 82[2] [send] via NET/IB/8/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-9:50:237 [4] NCCL INFO Channel 01/0 : 76[4] -> 77[5] [send] via NET/IB/15/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-21:49:236 [3] NCCL INFO Channel 02/0 : 171[3] -> 172[4] [send] via NET/IB/13/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-28:49:237 [3] NCCL INFO Channel 00/0 : 227[3] -> 228[4] [send] via NET/IB/13/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-26:47:230 [1] NCCL INFO Channel 00/0 : 208[0] -> 209[1] [receive] via NET/IB/9/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-27:48:366 [2] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-11:52:233 [6] NCCL INFO Channel 02/0 : 94[6] -> 95[7] [send] via NET/IB/4/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-23:46:230 [0] NCCL INFO Channel 02/0 : 183[7] -> 184[0] [receive] via NET/IB/0/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-7:50:371 [4] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-15:47:232 [1] NCCL INFO Channel 03/0 : 120[0] -> 121[1] [receive] via NET/IB/8/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-17:47:231 [1] NCCL INFO Channel 02/0 : 136[0] -> 137[1] [receive] via NET/IB/9/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-18:50:236 [4] NCCL INFO Channel 01/0 : 147[3] -> 148[4] [receive] via NET/IB/15/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-12:49:231 [3] NCCL INFO Channel 02/0 : 98[2] -> 99[3] [receive] via NET/IB/13/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-22:49:237 [3] NCCL INFO Channel 00/0 : 178[2] -> 179[3] [receive] via NET/IB/13/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-24:48:366 [2] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-14:48:237 [2] NCCL INFO Channel 00/0 : 113[1] -> 114[2] [receive] via NET/IB/10/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-13:50:236 [4] NCCL INFO Channel 03/0 : 108[4] -> 109[5] [send] via NET/IB/15/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-16:50:232 [4] NCCL INFO Channel 00/0 : 132[4] -> 133[5] [send] via NET/IB/14/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-20:47:234 [1] NCCL INFO Channel 02/0 : 160[0] -> 161[1] [receive] via NET/IB/9/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-19:50:237 [4] NCCL INFO Channel 03/0 : 156[4] -> 157[5] [send] via NET/IB/15/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-25:49:233 [3] NCCL INFO Channel 00/0 : 203[3] -> 204[4] [send] via NET/IB/13/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-8:47:370 [1] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-10:52:233 [6] NCCL INFO Channel 02/0 : 86[6] -> 87[7] [send] via NET/IB/4/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-9:52:234 [6] NCCL INFO Channel 03/0 : 78[6] -> 79[7] [send] via NET/IB/5/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-21:49:236 [3] NCCL INFO Channel 03/0 : 171[3] -> 172[4] [send] via NET/IB/12/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-28:46:233 [0] NCCL INFO Channel 03/0 : 223[7] -> 224[0] [receive] via NET/IB/1/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-26:48:235 [2] NCCL INFO Channel 02/0 : 210[2] -> 211[3] [send] via NET/IB/10/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-27:52:230 [6] NCCL INFO Channel 01/0 : 222[6] -> 223[7] [send] via NET/IB/5/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-11:47:235 [1] NCCL INFO Channel 00/0 : 89[1] -> 90[2] [send] via NET/IB/9/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-23:47:233 [1] NCCL INFO Channel 03/0 : 185[1] -> 186[2] [send] via NET/IB/8/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-7:49:373 [3] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-15:47:232 [1] NCCL INFO Channel 00/0 : 121[1] -> 122[2] [send] via NET/IB/9/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-17:47:231 [1] NCCL INFO Channel 03/0 : 136[0] -> 137[1] [receive] via NET/IB/8/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-18:47:232 [1] NCCL INFO Channel 03/0 : 144[0] -> 145[1] [receive] via NET/IB/8/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-12:54:233 [7] NCCL INFO Channel 01/0 : 103[7] -> 104[0] [send] via NET/IB/6/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-22:47:230 [1] NCCL INFO Channel 00/0 : 176[0] -> 177[1] [receive] via NET/IB/9/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-24:50:236 [4] NCCL INFO Channel 00/0 : 195[3] -> 196[4] [receive] via NET/IB/14/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-14:48:237 [2] NCCL INFO Channel 01/0 : 113[1] -> 114[2] [receive] via NET/IB/11/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-13:51:370 [5] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-16:52:237 [6] NCCL INFO Channel 02/0 : 134[6] -> 135[7] [send] via NET/IB/4/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-20:48:236 [2] NCCL INFO Channel 00/0 : 162[2] -> 163[3] [send] via NET/IB/10/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-19:52:234 [6] NCCL INFO Channel 02/0 : 157[5] -> 158[6] [receive] via NET/IB/4/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-25:52:236 [6] NCCL INFO Channel 03/0 : 205[5] -> 206[6] [receive] via NET/IB/5/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-8:53:369 [7] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-10:52:233 [6] NCCL INFO Channel 03/0 : 86[6] -> 87[7] [send] via NET/IB/5/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-9:50:237 [4] NCCL INFO Channel 02/0 : 76[4] -> 77[5] [send] via NET/IB/14/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-21:47:371 [1] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-28:49:237 [3] NCCL INFO Channel 01/0 : 227[3] -> 228[4] [send] via NET/IB/12/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-26:47:230 [1] NCCL INFO Channel 01/0 : 208[0] -> 209[1] [receive] via NET/IB/8/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-27:52:230 [6] NCCL INFO Channel 02/0 : 222[6] -> 223[7] [send] via NET/IB/4/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-11:49:232 [3] NCCL INFO Channel 02/0 : 91[3] -> 92[4] [send] via NET/IB/13/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-23:49:236 [3] NCCL INFO Channel 02/0 : 186[2] -> 187[3] [receive] via NET/IB/13/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-7:46:366 [0] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-15:47:232 [1] NCCL INFO Channel 01/0 : 121[1] -> 122[2] [send] via NET/IB/8/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-17:47:231 [1] NCCL INFO Channel 00/0 : 137[1] -> 138[2] [send] via NET/IB/9/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-18:49:234 [3] NCCL INFO Channel 02/0 : 146[2] -> 147[3] [receive] via NET/IB/13/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-12:51:234 [5] NCCL INFO Channel 01/0 : 101[5] -> 102[6] [send] via NET/IB/2/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-22:49:237 [3] NCCL INFO Channel 01/0 : 178[2] -> 179[3] [receive] via NET/IB/12/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-24:52:232 [6] NCCL INFO Channel 00/0 : 197[5] -> 198[6] [receive] via NET/IB/4/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-14:48:237 [2] NCCL INFO Channel 02/0 : 113[1] -> 114[2] [receive] via NET/IB/10/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-13:52:368 [6] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-16:50:232 [4] NCCL INFO Channel 01/0 : 132[4] -> 133[5] [send] via NET/IB/15/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-20:47:234 [1] NCCL INFO Channel 03/0 : 160[0] -> 161[1] [receive] via NET/IB/8/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-19:51:235 [5] NCCL INFO Channel 00/0 : 156[4] -> 157[5] [receive] via NET/IB/3/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-25:50:235 [4] NCCL INFO Channel 02/0 : 204[4] -> 205[5] [send] via NET/IB/14/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-10:50:370 [4] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-9:50:237 [4] NCCL INFO Channel 03/0 : 76[4] -> 77[5] [send] via NET/IB/15/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-21:48:370 [2] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-28:46:233 [0] NCCL INFO Channel 00/0 : 224[0] -> 225[1] [send] via NET/IB/0/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-26:50:370 [4] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-27:52:230 [6] NCCL INFO Channel 03/0 : 222[6] -> 223[7] [send] via NET/IB/5/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-11:50:237 [4] NCCL INFO Channel 02/0 : 92[4] -> 93[5] [send] via NET/IB/14/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-23:46:230 [0] NCCL INFO Channel 03/0 : 183[7] -> 184[0] [receive] via NET/IB/1/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-7:47:368 [1] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-15:47:232 [1] NCCL INFO Channel 02/0 : 121[1] -> 122[2] [send] via NET/IB/9/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-17:47:231 [1] NCCL INFO Channel 01/0 : 137[1] -> 138[2] [send] via NET/IB/8/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-18:52:237 [6] NCCL INFO Channel 01/0 : 149[5] -> 150[6] [receive] via NET/IB/5/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-12:49:231 [3] NCCL INFO Channel 03/0 : 98[2] -> 99[3] [receive] via NET/IB/12/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-22:47:230 [1] NCCL INFO Channel 01/0 : 176[0] -> 177[1] [receive] via NET/IB/8/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-24:50:236 [4] NCCL INFO Channel 01/0 : 195[3] -> 196[4] [receive] via NET/IB/15/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-14:46:366 [0] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-13:53:371 [7] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-16:52:237 [6] NCCL INFO Channel 03/0 : 134[6] -> 135[7] [send] via NET/IB/5/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-20:48:236 [2] NCCL INFO Channel 01/0 : 162[2] -> 163[3] [send] via NET/IB/11/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-19:47:232 [1] NCCL INFO Channel 00/0 : 153[1] -> 154[2] [send] via NET/IB/9/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-25:49:233 [3] NCCL INFO Channel 01/0 : 203[3] -> 204[4] [send] via NET/IB/12/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-10:51:373 [5] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-9:49:369 [3] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-21:50:369 [4] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-28:49:237 [3] NCCL INFO Channel 02/0 : 227[3] -> 228[4] [send] via NET/IB/13/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-26:48:235 [2] NCCL INFO Channel 03/0 : 210[2] -> 211[3] [send] via NET/IB/11/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-27:46:370 [0] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-11:53:236 [7] NCCL INFO Channel 03/0 : 95[7] -> 96[0] [send] via NET/IB/6/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-23:53:368 [7] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-7:48:367 [2] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-15:47:232 [1] NCCL INFO Channel 03/0 : 121[1] -> 122[2] [send] via NET/IB/8/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-17:47:231 [1] NCCL INFO Channel 02/0 : 137[1] -> 138[2] [send] via NET/IB/9/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-18:47:232 [1] NCCL INFO Channel 00/0 : 145[1] -> 146[2] [send] via NET/IB/9/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-12:54:233 [7] NCCL INFO Channel 02/0 : 103[7] -> 104[0] [send] via NET/IB/7/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-22:49:237 [3] NCCL INFO Channel 02/0 : 178[2] -> 179[3] [receive] via NET/IB/13/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-24:52:232 [6] NCCL INFO Channel 01/0 : 197[5] -> 198[6] [receive] via NET/IB/5/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-14:48:237 [2] NCCL INFO Channel 03/0 : 113[1] -> 114[2] [receive] via NET/IB/11/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-13:49:234 [3] NCCL INFO Channel 00/0 : 106[2] -> 107[3] [receive] via NET/IB/13/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-16:50:232 [4] NCCL INFO Channel 02/0 : 132[4] -> 133[5] [send] via NET/IB/14/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-20:47:234 [1] NCCL INFO Channel 00/0 : 161[1] -> 162[2] [send] via NET/IB/9/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-19:49:236 [3] NCCL INFO Channel 00/0 : 154[2] -> 155[3] [receive] via NET/IB/13/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-25:52:236 [6] NCCL INFO Channel 00/0 : 206[6] -> 207[7] [send] via NET/IB/4/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-10:48:368 [2] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-9:51:370 [5] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-21:46:372 [0] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-28:46:233 [0] NCCL INFO Channel 01/0 : 224[0] -> 225[1] [send] via NET/IB/1/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-26:47:230 [1] NCCL INFO Channel 02/0 : 208[0] -> 209[1] [receive] via NET/IB/9/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-27:52:373 [6] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-11:52:233 [6] NCCL INFO Channel 03/0 : 94[6] -> 95[7] [send] via NET/IB/5/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-23:49:236 [3] NCCL INFO Channel 03/0 : 186[2] -> 187[3] [receive] via NET/IB/12/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-15:46:366 [0] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-17:47:231 [1] NCCL INFO Channel 03/0 : 137[1] -> 138[2] [send] via NET/IB/8/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-18:50:236 [4] NCCL INFO Channel 02/0 : 147[3] -> 148[4] [receive] via NET/IB/14/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-12:51:234 [5] NCCL INFO Channel 02/0 : 101[5] -> 102[6] [send] via NET/IB/3/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-22:47:230 [1] NCCL INFO Channel 02/0 : 176[0] -> 177[1] [receive] via NET/IB/9/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-24:50:236 [4] NCCL INFO Channel 02/0 : 195[3] -> 196[4] [receive] via NET/IB/14/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-14:49:232 [3] NCCL INFO Channel 00/0 : 114[2] -> 115[3] [receive] via NET/IB/13/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-13:47:231 [1] NCCL INFO Channel 00/0 : 104[0] -> 105[1] [receive] via NET/IB/9/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-16:50:232 [4] NCCL INFO Channel 03/0 : 132[4] -> 133[5] [send] via NET/IB/15/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-20:48:236 [2] NCCL INFO Channel 02/0 : 162[2] -> 163[3] [send] via NET/IB/10/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-19:52:234 [6] NCCL INFO Channel 03/0 : 157[5] -> 158[6] [receive] via NET/IB/5/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-25:50:235 [4] NCCL INFO Channel 03/0 : 204[4] -> 205[5] [send] via NET/IB/15/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-10:52:371 [6] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-9:52:372 [6] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-21:49:373 [3] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-28:49:237 [3] NCCL INFO Channel 03/0 : 227[3] -> 228[4] [send] via NET/IB/12/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-26:47:230 [1] NCCL INFO Channel 03/0 : 208[0] -> 209[1] [receive] via NET/IB/8/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-27:53:367 [7] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-11:49:232 [3] NCCL INFO Channel 03/0 : 91[3] -> 92[4] [send] via NET/IB/12/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-23:46:230 [0] NCCL INFO Channel 00/0 : 184[0] -> 185[1] [send] via NET/IB/0/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-15:48:368 [2] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-17:46:366 [0] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-18:49:234 [3] NCCL INFO Channel 03/0 : 146[2] -> 147[3] [receive] via NET/IB/12/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-12:49:231 [3] NCCL INFO Channel 00/0 : 99[3] -> 100[4] [send] via NET/IB/13/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-22:49:237 [3] NCCL INFO Channel 03/0 : 178[2] -> 179[3] [receive] via NET/IB/12/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-24:52:232 [6] NCCL INFO Channel 02/0 : 197[5] -> 198[6] [receive] via NET/IB/4/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-14:51:370 [5] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-13:49:234 [3] NCCL INFO Channel 01/0 : 106[2] -> 107[3] [receive] via NET/IB/12/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-16:52:369 [6] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-20:47:234 [1] NCCL INFO Channel 01/0 : 161[1] -> 162[2] [send] via NET/IB/8/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-19:51:235 [5] NCCL INFO Channel 01/0 : 156[4] -> 157[5] [receive] via NET/IB/2/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-25:49:233 [3] NCCL INFO Channel 02/0 : 203[3] -> 204[4] [send] via NET/IB/13/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-10:53:372 [7] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-9:53:371 [7] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-28:46:233 [0] NCCL INFO Channel 02/0 : 224[0] -> 225[1] [send] via NET/IB/0/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-26:47:230 [1] NCCL INFO Channel 00/0 : 209[1] -> 210[2] [send] via NET/IB/9/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-27:47:371 [1] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-11:47:235 [1] NCCL INFO Channel 01/0 : 89[1] -> 90[2] [send] via NET/IB/8/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-23:49:236 [3] NCCL INFO Channel 00/0 : 187[3] -> 188[4] [send] via NET/IB/13/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-15:50:234 [4] NCCL INFO Channel 00/0 : 123[3] -> 124[4] [receive] via NET/IB/14/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-17:48:368 [2] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-18:52:237 [6] NCCL INFO Channel 02/0 : 149[5] -> 150[6] [receive] via NET/IB/4/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-12:54:233 [7] NCCL INFO Channel 03/0 : 103[7] -> 104[0] [send] via NET/IB/6/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-22:47:230 [1] NCCL INFO Channel 03/0 : 176[0] -> 177[1] [receive] via NET/IB/8/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-24:50:236 [4] NCCL INFO Channel 03/0 : 195[3] -> 196[4] [receive] via NET/IB/15/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-14:48:237 [2] NCCL INFO Channel 00/0 : 114[2] -> 115[3] [send] via NET/IB/10/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-13:47:231 [1] NCCL INFO Channel 01/0 : 104[0] -> 105[1] [receive] via NET/IB/8/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-16:51:370 [5] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-20:52:232 [6] NCCL INFO Channel 00/0 : 165[5] -> 166[6] [receive] via NET/IB/4/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-19:47:232 [1] NCCL INFO Channel 01/0 : 153[1] -> 154[2] [send] via NET/IB/8/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-25:52:236 [6] NCCL INFO Channel 01/0 : 206[6] -> 207[7] [send] via NET/IB/5/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-10:46:366 [0] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-9:50:373 [4] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-28:53:367 [7] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-26:47:230 [1] NCCL INFO Channel 01/0 : 209[1] -> 210[2] [send] via NET/IB/8/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-27:50:232 [4] NCCL INFO Channel 00/0 : 219[3] -> 220[4] [receive] via NET/IB/14/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-11:50:237 [4] NCCL INFO Channel 03/0 : 92[4] -> 93[5] [send] via NET/IB/15/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-23:46:230 [0] NCCL INFO Channel 01/0 : 184[0] -> 185[1] [send] via NET/IB/1/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-15:52:237 [6] NCCL INFO Channel 00/0 : 125[5] -> 126[6] [receive] via NET/IB/4/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-17:52:237 [6] NCCL INFO Channel 00/0 : 141[5] -> 142[6] [receive] via NET/IB/4/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-18:47:232 [1] NCCL INFO Channel 01/0 : 145[1] -> 146[2] [send] via NET/IB/8/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-12:51:234 [5] NCCL INFO Channel 03/0 : 101[5] -> 102[6] [send] via NET/IB/2/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-22:49:237 [3] NCCL INFO Channel 00/0 : 179[3] -> 180[4] [send] via NET/IB/13/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-24:52:232 [6] NCCL INFO Channel 03/0 : 197[5] -> 198[6] [receive] via NET/IB/5/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-14:52:372 [6] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-13:49:234 [3] NCCL INFO Channel 02/0 : 106[2] -> 107[3] [receive] via NET/IB/13/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-16:49:233 [3] NCCL INFO Channel 00/0 : 130[2] -> 131[3] [receive] via NET/IB/13/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-20:50:237 [4] NCCL INFO Channel 00/0 : 163[3] -> 164[4] [receive] via NET/IB/14/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-19:49:236 [3] NCCL INFO Channel 01/0 : 154[2] -> 155[3] [receive] via NET/IB/12/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-25:49:233 [3] NCCL INFO Channel 03/0 : 203[3] -> 204[4] [send] via NET/IB/12/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-10:49:369 [3] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-28:46:233 [0] NCCL INFO Channel 03/0 : 224[0] -> 225[1] [send] via NET/IB/1/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-26:47:230 [1] NCCL INFO Channel 02/0 : 209[1] -> 210[2] [send] via NET/IB/9/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-27:50:232 [4] NCCL INFO Channel 01/0 : 219[3] -> 220[4] [receive] via NET/IB/15/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-11:47:235 [1] NCCL INFO Channel 02/0 : 89[1] -> 90[2] [send] via NET/IB/9/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-23:49:236 [3] NCCL INFO Channel 01/0 : 187[3] -> 188[4] [send] via NET/IB/12/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-15:50:234 [4] NCCL INFO Channel 01/0 : 123[3] -> 124[4] [receive] via NET/IB/15/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-17:47:367 [1] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-18:49:234 [3] NCCL INFO Channel 00/0 : 147[3] -> 148[4] [send] via NET/IB/13/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-12:49:231 [3] NCCL INFO Channel 01/0 : 99[3] -> 100[4] [send] via NET/IB/12/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-22:47:230 [1] NCCL INFO Channel 00/0 : 177[1] -> 178[2] [send] via NET/IB/9/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-24:50:236 [4] NCCL INFO Channel 00/0 : 196[4] -> 197[5] [send] via NET/IB/14/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-14:49:232 [3] NCCL INFO Channel 01/0 : 114[2] -> 115[3] [receive] via NET/IB/12/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-13:47:231 [1] NCCL INFO Channel 02/0 : 104[0] -> 105[1] [receive] via NET/IB/9/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-16:47:234 [1] NCCL INFO Channel 00/0 : 128[0] -> 129[1] [receive] via NET/IB/9/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-20:48:236 [2] NCCL INFO Channel 03/0 : 162[2] -> 163[3] [send] via NET/IB/11/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-19:47:232 [1] NCCL INFO Channel 02/0 : 153[1] -> 154[2] [send] via NET/IB/9/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-25:52:236 [6] NCCL INFO Channel 02/0 : 206[6] -> 207[7] [send] via NET/IB/4/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-10:47:367 [1] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-28:48:366 [2] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-26:47:230 [1] NCCL INFO Channel 03/0 : 209[1] -> 210[2] [send] via NET/IB/8/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-27:50:232 [4] NCCL INFO Channel 02/0 : 219[3] -> 220[4] [receive] via NET/IB/14/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-11:47:235 [1] NCCL INFO Channel 03/0 : 89[1] -> 90[2] [send] via NET/IB/8/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-23:46:230 [0] NCCL INFO Channel 02/0 : 184[0] -> 185[1] [send] via NET/IB/0/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-15:52:237 [6] NCCL INFO Channel 01/0 : 125[5] -> 126[6] [receive] via NET/IB/5/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-17:52:237 [6] NCCL INFO Channel 01/0 : 141[5] -> 142[6] [receive] via NET/IB/5/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-18:50:236 [4] NCCL INFO Channel 03/0 : 147[3] -> 148[4] [receive] via NET/IB/15/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-12:49:231 [3] NCCL INFO Channel 02/0 : 99[3] -> 100[4] [send] via NET/IB/13/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-22:49:237 [3] NCCL INFO Channel 01/0 : 179[3] -> 180[4] [send] via NET/IB/12/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-24:52:232 [6] NCCL INFO Channel 00/0 : 198[6] -> 199[7] [send] via NET/IB/4/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-14:48:237 [2] NCCL INFO Channel 01/0 : 114[2] -> 115[3] [send] via NET/IB/11/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-13:49:234 [3] NCCL INFO Channel 03/0 : 106[2] -> 107[3] [receive] via NET/IB/12/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-16:49:233 [3] NCCL INFO Channel 01/0 : 130[2] -> 131[3] [receive] via NET/IB/12/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-20:47:234 [1] NCCL INFO Channel 02/0 : 161[1] -> 162[2] [send] via NET/IB/9/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-19:52:234 [6] NCCL INFO Channel 00/0 : 158[6] -> 159[7] [send] via NET/IB/4/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-25:52:236 [6] NCCL INFO Channel 03/0 : 206[6] -> 207[7] [send] via NET/IB/5/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-28:50:370 [4] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-26:49:371 [3] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-27:50:232 [4] NCCL INFO Channel 03/0 : 219[3] -> 220[4] [receive] via NET/IB/15/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-11:52:370 [6] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-23:49:236 [3] NCCL INFO Channel 02/0 : 187[3] -> 188[4] [send] via NET/IB/13/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-15:50:234 [4] NCCL INFO Channel 02/0 : 123[3] -> 124[4] [receive] via NET/IB/14/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-17:52:237 [6] NCCL INFO Channel 02/0 : 141[5] -> 142[6] [receive] via NET/IB/4/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-18:52:237 [6] NCCL INFO Channel 03/0 : 149[5] -> 150[6] [receive] via NET/IB/5/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-12:49:231 [3] NCCL INFO Channel 03/0 : 99[3] -> 100[4] [send] via NET/IB/12/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-22:47:230 [1] NCCL INFO Channel 01/0 : 177[1] -> 178[2] [send] via NET/IB/8/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-24:50:236 [4] NCCL INFO Channel 01/0 : 196[4] -> 197[5] [send] via NET/IB/15/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-14:49:232 [3] NCCL INFO Channel 02/0 : 114[2] -> 115[3] [receive] via NET/IB/13/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-13:47:231 [1] NCCL INFO Channel 03/0 : 104[0] -> 105[1] [receive] via NET/IB/8/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-16:47:234 [1] NCCL INFO Channel 01/0 : 128[0] -> 129[1] [receive] via NET/IB/8/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-20:52:232 [6] NCCL INFO Channel 01/0 : 165[5] -> 166[6] [receive] via NET/IB/5/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-19:51:235 [5] NCCL INFO Channel 02/0 : 156[4] -> 157[5] [receive] via NET/IB/3/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-25:48:237 [2] NCCL INFO Channel 00/0 : 201[1] -> 202[2] [receive] via NET/IB/10/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-28:47:372 [1] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-26:48:372 [2] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-27:50:232 [4] NCCL INFO Channel 00/0 : 220[4] -> 221[5] [send] via NET/IB/14/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-11:51:373 [5] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-23:46:230 [0] NCCL INFO Channel 03/0 : 184[0] -> 185[1] [send] via NET/IB/1/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-15:52:237 [6] NCCL INFO Channel 02/0 : 125[5] -> 126[6] [receive] via NET/IB/4/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-17:52:237 [6] NCCL INFO Channel 03/0 : 141[5] -> 142[6] [receive] via NET/IB/5/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-18:47:232 [1] NCCL INFO Channel 02/0 : 145[1] -> 146[2] [send] via NET/IB/9/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-12:47:367 [1] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-22:49:237 [3] NCCL INFO Channel 02/0 : 179[3] -> 180[4] [send] via NET/IB/13/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-24:52:232 [6] NCCL INFO Channel 01/0 : 198[6] -> 199[7] [send] via NET/IB/5/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-14:48:237 [2] NCCL INFO Channel 02/0 : 114[2] -> 115[3] [send] via NET/IB/10/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-13:49:234 [3] NCCL INFO Channel 00/0 : 107[3] -> 108[4] [send] via NET/IB/13/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-16:49:233 [3] NCCL INFO Channel 02/0 : 130[2] -> 131[3] [receive] via NET/IB/13/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-20:50:237 [4] NCCL INFO Channel 01/0 : 163[3] -> 164[4] [receive] via NET/IB/15/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-19:49:236 [3] NCCL INFO Channel 02/0 : 154[2] -> 155[3] [receive] via NET/IB/13/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-25:48:237 [2] NCCL INFO Channel 01/0 : 201[1] -> 202[2] [receive] via NET/IB/11/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-28:46:373 [0] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-26:47:373 [1] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-27:50:232 [4] NCCL INFO Channel 01/0 : 220[4] -> 221[5] [send] via NET/IB/15/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-11:50:369 [4] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-23:49:236 [3] NCCL INFO Channel 03/0 : 187[3] -> 188[4] [send] via NET/IB/12/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-15:50:234 [4] NCCL INFO Channel 03/0 : 123[3] -> 124[4] [receive] via NET/IB/15/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-17:52:237 [6] NCCL INFO Channel 00/0 : 142[6] -> 143[7] [send] via NET/IB/4/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-18:49:234 [3] NCCL INFO Channel 01/0 : 147[3] -> 148[4] [send] via NET/IB/12/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-12:46:366 [0] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-22:47:230 [1] NCCL INFO Channel 02/0 : 177[1] -> 178[2] [send] via NET/IB/9/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-24:52:232 [6] NCCL INFO Channel 02/0 : 198[6] -> 199[7] [send] via NET/IB/4/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-14:49:232 [3] NCCL INFO Channel 03/0 : 114[2] -> 115[3] [receive] via NET/IB/12/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-13:47:231 [1] NCCL INFO Channel 00/0 : 105[1] -> 106[2] [send] via NET/IB/9/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-16:47:234 [1] NCCL INFO Channel 02/0 : 128[0] -> 129[1] [receive] via NET/IB/9/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-20:47:234 [1] NCCL INFO Channel 03/0 : 161[1] -> 162[2] [send] via NET/IB/8/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-19:47:232 [1] NCCL INFO Channel 03/0 : 153[1] -> 154[2] [send] via NET/IB/8/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-25:46:230 [0] NCCL INFO Channel 00/0 : 199[7] -> 200[0] [receive] via NET/IB/0/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-28:49:371 [3] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-26:53:368 [7] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-27:50:232 [4] NCCL INFO Channel 02/0 : 220[4] -> 221[5] [send] via NET/IB/14/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-11:49:371 [3] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-23:48:366 [2] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-15:52:237 [6] NCCL INFO Channel 03/0 : 125[5] -> 126[6] [receive] via NET/IB/5/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-17:52:237 [6] NCCL INFO Channel 01/0 : 142[6] -> 143[7] [send] via NET/IB/5/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-18:50:236 [4] NCCL INFO Channel 00/0 : 148[4] -> 149[5] [send] via NET/IB/14/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-12:48:368 [2] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-22:49:237 [3] NCCL INFO Channel 03/0 : 179[3] -> 180[4] [send] via NET/IB/12/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-24:50:236 [4] NCCL INFO Channel 02/0 : 196[4] -> 197[5] [send] via NET/IB/14/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-14:49:232 [3] NCCL INFO Channel 00/0 : 115[3] -> 116[4] [send] via NET/IB/13/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-13:49:234 [3] NCCL INFO Channel 01/0 : 107[3] -> 108[4] [send] via NET/IB/12/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-16:47:234 [1] NCCL INFO Channel 03/0 : 128[0] -> 129[1] [receive] via NET/IB/8/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-20:52:232 [6] NCCL INFO Channel 02/0 : 165[5] -> 166[6] [receive] via NET/IB/4/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-19:52:234 [6] NCCL INFO Channel 01/0 : 158[6] -> 159[7] [send] via NET/IB/5/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-25:48:237 [2] NCCL INFO Channel 02/0 : 201[1] -> 202[2] [receive] via NET/IB/10/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-26:46:366 [0] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-27:50:232 [4] NCCL INFO Channel 03/0 : 220[4] -> 221[5] [send] via NET/IB/15/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-11:46:366 [0] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-23:50:371 [4] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-15:52:237 [6] NCCL INFO Channel 00/0 : 126[6] -> 127[7] [send] via NET/IB/4/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-17:52:237 [6] NCCL INFO Channel 02/0 : 142[6] -> 143[7] [send] via NET/IB/4/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-18:52:237 [6] NCCL INFO Channel 00/0 : 150[6] -> 151[7] [send] via NET/IB/4/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-12:52:235 [6] NCCL INFO Channel 00/0 : 101[5] -> 102[6] [receive] via NET/IB/4/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-22:47:230 [1] NCCL INFO Channel 03/0 : 177[1] -> 178[2] [send] via NET/IB/8/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-24:52:232 [6] NCCL INFO Channel 03/0 : 198[6] -> 199[7] [send] via NET/IB/5/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-14:48:237 [2] NCCL INFO Channel 03/0 : 114[2] -> 115[3] [send] via NET/IB/11/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-13:47:231 [1] NCCL INFO Channel 01/0 : 105[1] -> 106[2] [send] via NET/IB/8/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-16:49:233 [3] NCCL INFO Channel 03/0 : 130[2] -> 131[3] [receive] via NET/IB/12/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-20:50:237 [4] NCCL INFO Channel 02/0 : 163[3] -> 164[4] [receive] via NET/IB/14/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-19:51:235 [5] NCCL INFO Channel 03/0 : 156[4] -> 157[5] [receive] via NET/IB/2/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-25:46:230 [0] NCCL INFO Channel 01/0 : 199[7] -> 200[0] [receive] via NET/IB/1/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-27:51:368 [5] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-11:48:368 [2] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-23:49:373 [3] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-15:50:234 [4] NCCL INFO Channel 00/0 : 124[4] -> 125[5] [send] via NET/IB/14/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-17:52:237 [6] NCCL INFO Channel 03/0 : 142[6] -> 143[7] [send] via NET/IB/5/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-18:47:232 [1] NCCL INFO Channel 03/0 : 145[1] -> 146[2] [send] via NET/IB/8/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-12:50:232 [4] NCCL INFO Channel 00/0 : 99[3] -> 100[4] [receive] via NET/IB/14/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-22:53:368 [7] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-24:50:236 [4] NCCL INFO Channel 03/0 : 196[4] -> 197[5] [send] via NET/IB/15/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-14:49:232 [3] NCCL INFO Channel 01/0 : 115[3] -> 116[4] [send] via NET/IB/12/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-13:49:234 [3] NCCL INFO Channel 02/0 : 107[3] -> 108[4] [send] via NET/IB/13/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-16:49:233 [3] NCCL INFO Channel 00/0 : 131[3] -> 132[4] [send] via NET/IB/13/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-20:52:232 [6] NCCL INFO Channel 03/0 : 165[5] -> 166[6] [receive] via NET/IB/5/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-19:49:236 [3] NCCL INFO Channel 03/0 : 154[2] -> 155[3] [receive] via NET/IB/12/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-25:48:237 [2] NCCL INFO Channel 03/0 : 201[1] -> 202[2] [receive] via NET/IB/11/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-27:49:369 [3] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-11:47:367 [1] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-23:47:367 [1] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-15:52:237 [6] NCCL INFO Channel 01/0 : 126[6] -> 127[7] [send] via NET/IB/5/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-17:52:373 [6] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-18:49:234 [3] NCCL INFO Channel 02/0 : 147[3] -> 148[4] [send] via NET/IB/13/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-12:50:232 [4] NCCL INFO Channel 01/0 : 99[3] -> 100[4] [receive] via NET/IB/15/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-22:46:367 [0] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-24:49:369 [3] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-14:49:232 [3] NCCL INFO Channel 02/0 : 115[3] -> 116[4] [send] via NET/IB/13/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-13:47:231 [1] NCCL INFO Channel 02/0 : 105[1] -> 106[2] [send] via NET/IB/9/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-16:47:234 [1] NCCL INFO Channel 00/0 : 129[1] -> 130[2] [send] via NET/IB/9/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-20:50:237 [4] NCCL INFO Channel 03/0 : 163[3] -> 164[4] [receive] via NET/IB/15/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-19:51:235 [5] NCCL INFO Channel 00/0 : 157[5] -> 158[6] [send] via NET/IB/3/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-25:46:230 [0] NCCL INFO Channel 02/0 : 199[7] -> 200[0] [receive] via NET/IB/0/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-27:50:372 [4] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-11:53:372 [7] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-23:46:372 [0] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-15:50:234 [4] NCCL INFO Channel 01/0 : 124[4] -> 125[5] [send] via NET/IB/15/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-17:50:235 [4] NCCL INFO Channel 00/0 : 139[3] -> 140[4] [receive] via NET/IB/14/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-18:50:236 [4] NCCL INFO Channel 01/0 : 148[4] -> 149[5] [send] via NET/IB/15/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-12:52:235 [6] NCCL INFO Channel 01/0 : 101[5] -> 102[6] [receive] via NET/IB/5/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-22:50:371 [4] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-24:51:371 [5] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-14:49:232 [3] NCCL INFO Channel 03/0 : 115[3] -> 116[4] [send] via NET/IB/12/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-13:49:234 [3] NCCL INFO Channel 03/0 : 107[3] -> 108[4] [send] via NET/IB/12/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-16:47:234 [1] NCCL INFO Channel 01/0 : 129[1] -> 130[2] [send] via NET/IB/8/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-20:52:232 [6] NCCL INFO Channel 00/0 : 166[6] -> 167[7] [send] via NET/IB/4/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-19:52:234 [6] NCCL INFO Channel 02/0 : 158[6] -> 159[7] [send] via NET/IB/4/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-25:48:237 [2] NCCL INFO Channel 00/0 : 202[2] -> 203[3] [send] via NET/IB/10/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-15:52:237 [6] NCCL INFO Channel 02/0 : 126[6] -> 127[7] [send] via NET/IB/4/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-17:50:235 [4] NCCL INFO Channel 01/0 : 139[3] -> 140[4] [receive] via NET/IB/15/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-18:52:237 [6] NCCL INFO Channel 01/0 : 150[6] -> 151[7] [send] via NET/IB/5/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-12:50:232 [4] NCCL INFO Channel 02/0 : 99[3] -> 100[4] [receive] via NET/IB/14/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-22:48:366 [2] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-24:52:373 [6] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-14:47:367 [1] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-13:47:231 [1] NCCL INFO Channel 03/0 : 105[1] -> 106[2] [send] via NET/IB/8/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-16:49:233 [3] NCCL INFO Channel 01/0 : 131[3] -> 132[4] [send] via NET/IB/12/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-20:50:237 [4] NCCL INFO Channel 00/0 : 164[4] -> 165[5] [send] via NET/IB/14/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-19:49:236 [3] NCCL INFO Channel 00/0 : 155[3] -> 156[4] [send] via NET/IB/13/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-25:46:230 [0] NCCL INFO Channel 03/0 : 199[7] -> 200[0] [receive] via NET/IB/1/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-15:50:234 [4] NCCL INFO Channel 02/0 : 124[4] -> 125[5] [send] via NET/IB/14/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-17:50:235 [4] NCCL INFO Channel 02/0 : 139[3] -> 140[4] [receive] via NET/IB/14/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-18:49:234 [3] NCCL INFO Channel 03/0 : 147[3] -> 148[4] [send] via NET/IB/12/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-12:52:235 [6] NCCL INFO Channel 02/0 : 101[5] -> 102[6] [receive] via NET/IB/4/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-22:49:373 [3] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-24:50:372 [4] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-14:48:368 [2] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-13:48:366 [2] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-16:49:233 [3] NCCL INFO Channel 02/0 : 131[3] -> 132[4] [send] via NET/IB/13/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-20:52:232 [6] NCCL INFO Channel 01/0 : 166[6] -> 167[7] [send] via NET/IB/5/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-19:51:235 [5] NCCL INFO Channel 01/0 : 157[5] -> 158[6] [send] via NET/IB/2/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-25:48:237 [2] NCCL INFO Channel 01/0 : 202[2] -> 203[3] [send] via NET/IB/11/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-15:52:237 [6] NCCL INFO Channel 03/0 : 126[6] -> 127[7] [send] via NET/IB/5/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-17:50:235 [4] NCCL INFO Channel 03/0 : 139[3] -> 140[4] [receive] via NET/IB/15/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-18:50:236 [4] NCCL INFO Channel 02/0 : 148[4] -> 149[5] [send] via NET/IB/14/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-12:50:232 [4] NCCL INFO Channel 03/0 : 99[3] -> 100[4] [receive] via NET/IB/15/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-22:47:372 [1] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-24:53:370 [7] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-14:50:371 [4] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-13:50:369 [4] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-16:47:234 [1] NCCL INFO Channel 02/0 : 129[1] -> 130[2] [send] via NET/IB/9/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-20:50:237 [4] NCCL INFO Channel 01/0 : 164[4] -> 165[5] [send] via NET/IB/15/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-19:49:236 [3] NCCL INFO Channel 01/0 : 155[3] -> 156[4] [send] via NET/IB/12/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-25:46:230 [0] NCCL INFO Channel 00/0 : 200[0] -> 201[1] [send] via NET/IB/0/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-15:50:234 [4] NCCL INFO Channel 03/0 : 124[4] -> 125[5] [send] via NET/IB/15/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-17:50:235 [4] NCCL INFO Channel 00/0 : 140[4] -> 141[5] [send] via NET/IB/14/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-18:52:237 [6] NCCL INFO Channel 02/0 : 150[6] -> 151[7] [send] via NET/IB/4/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-12:52:235 [6] NCCL INFO Channel 03/0 : 101[5] -> 102[6] [receive] via NET/IB/5/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-14:53:369 [7] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-13:49:373 [3] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-16:49:233 [3] NCCL INFO Channel 03/0 : 131[3] -> 132[4] [send] via NET/IB/12/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-20:52:232 [6] NCCL INFO Channel 02/0 : 166[6] -> 167[7] [send] via NET/IB/4/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-19:52:234 [6] NCCL INFO Channel 03/0 : 158[6] -> 159[7] [send] via NET/IB/5/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-25:48:237 [2] NCCL INFO Channel 02/0 : 202[2] -> 203[3] [send] via NET/IB/10/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-15:49:367 [3] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-17:50:235 [4] NCCL INFO Channel 01/0 : 140[4] -> 141[5] [send] via NET/IB/15/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-18:50:236 [4] NCCL INFO Channel 03/0 : 148[4] -> 149[5] [send] via NET/IB/15/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-12:50:232 [4] NCCL INFO Channel 00/0 : 100[4] -> 101[5] [send] via NET/IB/14/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-14:49:373 [3] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-13:47:372 [1] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-16:47:234 [1] NCCL INFO Channel 03/0 : 129[1] -> 130[2] [send] via NET/IB/8/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-20:50:237 [4] NCCL INFO Channel 02/0 : 164[4] -> 165[5] [send] via NET/IB/14/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-19:51:235 [5] NCCL INFO Channel 02/0 : 157[5] -> 158[6] [send] via NET/IB/3/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-25:50:369 [4] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-15:52:372 [6] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-17:50:235 [4] NCCL INFO Channel 02/0 : 140[4] -> 141[5] [send] via NET/IB/14/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-18:52:237 [6] NCCL INFO Channel 03/0 : 150[6] -> 151[7] [send] via NET/IB/5/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-12:52:235 [6] NCCL INFO Channel 00/0 : 102[6] -> 103[7] [send] via NET/IB/4/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-13:46:367 [0] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-16:46:366 [0] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-20:52:232 [6] NCCL INFO Channel 03/0 : 166[6] -> 167[7] [send] via NET/IB/5/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-19:49:236 [3] NCCL INFO Channel 02/0 : 155[3] -> 156[4] [send] via NET/IB/13/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-25:46:230 [0] NCCL INFO Channel 01/0 : 200[0] -> 201[1] [send] via NET/IB/1/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-15:50:373 [4] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-17:50:235 [4] NCCL INFO Channel 03/0 : 140[4] -> 141[5] [send] via NET/IB/15/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-18:46:366 [0] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-12:50:232 [4] NCCL INFO Channel 01/0 : 100[4] -> 101[5] [send] via NET/IB/15/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-16:48:368 [2] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-20:50:237 [4] NCCL INFO Channel 03/0 : 164[4] -> 165[5] [send] via NET/IB/15/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-19:49:236 [3] NCCL INFO Channel 03/0 : 155[3] -> 156[4] [send] via NET/IB/12/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-25:48:237 [2] NCCL INFO Channel 03/0 : 202[2] -> 203[3] [send] via NET/IB/11/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-15:53:371 [7] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-17:53:371 [7] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-18:48:369 [2] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-12:52:235 [6] NCCL INFO Channel 01/0 : 102[6] -> 103[7] [send] via NET/IB/5/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-16:50:371 [4] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-20:46:370 [0] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-19:51:235 [5] NCCL INFO Channel 03/0 : 157[5] -> 158[6] [send] via NET/IB/2/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-25:52:368 [6] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-0:46:373 [0] NCCL INFO NCCL_IB_GID_INDEX set by environment to 3. | |
| nccl-alltoall-job-oguz-mpiworker-15:47:369 [1] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-17:49:369 [3] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-18:51:368 [5] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-12:50:232 [4] NCCL INFO Channel 02/0 : 100[4] -> 101[5] [send] via NET/IB/14/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-16:47:372 [1] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-20:48:371 [2] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-19:46:371 [0] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-25:47:234 [1] NCCL INFO Channel 00/0 : 200[0] -> 201[1] [receive] via NET/IB/9/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-0:48:372 [2] NCCL INFO NCCL_IB_GID_INDEX set by environment to 3. | |
| nccl-alltoall-job-oguz-mpiworker-15:51:370 [5] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-17:51:370 [5] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-18:50:371 [4] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-12:52:235 [6] NCCL INFO Channel 02/0 : 102[6] -> 103[7] [send] via NET/IB/4/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-16:49:373 [3] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-20:51:367 [5] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-19:47:372 [1] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-25:46:230 [0] NCCL INFO Channel 02/0 : 200[0] -> 201[1] [send] via NET/IB/0/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-17:50:372 [4] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-18:52:372 [6] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-12:50:232 [4] NCCL INFO Channel 03/0 : 100[4] -> 101[5] [send] via NET/IB/15/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-16:53:367 [7] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-20:52:369 [6] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-19:48:370 [2] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-25:46:230 [0] NCCL INFO Channel 03/0 : 200[0] -> 201[1] [send] via NET/IB/1/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-18:53:367 [7] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-12:52:235 [6] NCCL INFO Channel 03/0 : 102[6] -> 103[7] [send] via NET/IB/5/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-20:50:372 [4] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-19:52:368 [6] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-25:47:234 [1] NCCL INFO Channel 01/0 : 200[0] -> 201[1] [receive] via NET/IB/8/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-18:49:373 [3] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-12:49:369 [3] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-20:53:366 [7] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-19:51:367 [5] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-25:47:234 [1] NCCL INFO Channel 02/0 : 200[0] -> 201[1] [receive] via NET/IB/9/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-31:52:366 [6] NCCL INFO NCCL_IB_GID_INDEX set by environment to 3. | |
| nccl-alltoall-job-oguz-mpiworker-18:47:370 [1] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-12:51:370 [5] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-20:47:373 [1] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-19:50:369 [4] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-25:53:367 [7] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-12:52:373 [6] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-20:49:368 [3] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-19:49:373 [3] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-25:47:234 [1] NCCL INFO Channel 03/0 : 200[0] -> 201[1] [receive] via NET/IB/8/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-12:54:371 [7] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-19:53:366 [7] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-25:47:234 [1] NCCL INFO Channel 00/0 : 201[1] -> 202[2] [send] via NET/IB/9/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-12:50:372 [4] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-25:47:234 [1] NCCL INFO Channel 01/0 : 201[1] -> 202[2] [send] via NET/IB/8/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-25:47:234 [1] NCCL INFO Channel 02/0 : 201[1] -> 202[2] [send] via NET/IB/9/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-25:47:234 [1] NCCL INFO Channel 03/0 : 201[1] -> 202[2] [send] via NET/IB/8/GDRDMA | |
| nccl-alltoall-job-oguz-mpiworker-31:46:372 [0] NCCL INFO NCCL_IB_GID_INDEX set by environment to 3. | |
| nccl-alltoall-job-oguz-mpiworker-25:46:372 [0] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-25:48:371 [2] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-25:47:373 [1] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-25:51:366 [5] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-25:49:370 [3] NCCL INFO NCCL_IB_QPS_PER_CONNECTION set by environment to 16. | |
| nccl-alltoall-job-oguz-mpiworker-31:48:371 [2] NCCL INFO NCCL_IB_GID_INDEX set by environment to 3. | |
| nccl-alltoall-job-oguz-mpiworker-0:50:369 [4] NCCL INFO NCCL_IB_GID_INDEX set by environment to 3. | |
| nccl-alltoall-job-oguz-mpiworker-31:50:369 [4] NCCL INFO NCCL_IB_GID_INDEX set by environment to 3. | |
| nccl-alltoall-job-oguz-mpiworker-1:46:366 [0] NCCL INFO NCCL_IB_GID_INDEX set by environment to 3. | |
| nccl-alltoall-job-oguz-mpiworker-1:48:368 [2] NCCL INFO NCCL_IB_GID_INDEX set by environment to 3. | |
| nccl-alltoall-job-oguz-mpiworker-29:52:367 [6] NCCL INFO NCCL_IB_GID_INDEX set by environment to 3. | |
| nccl-alltoall-job-oguz-mpiworker-0:52:370 [6] NCCL INFO NCCL_IB_GID_INDEX set by environment to 3. | |
| nccl-alltoall-job-oguz-mpiworker-0:49:371 [3] NCCL INFO NCCL_IB_GID_INDEX set by environment to 3. | |
| nccl-alltoall-job-oguz-mpiworker-1:52:372 [6] NCCL INFO NCCL_IB_GID_INDEX set by environment to 3. | |
| nccl-alltoall-job-oguz-mpiworker-31:53:367 [7] NCCL INFO NCCL_IB_GID_INDEX set by environment to 3. | |
| nccl-alltoall-job-oguz-mpiworker-2:52:369 [6] NCCL INFO NCCL_IB_GID_INDEX set by environment to 3. | |
| nccl-alltoall-job-oguz-mpiworker-0:51:367 [5] NCCL INFO NCCL_IB_GID_INDEX set by environment to 3. | |
| nccl-alltoall-job-oguz-mpiworker-0:53:368 [7] NCCL INFO NCCL_IB_GID_INDEX set by environment to 3. | |
| nccl-alltoall-job-oguz-mpiworker-31:51:368 [5] NCCL INFO NCCL_IB_GID_INDEX set by environment to 3. | |
| nccl-alltoall-job-oguz-mpiworker-3:46:366 [0] NCCL INFO NCCL_IB_GID_INDEX set by environment to 3. | |
| nccl-alltoall-job-oguz-mpiworker-29:51:368 [5] NCCL INFO NCCL_IB_GID_INDEX set by environment to 3. | |
| nccl-alltoall-job-oguz-mpiworker-31:47:373 [1] NCCL INFO NCCL_IB_GID_INDEX set by environment to 3. | |
| nccl-alltoall-job-oguz-mpiworker-31:49:370 [3] NCCL INFO NCCL_IB_GID_INDEX set by environment to 3. | |
| nccl-alltoall-job-oguz-mpiworker-2:51:371 [5] NCCL INFO NCCL_IB_GID_INDEX set by environment to 3. | |
| nccl-alltoall-job-oguz-mpiworker-1:50:371 [4] NCCL INFO NCCL_IB_GID_INDEX set by environment to 3. | |
| nccl-alltoall-job-oguz-mpiworker-3:52:369 [6] NCCL INFO NCCL_IB_GID_INDEX set by environment to 3. | |
| nccl-alltoall-job-oguz-mpiworker-2:53:370 [7] NCCL INFO NCCL_IB_GID_INDEX set by environment to 3. | |
| nccl-alltoall-job-oguz-mpiworker-1:49:370 [3] NCCL INFO NCCL_IB_GID_INDEX set by environment to 3. | |
| nccl-alltoall-job-oguz-mpiworker-29:53:366 [7] NCCL INFO NCCL_IB_GID_INDEX set by environment to 3. | |
| nccl-alltoall-job-oguz-mpiworker-3:47:367 [1] NCCL INFO NCCL_IB_GID_INDEX set by environment to 3. | |
| nccl-alltoall-job-oguz-mpiworker-1:51:373 [5] NCCL INFO NCCL_IB_GID_INDEX set by environment to 3. | |
| nccl-alltoall-job-oguz-mpiworker-3:50:371 [4] NCCL INFO NCCL_IB_GID_INDEX set by environment to 3. | |
| nccl-alltoall-job-oguz-mpiworker-9:46:366 [0] NCCL INFO NCCL_IB_GID_INDEX set by environment to 3. | |
| nccl-alltoall-job-oguz-mpiworker-4:46:371 [0] NCCL INFO NCCL_IB_GID_INDEX set by environment to 3. | |
| nccl-alltoall-job-oguz-mpiworker-2:46:366 [0] NCCL INFO NCCL_IB_GID_INDEX set by environment to 3. | |
| nccl-alltoall-job-oguz-mpiworker-6:48:370 [2] NCCL INFO NCCL_IB_GID_INDEX set by environment to 3. | |
| nccl-alltoall-job-oguz-mpiworker-6:50:369 [4] NCCL INFO NCCL_IB_GID_INDEX set by environment to 3. | |
| nccl-alltoall-job-oguz-mpiworker-1:54:369 [7] NCCL INFO NCCL_IB_GID_INDEX set by environment to 3. | |
| nccl-alltoall-job-oguz-mpiworker-2:48:367 [2] NCCL INFO NCCL_IB_GID_INDEX set by environment to 3. | |
| nccl-alltoall-job-oguz-mpiworker-4:50:368 [4] NCCL INFO NCCL_IB_GID_INDEX set by environment to 3. | |
| nccl-alltoall-job-oguz-mpiworker-3:48:368 [2] NCCL INFO NCCL_IB_GID_INDEX set by environment to 3. | |
| nccl-alltoall-job-oguz-mpiworker-6:46:372 [0] NCCL INFO NCCL_IB_GID_INDEX set by environment to 3. | |
| nccl-alltoall-job-oguz-mpiworker-0:47:374 [1] NCCL INFO NCCL_IB_GID_INDEX set by environment to 3. | |
| nccl-alltoall-job-oguz-mpiworker-4:48:369 [2] NCCL INFO NCCL_IB_GID_INDEX set by environment to 3. | |
| nccl-alltoall-job-oguz-mpiworker-7:52:370 [6] NCCL INFO NCCL_IB_GID_INDEX set by environment to 3. | |
| nccl-alltoall-job-oguz-mpiworker-5:46:366 [0] NCCL INFO NCCL_IB_GID_INDEX set by environment to 3. | |
| nccl-alltoall-job-oguz-mpiworker-2:50:372 [4] NCCL INFO NCCL_IB_GID_INDEX set by environment to 3. | |
| nccl-alltoall-job-oguz-mpiworker-6:52:367 [6] NCCL INFO NCCL_IB_GID_INDEX set by environment to 3. | |
| nccl-all |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment