Created
July 24, 2019 01:10
-
-
Save jacobkahn/c9f1a3e5e0ce0491206346da25275625 to your computer and use it in GitHub Desktop.
~/mpirun.sh with 16 nodes
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| # nThread 1 nGpus 1 minBytes 8 maxBytes 1073741824 step: 2(factor) warmup iters: 5 iters: 100 validation: 1 | |
| # | |
| # Using devices | |
| # Rank 0 Pid 20303 on ip-172-31-39-141 device 0 [0x00] Tesla V100-SXM2-32GB | |
| # Rank 1 Pid 20304 on ip-172-31-39-141 device 1 [0x00] Tesla V100-SXM2-32GB | |
| # Rank 2 Pid 20305 on ip-172-31-39-141 device 2 [0x00] Tesla V100-SXM2-32GB | |
| # Rank 3 Pid 20306 on ip-172-31-39-141 device 3 [0x00] Tesla V100-SXM2-32GB | |
| # Rank 4 Pid 20307 on ip-172-31-39-141 device 4 [0x00] Tesla V100-SXM2-32GB | |
| # Rank 5 Pid 20308 on ip-172-31-39-141 device 5 [0x00] Tesla V100-SXM2-32GB | |
| # Rank 6 Pid 20309 on ip-172-31-39-141 device 6 [0x00] Tesla V100-SXM2-32GB | |
| # Rank 7 Pid 20310 on ip-172-31-39-141 device 7 [0x00] Tesla V100-SXM2-32GB | |
| # Rank 8 Pid 25298 on ip-172-31-38-14 device 0 [0x00] Tesla V100-SXM2-32GB | |
| # Rank 9 Pid 25299 on ip-172-31-38-14 device 1 [0x00] Tesla V100-SXM2-32GB | |
| # Rank 10 Pid 25300 on ip-172-31-38-14 device 2 [0x00] Tesla V100-SXM2-32GB | |
| # Rank 11 Pid 25301 on ip-172-31-38-14 device 3 [0x00] Tesla V100-SXM2-32GB | |
| # Rank 12 Pid 25302 on ip-172-31-38-14 device 4 [0x00] Tesla V100-SXM2-32GB | |
| # Rank 13 Pid 25303 on ip-172-31-38-14 device 5 [0x00] Tesla V100-SXM2-32GB | |
| # Rank 14 Pid 25304 on ip-172-31-38-14 device 6 [0x00] Tesla V100-SXM2-32GB | |
| # Rank 15 Pid 25306 on ip-172-31-38-14 device 7 [0x00] Tesla V100-SXM2-32GB | |
| # Rank 16 Pid 24929 on ip-172-31-44-142 device 0 [0x00] Tesla V100-SXM2-32GB | |
| # Rank 17 Pid 24930 on ip-172-31-44-142 device 1 [0x00] Tesla V100-SXM2-32GB | |
| # Rank 18 Pid 24931 on ip-172-31-44-142 device 2 [0x00] Tesla V100-SXM2-32GB | |
| # Rank 19 Pid 24932 on ip-172-31-44-142 device 3 [0x00] Tesla V100-SXM2-32GB | |
| # Rank 20 Pid 24933 on ip-172-31-44-142 device 4 [0x00] Tesla V100-SXM2-32GB | |
| # Rank 21 Pid 24934 on ip-172-31-44-142 device 5 [0x00] Tesla V100-SXM2-32GB | |
| # Rank 22 Pid 24935 on ip-172-31-44-142 device 6 [0x00] Tesla V100-SXM2-32GB | |
| # Rank 23 Pid 24937 on ip-172-31-44-142 device 7 [0x00] Tesla V100-SXM2-32GB | |
| # Rank 24 Pid 25058 on ip-172-31-45-171 device 0 [0x00] Tesla V100-SXM2-32GB | |
| # Rank 25 Pid 25059 on ip-172-31-45-171 device 1 [0x00] Tesla V100-SXM2-32GB | |
| # Rank 26 Pid 25060 on ip-172-31-45-171 device 2 [0x00] Tesla V100-SXM2-32GB | |
| # Rank 27 Pid 25061 on ip-172-31-45-171 device 3 [0x00] Tesla V100-SXM2-32GB | |
| # Rank 28 Pid 25062 on ip-172-31-45-171 device 4 [0x00] Tesla V100-SXM2-32GB | |
| # Rank 29 Pid 25063 on ip-172-31-45-171 device 5 [0x00] Tesla V100-SXM2-32GB | |
| # Rank 30 Pid 25064 on ip-172-31-45-171 device 6 [0x00] Tesla V100-SXM2-32GB | |
| # Rank 31 Pid 25065 on ip-172-31-45-171 device 7 [0x00] Tesla V100-SXM2-32GB | |
| # Rank 32 Pid 25824 on ip-172-31-37-141 device 0 [0x00] Tesla V100-SXM2-32GB | |
| # Rank 33 Pid 25825 on ip-172-31-37-141 device 1 [0x00] Tesla V100-SXM2-32GB | |
| # Rank 34 Pid 25826 on ip-172-31-37-141 device 2 [0x00] Tesla V100-SXM2-32GB | |
| # Rank 35 Pid 25827 on ip-172-31-37-141 device 3 [0x00] Tesla V100-SXM2-32GB | |
| # Rank 36 Pid 25828 on ip-172-31-37-141 device 4 [0x00] Tesla V100-SXM2-32GB | |
| # Rank 37 Pid 25829 on ip-172-31-37-141 device 5 [0x00] Tesla V100-SXM2-32GB | |
| # Rank 38 Pid 25830 on ip-172-31-37-141 device 6 [0x00] Tesla V100-SXM2-32GB | |
| # Rank 39 Pid 25831 on ip-172-31-37-141 device 7 [0x00] Tesla V100-SXM2-32GB | |
| # Rank 40 Pid 24978 on ip-172-31-38-82 device 0 [0x00] Tesla V100-SXM2-32GB | |
| # Rank 41 Pid 24979 on ip-172-31-38-82 device 1 [0x00] Tesla V100-SXM2-32GB | |
| # Rank 42 Pid 24980 on ip-172-31-38-82 device 2 [0x00] Tesla V100-SXM2-32GB | |
| # Rank 43 Pid 24981 on ip-172-31-38-82 device 3 [0x00] Tesla V100-SXM2-32GB | |
| # Rank 44 Pid 24982 on ip-172-31-38-82 device 4 [0x00] Tesla V100-SXM2-32GB | |
| # Rank 45 Pid 24983 on ip-172-31-38-82 device 5 [0x00] Tesla V100-SXM2-32GB | |
| # Rank 46 Pid 24984 on ip-172-31-38-82 device 6 [0x00] Tesla V100-SXM2-32GB | |
| # Rank 47 Pid 24985 on ip-172-31-38-82 device 7 [0x00] Tesla V100-SXM2-32GB | |
| # Rank 48 Pid 24691 on ip-172-31-32-86 device 0 [0x00] Tesla V100-SXM2-32GB | |
| # Rank 49 Pid 24692 on ip-172-31-32-86 device 1 [0x00] Tesla V100-SXM2-32GB | |
| # Rank 50 Pid 24693 on ip-172-31-32-86 device 2 [0x00] Tesla V100-SXM2-32GB | |
| # Rank 51 Pid 24694 on ip-172-31-32-86 device 3 [0x00] Tesla V100-SXM2-32GB | |
| # Rank 52 Pid 24695 on ip-172-31-32-86 device 4 [0x00] Tesla V100-SXM2-32GB | |
| # Rank 53 Pid 24696 on ip-172-31-32-86 device 5 [0x00] Tesla V100-SXM2-32GB | |
| # Rank 54 Pid 24697 on ip-172-31-32-86 device 6 [0x00] Tesla V100-SXM2-32GB | |
| # Rank 55 Pid 24698 on ip-172-31-32-86 device 7 [0x00] Tesla V100-SXM2-32GB | |
| # Rank 56 Pid 24689 on ip-172-31-47-240 device 0 [0x00] Tesla V100-SXM2-32GB | |
| # Rank 57 Pid 24690 on ip-172-31-47-240 device 1 [0x00] Tesla V100-SXM2-32GB | |
| # Rank 58 Pid 24691 on ip-172-31-47-240 device 2 [0x00] Tesla V100-SXM2-32GB | |
| # Rank 59 Pid 24692 on ip-172-31-47-240 device 3 [0x00] Tesla V100-SXM2-32GB | |
| # Rank 60 Pid 24693 on ip-172-31-47-240 device 4 [0x00] Tesla V100-SXM2-32GB | |
| # Rank 61 Pid 24694 on ip-172-31-47-240 device 5 [0x00] Tesla V100-SXM2-32GB | |
| # Rank 62 Pid 24695 on ip-172-31-47-240 device 6 [0x00] Tesla V100-SXM2-32GB | |
| # Rank 63 Pid 24696 on ip-172-31-47-240 device 7 [0x00] Tesla V100-SXM2-32GB | |
| # Rank 64 Pid 22040 on ip-172-31-46-50 device 0 [0x00] Tesla V100-SXM2-32GB | |
| # Rank 65 Pid 22041 on ip-172-31-46-50 device 1 [0x00] Tesla V100-SXM2-32GB | |
| # Rank 66 Pid 22042 on ip-172-31-46-50 device 2 [0x00] Tesla V100-SXM2-32GB | |
| # Rank 67 Pid 22043 on ip-172-31-46-50 device 3 [0x00] Tesla V100-SXM2-32GB | |
| # Rank 68 Pid 22044 on ip-172-31-46-50 device 4 [0x00] Tesla V100-SXM2-32GB | |
| # Rank 69 Pid 22045 on ip-172-31-46-50 device 5 [0x00] Tesla V100-SXM2-32GB | |
| # Rank 70 Pid 22046 on ip-172-31-46-50 device 6 [0x00] Tesla V100-SXM2-32GB | |
| # Rank 71 Pid 22047 on ip-172-31-46-50 device 7 [0x00] Tesla V100-SXM2-32GB | |
| # Rank 72 Pid 22008 on ip-172-31-47-188 device 0 [0x00] Tesla V100-SXM2-32GB | |
| # Rank 73 Pid 22009 on ip-172-31-47-188 device 1 [0x00] Tesla V100-SXM2-32GB | |
| # Rank 74 Pid 22010 on ip-172-31-47-188 device 2 [0x00] Tesla V100-SXM2-32GB | |
| # Rank 75 Pid 22011 on ip-172-31-47-188 device 3 [0x00] Tesla V100-SXM2-32GB | |
| # Rank 76 Pid 22012 on ip-172-31-47-188 device 4 [0x00] Tesla V100-SXM2-32GB | |
| # Rank 77 Pid 22013 on ip-172-31-47-188 device 5 [0x00] Tesla V100-SXM2-32GB | |
| # Rank 78 Pid 22014 on ip-172-31-47-188 device 6 [0x00] Tesla V100-SXM2-32GB | |
| # Rank 79 Pid 22015 on ip-172-31-47-188 device 7 [0x00] Tesla V100-SXM2-32GB | |
| # Rank 80 Pid 22560 on ip-172-31-47-156 device 0 [0x00] Tesla V100-SXM2-32GB | |
| # Rank 81 Pid 22561 on ip-172-31-47-156 device 1 [0x00] Tesla V100-SXM2-32GB | |
| # Rank 82 Pid 22562 on ip-172-31-47-156 device 2 [0x00] Tesla V100-SXM2-32GB | |
| # Rank 83 Pid 22563 on ip-172-31-47-156 device 3 [0x00] Tesla V100-SXM2-32GB | |
| # Rank 84 Pid 22564 on ip-172-31-47-156 device 4 [0x00] Tesla V100-SXM2-32GB | |
| # Rank 85 Pid 22565 on ip-172-31-47-156 device 5 [0x00] Tesla V100-SXM2-32GB | |
| # Rank 86 Pid 22566 on ip-172-31-47-156 device 6 [0x00] Tesla V100-SXM2-32GB | |
| # Rank 87 Pid 22569 on ip-172-31-47-156 device 7 [0x00] Tesla V100-SXM2-32GB | |
| # Rank 88 Pid 22207 on ip-172-31-36-188 device 0 [0x00] Tesla V100-SXM2-32GB | |
| # Rank 89 Pid 22208 on ip-172-31-36-188 device 1 [0x00] Tesla V100-SXM2-32GB | |
| # Rank 90 Pid 22209 on ip-172-31-36-188 device 2 [0x00] Tesla V100-SXM2-32GB | |
| # Rank 91 Pid 22210 on ip-172-31-36-188 device 3 [0x00] Tesla V100-SXM2-32GB | |
| # Rank 92 Pid 22211 on ip-172-31-36-188 device 4 [0x00] Tesla V100-SXM2-32GB | |
| # Rank 93 Pid 22212 on ip-172-31-36-188 device 5 [0x00] Tesla V100-SXM2-32GB | |
| # Rank 94 Pid 22213 on ip-172-31-36-188 device 6 [0x00] Tesla V100-SXM2-32GB | |
| # Rank 95 Pid 22214 on ip-172-31-36-188 device 7 [0x00] Tesla V100-SXM2-32GB | |
| # Rank 96 Pid 21987 on ip-172-31-44-230 device 0 [0x00] Tesla V100-SXM2-32GB | |
| # Rank 97 Pid 21988 on ip-172-31-44-230 device 1 [0x00] Tesla V100-SXM2-32GB | |
| # Rank 98 Pid 21989 on ip-172-31-44-230 device 2 [0x00] Tesla V100-SXM2-32GB | |
| # Rank 99 Pid 21990 on ip-172-31-44-230 device 3 [0x00] Tesla V100-SXM2-32GB | |
| # Rank 100 Pid 21991 on ip-172-31-44-230 device 4 [0x00] Tesla V100-SXM2-32GB | |
| # Rank 101 Pid 21992 on ip-172-31-44-230 device 5 [0x00] Tesla V100-SXM2-32GB | |
| # Rank 102 Pid 21993 on ip-172-31-44-230 device 6 [0x00] Tesla V100-SXM2-32GB | |
| # Rank 103 Pid 21994 on ip-172-31-44-230 device 7 [0x00] Tesla V100-SXM2-32GB | |
| # Rank 104 Pid 21959 on ip-172-31-33-235 device 0 [0x00] Tesla V100-SXM2-32GB | |
| # Rank 105 Pid 21960 on ip-172-31-33-235 device 1 [0x00] Tesla V100-SXM2-32GB | |
| # Rank 106 Pid 21961 on ip-172-31-33-235 device 2 [0x00] Tesla V100-SXM2-32GB | |
| # Rank 107 Pid 21962 on ip-172-31-33-235 device 3 [0x00] Tesla V100-SXM2-32GB | |
| # Rank 108 Pid 21963 on ip-172-31-33-235 device 4 [0x00] Tesla V100-SXM2-32GB | |
| # Rank 109 Pid 21964 on ip-172-31-33-235 device 5 [0x00] Tesla V100-SXM2-32GB | |
| # Rank 110 Pid 21965 on ip-172-31-33-235 device 6 [0x00] Tesla V100-SXM2-32GB | |
| # Rank 111 Pid 21966 on ip-172-31-33-235 device 7 [0x00] Tesla V100-SXM2-32GB | |
| # Rank 112 Pid 22021 on ip-172-31-36-128 device 0 [0x00] Tesla V100-SXM2-32GB | |
| # Rank 113 Pid 22022 on ip-172-31-36-128 device 1 [0x00] Tesla V100-SXM2-32GB | |
| # Rank 114 Pid 22023 on ip-172-31-36-128 device 2 [0x00] Tesla V100-SXM2-32GB | |
| # Rank 115 Pid 22024 on ip-172-31-36-128 device 3 [0x00] Tesla V100-SXM2-32GB | |
| # Rank 116 Pid 22025 on ip-172-31-36-128 device 4 [0x00] Tesla V100-SXM2-32GB | |
| # Rank 117 Pid 22026 on ip-172-31-36-128 device 5 [0x00] Tesla V100-SXM2-32GB | |
| # Rank 118 Pid 22027 on ip-172-31-36-128 device 6 [0x00] Tesla V100-SXM2-32GB | |
| # Rank 119 Pid 22030 on ip-172-31-36-128 device 7 [0x00] Tesla V100-SXM2-32GB | |
| # Rank 120 Pid 22492 on ip-172-31-44-4 device 0 [0x00] Tesla V100-SXM2-32GB | |
| # Rank 121 Pid 22493 on ip-172-31-44-4 device 1 [0x00] Tesla V100-SXM2-32GB | |
| # Rank 122 Pid 22494 on ip-172-31-44-4 device 2 [0x00] Tesla V100-SXM2-32GB | |
| # Rank 123 Pid 22495 on ip-172-31-44-4 device 3 [0x00] Tesla V100-SXM2-32GB | |
| # Rank 124 Pid 22496 on ip-172-31-44-4 device 4 [0x00] Tesla V100-SXM2-32GB | |
| # Rank 125 Pid 22497 on ip-172-31-44-4 device 5 [0x00] Tesla V100-SXM2-32GB | |
| # Rank 126 Pid 22498 on ip-172-31-44-4 device 6 [0x00] Tesla V100-SXM2-32GB | |
| # Rank 127 Pid 22499 on ip-172-31-44-4 device 7 [0x00] Tesla V100-SXM2-32GB | |
| ip-172-31-39-141:20303:20303 [0] NCCL INFO NET/Socket : Using [0]eth0:172.31.39.141<0> | |
| ip-172-31-39-141:20303:20303 [0] NCCL INFO NET/OFI Forcing AWS OFI ndev 4 | |
| ip-172-31-39-141:20303:20303 [0] NCCL INFO NET/OFI Selected Provider is efa;ofi_rxr | |
| ip-172-31-33-235:21961:21961 [2] NCCL INFO NET/Socket : Using [0]eth0:172.31.33.235<0> | |
| ip-172-31-33-235:21961:21961 [2] NCCL INFO NET/OFI Forcing AWS OFI ndev 4 | |
| ip-172-31-33-235:21961:21961 [2] NCCL INFO NET/OFI Selected Provider is efa;ofi_rxr | |
| ip-172-31-36-128:22024:22024 [3] NCCL INFO NET/Socket : Using [0]eth0:172.31.36.128<0> | |
| ip-172-31-36-128:22024:22024 [3] NCCL INFO NET/OFI Forcing AWS OFI ndev 4 | |
| ip-172-31-36-128:22024:22024 [3] NCCL INFO NET/OFI Selected Provider is efa;ofi_rxr | |
| NCCL version 2.4.6+cuda10.0 | |
| ip-172-31-32-86:24698:24698 [7] NCCL INFO NET/Socket : Using [0]eth0:172.31.32.86<0> | |
| ip-172-31-32-86:24698:24698 [7] NCCL INFO NET/OFI Forcing AWS OFI ndev 4 | |
| ip-172-31-32-86:24698:24698 [7] NCCL INFO NET/OFI Selected Provider is efa;ofi_rxr | |
| ip-172-31-32-86:24691:24691 [0] NCCL INFO NET/Socket : Using [0]eth0:172.31.32.86<0> | |
| ip-172-31-32-86:24691:24691 [0] NCCL INFO NET/OFI Forcing AWS OFI ndev 4 | |
| ip-172-31-32-86:24691:24691 [0] NCCL INFO NET/OFI Selected Provider is efa;ofi_rxr | |
| ip-172-31-37-141:25826:25826 [2] NCCL INFO NET/Socket : Using [0]eth0:172.31.37.141<0> | |
| ip-172-31-37-141:25831:25831 [7] NCCL INFO NET/Socket : Using [0]eth0:172.31.37.141<0> | |
| ip-172-31-37-141:25826:25826 [2] NCCL INFO NET/OFI Forcing AWS OFI ndev 4 | |
| ip-172-31-37-141:25826:25826 [2] NCCL INFO NET/OFI Selected Provider is efa;ofi_rxr | |
| ip-172-31-37-141:25831:25831 [7] NCCL INFO NET/OFI Forcing AWS OFI ndev 4 | |
| ip-172-31-37-141:25831:25831 [7] NCCL INFO NET/OFI Selected Provider is efa;ofi_rxr | |
| ip-172-31-36-188:22209:22209 [2] NCCL INFO NET/Socket : Using [0]eth0:172.31.36.188<0> | |
| ip-172-31-36-188:22214:22214 [7] NCCL INFO NET/Socket : Using [0]eth0:172.31.36.188<0> | |
| ip-172-31-36-188:22214:22214 [7] NCCL INFO NET/OFI Forcing AWS OFI ndev 4 | |
| ip-172-31-36-188:22214:22214 [7] NCCL INFO NET/OFI Selected Provider is efa;ofi_rxr | |
| ip-172-31-47-240:24692:24692 [3] NCCL INFO NET/Socket : Using [0]eth0:172.31.47.240<0> | |
| ip-172-31-36-188:22209:22209 [2] NCCL INFO NET/OFI Forcing AWS OFI ndev 4 | |
| ip-172-31-36-188:22209:22209 [2] NCCL INFO NET/OFI Selected Provider is efa;ofi_rxr | |
| ip-172-31-47-240:24691:24691 [2] NCCL INFO NET/Socket : Using [0]eth0:172.31.47.240<0> | |
| ip-172-31-47-240:24691:24691 [2] NCCL INFO NET/OFI Forcing AWS OFI ndev 4 | |
| ip-172-31-47-240:24691:24691 [2] NCCL INFO NET/OFI Selected Provider is efa;ofi_rxr | |
| ip-172-31-47-240:24692:24692 [3] NCCL INFO NET/OFI Forcing AWS OFI ndev 4 | |
| ip-172-31-47-240:24692:24692 [3] NCCL INFO NET/OFI Selected Provider is efa;ofi_rxr | |
| ip-172-31-47-240:24695:24695 [6] NCCL INFO NET/Socket : Using [0]eth0:172.31.47.240<0> | |
| ip-172-31-47-240:24695:24695 [6] NCCL INFO NET/OFI Forcing AWS OFI ndev 4 | |
| ip-172-31-47-240:24695:24695 [6] NCCL INFO NET/OFI Selected Provider is efa;ofi_rxr | |
| ip-172-31-45-171:25058:25058 [0] NCCL INFO NET/Socket : Using [0]eth0:172.31.45.171<0> | |
| ip-172-31-45-171:25064:25064 [6] NCCL INFO NET/Socket : Using [0]eth0:172.31.45.171<0> | |
| ip-172-31-45-171:25060:25060 [2] NCCL INFO NET/Socket : Using [0]eth0:172.31.45.171<0> | |
| ip-172-31-47-240:24696:24696 [7] NCCL INFO NET/Socket : Using [0]eth0:172.31.47.240<0> | |
| ip-172-31-47-240:24696:24696 [7] NCCL INFO NET/OFI Forcing AWS OFI ndev 4 | |
| ip-172-31-47-240:24696:24696 [7] NCCL INFO NET/OFI Selected Provider is efa;ofi_rxr | |
| ip-172-31-45-171:25064:25064 [6] NCCL INFO NET/OFI Forcing AWS OFI ndev 4 | |
| ip-172-31-45-171:25064:25064 [6] NCCL INFO NET/OFI Selected Provider is efa;ofi_rxr | |
| ip-172-31-45-171:25060:25060 [2] NCCL INFO NET/OFI Forcing AWS OFI ndev 4 | |
| ip-172-31-45-171:25060:25060 [2] NCCL INFO NET/OFI Selected Provider is efa;ofi_rxr | |
| ip-172-31-45-171:25058:25058 [0] NCCL INFO NET/OFI Forcing AWS OFI ndev 4 | |
| ip-172-31-45-171:25058:25058 [0] NCCL INFO NET/OFI Selected Provider is efa;ofi_rxr | |
| ip-172-31-32-86:24693:24693 [2] NCCL INFO NET/Socket : Using [0]eth0:172.31.32.86<0> | |
| ip-172-31-32-86:24692:24692 [1] NCCL INFO NET/Socket : Using [0]eth0:172.31.32.86<0> | |
| ip-172-31-32-86:24694:24694 [3] NCCL INFO NET/Socket : Using [0]eth0:172.31.32.86<0> | |
| ip-172-31-32-86:24693:24693 [2] NCCL INFO NET/OFI Forcing AWS OFI ndev 4 | |
| ip-172-31-32-86:24693:24693 [2] NCCL INFO NET/OFI Selected Provider is efa;ofi_rxr | |
| ip-172-31-32-86:24692:24692 [1] NCCL INFO NET/OFI Forcing AWS OFI ndev 4 | |
| ip-172-31-32-86:24692:24692 [1] NCCL INFO NET/OFI Selected Provider is efa;ofi_rxr | |
| ip-172-31-32-86:24694:24694 [3] NCCL INFO NET/OFI Forcing AWS OFI ndev 4 | |
| ip-172-31-32-86:24694:24694 [3] NCCL INFO NET/OFI Selected Provider is efa;ofi_rxr | |
| ip-172-31-47-156:22564:22564 [4] NCCL INFO NET/Socket : Using [0]eth0:172.31.47.156<0> | |
| ip-172-31-47-156:22562:22562 [2] NCCL INFO NET/Socket : Using [0]eth0:172.31.47.156<0> | |
| ip-172-31-47-156:22564:22564 [4] NCCL INFO NET/OFI Forcing AWS OFI ndev 4 | |
| ip-172-31-47-156:22564:22564 [4] NCCL INFO NET/OFI Selected Provider is efa;ofi_rxr | |
| ip-172-31-47-156:22562:22562 [2] NCCL INFO NET/OFI Forcing AWS OFI ndev 4 | |
| ip-172-31-47-156:22562:22562 [2] NCCL INFO NET/OFI Selected Provider is efa;ofi_rxr | |
| ip-172-31-32-86:24697:24697 [6] NCCL INFO NET/Socket : Using [0]eth0:172.31.32.86<0> | |
| ip-172-31-32-86:24695:24695 [4] NCCL INFO NET/Socket : Using [0]eth0:172.31.32.86<0> | |
| ip-172-31-32-86:24696:24696 [5] NCCL INFO NET/Socket : Using [0]eth0:172.31.32.86<0> | |
| ip-172-31-32-86:24697:24697 [6] NCCL INFO NET/OFI Forcing AWS OFI ndev 4 | |
| ip-172-31-32-86:24697:24697 [6] NCCL INFO NET/OFI Selected Provider is efa;ofi_rxr | |
| ip-172-31-32-86:24695:24695 [4] NCCL INFO NET/OFI Forcing AWS OFI ndev 4 | |
| ip-172-31-32-86:24695:24695 [4] NCCL INFO NET/OFI Selected Provider is efa;ofi_rxr | |
| ip-172-31-47-240:24694:24694 [5] NCCL INFO NET/Socket : Using [0]eth0:172.31.47.240<0> | |
| ip-172-31-32-86:24696:24696 [5] NCCL INFO NET/OFI Forcing AWS OFI ndev 4 | |
| ip-172-31-32-86:24696:24696 [5] NCCL INFO NET/OFI Selected Provider is efa;ofi_rxr | |
| ip-172-31-47-240:24694:24694 [5] NCCL INFO NET/OFI Forcing AWS OFI ndev 4 | |
| ip-172-31-47-240:24694:24694 [5] NCCL INFO NET/OFI Selected Provider is efa;ofi_rxr | |
| ip-172-31-47-240:24690:24690 [1] NCCL INFO NET/Socket : Using [0]eth0:172.31.47.240<0> | |
| ip-172-31-47-240:24689:24689 [0] NCCL INFO NET/Socket : Using [0]eth0:172.31.47.240<0> | |
| ip-172-31-47-240:24690:24690 [1] NCCL INFO NET/OFI Forcing AWS OFI ndev 4 | |
| ip-172-31-47-240:24690:24690 [1] NCCL INFO NET/OFI Selected Provider is efa;ofi_rxr | |
| ip-172-31-47-240:24689:24689 [0] NCCL INFO NET/OFI Forcing AWS OFI ndev 4 | |
| ip-172-31-47-240:24689:24689 [0] NCCL INFO NET/OFI Selected Provider is efa;ofi_rxr | |
| ip-172-31-47-240:24693:24693 [4] NCCL INFO NET/Socket : Using [0]eth0:172.31.47.240<0> | |
| ip-172-31-47-240:24693:24693 [4] NCCL INFO NET/OFI Forcing AWS OFI ndev 4 | |
| ip-172-31-47-240:24693:24693 [4] NCCL INFO NET/OFI Selected Provider is efa;ofi_rxr | |
| ip-172-31-45-171:25065:25065 [7] NCCL INFO NET/Socket : Using [0]eth0:172.31.45.171<0> | |
| ip-172-31-45-171:25061:25061 [3] NCCL INFO NET/Socket : Using [0]eth0:172.31.45.171<0> | |
| ip-172-31-45-171:25065:25065 [7] NCCL INFO NET/OFI Forcing AWS OFI ndev 4 | |
| ip-172-31-45-171:25065:25065 [7] NCCL INFO NET/OFI Selected Provider is efa;ofi_rxr | |
| ip-172-31-47-188:22012:22012 [4] NCCL INFO NET/Socket : Using [0]eth0:172.31.47.188<0> | |
| ip-172-31-45-171:25061:25061 [3] NCCL INFO NET/OFI Forcing AWS OFI ndev 4 | |
| ip-172-31-45-171:25061:25061 [3] NCCL INFO NET/OFI Selected Provider is efa;ofi_rxr | |
| ip-172-31-47-188:22015:22015 [7] NCCL INFO NET/Socket : Using [0]eth0:172.31.47.188<0> | |
| ip-172-31-45-171:25063:25063 [5] NCCL INFO NET/Socket : Using [0]eth0:172.31.45.171<0> | |
| ip-172-31-47-188:22013:22013 [5] NCCL INFO NET/Socket : Using [0]eth0:172.31.47.188<0> | |
| ip-172-31-47-188:22008:22008 [0] NCCL INFO NET/Socket : Using [0]eth0:172.31.47.188<0> | |
| ip-172-31-47-188:22009:22009 [1] NCCL INFO NET/Socket : Using [0]eth0:172.31.47.188<0> | |
| ip-172-31-37-141:25827:25827 [3] NCCL INFO NET/Socket : Using [0]eth0:172.31.37.141<0> | |
| ip-172-31-37-141:25825:25825 [1] NCCL INFO NET/Socket : Using [0]eth0:172.31.37.141<0> | |
| ip-172-31-45-171:25062:25062 [4] NCCL INFO NET/Socket : Using [0]eth0:172.31.45.171<0> | |
| ip-172-31-45-171:25063:25063 [5] NCCL INFO NET/OFI Forcing AWS OFI ndev 4 | |
| ip-172-31-45-171:25063:25063 [5] NCCL INFO NET/OFI Selected Provider is efa;ofi_rxr | |
| ip-172-31-47-188:22014:22014 [6] NCCL INFO NET/Socket : Using [0]eth0:172.31.47.188<0> | |
| ip-172-31-37-141:25827:25827 [3] NCCL INFO NET/OFI Forcing AWS OFI ndev 4 | |
| ip-172-31-37-141:25827:25827 [3] NCCL INFO NET/OFI Selected Provider is efa;ofi_rxr | |
| ip-172-31-47-188:22010:22010 [2] NCCL INFO NET/Socket : Using [0]eth0:172.31.47.188<0> | |
| ip-172-31-37-141:25830:25830 [6] NCCL INFO NET/Socket : Using [0]eth0:172.31.37.141<0> | |
| ip-172-31-47-188:22015:22015 [7] NCCL INFO NET/OFI Forcing AWS OFI ndev 4 | |
| ip-172-31-47-188:22015:22015 [7] NCCL INFO NET/OFI Selected Provider is efa;ofi_rxr | |
| ip-172-31-37-141:25825:25825 [1] NCCL INFO NET/OFI Forcing AWS OFI ndev 4 | |
| ip-172-31-37-141:25825:25825 [1] NCCL INFO NET/OFI Selected Provider is efa;ofi_rxr | |
| ip-172-31-47-188:22012:22012 [4] NCCL INFO NET/OFI Forcing AWS OFI ndev 4 | |
| ip-172-31-47-188:22012:22012 [4] NCCL INFO NET/OFI Selected Provider is efa;ofi_rxr | |
| ip-172-31-37-141:25829:25829 [5] NCCL INFO NET/Socket : Using [0]eth0:172.31.37.141<0> | |
| ip-172-31-47-188:22011:22011 [3] NCCL INFO NET/Socket : Using [0]eth0:172.31.47.188<0> | |
| ip-172-31-47-188:22013:22013 [5] NCCL INFO NET/OFI Forcing AWS OFI ndev 4 | |
| ip-172-31-47-188:22013:22013 [5] NCCL INFO NET/OFI Selected Provider is efa;ofi_rxr | |
| ip-172-31-45-171:25062:25062 [4] NCCL INFO NET/OFI Forcing AWS OFI ndev 4 | |
| ip-172-31-45-171:25062:25062 [4] NCCL INFO NET/OFI Selected Provider is efa;ofi_rxr | |
| ip-172-31-47-188:22008:22008 [0] NCCL INFO NET/OFI Forcing AWS OFI ndev 4 | |
| ip-172-31-47-188:22008:22008 [0] NCCL INFO NET/OFI Selected Provider is efa;ofi_rxr | |
| ip-172-31-37-141:25824:25824 [0] NCCL INFO NET/Socket : Using [0]eth0:172.31.37.141<0> | |
| ip-172-31-47-188:22009:22009 [1] NCCL INFO NET/OFI Forcing AWS OFI ndev 4 | |
| ip-172-31-47-188:22009:22009 [1] NCCL INFO NET/OFI Selected Provider is efa;ofi_rxr | |
| ip-172-31-37-141:25828:25828 [4] NCCL INFO NET/Socket : Using [0]eth0:172.31.37.141<0> | |
| ip-172-31-33-235:21959:21959 [0] NCCL INFO NET/Socket : Using [0]eth0:172.31.33.235<0> | |
| ip-172-31-47-188:22014:22014 [6] NCCL INFO NET/OFI Forcing AWS OFI ndev 4 | |
| ip-172-31-47-188:22014:22014 [6] NCCL INFO NET/OFI Selected Provider is efa;ofi_rxr | |
| ip-172-31-47-188:22010:22010 [2] NCCL INFO NET/OFI Forcing AWS OFI ndev 4 | |
| ip-172-31-47-188:22010:22010 [2] NCCL INFO NET/OFI Selected Provider is efa;ofi_rxr | |
| ip-172-31-37-141:25830:25830 [6] NCCL INFO NET/OFI Forcing AWS OFI ndev 4 | |
| ip-172-31-37-141:25830:25830 [6] NCCL INFO NET/OFI Selected Provider is efa;ofi_rxr | |
| ip-172-31-47-188:22011:22011 [3] NCCL INFO NET/OFI Forcing AWS OFI ndev 4 | |
| ip-172-31-47-188:22011:22011 [3] NCCL INFO NET/OFI Selected Provider is efa;ofi_rxr | |
| ip-172-31-37-141:25829:25829 [5] NCCL INFO NET/OFI Forcing AWS OFI ndev 4 | |
| ip-172-31-37-141:25829:25829 [5] NCCL INFO NET/OFI Selected Provider is efa;ofi_rxr | |
| ip-172-31-33-235:21959:21959 [0] NCCL INFO NET/OFI Forcing AWS OFI ndev 4 | |
| ip-172-31-33-235:21959:21959 [0] NCCL INFO NET/OFI Selected Provider is efa;ofi_rxr | |
| ip-172-31-37-141:25824:25824 [0] NCCL INFO NET/OFI Forcing AWS OFI ndev 4 | |
| ip-172-31-37-141:25824:25824 [0] NCCL INFO NET/OFI Selected Provider is efa;ofi_rxr | |
| ip-172-31-37-141:25828:25828 [4] NCCL INFO NET/OFI Forcing AWS OFI ndev 4 | |
| ip-172-31-37-141:25828:25828 [4] NCCL INFO NET/OFI Selected Provider is efa;ofi_rxr | |
| ip-172-31-33-235:21960:21960 [1] NCCL INFO NET/Socket : Using [0]eth0:172.31.33.235<0> | |
| ip-172-31-33-235:21966:21966 [7] NCCL INFO NET/Socket : Using [0]eth0:172.31.33.235<0> | |
| ip-172-31-33-235:21963:21963 [4] NCCL INFO NET/Socket : Using [0]eth0:172.31.33.235<0> | |
| ip-172-31-45-171:25059:25059 [1] NCCL INFO NET/Socket : Using [0]eth0:172.31.45.171<0> | |
| ip-172-31-33-235:21962:21962 [3] NCCL INFO NET/Socket : Using [0]eth0:172.31.33.235<0> | |
| ip-172-31-33-235:21960:21960 [1] NCCL INFO NET/OFI Forcing AWS OFI ndev 4 | |
| ip-172-31-33-235:21960:21960 [1] NCCL INFO NET/OFI Selected Provider is efa;ofi_rxr | |
| ip-172-31-39-141:20307:20307 [4] NCCL INFO NET/Socket : Using [0]eth0:172.31.39.141<0> | |
| ip-172-31-36-128:22027:22027 [6] NCCL INFO NET/Socket : Using [0]eth0:172.31.36.128<0> | |
| ip-172-31-33-235:21964:21964 [5] NCCL INFO NET/Socket : Using [0]eth0:172.31.33.235<0> | |
| ip-172-31-33-235:21966:21966 [7] NCCL INFO NET/OFI Forcing AWS OFI ndev 4 | |
| ip-172-31-33-235:21966:21966 [7] NCCL INFO NET/OFI Selected Provider is efa;ofi_rxr | |
| ip-172-31-33-235:21965:21965 [6] NCCL INFO NET/Socket : Using [0]eth0:172.31.33.235<0> | |
| ip-172-31-33-235:21963:21963 [4] NCCL INFO NET/OFI Forcing AWS OFI ndev 4 | |
| ip-172-31-33-235:21963:21963 [4] NCCL INFO NET/OFI Selected Provider is efa;ofi_rxr | |
| ip-172-31-33-235:21962:21962 [3] NCCL INFO NET/OFI Forcing AWS OFI ndev 4 | |
| ip-172-31-33-235:21962:21962 [3] NCCL INFO NET/OFI Selected Provider is efa;ofi_rxr | |
| ip-172-31-36-128:22027:22027 [6] NCCL INFO NET/OFI Forcing AWS OFI ndev 4 | |
| ip-172-31-36-128:22027:22027 [6] NCCL INFO NET/OFI Selected Provider is efa;ofi_rxr | |
| ip-172-31-45-171:25059:25059 [1] NCCL INFO NET/OFI Forcing AWS OFI ndev 4 | |
| ip-172-31-45-171:25059:25059 [1] NCCL INFO NET/OFI Selected Provider is efa;ofi_rxr | |
| ip-172-31-36-128:22026:22026 [5] NCCL INFO NET/Socket : Using [0]eth0:172.31.36.128<0> | |
| ip-172-31-47-156:22560:22560 [0] NCCL INFO NET/Socket : Using [0]eth0:172.31.47.156<0> | |
| ip-172-31-33-235:21964:21964 [5] NCCL INFO NET/OFI Forcing AWS OFI ndev 4 | |
| ip-172-31-33-235:21964:21964 [5] NCCL INFO NET/OFI Selected Provider is efa;ofi_rxr | |
| ip-172-31-36-128:22030:22030 [7] NCCL INFO NET/Socket : Using [0]eth0:172.31.36.128<0> | |
| ip-172-31-47-156:22566:22566 [6] NCCL INFO NET/Socket : Using [0]eth0:172.31.47.156<0> | |
| ip-172-31-33-235:21965:21965 [6] NCCL INFO NET/OFI Forcing AWS OFI ndev 4 | |
| ip-172-31-33-235:21965:21965 [6] NCCL INFO NET/OFI Selected Provider is efa;ofi_rxr | |
| ip-172-31-36-128:22023:22023 [2] NCCL INFO NET/Socket : Using [0]eth0:172.31.36.128<0> | |
| ip-172-31-38-14:25302:25302 [4] NCCL INFO NET/Socket : Using [0]eth0:172.31.38.14<0> | |
| ip-172-31-39-141:20307:20307 [4] NCCL INFO NET/OFI Forcing AWS OFI ndev 4 | |
| ip-172-31-39-141:20307:20307 [4] NCCL INFO NET/OFI Selected Provider is efa;ofi_rxr | |
| ip-172-31-38-14:25299:25299 [1] NCCL INFO NET/Socket : Using [0]eth0:172.31.38.14<0> | |
| ip-172-31-47-156:22561:22561 [1] NCCL INFO NET/Socket : Using [0]eth0:172.31.47.156<0> | |
| ip-172-31-44-4:22495:22495 [3] NCCL INFO NET/Socket : Using [0]eth0:172.31.44.4<0> | |
| ip-172-31-38-14:25304:25304 [6] NCCL INFO NET/Socket : Using [0]eth0:172.31.38.14<0> | |
| ip-172-31-44-4:22498:22498 [6] NCCL INFO NET/Socket : Using [0]eth0:172.31.44.4<0> | |
| ip-172-31-36-128:22025:22025 [4] NCCL INFO NET/Socket : Using [0]eth0:172.31.36.128<0> | |
| ip-172-31-47-156:22560:22560 [0] NCCL INFO NET/OFI Forcing AWS OFI ndev 4 | |
| ip-172-31-47-156:22560:22560 [0] NCCL INFO NET/OFI Selected Provider is efa;ofi_rxr | |
| ip-172-31-36-128:22026:22026 [5] NCCL INFO NET/OFI Forcing AWS OFI ndev 4 | |
| ip-172-31-36-128:22026:22026 [5] NCCL INFO NET/OFI Selected Provider is efa;ofi_rxr | |
| ip-172-31-39-141:20308:20308 [5] NCCL INFO NET/Socket : Using [0]eth0:172.31.39.141<0> | |
| ip-172-31-38-14:25298:25298 [0] NCCL INFO NET/Socket : Using [0]eth0:172.31.38.14<0> | |
| ip-172-31-44-4:22497:22497 [5] NCCL INFO NET/Socket : Using [0]eth0:172.31.44.4<0> | |
| ip-172-31-38-14:25303:25303 [5] NCCL INFO NET/Socket : Using [0]eth0:172.31.38.14<0> | |
| ip-172-31-47-156:22563:22563 [3] NCCL INFO NET/Socket : Using [0]eth0:172.31.47.156<0> | |
| ip-172-31-47-156:22566:22566 [6] NCCL INFO NET/OFI Forcing AWS OFI ndev 4 | |
| ip-172-31-47-156:22566:22566 [6] NCCL INFO NET/OFI Selected Provider is efa;ofi_rxr | |
| ip-172-31-38-14:25301:25301 [3] NCCL INFO NET/Socket : Using [0]eth0:172.31.38.14<0> | |
| ip-172-31-47-156:22569:22569 [7] NCCL INFO NET/Socket : Using [0]eth0:172.31.47.156<0> | |
| ip-172-31-36-128:22030:22030 [7] NCCL INFO NET/OFI Forcing AWS OFI ndev 4 | |
| ip-172-31-36-128:22030:22030 [7] NCCL INFO NET/OFI Selected Provider is efa;ofi_rxr | |
| ip-172-31-36-128:22023:22023 [2] NCCL INFO NET/OFI Forcing AWS OFI ndev 4 | |
| ip-172-31-36-128:22023:22023 [2] NCCL INFO NET/OFI Selected Provider is efa;ofi_rxr | |
| ip-172-31-47-156:22565:22565 [5] NCCL INFO NET/Socket : Using [0]eth0:172.31.47.156<0> | |
| ip-172-31-47-156:22561:22561 [1] NCCL INFO NET/OFI Forcing AWS OFI ndev 4 | |
| ip-172-31-47-156:22561:22561 [1] NCCL INFO NET/OFI Selected Provider is efa;ofi_rxr | |
| ip-172-31-36-128:22025:22025 [4] NCCL INFO NET/OFI Forcing AWS OFI ndev 4 | |
| ip-172-31-36-128:22025:22025 [4] NCCL INFO NET/OFI Selected Provider is efa;ofi_rxr | |
| ip-172-31-38-14:25299:25299 [1] NCCL INFO NET/OFI Forcing AWS OFI ndev 4 | |
| ip-172-31-38-14:25299:25299 [1] NCCL INFO NET/OFI Selected Provider is efa;ofi_rxr | |
| ip-172-31-44-4:22498:22498 [6] NCCL INFO NET/OFI Forcing AWS OFI ndev 4 | |
| ip-172-31-44-4:22498:22498 [6] NCCL INFO NET/OFI Selected Provider is efa;ofi_rxr | |
| ip-172-31-38-14:25306:25306 [7] NCCL INFO NET/Socket : Using [0]eth0:172.31.38.14<0> | |
| ip-172-31-39-141:20308:20308 [5] NCCL INFO NET/OFI Forcing AWS OFI ndev 4 | |
| ip-172-31-39-141:20308:20308 [5] NCCL INFO NET/OFI Selected Provider is efa;ofi_rxr | |
| ip-172-31-38-14:25300:25300 [2] NCCL INFO NET/Socket : Using [0]eth0:172.31.38.14<0> | |
| ip-172-31-38-14:25304:25304 [6] NCCL INFO NET/OFI Forcing AWS OFI ndev 4 | |
| ip-172-31-38-14:25304:25304 [6] NCCL INFO NET/OFI Selected Provider is efa;ofi_rxr | |
| ip-172-31-44-4:22497:22497 [5] NCCL INFO NET/OFI Forcing AWS OFI ndev 4 | |
| ip-172-31-44-4:22497:22497 [5] NCCL INFO NET/OFI Selected Provider is efa;ofi_rxr | |
| ip-172-31-47-156:22563:22563 [3] NCCL INFO NET/OFI Forcing AWS OFI ndev 4 | |
| ip-172-31-47-156:22563:22563 [3] NCCL INFO NET/OFI Selected Provider is efa;ofi_rxr | |
| ip-172-31-44-4:22495:22495 [3] NCCL INFO NET/OFI Forcing AWS OFI ndev 4 | |
| ip-172-31-44-4:22495:22495 [3] NCCL INFO NET/OFI Selected Provider is efa;ofi_rxr | |
| ip-172-31-47-156:22569:22569 [7] NCCL INFO NET/OFI Forcing AWS OFI ndev 4 | |
| ip-172-31-47-156:22569:22569 [7] NCCL INFO NET/OFI Selected Provider is efa;ofi_rxr | |
| ip-172-31-47-156:22565:22565 [5] NCCL INFO NET/OFI Forcing AWS OFI ndev 4 | |
| ip-172-31-47-156:22565:22565 [5] NCCL INFO NET/OFI Selected Provider is efa;ofi_rxr | |
| ip-172-31-36-188:22212:22212 [5] NCCL INFO NET/Socket : Using [0]eth0:172.31.36.188<0> | |
| ip-172-31-38-14:25302:25302 [4] NCCL INFO NET/OFI Forcing AWS OFI ndev 4 | |
| ip-172-31-38-14:25302:25302 [4] NCCL INFO NET/OFI Selected Provider is efa;ofi_rxr | |
| ip-172-31-36-188:22210:22210 [3] NCCL INFO NET/Socket : Using [0]eth0:172.31.36.188<0> | |
| ip-172-31-38-14:25298:25298 [0] NCCL INFO NET/OFI Forcing AWS OFI ndev 4 | |
| ip-172-31-38-14:25298:25298 [0] NCCL INFO NET/OFI Selected Provider is efa;ofi_rxr | |
| ip-172-31-38-14:25303:25303 [5] NCCL INFO NET/OFI Forcing AWS OFI ndev 4 | |
| ip-172-31-38-14:25303:25303 [5] NCCL INFO NET/OFI Selected Provider is efa;ofi_rxr | |
| ip-172-31-38-14:25301:25301 [3] NCCL INFO NET/OFI Forcing AWS OFI ndev 4 | |
| ip-172-31-38-14:25301:25301 [3] NCCL INFO NET/OFI Selected Provider is efa;ofi_rxr | |
| ip-172-31-36-188:22212:22212 [5] NCCL INFO NET/OFI Forcing AWS OFI ndev 4 | |
| ip-172-31-36-188:22212:22212 [5] NCCL INFO NET/OFI Selected Provider is efa;ofi_rxr | |
| ip-172-31-38-14:25300:25300 [2] NCCL INFO NET/OFI Forcing AWS OFI ndev 4 | |
| ip-172-31-38-14:25300:25300 [2] NCCL INFO NET/OFI Selected Provider is efa;ofi_rxr | |
| ip-172-31-38-14:25306:25306 [7] NCCL INFO NET/OFI Forcing AWS OFI ndev 4 | |
| ip-172-31-38-14:25306:25306 [7] NCCL INFO NET/OFI Selected Provider is efa;ofi_rxr | |
| ip-172-31-36-188:22207:22207 [0] NCCL INFO NET/Socket : Using [0]eth0:172.31.36.188<0> | |
| ip-172-31-36-188:22210:22210 [3] NCCL INFO NET/OFI Forcing AWS OFI ndev 4 | |
| ip-172-31-36-188:22210:22210 [3] NCCL INFO NET/OFI Selected Provider is efa;ofi_rxr | |
| ip-172-31-36-188:22213:22213 [6] NCCL INFO NET/Socket : Using [0]eth0:172.31.36.188<0> | |
| ip-172-31-36-188:22208:22208 [1] NCCL INFO NET/Socket : Using [0]eth0:172.31.36.188<0> | |
| ip-172-31-36-188:22211:22211 [4] NCCL INFO NET/Socket : Using [0]eth0:172.31.36.188<0> | |
| ip-172-31-36-188:22207:22207 [0] NCCL INFO NET/OFI Forcing AWS OFI ndev 4 | |
| ip-172-31-36-188:22207:22207 [0] NCCL INFO NET/OFI Selected Provider is efa;ofi_rxr | |
| ip-172-31-36-188:22213:22213 [6] NCCL INFO NET/OFI Forcing AWS OFI ndev 4 | |
| ip-172-31-36-188:22213:22213 [6] NCCL INFO NET/OFI Selected Provider is efa;ofi_rxr | |
| ip-172-31-36-188:22208:22208 [1] NCCL INFO NET/OFI Forcing AWS OFI ndev 4 | |
| ip-172-31-36-188:22208:22208 [1] NCCL INFO NET/OFI Selected Provider is efa;ofi_rxr | |
| ip-172-31-36-188:22211:22211 [4] NCCL INFO NET/OFI Forcing AWS OFI ndev 4 | |
| ip-172-31-36-188:22211:22211 [4] NCCL INFO NET/OFI Selected Provider is efa;ofi_rxr | |
| ip-172-31-44-230:21987:21987 [0] NCCL INFO NET/Socket : Using [0]eth0:172.31.44.230<0> | |
| ip-172-31-44-230:21989:21989 [2] NCCL INFO NET/Socket : Using [0]eth0:172.31.44.230<0> | |
| ip-172-31-44-230:21988:21988 [1] NCCL INFO NET/Socket : Using [0]eth0:172.31.44.230<0> | |
| ip-172-31-44-230:21987:21987 [0] NCCL INFO NET/OFI Forcing AWS OFI ndev 4 | |
| ip-172-31-44-230:21987:21987 [0] NCCL INFO NET/OFI Selected Provider is efa;ofi_rxr | |
| ip-172-31-44-230:21991:21991 [4] NCCL INFO NET/Socket : Using [0]eth0:172.31.44.230<0> | |
| ip-172-31-44-230:21989:21989 [2] NCCL INFO NET/OFI Forcing AWS OFI ndev 4 | |
| ip-172-31-44-230:21989:21989 [2] NCCL INFO NET/OFI Selected Provider is efa;ofi_rxr | |
| ip-172-31-44-230:21992:21992 [5] NCCL INFO NET/Socket : Using [0]eth0:172.31.44.230<0> | |
| ip-172-31-44-230:21994:21994 [7] NCCL INFO NET/Socket : Using [0]eth0:172.31.44.230<0> | |
| ip-172-31-39-141:20306:20306 [3] NCCL INFO NET/Socket : Using [0]eth0:172.31.39.141<0> | |
| ip-172-31-44-230:21988:21988 [1] NCCL INFO NET/OFI Forcing AWS OFI ndev 4 | |
| ip-172-31-44-230:21988:21988 [1] NCCL INFO NET/OFI Selected Provider is efa;ofi_rxr | |
| ip-172-31-44-230:21991:21991 [4] NCCL INFO NET/OFI Forcing AWS OFI ndev 4 | |
| ip-172-31-44-230:21991:21991 [4] NCCL INFO NET/OFI Selected Provider is efa;ofi_rxr | |
| ip-172-31-44-230:21990:21990 [3] NCCL INFO NET/Socket : Using [0]eth0:172.31.44.230<0> | |
| ip-172-31-44-230:21992:21992 [5] NCCL INFO NET/OFI Forcing AWS OFI ndev 4 | |
| ip-172-31-44-230:21992:21992 [5] NCCL INFO NET/OFI Selected Provider is efa;ofi_rxr | |
| ip-172-31-44-230:21993:21993 [6] NCCL INFO NET/Socket : Using [0]eth0:172.31.44.230<0> | |
| ip-172-31-44-230:21994:21994 [7] NCCL INFO NET/OFI Forcing AWS OFI ndev 4 | |
| ip-172-31-44-230:21994:21994 [7] NCCL INFO NET/OFI Selected Provider is efa;ofi_rxr | |
| ip-172-31-39-141:20306:20306 [3] NCCL INFO NET/OFI Forcing AWS OFI ndev 4 | |
| ip-172-31-39-141:20306:20306 [3] NCCL INFO NET/OFI Selected Provider is efa;ofi_rxr | |
| ip-172-31-44-230:21990:21990 [3] NCCL INFO NET/OFI Forcing AWS OFI ndev 4 | |
| ip-172-31-44-230:21990:21990 [3] NCCL INFO NET/OFI Selected Provider is efa;ofi_rxr | |
| ip-172-31-44-230:21993:21993 [6] NCCL INFO NET/OFI Forcing AWS OFI ndev 4 | |
| ip-172-31-44-230:21993:21993 [6] NCCL INFO NET/OFI Selected Provider is efa;ofi_rxr | |
| ip-172-31-46-50:22042:22042 [2] NCCL INFO NET/Socket : Using [0]eth0:172.31.46.50<0> | |
| ip-172-31-44-142:24935:24935 [6] NCCL INFO NET/Socket : Using [0]eth0:172.31.44.142<0> | |
| ip-172-31-46-50:22044:22044 [4] NCCL INFO NET/Socket : Using [0]eth0:172.31.46.50<0> | |
| ip-172-31-46-50:22042:22042 [2] NCCL INFO NET/OFI Forcing AWS OFI ndev 4 | |
| ip-172-31-46-50:22042:22042 [2] NCCL INFO NET/OFI Selected Provider is efa;ofi_rxr | |
| ip-172-31-46-50:22047:22047 [7] NCCL INFO NET/Socket : Using [0]eth0:172.31.46.50<0> | |
| ip-172-31-46-50:22043:22043 [3] NCCL INFO NET/Socket : Using [0]eth0:172.31.46.50<0> | |
| ip-172-31-46-50:22044:22044 [4] NCCL INFO NET/OFI Forcing AWS OFI ndev 4 | |
| ip-172-31-46-50:22044:22044 [4] NCCL INFO NET/OFI Selected Provider is efa;ofi_rxr | |
| ip-172-31-44-142:24935:24935 [6] NCCL INFO NET/OFI Forcing AWS OFI ndev 4 | |
| ip-172-31-44-142:24935:24935 [6] NCCL INFO NET/OFI Selected Provider is efa;ofi_rxr | |
| ip-172-31-46-50:22045:22045 [5] NCCL INFO NET/Socket : Using [0]eth0:172.31.46.50<0> | |
| ip-172-31-46-50:22047:22047 [7] NCCL INFO NET/OFI Forcing AWS OFI ndev 4 | |
| ip-172-31-46-50:22047:22047 [7] NCCL INFO NET/OFI Selected Provider is efa;ofi_rxr | |
| ip-172-31-44-142:24930:24930 [1] NCCL INFO NET/Socket : Using [0]eth0:172.31.44.142<0> | |
| ip-172-31-44-142:24931:24931 [2] NCCL INFO NET/Socket : Using [0]eth0:172.31.44.142<0> | |
| ip-172-31-46-50:22043:22043 [3] NCCL INFO NET/OFI Forcing AWS OFI ndev 4 | |
| ip-172-31-46-50:22043:22043 [3] NCCL INFO NET/OFI Selected Provider is efa;ofi_rxr | |
| ip-172-31-46-50:22046:22046 [6] NCCL INFO NET/Socket : Using [0]eth0:172.31.46.50<0> | |
| ip-172-31-44-142:24937:24937 [7] NCCL INFO NET/Socket : Using [0]eth0:172.31.44.142<0> | |
| ip-172-31-44-142:24929:24929 [0] NCCL INFO NET/Socket : Using [0]eth0:172.31.44.142<0> | |
| ip-172-31-46-50:22045:22045 [5] NCCL INFO NET/OFI Forcing AWS OFI ndev 4 | |
| ip-172-31-46-50:22045:22045 [5] NCCL INFO NET/OFI Selected Provider is efa;ofi_rxr | |
| ip-172-31-44-142:24930:24930 [1] NCCL INFO NET/OFI Forcing AWS OFI ndev 4 | |
| ip-172-31-44-142:24930:24930 [1] NCCL INFO NET/OFI Selected Provider is efa;ofi_rxr | |
| ip-172-31-44-142:24933:24933 [4] NCCL INFO NET/Socket : Using [0]eth0:172.31.44.142<0> | |
| ip-172-31-44-142:24931:24931 [2] NCCL INFO NET/OFI Forcing AWS OFI ndev 4 | |
| ip-172-31-44-142:24931:24931 [2] NCCL INFO NET/OFI Selected Provider is efa;ofi_rxr | |
| ip-172-31-46-50:22046:22046 [6] NCCL INFO NET/OFI Forcing AWS OFI ndev 4 | |
| ip-172-31-46-50:22046:22046 [6] NCCL INFO NET/OFI Selected Provider is efa;ofi_rxr | |
| ip-172-31-44-142:24934:24934 [5] NCCL INFO NET/Socket : Using [0]eth0:172.31.44.142<0> | |
| ip-172-31-44-142:24937:24937 [7] NCCL INFO NET/OFI Forcing AWS OFI ndev 4 | |
| ip-172-31-44-142:24937:24937 [7] NCCL INFO NET/OFI Selected Provider is efa;ofi_rxr | |
| ip-172-31-44-142:24929:24929 [0] NCCL INFO NET/OFI Forcing AWS OFI ndev 4 | |
| ip-172-31-44-142:24929:24929 [0] NCCL INFO NET/OFI Selected Provider is efa;ofi_rxr | |
| ip-172-31-44-142:24933:24933 [4] NCCL INFO NET/OFI Forcing AWS OFI ndev 4 | |
| ip-172-31-44-142:24933:24933 [4] NCCL INFO NET/OFI Selected Provider is efa;ofi_rxr | |
| ip-172-31-44-142:24934:24934 [5] NCCL INFO NET/OFI Forcing AWS OFI ndev 4 | |
| ip-172-31-44-142:24934:24934 [5] NCCL INFO NET/OFI Selected Provider is efa;ofi_rxr | |
| ip-172-31-44-142:24932:24932 [3] NCCL INFO NET/Socket : Using [0]eth0:172.31.44.142<0> | |
| ip-172-31-44-142:24932:24932 [3] NCCL INFO NET/OFI Forcing AWS OFI ndev 4 | |
| ip-172-31-44-142:24932:24932 [3] NCCL INFO NET/OFI Selected Provider is efa;ofi_rxr | |
| ip-172-31-46-50:22041:22041 [1] NCCL INFO NET/Socket : Using [0]eth0:172.31.46.50<0> | |
| ip-172-31-46-50:22040:22040 [0] NCCL INFO NET/Socket : Using [0]eth0:172.31.46.50<0> | |
| ip-172-31-46-50:22041:22041 [1] NCCL INFO NET/OFI Forcing AWS OFI ndev 4 | |
| ip-172-31-46-50:22041:22041 [1] NCCL INFO NET/OFI Selected Provider is efa;ofi_rxr | |
| ip-172-31-46-50:22040:22040 [0] NCCL INFO NET/OFI Forcing AWS OFI ndev 4 | |
| ip-172-31-46-50:22040:22040 [0] NCCL INFO NET/OFI Selected Provider is efa;ofi_rxr | |
| ip-172-31-36-128:22022:22022 [1] NCCL INFO NET/Socket : Using [0]eth0:172.31.36.128<0> | |
| ip-172-31-36-128:22021:22021 [0] NCCL INFO NET/Socket : Using [0]eth0:172.31.36.128<0> | |
| ip-172-31-36-128:22022:22022 [1] NCCL INFO NET/OFI Forcing AWS OFI ndev 4 | |
| ip-172-31-36-128:22022:22022 [1] NCCL INFO NET/OFI Selected Provider is efa;ofi_rxr | |
| ip-172-31-36-128:22021:22021 [0] NCCL INFO NET/OFI Forcing AWS OFI ndev 4 | |
| ip-172-31-36-128:22021:22021 [0] NCCL INFO NET/OFI Selected Provider is efa;ofi_rxr | |
| ip-172-31-38-82:24978:24978 [0] NCCL INFO NET/Socket : Using [0]eth0:172.31.38.82<0> | |
| ip-172-31-38-82:24981:24981 [3] NCCL INFO NET/Socket : Using [0]eth0:172.31.38.82<0> | |
| ip-172-31-38-82:24984:24984 [6] NCCL INFO NET/Socket : Using [0]eth0:172.31.38.82<0> | |
| ip-172-31-38-82:24978:24978 [0] NCCL INFO NET/OFI Forcing AWS OFI ndev 4 | |
| ip-172-31-38-82:24978:24978 [0] NCCL INFO NET/OFI Selected Provider is efa;ofi_rxr | |
| ip-172-31-38-82:24981:24981 [3] NCCL INFO NET/OFI Forcing AWS OFI ndev 4 | |
| ip-172-31-38-82:24981:24981 [3] NCCL INFO NET/OFI Selected Provider is efa;ofi_rxr | |
| ip-172-31-38-82:24984:24984 [6] NCCL INFO NET/OFI Forcing AWS OFI ndev 4 | |
| ip-172-31-38-82:24984:24984 [6] NCCL INFO NET/OFI Selected Provider is efa;ofi_rxr | |
| ip-172-31-44-4:22492:22492 [0] NCCL INFO NET/Socket : Using [0]eth0:172.31.44.4<0> | |
| ip-172-31-44-4:22494:22494 [2] NCCL INFO NET/Socket : Using [0]eth0:172.31.44.4<0> | |
| ip-172-31-39-141:20305:20305 [2] NCCL INFO NET/Socket : Using [0]eth0:172.31.39.141<0> | |
| ip-172-31-44-4:22499:22499 [7] NCCL INFO NET/Socket : Using [0]eth0:172.31.44.4<0> | |
| ip-172-31-44-4:22493:22493 [1] NCCL INFO NET/Socket : Using [0]eth0:172.31.44.4<0> | |
| ip-172-31-38-82:24983:24983 [5] NCCL INFO NET/Socket : Using [0]eth0:172.31.38.82<0> | |
| ip-172-31-38-82:24979:24979 [1] NCCL INFO NET/Socket : Using [0]eth0:172.31.38.82<0> | |
| ip-172-31-44-4:22492:22492 [0] NCCL INFO NET/OFI Forcing AWS OFI ndev 4 | |
| ip-172-31-44-4:22492:22492 [0] NCCL INFO NET/OFI Selected Provider is efa;ofi_rxr | |
| ip-172-31-44-4:22496:22496 [4] NCCL INFO NET/Socket : Using [0]eth0:172.31.44.4<0> | |
| ip-172-31-38-82:24980:24980 [2] NCCL INFO NET/Socket : Using [0]eth0:172.31.38.82<0> | |
| ip-172-31-44-4:22494:22494 [2] NCCL INFO NET/OFI Forcing AWS OFI ndev 4 | |
| ip-172-31-44-4:22494:22494 [2] NCCL INFO NET/OFI Selected Provider is efa;ofi_rxr | |
| ip-172-31-39-141:20305:20305 [2] NCCL INFO NET/OFI Forcing AWS OFI ndev 4 | |
| ip-172-31-39-141:20305:20305 [2] NCCL INFO NET/OFI Selected Provider is efa;ofi_rxr | |
| ip-172-31-39-141:20310:20310 [7] NCCL INFO NET/Socket : Using [0]eth0:172.31.39.141<0> | |
| ip-172-31-38-82:24985:24985 [7] NCCL INFO NET/Socket : Using [0]eth0:172.31.38.82<0> | |
| ip-172-31-38-82:24983:24983 [5] NCCL INFO NET/OFI Forcing AWS OFI ndev 4 | |
| ip-172-31-38-82:24983:24983 [5] NCCL INFO NET/OFI Selected Provider is efa;ofi_rxr | |
| ip-172-31-38-82:24982:24982 [4] NCCL INFO NET/Socket : Using [0]eth0:172.31.38.82<0> | |
| ip-172-31-44-4:22499:22499 [7] NCCL INFO NET/OFI Forcing AWS OFI ndev 4 | |
| ip-172-31-44-4:22499:22499 [7] NCCL INFO NET/OFI Selected Provider is efa;ofi_rxr | |
| ip-172-31-39-141:20309:20309 [6] NCCL INFO NET/Socket : Using [0]eth0:172.31.39.141<0> | |
| ip-172-31-38-82:24979:24979 [1] NCCL INFO NET/OFI Forcing AWS OFI ndev 4 | |
| ip-172-31-38-82:24979:24979 [1] NCCL INFO NET/OFI Selected Provider is efa;ofi_rxr | |
| ip-172-31-39-141:20304:20304 [1] NCCL INFO NET/Socket : Using [0]eth0:172.31.39.141<0> | |
| ip-172-31-44-4:22493:22493 [1] NCCL INFO NET/OFI Forcing AWS OFI ndev 4 | |
| ip-172-31-44-4:22493:22493 [1] NCCL INFO NET/OFI Selected Provider is efa;ofi_rxr | |
| ip-172-31-38-82:24980:24980 [2] NCCL INFO NET/OFI Forcing AWS OFI ndev 4 | |
| ip-172-31-38-82:24980:24980 [2] NCCL INFO NET/OFI Selected Provider is efa;ofi_rxr | |
| ip-172-31-44-4:22496:22496 [4] NCCL INFO NET/OFI Forcing AWS OFI ndev 4 | |
| ip-172-31-44-4:22496:22496 [4] NCCL INFO NET/OFI Selected Provider is efa;ofi_rxr | |
| ip-172-31-38-82:24985:24985 [7] NCCL INFO NET/OFI Forcing AWS OFI ndev 4 | |
| ip-172-31-38-82:24985:24985 [7] NCCL INFO NET/OFI Selected Provider is efa;ofi_rxr | |
| ip-172-31-38-82:24982:24982 [4] NCCL INFO NET/OFI Forcing AWS OFI ndev 4 | |
| ip-172-31-38-82:24982:24982 [4] NCCL INFO NET/OFI Selected Provider is efa;ofi_rxr | |
| ip-172-31-39-141:20310:20310 [7] NCCL INFO NET/OFI Forcing AWS OFI ndev 4 | |
| ip-172-31-39-141:20310:20310 [7] NCCL INFO NET/OFI Selected Provider is efa;ofi_rxr | |
| ip-172-31-39-141:20309:20309 [6] NCCL INFO NET/OFI Forcing AWS OFI ndev 4 | |
| ip-172-31-39-141:20309:20309 [6] NCCL INFO NET/OFI Selected Provider is efa;ofi_rxr | |
| ip-172-31-39-141:20304:20304 [1] NCCL INFO NET/OFI Forcing AWS OFI ndev 4 | |
| ip-172-31-39-141:20304:20304 [1] NCCL INFO NET/OFI Selected Provider is efa;ofi_rxr | |
| ip-172-31-33-235:21961:22023 [2] NCCL INFO Setting affinity for GPU 2 to ffffffff,ffffffff,ffffffff | |
| ip-172-31-33-235:21961:22023 [2] NCCL INFO NCCL_TREE_THRESHOLD set by environment to 0. | |
| ip-172-31-36-128:22024:22091 [3] NCCL INFO Setting affinity for GPU 3 to ffffffff,ffffffff,ffffffff | |
| ip-172-31-36-128:22024:22091 [3] NCCL INFO NCCL_TREE_THRESHOLD set by environment to 0. | |
| ip-172-31-39-141:20303:20372 [0] NCCL INFO Setting affinity for GPU 0 to ffffffff,ffffffff,ffffffff | |
| ip-172-31-39-141:20303:20372 [0] NCCL INFO NCCL_TREE_THRESHOLD set by environment to 0. | |
| ip-172-31-32-86:24698:24764 [7] NCCL INFO Setting affinity for GPU 7 to ffffffff,ffffffff,ffffffff | |
| ip-172-31-32-86:24698:24764 [7] NCCL INFO NCCL_TREE_THRESHOLD set by environment to 0. | |
| ip-172-31-32-86:24691:24765 [0] NCCL INFO Setting affinity for GPU 0 to ffffffff,ffffffff,ffffffff | |
| ip-172-31-32-86:24691:24765 [0] NCCL INFO NCCL_TREE_THRESHOLD set by environment to 0. | |
| ip-172-31-47-240:24692:24760 [3] NCCL INFO Setting affinity for GPU 3 to ffffffff,ffffffff,ffffffff | |
| ip-172-31-47-240:24692:24760 [3] NCCL INFO NCCL_TREE_THRESHOLD set by environment to 0. | |
| ip-172-31-37-141:25826:25891 [2] NCCL INFO Setting affinity for GPU 2 to ffffffff,ffffffff,ffffffff | |
| ip-172-31-37-141:25826:25891 [2] NCCL INFO NCCL_TREE_THRESHOLD set by environment to 0. | |
| ip-172-31-47-240:24691:24761 [2] NCCL INFO Setting affinity for GPU 2 to ffffffff,ffffffff,ffffffff | |
| ip-172-31-47-240:24691:24761 [2] NCCL INFO NCCL_TREE_THRESHOLD set by environment to 0. | |
| ip-172-31-36-188:22209:22274 [2] NCCL INFO Setting affinity for GPU 2 to ffffffff,ffffffff,ffffffff | |
| ip-172-31-36-188:22209:22274 [2] NCCL INFO NCCL_TREE_THRESHOLD set by environment to 0. | |
| ip-172-31-37-141:25831:25892 [7] NCCL INFO Setting affinity for GPU 7 to ffffffff,ffffffff,ffffffff | |
| ip-172-31-37-141:25831:25892 [7] NCCL INFO NCCL_TREE_THRESHOLD set by environment to 0. | |
| ip-172-31-36-188:22214:22275 [7] NCCL INFO Setting affinity for GPU 7 to ffffffff,ffffffff,ffffffff | |
| ip-172-31-36-188:22214:22275 [7] NCCL INFO NCCL_TREE_THRESHOLD set by environment to 0. | |
| ip-172-31-47-240:24695:24762 [6] NCCL INFO Setting affinity for GPU 6 to ffffffff,ffffffff,ffffffff | |
| ip-172-31-47-240:24695:24762 [6] NCCL INFO NCCL_TREE_THRESHOLD set by environment to 0. | |
| ip-172-31-45-171:25058:25128 [0] NCCL INFO Setting affinity for GPU 0 to ffffffff,ffffffff,ffffffff | |
| ip-172-31-45-171:25058:25128 [0] NCCL INFO NCCL_TREE_THRESHOLD set by environment to 0. | |
| ip-172-31-47-240:24696:24763 [7] NCCL INFO Setting affinity for GPU 7 to ffffffff,ffffffff,ffffffff | |
| ip-172-31-47-240:24696:24763 [7] NCCL INFO NCCL_TREE_THRESHOLD set by environment to 0. | |
| ip-172-31-45-171:25064:25129 [6] NCCL INFO Setting affinity for GPU 6 to ffffffff,ffffffff,ffffffff | |
| ip-172-31-45-171:25064:25129 [6] NCCL INFO NCCL_TREE_THRESHOLD set by environment to 0. | |
| ip-172-31-47-156:22564:22632 [4] NCCL INFO Setting affinity for GPU 4 to ffffffff,ffffffff,ffffffff | |
| ip-172-31-47-156:22564:22632 [4] NCCL INFO NCCL_TREE_THRESHOLD set by environment to 0. | |
| ip-172-31-47-156:22562:22633 [2] NCCL INFO Setting affinity for GPU 2 to ffffffff,ffffffff,ffffffff | |
| ip-172-31-47-156:22562:22633 [2] NCCL INFO NCCL_TREE_THRESHOLD set by environment to 0. | |
| ip-172-31-39-141:20307:20377 [4] NCCL INFO Setting affinity for GPU 4 to ffffffff,ffffffff,ffffffff | |
| ip-172-31-39-141:20307:20377 [4] NCCL INFO NCCL_TREE_THRESHOLD set by environment to 0. | |
| ip-172-31-44-4:22495:22558 [3] NCCL INFO Setting affinity for GPU 3 to ffffffff,ffffffff,ffffffff | |
| ip-172-31-44-4:22495:22558 [3] NCCL INFO NCCL_TREE_THRESHOLD set by environment to 0. | |
| ip-172-31-45-171:25063:25130 [5] NCCL INFO Setting affinity for GPU 5 to ffffffff,ffffffff,ffffffff | |
| ip-172-31-45-171:25063:25130 [5] NCCL INFO NCCL_TREE_THRESHOLD set by environment to 0. | |
| ip-172-31-45-171:25062:25131 [4] NCCL INFO Setting affinity for GPU 4 to ffffffff,ffffffff,ffffffff | |
| ip-172-31-45-171:25062:25131 [4] NCCL INFO NCCL_TREE_THRESHOLD set by environment to 0. | |
| ip-172-31-39-141:20308:20378 [5] NCCL INFO Setting affinity for GPU 5 to ffffffff,ffffffff,ffffffff | |
| ip-172-31-39-141:20308:20378 [5] NCCL INFO NCCL_TREE_THRESHOLD set by environment to 0. | |
| ip-172-31-39-141:20306:20379 [3] NCCL INFO Setting affinity for GPU 3 to ffffffff,ffffffff,ffffffff | |
| ip-172-31-39-141:20306:20379 [3] NCCL INFO NCCL_TREE_THRESHOLD set by environment to 0. | |
| ip-172-31-47-188:22012:22080 [4] NCCL INFO Setting affinity for GPU 4 to ffffffff,ffffffff,ffffffff | |
| ip-172-31-47-188:22012:22080 [4] NCCL INFO NCCL_TREE_THRESHOLD set by environment to 0. | |
| ip-172-31-44-4:22497:22559 [5] NCCL INFO Setting affinity for GPU 5 to ffffffff,ffffffff,ffffffff | |
| ip-172-31-44-4:22497:22559 [5] NCCL INFO NCCL_TREE_THRESHOLD set by environment to 0. | |
| ip-172-31-37-141:25827:25893 [3] NCCL INFO Setting affinity for GPU 3 to ffffffff,ffffffff,ffffffff | |
| ip-172-31-37-141:25827:25893 [3] NCCL INFO NCCL_TREE_THRESHOLD set by environment to 0. | |
| ip-172-31-44-4:22498:22560 [6] NCCL INFO Setting affinity for GPU 6 to ffffffff,ffffffff,ffffffff | |
| ip-172-31-44-4:22498:22560 [6] NCCL INFO NCCL_TREE_THRESHOLD set by environment to 0. | |
| ip-172-31-47-156:22561:22636 [1] NCCL INFO Setting affinity for GPU 1 to ffffffff,ffffffff,ffffffff | |
| ip-172-31-47-156:22561:22636 [1] NCCL INFO NCCL_TREE_THRESHOLD set by environment to 0. | |
| ip-172-31-36-128:22027:22092 [6] NCCL INFO Setting affinity for GPU 6 to ffffffff,ffffffff,ffffffff | |
| ip-172-31-36-128:22027:22092 [6] NCCL INFO NCCL_TREE_THRESHOLD set by environment to 0. | |
| ip-172-31-47-188:22008:22081 [0] NCCL INFO Setting affinity for GPU 0 to ffffffff,ffffffff,ffffffff | |
| ip-172-31-47-188:22008:22081 [0] NCCL INFO NCCL_TREE_THRESHOLD set by environment to 0. | |
| ip-172-31-47-156:22560:22634 [0] NCCL INFO Setting affinity for GPU 0 to ffffffff,ffffffff,ffffffff | |
| ip-172-31-47-156:22560:22634 [0] NCCL INFO NCCL_TREE_THRESHOLD set by environment to 0. | |
| ip-172-31-36-128:22023:22093 [2] NCCL INFO Setting affinity for GPU 2 to ffffffff,ffffffff,ffffffff | |
| ip-172-31-36-128:22023:22093 [2] NCCL INFO NCCL_TREE_THRESHOLD set by environment to 0. | |
| ip-172-31-38-14:25302:25368 [4] NCCL INFO Setting affinity for GPU 4 to ffffffff,ffffffff,ffffffff | |
| ip-172-31-38-14:25302:25368 [4] NCCL INFO NCCL_TREE_THRESHOLD set by environment to 0. | |
| ip-172-31-37-141:25825:25894 [1] NCCL INFO Setting affinity for GPU 1 to ffffffff,ffffffff,ffffffff | |
| ip-172-31-37-141:25825:25894 [1] NCCL INFO NCCL_TREE_THRESHOLD set by environment to 0. | |
| ip-172-31-46-50:22042:22112 [2] NCCL INFO Setting affinity for GPU 2 to ffffffff,ffffffff,ffffffff | |
| ip-172-31-46-50:22042:22112 [2] NCCL INFO NCCL_TREE_THRESHOLD set by environment to 0. | |
| ip-172-31-36-128:22026:22094 [5] NCCL INFO Setting affinity for GPU 5 to ffffffff,ffffffff,ffffffff | |
| ip-172-31-36-128:22026:22094 [5] NCCL INFO NCCL_TREE_THRESHOLD set by environment to 0. | |
| ip-172-31-33-235:21960:22031 [1] NCCL INFO Setting affinity for GPU 1 to ffffffff,ffffffff,ffffffff | |
| ip-172-31-33-235:21960:22031 [1] NCCL INFO NCCL_TREE_THRESHOLD set by environment to 0. | |
| ip-172-31-33-235:21959:22032 [0] NCCL INFO Setting affinity for GPU 0 to ffffffff,ffffffff,ffffffff | |
| ip-172-31-33-235:21959:22032 [0] NCCL INFO NCCL_TREE_THRESHOLD set by environment to 0. | |
| ip-172-31-36-188:22212:22276 [5] NCCL INFO Setting affinity for GPU 5 to ffffffff,ffffffff,ffffffff | |
| ip-172-31-36-188:22212:22276 [5] NCCL INFO NCCL_TREE_THRESHOLD set by environment to 0. | |
| ip-172-31-37-141:25830:25895 [6] NCCL INFO Setting affinity for GPU 6 to ffffffff,ffffffff,ffffffff | |
| ip-172-31-37-141:25830:25895 [6] NCCL INFO NCCL_TREE_THRESHOLD set by environment to 0. | |
| ip-172-31-47-156:22566:22635 [6] NCCL INFO Setting affinity for GPU 6 to ffffffff,ffffffff,ffffffff | |
| ip-172-31-47-156:22566:22635 [6] NCCL INFO NCCL_TREE_THRESHOLD set by environment to 0. | |
| ip-172-31-36-188:22210:22277 [3] NCCL INFO Setting affinity for GPU 3 to ffffffff,ffffffff,ffffffff | |
| ip-172-31-37-141:25828:25896 [4] NCCL INFO Setting affinity for GPU 4 to ffffffff,ffffffff,ffffffff | |
| ip-172-31-36-188:22210:22277 [3] NCCL INFO NCCL_TREE_THRESHOLD set by environment to 0. | |
| ip-172-31-37-141:25828:25896 [4] NCCL INFO NCCL_TREE_THRESHOLD set by environment to 0. | |
| ip-172-31-36-188:22208:22278 [1] NCCL INFO Setting affinity for GPU 1 to ffffffff,ffffffff,ffffffff | |
| ip-172-31-44-142:24935:25000 [6] NCCL INFO Setting affinity for GPU 6 to ffffffff,ffffffff,ffffffff | |
| ip-172-31-36-188:22208:22278 [1] NCCL INFO NCCL_TREE_THRESHOLD set by environment to 0. | |
| ip-172-31-36-128:22025:22095 [4] NCCL INFO Setting affinity for GPU 4 to ffffffff,ffffffff,ffffffff | |
| ip-172-31-44-142:24935:25000 [6] NCCL INFO NCCL_TREE_THRESHOLD set by environment to 0. | |
| ip-172-31-36-128:22025:22095 [4] NCCL INFO NCCL_TREE_THRESHOLD set by environment to 0. | |
| ip-172-31-45-171:25060:25132 [2] NCCL INFO Setting affinity for GPU 2 to ffffffff,ffffffff,ffffffff | |
| ip-172-31-45-171:25060:25132 [2] NCCL INFO NCCL_TREE_THRESHOLD set by environment to 0. | |
| ip-172-31-36-128:22030:22096 [7] NCCL INFO Setting affinity for GPU 7 to ffffffff,ffffffff,ffffffff | |
| ip-172-31-44-230:21987:22059 [0] NCCL INFO Setting affinity for GPU 0 to ffffffff,ffffffff,ffffffff | |
| ip-172-31-32-86:24693:24766 [2] NCCL INFO Setting affinity for GPU 2 to ffffffff,ffffffff,ffffffff | |
| ip-172-31-44-230:21987:22059 [0] NCCL INFO NCCL_TREE_THRESHOLD set by environment to 0. | |
| ip-172-31-36-128:22030:22096 [7] NCCL INFO NCCL_TREE_THRESHOLD set by environment to 0. | |
| ip-172-31-32-86:24693:24766 [2] NCCL INFO NCCL_TREE_THRESHOLD set by environment to 0. | |
| ip-172-31-47-156:22565:22637 [5] NCCL INFO Setting affinity for GPU 5 to ffffffff,ffffffff,ffffffff | |
| ip-172-31-47-156:22565:22637 [5] NCCL INFO NCCL_TREE_THRESHOLD set by environment to 0. | |
| ip-172-31-36-188:22213:22279 [6] NCCL INFO Setting affinity for GPU 6 to ffffffff,ffffffff,ffffffff | |
| ip-172-31-36-188:22213:22279 [6] NCCL INFO NCCL_TREE_THRESHOLD set by environment to 0. | |
| ip-172-31-46-50:22044:22113 [4] NCCL INFO Setting affinity for GPU 4 to ffffffff,ffffffff,ffffffff | |
| ip-172-31-46-50:22044:22113 [4] NCCL INFO NCCL_TREE_THRESHOLD set by environment to 0. | |
| ip-172-31-36-188:22211:22280 [4] NCCL INFO Setting affinity for GPU 4 to ffffffff,ffffffff,ffffffff | |
| ip-172-31-36-188:22211:22280 [4] NCCL INFO NCCL_TREE_THRESHOLD set by environment to 0. | |
| ip-172-31-36-188:22207:22281 [0] NCCL INFO Setting affinity for GPU 0 to ffffffff,ffffffff,ffffffff | |
| ip-172-31-36-188:22207:22281 [0] NCCL INFO NCCL_TREE_THRESHOLD set by environment to 0. | |
| ip-172-31-38-14:25300:25369 [2] NCCL INFO Setting affinity for GPU 2 to ffffffff,ffffffff,ffffffff | |
| ip-172-31-47-188:22009:22082 [1] NCCL INFO Setting affinity for GPU 1 to ffffffff,ffffffff,ffffffff | |
| ip-172-31-37-141:25824:25897 [0] NCCL INFO Setting affinity for GPU 0 to ffffffff,ffffffff,ffffffff | |
| ip-172-31-47-188:22009:22082 [1] NCCL INFO NCCL_TREE_THRESHOLD set by environment to 0. | |
| ip-172-31-37-141:25824:25897 [0] NCCL INFO NCCL_TREE_THRESHOLD set by environment to 0. | |
| ip-172-31-38-14:25300:25369 [2] NCCL INFO NCCL_TREE_THRESHOLD set by environment to 0. | |
| ip-172-31-33-235:21964:22033 [5] NCCL INFO Setting affinity for GPU 5 to ffffffff,ffffffff,ffffffff | |
| ip-172-31-44-230:21988:22060 [1] NCCL INFO Setting affinity for GPU 1 to ffffffff,ffffffff,ffffffff | |
| ip-172-31-33-235:21964:22033 [5] NCCL INFO NCCL_TREE_THRESHOLD set by environment to 0. | |
| ip-172-31-44-230:21988:22060 [1] NCCL INFO NCCL_TREE_THRESHOLD set by environment to 0. | |
| ip-172-31-47-188:22011:22083 [3] NCCL INFO Setting affinity for GPU 3 to ffffffff,ffffffff,ffffffff | |
| ip-172-31-47-188:22011:22083 [3] NCCL INFO NCCL_TREE_THRESHOLD set by environment to 0. | |
| ip-172-31-37-141:25829:25898 [5] NCCL INFO Setting affinity for GPU 5 to ffffffff,ffffffff,ffffffff | |
| ip-172-31-37-141:25829:25898 [5] NCCL INFO NCCL_TREE_THRESHOLD set by environment to 0. | |
| ip-172-31-47-240:24694:24764 [5] NCCL INFO Setting affinity for GPU 5 to ffffffff,ffffffff,ffffffff | |
| ip-172-31-47-240:24694:24764 [5] NCCL INFO NCCL_TREE_THRESHOLD set by environment to 0. | |
| ip-172-31-32-86:24696:24767 [5] NCCL INFO Setting affinity for GPU 5 to ffffffff,ffffffff,ffffffff | |
| ip-172-31-32-86:24696:24767 [5] NCCL INFO NCCL_TREE_THRESHOLD set by environment to 0. | |
| ip-172-31-44-230:21989:22061 [2] NCCL INFO Setting affinity for GPU 2 to ffffffff,ffffffff,ffffffff | |
| ip-172-31-47-188:22013:22084 [5] NCCL INFO Setting affinity for GPU 5 to ffffffff,ffffffff,ffffffff | |
| ip-172-31-47-188:22013:22084 [5] NCCL INFO NCCL_TREE_THRESHOLD set by environment to 0. | |
| ip-172-31-44-230:21989:22061 [2] NCCL INFO NCCL_TREE_THRESHOLD set by environment to 0. | |
| ip-172-31-47-188:22015:22086 [7] NCCL INFO Setting affinity for GPU 7 to ffffffff,ffffffff,ffffffff | |
| ip-172-31-47-188:22015:22086 [7] NCCL INFO NCCL_TREE_THRESHOLD set by environment to 0. | |
| ip-172-31-47-240:24690:24765 [1] NCCL INFO Setting affinity for GPU 1 to ffffffff,ffffffff,ffffffff | |
| ip-172-31-47-240:24690:24765 [1] NCCL INFO NCCL_TREE_THRESHOLD set by environment to 0. | |
| ip-172-31-33-235:21966:22034 [7] NCCL INFO Setting affinity for GPU 7 to ffffffff,ffffffff,ffffffff | |
| ip-172-31-33-235:21966:22034 [7] NCCL INFO NCCL_TREE_THRESHOLD set by environment to 0. | |
| ip-172-31-33-235:21963:22035 [4] NCCL INFO Setting affinity for GPU 4 to ffffffff,ffffffff,ffffffff | |
| ip-172-31-33-235:21963:22035 [4] NCCL INFO NCCL_TREE_THRESHOLD set by environment to 0. | |
| ip-172-31-47-188:22010:22085 [2] NCCL INFO Setting affinity for GPU 2 to ffffffff,ffffffff,ffffffff | |
| ip-172-31-47-188:22010:22085 [2] NCCL INFO NCCL_TREE_THRESHOLD set by environment to 0. | |
| ip-172-31-47-240:24689:24766 [0] NCCL INFO Setting affinity for GPU 0 to ffffffff,ffffffff,ffffffff | |
| ip-172-31-47-240:24689:24766 [0] NCCL INFO NCCL_TREE_THRESHOLD set by environment to 0. | |
| ip-172-31-38-14:25306:25370 [7] NCCL INFO Setting affinity for GPU 7 to ffffffff,ffffffff,ffffffff | |
| ip-172-31-38-14:25306:25370 [7] NCCL INFO NCCL_TREE_THRESHOLD set by environment to 0. | |
| ip-172-31-33-235:21962:22036 [3] NCCL INFO Setting affinity for GPU 3 to ffffffff,ffffffff,ffffffff | |
| ip-172-31-33-235:21962:22036 [3] NCCL INFO NCCL_TREE_THRESHOLD set by environment to 0. | |
| ip-172-31-32-86:24697:24768 [6] NCCL INFO Setting affinity for GPU 6 to ffffffff,ffffffff,ffffffff | |
| ip-172-31-32-86:24697:24768 [6] NCCL INFO NCCL_TREE_THRESHOLD set by environment to 0. | |
| ip-172-31-33-235:21965:22037 [6] NCCL INFO Setting affinity for GPU 6 to ffffffff,ffffffff,ffffffff | |
| ip-172-31-33-235:21965:22037 [6] NCCL INFO NCCL_TREE_THRESHOLD set by environment to 0. | |
| ip-172-31-47-188:22014:22087 [6] NCCL INFO Setting affinity for GPU 6 to ffffffff,ffffffff,ffffffff | |
| ip-172-31-47-188:22014:22087 [6] NCCL INFO NCCL_TREE_THRESHOLD set by environment to 0. | |
| ip-172-31-47-240:24693:24767 [4] NCCL INFO Setting affinity for GPU 4 to ffffffff,ffffffff,ffffffff | |
| ip-172-31-47-240:24693:24767 [4] NCCL INFO NCCL_TREE_THRESHOLD set by environment to 0. | |
| ip-172-31-38-14:25304:25371 [6] NCCL INFO Setting affinity for GPU 6 to ffffffff,ffffffff,ffffffff | |
| ip-172-31-38-14:25304:25371 [6] NCCL INFO NCCL_TREE_THRESHOLD set by environment to 0. | |
| ip-172-31-38-14:25299:25372 [1] NCCL INFO Setting affinity for GPU 1 to ffffffff,ffffffff,ffffffff | |
| ip-172-31-32-86:24692:24769 [1] NCCL INFO Setting affinity for GPU 1 to ffffffff,ffffffff,ffffffff | |
| ip-172-31-32-86:24692:24769 [1] NCCL INFO NCCL_TREE_THRESHOLD set by environment to 0. | |
| ip-172-31-38-14:25299:25372 [1] NCCL INFO NCCL_TREE_THRESHOLD set by environment to 0. | |
| ip-172-31-46-50:22047:22114 [7] NCCL INFO Setting affinity for GPU 7 to ffffffff,ffffffff,ffffffff | |
| ip-172-31-46-50:22047:22114 [7] NCCL INFO NCCL_TREE_THRESHOLD set by environment to 0. | |
| ip-172-31-44-230:21991:22062 [4] NCCL INFO Setting affinity for GPU 4 to ffffffff,ffffffff,ffffffff | |
| ip-172-31-44-230:21991:22062 [4] NCCL INFO NCCL_TREE_THRESHOLD set by environment to 0. | |
| ip-172-31-46-50:22045:22115 [5] NCCL INFO Setting affinity for GPU 5 to ffffffff,ffffffff,ffffffff | |
| ip-172-31-46-50:22045:22115 [5] NCCL INFO NCCL_TREE_THRESHOLD set by environment to 0. | |
| ip-172-31-44-230:21994:22063 [7] NCCL INFO Setting affinity for GPU 7 to ffffffff,ffffffff,ffffffff | |
| ip-172-31-38-82:24978:25047 [0] NCCL INFO Setting affinity for GPU 0 to ffffffff,ffffffff,ffffffff | |
| ip-172-31-44-230:21994:22063 [7] NCCL INFO NCCL_TREE_THRESHOLD set by environment to 0. | |
| ip-172-31-38-82:24978:25047 [0] NCCL INFO NCCL_TREE_THRESHOLD set by environment to 0. | |
| ip-172-31-45-171:25065:25133 [7] NCCL INFO Setting affinity for GPU 7 to ffffffff,ffffffff,ffffffff | |
| ip-172-31-45-171:25065:25133 [7] NCCL INFO NCCL_TREE_THRESHOLD set by environment to 0. | |
| ip-172-31-32-86:24694:24770 [3] NCCL INFO Setting affinity for GPU 3 to ffffffff,ffffffff,ffffffff | |
| ip-172-31-32-86:24694:24770 [3] NCCL INFO NCCL_TREE_THRESHOLD set by environment to 0. | |
| ip-172-31-46-50:22046:22116 [6] NCCL INFO Setting affinity for GPU 6 to ffffffff,ffffffff,ffffffff | |
| ip-172-31-46-50:22046:22116 [6] NCCL INFO NCCL_TREE_THRESHOLD set by environment to 0. | |
| ip-172-31-44-230:21992:22064 [5] NCCL INFO Setting affinity for GPU 5 to ffffffff,ffffffff,ffffffff | |
| ip-172-31-44-230:21992:22064 [5] NCCL INFO NCCL_TREE_THRESHOLD set by environment to 0. | |
| ip-172-31-46-50:22041:22117 [1] NCCL INFO Setting affinity for GPU 1 to ffffffff,ffffffff,ffffffff | |
| ip-172-31-46-50:22041:22117 [1] NCCL INFO NCCL_TREE_THRESHOLD set by environment to 0. | |
| ip-172-31-32-86:24695:24771 [4] NCCL INFO Setting affinity for GPU 4 to ffffffff,ffffffff,ffffffff | |
| ip-172-31-32-86:24695:24771 [4] NCCL INFO NCCL_TREE_THRESHOLD set by environment to 0. | |
| ip-172-31-44-142:24933:25001 [4] NCCL INFO Setting affinity for GPU 4 to ffffffff,ffffffff,ffffffff | |
| ip-172-31-44-142:24933:25001 [4] NCCL INFO NCCL_TREE_THRESHOLD set by environment to 0. | |
| ip-172-31-44-142:24929:25002 [0] NCCL INFO Setting affinity for GPU 0 to ffffffff,ffffffff,ffffffff | |
| ip-172-31-44-142:24929:25002 [0] NCCL INFO NCCL_TREE_THRESHOLD set by environment to 0. | |
| ip-172-31-38-14:25303:25373 [5] NCCL INFO Setting affinity for GPU 5 to ffffffff,ffffffff,ffffffff | |
| ip-172-31-38-14:25303:25373 [5] NCCL INFO NCCL_TREE_THRESHOLD set by environment to 0. | |
| ip-172-31-44-230:21990:22065 [3] NCCL INFO Setting affinity for GPU 3 to ffffffff,ffffffff,ffffffff | |
| ip-172-31-44-230:21990:22065 [3] NCCL INFO NCCL_TREE_THRESHOLD set by environment to 0. | |
| ip-172-31-44-142:24930:25003 [1] NCCL INFO Setting affinity for GPU 1 to ffffffff,ffffffff,ffffffff | |
| ip-172-31-44-142:24931:25004 [2] NCCL INFO Setting affinity for GPU 2 to ffffffff,ffffffff,ffffffff | |
| ip-172-31-44-142:24930:25003 [1] NCCL INFO NCCL_TREE_THRESHOLD set by environment to 0. | |
| ip-172-31-44-142:24931:25004 [2] NCCL INFO NCCL_TREE_THRESHOLD set by environment to 0. | |
| ip-172-31-44-230:21993:22066 [6] NCCL INFO Setting affinity for GPU 6 to ffffffff,ffffffff,ffffffff | |
| ip-172-31-44-230:21993:22066 [6] NCCL INFO NCCL_TREE_THRESHOLD set by environment to 0. | |
| ip-172-31-38-14:25298:25374 [0] NCCL INFO Setting affinity for GPU 0 to ffffffff,ffffffff,ffffffff | |
| ip-172-31-38-14:25298:25374 [0] NCCL INFO NCCL_TREE_THRESHOLD set by environment to 0. | |
| ip-172-31-38-14:25301:25375 [3] NCCL INFO Setting affinity for GPU 3 to ffffffff,ffffffff,ffffffff | |
| ip-172-31-38-14:25301:25375 [3] NCCL INFO NCCL_TREE_THRESHOLD set by environment to 0. | |
| ip-172-31-44-4:22496:22561 [4] NCCL INFO Setting affinity for GPU 4 to ffffffff,ffffffff,ffffffff | |
| ip-172-31-45-171:25061:25134 [3] NCCL INFO Setting affinity for GPU 3 to ffffffff,ffffffff,ffffffff | |
| ip-172-31-45-171:25061:25134 [3] NCCL INFO NCCL_TREE_THRESHOLD set by environment to 0. | |
| ip-172-31-44-4:22496:22561 [4] NCCL INFO NCCL_TREE_THRESHOLD set by environment to 0. | |
| ip-172-31-44-142:24934:25005 [5] NCCL INFO Setting affinity for GPU 5 to ffffffff,ffffffff,ffffffff | |
| ip-172-31-44-142:24934:25005 [5] NCCL INFO NCCL_TREE_THRESHOLD set by environment to 0. | |
| ip-172-31-44-4:22499:22562 [7] NCCL INFO Setting affinity for GPU 7 to ffffffff,ffffffff,ffffffff | |
| ip-172-31-44-4:22499:22562 [7] NCCL INFO NCCL_TREE_THRESHOLD set by environment to 0. | |
| ip-172-31-38-82:24981:25048 [3] NCCL INFO Setting affinity for GPU 3 to ffffffff,ffffffff,ffffffff | |
| ip-172-31-38-82:24981:25048 [3] NCCL INFO NCCL_TREE_THRESHOLD set by environment to 0. | |
| ip-172-31-45-171:25059:25135 [1] NCCL INFO Setting affinity for GPU 1 to ffffffff,ffffffff,ffffffff | |
| ip-172-31-45-171:25059:25135 [1] NCCL INFO NCCL_TREE_THRESHOLD set by environment to 0. | |
| ip-172-31-44-4:22492:22563 [0] NCCL INFO Setting affinity for GPU 0 to ffffffff,ffffffff,ffffffff | |
| ip-172-31-44-142:24932:25006 [3] NCCL INFO Setting affinity for GPU 3 to ffffffff,ffffffff,ffffffff | |
| ip-172-31-44-4:22492:22563 [0] NCCL INFO NCCL_TREE_THRESHOLD set by environment to 0. | |
| ip-172-31-44-142:24932:25006 [3] NCCL INFO NCCL_TREE_THRESHOLD set by environment to 0. | |
| ip-172-31-44-142:24937:25007 [7] NCCL INFO Setting affinity for GPU 7 to ffffffff,ffffffff,ffffffff | |
| ip-172-31-44-142:24937:25007 [7] NCCL INFO NCCL_TREE_THRESHOLD set by environment to 0. | |
| ip-172-31-44-4:22494:22564 [2] NCCL INFO Setting affinity for GPU 2 to ffffffff,ffffffff,ffffffff | |
| ip-172-31-44-4:22494:22564 [2] NCCL INFO NCCL_TREE_THRESHOLD set by environment to 0. | |
| ip-172-31-38-82:24984:25049 [6] NCCL INFO Setting affinity for GPU 6 to ffffffff,ffffffff,ffffffff | |
| ip-172-31-38-82:24984:25049 [6] NCCL INFO NCCL_TREE_THRESHOLD set by environment to 0. | |
| ip-172-31-44-4:22493:22565 [1] NCCL INFO Setting affinity for GPU 1 to ffffffff,ffffffff,ffffffff | |
| ip-172-31-44-4:22493:22565 [1] NCCL INFO NCCL_TREE_THRESHOLD set by environment to 0. | |
| ip-172-31-36-128:22022:22097 [1] NCCL INFO Setting affinity for GPU 1 to ffffffff,ffffffff,ffffffff | |
| ip-172-31-36-128:22022:22097 [1] NCCL INFO NCCL_TREE_THRESHOLD set by environment to 0. | |
| ip-172-31-36-128:22021:22098 [0] NCCL INFO Setting affinity for GPU 0 to ffffffff,ffffffff,ffffffff | |
| ip-172-31-36-128:22021:22098 [0] NCCL INFO NCCL_TREE_THRESHOLD set by environment to 0. | |
| ip-172-31-47-156:22563:22638 [3] NCCL INFO Setting affinity for GPU 3 to ffffffff,ffffffff,ffffffff | |
| ip-172-31-47-156:22563:22638 [3] NCCL INFO NCCL_TREE_THRESHOLD set by environment to 0. | |
| ip-172-31-46-50:22043:22118 [3] NCCL INFO Setting affinity for GPU 3 to ffffffff,ffffffff,ffffffff | |
| ip-172-31-46-50:22043:22118 [3] NCCL INFO NCCL_TREE_THRESHOLD set by environment to 0. | |
| ip-172-31-47-156:22569:22639 [7] NCCL INFO Setting affinity for GPU 7 to ffffffff,ffffffff,ffffffff | |
| ip-172-31-47-156:22569:22639 [7] NCCL INFO NCCL_TREE_THRESHOLD set by environment to 0. | |
| ip-172-31-38-82:24983:25050 [5] NCCL INFO Setting affinity for GPU 5 to ffffffff,ffffffff,ffffffff | |
| ip-172-31-38-82:24983:25050 [5] NCCL INFO NCCL_TREE_THRESHOLD set by environment to 0. | |
| ip-172-31-38-82:24980:25051 [2] NCCL INFO Setting affinity for GPU 2 to ffffffff,ffffffff,ffffffff | |
| ip-172-31-38-82:24980:25051 [2] NCCL INFO NCCL_TREE_THRESHOLD set by environment to 0. | |
| ip-172-31-38-82:24979:25052 [1] NCCL INFO Setting affinity for GPU 1 to ffffffff,ffffffff,ffffffff | |
| ip-172-31-38-82:24979:25052 [1] NCCL INFO NCCL_TREE_THRESHOLD set by environment to 0. | |
| ip-172-31-38-82:24985:25053 [7] NCCL INFO Setting affinity for GPU 7 to ffffffff,ffffffff,ffffffff | |
| ip-172-31-38-82:24985:25053 [7] NCCL INFO NCCL_TREE_THRESHOLD set by environment to 0. | |
| ip-172-31-46-50:22040:22119 [0] NCCL INFO Setting affinity for GPU 0 to ffffffff,ffffffff,ffffffff | |
| ip-172-31-46-50:22040:22119 [0] NCCL INFO NCCL_TREE_THRESHOLD set by environment to 0. | |
| ip-172-31-38-82:24982:25054 [4] NCCL INFO Setting affinity for GPU 4 to ffffffff,ffffffff,ffffffff | |
| ip-172-31-38-82:24982:25054 [4] NCCL INFO NCCL_TREE_THRESHOLD set by environment to 0. | |
| ip-172-31-39-141:20305:20380 [2] NCCL INFO Setting affinity for GPU 2 to ffffffff,ffffffff,ffffffff | |
| ip-172-31-39-141:20305:20380 [2] NCCL INFO NCCL_TREE_THRESHOLD set by environment to 0. | |
| ip-172-31-39-141:20304:20381 [1] NCCL INFO Setting affinity for GPU 1 to ffffffff,ffffffff,ffffffff | |
| ip-172-31-39-141:20304:20381 [1] NCCL INFO NCCL_TREE_THRESHOLD set by environment to 0. | |
| ip-172-31-39-141:20309:20382 [6] NCCL INFO Setting affinity for GPU 6 to ffffffff,ffffffff,ffffffff | |
| ip-172-31-39-141:20309:20382 [6] NCCL INFO NCCL_TREE_THRESHOLD set by environment to 0. | |
| ip-172-31-39-141:20310:20383 [7] NCCL INFO Setting affinity for GPU 7 to ffffffff,ffffffff,ffffffff | |
| ip-172-31-39-141:20310:20383 [7] NCCL INFO NCCL_TREE_THRESHOLD set by environment to 0. | |
| ip-172-31-38-82:24985:25053 [7] NCCL INFO NET/OFI [7] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00 | |
| ip-172-31-38-82:24984:25049 [6] NCCL INFO NET/OFI [6] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00 | |
| ip-172-31-38-82:24985:25053 [7] NCCL INFO NET/OFI [7] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00 | |
| ip-172-31-38-82:24983:25050 [5] NCCL INFO NET/OFI [5] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00 | |
| ip-172-31-38-82:24985:25053 [7] NCCL INFO NET/OFI [7] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00 | |
| ip-172-31-38-82:24984:25049 [6] NCCL INFO NET/OFI [6] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00 | |
| ip-172-31-38-82:24982:25054 [4] NCCL INFO NET/OFI [4] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00 | |
| ip-172-31-37-141:25829:25898 [5] NCCL INFO NET/OFI [5] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00 | |
| ip-172-31-37-141:25829:25898 [5] NCCL INFO NET/OFI [5] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00 | |
| ip-172-31-38-82:24983:25050 [5] NCCL INFO NET/OFI [5] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00 | |
| ip-172-31-38-82:24985:25053 [7] NCCL INFO NET/OFI [7] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-38-82:24985:25053 [7] NCCL INFO CUDA Dev 7[7], AWS Libfabric NIC distance : NODE NODE NODE PHB | |
| ip-172-31-38-82:24981:25048 [3] NCCL INFO NET/OFI [3] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00 | |
| ip-172-31-37-141:25828:25896 [4] NCCL INFO NET/OFI [4] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00 | |
| ip-172-31-38-82:24984:25049 [6] NCCL INFO NET/OFI [6] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00 | |
| ip-172-31-37-141:25829:25898 [5] NCCL INFO NET/OFI [5] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-38-82:24980:25051 [2] NCCL INFO NET/OFI [2] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00 | |
| ip-172-31-44-230:21991:22062 [4] NCCL INFO NET/OFI [4] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00 | |
| ip-172-31-37-141:25830:25895 [6] NCCL INFO NET/OFI [6] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00 | |
| ip-172-31-38-82:24982:25054 [4] NCCL INFO NET/OFI [4] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00 | |
| ip-172-31-38-14:25304:25371 [6] NCCL INFO NET/OFI [6] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00 | |
| ip-172-31-33-235:21960:22031 [1] NCCL INFO NET/OFI [1] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-33-235:21960:22031 [1] NCCL INFO NET/OFI [1] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00 | |
| ip-172-31-44-230:21990:22065 [3] NCCL INFO NET/OFI [3] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00 | |
| ip-172-31-45-171:25064:25129 [6] NCCL INFO NET/OFI [6] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00 | |
| ip-172-31-45-171:25064:25129 [6] NCCL INFO NET/OFI [6] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00 | |
| ip-172-31-37-141:25828:25896 [4] NCCL INFO NET/OFI [4] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00 | |
| ip-172-31-36-188:22214:22275 [7] NCCL INFO NET/OFI [7] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00 | |
| ip-172-31-36-128:22030:22096 [7] NCCL INFO NET/OFI [7] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00 | |
| ip-172-31-36-128:22030:22096 [7] NCCL INFO NET/OFI [7] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00 | |
| ip-172-31-38-14:25306:25370 [7] NCCL INFO NET/OFI [7] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00 | |
| ip-172-31-32-86:24697:24768 [6] NCCL INFO NET/OFI [6] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00 | |
| ip-172-31-33-235:21959:22032 [0] NCCL INFO NET/OFI [0] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-44-230:21991:22062 [4] NCCL INFO NET/OFI [4] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00 | |
| ip-172-31-44-4:22492:22563 [0] NCCL INFO NET/OFI [0] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-44-4:22492:22563 [0] NCCL INFO NET/OFI [0] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00 | |
| ip-172-31-45-171:25065:25133 [7] NCCL INFO NET/OFI [7] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00 | |
| ip-172-31-47-156:22566:22635 [6] NCCL INFO NET/OFI [6] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00 | |
| ip-172-31-37-141:25829:25898 [5] NCCL INFO NET/OFI [5] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00 | |
| ip-172-31-38-82:24979:25052 [1] NCCL INFO NET/OFI [1] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-36-188:22212:22276 [5] NCCL INFO NET/OFI [5] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00 | |
| ip-172-31-36-188:22212:22276 [5] NCCL INFO NET/OFI [5] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00 | |
| ip-172-31-36-128:22027:22092 [6] NCCL INFO NET/OFI [6] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00 | |
| ip-172-31-38-14:25304:25371 [6] NCCL INFO NET/OFI [6] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00 | |
| ip-172-31-32-86:24696:24767 [5] NCCL INFO NET/OFI [5] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00 | |
| ip-172-31-33-235:21960:22031 [1] NCCL INFO NET/OFI [1] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00 | |
| ip-172-31-47-188:22009:22082 [1] NCCL INFO NET/OFI [1] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-47-188:22009:22082 [1] NCCL INFO NET/OFI [1] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00 | |
| ip-172-31-44-230:21994:22063 [7] NCCL INFO NET/OFI [7] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00 | |
| ip-172-31-44-4:22492:22563 [0] NCCL INFO NET/OFI [0] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00 | |
| ip-172-31-47-240:24696:24763 [7] NCCL INFO NET/OFI [7] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00 | |
| ip-172-31-46-50:22046:22116 [6] NCCL INFO NET/OFI [6] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00 | |
| ip-172-31-45-171:25063:25130 [5] NCCL INFO NET/OFI [5] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00 | |
| ip-172-31-47-156:22569:22639 [7] NCCL INFO NET/OFI [7] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00 | |
| ip-172-31-47-156:22569:22639 [7] NCCL INFO NET/OFI [7] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00 | |
| ip-172-31-37-141:25829:25898 [5] NCCL INFO CUDA Dev 5[5], AWS Libfabric NIC distance : NODE NODE PHB NODE | |
| ip-172-31-38-82:24983:25050 [5] NCCL INFO NET/OFI [5] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-36-188:22211:22280 [4] NCCL INFO NET/OFI [4] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00 | |
| ip-172-31-36-128:22026:22094 [5] NCCL INFO NET/OFI [5] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00 | |
| ip-172-31-38-14:25304:25371 [6] NCCL INFO NET/OFI [6] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00 | |
| ip-172-31-32-86:24697:24768 [6] NCCL INFO NET/OFI [6] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00 | |
| ip-172-31-33-235:21959:22032 [0] NCCL INFO NET/OFI [0] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00 | |
| ip-172-31-47-188:22010:22085 [2] NCCL INFO NET/OFI [2] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00 | |
| ip-172-31-47-188:22010:22085 [2] NCCL INFO NET/OFI [2] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-44-230:21991:22062 [4] NCCL INFO NET/OFI [4] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-44-142:24935:25000 [6] NCCL INFO NET/OFI [6] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00 | |
| ip-172-31-44-4:22492:22563 [0] NCCL INFO NET/OFI [0] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00 | |
| ip-172-31-47-240:24695:24762 [6] NCCL INFO NET/OFI [6] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00 | |
| ip-172-31-46-50:22047:22114 [7] NCCL INFO NET/OFI [7] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00 | |
| ip-172-31-46-50:22047:22114 [7] NCCL INFO NET/OFI [7] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00 | |
| ip-172-31-45-171:25064:25129 [6] NCCL INFO NET/OFI [6] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00 | |
| ip-172-31-47-156:22565:22637 [5] NCCL INFO NET/OFI [5] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00 | |
| ip-172-31-37-141:25831:25892 [7] NCCL INFO NET/OFI [7] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00 | |
| ip-172-31-38-82:24984:25049 [6] NCCL INFO NET/OFI [6] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-38-82:24984:25049 [6] NCCL INFO CUDA Dev 6[6], AWS Libfabric NIC distance : NODE NODE NODE PHB | |
| ip-172-31-36-188:22214:22275 [7] NCCL INFO NET/OFI [7] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00 | |
| ip-172-31-36-128:22027:22092 [6] NCCL INFO NET/OFI [6] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00 | |
| ip-172-31-38-14:25306:25370 [7] NCCL INFO NET/OFI [7] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00 | |
| ip-172-31-32-86:24693:24766 [2] NCCL INFO NET/OFI [2] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00 | |
| ip-172-31-33-235:21966:22034 [7] NCCL INFO NET/OFI [7] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00 | |
| ip-172-31-47-188:22014:22087 [6] NCCL INFO NET/OFI [6] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00 | |
| ip-172-31-44-230:21990:22065 [3] NCCL INFO NET/OFI [3] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-44-142:24934:25005 [5] NCCL INFO NET/OFI [5] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00 | |
| ip-172-31-44-142:24934:25005 [5] NCCL INFO NET/OFI [5] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00 | |
| ip-172-31-44-4:22492:22563 [0] NCCL INFO CUDA Dev 0[0], AWS Libfabric NIC distance : PHB NODE NODE NODE | |
| ip-172-31-47-240:24696:24763 [7] NCCL INFO NET/OFI [7] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00 | |
| ip-172-31-46-50:22046:22116 [6] NCCL INFO NET/OFI [6] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00 | |
| ip-172-31-45-171:25065:25133 [7] NCCL INFO NET/OFI [7] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00 | |
| ip-172-31-47-156:22564:22632 [4] NCCL INFO NET/OFI [4] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00 | |
| ip-172-31-37-141:25828:25896 [4] NCCL INFO NET/OFI [4] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-38-82:24981:25048 [3] NCCL INFO NET/OFI [3] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-36-188:22212:22276 [5] NCCL INFO NET/OFI [5] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-36-188:22212:22276 [5] NCCL INFO NET/OFI [5] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00 | |
| ip-172-31-36-188:22212:22276 [5] NCCL INFO CUDA Dev 5[5], AWS Libfabric NIC distance : NODE NODE PHB NODE | |
| ip-172-31-36-128:22030:22096 [7] NCCL INFO NET/OFI [7] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00 | |
| ip-172-31-38-14:25300:25369 [2] NCCL INFO NET/OFI [2] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00 | |
| ip-172-31-32-86:24696:24767 [5] NCCL INFO NET/OFI [5] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00 | |
| ip-172-31-33-235:21963:22035 [4] NCCL INFO NET/OFI [4] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00 | |
| ip-172-31-47-188:22010:22085 [2] NCCL INFO NET/OFI [2] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00 | |
| ip-172-31-44-230:21991:22062 [4] NCCL INFO NET/OFI [4] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00 | |
| ip-172-31-44-142:24931:25004 [2] NCCL INFO NET/OFI [2] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00 | |
| ip-172-31-44-4:22498:22560 [6] NCCL INFO NET/OFI [6] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00 | |
| ip-172-31-47-240:24694:24764 [5] NCCL INFO NET/OFI [5] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00 | |
| ip-172-31-46-50:22044:22113 [4] NCCL INFO NET/OFI [4] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00 | |
| ip-172-31-45-171:25062:25131 [4] NCCL INFO NET/OFI [4] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00 | |
| ip-172-31-47-156:22566:22635 [6] NCCL INFO NET/OFI [6] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00 | |
| ip-172-31-37-141:25830:25895 [6] NCCL INFO NET/OFI [6] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00 | |
| ip-172-31-38-82:24980:25051 [2] NCCL INFO NET/OFI [2] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-36-188:22213:22279 [6] NCCL INFO NET/OFI [6] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00 | |
| ip-172-31-36-128:22024:22091 [3] NCCL INFO NET/OFI [3] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00 | |
| ip-172-31-38-14:25304:25371 [6] NCCL INFO NET/OFI [6] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-38-14:25304:25371 [6] NCCL INFO CUDA Dev 6[6], AWS Libfabric NIC distance : NODE NODE NODE PHB | |
| ip-172-31-32-86:24697:24768 [6] NCCL INFO NET/OFI [6] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00 | |
| ip-172-31-33-235:21965:22037 [6] NCCL INFO NET/OFI [6] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00 | |
| ip-172-31-47-188:22015:22086 [7] NCCL INFO NET/OFI [7] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00 | |
| ip-172-31-44-230:21990:22065 [3] NCCL INFO NET/OFI [3] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00 | |
| ip-172-31-44-142:24935:25000 [6] NCCL INFO NET/OFI [6] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00 | |
| ip-172-31-44-4:22499:22562 [7] NCCL INFO NET/OFI [7] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00 | |
| ip-172-31-47-240:24691:24761 [2] NCCL INFO NET/OFI [2] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00 | |
| ip-172-31-46-50:22047:22114 [7] NCCL INFO NET/OFI [7] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00 | |
| ip-172-31-45-171:25063:25130 [5] NCCL INFO NET/OFI [5] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00 | |
| ip-172-31-47-156:22569:22639 [7] NCCL INFO NET/OFI [7] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00 | |
| ip-172-31-37-141:25827:25893 [3] NCCL INFO NET/OFI [3] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00 | |
| ip-172-31-38-82:24982:25054 [4] NCCL INFO NET/OFI [4] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-36-188:22211:22280 [4] NCCL INFO NET/OFI [4] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00 | |
| ip-172-31-36-128:22030:22096 [7] NCCL INFO NET/OFI [7] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-36-128:22030:22096 [7] NCCL INFO CUDA Dev 7[7], AWS Libfabric NIC distance : NODE NODE NODE PHB | |
| ip-172-31-38-14:25306:25370 [7] NCCL INFO NET/OFI [7] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00 | |
| ip-172-31-32-86:24697:24768 [6] NCCL INFO NET/OFI [6] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-32-86:24697:24768 [6] NCCL INFO CUDA Dev 6[6], AWS Libfabric NIC distance : NODE NODE NODE PHB | |
| ip-172-31-33-235:21960:22031 [1] NCCL INFO NET/OFI [1] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00 | |
| ip-172-31-47-188:22009:22082 [1] NCCL INFO NET/OFI [1] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00 | |
| ip-172-31-44-230:21991:22062 [4] NCCL INFO CUDA Dev 4[4], AWS Libfabric NIC distance : NODE NODE PHB NODE | |
| ip-172-31-44-142:24934:25005 [5] NCCL INFO NET/OFI [5] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-44-4:22498:22560 [6] NCCL INFO NET/OFI [6] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00 | |
| ip-172-31-44-4:22498:22560 [6] NCCL INFO NET/OFI [6] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00 | |
| ip-172-31-47-240:24695:24762 [6] NCCL INFO NET/OFI [6] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00 | |
| ip-172-31-46-50:22045:22115 [5] NCCL INFO NET/OFI [5] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00 | |
| ip-172-31-45-171:25064:25129 [6] NCCL INFO NET/OFI [6] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-45-171:25064:25129 [6] NCCL INFO CUDA Dev 6[6], AWS Libfabric NIC distance : NODE NODE NODE PHB | |
| ip-172-31-47-156:22565:22637 [5] NCCL INFO NET/OFI [5] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00 | |
| ip-172-31-37-141:25828:25896 [4] NCCL INFO NET/OFI [4] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00 | |
| ip-172-31-38-82:24978:25047 [0] NCCL INFO NET/OFI [0] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-36-188:22209:22274 [2] NCCL INFO NET/OFI [2] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00 | |
| ip-172-31-36-128:22027:22092 [6] NCCL INFO NET/OFI [6] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00 | |
| ip-172-31-38-14:25303:25373 [5] NCCL INFO NET/OFI [5] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00 | |
| ip-172-31-32-86:24696:24767 [5] NCCL INFO NET/OFI [5] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-33-235:21963:22035 [4] NCCL INFO NET/OFI [4] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00 | |
| ip-172-31-47-188:22010:22085 [2] NCCL INFO NET/OFI [2] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00 | |
| ip-172-31-44-230:21993:22066 [6] NCCL INFO NET/OFI [6] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00 | |
| ip-172-31-44-142:24929:25002 [0] NCCL INFO NET/OFI [0] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-44-4:22497:22559 [5] NCCL INFO NET/OFI [5] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00 | |
| ip-172-31-47-240:24696:24763 [7] NCCL INFO NET/OFI [7] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00 | |
| ip-172-31-46-50:22042:22112 [2] NCCL INFO NET/OFI [2] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00 | |
| ip-172-31-45-171:25060:25132 [2] NCCL INFO NET/OFI [2] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00 | |
| ip-172-31-47-156:22562:22633 [2] NCCL INFO NET/OFI [2] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00 | |
| ip-172-31-37-141:25828:25896 [4] NCCL INFO CUDA Dev 4[4], AWS Libfabric NIC distance : NODE NODE PHB NODE | |
| ip-172-31-38-82:24983:25050 [5] NCCL INFO NET/OFI [5] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00 | |
| ip-172-31-36-188:22210:22277 [3] NCCL INFO NET/OFI [3] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00 | |
| ip-172-31-36-128:22026:22094 [5] NCCL INFO NET/OFI [5] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00 | |
| ip-172-31-38-14:25300:25369 [2] NCCL INFO NET/OFI [2] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-32-86:24695:24771 [4] NCCL INFO NET/OFI [4] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00 | |
| ip-172-31-33-235:21966:22034 [7] NCCL INFO NET/OFI [7] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00 | |
| ip-172-31-47-188:22014:22087 [6] NCCL INFO NET/OFI [6] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00 | |
| ip-172-31-44-230:21989:22061 [2] NCCL INFO NET/OFI [2] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00 | |
| ip-172-31-44-142:24931:25004 [2] NCCL INFO NET/OFI [2] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-44-4:22499:22562 [7] NCCL INFO NET/OFI [7] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00 | |
| ip-172-31-47-240:24694:24764 [5] NCCL INFO NET/OFI [5] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00 | |
| ip-172-31-46-50:22047:22114 [7] NCCL INFO NET/OFI [7] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-46-50:22047:22114 [7] NCCL INFO CUDA Dev 7[7], AWS Libfabric NIC distance : NODE NODE NODE PHB | |
| ip-172-31-45-171:25061:25134 [3] NCCL INFO NET/OFI [3] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00 | |
| ip-172-31-47-156:22560:22634 [0] NCCL INFO NET/OFI [0] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-37-141:25825:25894 [1] NCCL INFO NET/OFI [1] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-38-82:24979:25052 [1] NCCL INFO NET/OFI [1] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00 | |
| ip-172-31-36-188:22214:22275 [7] NCCL INFO NET/OFI [7] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00 | |
| ip-172-31-36-128:22023:22093 [2] NCCL INFO NET/OFI [2] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00 | |
| ip-172-31-38-14:25298:25374 [0] NCCL INFO NET/OFI [0] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-32-86:24693:24766 [2] NCCL INFO NET/OFI [2] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-33-235:21959:22032 [0] NCCL INFO NET/OFI [0] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00 | |
| ip-172-31-47-188:22010:22085 [2] NCCL INFO CUDA Dev 2[2], AWS Libfabric NIC distance : NODE PHB NODE NODE | |
| ip-172-31-44-230:21994:22063 [7] NCCL INFO NET/OFI [7] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00 | |
| ip-172-31-44-142:24935:25000 [6] NCCL INFO NET/OFI [6] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00 | |
| ip-172-31-44-4:22498:22560 [6] NCCL INFO NET/OFI [6] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-44-4:22498:22560 [6] NCCL INFO CUDA Dev 6[6], AWS Libfabric NIC distance : NODE NODE NODE PHB | |
| ip-172-31-47-240:24690:24765 [1] NCCL INFO NET/OFI [1] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-46-50:22046:22116 [6] NCCL INFO NET/OFI [6] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00 | |
| ip-172-31-45-171:25059:25135 [1] NCCL INFO NET/OFI [1] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-47-156:22564:22632 [4] NCCL INFO NET/OFI [4] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00 | |
| ip-172-31-37-141:25831:25892 [7] NCCL INFO NET/OFI [7] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00 | |
| ip-172-31-38-82:24983:25050 [5] NCCL INFO CUDA Dev 5[5], AWS Libfabric NIC distance : NODE NODE PHB NODE | |
| ip-172-31-36-188:22207:22281 [0] NCCL INFO NET/OFI [0] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-36-128:22025:22095 [4] NCCL INFO NET/OFI [4] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00 | |
| ip-172-31-38-14:25306:25370 [7] NCCL INFO NET/OFI [7] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-38-14:25306:25370 [7] NCCL INFO CUDA Dev 7[7], AWS Libfabric NIC distance : NODE NODE NODE PHB | |
| ip-172-31-32-86:24692:24769 [1] NCCL INFO NET/OFI [1] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-33-235:21960:22031 [1] NCCL INFO CUDA Dev 1[1], AWS Libfabric NIC distance : PHB NODE NODE NODE | |
| ip-172-31-47-188:22008:22081 [0] NCCL INFO NET/OFI [0] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-44-230:21993:22066 [6] NCCL INFO NET/OFI [6] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00 | |
| ip-172-31-44-142:24933:25001 [4] NCCL INFO NET/OFI [4] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00 | |
| ip-172-31-44-4:22494:22564 [2] NCCL INFO NET/OFI [2] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00 | |
| ip-172-31-47-240:24689:24766 [0] NCCL INFO NET/OFI [0] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-46-50:22044:22113 [4] NCCL INFO NET/OFI [4] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00 | |
| ip-172-31-45-171:25058:25128 [0] NCCL INFO NET/OFI [0] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-47-156:22569:22639 [7] NCCL INFO NET/OFI [7] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-47-156:22569:22639 [7] NCCL INFO CUDA Dev 7[7], AWS Libfabric NIC distance : NODE NODE NODE PHB | |
| ip-172-31-37-141:25827:25893 [3] NCCL INFO NET/OFI [3] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-38-82:24982:25054 [4] NCCL INFO NET/OFI [4] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00 | |
| ip-172-31-36-188:22208:22278 [1] NCCL INFO NET/OFI [1] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-36-128:22022:22097 [1] NCCL INFO NET/OFI [1] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-38-14:25299:25372 [1] NCCL INFO NET/OFI [1] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-32-86:24691:24765 [0] NCCL INFO NET/OFI [0] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-33-235:21962:22036 [3] NCCL INFO NET/OFI [3] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00 | |
| ip-172-31-47-188:22012:22080 [4] NCCL INFO NET/OFI [4] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00 | |
| ip-172-31-44-230:21990:22065 [3] NCCL INFO NET/OFI [3] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00 | |
| ip-172-31-44-142:24937:25007 [7] NCCL INFO NET/OFI [7] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00 | |
| ip-172-31-44-4:22496:22561 [4] NCCL INFO NET/OFI [4] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00 | |
| ip-172-31-47-240:24695:24762 [6] NCCL INFO NET/OFI [6] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00 | |
| ip-172-31-46-50:22045:22115 [5] NCCL INFO NET/OFI [5] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00 | |
| ip-172-31-45-171:25065:25133 [7] NCCL INFO NET/OFI [7] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00 | |
| ip-172-31-47-156:22563:22638 [3] NCCL INFO NET/OFI [3] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00 | |
| ip-172-31-37-141:25830:25895 [6] NCCL INFO NET/OFI [6] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00 | |
| ip-172-31-38-82:24981:25048 [3] NCCL INFO NET/OFI [3] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00 | |
| ip-172-31-36-188:22213:22279 [6] NCCL INFO NET/OFI [6] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00 | |
| ip-172-31-36-128:22024:22091 [3] NCCL INFO NET/OFI [3] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-38-14:25298:25374 [0] NCCL INFO NET/OFI [0] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00 | |
| ip-172-31-32-86:24698:24764 [7] NCCL INFO NET/OFI [7] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00 | |
| ip-172-31-33-235:21964:22033 [5] NCCL INFO NET/OFI [5] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00 | |
| ip-172-31-47-188:22015:22086 [7] NCCL INFO NET/OFI [7] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00 | |
| ip-172-31-44-230:21990:22065 [3] NCCL INFO CUDA Dev 3[3], AWS Libfabric NIC distance : NODE PHB NODE NODE | |
| ip-172-31-44-142:24934:25005 [5] NCCL INFO NET/OFI [5] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00 | |
| ip-172-31-44-4:22497:22559 [5] NCCL INFO NET/OFI [5] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00 | |
| ip-172-31-47-240:24691:24761 [2] NCCL INFO NET/OFI [2] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-46-50:22043:22118 [3] NCCL INFO NET/OFI [3] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00 | |
| ip-172-31-45-171:25062:25131 [4] NCCL INFO NET/OFI [4] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00 | |
| ip-172-31-47-156:22565:22637 [5] NCCL INFO NET/OFI [5] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-37-141:25826:25891 [2] NCCL INFO NET/OFI [2] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00 | |
| ip-172-31-38-82:24982:25054 [4] NCCL INFO CUDA Dev 4[4], AWS Libfabric NIC distance : NODE NODE PHB NODE | |
| ip-172-31-36-188:22209:22274 [2] NCCL INFO NET/OFI [2] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-36-128:22021:22098 [0] NCCL INFO NET/OFI [0] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-38-14:25301:25375 [3] NCCL INFO NET/OFI [3] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00 | |
| ip-172-31-32-86:24696:24767 [5] NCCL INFO NET/OFI [5] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00 | |
| ip-172-31-33-235:21965:22037 [6] NCCL INFO NET/OFI [6] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00 | |
| ip-172-31-47-188:22013:22084 [5] NCCL INFO NET/OFI [5] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00 | |
| ip-172-31-44-230:21992:22064 [5] NCCL INFO NET/OFI [5] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00 | |
| ip-172-31-44-142:24929:25002 [0] NCCL INFO NET/OFI [0] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00 | |
| ip-172-31-44-4:22495:22558 [3] NCCL INFO NET/OFI [3] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00 | |
| ip-172-31-47-240:24693:24767 [4] NCCL INFO NET/OFI [4] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00 | |
| ip-172-31-46-50:22041:22117 [1] NCCL INFO NET/OFI [1] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-45-171:25063:25130 [5] NCCL INFO NET/OFI [5] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-47-156:22566:22635 [6] NCCL INFO NET/OFI [6] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00 | |
| ip-172-31-37-141:25824:25897 [0] NCCL INFO NET/OFI [0] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-38-82:24980:25051 [2] NCCL INFO NET/OFI [2] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00 | |
| ip-172-31-36-188:22211:22280 [4] NCCL INFO NET/OFI [4] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-36-128:22027:22092 [6] NCCL INFO NET/OFI [6] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-36-128:22027:22092 [6] NCCL INFO CUDA Dev 6[6], AWS Libfabric NIC distance : NODE NODE NODE PHB | |
| ip-172-31-38-14:25300:25369 [2] NCCL INFO NET/OFI [2] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00 | |
| ip-172-31-32-86:24696:24767 [5] NCCL INFO CUDA Dev 5[5], AWS Libfabric NIC distance : NODE NODE PHB NODE | |
| ip-172-31-33-235:21961:22023 [2] NCCL INFO NET/OFI [2] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00 | |
| ip-172-31-47-188:22009:22082 [1] NCCL INFO NET/OFI [1] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00 | |
| ip-172-31-44-230:21994:22063 [7] NCCL INFO NET/OFI [7] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00 | |
| ip-172-31-44-142:24934:25005 [5] NCCL INFO CUDA Dev 5[5], AWS Libfabric NIC distance : NODE NODE PHB NODE | |
| ip-172-31-44-4:22499:22562 [7] NCCL INFO NET/OFI [7] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00 | |
| ip-172-31-47-240:24692:24760 [3] NCCL INFO NET/OFI [3] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00 | |
| ip-172-31-46-50:22042:22112 [2] NCCL INFO NET/OFI [2] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-45-171:25061:25134 [3] NCCL INFO NET/OFI [3] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-47-156:22561:22636 [1] NCCL INFO NET/OFI [1] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-37-141:25825:25894 [1] NCCL INFO NET/OFI [1] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00 | |
| ip-172-31-38-82:24979:25052 [1] NCCL INFO NET/OFI [1] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00 | |
| ip-172-31-36-188:22208:22278 [1] NCCL INFO NET/OFI [1] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00 | |
| ip-172-31-36-128:22023:22093 [2] NCCL INFO NET/OFI [2] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-38-14:25303:25373 [5] NCCL INFO NET/OFI [5] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00 | |
| ip-172-31-32-86:24692:24769 [1] NCCL INFO NET/OFI [1] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00 | |
| ip-172-31-33-235:21963:22035 [4] NCCL INFO NET/OFI [4] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-47-188:22009:22082 [1] NCCL INFO CUDA Dev 1[1], AWS Libfabric NIC distance : PHB NODE NODE NODE | |
| ip-172-31-44-230:21988:22060 [1] NCCL INFO NET/OFI [1] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-44-142:24932:25006 [3] NCCL INFO NET/OFI [3] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00 | |
| ip-172-31-44-4:22497:22559 [5] NCCL INFO NET/OFI [5] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-47-240:24696:24763 [7] NCCL INFO NET/OFI [7] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-47-240:24696:24763 [7] NCCL INFO CUDA Dev 7[7], AWS Libfabric NIC distance : NODE NODE NODE PHB | |
| ip-172-31-46-50:22040:22119 [0] NCCL INFO NET/OFI [0] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-45-171:25059:25135 [1] NCCL INFO NET/OFI [1] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00 | |
| ip-172-31-47-156:22562:22633 [2] NCCL INFO NET/OFI [2] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-37-141:25831:25892 [7] NCCL INFO NET/OFI [7] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00 | |
| ip-172-31-38-82:24978:25047 [0] NCCL INFO NET/OFI [0] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00 | |
| ip-172-31-36-188:22207:22281 [0] NCCL INFO NET/OFI [0] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00 | |
| ip-172-31-36-128:22026:22094 [5] NCCL INFO NET/OFI [5] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-38-14:25302:25368 [4] NCCL INFO NET/OFI [4] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00 | |
| ip-172-31-32-86:24694:24770 [3] NCCL INFO NET/OFI [3] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00 | |
| ip-172-31-33-235:21966:22034 [7] NCCL INFO NET/OFI [7] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00 | |
| ip-172-31-47-188:22011:22083 [3] NCCL INFO NET/OFI [3] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00 | |
| ip-172-31-44-230:21989:22061 [2] NCCL INFO NET/OFI [2] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-44-142:24930:25003 [1] NCCL INFO NET/OFI [1] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-44-4:22496:22561 [4] NCCL INFO NET/OFI [4] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00 | |
| ip-172-31-47-240:24694:24764 [5] NCCL INFO NET/OFI [5] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-46-50:22046:22116 [6] NCCL INFO NET/OFI [6] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-46-50:22046:22116 [6] NCCL INFO CUDA Dev 6[6], AWS Libfabric NIC distance : NODE NODE NODE PHB | |
| ip-172-31-45-171:25060:25132 [2] NCCL INFO NET/OFI [2] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-47-156:22560:22634 [0] NCCL INFO NET/OFI [0] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00 | |
| ip-172-31-37-141:25827:25893 [3] NCCL INFO NET/OFI [3] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00 | |
| ip-172-31-38-82:24981:25048 [3] NCCL INFO NET/OFI [3] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00 | |
| ip-172-31-36-188:22209:22274 [2] NCCL INFO NET/OFI [2] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00 | |
| ip-172-31-36-128:22022:22097 [1] NCCL INFO NET/OFI [1] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00 | |
| ip-172-31-38-14:25299:25372 [1] NCCL INFO NET/OFI [1] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00 | |
| ip-172-31-32-86:24693:24766 [2] NCCL INFO NET/OFI [2] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00 | |
| ip-172-31-33-235:21959:22032 [0] NCCL INFO NET/OFI [0] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00 | |
| ip-172-31-47-188:22014:22087 [6] NCCL INFO NET/OFI [6] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00 | |
| ip-172-31-44-230:21987:22059 [0] NCCL INFO NET/OFI [0] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-44-142:24931:25004 [2] NCCL INFO NET/OFI [2] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00 | |
| ip-172-31-44-4:22493:22565 [1] NCCL INFO NET/OFI [1] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-47-240:24690:24765 [1] NCCL INFO NET/OFI [1] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00 | |
| ip-172-31-46-50:22044:22113 [4] NCCL INFO NET/OFI [4] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-45-171:25058:25128 [0] NCCL INFO NET/OFI [0] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00 | |
| ip-172-31-47-156:22565:22637 [5] NCCL INFO NET/OFI [5] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00 | |
| ip-172-31-37-141:25830:25895 [6] NCCL INFO NET/OFI [6] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-37-141:25830:25895 [6] NCCL INFO CUDA Dev 6[6], AWS Libfabric NIC distance : NODE NODE NODE PHB | |
| ip-172-31-38-82:24980:25051 [2] NCCL INFO NET/OFI [2] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00 | |
| ip-172-31-38-82:24980:25051 [2] NCCL INFO CUDA Dev 2[2], AWS Libfabric NIC distance : NODE PHB NODE NODE | |
| ip-172-31-36-188:22210:22277 [3] NCCL INFO NET/OFI [3] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-36-128:22021:22098 [0] NCCL INFO NET/OFI [0] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00 | |
| ip-172-31-38-14:25301:25375 [3] NCCL INFO NET/OFI [3] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-32-86:24691:24765 [0] NCCL INFO NET/OFI [0] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00 | |
| ip-172-31-33-235:21962:22036 [3] NCCL INFO NET/OFI [3] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-47-188:22008:22081 [0] NCCL INFO NET/OFI [0] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00 | |
| ip-172-31-44-230:21993:22066 [6] NCCL INFO NET/OFI [6] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00 | |
| ip-172-31-44-142:24933:25001 [4] NCCL INFO NET/OFI [4] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00 | |
| ip-172-31-44-4:22494:22564 [2] NCCL INFO NET/OFI [2] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-47-240:24689:24766 [0] NCCL INFO NET/OFI [0] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00 | |
| ip-172-31-46-50:22045:22115 [5] NCCL INFO NET/OFI [5] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-45-171:25065:25133 [7] NCCL INFO NET/OFI [7] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-45-171:25065:25133 [7] NCCL INFO CUDA Dev 7[7], AWS Libfabric NIC distance : NODE NODE NODE PHB | |
| ip-172-31-47-156:22564:22632 [4] NCCL INFO NET/OFI [4] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-37-141:25824:25897 [0] NCCL INFO NET/OFI [0] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00 | |
| ip-172-31-38-82:24981:25048 [3] NCCL INFO CUDA Dev 3[3], AWS Libfabric NIC distance : NODE PHB NODE NODE | |
| ip-172-31-36-188:22214:22275 [7] NCCL INFO NET/OFI [7] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-36-188:22214:22275 [7] NCCL INFO CUDA Dev 7[7], AWS Libfabric NIC distance : NODE NODE NODE PHB | |
| ip-172-31-36-128:22024:22091 [3] NCCL INFO NET/OFI [3] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00 | |
| ip-172-31-38-14:25300:25369 [2] NCCL INFO NET/OFI [2] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00 | |
| ip-172-31-32-86:24695:24771 [4] NCCL INFO NET/OFI [4] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00 | |
| ip-172-31-33-235:21959:22032 [0] NCCL INFO CUDA Dev 0[0], AWS Libfabric NIC distance : PHB NODE NODE NODE | |
| ip-172-31-47-188:22013:22084 [5] NCCL INFO NET/OFI [5] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00 | |
| ip-172-31-44-230:21989:22061 [2] NCCL INFO NET/OFI [2] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00 | |
| ip-172-31-44-142:24935:25000 [6] NCCL INFO NET/OFI [6] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-44-142:24935:25000 [6] NCCL INFO CUDA Dev 6[6], AWS Libfabric NIC distance : NODE NODE NODE PHB | |
| ip-172-31-44-4:22499:22562 [7] NCCL INFO NET/OFI [7] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-44-4:22499:22562 [7] NCCL INFO CUDA Dev 7[7], AWS Libfabric NIC distance : NODE NODE NODE PHB | |
| ip-172-31-47-240:24695:24762 [6] NCCL INFO NET/OFI [6] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-47-240:24695:24762 [6] NCCL INFO CUDA Dev 6[6], AWS Libfabric NIC distance : NODE NODE NODE PHB | |
| ip-172-31-46-50:22042:22112 [2] NCCL INFO NET/OFI [2] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00 | |
| ip-172-31-45-171:25063:25130 [5] NCCL INFO NET/OFI [5] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00 | |
| ip-172-31-47-156:22562:22633 [2] NCCL INFO NET/OFI [2] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00 | |
| ip-172-31-37-141:25826:25891 [2] NCCL INFO NET/OFI [2] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-38-82:24979:25052 [1] NCCL INFO NET/OFI [1] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00 | |
| ip-172-31-36-188:22213:22279 [6] NCCL INFO NET/OFI [6] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00 | |
| ip-172-31-36-128:22023:22093 [2] NCCL INFO NET/OFI [2] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00 | |
| ip-172-31-38-14:25298:25374 [0] NCCL INFO NET/OFI [0] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00 | |
| ip-172-31-32-86:24698:24764 [7] NCCL INFO NET/OFI [7] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00 | |
| ip-172-31-33-235:21964:22033 [5] NCCL INFO NET/OFI [5] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00 | |
| ip-172-31-47-188:22012:22080 [4] NCCL INFO NET/OFI [4] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00 | |
| ip-172-31-44-230:21994:22063 [7] NCCL INFO NET/OFI [7] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-44-230:21994:22063 [7] NCCL INFO CUDA Dev 7[7], AWS Libfabric NIC distance : NODE NODE NODE PHB | |
| ip-172-31-44-142:24937:25007 [7] NCCL INFO NET/OFI [7] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00 | |
| ip-172-31-44-4:22496:22561 [4] NCCL INFO NET/OFI [4] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-47-240:24691:24761 [2] NCCL INFO NET/OFI [2] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00 | |
| ip-172-31-46-50:22041:22117 [1] NCCL INFO NET/OFI [1] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00 | |
| ip-172-31-45-171:25061:25134 [3] NCCL INFO NET/OFI [3] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00 | |
| ip-172-31-47-156:22566:22635 [6] NCCL INFO NET/OFI [6] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-47-156:22566:22635 [6] NCCL INFO CUDA Dev 6[6], AWS Libfabric NIC distance : NODE NODE NODE PHB | |
| ip-172-31-37-141:25825:25894 [1] NCCL INFO NET/OFI [1] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00 | |
| ip-172-31-38-82:24979:25052 [1] NCCL INFO CUDA Dev 1[1], AWS Libfabric NIC distance : PHB NODE NODE NODE | |
| ip-172-31-36-188:22211:22280 [4] NCCL INFO NET/OFI [4] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00 | |
| ip-172-31-36-128:22025:22095 [4] NCCL INFO NET/OFI [4] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00 | |
| ip-172-31-38-14:25300:25369 [2] NCCL INFO CUDA Dev 2[2], AWS Libfabric NIC distance : NODE PHB NODE NODE | |
| ip-172-31-32-86:24693:24766 [2] NCCL INFO NET/OFI [2] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00 | |
| ip-172-31-33-235:21965:22037 [6] NCCL INFO NET/OFI [6] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00 | |
| ip-172-31-47-188:22014:22087 [6] NCCL INFO NET/OFI [6] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-47-188:22014:22087 [6] NCCL INFO CUDA Dev 6[6], AWS Libfabric NIC distance : NODE NODE NODE PHB | |
| ip-172-31-44-230:21988:22060 [1] NCCL INFO NET/OFI [1] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00 | |
| ip-172-31-44-142:24929:25002 [0] NCCL INFO NET/OFI [0] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00 | |
| ip-172-31-44-4:22493:22565 [1] NCCL INFO NET/OFI [1] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00 | |
| ip-172-31-47-240:24694:24764 [5] NCCL INFO NET/OFI [5] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00 | |
| ip-172-31-46-50:22043:22118 [3] NCCL INFO NET/OFI [3] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-45-171:25062:25131 [4] NCCL INFO NET/OFI [4] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-47-156:22561:22636 [1] NCCL INFO NET/OFI [1] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00 | |
| ip-172-31-37-141:25827:25893 [3] NCCL INFO NET/OFI [3] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00 | |
| ip-172-31-38-82:24978:25047 [0] NCCL INFO NET/OFI [0] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00 | |
| ip-172-31-36-188:22211:22280 [4] NCCL INFO CUDA Dev 4[4], AWS Libfabric NIC distance : NODE NODE PHB NODE | |
| ip-172-31-36-128:22026:22094 [5] NCCL INFO NET/OFI [5] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00 | |
| ip-172-31-38-14:25303:25373 [5] NCCL INFO NET/OFI [5] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-32-86:24693:24766 [2] NCCL INFO CUDA Dev 2[2], AWS Libfabric NIC distance : NODE PHB NODE NODE | |
| ip-172-31-33-235:21963:22035 [4] NCCL INFO NET/OFI [4] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00 | |
| ip-172-31-47-188:22008:22081 [0] NCCL INFO NET/OFI [0] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00 | |
| ip-172-31-44-230:21992:22064 [5] NCCL INFO NET/OFI [5] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00 | |
| ip-172-31-44-142:24930:25003 [1] NCCL INFO NET/OFI [1] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00 | |
| ip-172-31-44-4:22497:22559 [5] NCCL INFO NET/OFI [5] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00 | |
| ip-172-31-44-4:22497:22559 [5] NCCL INFO CUDA Dev 5[5], AWS Libfabric NIC distance : NODE NODE PHB NODE | |
| ip-172-31-47-240:24693:24767 [4] NCCL INFO NET/OFI [4] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00 | |
| ip-172-31-46-50:22040:22119 [0] NCCL INFO NET/OFI [0] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00 | |
| ip-172-31-45-171:25063:25130 [5] NCCL INFO CUDA Dev 5[5], AWS Libfabric NIC distance : NODE NODE PHB NODE | |
| ip-172-31-47-156:22565:22637 [5] NCCL INFO CUDA Dev 5[5], AWS Libfabric NIC distance : NODE NODE PHB NODE | |
| ip-172-31-37-141:25831:25892 [7] NCCL INFO NET/OFI [7] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-37-141:25831:25892 [7] NCCL INFO CUDA Dev 7[7], AWS Libfabric NIC distance : NODE NODE NODE PHB | |
| ip-172-31-38-82:24978:25047 [0] NCCL INFO NET/OFI [0] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00 | |
| ip-172-31-36-188:22208:22278 [1] NCCL INFO NET/OFI [1] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00 | |
| ip-172-31-36-128:22026:22094 [5] NCCL INFO CUDA Dev 5[5], AWS Libfabric NIC distance : NODE NODE PHB NODE | |
| ip-172-31-38-14:25299:25372 [1] NCCL INFO NET/OFI [1] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00 | |
| ip-172-31-32-86:24692:24769 [1] NCCL INFO NET/OFI [1] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00 | |
| ip-172-31-33-235:21966:22034 [7] NCCL INFO NET/OFI [7] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-33-235:21966:22034 [7] NCCL INFO CUDA Dev 7[7], AWS Libfabric NIC distance : NODE NODE NODE PHB | |
| ip-172-31-47-188:22015:22086 [7] NCCL INFO NET/OFI [7] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00 | |
| ip-172-31-44-230:21987:22059 [0] NCCL INFO NET/OFI [0] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00 | |
| ip-172-31-44-142:24931:25004 [2] NCCL INFO NET/OFI [2] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00 | |
| ip-172-31-44-4:22495:22558 [3] NCCL INFO NET/OFI [3] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-44-4:22495:22558 [3] NCCL INFO NET/OFI [3] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00 | |
| ip-172-31-47-240:24694:24764 [5] NCCL INFO CUDA Dev 5[5], AWS Libfabric NIC distance : NODE NODE PHB NODE | |
| ip-172-31-46-50:22044:22113 [4] NCCL INFO NET/OFI [4] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00 | |
| ip-172-31-45-171:25060:25132 [2] NCCL INFO NET/OFI [2] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00 | |
| ip-172-31-47-156:22563:22638 [3] NCCL INFO NET/OFI [3] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-37-141:25827:25893 [3] NCCL INFO CUDA Dev 3[3], AWS Libfabric NIC distance : NODE PHB NODE NODE | |
| ip-172-31-38-82:24978:25047 [0] NCCL INFO CUDA Dev 0[0], AWS Libfabric NIC distance : PHB NODE NODE NODE | |
| ip-172-31-36-188:22210:22277 [3] NCCL INFO NET/OFI [3] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00 | |
| ip-172-31-36-128:22024:22091 [3] NCCL INFO NET/OFI [3] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00 | |
| ip-172-31-38-14:25302:25368 [4] NCCL INFO NET/OFI [4] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00 | |
| ip-172-31-32-86:24694:24770 [3] NCCL INFO NET/OFI [3] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-33-235:21962:22036 [3] NCCL INFO NET/OFI [3] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00 | |
| ip-172-31-47-188:22011:22083 [3] NCCL INFO NET/OFI [3] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-44-230:21993:22066 [6] NCCL INFO NET/OFI [6] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-44-230:21993:22066 [6] NCCL INFO CUDA Dev 6[6], AWS Libfabric NIC distance : NODE NODE NODE PHB | |
| ip-172-31-44-142:24932:25006 [3] NCCL INFO NET/OFI [3] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-44-4:22494:22564 [2] NCCL INFO NET/OFI [2] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00 | |
| ip-172-31-47-240:24692:24760 [3] NCCL INFO NET/OFI [3] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-46-50:22045:22115 [5] NCCL INFO NET/OFI [5] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00 | |
| ip-172-31-45-171:25059:25135 [1] NCCL INFO NET/OFI [1] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00 | |
| ip-172-31-47-156:22564:22632 [4] NCCL INFO NET/OFI [4] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00 | |
| ip-172-31-37-141:25826:25891 [2] NCCL INFO NET/OFI [2] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00 | |
| ip-172-31-36-188:22209:22274 [2] NCCL INFO NET/OFI [2] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00 | |
| ip-172-31-36-188:22209:22274 [2] NCCL INFO CUDA Dev 2[2], AWS Libfabric NIC distance : NODE PHB NODE NODE | |
| ip-172-31-36-128:22023:22093 [2] NCCL INFO NET/OFI [2] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00 | |
| ip-172-31-38-14:25301:25375 [3] NCCL INFO NET/OFI [3] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00 | |
| ip-172-31-32-86:24691:24765 [0] NCCL INFO NET/OFI [0] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00 | |
| ip-172-31-33-235:21963:22035 [4] NCCL INFO CUDA Dev 4[4], AWS Libfabric NIC distance : NODE NODE PHB NODE | |
| ip-172-31-47-188:22013:22084 [5] NCCL INFO NET/OFI [5] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-44-230:21989:22061 [2] NCCL INFO NET/OFI [2] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00 | |
| ip-172-31-44-142:24931:25004 [2] NCCL INFO CUDA Dev 2[2], AWS Libfabric NIC distance : NODE PHB NODE NODE | |
| ip-172-31-44-4:22496:22561 [4] NCCL INFO NET/OFI [4] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00 | |
| ip-172-31-47-240:24690:24765 [1] NCCL INFO NET/OFI [1] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00 | |
| ip-172-31-46-50:22042:22112 [2] NCCL INFO NET/OFI [2] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00 | |
| ip-172-31-45-171:25058:25128 [0] NCCL INFO NET/OFI [0] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00 | |
| ip-172-31-47-156:22560:22634 [0] NCCL INFO NET/OFI [0] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00 | |
| ip-172-31-37-141:25825:25894 [1] NCCL INFO NET/OFI [1] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00 | |
| ip-172-31-36-188:22213:22279 [6] NCCL INFO NET/OFI [6] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-36-188:22213:22279 [6] NCCL INFO CUDA Dev 6[6], AWS Libfabric NIC distance : NODE NODE NODE PHB | |
| ip-172-31-36-128:22022:22097 [1] NCCL INFO NET/OFI [1] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00 | |
| ip-172-31-38-14:25298:25374 [0] NCCL INFO NET/OFI [0] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00 | |
| ip-172-31-32-86:24695:24771 [4] NCCL INFO NET/OFI [4] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-33-235:21961:22023 [2] NCCL INFO NET/OFI [2] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-47-188:22012:22080 [4] NCCL INFO NET/OFI [4] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-44-230:21988:22060 [1] NCCL INFO NET/OFI [1] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00 | |
| ip-172-31-44-142:24933:25001 [4] NCCL INFO NET/OFI [4] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-44-4:22494:22564 [2] NCCL INFO NET/OFI [2] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00 | |
| ip-172-31-47-240:24691:24761 [2] NCCL INFO NET/OFI [2] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00 | |
| ip-172-31-46-50:22045:22115 [5] NCCL INFO CUDA Dev 5[5], AWS Libfabric NIC distance : NODE NODE PHB NODE | |
| ip-172-31-45-171:25062:25131 [4] NCCL INFO NET/OFI [4] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00 | |
| ip-172-31-47-156:22564:22632 [4] NCCL INFO CUDA Dev 4[4], AWS Libfabric NIC distance : NODE NODE PHB NODE | |
| ip-172-31-37-141:25824:25897 [0] NCCL INFO NET/OFI [0] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00 | |
| ip-172-31-36-188:22207:22281 [0] NCCL INFO NET/OFI [0] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00 | |
| ip-172-31-36-188:22207:22281 [0] NCCL INFO NET/OFI [0] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00 | |
| ip-172-31-36-128:22023:22093 [2] NCCL INFO CUDA Dev 2[2], AWS Libfabric NIC distance : NODE PHB NODE NODE | |
| ip-172-31-38-14:25303:25373 [5] NCCL INFO NET/OFI [5] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00 | |
| ip-172-31-32-86:24698:24764 [7] NCCL INFO NET/OFI [7] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00 | |
| ip-172-31-33-235:21964:22033 [5] NCCL INFO NET/OFI [5] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-47-188:22008:22081 [0] NCCL INFO NET/OFI [0] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00 | |
| ip-172-31-44-230:21989:22061 [2] NCCL INFO CUDA Dev 2[2], AWS Libfabric NIC distance : NODE PHB NODE NODE | |
| ip-172-31-44-142:24937:25007 [7] NCCL INFO NET/OFI [7] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00 | |
| ip-172-31-44-4:22496:22561 [4] NCCL INFO CUDA Dev 4[4], AWS Libfabric NIC distance : NODE NODE PHB NODE | |
| ip-172-31-47-240:24689:24766 [0] NCCL INFO NET/OFI [0] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00 | |
| ip-172-31-46-50:22044:22113 [4] NCCL INFO CUDA Dev 4[4], AWS Libfabric NIC distance : NODE NODE PHB NODE | |
| ip-172-31-45-171:25061:25134 [3] NCCL INFO NET/OFI [3] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00 | |
| ip-172-31-47-156:22562:22633 [2] NCCL INFO NET/OFI [2] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00 | |
| ip-172-31-37-141:25825:25894 [1] NCCL INFO CUDA Dev 1[1], AWS Libfabric NIC distance : PHB NODE NODE NODE | |
| ip-172-31-36-188:22208:22278 [1] NCCL INFO NET/OFI [1] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00 | |
| ip-172-31-36-188:22208:22278 [1] NCCL INFO CUDA Dev 1[1], AWS Libfabric NIC distance : PHB NODE NODE NODE | |
| ip-172-31-36-128:22021:22098 [0] NCCL INFO NET/OFI [0] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00 | |
| ip-172-31-38-14:25298:25374 [0] NCCL INFO CUDA Dev 0[0], AWS Libfabric NIC distance : PHB NODE NODE NODE | |
| ip-172-31-32-86:24694:24770 [3] NCCL INFO NET/OFI [3] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00 | |
| ip-172-31-33-235:21965:22037 [6] NCCL INFO NET/OFI [6] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-33-235:21965:22037 [6] NCCL INFO CUDA Dev 6[6], AWS Libfabric NIC distance : NODE NODE NODE PHB | |
| ip-172-31-47-188:22008:22081 [0] NCCL INFO CUDA Dev 0[0], AWS Libfabric NIC distance : PHB NODE NODE NODE | |
| ip-172-31-44-230:21987:22059 [0] NCCL INFO NET/OFI [0] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00 | |
| ip-172-31-44-142:24932:25006 [3] NCCL INFO NET/OFI [3] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00 | |
| ip-172-31-44-4:22493:22565 [1] NCCL INFO NET/OFI [1] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00 | |
| ip-172-31-47-240:24691:24761 [2] NCCL INFO CUDA Dev 2[2], AWS Libfabric NIC distance : NODE PHB NODE NODE | |
| ip-172-31-46-50:22042:22112 [2] NCCL INFO CUDA Dev 2[2], AWS Libfabric NIC distance : NODE PHB NODE NODE | |
| ip-172-31-45-171:25059:25135 [1] NCCL INFO NET/OFI [1] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00 | |
| ip-172-31-47-156:22561:22636 [1] NCCL INFO NET/OFI [1] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00 | |
| ip-172-31-37-141:25824:25897 [0] NCCL INFO NET/OFI [0] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00 | |
| ip-172-31-36-188:22210:22277 [3] NCCL INFO NET/OFI [3] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00 | |
| ip-172-31-36-128:22024:22091 [3] NCCL INFO CUDA Dev 3[3], AWS Libfabric NIC distance : NODE PHB NODE NODE | |
| ip-172-31-38-14:25303:25373 [5] NCCL INFO CUDA Dev 5[5], AWS Libfabric NIC distance : NODE NODE PHB NODE | |
| ip-172-31-32-86:24692:24769 [1] NCCL INFO NET/OFI [1] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00 | |
| ip-172-31-33-235:21962:22036 [3] NCCL INFO NET/OFI [3] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00 | |
| ip-172-31-47-188:22013:22084 [5] NCCL INFO NET/OFI [5] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00 | |
| ip-172-31-44-230:21992:22064 [5] NCCL INFO NET/OFI [5] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-44-142:24933:25001 [4] NCCL INFO NET/OFI [4] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00 | |
| ip-172-31-44-4:22494:22564 [2] NCCL INFO CUDA Dev 2[2], AWS Libfabric NIC distance : NODE PHB NODE NODE | |
| ip-172-31-47-240:24692:24760 [3] NCCL INFO NET/OFI [3] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00 | |
| ip-172-31-46-50:22043:22118 [3] NCCL INFO NET/OFI [3] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00 | |
| ip-172-31-45-171:25062:25131 [4] NCCL INFO CUDA Dev 4[4], AWS Libfabric NIC distance : NODE NODE PHB NODE | |
| ip-172-31-47-156:22563:22638 [3] NCCL INFO NET/OFI [3] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00 | |
| ip-172-31-37-141:25826:25891 [2] NCCL INFO NET/OFI [2] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00 | |
| ip-172-31-36-188:22207:22281 [0] NCCL INFO CUDA Dev 0[0], AWS Libfabric NIC distance : PHB NODE NODE NODE | |
| ip-172-31-36-128:22025:22095 [4] NCCL INFO NET/OFI [4] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-38-14:25299:25372 [1] NCCL INFO NET/OFI [1] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00 | |
| ip-172-31-32-86:24692:24769 [1] NCCL INFO CUDA Dev 1[1], AWS Libfabric NIC distance : PHB NODE NODE NODE | |
| ip-172-31-33-235:21961:22023 [2] NCCL INFO NET/OFI [2] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00 | |
| ip-172-31-47-188:22015:22086 [7] NCCL INFO NET/OFI [7] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-47-188:22015:22086 [7] NCCL INFO CUDA Dev 7[7], AWS Libfabric NIC distance : NODE NODE NODE PHB | |
| ip-172-31-44-230:21992:22064 [5] NCCL INFO NET/OFI [5] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00 | |
| ip-172-31-44-142:24929:25002 [0] NCCL INFO NET/OFI [0] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00 | |
| ip-172-31-44-4:22495:22558 [3] NCCL INFO NET/OFI [3] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00 | |
| ip-172-31-47-240:24693:24767 [4] NCCL INFO NET/OFI [4] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-46-50:22041:22117 [1] NCCL INFO NET/OFI [1] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00 | |
| ip-172-31-45-171:25058:25128 [0] NCCL INFO NET/OFI [0] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00 | |
| ip-172-31-47-156:22562:22633 [2] NCCL INFO CUDA Dev 2[2], AWS Libfabric NIC distance : NODE PHB NODE NODE | |
| ip-172-31-37-141:25824:25897 [0] NCCL INFO CUDA Dev 0[0], AWS Libfabric NIC distance : PHB NODE NODE NODE | |
| ip-172-31-36-188:22210:22277 [3] NCCL INFO CUDA Dev 3[3], AWS Libfabric NIC distance : NODE PHB NODE NODE | |
| ip-172-31-36-128:22021:22098 [0] NCCL INFO NET/OFI [0] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00 | |
| ip-172-31-38-14:25302:25368 [4] NCCL INFO NET/OFI [4] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-32-86:24691:24765 [0] NCCL INFO NET/OFI [0] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00 | |
| ip-172-31-33-235:21964:22033 [5] NCCL INFO NET/OFI [5] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00 | |
| ip-172-31-47-188:22011:22083 [3] NCCL INFO NET/OFI [3] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00 | |
| ip-172-31-44-230:21988:22060 [1] NCCL INFO NET/OFI [1] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00 | |
| ip-172-31-44-142:24930:25003 [1] NCCL INFO NET/OFI [1] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00 | |
| ip-172-31-44-4:22495:22558 [3] NCCL INFO CUDA Dev 3[3], AWS Libfabric NIC distance : NODE PHB NODE NODE | |
| ip-172-31-47-240:24690:24765 [1] NCCL INFO NET/OFI [1] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00 | |
| ip-172-31-46-50:22040:22119 [0] NCCL INFO NET/OFI [0] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00 | |
| ip-172-31-45-171:25061:25134 [3] NCCL INFO CUDA Dev 3[3], AWS Libfabric NIC distance : NODE PHB NODE NODE | |
| ip-172-31-47-156:22560:22634 [0] NCCL INFO NET/OFI [0] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00 | |
| ip-172-31-37-141:25826:25891 [2] NCCL INFO CUDA Dev 2[2], AWS Libfabric NIC distance : NODE PHB NODE NODE | |
| ip-172-31-36-128:22022:22097 [1] NCCL INFO NET/OFI [1] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00 | |
| ip-172-31-38-14:25301:25375 [3] NCCL INFO NET/OFI [3] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00 | |
| ip-172-31-32-86:24691:24765 [0] NCCL INFO CUDA Dev 0[0], AWS Libfabric NIC distance : PHB NODE NODE NODE | |
| ip-172-31-33-235:21962:22036 [3] NCCL INFO CUDA Dev 3[3], AWS Libfabric NIC distance : NODE PHB NODE NODE | |
| ip-172-31-47-188:22012:22080 [4] NCCL INFO NET/OFI [4] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00 | |
| ip-172-31-47-188:22012:22080 [4] NCCL INFO CUDA Dev 4[4], AWS Libfabric NIC distance : NODE NODE PHB NODE | |
| ip-172-31-44-230:21992:22064 [5] NCCL INFO CUDA Dev 5[5], AWS Libfabric NIC distance : NODE NODE PHB NODE | |
| ip-172-31-44-142:24929:25002 [0] NCCL INFO CUDA Dev 0[0], AWS Libfabric NIC distance : PHB NODE NODE NODE | |
| ip-172-31-44-4:22493:22565 [1] NCCL INFO NET/OFI [1] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00 | |
| ip-172-31-47-240:24689:24766 [0] NCCL INFO NET/OFI [0] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00 | |
| ip-172-31-47-240:24689:24766 [0] NCCL INFO CUDA Dev 0[0], AWS Libfabric NIC distance : PHB NODE NODE NODE | |
| ip-172-31-46-50:22043:22118 [3] NCCL INFO NET/OFI [3] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00 | |
| ip-172-31-45-171:25060:25132 [2] NCCL INFO NET/OFI [2] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00 | |
| ip-172-31-45-171:25060:25132 [2] NCCL INFO CUDA Dev 2[2], AWS Libfabric NIC distance : NODE PHB NODE NODE | |
| ip-172-31-47-156:22563:22638 [3] NCCL INFO NET/OFI [3] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00 | |
| ip-172-31-36-128:22021:22098 [0] NCCL INFO CUDA Dev 0[0], AWS Libfabric NIC distance : PHB NODE NODE NODE | |
| ip-172-31-38-14:25299:25372 [1] NCCL INFO CUDA Dev 1[1], AWS Libfabric NIC distance : PHB NODE NODE NODE | |
| ip-172-31-32-86:24698:24764 [7] NCCL INFO NET/OFI [7] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-32-86:24698:24764 [7] NCCL INFO CUDA Dev 7[7], AWS Libfabric NIC distance : NODE NODE NODE PHB | |
| ip-172-31-33-235:21964:22033 [5] NCCL INFO CUDA Dev 5[5], AWS Libfabric NIC distance : NODE NODE PHB NODE | |
| ip-172-31-47-188:22013:22084 [5] NCCL INFO CUDA Dev 5[5], AWS Libfabric NIC distance : NODE NODE PHB NODE | |
| ip-172-31-44-230:21987:22059 [0] NCCL INFO NET/OFI [0] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00 | |
| ip-172-31-44-142:24933:25001 [4] NCCL INFO CUDA Dev 4[4], AWS Libfabric NIC distance : NODE NODE PHB NODE | |
| ip-172-31-44-4:22493:22565 [1] NCCL INFO CUDA Dev 1[1], AWS Libfabric NIC distance : PHB NODE NODE NODE | |
| ip-172-31-47-240:24693:24767 [4] NCCL INFO NET/OFI [4] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00 | |
| ip-172-31-47-240:24693:24767 [4] NCCL INFO CUDA Dev 4[4], AWS Libfabric NIC distance : NODE NODE PHB NODE | |
| ip-172-31-46-50:22040:22119 [0] NCCL INFO NET/OFI [0] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00 | |
| ip-172-31-45-171:25059:25135 [1] NCCL INFO CUDA Dev 1[1], AWS Libfabric NIC distance : PHB NODE NODE NODE | |
| ip-172-31-47-156:22561:22636 [1] NCCL INFO NET/OFI [1] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00 | |
| ip-172-31-47-156:22561:22636 [1] NCCL INFO CUDA Dev 1[1], AWS Libfabric NIC distance : PHB NODE NODE NODE | |
| ip-172-31-36-128:22022:22097 [1] NCCL INFO CUDA Dev 1[1], AWS Libfabric NIC distance : PHB NODE NODE NODE | |
| ip-172-31-38-14:25301:25375 [3] NCCL INFO CUDA Dev 3[3], AWS Libfabric NIC distance : NODE PHB NODE NODE | |
| ip-172-31-32-86:24694:24770 [3] NCCL INFO NET/OFI [3] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00 | |
| ip-172-31-33-235:21961:22023 [2] NCCL INFO NET/OFI [2] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00 | |
| ip-172-31-47-188:22011:22083 [3] NCCL INFO NET/OFI [3] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00 | |
| ip-172-31-44-230:21988:22060 [1] NCCL INFO CUDA Dev 1[1], AWS Libfabric NIC distance : PHB NODE NODE NODE | |
| ip-172-31-44-142:24937:25007 [7] NCCL INFO NET/OFI [7] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-44-142:24937:25007 [7] NCCL INFO CUDA Dev 7[7], AWS Libfabric NIC distance : NODE NODE NODE PHB | |
| ip-172-31-47-240:24690:24765 [1] NCCL INFO CUDA Dev 1[1], AWS Libfabric NIC distance : PHB NODE NODE NODE | |
| ip-172-31-46-50:22043:22118 [3] NCCL INFO CUDA Dev 3[3], AWS Libfabric NIC distance : NODE PHB NODE NODE | |
| ip-172-31-45-171:25058:25128 [0] NCCL INFO CUDA Dev 0[0], AWS Libfabric NIC distance : PHB NODE NODE NODE | |
| ip-172-31-47-156:22560:22634 [0] NCCL INFO CUDA Dev 0[0], AWS Libfabric NIC distance : PHB NODE NODE NODE | |
| ip-172-31-36-128:22025:22095 [4] NCCL INFO NET/OFI [4] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00 | |
| ip-172-31-38-14:25302:25368 [4] NCCL INFO NET/OFI [4] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00 | |
| ip-172-31-32-86:24695:24771 [4] NCCL INFO NET/OFI [4] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00 | |
| ip-172-31-33-235:21961:22023 [2] NCCL INFO CUDA Dev 2[2], AWS Libfabric NIC distance : NODE PHB NODE NODE | |
| ip-172-31-47-188:22011:22083 [3] NCCL INFO CUDA Dev 3[3], AWS Libfabric NIC distance : NODE PHB NODE NODE | |
| ip-172-31-44-230:21987:22059 [0] NCCL INFO CUDA Dev 0[0], AWS Libfabric NIC distance : PHB NODE NODE NODE | |
| ip-172-31-44-142:24932:25006 [3] NCCL INFO NET/OFI [3] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00 | |
| ip-172-31-47-240:24692:24760 [3] NCCL INFO NET/OFI [3] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00 | |
| ip-172-31-47-240:24692:24760 [3] NCCL INFO CUDA Dev 3[3], AWS Libfabric NIC distance : NODE PHB NODE NODE | |
| ip-172-31-46-50:22041:22117 [1] NCCL INFO NET/OFI [1] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00 | |
| ip-172-31-47-156:22563:22638 [3] NCCL INFO CUDA Dev 3[3], AWS Libfabric NIC distance : NODE PHB NODE NODE | |
| ip-172-31-36-128:22025:22095 [4] NCCL INFO CUDA Dev 4[4], AWS Libfabric NIC distance : NODE NODE PHB NODE | |
| ip-172-31-38-14:25302:25368 [4] NCCL INFO CUDA Dev 4[4], AWS Libfabric NIC distance : NODE NODE PHB NODE | |
| ip-172-31-32-86:24694:24770 [3] NCCL INFO CUDA Dev 3[3], AWS Libfabric NIC distance : NODE PHB NODE NODE | |
| ip-172-31-44-142:24930:25003 [1] NCCL INFO NET/OFI [1] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00 | |
| ip-172-31-46-50:22040:22119 [0] NCCL INFO CUDA Dev 0[0], AWS Libfabric NIC distance : PHB NODE NODE NODE | |
| ip-172-31-32-86:24695:24771 [4] NCCL INFO CUDA Dev 4[4], AWS Libfabric NIC distance : NODE NODE PHB NODE | |
| ip-172-31-44-142:24932:25006 [3] NCCL INFO CUDA Dev 3[3], AWS Libfabric NIC distance : NODE PHB NODE NODE | |
| ip-172-31-46-50:22041:22117 [1] NCCL INFO CUDA Dev 1[1], AWS Libfabric NIC distance : PHB NODE NODE NODE | |
| ip-172-31-44-142:24930:25003 [1] NCCL INFO CUDA Dev 1[1], AWS Libfabric NIC distance : PHB NODE NODE NODE | |
| ip-172-31-39-141:20309:20382 [6] NCCL INFO NET/OFI [6] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00 | |
| ip-172-31-39-141:20309:20382 [6] NCCL INFO NET/OFI [6] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00 | |
| ip-172-31-39-141:20309:20382 [6] NCCL INFO NET/OFI [6] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00 | |
| ip-172-31-39-141:20309:20382 [6] NCCL INFO NET/OFI [6] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-39-141:20309:20382 [6] NCCL INFO CUDA Dev 6[6], AWS Libfabric NIC distance : NODE NODE NODE PHB | |
| ip-172-31-39-141:20310:20383 [7] NCCL INFO NET/OFI [7] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00 | |
| ip-172-31-39-141:20308:20378 [5] NCCL INFO NET/OFI [5] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00 | |
| ip-172-31-39-141:20310:20383 [7] NCCL INFO NET/OFI [7] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00 | |
| ip-172-31-39-141:20307:20377 [4] NCCL INFO NET/OFI [4] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00 | |
| ip-172-31-39-141:20308:20378 [5] NCCL INFO NET/OFI [5] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00 | |
| ip-172-31-39-141:20310:20383 [7] NCCL INFO NET/OFI [7] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00 | |
| ip-172-31-39-141:20306:20379 [3] NCCL INFO NET/OFI [3] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00 | |
| ip-172-31-39-141:20307:20377 [4] NCCL INFO NET/OFI [4] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00 | |
| ip-172-31-39-141:20310:20383 [7] NCCL INFO NET/OFI [7] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-39-141:20310:20383 [7] NCCL INFO CUDA Dev 7[7], AWS Libfabric NIC distance : NODE NODE NODE PHB | |
| ip-172-31-39-141:20308:20378 [5] NCCL INFO NET/OFI [5] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-39-141:20305:20380 [2] NCCL INFO NET/OFI [2] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00 | |
| ip-172-31-39-141:20306:20379 [3] NCCL INFO NET/OFI [3] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-39-141:20304:20381 [1] NCCL INFO NET/OFI [1] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-39-141:20308:20378 [5] NCCL INFO NET/OFI [5] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00 | |
| ip-172-31-39-141:20307:20377 [4] NCCL INFO NET/OFI [4] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-39-141:20303:20372 [0] NCCL INFO NET/OFI [0] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-39-141:20308:20378 [5] NCCL INFO CUDA Dev 5[5], AWS Libfabric NIC distance : NODE NODE PHB NODE | |
| ip-172-31-39-141:20305:20380 [2] NCCL INFO NET/OFI [2] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-39-141:20304:20381 [1] NCCL INFO NET/OFI [1] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00 | |
| ip-172-31-39-141:20306:20379 [3] NCCL INFO NET/OFI [3] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00 | |
| ip-172-31-39-141:20307:20377 [4] NCCL INFO NET/OFI [4] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00 | |
| ip-172-31-39-141:20303:20372 [0] NCCL INFO NET/OFI [0] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00 | |
| ip-172-31-39-141:20307:20377 [4] NCCL INFO CUDA Dev 4[4], AWS Libfabric NIC distance : NODE NODE PHB NODE | |
| ip-172-31-39-141:20305:20380 [2] NCCL INFO NET/OFI [2] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00 | |
| ip-172-31-39-141:20306:20379 [3] NCCL INFO NET/OFI [3] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00 | |
| ip-172-31-39-141:20304:20381 [1] NCCL INFO NET/OFI [1] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00 | |
| ip-172-31-39-141:20306:20379 [3] NCCL INFO CUDA Dev 3[3], AWS Libfabric NIC distance : NODE PHB NODE NODE | |
| ip-172-31-39-141:20305:20380 [2] NCCL INFO NET/OFI [2] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00 | |
| ip-172-31-39-141:20304:20381 [1] NCCL INFO NET/OFI [1] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00 | |
| ip-172-31-39-141:20303:20372 [0] NCCL INFO NET/OFI [0] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00 | |
| ip-172-31-39-141:20305:20380 [2] NCCL INFO CUDA Dev 2[2], AWS Libfabric NIC distance : NODE PHB NODE NODE | |
| ip-172-31-39-141:20304:20381 [1] NCCL INFO CUDA Dev 1[1], AWS Libfabric NIC distance : PHB NODE NODE NODE | |
| ip-172-31-39-141:20303:20372 [0] NCCL INFO NET/OFI [0] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00 | |
| ip-172-31-39-141:20303:20372 [0] NCCL INFO CUDA Dev 0[0], AWS Libfabric NIC distance : PHB NODE NODE NODE | |
| ip-172-31-39-141:20303:20372 [0] NCCL INFO Channel 00 : 0 2 3 7 4 6 5 1 8 10 11 15 12 14 13 9 16 18 19 23 | |
| ip-172-31-39-141:20303:20372 [0] NCCL INFO Channel 01 : 0 1 5 6 4 7 3 10 8 9 13 14 12 15 11 18 16 17 21 22 | |
| ip-172-31-39-141:20303:20372 [0] NCCL INFO Channel 02 : 0 3 1 2 6 7 5 12 8 11 9 10 14 15 13 20 16 19 17 18 | |
| ip-172-31-39-141:20303:20372 [0] NCCL INFO Channel 03 : 0 4 5 7 14 10 9 11 8 12 13 15 22 18 17 19 16 20 21 23 | |
| ip-172-31-39-141:20303:20372 [0] NCCL INFO Channel 04 : 0 2 3 7 4 6 5 1 8 10 11 15 12 14 13 9 16 18 19 23 | |
| ip-172-31-39-141:20303:20372 [0] NCCL INFO Channel 05 : 0 1 5 6 4 7 3 10 8 9 13 14 12 15 11 18 16 17 21 22 | |
| ip-172-31-39-141:20303:20372 [0] NCCL INFO Channel 06 : 0 3 1 2 6 7 5 12 8 11 9 10 14 15 13 20 16 19 17 18 | |
| ip-172-31-39-141:20303:20372 [0] NCCL INFO Channel 07 : 0 4 5 7 14 10 9 11 8 12 13 15 22 18 17 19 16 20 21 23 | |
| ip-172-31-38-14:25298:25374 [0] NCCL INFO NET/OFI [0] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-39-141:20303:20372 [0] NCCL INFO NET/OFI [0] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-33-235:21959:22032 [0] NCCL INFO NET/OFI [0] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-44-142:24929:25002 [0] NCCL INFO NET/OFI [0] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-37-141:25824:25897 [0] NCCL INFO NET/OFI [0] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-38-82:24978:25047 [0] NCCL INFO NET/OFI [0] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-45-171:25058:25128 [0] NCCL INFO NET/OFI [0] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-44-4:22492:22563 [0] NCCL INFO NET/OFI [0] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-36-188:22207:22281 [0] NCCL INFO NET/OFI [0] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-36-128:22021:22098 [0] NCCL INFO NET/OFI [0] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-32-86:24691:24765 [0] NCCL INFO NET/OFI [0] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-44-230:21987:22059 [0] NCCL INFO NET/OFI [0] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-47-240:24689:24766 [0] NCCL INFO NET/OFI [0] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-47-156:22560:22634 [0] NCCL INFO NET/OFI [0] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-47-188:22008:22081 [0] NCCL INFO NET/OFI [0] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-46-50:22040:22119 [0] NCCL INFO NET/OFI [0] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-38-14:25298:25374 [0] NCCL INFO Ring 00 : 1 -> 8 [receive] via NET/AWS Libfabric/0 | |
| ip-172-31-39-141:20303:20372 [0] NCCL INFO Ring 00 : 121 -> 0 [receive] via NET/AWS Libfabric/0 | |
| ip-172-31-38-82:24978:25047 [0] NCCL INFO Ring 00 : 33 -> 40 [receive] via NET/AWS Libfabric/0 | |
| ip-172-31-44-142:24929:25002 [0] NCCL INFO Ring 00 : 9 -> 16 [receive] via NET/AWS Libfabric/0 | |
| ip-172-31-33-235:21959:22032 [0] NCCL INFO Ring 00 : 97 -> 104 [receive] via NET/AWS Libfabric/0 | |
| ip-172-31-44-4:22492:22563 [0] NCCL INFO Ring 00 : 113 -> 120 [receive] via NET/AWS Libfabric/0 | |
| ip-172-31-37-141:25824:25897 [0] NCCL INFO Ring 00 : 25 -> 32 [receive] via NET/AWS Libfabric/0 | |
| ip-172-31-45-171:25058:25128 [0] NCCL INFO Ring 00 : 17 -> 24 [receive] via NET/AWS Libfabric/0 | |
| ip-172-31-36-128:22021:22098 [0] NCCL INFO Ring 00 : 105 -> 112 [receive] via NET/AWS Libfabric/0 | |
| ip-172-31-36-188:22207:22281 [0] NCCL INFO Ring 00 : 81 -> 88 [receive] via NET/AWS Libfabric/0 | |
| ip-172-31-44-230:21987:22059 [0] NCCL INFO Ring 00 : 89 -> 96 [receive] via NET/AWS Libfabric/0 | |
| ip-172-31-32-86:24691:24765 [0] NCCL INFO Ring 00 : 41 -> 48 [receive] via NET/AWS Libfabric/0 | |
| ip-172-31-47-240:24689:24766 [0] NCCL INFO Ring 00 : 49 -> 56 [receive] via NET/AWS Libfabric/0 | |
| ip-172-31-47-156:22560:22634 [0] NCCL INFO Ring 00 : 73 -> 80 [receive] via NET/AWS Libfabric/0 | |
| ip-172-31-46-50:22040:22119 [0] NCCL INFO Ring 00 : 57 -> 64 [receive] via NET/AWS Libfabric/0 | |
| ip-172-31-47-188:22008:22081 [0] NCCL INFO Ring 00 : 65 -> 72 [receive] via NET/AWS Libfabric/0 | |
| ip-172-31-38-82:24978:25047 [0] NCCL INFO Ring 00 : 40[0] -> 42[2] via P2P/IPC | |
| ip-172-31-44-142:24929:25002 [0] NCCL INFO Ring 00 : 16[0] -> 18[2] via P2P/IPC | |
| ip-172-31-39-141:20303:20372 [0] NCCL INFO Ring 00 : 0[0] -> 2[2] via P2P/IPC | |
| ip-172-31-37-141:25824:25897 [0] NCCL INFO Ring 00 : 32[0] -> 34[2] via P2P/IPC | |
| ip-172-31-44-4:22492:22563 [0] NCCL INFO Ring 00 : 120[0] -> 122[2] via P2P/IPC | |
| ip-172-31-47-240:24689:24766 [0] NCCL INFO Ring 00 : 56[0] -> 58[2] via P2P/IPC | |
| ip-172-31-45-171:25058:25128 [0] NCCL INFO Ring 00 : 24[0] -> 26[2] via P2P/IPC | |
| ip-172-31-46-50:22040:22119 [0] NCCL INFO Ring 00 : 64[0] -> 66[2] via P2P/IPC | |
| ip-172-31-44-230:21987:22059 [0] NCCL INFO Ring 00 : 96[0] -> 98[2] via P2P/IPC | |
| ip-172-31-32-86:24691:24765 [0] NCCL INFO Ring 00 : 48[0] -> 50[2] via P2P/IPC | |
| ip-172-31-47-188:22008:22081 [0] NCCL INFO Ring 00 : 72[0] -> 74[2] via P2P/IPC | |
| ip-172-31-38-14:25298:25374 [0] NCCL INFO Ring 00 : 8[0] -> 10[2] via P2P/IPC | |
| ip-172-31-47-156:22560:22634 [0] NCCL INFO Ring 00 : 80[0] -> 82[2] via P2P/IPC | |
| ip-172-31-36-188:22207:22281 [0] NCCL INFO Ring 00 : 88[0] -> 90[2] via P2P/IPC | |
| ip-172-31-33-235:21959:22032 [0] NCCL INFO Ring 00 : 104[0] -> 106[2] via P2P/IPC | |
| ip-172-31-36-128:22021:22098 [0] NCCL INFO Ring 00 : 112[0] -> 114[2] via P2P/IPC | |
| ip-172-31-38-82:24982:25054 [4] NCCL INFO Ring 00 : 44[4] -> 46[6] via P2P/IPC | |
| ip-172-31-38-82:24983:25050 [5] NCCL INFO Ring 00 : 45[5] -> 41[1] via P2P/IPC | |
| ip-172-31-38-82:24985:25053 [7] NCCL INFO Ring 00 : 47[7] -> 44[4] via P2P/IPC | |
| ip-172-31-38-82:24981:25048 [3] NCCL INFO Ring 00 : 43[3] -> 47[7] via P2P/IPC | |
| ip-172-31-38-82:24980:25051 [2] NCCL INFO Ring 00 : 42[2] -> 43[3] via P2P/IPC | |
| ip-172-31-39-141:20310:20383 [7] NCCL INFO Ring 00 : 7[7] -> 4[4] via P2P/IPC | |
| ip-172-31-38-82:24984:25049 [6] NCCL INFO Ring 00 : 46[6] -> 45[5] via P2P/IPC | |
| ip-172-31-39-141:20309:20382 [6] NCCL INFO Ring 00 : 6[6] -> 5[5] via P2P/IPC | |
| ip-172-31-39-141:20308:20378 [5] NCCL INFO Ring 00 : 5[5] -> 1[1] via P2P/IPC | |
| ip-172-31-39-141:20307:20377 [4] NCCL INFO Ring 00 : 4[4] -> 6[6] via P2P/IPC | |
| ip-172-31-39-141:20305:20380 [2] NCCL INFO Ring 00 : 2[2] -> 3[3] via P2P/IPC | |
| ip-172-31-38-82:24979:25052 [1] NCCL INFO NET/OFI [1] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-39-141:20306:20379 [3] NCCL INFO Ring 00 : 3[3] -> 7[7] via P2P/IPC | |
| ip-172-31-44-4:22499:22562 [7] NCCL INFO Ring 00 : 127[7] -> 124[4] via P2P/IPC | |
| ip-172-31-44-4:22498:22560 [6] NCCL INFO Ring 00 : 126[6] -> 125[5] via P2P/IPC | |
| ip-172-31-44-4:22497:22559 [5] NCCL INFO Ring 00 : 125[5] -> 121[1] via P2P/IPC | |
| ip-172-31-44-4:22494:22564 [2] NCCL INFO Ring 00 : 122[2] -> 123[3] via P2P/IPC | |
| ip-172-31-39-141:20304:20381 [1] NCCL INFO NET/OFI [1] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-44-142:24931:25004 [2] NCCL INFO Ring 00 : 18[2] -> 19[3] via P2P/IPC | |
| ip-172-31-44-142:24933:25001 [4] NCCL INFO Ring 00 : 20[4] -> 22[6] via P2P/IPC | |
| ip-172-31-47-156:22569:22639 [7] NCCL INFO Ring 00 : 87[7] -> 84[4] via P2P/IPC | |
| ip-172-31-44-4:22496:22561 [4] NCCL INFO Ring 00 : 124[4] -> 126[6] via P2P/IPC | |
| ip-172-31-44-142:24934:25005 [5] NCCL INFO Ring 00 : 21[5] -> 17[1] via P2P/IPC | |
| ip-172-31-45-171:25064:25129 [6] NCCL INFO Ring 00 : 30[6] -> 29[5] via P2P/IPC | |
| ip-172-31-45-171:25065:25133 [7] NCCL INFO Ring 00 : 31[7] -> 28[4] via P2P/IPC | |
| ip-172-31-32-86:24697:24768 [6] NCCL INFO Ring 00 : 54[6] -> 53[5] via P2P/IPC | |
| ip-172-31-46-50:22047:22114 [7] NCCL INFO Ring 00 : 71[7] -> 68[4] via P2P/IPC | |
| ip-172-31-45-171:25063:25130 [5] NCCL INFO Ring 00 : 29[5] -> 25[1] via P2P/IPC | |
| ip-172-31-47-156:22566:22635 [6] NCCL INFO Ring 00 : 86[6] -> 85[5] via P2P/IPC | |
| ip-172-31-37-141:25829:25898 [5] NCCL INFO Ring 00 : 37[5] -> 33[1] via P2P/IPC | |
| ip-172-31-32-86:24696:24767 [5] NCCL INFO Ring 00 : 53[5] -> 49[1] via P2P/IPC | |
| ip-172-31-44-142:24935:25000 [6] NCCL INFO Ring 00 : 22[6] -> 21[5] via P2P/IPC | |
| ip-172-31-46-50:22046:22116 [6] NCCL INFO Ring 00 : 70[6] -> 69[5] via P2P/IPC | |
| ip-172-31-47-156:22565:22637 [5] NCCL INFO Ring 00 : 85[5] -> 81[1] via P2P/IPC | |
| ip-172-31-44-4:22495:22558 [3] NCCL INFO Ring 00 : 123[3] -> 127[7] via P2P/IPC | |
| ip-172-31-47-240:24695:24762 [6] NCCL INFO Ring 00 : 62[6] -> 61[5] via P2P/IPC | |
| ip-172-31-45-171:25062:25131 [4] NCCL INFO Ring 00 : 28[4] -> 30[6] via P2P/IPC | |
| ip-172-31-47-156:22564:22632 [4] NCCL INFO Ring 00 : 84[4] -> 86[6] via P2P/IPC | |
| ip-172-31-37-141:25828:25896 [4] NCCL INFO Ring 00 : 36[4] -> 38[6] via P2P/IPC | |
| ip-172-31-46-50:22044:22113 [4] NCCL INFO Ring 00 : 68[4] -> 70[6] via P2P/IPC | |
| ip-172-31-46-50:22045:22115 [5] NCCL INFO Ring 00 : 69[5] -> 65[1] via P2P/IPC | |
| ip-172-31-45-171:25061:25134 [3] NCCL INFO Ring 00 : 27[3] -> 31[7] via P2P/IPC | |
| ip-172-31-37-141:25827:25893 [3] NCCL INFO Ring 00 : 35[3] -> 39[7] via P2P/IPC | |
| ip-172-31-32-86:24695:24771 [4] NCCL INFO Ring 00 : 52[4] -> 54[6] via P2P/IPC | |
| ip-172-31-44-142:24937:25007 [7] NCCL INFO Ring 00 : 23[7] -> 20[4] via P2P/IPC | |
| ip-172-31-47-156:22562:22633 [2] NCCL INFO Ring 00 : 82[2] -> 83[3] via P2P/IPC | |
| ip-172-31-37-141:25831:25892 [7] NCCL INFO Ring 00 : 39[7] -> 36[4] via P2P/IPC | |
| ip-172-31-46-50:22043:22118 [3] NCCL INFO Ring 00 : 67[3] -> 71[7] via P2P/IPC | |
| ip-172-31-32-86:24698:24764 [7] NCCL INFO Ring 00 : 55[7] -> 52[4] via P2P/IPC | |
| ip-172-31-44-230:21994:22063 [7] NCCL INFO Ring 00 : 103[7] -> 100[4] via P2P/IPC | |
| ip-172-31-47-240:24696:24763 [7] NCCL INFO Ring 00 : 63[7] -> 60[4] via P2P/IPC | |
| ip-172-31-47-156:22563:22638 [3] NCCL INFO Ring 00 : 83[3] -> 87[7] via P2P/IPC | |
| ip-172-31-32-86:24693:24766 [2] NCCL INFO Ring 00 : 50[2] -> 51[3] via P2P/IPC | |
| ip-172-31-44-142:24932:25006 [3] NCCL INFO Ring 00 : 19[3] -> 23[7] via P2P/IPC | |
| ip-172-31-47-240:24694:24764 [5] NCCL INFO Ring 00 : 61[5] -> 57[1] via P2P/IPC | |
| ip-172-31-46-50:22042:22112 [2] NCCL INFO Ring 00 : 66[2] -> 67[3] via P2P/IPC | |
| ip-172-31-44-230:21991:22062 [4] NCCL INFO Ring 00 : 100[4] -> 102[6] via P2P/IPC | |
| ip-172-31-45-171:25060:25132 [2] NCCL INFO Ring 00 : 26[2] -> 27[3] via P2P/IPC | |
| ip-172-31-36-128:22030:22096 [7] NCCL INFO Ring 00 : 119[7] -> 116[4] via P2P/IPC | |
| ip-172-31-37-141:25830:25895 [6] NCCL INFO Ring 00 : 38[6] -> 37[5] via P2P/IPC | |
| ip-172-31-36-128:22027:22092 [6] NCCL INFO Ring 00 : 118[6] -> 117[5] via P2P/IPC | |
| ip-172-31-47-240:24691:24761 [2] NCCL INFO Ring 00 : 58[2] -> 59[3] via P2P/IPC | |
| ip-172-31-32-86:24694:24770 [3] NCCL INFO Ring 00 : 51[3] -> 55[7] via P2P/IPC | |
| ip-172-31-44-230:21990:22065 [3] NCCL INFO Ring 00 : 99[3] -> 103[7] via P2P/IPC | |
| ip-172-31-36-128:22026:22094 [5] NCCL INFO Ring 00 : 117[5] -> 113[1] via P2P/IPC | |
| ip-172-31-37-141:25826:25891 [2] NCCL INFO Ring 00 : 34[2] -> 35[3] via P2P/IPC | |
| ip-172-31-44-230:21993:22066 [6] NCCL INFO Ring 00 : 102[6] -> 101[5] via P2P/IPC | |
| ip-172-31-47-240:24693:24767 [4] NCCL INFO Ring 00 : 60[4] -> 62[6] via P2P/IPC | |
| ip-172-31-36-128:22024:22091 [3] NCCL INFO Ring 00 : 115[3] -> 119[7] via P2P/IPC | |
| ip-172-31-44-142:24930:25003 [1] NCCL INFO NET/OFI [1] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-36-128:22023:22093 [2] NCCL INFO Ring 00 : 114[2] -> 115[3] via P2P/IPC | |
| ip-172-31-38-14:25300:25369 [2] NCCL INFO Ring 00 : 10[2] -> 11[3] via P2P/IPC | |
| ip-172-31-44-230:21992:22064 [5] NCCL INFO Ring 00 : 101[5] -> 97[1] via P2P/IPC | |
| ip-172-31-37-141:25825:25894 [1] NCCL INFO NET/OFI [1] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-44-230:21989:22061 [2] NCCL INFO Ring 00 : 98[2] -> 99[3] via P2P/IPC | |
| ip-172-31-36-188:22212:22276 [5] NCCL INFO Ring 00 : 93[5] -> 89[1] via P2P/IPC | |
| ip-172-31-45-171:25059:25135 [1] NCCL INFO NET/OFI [1] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-47-240:24692:24760 [3] NCCL INFO Ring 00 : 59[3] -> 63[7] via P2P/IPC | |
| ip-172-31-36-188:22214:22275 [7] NCCL INFO Ring 00 : 95[7] -> 92[4] via P2P/IPC | |
| ip-172-31-36-128:22025:22095 [4] NCCL INFO Ring 00 : 116[4] -> 118[6] via P2P/IPC | |
| ip-172-31-47-188:22015:22086 [7] NCCL INFO Ring 00 : 79[7] -> 76[4] via P2P/IPC | |
| ip-172-31-32-86:24692:24769 [1] NCCL INFO NET/OFI [1] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-44-4:22493:22565 [1] NCCL INFO NET/OFI [1] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-47-240:24690:24765 [1] NCCL INFO NET/OFI [1] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-47-188:22014:22087 [6] NCCL INFO Ring 00 : 78[6] -> 77[5] via P2P/IPC | |
| ip-172-31-47-156:22561:22636 [1] NCCL INFO NET/OFI [1] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-36-188:22213:22279 [6] NCCL INFO Ring 00 : 94[6] -> 93[5] via P2P/IPC | |
| ip-172-31-33-235:21966:22034 [7] NCCL INFO Ring 00 : 111[7] -> 108[4] via P2P/IPC | |
| ip-172-31-46-50:22041:22117 [1] NCCL INFO NET/OFI [1] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-36-188:22211:22280 [4] NCCL INFO Ring 00 : 92[4] -> 94[6] via P2P/IPC | |
| ip-172-31-33-235:21963:22035 [4] NCCL INFO Ring 00 : 108[4] -> 110[6] via P2P/IPC | |
| ip-172-31-36-188:22209:22274 [2] NCCL INFO Ring 00 : 90[2] -> 91[3] via P2P/IPC | |
| ip-172-31-38-14:25299:25372 [1] NCCL INFO NET/OFI [1] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-47-188:22013:22084 [5] NCCL INFO Ring 00 : 77[5] -> 73[1] via P2P/IPC | |
| ip-172-31-33-235:21962:22036 [3] NCCL INFO Ring 00 : 107[3] -> 111[7] via P2P/IPC | |
| ip-172-31-47-188:22012:22080 [4] NCCL INFO Ring 00 : 76[4] -> 78[6] via P2P/IPC | |
| ip-172-31-36-128:22022:22097 [1] NCCL INFO NET/OFI [1] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-44-230:21988:22060 [1] NCCL INFO NET/OFI [1] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-47-188:22010:22085 [2] NCCL INFO Ring 00 : 74[2] -> 75[3] via P2P/IPC | |
| ip-172-31-33-235:21965:22037 [6] NCCL INFO Ring 00 : 110[6] -> 109[5] via P2P/IPC | |
| ip-172-31-33-235:21964:22033 [5] NCCL INFO Ring 00 : 109[5] -> 105[1] via P2P/IPC | |
| ip-172-31-38-14:25306:25370 [7] NCCL INFO Ring 00 : 15[7] -> 12[4] via P2P/IPC | |
| ip-172-31-38-14:25304:25371 [6] NCCL INFO Ring 00 : 14[6] -> 13[5] via P2P/IPC | |
| ip-172-31-47-188:22011:22083 [3] NCCL INFO Ring 00 : 75[3] -> 79[7] via P2P/IPC | |
| ip-172-31-36-188:22210:22277 [3] NCCL INFO Ring 00 : 91[3] -> 95[7] via P2P/IPC | |
| ip-172-31-33-235:21961:22023 [2] NCCL INFO Ring 00 : 106[2] -> 107[3] via P2P/IPC | |
| ip-172-31-38-14:25301:25375 [3] NCCL INFO Ring 00 : 11[3] -> 15[7] via P2P/IPC | |
| ip-172-31-36-188:22208:22278 [1] NCCL INFO NET/OFI [1] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-33-235:21960:22031 [1] NCCL INFO NET/OFI [1] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-38-14:25303:25373 [5] NCCL INFO Ring 00 : 13[5] -> 9[1] via P2P/IPC | |
| ip-172-31-38-82:24979:25052 [1] NCCL INFO Ring 00 : 41 -> 48 [send] via NET/AWS Libfabric/0 | |
| ip-172-31-47-188:22009:22082 [1] NCCL INFO NET/OFI [1] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-38-14:25302:25368 [4] NCCL INFO Ring 00 : 12[4] -> 14[6] via P2P/IPC | |
| ip-172-31-39-141:20304:20381 [1] NCCL INFO Ring 00 : 1 -> 8 [send] via NET/AWS Libfabric/0 | |
| ip-172-31-44-142:24930:25003 [1] NCCL INFO Ring 00 : 17 -> 24 [send] via NET/AWS Libfabric/0 | |
| ip-172-31-37-141:25825:25894 [1] NCCL INFO Ring 00 : 33 -> 40 [send] via NET/AWS Libfabric/0 | |
| ip-172-31-45-171:25059:25135 [1] NCCL INFO Ring 00 : 25 -> 32 [send] via NET/AWS Libfabric/0 | |
| ip-172-31-32-86:24692:24769 [1] NCCL INFO Ring 00 : 49 -> 56 [send] via NET/AWS Libfabric/0 | |
| ip-172-31-47-240:24690:24765 [1] NCCL INFO Ring 00 : 57 -> 64 [send] via NET/AWS Libfabric/0 | |
| ip-172-31-46-50:22041:22117 [1] NCCL INFO Ring 00 : 65 -> 72 [send] via NET/AWS Libfabric/0 | |
| ip-172-31-47-156:22561:22636 [1] NCCL INFO Ring 00 : 81 -> 88 [send] via NET/AWS Libfabric/0 | |
| ip-172-31-44-4:22493:22565 [1] NCCL INFO Ring 00 : 121 -> 0 [send] via NET/AWS Libfabric/0 | |
| ip-172-31-38-82:24980:25051 [2] NCCL INFO NET/OFI [2] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-44-230:21988:22060 [1] NCCL INFO Ring 00 : 97 -> 104 [send] via NET/AWS Libfabric/0 | |
| ip-172-31-36-128:22022:22097 [1] NCCL INFO Ring 00 : 113 -> 120 [send] via NET/AWS Libfabric/0 | |
| ip-172-31-38-14:25299:25372 [1] NCCL INFO Ring 00 : 9 -> 16 [send] via NET/AWS Libfabric/0 | |
| ip-172-31-33-235:21960:22031 [1] NCCL INFO Ring 00 : 105 -> 112 [send] via NET/AWS Libfabric/0 | |
| ip-172-31-36-188:22208:22278 [1] NCCL INFO Ring 00 : 89 -> 96 [send] via NET/AWS Libfabric/0 | |
| ip-172-31-39-141:20305:20380 [2] NCCL INFO NET/OFI [2] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-47-188:22009:22082 [1] NCCL INFO Ring 00 : 73 -> 80 [send] via NET/AWS Libfabric/0 | |
| ip-172-31-44-4:22494:22564 [2] NCCL INFO NET/OFI [2] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-44-142:24931:25004 [2] NCCL INFO NET/OFI [2] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-37-141:25826:25891 [2] NCCL INFO NET/OFI [2] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-47-240:24691:24761 [2] NCCL INFO NET/OFI [2] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-32-86:24693:24766 [2] NCCL INFO NET/OFI [2] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-45-171:25060:25132 [2] NCCL INFO NET/OFI [2] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-46-50:22042:22112 [2] NCCL INFO NET/OFI [2] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-47-156:22562:22633 [2] NCCL INFO NET/OFI [2] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-44-230:21989:22061 [2] NCCL INFO NET/OFI [2] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-38-14:25300:25369 [2] NCCL INFO NET/OFI [2] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-36-128:22023:22093 [2] NCCL INFO NET/OFI [2] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-38-82:24980:25051 [2] NCCL INFO Ring 01 : 35 -> 42 [receive] via NET/AWS Libfabric/1 | |
| ip-172-31-36-188:22209:22274 [2] NCCL INFO NET/OFI [2] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-47-188:22010:22085 [2] NCCL INFO NET/OFI [2] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-33-235:21961:22023 [2] NCCL INFO NET/OFI [2] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-39-141:20305:20380 [2] NCCL INFO Ring 01 : 123 -> 2 [receive] via NET/AWS Libfabric/1 | |
| ip-172-31-44-4:22494:22564 [2] NCCL INFO Ring 01 : 115 -> 122 [receive] via NET/AWS Libfabric/1 | |
| ip-172-31-44-142:24931:25004 [2] NCCL INFO Ring 01 : 11 -> 18 [receive] via NET/AWS Libfabric/1 | |
| ip-172-31-47-240:24691:24761 [2] NCCL INFO Ring 01 : 51 -> 58 [receive] via NET/AWS Libfabric/1 | |
| ip-172-31-45-171:25060:25132 [2] NCCL INFO Ring 01 : 19 -> 26 [receive] via NET/AWS Libfabric/1 | |
| ip-172-31-32-86:24693:24766 [2] NCCL INFO Ring 01 : 43 -> 50 [receive] via NET/AWS Libfabric/1 | |
| ip-172-31-37-141:25826:25891 [2] NCCL INFO Ring 01 : 27 -> 34 [receive] via NET/AWS Libfabric/1 | |
| ip-172-31-44-230:21989:22061 [2] NCCL INFO Ring 01 : 91 -> 98 [receive] via NET/AWS Libfabric/1 | |
| ip-172-31-47-156:22562:22633 [2] NCCL INFO Ring 01 : 75 -> 82 [receive] via NET/AWS Libfabric/1 | |
| ip-172-31-38-14:25300:25369 [2] NCCL INFO Ring 01 : 3 -> 10 [receive] via NET/AWS Libfabric/1 | |
| ip-172-31-36-128:22023:22093 [2] NCCL INFO Ring 01 : 107 -> 114 [receive] via NET/AWS Libfabric/1 | |
| ip-172-31-36-188:22209:22274 [2] NCCL INFO Ring 01 : 83 -> 90 [receive] via NET/AWS Libfabric/1 | |
| ip-172-31-47-188:22010:22085 [2] NCCL INFO Ring 01 : 67 -> 74 [receive] via NET/AWS Libfabric/1 | |
| ip-172-31-46-50:22042:22112 [2] NCCL INFO Ring 01 : 59 -> 66 [receive] via NET/AWS Libfabric/1 | |
| ip-172-31-33-235:21961:22023 [2] NCCL INFO Ring 01 : 99 -> 106 [receive] via NET/AWS Libfabric/1 | |
| ip-172-31-38-82:24982:25054 [4] NCCL INFO Ring 01 : 44[4] -> 47[7] via P2P/IPC | |
| ip-172-31-39-141:20310:20383 [7] NCCL INFO Ring 01 : 7[7] -> 3[3] via P2P/IPC | |
| ip-172-31-38-82:24983:25050 [5] NCCL INFO Ring 01 : 45[5] -> 46[6] via P2P/IPC | |
| ip-172-31-38-82:24985:25053 [7] NCCL INFO Ring 01 : 47[7] -> 43[3] via P2P/IPC | |
| ip-172-31-39-141:20309:20382 [6] NCCL INFO Ring 01 : 6[6] -> 4[4] via P2P/IPC | |
| ip-172-31-39-141:20308:20378 [5] NCCL INFO Ring 01 : 5[5] -> 6[6] via P2P/IPC | |
| ip-172-31-38-82:24984:25049 [6] NCCL INFO Ring 01 : 46[6] -> 44[4] via P2P/IPC | |
| ip-172-31-38-82:24981:25048 [3] NCCL INFO NET/OFI [3] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-46-50:22047:22114 [7] NCCL INFO Ring 01 : 71[7] -> 67[3] via P2P/IPC | |
| ip-172-31-46-50:22046:22116 [6] NCCL INFO Ring 01 : 70[6] -> 68[4] via P2P/IPC | |
| ip-172-31-46-50:22044:22113 [4] NCCL INFO Ring 01 : 68[4] -> 71[7] via P2P/IPC | |
| ip-172-31-39-141:20307:20377 [4] NCCL INFO Ring 01 : 4[4] -> 7[7] via P2P/IPC | |
| ip-172-31-46-50:22045:22115 [5] NCCL INFO Ring 01 : 69[5] -> 70[6] via P2P/IPC | |
| ip-172-31-37-141:25829:25898 [5] NCCL INFO Ring 01 : 37[5] -> 38[6] via P2P/IPC | |
| ip-172-31-37-141:25828:25896 [4] NCCL INFO Ring 01 : 36[4] -> 39[7] via P2P/IPC | |
| ip-172-31-44-4:22499:22562 [7] NCCL INFO Ring 01 : 127[7] -> 123[3] via P2P/IPC | |
| ip-172-31-46-50:22043:22118 [3] NCCL INFO NET/OFI [3] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-39-141:20306:20379 [3] NCCL INFO NET/OFI [3] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-44-4:22498:22560 [6] NCCL INFO Ring 01 : 126[6] -> 124[4] via P2P/IPC | |
| ip-172-31-37-141:25831:25892 [7] NCCL INFO Ring 01 : 39[7] -> 35[3] via P2P/IPC | |
| ip-172-31-44-4:22497:22559 [5] NCCL INFO Ring 01 : 125[5] -> 126[6] via P2P/IPC | |
| ip-172-31-38-82:24980:25051 [2] NCCL INFO Ring 01 : 42[2] -> 40[0] via P2P/IPC | |
| ip-172-31-37-141:25830:25895 [6] NCCL INFO Ring 01 : 38[6] -> 36[4] via P2P/IPC | |
| ip-172-31-44-4:22496:22561 [4] NCCL INFO Ring 01 : 124[4] -> 127[7] via P2P/IPC | |
| ip-172-31-37-141:25827:25893 [3] NCCL INFO NET/OFI [3] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-44-4:22495:22558 [3] NCCL INFO NET/OFI [3] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-39-141:20305:20380 [2] NCCL INFO Ring 01 : 2[2] -> 0[0] via P2P/IPC | |
| ip-172-31-45-171:25064:25129 [6] NCCL INFO Ring 01 : 30[6] -> 28[4] via P2P/IPC | |
| ip-172-31-38-82:24981:25048 [3] NCCL INFO Ring 01 : 43 -> 50 [send] via NET/AWS Libfabric/1 | |
| ip-172-31-32-86:24697:24768 [6] NCCL INFO Ring 01 : 54[6] -> 52[4] via P2P/IPC | |
| ip-172-31-47-240:24695:24762 [6] NCCL INFO Ring 01 : 62[6] -> 60[4] via P2P/IPC | |
| ip-172-31-45-171:25063:25130 [5] NCCL INFO Ring 01 : 29[5] -> 30[6] via P2P/IPC | |
| ip-172-31-44-230:21994:22063 [7] NCCL INFO Ring 01 : 103[7] -> 99[3] via P2P/IPC | |
| ip-172-31-44-230:21991:22062 [4] NCCL INFO Ring 01 : 100[4] -> 103[7] via P2P/IPC | |
| ip-172-31-47-240:24696:24763 [7] NCCL INFO Ring 01 : 63[7] -> 59[3] via P2P/IPC | |
| ip-172-31-45-171:25065:25133 [7] NCCL INFO Ring 01 : 31[7] -> 27[3] via P2P/IPC | |
| ip-172-31-45-171:25062:25131 [4] NCCL INFO Ring 01 : 28[4] -> 31[7] via P2P/IPC | |
| ip-172-31-47-156:22569:22639 [7] NCCL INFO Ring 01 : 87[7] -> 83[3] via P2P/IPC | |
| ip-172-31-44-230:21993:22066 [6] NCCL INFO Ring 01 : 102[6] -> 100[4] via P2P/IPC | |
| ip-172-31-47-240:24694:24764 [5] NCCL INFO Ring 01 : 61[5] -> 62[6] via P2P/IPC | |
| ip-172-31-47-156:22566:22635 [6] NCCL INFO Ring 01 : 86[6] -> 84[4] via P2P/IPC | |
| ip-172-31-32-86:24696:24767 [5] NCCL INFO Ring 01 : 53[5] -> 54[6] via P2P/IPC | |
| ip-172-31-32-86:24695:24771 [4] NCCL INFO Ring 01 : 52[4] -> 55[7] via P2P/IPC | |
| ip-172-31-47-240:24693:24767 [4] NCCL INFO Ring 01 : 60[4] -> 63[7] via P2P/IPC | |
| ip-172-31-47-188:22015:22086 [7] NCCL INFO Ring 01 : 79[7] -> 75[3] via P2P/IPC | |
| ip-172-31-47-156:22565:22637 [5] NCCL INFO Ring 01 : 85[5] -> 86[6] via P2P/IPC | |
| ip-172-31-47-188:22014:22087 [6] NCCL INFO Ring 01 : 78[6] -> 76[4] via P2P/IPC | |
| ip-172-31-44-4:22494:22564 [2] NCCL INFO Ring 01 : 122[2] -> 120[0] via P2P/IPC | |
| ip-172-31-44-230:21992:22064 [5] NCCL INFO Ring 01 : 101[5] -> 102[6] via P2P/IPC | |
| ip-172-31-44-230:21990:22065 [3] NCCL INFO NET/OFI [3] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-47-156:22564:22632 [4] NCCL INFO Ring 01 : 84[4] -> 87[7] via P2P/IPC | |
| ip-172-31-47-188:22013:22084 [5] NCCL INFO Ring 01 : 77[5] -> 78[6] via P2P/IPC | |
| ip-172-31-47-188:22012:22080 [4] NCCL INFO Ring 01 : 76[4] -> 79[7] via P2P/IPC | |
| ip-172-31-32-86:24698:24764 [7] NCCL INFO Ring 01 : 55[7] -> 51[3] via P2P/IPC | |
| ip-172-31-44-142:24933:25001 [4] NCCL INFO Ring 01 : 20[4] -> 23[7] via P2P/IPC | |
| ip-172-31-44-142:24934:25005 [5] NCCL INFO Ring 01 : 21[5] -> 22[6] via P2P/IPC | |
| ip-172-31-32-86:24694:24770 [3] NCCL INFO NET/OFI [3] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-44-142:24935:25000 [6] NCCL INFO Ring 01 : 22[6] -> 20[4] via P2P/IPC | |
| ip-172-31-45-171:25061:25134 [3] NCCL INFO NET/OFI [3] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-46-50:22043:22118 [3] NCCL INFO Ring 01 : 67 -> 74 [send] via NET/AWS Libfabric/1 | |
| ip-172-31-44-142:24931:25004 [2] NCCL INFO Ring 01 : 18[2] -> 16[0] via P2P/IPC | |
| ip-172-31-39-141:20306:20379 [3] NCCL INFO Ring 01 : 3 -> 10 [send] via NET/AWS Libfabric/1 | |
| ip-172-31-47-240:24692:24760 [3] NCCL INFO NET/OFI [3] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-38-82:24982:25054 [4] NCCL INFO NET/OFI [4] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-44-142:24937:25007 [7] NCCL INFO Ring 01 : 23[7] -> 19[3] via P2P/IPC | |
| ip-172-31-47-240:24691:24761 [2] NCCL INFO Ring 01 : 58[2] -> 56[0] via P2P/IPC | |
| ip-172-31-36-188:22212:22276 [5] NCCL INFO Ring 01 : 93[5] -> 94[6] via P2P/IPC | |
| ip-172-31-47-156:22563:22638 [3] NCCL INFO NET/OFI [3] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-36-128:22030:22096 [7] NCCL INFO Ring 01 : 119[7] -> 115[3] via P2P/IPC | |
| ip-172-31-33-235:21966:22034 [7] NCCL INFO Ring 01 : 111[7] -> 107[3] via P2P/IPC | |
| ip-172-31-36-188:22214:22275 [7] NCCL INFO Ring 01 : 95[7] -> 91[3] via P2P/IPC | |
| ip-172-31-47-188:22011:22083 [3] NCCL INFO NET/OFI [3] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-33-235:21963:22035 [4] NCCL INFO Ring 01 : 108[4] -> 111[7] via P2P/IPC | |
| ip-172-31-37-141:25827:25893 [3] NCCL INFO Ring 01 : 35 -> 42 [send] via NET/AWS Libfabric/1 | |
| ip-172-31-44-142:24932:25006 [3] NCCL INFO NET/OFI [3] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-36-128:22027:22092 [6] NCCL INFO Ring 01 : 118[6] -> 116[4] via P2P/IPC | |
| ip-172-31-47-188:22010:22085 [2] NCCL INFO Ring 01 : 74[2] -> 72[0] via P2P/IPC | |
| ip-172-31-45-171:25060:25132 [2] NCCL INFO Ring 01 : 26[2] -> 24[0] via P2P/IPC | |
| ip-172-31-36-188:22213:22279 [6] NCCL INFO Ring 01 : 94[6] -> 92[4] via P2P/IPC | |
| ip-172-31-36-128:22026:22094 [5] NCCL INFO Ring 01 : 117[5] -> 118[6] via P2P/IPC | |
| ip-172-31-32-86:24693:24766 [2] NCCL INFO Ring 01 : 50[2] -> 48[0] via P2P/IPC | |
| ip-172-31-33-235:21965:22037 [6] NCCL INFO Ring 01 : 110[6] -> 108[4] via P2P/IPC | |
| ip-172-31-44-4:22495:22558 [3] NCCL INFO Ring 01 : 123 -> 2 [send] via NET/AWS Libfabric/1 | |
| ip-172-31-44-230:21989:22061 [2] NCCL INFO Ring 01 : 98[2] -> 96[0] via P2P/IPC | |
| ip-172-31-36-188:22211:22280 [4] NCCL INFO Ring 01 : 92[4] -> 95[7] via P2P/IPC | |
| ip-172-31-33-235:21964:22033 [5] NCCL INFO Ring 01 : 109[5] -> 110[6] via P2P/IPC | |
| ip-172-31-33-235:21962:22036 [3] NCCL INFO NET/OFI [3] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-47-156:22562:22633 [2] NCCL INFO Ring 01 : 82[2] -> 80[0] via P2P/IPC | |
| ip-172-31-39-141:20307:20377 [4] NCCL INFO NET/OFI [4] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-38-14:25300:25369 [2] NCCL INFO Ring 01 : 10[2] -> 8[0] via P2P/IPC | |
| ip-172-31-36-188:22209:22274 [2] NCCL INFO Ring 01 : 90[2] -> 88[0] via P2P/IPC | |
| ip-172-31-46-50:22044:22113 [4] NCCL INFO NET/OFI [4] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-36-188:22210:22277 [3] NCCL INFO NET/OFI [3] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-36-128:22025:22095 [4] NCCL INFO Ring 01 : 116[4] -> 119[7] via P2P/IPC | |
| ip-172-31-36-128:22024:22091 [3] NCCL INFO NET/OFI [3] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-37-141:25826:25891 [2] NCCL INFO Ring 01 : 34[2] -> 32[0] via P2P/IPC | |
| ip-172-31-36-128:22023:22093 [2] NCCL INFO Ring 01 : 114[2] -> 112[0] via P2P/IPC | |
| ip-172-31-38-14:25306:25370 [7] NCCL INFO Ring 01 : 15[7] -> 11[3] via P2P/IPC | |
| ip-172-31-38-14:25304:25371 [6] NCCL INFO Ring 01 : 14[6] -> 12[4] via P2P/IPC | |
| ip-172-31-44-230:21990:22065 [3] NCCL INFO Ring 01 : 99 -> 106 [send] via NET/AWS Libfabric/1 | |
| ip-172-31-37-141:25828:25896 [4] NCCL INFO NET/OFI [4] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-44-4:22496:22561 [4] NCCL INFO NET/OFI [4] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-38-14:25303:25373 [5] NCCL INFO Ring 01 : 13[5] -> 14[6] via P2P/IPC | |
| ip-172-31-45-171:25061:25134 [3] NCCL INFO Ring 01 : 27 -> 34 [send] via NET/AWS Libfabric/1 | |
| ip-172-31-32-86:24694:24770 [3] NCCL INFO Ring 01 : 51 -> 58 [send] via NET/AWS Libfabric/1 | |
| ip-172-31-38-14:25302:25368 [4] NCCL INFO Ring 01 : 12[4] -> 15[7] via P2P/IPC | |
| ip-172-31-38-82:24982:25054 [4] NCCL INFO Ring 02 : 37 -> 44 [receive] via NET/AWS Libfabric/2 | |
| ip-172-31-38-14:25301:25375 [3] NCCL INFO NET/OFI [3] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-47-240:24692:24760 [3] NCCL INFO Ring 01 : 59 -> 66 [send] via NET/AWS Libfabric/1 | |
| ip-172-31-47-156:22563:22638 [3] NCCL INFO Ring 01 : 83 -> 90 [send] via NET/AWS Libfabric/1 | |
| ip-172-31-47-188:22011:22083 [3] NCCL INFO Ring 01 : 75 -> 82 [send] via NET/AWS Libfabric/1 | |
| ip-172-31-44-142:24932:25006 [3] NCCL INFO Ring 01 : 19 -> 26 [send] via NET/AWS Libfabric/1 | |
| ip-172-31-44-230:21991:22062 [4] NCCL INFO NET/OFI [4] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-39-141:20307:20377 [4] NCCL INFO Ring 02 : 125 -> 4 [receive] via NET/AWS Libfabric/2 | |
| ip-172-31-45-171:25062:25131 [4] NCCL INFO NET/OFI [4] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-32-86:24695:24771 [4] NCCL INFO NET/OFI [4] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-47-240:24693:24767 [4] NCCL INFO NET/OFI [4] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-33-235:21962:22036 [3] NCCL INFO Ring 01 : 107 -> 114 [send] via NET/AWS Libfabric/1 | |
| ip-172-31-33-235:21961:22023 [2] NCCL INFO Ring 01 : 106[2] -> 104[0] via P2P/IPC | |
| ip-172-31-47-188:22012:22080 [4] NCCL INFO NET/OFI [4] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-46-50:22044:22113 [4] NCCL INFO Ring 02 : 61 -> 68 [receive] via NET/AWS Libfabric/2 | |
| ip-172-31-47-156:22564:22632 [4] NCCL INFO NET/OFI [4] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-44-142:24933:25001 [4] NCCL INFO NET/OFI [4] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-36-128:22024:22091 [3] NCCL INFO Ring 01 : 115 -> 122 [send] via NET/AWS Libfabric/1 | |
| ip-172-31-44-4:22496:22561 [4] NCCL INFO Ring 02 : 117 -> 124 [receive] via NET/AWS Libfabric/2 | |
| ip-172-31-36-188:22210:22277 [3] NCCL INFO Ring 01 : 91 -> 98 [send] via NET/AWS Libfabric/1 | |
| ip-172-31-37-141:25828:25896 [4] NCCL INFO Ring 02 : 29 -> 36 [receive] via NET/AWS Libfabric/2 | |
| ip-172-31-38-14:25301:25375 [3] NCCL INFO Ring 01 : 11 -> 18 [send] via NET/AWS Libfabric/1 | |
| ip-172-31-36-188:22211:22280 [4] NCCL INFO NET/OFI [4] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-33-235:21963:22035 [4] NCCL INFO NET/OFI [4] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-46-50:22042:22112 [2] NCCL INFO Ring 01 : 66[2] -> 64[0] via P2P/IPC | |
| ip-172-31-44-230:21991:22062 [4] NCCL INFO Ring 02 : 93 -> 100 [receive] via NET/AWS Libfabric/2 | |
| ip-172-31-36-128:22025:22095 [4] NCCL INFO NET/OFI [4] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-45-171:25062:25131 [4] NCCL INFO Ring 02 : 21 -> 28 [receive] via NET/AWS Libfabric/2 | |
| ip-172-31-47-240:24693:24767 [4] NCCL INFO Ring 02 : 53 -> 60 [receive] via NET/AWS Libfabric/2 | |
| ip-172-31-47-156:22564:22632 [4] NCCL INFO Ring 02 : 77 -> 84 [receive] via NET/AWS Libfabric/2 | |
| ip-172-31-32-86:24695:24771 [4] NCCL INFO Ring 02 : 45 -> 52 [receive] via NET/AWS Libfabric/2 | |
| ip-172-31-47-188:22012:22080 [4] NCCL INFO Ring 02 : 69 -> 76 [receive] via NET/AWS Libfabric/2 | |
| ip-172-31-44-142:24933:25001 [4] NCCL INFO Ring 02 : 13 -> 20 [receive] via NET/AWS Libfabric/2 | |
| ip-172-31-38-14:25302:25368 [4] NCCL INFO NET/OFI [4] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-36-188:22211:22280 [4] NCCL INFO Ring 02 : 85 -> 92 [receive] via NET/AWS Libfabric/2 | |
| ip-172-31-33-235:21963:22035 [4] NCCL INFO Ring 02 : 101 -> 108 [receive] via NET/AWS Libfabric/2 | |
| ip-172-31-36-128:22025:22095 [4] NCCL INFO Ring 02 : 109 -> 116 [receive] via NET/AWS Libfabric/2 | |
| ip-172-31-38-14:25302:25368 [4] NCCL INFO Ring 02 : 5 -> 12 [receive] via NET/AWS Libfabric/2 | |
| ip-172-31-38-82:24979:25052 [1] NCCL INFO Ring 01 : 41[1] -> 45[5] via P2P/IPC | |
| ip-172-31-39-141:20304:20381 [1] NCCL INFO Ring 01 : 1[1] -> 5[5] via P2P/IPC | |
| ip-172-31-37-141:25825:25894 [1] NCCL INFO Ring 01 : 33[1] -> 37[5] via P2P/IPC | |
| ip-172-31-47-156:22561:22636 [1] NCCL INFO Ring 01 : 81[1] -> 85[5] via P2P/IPC | |
| ip-172-31-47-240:24690:24765 [1] NCCL INFO Ring 01 : 57[1] -> 61[5] via P2P/IPC | |
| ip-172-31-44-230:21988:22060 [1] NCCL INFO Ring 01 : 97[1] -> 101[5] via P2P/IPC | |
| ip-172-31-38-82:24978:25047 [0] NCCL INFO Ring 01 : 40[0] -> 41[1] via P2P/IPC | |
| ip-172-31-46-50:22040:22119 [0] NCCL INFO Ring 01 : 64[0] -> 65[1] via P2P/IPC | |
| ip-172-31-45-171:25059:25135 [1] NCCL INFO Ring 01 : 25[1] -> 29[5] via P2P/IPC | |
| ip-172-31-47-188:22009:22082 [1] NCCL INFO Ring 01 : 73[1] -> 77[5] via P2P/IPC | |
| ip-172-31-32-86:24692:24769 [1] NCCL INFO Ring 01 : 49[1] -> 53[5] via P2P/IPC | |
| ip-172-31-44-142:24930:25003 [1] NCCL INFO Ring 01 : 17[1] -> 21[5] via P2P/IPC | |
| ip-172-31-44-4:22493:22565 [1] NCCL INFO Ring 01 : 121[1] -> 125[5] via P2P/IPC | |
| ip-172-31-36-188:22208:22278 [1] NCCL INFO Ring 01 : 89[1] -> 93[5] via P2P/IPC | |
| ip-172-31-44-4:22492:22563 [0] NCCL INFO Ring 01 : 120[0] -> 121[1] via P2P/IPC | |
| ip-172-31-33-235:21960:22031 [1] NCCL INFO Ring 01 : 105[1] -> 109[5] via P2P/IPC | |
| ip-172-31-37-141:25824:25897 [0] NCCL INFO Ring 01 : 32[0] -> 33[1] via P2P/IPC | |
| ip-172-31-32-86:24691:24765 [0] NCCL INFO Ring 01 : 48[0] -> 49[1] via P2P/IPC | |
| ip-172-31-36-128:22022:22097 [1] NCCL INFO Ring 01 : 113[1] -> 117[5] via P2P/IPC | |
| ip-172-31-38-14:25299:25372 [1] NCCL INFO Ring 01 : 9[1] -> 13[5] via P2P/IPC | |
| ip-172-31-45-171:25058:25128 [0] NCCL INFO Ring 01 : 24[0] -> 25[1] via P2P/IPC | |
| ip-172-31-47-240:24689:24766 [0] NCCL INFO Ring 01 : 56[0] -> 57[1] via P2P/IPC | |
| ip-172-31-44-142:24929:25002 [0] NCCL INFO Ring 01 : 16[0] -> 17[1] via P2P/IPC | |
| ip-172-31-36-188:22207:22281 [0] NCCL INFO Ring 01 : 88[0] -> 89[1] via P2P/IPC | |
| ip-172-31-47-156:22560:22634 [0] NCCL INFO Ring 01 : 80[0] -> 81[1] via P2P/IPC | |
| ip-172-31-38-14:25298:25374 [0] NCCL INFO Ring 01 : 8[0] -> 9[1] via P2P/IPC | |
| ip-172-31-39-141:20303:20372 [0] NCCL INFO Ring 01 : 0[0] -> 1[1] via P2P/IPC | |
| ip-172-31-33-235:21959:22032 [0] NCCL INFO Ring 01 : 104[0] -> 105[1] via P2P/IPC | |
| ip-172-31-44-230:21987:22059 [0] NCCL INFO Ring 01 : 96[0] -> 97[1] via P2P/IPC | |
| ip-172-31-36-128:22021:22098 [0] NCCL INFO Ring 01 : 112[0] -> 113[1] via P2P/IPC | |
| ip-172-31-46-50:22041:22117 [1] NCCL INFO Ring 01 : 65[1] -> 69[5] via P2P/IPC | |
| ip-172-31-47-188:22008:22081 [0] NCCL INFO Ring 01 : 72[0] -> 73[1] via P2P/IPC | |
| ip-172-31-38-82:24985:25053 [7] NCCL INFO Ring 02 : 47[7] -> 45[5] via P2P/IPC | |
| ip-172-31-46-50:22047:22114 [7] NCCL INFO Ring 02 : 71[7] -> 69[5] via P2P/IPC | |
| ip-172-31-46-50:22046:22116 [6] NCCL INFO Ring 02 : 70[6] -> 71[7] via P2P/IPC | |
| ip-172-31-38-82:24984:25049 [6] NCCL INFO Ring 02 : 46[6] -> 47[7] via P2P/IPC | |
| ip-172-31-39-141:20310:20383 [7] NCCL INFO Ring 02 : 7[7] -> 5[5] via P2P/IPC | |
| ip-172-31-39-141:20309:20382 [6] NCCL INFO Ring 02 : 6[6] -> 7[7] via P2P/IPC | |
| ip-172-31-44-4:22499:22562 [7] NCCL INFO Ring 02 : 127[7] -> 125[5] via P2P/IPC | |
| ip-172-31-44-4:22498:22560 [6] NCCL INFO Ring 02 : 126[6] -> 127[7] via P2P/IPC | |
| ip-172-31-37-141:25831:25892 [7] NCCL INFO Ring 02 : 39[7] -> 37[5] via P2P/IPC | |
| ip-172-31-37-141:25830:25895 [6] NCCL INFO Ring 02 : 38[6] -> 39[7] via P2P/IPC | |
| ip-172-31-38-82:24982:25054 [4] NCCL INFO Ring 02 : 44[4] -> 40[0] via P2P/IPC | |
| ip-172-31-47-188:22015:22086 [7] NCCL INFO Ring 02 : 79[7] -> 77[5] via P2P/IPC | |
| ip-172-31-47-188:22014:22087 [6] NCCL INFO Ring 02 : 78[6] -> 79[7] via P2P/IPC | |
| ip-172-31-44-230:21994:22063 [7] NCCL INFO Ring 02 : 103[7] -> 101[5] via P2P/IPC | |
| ip-172-31-45-171:25064:25129 [6] NCCL INFO Ring 02 : 30[6] -> 31[7] via P2P/IPC | |
| ip-172-31-45-171:25065:25133 [7] NCCL INFO Ring 02 : 31[7] -> 29[5] via P2P/IPC | |
| ip-172-31-46-50:22044:22113 [4] NCCL INFO Ring 02 : 68[4] -> 64[0] via P2P/IPC | |
| ip-172-31-39-141:20307:20377 [4] NCCL INFO Ring 02 : 4[4] -> 0[0] via P2P/IPC | |
| ip-172-31-47-156:22569:22639 [7] NCCL INFO Ring 02 : 87[7] -> 85[5] via P2P/IPC | |
| ip-172-31-32-86:24697:24768 [6] NCCL INFO Ring 02 : 54[6] -> 55[7] via P2P/IPC | |
| ip-172-31-47-156:22566:22635 [6] NCCL INFO Ring 02 : 86[6] -> 87[7] via P2P/IPC | |
| ip-172-31-47-240:24695:24762 [6] NCCL INFO Ring 02 : 62[6] -> 63[7] via P2P/IPC | |
| ip-172-31-44-230:21993:22066 [6] NCCL INFO Ring 02 : 102[6] -> 103[7] via P2P/IPC | |
| ip-172-31-47-240:24696:24763 [7] NCCL INFO Ring 02 : 63[7] -> 61[5] via P2P/IPC | |
| ip-172-31-44-142:24935:25000 [6] NCCL INFO Ring 02 : 22[6] -> 23[7] via P2P/IPC | |
| ip-172-31-32-86:24698:24764 [7] NCCL INFO Ring 02 : 55[7] -> 53[5] via P2P/IPC | |
| ip-172-31-44-4:22496:22561 [4] NCCL INFO Ring 02 : 124[4] -> 120[0] via P2P/IPC | |
| ip-172-31-36-188:22214:22275 [7] NCCL INFO Ring 02 : 95[7] -> 93[5] via P2P/IPC | |
| ip-172-31-36-188:22213:22279 [6] NCCL INFO Ring 02 : 94[6] -> 95[7] via P2P/IPC | |
| ip-172-31-36-128:22030:22096 [7] NCCL INFO Ring 02 : 119[7] -> 117[5] via P2P/IPC | |
| ip-172-31-36-128:22027:22092 [6] NCCL INFO Ring 02 : 118[6] -> 119[7] via P2P/IPC | |
| ip-172-31-44-142:24937:25007 [7] NCCL INFO Ring 02 : 23[7] -> 21[5] via P2P/IPC | |
| ip-172-31-37-141:25828:25896 [4] NCCL INFO Ring 02 : 36[4] -> 32[0] via P2P/IPC | |
| ip-172-31-33-235:21966:22034 [7] NCCL INFO Ring 02 : 111[7] -> 109[5] via P2P/IPC | |
| ip-172-31-33-235:21965:22037 [6] NCCL INFO Ring 02 : 110[6] -> 111[7] via P2P/IPC | |
| ip-172-31-45-171:25062:25131 [4] NCCL INFO Ring 02 : 28[4] -> 24[0] via P2P/IPC | |
| ip-172-31-38-14:25306:25370 [7] NCCL INFO Ring 02 : 15[7] -> 13[5] via P2P/IPC | |
| ip-172-31-38-14:25304:25371 [6] NCCL INFO Ring 02 : 14[6] -> 15[7] via P2P/IPC | |
| ip-172-31-44-142:24933:25001 [4] NCCL INFO Ring 02 : 20[4] -> 16[0] via P2P/IPC | |
| ip-172-31-44-230:21991:22062 [4] NCCL INFO Ring 02 : 100[4] -> 96[0] via P2P/IPC | |
| ip-172-31-47-240:24693:24767 [4] NCCL INFO Ring 02 : 60[4] -> 56[0] via P2P/IPC | |
| ip-172-31-47-188:22012:22080 [4] NCCL INFO Ring 02 : 76[4] -> 72[0] via P2P/IPC | |
| ip-172-31-47-156:22564:22632 [4] NCCL INFO Ring 02 : 84[4] -> 80[0] via P2P/IPC | |
| ip-172-31-32-86:24695:24771 [4] NCCL INFO Ring 02 : 52[4] -> 48[0] via P2P/IPC | |
| ip-172-31-36-188:22211:22280 [4] NCCL INFO Ring 02 : 92[4] -> 88[0] via P2P/IPC | |
| ip-172-31-33-235:21963:22035 [4] NCCL INFO Ring 02 : 108[4] -> 104[0] via P2P/IPC | |
| ip-172-31-36-128:22025:22095 [4] NCCL INFO Ring 02 : 116[4] -> 112[0] via P2P/IPC | |
| ip-172-31-38-14:25302:25368 [4] NCCL INFO Ring 02 : 12[4] -> 8[0] via P2P/IPC | |
| ip-172-31-38-82:24983:25050 [5] NCCL INFO NET/OFI [5] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-39-141:20308:20378 [5] NCCL INFO NET/OFI [5] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-38-82:24983:25050 [5] NCCL INFO Ring 02 : 45 -> 52 [send] via NET/AWS Libfabric/2 | |
| ip-172-31-39-141:20308:20378 [5] NCCL INFO Ring 02 : 5 -> 12 [send] via NET/AWS Libfabric/2 | |
| ip-172-31-47-156:22565:22637 [5] NCCL INFO NET/OFI [5] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-37-141:25829:25898 [5] NCCL INFO NET/OFI [5] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-47-188:22013:22084 [5] NCCL INFO NET/OFI [5] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-46-50:22043:22118 [3] NCCL INFO Ring 02 : 67[3] -> 65[1] via P2P/IPC | |
| ip-172-31-46-50:22040:22119 [0] NCCL INFO Ring 02 : 64[0] -> 67[3] via P2P/IPC | |
| ip-172-31-38-82:24979:25052 [1] NCCL INFO Ring 02 : 41[1] -> 42[2] via P2P/IPC | |
| ip-172-31-44-142:24934:25005 [5] NCCL INFO NET/OFI [5] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-44-4:22497:22559 [5] NCCL INFO NET/OFI [5] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-44-230:21992:22064 [5] NCCL INFO NET/OFI [5] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-45-171:25063:25130 [5] NCCL INFO NET/OFI [5] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-38-82:24981:25048 [3] NCCL INFO Ring 02 : 43[3] -> 41[1] via P2P/IPC | |
| ip-172-31-47-240:24694:24764 [5] NCCL INFO NET/OFI [5] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-36-188:22212:22276 [5] NCCL INFO NET/OFI [5] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-44-4:22495:22558 [3] NCCL INFO Ring 02 : 123[3] -> 121[1] via P2P/IPC | |
| ip-172-31-32-86:24696:24767 [5] NCCL INFO NET/OFI [5] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-33-235:21964:22033 [5] NCCL INFO NET/OFI [5] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-39-141:20306:20379 [3] NCCL INFO Ring 02 : 3[3] -> 1[1] via P2P/IPC | |
| ip-172-31-47-156:22565:22637 [5] NCCL INFO Ring 02 : 85 -> 92 [send] via NET/AWS Libfabric/2 | |
| ip-172-31-36-128:22026:22094 [5] NCCL INFO NET/OFI [5] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-37-141:25829:25898 [5] NCCL INFO Ring 02 : 37 -> 44 [send] via NET/AWS Libfabric/2 | |
| ip-172-31-46-50:22045:22115 [5] NCCL INFO NET/OFI [5] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-37-141:25825:25894 [1] NCCL INFO Ring 02 : 33[1] -> 34[2] via P2P/IPC | |
| ip-172-31-47-188:22013:22084 [5] NCCL INFO Ring 02 : 77 -> 84 [send] via NET/AWS Libfabric/2 | |
| ip-172-31-38-82:24978:25047 [0] NCCL INFO Ring 02 : 40[0] -> 43[3] via P2P/IPC | |
| ip-172-31-45-171:25059:25135 [1] NCCL INFO Ring 02 : 25[1] -> 26[2] via P2P/IPC | |
| ip-172-31-44-4:22497:22559 [5] NCCL INFO Ring 02 : 125 -> 4 [send] via NET/AWS Libfabric/2 | |
| ip-172-31-39-141:20304:20381 [1] NCCL INFO Ring 02 : 1[1] -> 2[2] via P2P/IPC | |
| ip-172-31-47-156:22561:22636 [1] NCCL INFO Ring 02 : 81[1] -> 82[2] via P2P/IPC | |
| ip-172-31-45-171:25063:25130 [5] NCCL INFO Ring 02 : 29 -> 36 [send] via NET/AWS Libfabric/2 | |
| ip-172-31-44-142:24934:25005 [5] NCCL INFO Ring 02 : 21 -> 28 [send] via NET/AWS Libfabric/2 | |
| ip-172-31-44-230:21992:22064 [5] NCCL INFO Ring 02 : 101 -> 108 [send] via NET/AWS Libfabric/2 | |
| ip-172-31-44-4:22493:22565 [1] NCCL INFO Ring 02 : 121[1] -> 122[2] via P2P/IPC | |
| ip-172-31-44-142:24930:25003 [1] NCCL INFO Ring 02 : 17[1] -> 18[2] via P2P/IPC | |
| ip-172-31-38-14:25303:25373 [5] NCCL INFO NET/OFI [5] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-47-240:24694:24764 [5] NCCL INFO Ring 02 : 61 -> 68 [send] via NET/AWS Libfabric/2 | |
| ip-172-31-44-4:22492:22563 [0] NCCL INFO Ring 02 : 120[0] -> 123[3] via P2P/IPC | |
| ip-172-31-47-240:24690:24765 [1] NCCL INFO Ring 02 : 57[1] -> 58[2] via P2P/IPC | |
| ip-172-31-36-188:22212:22276 [5] NCCL INFO Ring 02 : 93 -> 100 [send] via NET/AWS Libfabric/2 | |
| ip-172-31-45-171:25061:25134 [3] NCCL INFO Ring 02 : 27[3] -> 25[1] via P2P/IPC | |
| ip-172-31-44-230:21988:22060 [1] NCCL INFO Ring 02 : 97[1] -> 98[2] via P2P/IPC | |
| ip-172-31-36-188:22208:22278 [1] NCCL INFO Ring 02 : 89[1] -> 90[2] via P2P/IPC | |
| ip-172-31-47-156:22563:22638 [3] NCCL INFO Ring 02 : 83[3] -> 81[1] via P2P/IPC | |
| ip-172-31-32-86:24696:24767 [5] NCCL INFO Ring 02 : 53 -> 60 [send] via NET/AWS Libfabric/2 | |
| ip-172-31-39-141:20305:20380 [2] NCCL INFO Ring 02 : 2[2] -> 6[6] via P2P/IPC | |
| ip-172-31-46-50:22042:22112 [2] NCCL INFO Ring 02 : 66[2] -> 70[6] via P2P/IPC | |
| ip-172-31-36-128:22026:22094 [5] NCCL INFO Ring 02 : 117 -> 124 [send] via NET/AWS Libfabric/2 | |
| ip-172-31-33-235:21964:22033 [5] NCCL INFO Ring 02 : 109 -> 116 [send] via NET/AWS Libfabric/2 | |
| ip-172-31-33-235:21960:22031 [1] NCCL INFO Ring 02 : 105[1] -> 106[2] via P2P/IPC | |
| ip-172-31-32-86:24692:24769 [1] NCCL INFO Ring 02 : 49[1] -> 50[2] via P2P/IPC | |
| ip-172-31-46-50:22045:22115 [5] NCCL INFO Ring 02 : 69 -> 76 [send] via NET/AWS Libfabric/2 | |
| ip-172-31-44-142:24929:25002 [0] NCCL INFO Ring 02 : 16[0] -> 19[3] via P2P/IPC | |
| ip-172-31-36-128:22022:22097 [1] NCCL INFO Ring 02 : 113[1] -> 114[2] via P2P/IPC | |
| ip-172-31-44-230:21990:22065 [3] NCCL INFO Ring 02 : 99[3] -> 97[1] via P2P/IPC | |
| ip-172-31-44-142:24932:25006 [3] NCCL INFO Ring 02 : 19[3] -> 17[1] via P2P/IPC | |
| ip-172-31-47-240:24692:24760 [3] NCCL INFO Ring 02 : 59[3] -> 57[1] via P2P/IPC | |
| ip-172-31-46-50:22041:22117 [1] NCCL INFO Ring 02 : 65[1] -> 66[2] via P2P/IPC | |
| ip-172-31-37-141:25824:25897 [0] NCCL INFO Ring 02 : 32[0] -> 35[3] via P2P/IPC | |
| ip-172-31-37-141:25827:25893 [3] NCCL INFO Ring 02 : 35[3] -> 33[1] via P2P/IPC | |
| ip-172-31-45-171:25058:25128 [0] NCCL INFO Ring 02 : 24[0] -> 27[3] via P2P/IPC | |
| ip-172-31-47-156:22560:22634 [0] NCCL INFO Ring 02 : 80[0] -> 83[3] via P2P/IPC | |
| ip-172-31-39-141:20303:20372 [0] NCCL INFO Ring 02 : 0[0] -> 3[3] via P2P/IPC | |
| ip-172-31-38-82:24980:25051 [2] NCCL INFO Ring 02 : 42[2] -> 46[6] via P2P/IPC | |
| ip-172-31-47-188:22011:22083 [3] NCCL INFO Ring 02 : 75[3] -> 73[1] via P2P/IPC | |
| ip-172-31-47-188:22010:22085 [2] NCCL INFO Ring 02 : 74[2] -> 78[6] via P2P/IPC | |
| ip-172-31-32-86:24693:24766 [2] NCCL INFO Ring 02 : 50[2] -> 54[6] via P2P/IPC | |
| ip-172-31-32-86:24691:24765 [0] NCCL INFO Ring 02 : 48[0] -> 51[3] via P2P/IPC | |
| ip-172-31-38-14:25303:25373 [5] NCCL INFO Ring 02 : 13 -> 20 [send] via NET/AWS Libfabric/2 | |
| ip-172-31-36-188:22207:22281 [0] NCCL INFO Ring 02 : 88[0] -> 91[3] via P2P/IPC | |
| ip-172-31-38-14:25299:25372 [1] NCCL INFO Ring 02 : 9[1] -> 10[2] via P2P/IPC | |
| ip-172-31-32-86:24694:24770 [3] NCCL INFO Ring 02 : 51[3] -> 49[1] via P2P/IPC | |
| ip-172-31-47-240:24689:24766 [0] NCCL INFO Ring 02 : 56[0] -> 59[3] via P2P/IPC | |
| ip-172-31-46-50:22046:22116 [6] NCCL INFO NET/OFI [6] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-39-141:20309:20382 [6] NCCL INFO NET/OFI [6] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-47-156:22562:22633 [2] NCCL INFO Ring 02 : 82[2] -> 86[6] via P2P/IPC | |
| ip-172-31-44-4:22494:22564 [2] NCCL INFO Ring 02 : 122[2] -> 126[6] via P2P/IPC | |
| ip-172-31-44-230:21987:22059 [0] NCCL INFO Ring 02 : 96[0] -> 99[3] via P2P/IPC | |
| ip-172-31-36-128:22021:22098 [0] NCCL INFO Ring 02 : 112[0] -> 115[3] via P2P/IPC | |
| ip-172-31-37-141:25826:25891 [2] NCCL INFO Ring 02 : 34[2] -> 38[6] via P2P/IPC | |
| ip-172-31-33-235:21959:22032 [0] NCCL INFO Ring 02 : 104[0] -> 107[3] via P2P/IPC | |
| ip-172-31-36-188:22210:22277 [3] NCCL INFO Ring 02 : 91[3] -> 89[1] via P2P/IPC | |
| ip-172-31-47-188:22009:22082 [1] NCCL INFO Ring 02 : 73[1] -> 74[2] via P2P/IPC | |
| ip-172-31-38-14:25298:25374 [0] NCCL INFO Ring 02 : 8[0] -> 11[3] via P2P/IPC | |
| ip-172-31-38-82:24984:25049 [6] NCCL INFO NET/OFI [6] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-33-235:21962:22036 [3] NCCL INFO Ring 02 : 107[3] -> 105[1] via P2P/IPC | |
| ip-172-31-47-240:24691:24761 [2] NCCL INFO Ring 02 : 58[2] -> 62[6] via P2P/IPC | |
| ip-172-31-47-188:22014:22087 [6] NCCL INFO NET/OFI [6] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-32-86:24697:24768 [6] NCCL INFO NET/OFI [6] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-36-128:22024:22091 [3] NCCL INFO Ring 02 : 115[3] -> 113[1] via P2P/IPC | |
| ip-172-31-45-171:25060:25132 [2] NCCL INFO Ring 02 : 26[2] -> 30[6] via P2P/IPC | |
| ip-172-31-38-14:25300:25369 [2] NCCL INFO Ring 02 : 10[2] -> 14[6] via P2P/IPC | |
| ip-172-31-47-156:22566:22635 [6] NCCL INFO NET/OFI [6] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-33-235:21961:22023 [2] NCCL INFO Ring 02 : 106[2] -> 110[6] via P2P/IPC | |
| ip-172-31-44-4:22498:22560 [6] NCCL INFO NET/OFI [6] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-36-188:22209:22274 [2] NCCL INFO Ring 02 : 90[2] -> 94[6] via P2P/IPC | |
| ip-172-31-46-50:22046:22116 [6] NCCL INFO Ring 03 : 63 -> 70 [receive] via NET/AWS Libfabric/3 | |
| ip-172-31-44-230:21989:22061 [2] NCCL INFO Ring 02 : 98[2] -> 102[6] via P2P/IPC | |
| ip-172-31-37-141:25830:25895 [6] NCCL INFO NET/OFI [6] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-39-141:20309:20382 [6] NCCL INFO Ring 03 : 127 -> 6 [receive] via NET/AWS Libfabric/3 | |
| ip-172-31-47-240:24695:24762 [6] NCCL INFO NET/OFI [6] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-45-171:25064:25129 [6] NCCL INFO NET/OFI [6] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-36-128:22023:22093 [2] NCCL INFO Ring 02 : 114[2] -> 118[6] via P2P/IPC | |
| ip-172-31-47-188:22008:22081 [0] NCCL INFO Ring 02 : 72[0] -> 75[3] via P2P/IPC | |
| ip-172-31-44-142:24931:25004 [2] NCCL INFO Ring 02 : 18[2] -> 22[6] via P2P/IPC | |
| ip-172-31-47-188:22014:22087 [6] NCCL INFO Ring 03 : 71 -> 78 [receive] via NET/AWS Libfabric/3 | |
| ip-172-31-38-14:25304:25371 [6] NCCL INFO NET/OFI [6] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-38-82:24984:25049 [6] NCCL INFO Ring 03 : 39 -> 46 [receive] via NET/AWS Libfabric/3 | |
| ip-172-31-32-86:24697:24768 [6] NCCL INFO Ring 03 : 47 -> 54 [receive] via NET/AWS Libfabric/3 | |
| ip-172-31-33-235:21965:22037 [6] NCCL INFO NET/OFI [6] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-36-188:22213:22279 [6] NCCL INFO NET/OFI [6] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-44-230:21993:22066 [6] NCCL INFO NET/OFI [6] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-44-4:22498:22560 [6] NCCL INFO Ring 03 : 119 -> 126 [receive] via NET/AWS Libfabric/3 | |
| ip-172-31-47-156:22566:22635 [6] NCCL INFO Ring 03 : 79 -> 86 [receive] via NET/AWS Libfabric/3 | |
| ip-172-31-38-14:25301:25375 [3] NCCL INFO Ring 02 : 11[3] -> 9[1] via P2P/IPC | |
| ip-172-31-36-128:22027:22092 [6] NCCL INFO NET/OFI [6] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-37-141:25830:25895 [6] NCCL INFO Ring 03 : 31 -> 38 [receive] via NET/AWS Libfabric/3 | |
| ip-172-31-44-142:24935:25000 [6] NCCL INFO NET/OFI [6] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-47-240:24695:24762 [6] NCCL INFO Ring 03 : 55 -> 62 [receive] via NET/AWS Libfabric/3 | |
| ip-172-31-45-171:25064:25129 [6] NCCL INFO Ring 03 : 23 -> 30 [receive] via NET/AWS Libfabric/3 | |
| ip-172-31-38-14:25304:25371 [6] NCCL INFO Ring 03 : 7 -> 14 [receive] via NET/AWS Libfabric/3 | |
| ip-172-31-33-235:21965:22037 [6] NCCL INFO Ring 03 : 103 -> 110 [receive] via NET/AWS Libfabric/3 | |
| ip-172-31-36-188:22213:22279 [6] NCCL INFO Ring 03 : 87 -> 94 [receive] via NET/AWS Libfabric/3 | |
| ip-172-31-44-230:21993:22066 [6] NCCL INFO Ring 03 : 95 -> 102 [receive] via NET/AWS Libfabric/3 | |
| ip-172-31-38-82:24985:25053 [7] NCCL INFO NET/OFI [7] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-36-128:22027:22092 [6] NCCL INFO Ring 03 : 111 -> 118 [receive] via NET/AWS Libfabric/3 | |
| ip-172-31-44-142:24935:25000 [6] NCCL INFO Ring 03 : 15 -> 22 [receive] via NET/AWS Libfabric/3 | |
| ip-172-31-39-141:20310:20383 [7] NCCL INFO NET/OFI [7] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-47-156:22569:22639 [7] NCCL INFO NET/OFI [7] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-38-82:24985:25053 [7] NCCL INFO Ring 03 : 47 -> 54 [send] via NET/AWS Libfabric/3 | |
| ip-172-31-44-142:24937:25007 [7] NCCL INFO NET/OFI [7] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-39-141:20310:20383 [7] NCCL INFO Ring 03 : 7 -> 14 [send] via NET/AWS Libfabric/3 | |
| ip-172-31-44-4:22499:22562 [7] NCCL INFO NET/OFI [7] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-37-141:25831:25892 [7] NCCL INFO NET/OFI [7] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-47-188:22015:22086 [7] NCCL INFO NET/OFI [7] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-44-230:21994:22063 [7] NCCL INFO NET/OFI [7] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-45-171:25065:25133 [7] NCCL INFO NET/OFI [7] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-47-240:24696:24763 [7] NCCL INFO NET/OFI [7] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-36-188:22214:22275 [7] NCCL INFO NET/OFI [7] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-47-156:22569:22639 [7] NCCL INFO Ring 03 : 87 -> 94 [send] via NET/AWS Libfabric/3 | |
| ip-172-31-46-50:22047:22114 [7] NCCL INFO NET/OFI [7] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-44-142:24937:25007 [7] NCCL INFO Ring 03 : 23 -> 30 [send] via NET/AWS Libfabric/3 | |
| ip-172-31-36-128:22030:22096 [7] NCCL INFO NET/OFI [7] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-33-235:21966:22034 [7] NCCL INFO NET/OFI [7] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-44-4:22499:22562 [7] NCCL INFO Ring 03 : 127 -> 6 [send] via NET/AWS Libfabric/3 | |
| ip-172-31-37-141:25831:25892 [7] NCCL INFO Ring 03 : 39 -> 46 [send] via NET/AWS Libfabric/3 | |
| ip-172-31-32-86:24698:24764 [7] NCCL INFO NET/OFI [7] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-45-171:25065:25133 [7] NCCL INFO Ring 03 : 31 -> 38 [send] via NET/AWS Libfabric/3 | |
| ip-172-31-44-230:21994:22063 [7] NCCL INFO Ring 03 : 103 -> 110 [send] via NET/AWS Libfabric/3 | |
| ip-172-31-38-14:25306:25370 [7] NCCL INFO NET/OFI [7] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-47-240:24696:24763 [7] NCCL INFO Ring 03 : 63 -> 70 [send] via NET/AWS Libfabric/3 | |
| ip-172-31-47-188:22015:22086 [7] NCCL INFO Ring 03 : 79 -> 86 [send] via NET/AWS Libfabric/3 | |
| ip-172-31-36-188:22214:22275 [7] NCCL INFO Ring 03 : 95 -> 102 [send] via NET/AWS Libfabric/3 | |
| ip-172-31-46-50:22047:22114 [7] NCCL INFO Ring 03 : 71 -> 78 [send] via NET/AWS Libfabric/3 | |
| ip-172-31-36-128:22030:22096 [7] NCCL INFO Ring 03 : 119 -> 126 [send] via NET/AWS Libfabric/3 | |
| ip-172-31-33-235:21966:22034 [7] NCCL INFO Ring 03 : 111 -> 118 [send] via NET/AWS Libfabric/3 | |
| ip-172-31-32-86:24698:24764 [7] NCCL INFO Ring 03 : 55 -> 62 [send] via NET/AWS Libfabric/3 | |
| ip-172-31-38-14:25306:25370 [7] NCCL INFO Ring 03 : 15 -> 22 [send] via NET/AWS Libfabric/3 | |
| ip-172-31-46-50:22043:22118 [3] NCCL INFO Ring 03 : 67[3] -> 64[0] via P2P/IPC | |
| ip-172-31-38-82:24979:25052 [1] NCCL INFO Ring 03 : 41[1] -> 43[3] via P2P/IPC | |
| ip-172-31-38-82:24981:25048 [3] NCCL INFO Ring 03 : 43[3] -> 40[0] via P2P/IPC | |
| ip-172-31-46-50:22040:22119 [0] NCCL INFO Ring 03 : 64[0] -> 68[4] via P2P/IPC | |
| ip-172-31-44-4:22495:22558 [3] NCCL INFO Ring 03 : 123[3] -> 120[0] via P2P/IPC | |
| ip-172-31-47-156:22561:22636 [1] NCCL INFO Ring 03 : 81[1] -> 83[3] via P2P/IPC | |
| ip-172-31-44-142:24930:25003 [1] NCCL INFO Ring 03 : 17[1] -> 19[3] via P2P/IPC | |
| ip-172-31-44-4:22493:22565 [1] NCCL INFO Ring 03 : 121[1] -> 123[3] via P2P/IPC | |
| ip-172-31-45-171:25059:25135 [1] NCCL INFO Ring 03 : 25[1] -> 27[3] via P2P/IPC | |
| ip-172-31-46-50:22046:22116 [6] NCCL INFO Ring 03 : 70[6] -> 66[2] via P2P/IPC | |
| ip-172-31-44-4:22492:22563 [0] NCCL INFO Ring 03 : 120[0] -> 124[4] via P2P/IPC | |
| ip-172-31-38-82:24978:25047 [0] NCCL INFO Ring 03 : 40[0] -> 44[4] via P2P/IPC | |
| ip-172-31-39-141:20306:20379 [3] NCCL INFO Ring 03 : 3[3] -> 0[0] via P2P/IPC | |
| ip-172-31-39-141:20304:20381 [1] NCCL INFO Ring 03 : 1[1] -> 3[3] via P2P/IPC | |
| ip-172-31-47-156:22563:22638 [3] NCCL INFO Ring 03 : 83[3] -> 80[0] via P2P/IPC | |
| ip-172-31-39-141:20309:20382 [6] NCCL INFO Ring 03 : 6[6] -> 2[2] via P2P/IPC | |
| ip-172-31-44-142:24929:25002 [0] NCCL INFO Ring 03 : 16[0] -> 20[4] via P2P/IPC | |
| ip-172-31-37-141:25825:25894 [1] NCCL INFO Ring 03 : 33[1] -> 35[3] via P2P/IPC | |
| ip-172-31-47-240:24690:24765 [1] NCCL INFO Ring 03 : 57[1] -> 59[3] via P2P/IPC | |
| ip-172-31-38-82:24984:25049 [6] NCCL INFO Ring 03 : 46[6] -> 42[2] via P2P/IPC | |
| ip-172-31-46-50:22042:22112 [2] NCCL INFO Ring 03 : 66[2] -> 65[1] via P2P/IPC | |
| ip-172-31-45-171:25061:25134 [3] NCCL INFO Ring 03 : 27[3] -> 24[0] via P2P/IPC | |
| ip-172-31-44-4:22498:22560 [6] NCCL INFO Ring 03 : 126[6] -> 122[2] via P2P/IPC | |
| ip-172-31-44-230:21988:22060 [1] NCCL INFO Ring 03 : 97[1] -> 99[3] via P2P/IPC | |
| ip-172-31-44-142:24932:25006 [3] NCCL INFO Ring 03 : 19[3] -> 16[0] via P2P/IPC | |
| ip-172-31-47-156:22560:22634 [0] NCCL INFO Ring 03 : 80[0] -> 84[4] via P2P/IPC | |
| ip-172-31-47-188:22014:22087 [6] NCCL INFO Ring 03 : 78[6] -> 74[2] via P2P/IPC | |
| ip-172-31-36-128:22022:22097 [1] NCCL INFO Ring 03 : 113[1] -> 115[3] via P2P/IPC | |
| ip-172-31-36-188:22208:22278 [1] NCCL INFO Ring 03 : 89[1] -> 91[3] via P2P/IPC | |
| ip-172-31-46-50:22041:22117 [1] NCCL INFO Ring 03 : 65[1] -> 67[3] via P2P/IPC | |
| ip-172-31-32-86:24697:24768 [6] NCCL INFO Ring 03 : 54[6] -> 50[2] via P2P/IPC | |
| ip-172-31-44-230:21990:22065 [3] NCCL INFO Ring 03 : 99[3] -> 96[0] via P2P/IPC | |
| ip-172-31-47-240:24692:24760 [3] NCCL INFO Ring 03 : 59[3] -> 56[0] via P2P/IPC | |
| ip-172-31-39-141:20305:20380 [2] NCCL INFO Ring 03 : 2[2] -> 1[1] via P2P/IPC | |
| ip-172-31-47-240:24695:24762 [6] NCCL INFO Ring 03 : 62[6] -> 58[2] via P2P/IPC | |
| ip-172-31-38-82:24980:25051 [2] NCCL INFO Ring 03 : 42[2] -> 41[1] via P2P/IPC | |
| ip-172-31-33-235:21960:22031 [1] NCCL INFO Ring 03 : 105[1] -> 107[3] via P2P/IPC | |
| ip-172-31-45-171:25058:25128 [0] NCCL INFO Ring 03 : 24[0] -> 28[4] via P2P/IPC | |
| ip-172-31-44-4:22494:22564 [2] NCCL INFO Ring 03 : 122[2] -> 121[1] via P2P/IPC | |
| ip-172-31-45-171:25064:25129 [6] NCCL INFO Ring 03 : 30[6] -> 26[2] via P2P/IPC | |
| ip-172-31-39-141:20303:20372 [0] NCCL INFO Ring 03 : 0[0] -> 4[4] via P2P/IPC | |
| ip-172-31-47-156:22562:22633 [2] NCCL INFO Ring 03 : 82[2] -> 81[1] via P2P/IPC | |
| ip-172-31-37-141:25830:25895 [6] NCCL INFO Ring 03 : 38[6] -> 34[2] via P2P/IPC | |
| ip-172-31-36-188:22207:22281 [0] NCCL INFO Ring 03 : 88[0] -> 92[4] via P2P/IPC | |
| ip-172-31-47-188:22010:22085 [2] NCCL INFO Ring 03 : 74[2] -> 73[1] via P2P/IPC | |
| ip-172-31-38-14:25304:25371 [6] NCCL INFO Ring 03 : 14[6] -> 10[2] via P2P/IPC | |
| ip-172-31-47-240:24689:24766 [0] NCCL INFO Ring 03 : 56[0] -> 60[4] via P2P/IPC | |
| ip-172-31-37-141:25824:25897 [0] NCCL INFO Ring 03 : 32[0] -> 36[4] via P2P/IPC | |
| ip-172-31-37-141:25827:25893 [3] NCCL INFO Ring 03 : 35[3] -> 32[0] via P2P/IPC | |
| ip-172-31-44-142:24935:25000 [6] NCCL INFO Ring 03 : 22[6] -> 18[2] via P2P/IPC | |
| ip-172-31-33-235:21965:22037 [6] NCCL INFO Ring 03 : 110[6] -> 106[2] via P2P/IPC | |
| ip-172-31-36-128:22021:22098 [0] NCCL INFO Ring 03 : 112[0] -> 116[4] via P2P/IPC | |
| ip-172-31-47-188:22011:22083 [3] NCCL INFO Ring 03 : 75[3] -> 72[0] via P2P/IPC | |
| ip-172-31-36-188:22213:22279 [6] NCCL INFO Ring 03 : 94[6] -> 90[2] via P2P/IPC | |
| ip-172-31-47-156:22566:22635 [6] NCCL INFO Ring 03 : 86[6] -> 82[2] via P2P/IPC | |
| ip-172-31-44-230:21993:22066 [6] NCCL INFO Ring 03 : 102[6] -> 98[2] via P2P/IPC | |
| ip-172-31-36-128:22027:22092 [6] NCCL INFO Ring 03 : 118[6] -> 114[2] via P2P/IPC | |
| ip-172-31-44-230:21987:22059 [0] NCCL INFO Ring 03 : 96[0] -> 100[4] via P2P/IPC | |
| ip-172-31-47-188:22009:22082 [1] NCCL INFO Ring 03 : 73[1] -> 75[3] via P2P/IPC | |
| ip-172-31-45-171:25060:25132 [2] NCCL INFO Ring 03 : 26[2] -> 25[1] via P2P/IPC | |
| ip-172-31-32-86:24692:24769 [1] NCCL INFO Ring 03 : 49[1] -> 51[3] via P2P/IPC | |
| ip-172-31-47-240:24691:24761 [2] NCCL INFO Ring 03 : 58[2] -> 57[1] via P2P/IPC | |
| ip-172-31-36-128:22024:22091 [3] NCCL INFO Ring 03 : 115[3] -> 112[0] via P2P/IPC | |
| ip-172-31-33-235:21959:22032 [0] NCCL INFO Ring 03 : 104[0] -> 108[4] via P2P/IPC | |
| ip-172-31-36-188:22210:22277 [3] NCCL INFO Ring 03 : 91[3] -> 88[0] via P2P/IPC | |
| ip-172-31-32-86:24693:24766 [2] NCCL INFO Ring 03 : 50[2] -> 49[1] via P2P/IPC | |
| ip-172-31-37-141:25826:25891 [2] NCCL INFO Ring 03 : 34[2] -> 33[1] via P2P/IPC | |
| ip-172-31-32-86:24691:24765 [0] NCCL INFO Ring 03 : 48[0] -> 52[4] via P2P/IPC | |
| ip-172-31-33-235:21962:22036 [3] NCCL INFO Ring 03 : 107[3] -> 104[0] via P2P/IPC | |
| ip-172-31-44-142:24931:25004 [2] NCCL INFO Ring 03 : 18[2] -> 17[1] via P2P/IPC | |
| ip-172-31-38-14:25298:25374 [0] NCCL INFO Ring 03 : 8[0] -> 12[4] via P2P/IPC | |
| ip-172-31-44-230:21989:22061 [2] NCCL INFO Ring 03 : 98[2] -> 97[1] via P2P/IPC | |
| ip-172-31-32-86:24694:24770 [3] NCCL INFO Ring 03 : 51[3] -> 48[0] via P2P/IPC | |
| ip-172-31-33-235:21961:22023 [2] NCCL INFO Ring 03 : 106[2] -> 105[1] via P2P/IPC | |
| ip-172-31-36-188:22209:22274 [2] NCCL INFO Ring 03 : 90[2] -> 89[1] via P2P/IPC | |
| ip-172-31-36-128:22023:22093 [2] NCCL INFO Ring 03 : 114[2] -> 113[1] via P2P/IPC | |
| ip-172-31-38-14:25300:25369 [2] NCCL INFO Ring 03 : 10[2] -> 9[1] via P2P/IPC | |
| ip-172-31-38-82:24978:25047 [0] NCCL INFO NET/OFI [0] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-38-14:25299:25372 [1] NCCL INFO Ring 03 : 9[1] -> 11[3] via P2P/IPC | |
| ip-172-31-47-188:22008:22081 [0] NCCL INFO Ring 03 : 72[0] -> 76[4] via P2P/IPC | |
| ip-172-31-46-50:22040:22119 [0] NCCL INFO NET/OFI [0] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-38-82:24983:25050 [5] NCCL INFO Ring 03 : 45[5] -> 47[7] via P2P/IPC | |
| ip-172-31-45-171:25058:25128 [0] NCCL INFO NET/OFI [0] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-39-141:20303:20372 [0] NCCL INFO NET/OFI [0] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-36-188:22207:22281 [0] NCCL INFO NET/OFI [0] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-33-235:21959:22032 [0] NCCL INFO NET/OFI [0] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-32-86:24691:24765 [0] NCCL INFO NET/OFI [0] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-38-82:24978:25047 [0] NCCL INFO Ring 04 : 33 -> 40 [receive] via NET/AWS Libfabric/0 | |
| ip-172-31-44-230:21987:22059 [0] NCCL INFO NET/OFI [0] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-44-4:22492:22563 [0] NCCL INFO NET/OFI [0] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-39-141:20308:20378 [5] NCCL INFO Ring 03 : 5[5] -> 7[7] via P2P/IPC | |
| ip-172-31-37-141:25824:25897 [0] NCCL INFO NET/OFI [0] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-36-128:22021:22098 [0] NCCL INFO NET/OFI [0] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-38-14:25301:25375 [3] NCCL INFO Ring 03 : 11[3] -> 8[0] via P2P/IPC | |
| ip-172-31-47-240:24689:24766 [0] NCCL INFO NET/OFI [0] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-44-142:24929:25002 [0] NCCL INFO NET/OFI [0] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-47-156:22560:22634 [0] NCCL INFO NET/OFI [0] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-46-50:22040:22119 [0] NCCL INFO Ring 04 : 57 -> 64 [receive] via NET/AWS Libfabric/0 | |
| ip-172-31-45-171:25058:25128 [0] NCCL INFO Ring 04 : 17 -> 24 [receive] via NET/AWS Libfabric/0 | |
| ip-172-31-47-156:22565:22637 [5] NCCL INFO Ring 03 : 85[5] -> 87[7] via P2P/IPC | |
| ip-172-31-44-142:24934:25005 [5] NCCL INFO Ring 03 : 21[5] -> 23[7] via P2P/IPC | |
| ip-172-31-39-141:20303:20372 [0] NCCL INFO Ring 04 : 121 -> 0 [receive] via NET/AWS Libfabric/0 | |
| ip-172-31-44-4:22497:22559 [5] NCCL INFO Ring 03 : 125[5] -> 127[7] via P2P/IPC | |
| ip-172-31-38-14:25298:25374 [0] NCCL INFO NET/OFI [0] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-36-188:22207:22281 [0] NCCL INFO Ring 04 : 81 -> 88 [receive] via NET/AWS Libfabric/0 | |
| ip-172-31-47-188:22008:22081 [0] NCCL INFO NET/OFI [0] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-33-235:21959:22032 [0] NCCL INFO Ring 04 : 97 -> 104 [receive] via NET/AWS Libfabric/0 | |
| ip-172-31-44-230:21987:22059 [0] NCCL INFO Ring 04 : 89 -> 96 [receive] via NET/AWS Libfabric/0 | |
| ip-172-31-32-86:24691:24765 [0] NCCL INFO Ring 04 : 41 -> 48 [receive] via NET/AWS Libfabric/0 | |
| ip-172-31-44-4:22492:22563 [0] NCCL INFO Ring 04 : 113 -> 120 [receive] via NET/AWS Libfabric/0 | |
| ip-172-31-44-230:21992:22064 [5] NCCL INFO Ring 03 : 101[5] -> 103[7] via P2P/IPC | |
| ip-172-31-36-128:22021:22098 [0] NCCL INFO Ring 04 : 105 -> 112 [receive] via NET/AWS Libfabric/0 | |
| ip-172-31-37-141:25829:25898 [5] NCCL INFO Ring 03 : 37[5] -> 39[7] via P2P/IPC | |
| ip-172-31-37-141:25824:25897 [0] NCCL INFO Ring 04 : 25 -> 32 [receive] via NET/AWS Libfabric/0 | |
| ip-172-31-47-240:24694:24764 [5] NCCL INFO Ring 03 : 61[5] -> 63[7] via P2P/IPC | |
| ip-172-31-47-240:24689:24766 [0] NCCL INFO Ring 04 : 49 -> 56 [receive] via NET/AWS Libfabric/0 | |
| ip-172-31-47-156:22560:22634 [0] NCCL INFO Ring 04 : 73 -> 80 [receive] via NET/AWS Libfabric/0 | |
| ip-172-31-44-142:24929:25002 [0] NCCL INFO Ring 04 : 9 -> 16 [receive] via NET/AWS Libfabric/0 | |
| ip-172-31-47-188:22013:22084 [5] NCCL INFO Ring 03 : 77[5] -> 79[7] via P2P/IPC | |
| ip-172-31-45-171:25063:25130 [5] NCCL INFO Ring 03 : 29[5] -> 31[7] via P2P/IPC | |
| ip-172-31-47-188:22008:22081 [0] NCCL INFO Ring 04 : 65 -> 72 [receive] via NET/AWS Libfabric/0 | |
| ip-172-31-38-14:25298:25374 [0] NCCL INFO Ring 04 : 1 -> 8 [receive] via NET/AWS Libfabric/0 | |
| ip-172-31-36-128:22026:22094 [5] NCCL INFO Ring 03 : 117[5] -> 119[7] via P2P/IPC | |
| ip-172-31-36-188:22212:22276 [5] NCCL INFO Ring 03 : 93[5] -> 95[7] via P2P/IPC | |
| ip-172-31-46-50:22045:22115 [5] NCCL INFO Ring 03 : 69[5] -> 71[7] via P2P/IPC | |
| ip-172-31-38-14:25302:25368 [4] NCCL INFO Ring 03 : 12[4] -> 13[5] via P2P/IPC | |
| ip-172-31-32-86:24695:24771 [4] NCCL INFO Ring 03 : 52[4] -> 53[5] via P2P/IPC | |
| ip-172-31-33-235:21964:22033 [5] NCCL INFO Ring 03 : 109[5] -> 111[7] via P2P/IPC | |
| ip-172-31-38-82:24982:25054 [4] NCCL INFO Ring 03 : 44[4] -> 45[5] via P2P/IPC | |
| ip-172-31-45-171:25062:25131 [4] NCCL INFO Ring 03 : 28[4] -> 29[5] via P2P/IPC | |
| ip-172-31-46-50:22044:22113 [4] NCCL INFO Ring 03 : 68[4] -> 69[5] via P2P/IPC | |
| ip-172-31-32-86:24696:24767 [5] NCCL INFO Ring 03 : 53[5] -> 55[7] via P2P/IPC | |
| ip-172-31-39-141:20307:20377 [4] NCCL INFO Ring 03 : 4[4] -> 5[5] via P2P/IPC | |
| ip-172-31-38-14:25303:25373 [5] NCCL INFO Ring 03 : 13[5] -> 15[7] via P2P/IPC | |
| ip-172-31-36-188:22211:22280 [4] NCCL INFO Ring 03 : 92[4] -> 93[5] via P2P/IPC | |
| ip-172-31-33-235:21963:22035 [4] NCCL INFO Ring 03 : 108[4] -> 109[5] via P2P/IPC | |
| ip-172-31-44-230:21991:22062 [4] NCCL INFO Ring 03 : 100[4] -> 101[5] via P2P/IPC | |
| ip-172-31-36-128:22025:22095 [4] NCCL INFO Ring 03 : 116[4] -> 117[5] via P2P/IPC | |
| ip-172-31-47-156:22564:22632 [4] NCCL INFO Ring 03 : 84[4] -> 85[5] via P2P/IPC | |
| ip-172-31-44-142:24933:25001 [4] NCCL INFO Ring 03 : 20[4] -> 21[5] via P2P/IPC | |
| ip-172-31-47-188:22012:22080 [4] NCCL INFO Ring 03 : 76[4] -> 77[5] via P2P/IPC | |
| ip-172-31-44-4:22496:22561 [4] NCCL INFO Ring 03 : 124[4] -> 125[5] via P2P/IPC | |
| ip-172-31-37-141:25828:25896 [4] NCCL INFO Ring 03 : 36[4] -> 37[5] via P2P/IPC | |
| ip-172-31-47-240:24693:24767 [4] NCCL INFO Ring 03 : 60[4] -> 61[5] via P2P/IPC | |
| ip-172-31-38-82:24978:25047 [0] NCCL INFO Ring 04 : 40[0] -> 42[2] via P2P/IPC | |
| ip-172-31-46-50:22040:22119 [0] NCCL INFO Ring 04 : 64[0] -> 66[2] via P2P/IPC | |
| ip-172-31-39-141:20303:20372 [0] NCCL INFO Ring 04 : 0[0] -> 2[2] via P2P/IPC | |
| ip-172-31-45-171:25058:25128 [0] NCCL INFO Ring 04 : 24[0] -> 26[2] via P2P/IPC | |
| ip-172-31-32-86:24691:24765 [0] NCCL INFO Ring 04 : 48[0] -> 50[2] via P2P/IPC | |
| ip-172-31-36-188:22207:22281 [0] NCCL INFO Ring 04 : 88[0] -> 90[2] via P2P/IPC | |
| ip-172-31-33-235:21959:22032 [0] NCCL INFO Ring 04 : 104[0] -> 106[2] via P2P/IPC | |
| ip-172-31-44-230:21987:22059 [0] NCCL INFO Ring 04 : 96[0] -> 98[2] via P2P/IPC | |
| ip-172-31-44-4:22492:22563 [0] NCCL INFO Ring 04 : 120[0] -> 122[2] via P2P/IPC | |
| ip-172-31-47-156:22560:22634 [0] NCCL INFO Ring 04 : 80[0] -> 82[2] via P2P/IPC | |
| ip-172-31-37-141:25824:25897 [0] NCCL INFO Ring 04 : 32[0] -> 34[2] via P2P/IPC | |
| ip-172-31-47-240:24689:24766 [0] NCCL INFO Ring 04 : 56[0] -> 58[2] via P2P/IPC | |
| ip-172-31-36-128:22021:22098 [0] NCCL INFO Ring 04 : 112[0] -> 114[2] via P2P/IPC | |
| ip-172-31-44-142:24929:25002 [0] NCCL INFO Ring 04 : 16[0] -> 18[2] via P2P/IPC | |
| ip-172-31-38-82:24981:25048 [3] NCCL INFO Ring 04 : 43[3] -> 47[7] via P2P/IPC | |
| ip-172-31-47-188:22008:22081 [0] NCCL INFO Ring 04 : 72[0] -> 74[2] via P2P/IPC | |
| ip-172-31-38-14:25298:25374 [0] NCCL INFO Ring 04 : 8[0] -> 10[2] via P2P/IPC | |
| ip-172-31-44-4:22495:22558 [3] NCCL INFO Ring 04 : 123[3] -> 127[7] via P2P/IPC | |
| ip-172-31-46-50:22043:22118 [3] NCCL INFO Ring 04 : 67[3] -> 71[7] via P2P/IPC | |
| ip-172-31-39-141:20306:20379 [3] NCCL INFO Ring 04 : 3[3] -> 7[7] via P2P/IPC | |
| ip-172-31-47-156:22563:22638 [3] NCCL INFO Ring 04 : 83[3] -> 87[7] via P2P/IPC | |
| ip-172-31-47-156:22561:22636 [1] NCCL INFO NET/OFI [1] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-38-82:24979:25052 [1] NCCL INFO NET/OFI [1] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-39-141:20304:20381 [1] NCCL INFO NET/OFI [1] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-44-4:22493:22565 [1] NCCL INFO NET/OFI [1] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-45-171:25061:25134 [3] NCCL INFO Ring 04 : 27[3] -> 31[7] via P2P/IPC | |
| ip-172-31-46-50:22042:22112 [2] NCCL INFO Ring 04 : 66[2] -> 67[3] via P2P/IPC | |
| ip-172-31-44-142:24932:25006 [3] NCCL INFO Ring 04 : 19[3] -> 23[7] via P2P/IPC | |
| ip-172-31-46-50:22041:22117 [1] NCCL INFO NET/OFI [1] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-38-82:24980:25051 [2] NCCL INFO Ring 04 : 42[2] -> 43[3] via P2P/IPC | |
| ip-172-31-39-141:20305:20380 [2] NCCL INFO Ring 04 : 2[2] -> 3[3] via P2P/IPC | |
| ip-172-31-44-230:21990:22065 [3] NCCL INFO Ring 04 : 99[3] -> 103[7] via P2P/IPC | |
| ip-172-31-38-82:24979:25052 [1] NCCL INFO Ring 04 : 41 -> 48 [send] via NET/AWS Libfabric/0 | |
| ip-172-31-47-240:24692:24760 [3] NCCL INFO Ring 04 : 59[3] -> 63[7] via P2P/IPC | |
| ip-172-31-45-171:25059:25135 [1] NCCL INFO NET/OFI [1] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-39-141:20304:20381 [1] NCCL INFO Ring 04 : 1 -> 8 [send] via NET/AWS Libfabric/0 | |
| ip-172-31-47-156:22561:22636 [1] NCCL INFO Ring 04 : 81 -> 88 [send] via NET/AWS Libfabric/0 | |
| ip-172-31-47-240:24690:24765 [1] NCCL INFO NET/OFI [1] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-44-4:22493:22565 [1] NCCL INFO Ring 04 : 121 -> 0 [send] via NET/AWS Libfabric/0 | |
| ip-172-31-44-4:22494:22564 [2] NCCL INFO Ring 04 : 122[2] -> 123[3] via P2P/IPC | |
| ip-172-31-47-156:22562:22633 [2] NCCL INFO Ring 04 : 82[2] -> 83[3] via P2P/IPC | |
| ip-172-31-44-142:24930:25003 [1] NCCL INFO NET/OFI [1] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-37-141:25825:25894 [1] NCCL INFO NET/OFI [1] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-47-188:22010:22085 [2] NCCL INFO Ring 04 : 74[2] -> 75[3] via P2P/IPC | |
| ip-172-31-46-50:22046:22116 [6] NCCL INFO Ring 04 : 70[6] -> 69[5] via P2P/IPC | |
| ip-172-31-44-230:21988:22060 [1] NCCL INFO NET/OFI [1] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-46-50:22042:22112 [2] NCCL INFO NET/OFI [2] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-38-82:24984:25049 [6] NCCL INFO Ring 04 : 46[6] -> 45[5] via P2P/IPC | |
| ip-172-31-46-50:22041:22117 [1] NCCL INFO Ring 04 : 65 -> 72 [send] via NET/AWS Libfabric/0 | |
| ip-172-31-37-141:25827:25893 [3] NCCL INFO Ring 04 : 35[3] -> 39[7] via P2P/IPC | |
| ip-172-31-45-171:25059:25135 [1] NCCL INFO Ring 04 : 25 -> 32 [send] via NET/AWS Libfabric/0 | |
| ip-172-31-47-188:22011:22083 [3] NCCL INFO Ring 04 : 75[3] -> 79[7] via P2P/IPC | |
| ip-172-31-45-171:25060:25132 [2] NCCL INFO Ring 04 : 26[2] -> 27[3] via P2P/IPC | |
| ip-172-31-39-141:20309:20382 [6] NCCL INFO Ring 04 : 6[6] -> 5[5] via P2P/IPC | |
| ip-172-31-36-128:22022:22097 [1] NCCL INFO NET/OFI [1] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-33-235:21960:22031 [1] NCCL INFO NET/OFI [1] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-47-240:24691:24761 [2] NCCL INFO Ring 04 : 58[2] -> 59[3] via P2P/IPC | |
| ip-172-31-47-240:24690:24765 [1] NCCL INFO Ring 04 : 57 -> 64 [send] via NET/AWS Libfabric/0 | |
| ip-172-31-38-82:24980:25051 [2] NCCL INFO NET/OFI [2] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-47-188:22009:22082 [1] NCCL INFO NET/OFI [1] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-36-128:22024:22091 [3] NCCL INFO Ring 04 : 115[3] -> 119[7] via P2P/IPC | |
| ip-172-31-36-188:22208:22278 [1] NCCL INFO NET/OFI [1] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-44-4:22498:22560 [6] NCCL INFO Ring 04 : 126[6] -> 125[5] via P2P/IPC | |
| ip-172-31-39-141:20305:20380 [2] NCCL INFO NET/OFI [2] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-38-82:24985:25053 [7] NCCL INFO Ring 04 : 47[7] -> 44[4] via P2P/IPC | |
| ip-172-31-32-86:24692:24769 [1] NCCL INFO NET/OFI [1] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-44-4:22494:22564 [2] NCCL INFO NET/OFI [2] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-47-156:22562:22633 [2] NCCL INFO NET/OFI [2] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-44-142:24930:25003 [1] NCCL INFO Ring 04 : 17 -> 24 [send] via NET/AWS Libfabric/0 | |
| ip-172-31-44-142:24931:25004 [2] NCCL INFO Ring 04 : 18[2] -> 19[3] via P2P/IPC | |
| ip-172-31-32-86:24693:24766 [2] NCCL INFO Ring 04 : 50[2] -> 51[3] via P2P/IPC | |
| ip-172-31-44-230:21989:22061 [2] NCCL INFO Ring 04 : 98[2] -> 99[3] via P2P/IPC | |
| ip-172-31-44-230:21988:22060 [1] NCCL INFO Ring 04 : 97 -> 104 [send] via NET/AWS Libfabric/0 | |
| ip-172-31-46-50:22042:22112 [2] NCCL INFO Ring 05 : 59 -> 66 [receive] via NET/AWS Libfabric/1 | |
| ip-172-31-37-141:25825:25894 [1] NCCL INFO Ring 04 : 33 -> 40 [send] via NET/AWS Libfabric/0 | |
| ip-172-31-33-235:21962:22036 [3] NCCL INFO Ring 04 : 107[3] -> 111[7] via P2P/IPC | |
| ip-172-31-37-141:25826:25891 [2] NCCL INFO Ring 04 : 34[2] -> 35[3] via P2P/IPC | |
| ip-172-31-45-171:25064:25129 [6] NCCL INFO Ring 04 : 30[6] -> 29[5] via P2P/IPC | |
| ip-172-31-36-188:22210:22277 [3] NCCL INFO Ring 04 : 91[3] -> 95[7] via P2P/IPC | |
| ip-172-31-47-188:22014:22087 [6] NCCL INFO Ring 04 : 78[6] -> 77[5] via P2P/IPC | |
| ip-172-31-47-240:24695:24762 [6] NCCL INFO Ring 04 : 62[6] -> 61[5] via P2P/IPC | |
| ip-172-31-38-82:24980:25051 [2] NCCL INFO Ring 05 : 35 -> 42 [receive] via NET/AWS Libfabric/1 | |
| ip-172-31-32-86:24694:24770 [3] NCCL INFO Ring 04 : 51[3] -> 55[7] via P2P/IPC | |
| ip-172-31-36-128:22022:22097 [1] NCCL INFO Ring 04 : 113 -> 120 [send] via NET/AWS Libfabric/0 | |
| ip-172-31-36-128:22023:22093 [2] NCCL INFO Ring 04 : 114[2] -> 115[3] via P2P/IPC | |
| ip-172-31-47-188:22010:22085 [2] NCCL INFO NET/OFI [2] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-33-235:21960:22031 [1] NCCL INFO Ring 04 : 105 -> 112 [send] via NET/AWS Libfabric/0 | |
| ip-172-31-33-235:21961:22023 [2] NCCL INFO Ring 04 : 106[2] -> 107[3] via P2P/IPC | |
| ip-172-31-39-141:20310:20383 [7] NCCL INFO Ring 04 : 7[7] -> 4[4] via P2P/IPC | |
| ip-172-31-47-188:22009:22082 [1] NCCL INFO Ring 04 : 73 -> 80 [send] via NET/AWS Libfabric/0 | |
| ip-172-31-32-86:24697:24768 [6] NCCL INFO Ring 04 : 54[6] -> 53[5] via P2P/IPC | |
| ip-172-31-47-156:22566:22635 [6] NCCL INFO Ring 04 : 86[6] -> 85[5] via P2P/IPC | |
| ip-172-31-39-141:20305:20380 [2] NCCL INFO Ring 05 : 123 -> 2 [receive] via NET/AWS Libfabric/1 | |
| ip-172-31-36-188:22208:22278 [1] NCCL INFO Ring 04 : 89 -> 96 [send] via NET/AWS Libfabric/0 | |
| ip-172-31-47-156:22562:22633 [2] NCCL INFO Ring 05 : 75 -> 82 [receive] via NET/AWS Libfabric/1 | |
| ip-172-31-32-86:24692:24769 [1] NCCL INFO Ring 04 : 49 -> 56 [send] via NET/AWS Libfabric/0 | |
| ip-172-31-44-4:22494:22564 [2] NCCL INFO Ring 05 : 115 -> 122 [receive] via NET/AWS Libfabric/1 | |
| ip-172-31-47-156:22569:22639 [7] NCCL INFO Ring 04 : 87[7] -> 84[4] via P2P/IPC | |
| ip-172-31-44-142:24935:25000 [6] NCCL INFO Ring 04 : 22[6] -> 21[5] via P2P/IPC | |
| ip-172-31-45-171:25060:25132 [2] NCCL INFO NET/OFI [2] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-36-188:22209:22274 [2] NCCL INFO Ring 04 : 90[2] -> 91[3] via P2P/IPC | |
| ip-172-31-47-240:24691:24761 [2] NCCL INFO NET/OFI [2] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-37-141:25830:25895 [6] NCCL INFO Ring 04 : 38[6] -> 37[5] via P2P/IPC | |
| ip-172-31-44-230:21993:22066 [6] NCCL INFO Ring 04 : 102[6] -> 101[5] via P2P/IPC | |
| ip-172-31-38-14:25300:25369 [2] NCCL INFO Ring 04 : 10[2] -> 11[3] via P2P/IPC | |
| ip-172-31-44-4:22499:22562 [7] NCCL INFO Ring 04 : 127[7] -> 124[4] via P2P/IPC | |
| ip-172-31-44-142:24937:25007 [7] NCCL INFO Ring 04 : 23[7] -> 20[4] via P2P/IPC | |
| ip-172-31-44-142:24931:25004 [2] NCCL INFO NET/OFI [2] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-36-128:22027:22092 [6] NCCL INFO Ring 04 : 118[6] -> 117[5] via P2P/IPC | |
| ip-172-31-44-230:21989:22061 [2] NCCL INFO NET/OFI [2] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-38-14:25299:25372 [1] NCCL INFO NET/OFI [1] getCudaPath dev 0 busId 0000:00:16.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-32-86:24693:24766 [2] NCCL INFO NET/OFI [2] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-33-235:21965:22037 [6] NCCL INFO Ring 04 : 110[6] -> 109[5] via P2P/IPC | |
| ip-172-31-47-188:22010:22085 [2] NCCL INFO Ring 05 : 67 -> 74 [receive] via NET/AWS Libfabric/1 | |
| ip-172-31-44-230:21994:22063 [7] NCCL INFO Ring 04 : 103[7] -> 100[4] via P2P/IPC | |
| ip-172-31-36-188:22213:22279 [6] NCCL INFO Ring 04 : 94[6] -> 93[5] via P2P/IPC | |
| ip-172-31-37-141:25826:25891 [2] NCCL INFO NET/OFI [2] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-45-171:25060:25132 [2] NCCL INFO Ring 05 : 19 -> 26 [receive] via NET/AWS Libfabric/1 | |
| ip-172-31-38-14:25304:25371 [6] NCCL INFO Ring 04 : 14[6] -> 13[5] via P2P/IPC | |
| ip-172-31-47-240:24691:24761 [2] NCCL INFO Ring 05 : 51 -> 58 [receive] via NET/AWS Libfabric/1 | |
| ip-172-31-47-240:24696:24763 [7] NCCL INFO Ring 04 : 63[7] -> 60[4] via P2P/IPC | |
| ip-172-31-45-171:25065:25133 [7] NCCL INFO Ring 04 : 31[7] -> 28[4] via P2P/IPC | |
| ip-172-31-36-128:22023:22093 [2] NCCL INFO NET/OFI [2] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-33-235:21961:22023 [2] NCCL INFO NET/OFI [2] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-44-142:24931:25004 [2] NCCL INFO Ring 05 : 11 -> 18 [receive] via NET/AWS Libfabric/1 | |
| ip-172-31-37-141:25831:25892 [7] NCCL INFO Ring 04 : 39[7] -> 36[4] via P2P/IPC | |
| ip-172-31-38-82:24983:25050 [5] NCCL INFO Ring 04 : 45[5] -> 41[1] via P2P/IPC | |
| ip-172-31-47-188:22015:22086 [7] NCCL INFO Ring 04 : 79[7] -> 76[4] via P2P/IPC | |
| ip-172-31-44-230:21989:22061 [2] NCCL INFO Ring 05 : 91 -> 98 [receive] via NET/AWS Libfabric/1 | |
| ip-172-31-46-50:22047:22114 [7] NCCL INFO Ring 04 : 71[7] -> 68[4] via P2P/IPC | |
| ip-172-31-36-188:22209:22274 [2] NCCL INFO NET/OFI [2] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-32-86:24693:24766 [2] NCCL INFO Ring 05 : 43 -> 50 [receive] via NET/AWS Libfabric/1 | |
| ip-172-31-38-14:25299:25372 [1] NCCL INFO Ring 04 : 9 -> 16 [send] via NET/AWS Libfabric/0 | |
| ip-172-31-37-141:25826:25891 [2] NCCL INFO Ring 05 : 27 -> 34 [receive] via NET/AWS Libfabric/1 | |
| ip-172-31-38-14:25300:25369 [2] NCCL INFO NET/OFI [2] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-38-14:25301:25375 [3] NCCL INFO Ring 04 : 11[3] -> 15[7] via P2P/IPC | |
| ip-172-31-36-128:22030:22096 [7] NCCL INFO Ring 04 : 119[7] -> 116[4] via P2P/IPC | |
| ip-172-31-36-128:22023:22093 [2] NCCL INFO Ring 05 : 107 -> 114 [receive] via NET/AWS Libfabric/1 | |
| ip-172-31-33-235:21961:22023 [2] NCCL INFO Ring 05 : 99 -> 106 [receive] via NET/AWS Libfabric/1 | |
| ip-172-31-36-188:22214:22275 [7] NCCL INFO Ring 04 : 95[7] -> 92[4] via P2P/IPC | |
| ip-172-31-45-171:25063:25130 [5] NCCL INFO Ring 04 : 29[5] -> 25[1] via P2P/IPC | |
| ip-172-31-39-141:20308:20378 [5] NCCL INFO Ring 04 : 5[5] -> 1[1] via P2P/IPC | |
| ip-172-31-33-235:21966:22034 [7] NCCL INFO Ring 04 : 111[7] -> 108[4] via P2P/IPC | |
| ip-172-31-36-188:22209:22274 [2] NCCL INFO Ring 05 : 83 -> 90 [receive] via NET/AWS Libfabric/1 | |
| ip-172-31-38-82:24982:25054 [4] NCCL INFO Ring 04 : 44[4] -> 46[6] via P2P/IPC | |
| ip-172-31-44-142:24934:25005 [5] NCCL INFO Ring 04 : 21[5] -> 17[1] via P2P/IPC | |
| ip-172-31-46-50:22045:22115 [5] NCCL INFO Ring 04 : 69[5] -> 65[1] via P2P/IPC | |
| ip-172-31-44-4:22497:22559 [5] NCCL INFO Ring 04 : 125[5] -> 121[1] via P2P/IPC | |
| ip-172-31-47-156:22565:22637 [5] NCCL INFO Ring 04 : 85[5] -> 81[1] via P2P/IPC | |
| ip-172-31-38-14:25300:25369 [2] NCCL INFO Ring 05 : 3 -> 10 [receive] via NET/AWS Libfabric/1 | |
| ip-172-31-32-86:24698:24764 [7] NCCL INFO Ring 04 : 55[7] -> 52[4] via P2P/IPC | |
| ip-172-31-44-230:21992:22064 [5] NCCL INFO Ring 04 : 101[5] -> 97[1] via P2P/IPC | |
| ip-172-31-47-240:24694:24764 [5] NCCL INFO Ring 04 : 61[5] -> 57[1] via P2P/IPC | |
| ip-172-31-32-86:24695:24771 [4] NCCL INFO Ring 04 : 52[4] -> 54[6] via P2P/IPC | |
| ip-172-31-36-128:22026:22094 [5] NCCL INFO Ring 04 : 117[5] -> 113[1] via P2P/IPC | |
| ip-172-31-47-188:22013:22084 [5] NCCL INFO Ring 04 : 77[5] -> 73[1] via P2P/IPC | |
| ip-172-31-46-50:22044:22113 [4] NCCL INFO Ring 04 : 68[4] -> 70[6] via P2P/IPC | |
| ip-172-31-45-171:25062:25131 [4] NCCL INFO Ring 04 : 28[4] -> 30[6] via P2P/IPC | |
| ip-172-31-37-141:25829:25898 [5] NCCL INFO Ring 04 : 37[5] -> 33[1] via P2P/IPC | |
| ip-172-31-36-188:22212:22276 [5] NCCL INFO Ring 04 : 93[5] -> 89[1] via P2P/IPC | |
| ip-172-31-39-141:20307:20377 [4] NCCL INFO Ring 04 : 4[4] -> 6[6] via P2P/IPC | |
| ip-172-31-38-14:25306:25370 [7] NCCL INFO Ring 04 : 15[7] -> 12[4] via P2P/IPC | |
| ip-172-31-33-235:21964:22033 [5] NCCL INFO Ring 04 : 109[5] -> 105[1] via P2P/IPC | |
| ip-172-31-38-14:25302:25368 [4] NCCL INFO Ring 04 : 12[4] -> 14[6] via P2P/IPC | |
| ip-172-31-44-142:24933:25001 [4] NCCL INFO Ring 04 : 20[4] -> 22[6] via P2P/IPC | |
| ip-172-31-47-156:22564:22632 [4] NCCL INFO Ring 04 : 84[4] -> 86[6] via P2P/IPC | |
| ip-172-31-44-4:22496:22561 [4] NCCL INFO Ring 04 : 124[4] -> 126[6] via P2P/IPC | |
| ip-172-31-44-230:21991:22062 [4] NCCL INFO Ring 04 : 100[4] -> 102[6] via P2P/IPC | |
| ip-172-31-32-86:24696:24767 [5] NCCL INFO Ring 04 : 53[5] -> 49[1] via P2P/IPC | |
| ip-172-31-47-240:24693:24767 [4] NCCL INFO Ring 04 : 60[4] -> 62[6] via P2P/IPC | |
| ip-172-31-46-50:22042:22112 [2] NCCL INFO Ring 05 : 66[2] -> 64[0] via P2P/IPC | |
| ip-172-31-36-128:22025:22095 [4] NCCL INFO Ring 04 : 116[4] -> 118[6] via P2P/IPC | |
| ip-172-31-33-235:21963:22035 [4] NCCL INFO Ring 04 : 108[4] -> 110[6] via P2P/IPC | |
| ip-172-31-47-188:22012:22080 [4] NCCL INFO Ring 04 : 76[4] -> 78[6] via P2P/IPC | |
| ip-172-31-36-188:22211:22280 [4] NCCL INFO Ring 04 : 92[4] -> 94[6] via P2P/IPC | |
| ip-172-31-37-141:25828:25896 [4] NCCL INFO Ring 04 : 36[4] -> 38[6] via P2P/IPC | |
| ip-172-31-38-14:25303:25373 [5] NCCL INFO Ring 04 : 13[5] -> 9[1] via P2P/IPC | |
| ip-172-31-39-141:20305:20380 [2] NCCL INFO Ring 05 : 2[2] -> 0[0] via P2P/IPC | |
| ip-172-31-38-82:24980:25051 [2] NCCL INFO Ring 05 : 42[2] -> 40[0] via P2P/IPC | |
| ip-172-31-47-156:22562:22633 [2] NCCL INFO Ring 05 : 82[2] -> 80[0] via P2P/IPC | |
| ip-172-31-44-4:22494:22564 [2] NCCL INFO Ring 05 : 122[2] -> 120[0] via P2P/IPC | |
| ip-172-31-47-188:22010:22085 [2] NCCL INFO Ring 05 : 74[2] -> 72[0] via P2P/IPC | |
| ip-172-31-47-240:24691:24761 [2] NCCL INFO Ring 05 : 58[2] -> 56[0] via P2P/IPC | |
| ip-172-31-44-142:24931:25004 [2] NCCL INFO Ring 05 : 18[2] -> 16[0] via P2P/IPC | |
| ip-172-31-45-171:25060:25132 [2] NCCL INFO Ring 05 : 26[2] -> 24[0] via P2P/IPC | |
| ip-172-31-32-86:24693:24766 [2] NCCL INFO Ring 05 : 50[2] -> 48[0] via P2P/IPC | |
| ip-172-31-44-230:21989:22061 [2] NCCL INFO Ring 05 : 98[2] -> 96[0] via P2P/IPC | |
| ip-172-31-37-141:25826:25891 [2] NCCL INFO Ring 05 : 34[2] -> 32[0] via P2P/IPC | |
| ip-172-31-36-128:22023:22093 [2] NCCL INFO Ring 05 : 114[2] -> 112[0] via P2P/IPC | |
| ip-172-31-33-235:21961:22023 [2] NCCL INFO Ring 05 : 106[2] -> 104[0] via P2P/IPC | |
| ip-172-31-36-188:22209:22274 [2] NCCL INFO Ring 05 : 90[2] -> 88[0] via P2P/IPC | |
| ip-172-31-38-14:25300:25369 [2] NCCL INFO Ring 05 : 10[2] -> 8[0] via P2P/IPC | |
| ip-172-31-47-156:22563:22638 [3] NCCL INFO NET/OFI [3] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-38-82:24981:25048 [3] NCCL INFO NET/OFI [3] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-44-4:22495:22558 [3] NCCL INFO NET/OFI [3] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-39-141:20303:20372 [0] NCCL INFO Ring 05 : 0[0] -> 1[1] via P2P/IPC | |
| ip-172-31-39-141:20306:20379 [3] NCCL INFO NET/OFI [3] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-46-50:22043:22118 [3] NCCL INFO NET/OFI [3] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-32-86:24691:24765 [0] NCCL INFO Ring 05 : 48[0] -> 49[1] via P2P/IPC | |
| ip-172-31-36-188:22207:22281 [0] NCCL INFO Ring 05 : 88[0] -> 89[1] via P2P/IPC | |
| ip-172-31-47-156:22563:22638 [3] NCCL INFO Ring 05 : 83 -> 90 [send] via NET/AWS Libfabric/1 | |
| ip-172-31-38-82:24981:25048 [3] NCCL INFO Ring 05 : 43 -> 50 [send] via NET/AWS Libfabric/1 | |
| ip-172-31-44-4:22495:22558 [3] NCCL INFO Ring 05 : 123 -> 2 [send] via NET/AWS Libfabric/1 | |
| ip-172-31-47-188:22008:22081 [0] NCCL INFO Ring 05 : 72[0] -> 73[1] via P2P/IPC | |
| ip-172-31-46-50:22040:22119 [0] NCCL INFO Ring 05 : 64[0] -> 65[1] via P2P/IPC | |
| ip-172-31-39-141:20306:20379 [3] NCCL INFO Ring 05 : 3 -> 10 [send] via NET/AWS Libfabric/1 | |
| ip-172-31-47-240:24692:24760 [3] NCCL INFO NET/OFI [3] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-46-50:22043:22118 [3] NCCL INFO Ring 05 : 67 -> 74 [send] via NET/AWS Libfabric/1 | |
| ip-172-31-38-14:25298:25374 [0] NCCL INFO Ring 05 : 8[0] -> 9[1] via P2P/IPC | |
| ip-172-31-44-142:24932:25006 [3] NCCL INFO NET/OFI [3] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-45-171:25061:25134 [3] NCCL INFO NET/OFI [3] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-37-141:25824:25897 [0] NCCL INFO Ring 05 : 32[0] -> 33[1] via P2P/IPC | |
| ip-172-31-45-171:25058:25128 [0] NCCL INFO Ring 05 : 24[0] -> 25[1] via P2P/IPC | |
| ip-172-31-38-82:24978:25047 [0] NCCL INFO Ring 05 : 40[0] -> 41[1] via P2P/IPC | |
| ip-172-31-47-156:22560:22634 [0] NCCL INFO Ring 05 : 80[0] -> 81[1] via P2P/IPC | |
| ip-172-31-47-240:24692:24760 [3] NCCL INFO Ring 05 : 59 -> 66 [send] via NET/AWS Libfabric/1 | |
| ip-172-31-44-4:22492:22563 [0] NCCL INFO Ring 05 : 120[0] -> 121[1] via P2P/IPC | |
| ip-172-31-44-230:21990:22065 [3] NCCL INFO NET/OFI [3] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-47-240:24689:24766 [0] NCCL INFO Ring 05 : 56[0] -> 57[1] via P2P/IPC | |
| ip-172-31-45-171:25061:25134 [3] NCCL INFO Ring 05 : 27 -> 34 [send] via NET/AWS Libfabric/1 | |
| ip-172-31-44-142:24932:25006 [3] NCCL INFO Ring 05 : 19 -> 26 [send] via NET/AWS Libfabric/1 | |
| ip-172-31-33-235:21959:22032 [0] NCCL INFO Ring 05 : 104[0] -> 105[1] via P2P/IPC | |
| ip-172-31-36-128:22021:22098 [0] NCCL INFO Ring 05 : 112[0] -> 113[1] via P2P/IPC | |
| ip-172-31-38-82:24985:25053 [7] NCCL INFO Ring 05 : 47[7] -> 43[3] via P2P/IPC | |
| ip-172-31-47-188:22011:22083 [3] NCCL INFO NET/OFI [3] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-37-141:25827:25893 [3] NCCL INFO NET/OFI [3] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-44-230:21987:22059 [0] NCCL INFO Ring 05 : 96[0] -> 97[1] via P2P/IPC | |
| ip-172-31-36-128:22024:22091 [3] NCCL INFO NET/OFI [3] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-44-230:21990:22065 [3] NCCL INFO Ring 05 : 99 -> 106 [send] via NET/AWS Libfabric/1 | |
| ip-172-31-47-156:22569:22639 [7] NCCL INFO Ring 05 : 87[7] -> 83[3] via P2P/IPC | |
| ip-172-31-39-141:20310:20383 [7] NCCL INFO Ring 05 : 7[7] -> 3[3] via P2P/IPC | |
| ip-172-31-44-142:24937:25007 [7] NCCL INFO Ring 05 : 23[7] -> 19[3] via P2P/IPC | |
| ip-172-31-47-188:22011:22083 [3] NCCL INFO Ring 05 : 75 -> 82 [send] via NET/AWS Libfabric/1 | |
| ip-172-31-44-4:22499:22562 [7] NCCL INFO Ring 05 : 127[7] -> 123[3] via P2P/IPC | |
| ip-172-31-32-86:24694:24770 [3] NCCL INFO NET/OFI [3] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-33-235:21962:22036 [3] NCCL INFO NET/OFI [3] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-37-141:25827:25893 [3] NCCL INFO Ring 05 : 35 -> 42 [send] via NET/AWS Libfabric/1 | |
| ip-172-31-36-128:22024:22091 [3] NCCL INFO Ring 05 : 115 -> 122 [send] via NET/AWS Libfabric/1 | |
| ip-172-31-36-188:22210:22277 [3] NCCL INFO NET/OFI [3] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-38-82:24979:25052 [1] NCCL INFO Ring 05 : 41[1] -> 45[5] via P2P/IPC | |
| ip-172-31-44-142:24929:25002 [0] NCCL INFO Ring 05 : 16[0] -> 17[1] via P2P/IPC | |
| ip-172-31-47-240:24696:24763 [7] NCCL INFO Ring 05 : 63[7] -> 59[3] via P2P/IPC | |
| ip-172-31-32-86:24694:24770 [3] NCCL INFO Ring 05 : 51 -> 58 [send] via NET/AWS Libfabric/1 | |
| ip-172-31-44-230:21994:22063 [7] NCCL INFO Ring 05 : 103[7] -> 99[3] via P2P/IPC | |
| ip-172-31-33-235:21962:22036 [3] NCCL INFO Ring 05 : 107 -> 114 [send] via NET/AWS Libfabric/1 | |
| ip-172-31-47-156:22561:22636 [1] NCCL INFO Ring 05 : 81[1] -> 85[5] via P2P/IPC | |
| ip-172-31-36-188:22210:22277 [3] NCCL INFO Ring 05 : 91 -> 98 [send] via NET/AWS Libfabric/1 | |
| ip-172-31-44-142:24930:25003 [1] NCCL INFO Ring 05 : 17[1] -> 21[5] via P2P/IPC | |
| ip-172-31-46-50:22047:22114 [7] NCCL INFO Ring 05 : 71[7] -> 67[3] via P2P/IPC | |
| ip-172-31-45-171:25065:25133 [7] NCCL INFO Ring 05 : 31[7] -> 27[3] via P2P/IPC | |
| ip-172-31-45-171:25059:25135 [1] NCCL INFO Ring 05 : 25[1] -> 29[5] via P2P/IPC | |
| ip-172-31-38-82:24984:25049 [6] NCCL INFO Ring 05 : 46[6] -> 44[4] via P2P/IPC | |
| ip-172-31-44-4:22493:22565 [1] NCCL INFO Ring 05 : 121[1] -> 125[5] via P2P/IPC | |
| ip-172-31-38-82:24983:25050 [5] NCCL INFO Ring 05 : 45[5] -> 46[6] via P2P/IPC | |
| ip-172-31-46-50:22041:22117 [1] NCCL INFO Ring 05 : 65[1] -> 69[5] via P2P/IPC | |
| ip-172-31-47-188:22015:22086 [7] NCCL INFO Ring 05 : 79[7] -> 75[3] via P2P/IPC | |
| ip-172-31-39-141:20304:20381 [1] NCCL INFO Ring 05 : 1[1] -> 5[5] via P2P/IPC | |
| ip-172-31-47-240:24690:24765 [1] NCCL INFO Ring 05 : 57[1] -> 61[5] via P2P/IPC | |
| ip-172-31-37-141:25831:25892 [7] NCCL INFO Ring 05 : 39[7] -> 35[3] via P2P/IPC | |
| ip-172-31-44-230:21988:22060 [1] NCCL INFO Ring 05 : 97[1] -> 101[5] via P2P/IPC | |
| ip-172-31-36-128:22030:22096 [7] NCCL INFO Ring 05 : 119[7] -> 115[3] via P2P/IPC | |
| ip-172-31-36-128:22022:22097 [1] NCCL INFO Ring 05 : 113[1] -> 117[5] via P2P/IPC | |
| ip-172-31-46-50:22046:22116 [6] NCCL INFO Ring 05 : 70[6] -> 68[4] via P2P/IPC | |
| ip-172-31-44-142:24934:25005 [5] NCCL INFO Ring 05 : 21[5] -> 22[6] via P2P/IPC | |
| ip-172-31-44-142:24935:25000 [6] NCCL INFO Ring 05 : 22[6] -> 20[4] via P2P/IPC | |
| ip-172-31-47-156:22566:22635 [6] NCCL INFO Ring 05 : 86[6] -> 84[4] via P2P/IPC | |
| ip-172-31-47-156:22565:22637 [5] NCCL INFO Ring 05 : 85[5] -> 86[6] via P2P/IPC | |
| ip-172-31-32-86:24697:24768 [6] NCCL INFO Ring 05 : 54[6] -> 52[4] via P2P/IPC | |
| ip-172-31-47-188:22009:22082 [1] NCCL INFO Ring 05 : 73[1] -> 77[5] via P2P/IPC | |
| ip-172-31-44-4:22498:22560 [6] NCCL INFO Ring 05 : 126[6] -> 124[4] via P2P/IPC | |
| ip-172-31-38-82:24982:25054 [4] NCCL INFO Ring 05 : 44[4] -> 47[7] via P2P/IPC | |
| ip-172-31-45-171:25064:25129 [6] NCCL INFO Ring 05 : 30[6] -> 28[4] via P2P/IPC | |
| ip-172-31-44-4:22497:22559 [5] NCCL INFO Ring 05 : 125[5] -> 126[6] via P2P/IPC | |
| ip-172-31-39-141:20309:20382 [6] NCCL INFO Ring 05 : 6[6] -> 4[4] via P2P/IPC | |
| ip-172-31-45-171:25063:25130 [5] NCCL INFO Ring 05 : 29[5] -> 30[6] via P2P/IPC | |
| ip-172-31-39-141:20308:20378 [5] NCCL INFO Ring 05 : 5[5] -> 6[6] via P2P/IPC | |
| ip-172-31-47-240:24695:24762 [6] NCCL INFO Ring 05 : 62[6] -> 60[4] via P2P/IPC | |
| ip-172-31-47-240:24694:24764 [5] NCCL INFO Ring 05 : 61[5] -> 62[6] via P2P/IPC | |
| ip-172-31-46-50:22045:22115 [5] NCCL INFO Ring 05 : 69[5] -> 70[6] via P2P/IPC | |
| ip-172-31-36-188:22214:22275 [7] NCCL INFO Ring 05 : 95[7] -> 91[3] via P2P/IPC | |
| ip-172-31-38-14:25301:25375 [3] NCCL INFO NET/OFI [3] getCudaPath dev 1 busId 0000:00:17.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-37-141:25825:25894 [1] NCCL INFO Ring 05 : 33[1] -> 37[5] via P2P/IPC | |
| ip-172-31-36-188:22208:22278 [1] NCCL INFO Ring 05 : 89[1] -> 93[5] via P2P/IPC | |
| ip-172-31-33-235:21966:22034 [7] NCCL INFO Ring 05 : 111[7] -> 107[3] via P2P/IPC | |
| ip-172-31-44-230:21993:22066 [6] NCCL INFO Ring 05 : 102[6] -> 100[4] via P2P/IPC | |
| ip-172-31-44-230:21992:22064 [5] NCCL INFO Ring 05 : 101[5] -> 102[6] via P2P/IPC | |
| ip-172-31-44-142:24933:25001 [4] NCCL INFO Ring 05 : 20[4] -> 23[7] via P2P/IPC | |
| ip-172-31-33-235:21960:22031 [1] NCCL INFO Ring 05 : 105[1] -> 109[5] via P2P/IPC | |
| ip-172-31-32-86:24698:24764 [7] NCCL INFO Ring 05 : 55[7] -> 51[3] via P2P/IPC | |
| ip-172-31-32-86:24692:24769 [1] NCCL INFO Ring 05 : 49[1] -> 53[5] via P2P/IPC | |
| ip-172-31-46-50:22044:22113 [4] NCCL INFO Ring 05 : 68[4] -> 71[7] via P2P/IPC | |
| ip-172-31-36-128:22027:22092 [6] NCCL INFO Ring 05 : 118[6] -> 116[4] via P2P/IPC | |
| ip-172-31-36-128:22026:22094 [5] NCCL INFO Ring 05 : 117[5] -> 118[6] via P2P/IPC | |
| ip-172-31-47-156:22564:22632 [4] NCCL INFO Ring 05 : 84[4] -> 87[7] via P2P/IPC | |
| ip-172-31-32-86:24695:24771 [4] NCCL INFO Ring 05 : 52[4] -> 55[7] via P2P/IPC | |
| ip-172-31-47-188:22013:22084 [5] NCCL INFO Ring 05 : 77[5] -> 78[6] via P2P/IPC | |
| ip-172-31-47-188:22014:22087 [6] NCCL INFO Ring 05 : 78[6] -> 76[4] via P2P/IPC | |
| ip-172-31-45-171:25062:25131 [4] NCCL INFO Ring 05 : 28[4] -> 31[7] via P2P/IPC | |
| ip-172-31-44-4:22496:22561 [4] NCCL INFO Ring 05 : 124[4] -> 127[7] via P2P/IPC | |
| ip-172-31-39-141:20307:20377 [4] NCCL INFO Ring 05 : 4[4] -> 7[7] via P2P/IPC | |
| ip-172-31-38-14:25304:25371 [6] NCCL INFO Ring 05 : 14[6] -> 12[4] via P2P/IPC | |
| ip-172-31-47-240:24693:24767 [4] NCCL INFO Ring 05 : 60[4] -> 63[7] via P2P/IPC | |
| ip-172-31-38-14:25301:25375 [3] NCCL INFO Ring 05 : 11 -> 18 [send] via NET/AWS Libfabric/1 | |
| ip-172-31-33-235:21965:22037 [6] NCCL INFO Ring 05 : 110[6] -> 108[4] via P2P/IPC | |
| ip-172-31-38-82:24982:25054 [4] NCCL INFO NET/OFI [4] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-37-141:25830:25895 [6] NCCL INFO Ring 05 : 38[6] -> 36[4] via P2P/IPC | |
| ip-172-31-37-141:25829:25898 [5] NCCL INFO Ring 05 : 37[5] -> 38[6] via P2P/IPC | |
| ip-172-31-36-188:22213:22279 [6] NCCL INFO Ring 05 : 94[6] -> 92[4] via P2P/IPC | |
| ip-172-31-36-188:22212:22276 [5] NCCL INFO Ring 05 : 93[5] -> 94[6] via P2P/IPC | |
| ip-172-31-44-230:21991:22062 [4] NCCL INFO Ring 05 : 100[4] -> 103[7] via P2P/IPC | |
| ip-172-31-33-235:21964:22033 [5] NCCL INFO Ring 05 : 109[5] -> 110[6] via P2P/IPC | |
| ip-172-31-44-142:24933:25001 [4] NCCL INFO NET/OFI [4] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-36-128:22025:22095 [4] NCCL INFO Ring 05 : 116[4] -> 119[7] via P2P/IPC | |
| ip-172-31-32-86:24696:24767 [5] NCCL INFO Ring 05 : 53[5] -> 54[6] via P2P/IPC | |
| ip-172-31-38-14:25306:25370 [7] NCCL INFO Ring 05 : 15[7] -> 11[3] via P2P/IPC | |
| ip-172-31-47-188:22012:22080 [4] NCCL INFO Ring 05 : 76[4] -> 79[7] via P2P/IPC | |
| ip-172-31-38-14:25299:25372 [1] NCCL INFO Ring 05 : 9[1] -> 13[5] via P2P/IPC | |
| ip-172-31-46-50:22044:22113 [4] NCCL INFO NET/OFI [4] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-32-86:24695:24771 [4] NCCL INFO NET/OFI [4] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-47-156:22564:22632 [4] NCCL INFO NET/OFI [4] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-38-14:25302:25368 [4] NCCL INFO Ring 05 : 12[4] -> 15[7] via P2P/IPC | |
| ip-172-31-38-82:24982:25054 [4] NCCL INFO Ring 06 : 37 -> 44 [receive] via NET/AWS Libfabric/2 | |
| ip-172-31-45-171:25062:25131 [4] NCCL INFO NET/OFI [4] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-37-141:25828:25896 [4] NCCL INFO Ring 05 : 36[4] -> 39[7] via P2P/IPC | |
| ip-172-31-33-235:21963:22035 [4] NCCL INFO Ring 05 : 108[4] -> 111[7] via P2P/IPC | |
| ip-172-31-44-4:22496:22561 [4] NCCL INFO NET/OFI [4] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-39-141:20307:20377 [4] NCCL INFO NET/OFI [4] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-36-188:22211:22280 [4] NCCL INFO Ring 05 : 92[4] -> 95[7] via P2P/IPC | |
| ip-172-31-47-240:24693:24767 [4] NCCL INFO NET/OFI [4] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-44-142:24933:25001 [4] NCCL INFO Ring 06 : 13 -> 20 [receive] via NET/AWS Libfabric/2 | |
| ip-172-31-32-86:24695:24771 [4] NCCL INFO Ring 06 : 45 -> 52 [receive] via NET/AWS Libfabric/2 | |
| ip-172-31-44-230:21991:22062 [4] NCCL INFO NET/OFI [4] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-46-50:22044:22113 [4] NCCL INFO Ring 06 : 61 -> 68 [receive] via NET/AWS Libfabric/2 | |
| ip-172-31-38-14:25303:25373 [5] NCCL INFO Ring 05 : 13[5] -> 14[6] via P2P/IPC | |
| ip-172-31-47-156:22564:22632 [4] NCCL INFO Ring 06 : 77 -> 84 [receive] via NET/AWS Libfabric/2 | |
| ip-172-31-45-171:25062:25131 [4] NCCL INFO Ring 06 : 21 -> 28 [receive] via NET/AWS Libfabric/2 | |
| ip-172-31-36-128:22025:22095 [4] NCCL INFO NET/OFI [4] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-47-188:22012:22080 [4] NCCL INFO NET/OFI [4] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-44-4:22496:22561 [4] NCCL INFO Ring 06 : 117 -> 124 [receive] via NET/AWS Libfabric/2 | |
| ip-172-31-39-141:20307:20377 [4] NCCL INFO Ring 06 : 125 -> 4 [receive] via NET/AWS Libfabric/2 | |
| ip-172-31-38-14:25302:25368 [4] NCCL INFO NET/OFI [4] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-47-240:24693:24767 [4] NCCL INFO Ring 06 : 53 -> 60 [receive] via NET/AWS Libfabric/2 | |
| ip-172-31-33-235:21963:22035 [4] NCCL INFO NET/OFI [4] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-37-141:25828:25896 [4] NCCL INFO NET/OFI [4] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-44-230:21991:22062 [4] NCCL INFO Ring 06 : 93 -> 100 [receive] via NET/AWS Libfabric/2 | |
| ip-172-31-36-188:22211:22280 [4] NCCL INFO NET/OFI [4] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-36-128:22025:22095 [4] NCCL INFO Ring 06 : 109 -> 116 [receive] via NET/AWS Libfabric/2 | |
| ip-172-31-47-188:22012:22080 [4] NCCL INFO Ring 06 : 69 -> 76 [receive] via NET/AWS Libfabric/2 | |
| ip-172-31-38-14:25302:25368 [4] NCCL INFO Ring 06 : 5 -> 12 [receive] via NET/AWS Libfabric/2 | |
| ip-172-31-33-235:21963:22035 [4] NCCL INFO Ring 06 : 101 -> 108 [receive] via NET/AWS Libfabric/2 | |
| ip-172-31-37-141:25828:25896 [4] NCCL INFO Ring 06 : 29 -> 36 [receive] via NET/AWS Libfabric/2 | |
| ip-172-31-36-188:22211:22280 [4] NCCL INFO Ring 06 : 85 -> 92 [receive] via NET/AWS Libfabric/2 | |
| ip-172-31-36-188:22209:22274 [2] NCCL INFO Ring 06 : 90[2] -> 94[6] via P2P/IPC | |
| ip-172-31-32-86:24693:24766 [2] NCCL INFO Ring 06 : 50[2] -> 54[6] via P2P/IPC | |
| ip-172-31-39-141:20305:20380 [2] NCCL INFO Ring 06 : 2[2] -> 6[6] via P2P/IPC | |
| ip-172-31-47-188:22010:22085 [2] NCCL INFO Ring 06 : 74[2] -> 78[6] via P2P/IPC | |
| ip-172-31-39-141:20303:20372 [0] NCCL INFO Ring 06 : 0[0] -> 3[3] via P2P/IPC | |
| ip-172-31-38-14:25300:25369 [2] NCCL INFO Ring 06 : 10[2] -> 14[6] via P2P/IPC | |
| ip-172-31-47-188:22008:22081 [0] NCCL INFO Ring 06 : 72[0] -> 75[3] via P2P/IPC | |
| ip-172-31-46-50:22042:22112 [2] NCCL INFO Ring 06 : 66[2] -> 70[6] via P2P/IPC | |
| ip-172-31-46-50:22040:22119 [0] NCCL INFO Ring 06 : 64[0] -> 67[3] via P2P/IPC | |
| ip-172-31-36-188:22207:22281 [0] NCCL INFO Ring 06 : 88[0] -> 91[3] via P2P/IPC | |
| ip-172-31-37-141:25826:25891 [2] NCCL INFO Ring 06 : 34[2] -> 38[6] via P2P/IPC | |
| ip-172-31-45-171:25060:25132 [2] NCCL INFO Ring 06 : 26[2] -> 30[6] via P2P/IPC | |
| ip-172-31-38-82:24981:25048 [3] NCCL INFO Ring 06 : 43[3] -> 41[1] via P2P/IPC | |
| ip-172-31-44-142:24932:25006 [3] NCCL INFO Ring 06 : 19[3] -> 17[1] via P2P/IPC | |
| ip-172-31-37-141:25824:25897 [0] NCCL INFO Ring 06 : 32[0] -> 35[3] via P2P/IPC | |
| ip-172-31-32-86:24691:24765 [0] NCCL INFO Ring 06 : 48[0] -> 51[3] via P2P/IPC | |
| ip-172-31-47-240:24689:24766 [0] NCCL INFO Ring 06 : 56[0] -> 59[3] via P2P/IPC | |
| ip-172-31-47-156:22560:22634 [0] NCCL INFO Ring 06 : 80[0] -> 83[3] via P2P/IPC | |
| ip-172-31-44-4:22492:22563 [0] NCCL INFO Ring 06 : 120[0] -> 123[3] via P2P/IPC | |
| ip-172-31-47-156:22563:22638 [3] NCCL INFO Ring 06 : 83[3] -> 81[1] via P2P/IPC | |
| ip-172-31-38-82:24982:25054 [4] NCCL INFO Ring 06 : 44[4] -> 40[0] via P2P/IPC | |
| ip-172-31-38-82:24978:25047 [0] NCCL INFO Ring 06 : 40[0] -> 43[3] via P2P/IPC | |
| ip-172-31-45-171:25058:25128 [0] NCCL INFO Ring 06 : 24[0] -> 27[3] via P2P/IPC | |
| ip-172-31-33-235:21961:22023 [2] NCCL INFO Ring 06 : 106[2] -> 110[6] via P2P/IPC | |
| ip-172-31-39-141:20306:20379 [3] NCCL INFO Ring 06 : 3[3] -> 1[1] via P2P/IPC | |
| ip-172-31-44-142:24933:25001 [4] NCCL INFO Ring 06 : 20[4] -> 16[0] via P2P/IPC | |
| ip-172-31-32-86:24695:24771 [4] NCCL INFO Ring 06 : 52[4] -> 48[0] via P2P/IPC | |
| ip-172-31-47-156:22562:22633 [2] NCCL INFO Ring 06 : 82[2] -> 86[6] via P2P/IPC | |
| ip-172-31-36-128:22021:22098 [0] NCCL INFO Ring 06 : 112[0] -> 115[3] via P2P/IPC | |
| ip-172-31-33-235:21959:22032 [0] NCCL INFO Ring 06 : 104[0] -> 107[3] via P2P/IPC | |
| ip-172-31-44-4:22495:22558 [3] NCCL INFO Ring 06 : 123[3] -> 121[1] via P2P/IPC | |
| ip-172-31-44-230:21987:22059 [0] NCCL INFO Ring 06 : 96[0] -> 99[3] via P2P/IPC | |
| ip-172-31-39-141:20307:20377 [4] NCCL INFO Ring 06 : 4[4] -> 0[0] via P2P/IPC | |
| ip-172-31-44-4:22494:22564 [2] NCCL INFO Ring 06 : 122[2] -> 126[6] via P2P/IPC | |
| ip-172-31-46-50:22044:22113 [4] NCCL INFO Ring 06 : 68[4] -> 64[0] via P2P/IPC | |
| ip-172-31-38-14:25298:25374 [0] NCCL INFO Ring 06 : 8[0] -> 11[3] via P2P/IPC | |
| ip-172-31-47-156:22564:22632 [4] NCCL INFO Ring 06 : 84[4] -> 80[0] via P2P/IPC | |
| ip-172-31-38-82:24980:25051 [2] NCCL INFO Ring 06 : 42[2] -> 46[6] via P2P/IPC | |
| ip-172-31-45-171:25062:25131 [4] NCCL INFO Ring 06 : 28[4] -> 24[0] via P2P/IPC | |
| ip-172-31-47-240:24692:24760 [3] NCCL INFO Ring 06 : 59[3] -> 57[1] via P2P/IPC | |
| ip-172-31-47-240:24693:24767 [4] NCCL INFO Ring 06 : 60[4] -> 56[0] via P2P/IPC | |
| ip-172-31-44-4:22496:22561 [4] NCCL INFO Ring 06 : 124[4] -> 120[0] via P2P/IPC | |
| ip-172-31-47-188:22012:22080 [4] NCCL INFO Ring 06 : 76[4] -> 72[0] via P2P/IPC | |
| ip-172-31-44-142:24929:25002 [0] NCCL INFO Ring 06 : 16[0] -> 19[3] via P2P/IPC | |
| ip-172-31-47-188:22011:22083 [3] NCCL INFO Ring 06 : 75[3] -> 73[1] via P2P/IPC | |
| ip-172-31-47-240:24691:24761 [2] NCCL INFO Ring 06 : 58[2] -> 62[6] via P2P/IPC | |
| ip-172-31-44-230:21990:22065 [3] NCCL INFO Ring 06 : 99[3] -> 97[1] via P2P/IPC | |
| ip-172-31-44-230:21991:22062 [4] NCCL INFO Ring 06 : 100[4] -> 96[0] via P2P/IPC | |
| ip-172-31-38-14:25302:25368 [4] NCCL INFO Ring 06 : 12[4] -> 8[0] via P2P/IPC | |
| ip-172-31-36-188:22211:22280 [4] NCCL INFO Ring 06 : 92[4] -> 88[0] via P2P/IPC | |
| ip-172-31-36-128:22023:22093 [2] NCCL INFO Ring 06 : 114[2] -> 118[6] via P2P/IPC | |
| ip-172-31-46-50:22043:22118 [3] NCCL INFO Ring 06 : 67[3] -> 65[1] via P2P/IPC | |
| ip-172-31-45-171:25061:25134 [3] NCCL INFO Ring 06 : 27[3] -> 25[1] via P2P/IPC | |
| ip-172-31-36-128:22025:22095 [4] NCCL INFO Ring 06 : 116[4] -> 112[0] via P2P/IPC | |
| ip-172-31-44-230:21989:22061 [2] NCCL INFO Ring 06 : 98[2] -> 102[6] via P2P/IPC | |
| ip-172-31-37-141:25827:25893 [3] NCCL INFO Ring 06 : 35[3] -> 33[1] via P2P/IPC | |
| ip-172-31-37-141:25828:25896 [4] NCCL INFO Ring 06 : 36[4] -> 32[0] via P2P/IPC | |
| ip-172-31-38-82:24979:25052 [1] NCCL INFO Ring 06 : 41[1] -> 42[2] via P2P/IPC | |
| ip-172-31-44-142:24930:25003 [1] NCCL INFO Ring 06 : 17[1] -> 18[2] via P2P/IPC | |
| ip-172-31-33-235:21963:22035 [4] NCCL INFO Ring 06 : 108[4] -> 104[0] via P2P/IPC | |
| ip-172-31-36-188:22210:22277 [3] NCCL INFO Ring 06 : 91[3] -> 89[1] via P2P/IPC | |
| ip-172-31-39-141:20304:20381 [1] NCCL INFO Ring 06 : 1[1] -> 2[2] via P2P/IPC | |
| ip-172-31-47-156:22561:22636 [1] NCCL INFO Ring 06 : 81[1] -> 82[2] via P2P/IPC | |
| ip-172-31-36-128:22024:22091 [3] NCCL INFO Ring 06 : 115[3] -> 113[1] via P2P/IPC | |
| ip-172-31-32-86:24694:24770 [3] NCCL INFO Ring 06 : 51[3] -> 49[1] via P2P/IPC | |
| ip-172-31-44-4:22493:22565 [1] NCCL INFO Ring 06 : 121[1] -> 122[2] via P2P/IPC | |
| ip-172-31-46-50:22041:22117 [1] NCCL INFO Ring 06 : 65[1] -> 66[2] via P2P/IPC | |
| ip-172-31-45-171:25059:25135 [1] NCCL INFO Ring 06 : 25[1] -> 26[2] via P2P/IPC | |
| ip-172-31-47-240:24690:24765 [1] NCCL INFO Ring 06 : 57[1] -> 58[2] via P2P/IPC | |
| ip-172-31-38-82:24985:25053 [7] NCCL INFO Ring 06 : 47[7] -> 45[5] via P2P/IPC | |
| ip-172-31-47-188:22009:22082 [1] NCCL INFO Ring 06 : 73[1] -> 74[2] via P2P/IPC | |
| ip-172-31-38-82:24984:25049 [6] NCCL INFO Ring 06 : 46[6] -> 47[7] via P2P/IPC | |
| ip-172-31-38-82:24983:25050 [5] NCCL INFO NET/OFI [5] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-44-142:24937:25007 [7] NCCL INFO Ring 06 : 23[7] -> 21[5] via P2P/IPC | |
| ip-172-31-33-235:21962:22036 [3] NCCL INFO Ring 06 : 107[3] -> 105[1] via P2P/IPC | |
| ip-172-31-44-142:24935:25000 [6] NCCL INFO Ring 06 : 22[6] -> 23[7] via P2P/IPC | |
| ip-172-31-44-142:24934:25005 [5] NCCL INFO NET/OFI [5] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-46-50:22047:22114 [7] NCCL INFO Ring 06 : 71[7] -> 69[5] via P2P/IPC | |
| ip-172-31-46-50:22046:22116 [6] NCCL INFO Ring 06 : 70[6] -> 71[7] via P2P/IPC | |
| ip-172-31-36-188:22208:22278 [1] NCCL INFO Ring 06 : 89[1] -> 90[2] via P2P/IPC | |
| ip-172-31-39-141:20310:20383 [7] NCCL INFO Ring 06 : 7[7] -> 5[5] via P2P/IPC | |
| ip-172-31-44-230:21988:22060 [1] NCCL INFO Ring 06 : 97[1] -> 98[2] via P2P/IPC | |
| ip-172-31-39-141:20309:20382 [6] NCCL INFO Ring 06 : 6[6] -> 7[7] via P2P/IPC | |
| ip-172-31-47-156:22569:22639 [7] NCCL INFO Ring 06 : 87[7] -> 85[5] via P2P/IPC | |
| ip-172-31-47-156:22566:22635 [6] NCCL INFO Ring 06 : 86[6] -> 87[7] via P2P/IPC | |
| ip-172-31-39-141:20308:20378 [5] NCCL INFO NET/OFI [5] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-45-171:25065:25133 [7] NCCL INFO Ring 06 : 31[7] -> 29[5] via P2P/IPC | |
| ip-172-31-47-156:22565:22637 [5] NCCL INFO NET/OFI [5] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-45-171:25064:25129 [6] NCCL INFO Ring 06 : 30[6] -> 31[7] via P2P/IPC | |
| ip-172-31-32-86:24698:24764 [7] NCCL INFO Ring 06 : 55[7] -> 53[5] via P2P/IPC | |
| ip-172-31-32-86:24692:24769 [1] NCCL INFO Ring 06 : 49[1] -> 50[2] via P2P/IPC | |
| ip-172-31-32-86:24697:24768 [6] NCCL INFO Ring 06 : 54[6] -> 55[7] via P2P/IPC | |
| ip-172-31-44-4:22499:22562 [7] NCCL INFO Ring 06 : 127[7] -> 125[5] via P2P/IPC | |
| ip-172-31-37-141:25825:25894 [1] NCCL INFO Ring 06 : 33[1] -> 34[2] via P2P/IPC | |
| ip-172-31-44-4:22498:22560 [6] NCCL INFO Ring 06 : 126[6] -> 127[7] via P2P/IPC | |
| ip-172-31-44-142:24931:25004 [2] NCCL INFO Ring 06 : 18[2] -> 22[6] via P2P/IPC | |
| ip-172-31-38-14:25301:25375 [3] NCCL INFO Ring 06 : 11[3] -> 9[1] via P2P/IPC | |
| ip-172-31-36-128:22022:22097 [1] NCCL INFO Ring 06 : 113[1] -> 114[2] via P2P/IPC | |
| ip-172-31-47-188:22015:22086 [7] NCCL INFO Ring 06 : 79[7] -> 77[5] via P2P/IPC | |
| ip-172-31-47-240:24696:24763 [7] NCCL INFO Ring 06 : 63[7] -> 61[5] via P2P/IPC | |
| ip-172-31-47-240:24695:24762 [6] NCCL INFO Ring 06 : 62[6] -> 63[7] via P2P/IPC | |
| ip-172-31-44-4:22497:22559 [5] NCCL INFO NET/OFI [5] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-46-50:22045:22115 [5] NCCL INFO NET/OFI [5] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-38-82:24983:25050 [5] NCCL INFO Ring 06 : 45 -> 52 [send] via NET/AWS Libfabric/2 | |
| ip-172-31-47-188:22014:22087 [6] NCCL INFO Ring 06 : 78[6] -> 79[7] via P2P/IPC | |
| ip-172-31-44-142:24934:25005 [5] NCCL INFO Ring 06 : 21 -> 28 [send] via NET/AWS Libfabric/2 | |
| ip-172-31-45-171:25063:25130 [5] NCCL INFO NET/OFI [5] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-47-188:22013:22084 [5] NCCL INFO NET/OFI [5] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-47-240:24694:24764 [5] NCCL INFO NET/OFI [5] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-44-142:24935:25000 [6] NCCL INFO NET/OFI [6] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-38-82:24984:25049 [6] NCCL INFO NET/OFI [6] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-33-235:21960:22031 [1] NCCL INFO Ring 06 : 105[1] -> 106[2] via P2P/IPC | |
| ip-172-31-36-188:22214:22275 [7] NCCL INFO Ring 06 : 95[7] -> 93[5] via P2P/IPC | |
| ip-172-31-39-141:20308:20378 [5] NCCL INFO Ring 06 : 5 -> 12 [send] via NET/AWS Libfabric/2 | |
| ip-172-31-36-188:22213:22279 [6] NCCL INFO Ring 06 : 94[6] -> 95[7] via P2P/IPC | |
| ip-172-31-44-230:21994:22063 [7] NCCL INFO Ring 06 : 103[7] -> 101[5] via P2P/IPC | |
| ip-172-31-44-230:21993:22066 [6] NCCL INFO Ring 06 : 102[6] -> 103[7] via P2P/IPC | |
| ip-172-31-36-188:22212:22276 [5] NCCL INFO NET/OFI [5] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-47-156:22565:22637 [5] NCCL INFO Ring 06 : 85 -> 92 [send] via NET/AWS Libfabric/2 | |
| ip-172-31-46-50:22046:22116 [6] NCCL INFO NET/OFI [6] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-39-141:20309:20382 [6] NCCL INFO NET/OFI [6] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-44-230:21992:22064 [5] NCCL INFO NET/OFI [5] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-47-156:22566:22635 [6] NCCL INFO NET/OFI [6] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-37-141:25831:25892 [7] NCCL INFO Ring 06 : 39[7] -> 37[5] via P2P/IPC | |
| ip-172-31-38-14:25306:25370 [7] NCCL INFO Ring 06 : 15[7] -> 13[5] via P2P/IPC | |
| ip-172-31-37-141:25830:25895 [6] NCCL INFO Ring 06 : 38[6] -> 39[7] via P2P/IPC | |
| ip-172-31-38-14:25304:25371 [6] NCCL INFO Ring 06 : 14[6] -> 15[7] via P2P/IPC | |
| ip-172-31-44-4:22497:22559 [5] NCCL INFO Ring 06 : 125 -> 4 [send] via NET/AWS Libfabric/2 | |
| ip-172-31-46-50:22045:22115 [5] NCCL INFO Ring 06 : 69 -> 76 [send] via NET/AWS Libfabric/2 | |
| ip-172-31-36-128:22030:22096 [7] NCCL INFO Ring 06 : 119[7] -> 117[5] via P2P/IPC | |
| ip-172-31-38-14:25299:25372 [1] NCCL INFO Ring 06 : 9[1] -> 10[2] via P2P/IPC | |
| ip-172-31-33-235:21965:22037 [6] NCCL INFO Ring 06 : 110[6] -> 111[7] via P2P/IPC | |
| ip-172-31-36-128:22027:22092 [6] NCCL INFO Ring 06 : 118[6] -> 119[7] via P2P/IPC | |
| ip-172-31-33-235:21966:22034 [7] NCCL INFO Ring 06 : 111[7] -> 109[5] via P2P/IPC | |
| ip-172-31-32-86:24696:24767 [5] NCCL INFO NET/OFI [5] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-47-240:24694:24764 [5] NCCL INFO Ring 06 : 61 -> 68 [send] via NET/AWS Libfabric/2 | |
| ip-172-31-45-171:25064:25129 [6] NCCL INFO NET/OFI [6] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-47-188:22013:22084 [5] NCCL INFO Ring 06 : 77 -> 84 [send] via NET/AWS Libfabric/2 | |
| ip-172-31-44-4:22498:22560 [6] NCCL INFO NET/OFI [6] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-45-171:25063:25130 [5] NCCL INFO Ring 06 : 29 -> 36 [send] via NET/AWS Libfabric/2 | |
| ip-172-31-36-128:22026:22094 [5] NCCL INFO NET/OFI [5] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-37-141:25829:25898 [5] NCCL INFO NET/OFI [5] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-38-82:24984:25049 [6] NCCL INFO Ring 07 : 39 -> 46 [receive] via NET/AWS Libfabric/3 | |
| ip-172-31-44-142:24935:25000 [6] NCCL INFO Ring 07 : 15 -> 22 [receive] via NET/AWS Libfabric/3 | |
| ip-172-31-32-86:24697:24768 [6] NCCL INFO NET/OFI [6] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-47-240:24695:24762 [6] NCCL INFO NET/OFI [6] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-47-188:22014:22087 [6] NCCL INFO NET/OFI [6] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-46-50:22046:22116 [6] NCCL INFO Ring 07 : 63 -> 70 [receive] via NET/AWS Libfabric/3 | |
| ip-172-31-33-235:21964:22033 [5] NCCL INFO NET/OFI [5] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-36-188:22212:22276 [5] NCCL INFO Ring 06 : 93 -> 100 [send] via NET/AWS Libfabric/2 | |
| ip-172-31-44-230:21992:22064 [5] NCCL INFO Ring 06 : 101 -> 108 [send] via NET/AWS Libfabric/2 | |
| ip-172-31-39-141:20309:20382 [6] NCCL INFO Ring 07 : 127 -> 6 [receive] via NET/AWS Libfabric/3 | |
| ip-172-31-47-156:22566:22635 [6] NCCL INFO Ring 07 : 79 -> 86 [receive] via NET/AWS Libfabric/3 | |
| ip-172-31-36-188:22213:22279 [6] NCCL INFO NET/OFI [6] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-44-230:21993:22066 [6] NCCL INFO NET/OFI [6] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-32-86:24696:24767 [5] NCCL INFO Ring 06 : 53 -> 60 [send] via NET/AWS Libfabric/2 | |
| ip-172-31-44-4:22498:22560 [6] NCCL INFO Ring 07 : 119 -> 126 [receive] via NET/AWS Libfabric/3 | |
| ip-172-31-36-128:22026:22094 [5] NCCL INFO Ring 06 : 117 -> 124 [send] via NET/AWS Libfabric/2 | |
| ip-172-31-37-141:25829:25898 [5] NCCL INFO Ring 06 : 37 -> 44 [send] via NET/AWS Libfabric/2 | |
| ip-172-31-45-171:25064:25129 [6] NCCL INFO Ring 07 : 23 -> 30 [receive] via NET/AWS Libfabric/3 | |
| ip-172-31-47-240:24695:24762 [6] NCCL INFO Ring 07 : 55 -> 62 [receive] via NET/AWS Libfabric/3 | |
| ip-172-31-32-86:24697:24768 [6] NCCL INFO Ring 07 : 47 -> 54 [receive] via NET/AWS Libfabric/3 | |
| ip-172-31-33-235:21965:22037 [6] NCCL INFO NET/OFI [6] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-36-128:22027:22092 [6] NCCL INFO NET/OFI [6] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-38-14:25303:25373 [5] NCCL INFO NET/OFI [5] getCudaPath dev 2 busId 0000:00:18.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-38-14:25304:25371 [6] NCCL INFO NET/OFI [6] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-47-188:22014:22087 [6] NCCL INFO Ring 07 : 71 -> 78 [receive] via NET/AWS Libfabric/3 | |
| ip-172-31-37-141:25830:25895 [6] NCCL INFO NET/OFI [6] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-33-235:21964:22033 [5] NCCL INFO Ring 06 : 109 -> 116 [send] via NET/AWS Libfabric/2 | |
| ip-172-31-36-188:22213:22279 [6] NCCL INFO Ring 07 : 87 -> 94 [receive] via NET/AWS Libfabric/3 | |
| ip-172-31-44-230:21993:22066 [6] NCCL INFO Ring 07 : 95 -> 102 [receive] via NET/AWS Libfabric/3 | |
| ip-172-31-33-235:21965:22037 [6] NCCL INFO Ring 07 : 103 -> 110 [receive] via NET/AWS Libfabric/3 | |
| ip-172-31-38-14:25303:25373 [5] NCCL INFO Ring 06 : 13 -> 20 [send] via NET/AWS Libfabric/2 | |
| ip-172-31-38-14:25304:25371 [6] NCCL INFO Ring 07 : 7 -> 14 [receive] via NET/AWS Libfabric/3 | |
| ip-172-31-36-128:22027:22092 [6] NCCL INFO Ring 07 : 111 -> 118 [receive] via NET/AWS Libfabric/3 | |
| ip-172-31-37-141:25830:25895 [6] NCCL INFO Ring 07 : 31 -> 38 [receive] via NET/AWS Libfabric/3 | |
| ip-172-31-44-142:24935:25000 [6] NCCL INFO Ring 07 : 22[6] -> 18[2] via P2P/IPC | |
| ip-172-31-38-82:24984:25049 [6] NCCL INFO Ring 07 : 46[6] -> 42[2] via P2P/IPC | |
| ip-172-31-39-141:20303:20372 [0] NCCL INFO Ring 07 : 0[0] -> 4[4] via P2P/IPC | |
| ip-172-31-46-50:22040:22119 [0] NCCL INFO Ring 07 : 64[0] -> 68[4] via P2P/IPC | |
| ip-172-31-32-86:24691:24765 [0] NCCL INFO Ring 07 : 48[0] -> 52[4] via P2P/IPC | |
| ip-172-31-46-50:22046:22116 [6] NCCL INFO Ring 07 : 70[6] -> 66[2] via P2P/IPC | |
| ip-172-31-39-141:20309:20382 [6] NCCL INFO Ring 07 : 6[6] -> 2[2] via P2P/IPC | |
| ip-172-31-38-82:24981:25048 [3] NCCL INFO Ring 07 : 43[3] -> 40[0] via P2P/IPC | |
| ip-172-31-47-156:22566:22635 [6] NCCL INFO Ring 07 : 86[6] -> 82[2] via P2P/IPC | |
| ip-172-31-44-4:22498:22560 [6] NCCL INFO Ring 07 : 126[6] -> 122[2] via P2P/IPC | |
| ip-172-31-44-4:22492:22563 [0] NCCL INFO Ring 07 : 120[0] -> 124[4] via P2P/IPC | |
| ip-172-31-47-240:24689:24766 [0] NCCL INFO Ring 07 : 56[0] -> 60[4] via P2P/IPC | |
| ip-172-31-47-156:22560:22634 [0] NCCL INFO Ring 07 : 80[0] -> 84[4] via P2P/IPC | |
| ip-172-31-39-141:20306:20379 [3] NCCL INFO Ring 07 : 3[3] -> 0[0] via P2P/IPC | |
| ip-172-31-45-171:25064:25129 [6] NCCL INFO Ring 07 : 30[6] -> 26[2] via P2P/IPC | |
| ip-172-31-44-142:24932:25006 [3] NCCL INFO Ring 07 : 19[3] -> 16[0] via P2P/IPC | |
| ip-172-31-47-240:24695:24762 [6] NCCL INFO Ring 07 : 62[6] -> 58[2] via P2P/IPC | |
| ip-172-31-32-86:24697:24768 [6] NCCL INFO Ring 07 : 54[6] -> 50[2] via P2P/IPC | |
| ip-172-31-47-156:22563:22638 [3] NCCL INFO Ring 07 : 83[3] -> 80[0] via P2P/IPC | |
| ip-172-31-38-82:24978:25047 [0] NCCL INFO Ring 07 : 40[0] -> 44[4] via P2P/IPC | |
| ip-172-31-45-171:25058:25128 [0] NCCL INFO Ring 07 : 24[0] -> 28[4] via P2P/IPC | |
| ip-172-31-47-188:22014:22087 [6] NCCL INFO Ring 07 : 78[6] -> 74[2] via P2P/IPC | |
| ip-172-31-44-4:22495:22558 [3] NCCL INFO Ring 07 : 123[3] -> 120[0] via P2P/IPC | |
| ip-172-31-47-188:22008:22081 [0] NCCL INFO Ring 07 : 72[0] -> 76[4] via P2P/IPC | |
| ip-172-31-36-188:22213:22279 [6] NCCL INFO Ring 07 : 94[6] -> 90[2] via P2P/IPC | |
| ip-172-31-36-188:22207:22281 [0] NCCL INFO Ring 07 : 88[0] -> 92[4] via P2P/IPC | |
| ip-172-31-38-14:25298:25374 [0] NCCL INFO Ring 07 : 8[0] -> 12[4] via P2P/IPC | |
| ip-172-31-39-141:20303:20372 [0] NCCL INFO Using 256 threads, Min Comp Cap 7, Trees disabled | |
| ip-172-31-44-230:21993:22066 [6] NCCL INFO Ring 07 : 102[6] -> 98[2] via P2P/IPC | |
| ip-172-31-38-82:24980:25051 [2] NCCL INFO Ring 07 : 42[2] -> 41[1] via P2P/IPC | |
| ip-172-31-44-230:21987:22059 [0] NCCL INFO Ring 07 : 96[0] -> 100[4] via P2P/IPC | |
| ip-172-31-47-240:24692:24760 [3] NCCL INFO Ring 07 : 59[3] -> 56[0] via P2P/IPC | |
| ip-172-31-37-141:25824:25897 [0] NCCL INFO Ring 07 : 32[0] -> 36[4] via P2P/IPC | |
| ip-172-31-38-14:25304:25371 [6] NCCL INFO Ring 07 : 14[6] -> 10[2] via P2P/IPC | |
| ip-172-31-44-142:24929:25002 [0] NCCL INFO Ring 07 : 16[0] -> 20[4] via P2P/IPC | |
| ip-172-31-37-141:25830:25895 [6] NCCL INFO Ring 07 : 38[6] -> 34[2] via P2P/IPC | |
| ip-172-31-33-235:21959:22032 [0] NCCL INFO Ring 07 : 104[0] -> 108[4] via P2P/IPC | |
| ip-172-31-33-235:21965:22037 [6] NCCL INFO Ring 07 : 110[6] -> 106[2] via P2P/IPC | |
| ip-172-31-36-128:22021:22098 [0] NCCL INFO Ring 07 : 112[0] -> 116[4] via P2P/IPC | |
| ip-172-31-36-128:22027:22092 [6] NCCL INFO Ring 07 : 118[6] -> 114[2] via P2P/IPC | |
| ip-172-31-39-141:20305:20380 [2] NCCL INFO Ring 07 : 2[2] -> 1[1] via P2P/IPC | |
| ip-172-31-47-156:22562:22633 [2] NCCL INFO Ring 07 : 82[2] -> 81[1] via P2P/IPC | |
| ip-172-31-47-188:22011:22083 [3] NCCL INFO Ring 07 : 75[3] -> 72[0] via P2P/IPC | |
| ip-172-31-44-4:22494:22564 [2] NCCL INFO Ring 07 : 122[2] -> 121[1] via P2P/IPC | |
| ip-172-31-46-50:22043:22118 [3] NCCL INFO Ring 07 : 67[3] -> 64[0] via P2P/IPC | |
| ip-172-31-38-82:24979:25052 [1] NCCL INFO Ring 07 : 41[1] -> 43[3] via P2P/IPC | |
| ip-172-31-46-50:22042:22112 [2] NCCL INFO Ring 07 : 66[2] -> 65[1] via P2P/IPC | |
| ip-172-31-44-142:24930:25003 [1] NCCL INFO Ring 07 : 17[1] -> 19[3] via P2P/IPC | |
| ip-172-31-44-230:21990:22065 [3] NCCL INFO Ring 07 : 99[3] -> 96[0] via P2P/IPC | |
| ip-172-31-47-240:24691:24761 [2] NCCL INFO Ring 07 : 58[2] -> 57[1] via P2P/IPC | |
| ip-172-31-45-171:25061:25134 [3] NCCL INFO Ring 07 : 27[3] -> 24[0] via P2P/IPC | |
| ip-172-31-45-171:25060:25132 [2] NCCL INFO Ring 07 : 26[2] -> 25[1] via P2P/IPC | |
| ip-172-31-39-141:20304:20381 [1] NCCL INFO Ring 07 : 1[1] -> 3[3] via P2P/IPC | |
| ip-172-31-47-188:22010:22085 [2] NCCL INFO Ring 07 : 74[2] -> 73[1] via P2P/IPC | |
| ip-172-31-37-141:25827:25893 [3] NCCL INFO Ring 07 : 35[3] -> 32[0] via P2P/IPC | |
| ip-172-31-36-188:22210:22277 [3] NCCL INFO Ring 07 : 91[3] -> 88[0] via P2P/IPC | |
| ip-172-31-36-188:22209:22274 [2] NCCL INFO Ring 07 : 90[2] -> 89[1] via P2P/IPC | |
| ip-172-31-44-4:22493:22565 [1] NCCL INFO Ring 07 : 121[1] -> 123[3] via P2P/IPC | |
| ip-172-31-47-156:22561:22636 [1] NCCL INFO Ring 07 : 81[1] -> 83[3] via P2P/IPC | |
| ip-172-31-38-82:24983:25050 [5] NCCL INFO Ring 07 : 45[5] -> 47[7] via P2P/IPC | |
| ip-172-31-38-82:24985:25053 [7] NCCL INFO NET/OFI [7] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-32-86:24694:24770 [3] NCCL INFO Ring 07 : 51[3] -> 48[0] via P2P/IPC | |
| ip-172-31-32-86:24693:24766 [2] NCCL INFO Ring 07 : 50[2] -> 49[1] via P2P/IPC | |
| ip-172-31-44-142:24934:25005 [5] NCCL INFO Ring 07 : 21[5] -> 23[7] via P2P/IPC | |
| ip-172-31-44-142:24937:25007 [7] NCCL INFO NET/OFI [7] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-44-230:21989:22061 [2] NCCL INFO Ring 07 : 98[2] -> 97[1] via P2P/IPC | |
| ip-172-31-46-50:22041:22117 [1] NCCL INFO Ring 07 : 65[1] -> 67[3] via P2P/IPC | |
| ip-172-31-47-240:24690:24765 [1] NCCL INFO Ring 07 : 57[1] -> 59[3] via P2P/IPC | |
| ip-172-31-39-141:20308:20378 [5] NCCL INFO Ring 07 : 5[5] -> 7[7] via P2P/IPC | |
| ip-172-31-32-86:24695:24771 [4] NCCL INFO Ring 07 : 52[4] -> 53[5] via P2P/IPC | |
| ip-172-31-39-141:20310:20383 [7] NCCL INFO NET/OFI [7] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-36-128:22024:22091 [3] NCCL INFO Ring 07 : 115[3] -> 112[0] via P2P/IPC | |
| ip-172-31-45-171:25059:25135 [1] NCCL INFO Ring 07 : 25[1] -> 27[3] via P2P/IPC | |
| ip-172-31-37-141:25826:25891 [2] NCCL INFO Ring 07 : 34[2] -> 33[1] via P2P/IPC | |
| ip-172-31-44-142:24931:25004 [2] NCCL INFO Ring 07 : 18[2] -> 17[1] via P2P/IPC | |
| ip-172-31-47-188:22009:22082 [1] NCCL INFO Ring 07 : 73[1] -> 75[3] via P2P/IPC | |
| ip-172-31-44-4:22497:22559 [5] NCCL INFO Ring 07 : 125[5] -> 127[7] via P2P/IPC | |
| ip-172-31-47-156:22565:22637 [5] NCCL INFO Ring 07 : 85[5] -> 87[7] via P2P/IPC | |
| ip-172-31-44-4:22499:22562 [7] NCCL INFO NET/OFI [7] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-38-82:24985:25053 [7] NCCL INFO Ring 07 : 47 -> 54 [send] via NET/AWS Libfabric/3 | |
| ip-172-31-39-141:20307:20377 [4] NCCL INFO Ring 07 : 4[4] -> 5[5] via P2P/IPC | |
| ip-172-31-46-50:22047:22114 [7] NCCL INFO NET/OFI [7] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-45-171:25062:25131 [4] NCCL INFO Ring 07 : 28[4] -> 29[5] via P2P/IPC | |
| ip-172-31-47-156:22569:22639 [7] NCCL INFO NET/OFI [7] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-36-188:22208:22278 [1] NCCL INFO Ring 07 : 89[1] -> 91[3] via P2P/IPC | |
| ip-172-31-36-128:22023:22093 [2] NCCL INFO Ring 07 : 114[2] -> 113[1] via P2P/IPC | |
| ip-172-31-44-142:24937:25007 [7] NCCL INFO Ring 07 : 23 -> 30 [send] via NET/AWS Libfabric/3 | |
| ip-172-31-36-188:22211:22280 [4] NCCL INFO Ring 07 : 92[4] -> 93[5] via P2P/IPC | |
| ip-172-31-33-235:21962:22036 [3] NCCL INFO Ring 07 : 107[3] -> 104[0] via P2P/IPC | |
| ip-172-31-38-82:24982:25054 [4] NCCL INFO Ring 07 : 44[4] -> 45[5] via P2P/IPC | |
| ip-172-31-46-50:22045:22115 [5] NCCL INFO Ring 07 : 69[5] -> 71[7] via P2P/IPC | |
| ip-172-31-47-156:22564:22632 [4] NCCL INFO Ring 07 : 84[4] -> 85[5] via P2P/IPC | |
| ip-172-31-38-14:25302:25368 [4] NCCL INFO Ring 07 : 12[4] -> 13[5] via P2P/IPC | |
| ip-172-31-46-50:22044:22113 [4] NCCL INFO Ring 07 : 68[4] -> 69[5] via P2P/IPC | |
| ip-172-31-33-235:21961:22023 [2] NCCL INFO Ring 07 : 106[2] -> 105[1] via P2P/IPC | |
| ip-172-31-47-240:24694:24764 [5] NCCL INFO Ring 07 : 61[5] -> 63[7] via P2P/IPC | |
| ip-172-31-44-230:21988:22060 [1] NCCL INFO Ring 07 : 97[1] -> 99[3] via P2P/IPC | |
| ip-172-31-39-141:20310:20383 [7] NCCL INFO Ring 07 : 7 -> 14 [send] via NET/AWS Libfabric/3 | |
| ip-172-31-32-86:24692:24769 [1] NCCL INFO Ring 07 : 49[1] -> 51[3] via P2P/IPC | |
| ip-172-31-47-240:24696:24763 [7] NCCL INFO NET/OFI [7] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-32-86:24698:24764 [7] NCCL INFO NET/OFI [7] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-44-4:22496:22561 [4] NCCL INFO Ring 07 : 124[4] -> 125[5] via P2P/IPC | |
| ip-172-31-44-4:22499:22562 [7] NCCL INFO Ring 07 : 127 -> 6 [send] via NET/AWS Libfabric/3 | |
| ip-172-31-38-14:25300:25369 [2] NCCL INFO Ring 07 : 10[2] -> 9[1] via P2P/IPC | |
| ip-172-31-47-188:22012:22080 [4] NCCL INFO Ring 07 : 76[4] -> 77[5] via P2P/IPC | |
| ip-172-31-45-171:25065:25133 [7] NCCL INFO NET/OFI [7] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-38-14:25301:25375 [3] NCCL INFO Ring 07 : 11[3] -> 8[0] via P2P/IPC | |
| ip-172-31-45-171:25063:25130 [5] NCCL INFO Ring 07 : 29[5] -> 31[7] via P2P/IPC | |
| ip-172-31-46-50:22047:22114 [7] NCCL INFO Ring 07 : 71 -> 78 [send] via NET/AWS Libfabric/3 | |
| ip-172-31-47-156:22569:22639 [7] NCCL INFO Ring 07 : 87 -> 94 [send] via NET/AWS Libfabric/3 | |
| ip-172-31-47-188:22013:22084 [5] NCCL INFO Ring 07 : 77[5] -> 79[7] via P2P/IPC | |
| ip-172-31-37-141:25825:25894 [1] NCCL INFO Ring 07 : 33[1] -> 35[3] via P2P/IPC | |
| ip-172-31-47-188:22015:22086 [7] NCCL INFO NET/OFI [7] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-36-188:22212:22276 [5] NCCL INFO Ring 07 : 93[5] -> 95[7] via P2P/IPC | |
| ip-172-31-36-188:22214:22275 [7] NCCL INFO NET/OFI [7] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-37-141:25828:25896 [4] NCCL INFO Ring 07 : 36[4] -> 37[5] via P2P/IPC | |
| ip-172-31-47-240:24693:24767 [4] NCCL INFO Ring 07 : 60[4] -> 61[5] via P2P/IPC | |
| ip-172-31-44-142:24933:25001 [4] NCCL INFO Ring 07 : 20[4] -> 21[5] via P2P/IPC | |
| ip-172-31-32-86:24696:24767 [5] NCCL INFO Ring 07 : 53[5] -> 55[7] via P2P/IPC | |
| ip-172-31-47-240:24696:24763 [7] NCCL INFO Ring 07 : 63 -> 70 [send] via NET/AWS Libfabric/3 | |
| ip-172-31-36-128:22022:22097 [1] NCCL INFO Ring 07 : 113[1] -> 115[3] via P2P/IPC | |
| ip-172-31-44-230:21991:22062 [4] NCCL INFO Ring 07 : 100[4] -> 101[5] via P2P/IPC | |
| ip-172-31-32-86:24698:24764 [7] NCCL INFO Ring 07 : 55 -> 62 [send] via NET/AWS Libfabric/3 | |
| ip-172-31-44-230:21992:22064 [5] NCCL INFO Ring 07 : 101[5] -> 103[7] via P2P/IPC | |
| ip-172-31-38-82:24978:25047 [0] NCCL INFO comm 0x7f42b8001cf0 rank 40 nranks 128 cudaDev 0 nvmlDev 0 - Init COMPLETE | |
| ip-172-31-39-141:20303:20372 [0] NCCL INFO comm 0x7fa56c001cf0 rank 0 nranks 128 cudaDev 0 nvmlDev 0 - Init COMPLETE | |
| ip-172-31-45-171:25065:25133 [7] NCCL INFO Ring 07 : 31 -> 38 [send] via NET/AWS Libfabric/3 | |
| ip-172-31-44-230:21994:22063 [7] NCCL INFO NET/OFI [7] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-33-235:21960:22031 [1] NCCL INFO Ring 07 : 105[1] -> 107[3] via P2P/IPC | |
| ip-172-31-33-235:21963:22035 [4] NCCL INFO Ring 07 : 108[4] -> 109[5] via P2P/IPC | |
| ip-172-31-37-141:25829:25898 [5] NCCL INFO Ring 07 : 37[5] -> 39[7] via P2P/IPC | |
| ip-172-31-47-188:22015:22086 [7] NCCL INFO Ring 07 : 79 -> 86 [send] via NET/AWS Libfabric/3 | |
| ip-172-31-36-188:22214:22275 [7] NCCL INFO Ring 07 : 95 -> 102 [send] via NET/AWS Libfabric/3 | |
| ip-172-31-47-156:22560:22634 [0] NCCL INFO comm 0x7fba40001cf0 rank 80 nranks 128 cudaDev 0 nvmlDev 0 - Init COMPLETE | |
| ip-172-31-37-141:25831:25892 [7] NCCL INFO NET/OFI [7] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-39-141:20303:20303 [0] NCCL INFO Launch mode Parallel | |
| ip-172-31-38-14:25299:25372 [1] NCCL INFO Ring 07 : 9[1] -> 11[3] via P2P/IPC | |
| ip-172-31-44-4:22492:22563 [0] NCCL INFO comm 0x7f0098001cf0 rank 120 nranks 128 cudaDev 0 nvmlDev 0 - Init COMPLETE | |
| ip-172-31-38-14:25306:25370 [7] NCCL INFO NET/OFI [7] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-38-82:24980:25051 [2] NCCL INFO comm 0x7f8df8001cf0 rank 42 nranks 128 cudaDev 2 nvmlDev 2 - Init COMPLETE | |
| ip-172-31-38-82:24981:25048 [3] NCCL INFO comm 0x7fb064001cf0 rank 43 nranks 128 cudaDev 3 nvmlDev 3 - Init COMPLETE | |
| ip-172-31-44-142:24932:25006 [3] NCCL INFO comm 0x7fddf4001cf0 rank 19 nranks 128 cudaDev 3 nvmlDev 3 - Init COMPLETE | |
| ip-172-31-36-128:22026:22094 [5] NCCL INFO Ring 07 : 117[5] -> 119[7] via P2P/IPC | |
| ip-172-31-36-128:22030:22096 [7] NCCL INFO NET/OFI [7] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-33-235:21966:22034 [7] NCCL INFO NET/OFI [7] getCudaPath dev 3 busId 0000:00:19.0 path /sys/devices/pci0000:00/ | |
| ip-172-31-44-230:21994:22063 [7] NCCL INFO Ring 07 : 103 -> 110 [send] via NET/AWS Libfabric/3 | |
| ip-172-31-33-235:21964:22033 [5] NCCL INFO Ring 07 : 109[5] -> 111[7] via P2P/IPC | |
| ip-172-31-46-50:22040:22119 [0] NCCL INFO comm 0x7f1274001cf0 rank 64 nranks 128 cudaDev 0 nvmlDev 0 - Init COMPLETE | |
| ip-172-31-36-128:22025:22095 [4] NCCL INFO Ring 07 : 116[4] -> 117[5] via P2P/IPC | |
| ip-172-31-47-240:24689:24766 [0] NCCL INFO comm 0x7f0eac001cf0 rank 56 nranks 128 cudaDev 0 nvmlDev 0 - Init COMPLETE | |
| ip-172-31-38-82:24979:25052 [1] NCCL INFO comm 0x7f51c4001cf0 rank 41 nranks 128 cudaDev 1 nvmlDev 1 - Init COMPLETE | |
| ip-172-31-37-141:25831:25892 [7] NCCL INFO Ring 07 : 39 -> 46 [send] via NET/AWS Libfabric/3 | |
| ip-172-31-44-4:22495:22558 [3] NCCL INFO comm 0x7f4eb4001cf0 rank 123 nranks 128 cudaDev 3 nvmlDev 3 - Init COMPLETE | |
| ip-172-31-44-4:22494:22564 [2] NCCL INFO comm 0x7fc938001cf0 rank 122 nranks 128 cudaDev 2 nvmlDev 2 - Init COMPLETE | |
| ip-172-31-44-142:24929:25002 [0] NCCL INFO comm 0x7f5ce4001cf0 rank 16 nranks 128 cudaDev 0 nvmlDev 0 - Init COMPLETE | |
| ip-172-31-38-14:25303:25373 [5] NCCL INFO Ring 07 : 13[5] -> 15[7] via P2P/IPC | |
| ip-172-31-38-14:25306:25370 [7] NCCL INFO Ring 07 : 15 -> 22 [send] via NET/AWS Libfabric/3 | |
| ip-172-31-39-141:20306:20379 [3] NCCL INFO comm 0x7f7c54001cf0 rank 3 nranks 128 cudaDev 3 nvmlDev 3 - Init COMPLETE | |
| ip-172-31-47-156:22562:22633 [2] NCCL INFO comm 0x7fa2a4001cf0 rank 82 nranks 128 cudaDev 2 nvmlDev 2 - Init COMPLETE | |
| ip-172-31-47-188:22008:22081 [0] NCCL INFO comm 0x7f8048001cf0 rank 72 nranks 128 cudaDev 0 nvmlDev 0 - Init COMPLETE | |
| ip-172-31-39-141:20305:20380 [2] NCCL INFO comm 0x7fba90001cf0 rank 2 nranks 128 cudaDev 2 nvmlDev 2 - Init COMPLETE | |
| ip-172-31-45-171:25058:25128 [0] NCCL INFO comm 0x7fa774001cf0 rank 24 nranks 128 cudaDev 0 nvmlDev 0 - Init COMPLETE | |
| ip-172-31-47-156:22563:22638 [3] NCCL INFO comm 0x7f92e0001cf0 rank 83 nranks 128 cudaDev 3 nvmlDev 3 - Init COMPLETE | |
| ip-172-31-36-128:22030:22096 [7] NCCL INFO Ring 07 : 119 -> 126 [send] via NET/AWS Libfabric/3 | |
| ip-172-31-33-235:21966:22034 [7] NCCL INFO Ring 07 : 111 -> 118 [send] via NET/AWS Libfabric/3 | |
| ip-172-31-36-188:22207:22281 [0] NCCL INFO comm 0x7f9780001cf0 rank 88 nranks 128 cudaDev 0 nvmlDev 0 - Init COMPLETE | |
| ip-172-31-38-82:24985:25053 [7] NCCL INFO comm 0x7f7cc4001cf0 rank 47 nranks 128 cudaDev 7 nvmlDev 7 - Init COMPLETE | |
| ip-172-31-44-230:21987:22059 [0] NCCL INFO comm 0x7f8f4c001cf0 rank 96 nranks 128 cudaDev 0 nvmlDev 0 - Init COMPLETE | |
| ip-172-31-44-4:22493:22565 [1] NCCL INFO comm 0x7f68cc001cf0 rank 121 nranks 128 cudaDev 1 nvmlDev 1 - Init COMPLETE | |
| ip-172-31-46-50:22043:22118 [3] NCCL INFO comm 0x7f79a0001cf0 rank 67 nranks 128 cudaDev 3 nvmlDev 3 - Init COMPLETE | |
| ip-172-31-46-50:22042:22112 [2] NCCL INFO comm 0x7f434c001cf0 rank 66 nranks 128 cudaDev 2 nvmlDev 2 - Init COMPLETE | |
| ip-172-31-32-86:24691:24765 [0] NCCL INFO comm 0x7f1ec0001cf0 rank 48 nranks 128 cudaDev 0 nvmlDev 0 - Init COMPLETE | |
| ip-172-31-37-141:25824:25897 [0] NCCL INFO comm 0x7f94e0001cf0 rank 32 nranks 128 cudaDev 0 nvmlDev 0 - Init COMPLETE | |
| ip-172-31-47-156:22561:22636 [1] NCCL INFO comm 0x7f20e4001cf0 rank 81 nranks 128 cudaDev 1 nvmlDev 1 - Init COMPLETE | |
| ip-172-31-38-82:24983:25050 [5] NCCL INFO comm 0x7f1f58001cf0 rank 45 nranks 128 cudaDev 5 nvmlDev 5 - Init COMPLETE | |
| ip-172-31-39-141:20304:20381 [1] NCCL INFO comm 0x7f18e4001cf0 rank 1 nranks 128 cudaDev 1 nvmlDev 1 - Init COMPLETE | |
| ip-172-31-44-142:24930:25003 [1] NCCL INFO comm 0x7fa534001cf0 rank 17 nranks 128 cudaDev 1 nvmlDev 1 - Init COMPLETE | |
| ip-172-31-47-240:24692:24760 [3] NCCL INFO comm 0x7fcde4001cf0 rank 59 nranks 128 cudaDev 3 nvmlDev 3 - Init COMPLETE | |
| ip-172-31-47-240:24691:24761 [2] NCCL INFO comm 0x7f25e4001cf0 rank 58 nranks 128 cudaDev 2 nvmlDev 2 - Init COMPLETE | |
| ip-172-31-36-188:22210:22277 [3] NCCL INFO comm 0x7f732c001cf0 rank 91 nranks 128 cudaDev 3 nvmlDev 3 - Init COMPLETE | |
| ip-172-31-36-188:22209:22274 [2] NCCL INFO comm 0x7f0094001cf0 rank 90 nranks 128 cudaDev 2 nvmlDev 2 - Init COMPLETE | |
| ip-172-31-44-4:22497:22559 [5] NCCL INFO comm 0x7f7a90001cf0 rank 125 nranks 128 cudaDev 5 nvmlDev 5 - Init COMPLETE | |
| ip-172-31-45-171:25061:25134 [3] NCCL INFO comm 0x7f0c74001cf0 rank 27 nranks 128 cudaDev 3 nvmlDev 3 - Init COMPLETE | |
| ip-172-31-46-50:22041:22117 [1] NCCL INFO comm 0x7f02d4001cf0 rank 65 nranks 128 cudaDev 1 nvmlDev 1 - Init COMPLETE | |
| ip-172-31-45-171:25060:25132 [2] NCCL INFO comm 0x7fee60001cf0 rank 26 nranks 128 cudaDev 2 nvmlDev 2 - Init COMPLETE | |
| ip-172-31-32-86:24697:24768 [6] NCCL INFO comm 0x7f46f4001cf0 rank 54 nranks 128 cudaDev 6 nvmlDev 6 - Init COMPLETE | |
| ip-172-31-39-141:20308:20378 [5] NCCL INFO comm 0x7f5b70001cf0 rank 5 nranks 128 cudaDev 5 nvmlDev 5 - Init COMPLETE | |
| ip-172-31-38-82:24982:25054 [4] NCCL INFO comm 0x7f3d90001cf0 rank 44 nranks 128 cudaDev 4 nvmlDev 4 - Init COMPLETE | |
| ip-172-31-47-156:22565:22637 [5] NCCL INFO comm 0x7f1834001cf0 rank 85 nranks 128 cudaDev 5 nvmlDev 5 - Init COMPLETE | |
| ip-172-31-47-188:22010:22085 [2] NCCL INFO comm 0x7efbfc001cf0 rank 74 nranks 128 cudaDev 2 nvmlDev 2 - Init COMPLETE | |
| ip-172-31-47-188:22011:22083 [3] NCCL INFO comm 0x7ff4f8001cf0 rank 75 nranks 128 cudaDev 3 nvmlDev 3 - Init COMPLETE | |
| ip-172-31-44-142:24937:25007 [7] NCCL INFO comm 0x7f1154001cf0 rank 23 nranks 128 cudaDev 7 nvmlDev 7 - Init COMPLETE | |
| ip-172-31-45-171:25064:25129 [6] NCCL INFO comm 0x7f10b0001cf0 rank 30 nranks 128 cudaDev 6 nvmlDev 6 - Init COMPLETE | |
| ip-172-31-44-4:22499:22562 [7] NCCL INFO comm 0x7fca44001cf0 rank 127 nranks 128 cudaDev 7 nvmlDev 7 - Init COMPLETE | |
| ip-172-31-46-50:22047:22114 [7] NCCL INFO comm 0x7fc07c001cf0 rank 71 nranks 128 cudaDev 7 nvmlDev 7 - Init COMPLETE | |
| ip-172-31-47-240:24690:24765 [1] NCCL INFO comm 0x7f20ac001cf0 rank 57 nranks 128 cudaDev 1 nvmlDev 1 - Init COMPLETE | |
| ip-172-31-44-230:21990:22065 [3] NCCL INFO comm 0x7fb224001cf0 rank 99 nranks 128 cudaDev 3 nvmlDev 3 - Init COMPLETE | |
| ip-172-31-44-230:21989:22061 [2] NCCL INFO comm 0x7f9e08001cf0 rank 98 nranks 128 cudaDev 2 nvmlDev 2 - Init COMPLETE | |
| ip-172-31-36-128:22021:22098 [0] NCCL INFO comm 0x7f0118001cf0 rank 112 nranks 128 cudaDev 0 nvmlDev 0 - Init COMPLETE | |
| ip-172-31-44-142:24931:25004 [2] NCCL INFO comm 0x7f29a4001cf0 rank 18 nranks 128 cudaDev 2 nvmlDev 2 - Init COMPLETE | |
| ip-172-31-39-141:20310:20383 [7] NCCL INFO comm 0x7f0ed4001cf0 rank 7 nranks 128 cudaDev 7 nvmlDev 7 - Init COMPLETE | |
| ip-172-31-39-141:20307:20377 [4] NCCL INFO comm 0x7f6d04001cf0 rank 4 nranks 128 cudaDev 4 nvmlDev 4 - Init COMPLETE | |
| ip-172-31-36-188:22213:22279 [6] NCCL INFO comm 0x7f2f40001cf0 rank 94 nranks 128 cudaDev 6 nvmlDev 6 - Init COMPLETE | |
| ip-172-31-36-188:22208:22278 [1] NCCL INFO comm 0x7f1a34001cf0 rank 89 nranks 128 cudaDev 1 nvmlDev 1 - Init COMPLETE | |
| ip-172-31-32-86:24694:24770 [3] NCCL INFO comm 0x7fb620001cf0 rank 51 nranks 128 cudaDev 3 nvmlDev 3 - Init COMPLETE | |
| ip-172-31-38-82:24984:25049 [6] NCCL INFO comm 0x7fb428001cf0 rank 46 nranks 128 cudaDev 6 nvmlDev 6 - Init COMPLETE | |
| ip-172-31-32-86:24693:24766 [2] NCCL INFO comm 0x7fea34001cf0 rank 50 nranks 128 cudaDev 2 nvmlDev 2 - Init COMPLETE | |
| ip-172-31-33-235:21959:22032 [0] NCCL INFO comm 0x7f6ee0001cf0 rank 104 nranks 128 cudaDev 0 nvmlDev 0 - Init COMPLETE | |
| ip-172-31-46-50:22045:22115 [5] NCCL INFO comm 0x7f3434001cf0 rank 69 nranks 128 cudaDev 5 nvmlDev 5 - Init COMPLETE | |
| ip-172-31-44-4:22496:22561 [4] NCCL INFO comm 0x7f7b54001cf0 rank 124 nranks 128 cudaDev 4 nvmlDev 4 - Init COMPLETE | |
| ip-172-31-47-156:22569:22639 [7] NCCL INFO comm 0x7f1eb4001cf0 rank 87 nranks 128 cudaDev 7 nvmlDev 7 - Init COMPLETE | |
| ip-172-31-38-14:25304:25371 [6] NCCL INFO comm 0x7f48e4001cf0 rank 14 nranks 128 cudaDev 6 nvmlDev 6 - Init COMPLETE | |
| ip-172-31-45-171:25059:25135 [1] NCCL INFO comm 0x7f030c001cf0 rank 25 nranks 128 cudaDev 1 nvmlDev 1 - Init COMPLETE | |
| ip-172-31-47-156:22564:22632 [4] NCCL INFO comm 0x7f5864001cf0 rank 84 nranks 128 cudaDev 4 nvmlDev 4 - Init COMPLETE | |
| ip-172-31-47-240:24694:24764 [5] NCCL INFO comm 0x7f2160001cf0 rank 61 nranks 128 cudaDev 5 nvmlDev 5 - Init COMPLETE | |
| ip-172-31-47-188:22014:22087 [6] NCCL INFO comm 0x7fb020001cf0 rank 78 nranks 128 cudaDev 6 nvmlDev 6 - Init COMPLETE | |
| ip-172-31-44-230:21988:22060 [1] NCCL INFO comm 0x7f0da0001cf0 rank 97 nranks 128 cudaDev 1 nvmlDev 1 - Init COMPLETE | |
| ip-172-31-47-188:22009:22082 [1] NCCL INFO comm 0x7f9f74001cf0 rank 73 nranks 128 cudaDev 1 nvmlDev 1 - Init COMPLETE | |
| ip-172-31-36-188:22211:22280 [4] NCCL INFO comm 0x7fd3f8001cf0 rank 92 nranks 128 cudaDev 4 nvmlDev 4 - Init COMPLETE | |
| ip-172-31-45-171:25062:25131 [4] NCCL INFO comm 0x7fed20001cf0 rank 28 nranks 128 cudaDev 4 nvmlDev 4 - Init COMPLETE | |
| ip-172-31-32-86:24695:24771 [4] NCCL INFO comm 0x7f8334001cf0 rank 52 nranks 128 cudaDev 4 nvmlDev 4 - Init COMPLETE | |
| ip-172-31-46-50:22044:22113 [4] NCCL INFO comm 0x7f790c001cf0 rank 68 nranks 128 cudaDev 4 nvmlDev 4 - Init COMPLETE | |
| ip-172-31-44-142:24934:25005 [5] NCCL INFO comm 0x7f616c001cf0 rank 21 nranks 128 cudaDev 5 nvmlDev 5 - Init COMPLETE | |
| ip-172-31-39-141:20309:20382 [6] NCCL INFO comm 0x7fe7c4001cf0 rank 6 nranks 128 cudaDev 6 nvmlDev 6 - Init COMPLETE | |
| ip-172-31-37-141:25827:25893 [3] NCCL INFO comm 0x7f0138001cf0 rank 35 nranks 128 cudaDev 3 nvmlDev 3 - Init COMPLETE | |
| ip-172-31-37-141:25826:25891 [2] NCCL INFO comm 0x7f9958001cf0 rank 34 nranks 128 cudaDev 2 nvmlDev 2 - Init COMPLETE | |
| ip-172-31-47-240:24696:24763 [7] NCCL INFO comm 0x7fee24001cf0 rank 63 nranks 128 cudaDev 7 nvmlDev 7 - Init COMPLETE | |
| ip-172-31-47-156:22566:22635 [6] NCCL INFO comm 0x7fa3f4001cf0 rank 86 nranks 128 cudaDev 6 nvmlDev 6 - Init COMPLETE | |
| ip-172-31-44-4:22498:22560 [6] NCCL INFO comm 0x7f6f0c001cf0 rank 126 nranks 128 cudaDev 6 nvmlDev 6 - Init COMPLETE | |
| ip-172-31-46-50:22046:22116 [6] NCCL INFO comm 0x7f624c001cf0 rank 70 nranks 128 cudaDev 6 nvmlDev 6 - Init COMPLETE | |
| ip-172-31-45-171:25065:25133 [7] NCCL INFO comm 0x7fc0d0001cf0 rank 31 nranks 128 cudaDev 7 nvmlDev 7 - Init COMPLETE | |
| ip-172-31-38-14:25298:25374 [0] NCCL INFO comm 0x7f1b60001cf0 rank 8 nranks 128 cudaDev 0 nvmlDev 0 - Init COMPLETE | |
| ip-172-31-44-230:21993:22066 [6] NCCL INFO comm 0x7fe228001cf0 rank 102 nranks 128 cudaDev 6 nvmlDev 6 - Init COMPLETE | |
| ip-172-31-32-86:24692:24769 [1] NCCL INFO comm 0x7feb0c001cf0 rank 49 nranks 128 cudaDev 1 nvmlDev 1 - Init COMPLETE | |
| ip-172-31-44-142:24933:25001 [4] NCCL INFO comm 0x7f9574001cf0 rank 20 nranks 128 cudaDev 4 nvmlDev 4 - Init COMPLETE | |
| ip-172-31-36-188:22212:22276 [5] NCCL INFO comm 0x7fd854001cf0 rank 93 nranks 128 cudaDev 5 nvmlDev 5 - Init COMPLETE | |
| ip-172-31-47-240:24695:24762 [6] NCCL INFO comm 0x7f10c0001cf0 rank 62 nranks 128 cudaDev 6 nvmlDev 6 - Init COMPLETE | |
| ip-172-31-47-188:22012:22080 [4] NCCL INFO comm 0x7f6cb0001cf0 rank 76 nranks 128 cudaDev 4 nvmlDev 4 - Init COMPLETE | |
| ip-172-31-37-141:25830:25895 [6] NCCL INFO comm 0x7f2078001cf0 rank 38 nranks 128 cudaDev 6 nvmlDev 6 - Init COMPLETE | |
| ip-172-31-32-86:24698:24764 [7] NCCL INFO comm 0x7f89a0001cf0 rank 55 nranks 128 cudaDev 7 nvmlDev 7 - Init COMPLETE | |
| ip-172-31-36-128:22024:22091 [3] NCCL INFO comm 0x7f8d90001cf0 rank 115 nranks 128 cudaDev 3 nvmlDev 3 - Init COMPLETE | |
| ip-172-31-36-128:22023:22093 [2] NCCL INFO comm 0x7fc3f4001cf0 rank 114 nranks 128 cudaDev 2 nvmlDev 2 - Init COMPLETE | |
| ip-172-31-44-230:21991:22062 [4] NCCL INFO comm 0x7fa934001cf0 rank 100 nranks 128 cudaDev 4 nvmlDev 4 - Init COMPLETE | |
| ip-172-31-45-171:25063:25130 [5] NCCL INFO comm 0x7f7004001cf0 rank 29 nranks 128 cudaDev 5 nvmlDev 5 - Init COMPLETE | |
| ip-172-31-36-188:22214:22275 [7] NCCL INFO comm 0x7f8da4001cf0 rank 95 nranks 128 cudaDev 7 nvmlDev 7 - Init COMPLETE | |
| ip-172-31-33-235:21962:22036 [3] NCCL INFO comm 0x7f5f44001cf0 rank 107 nranks 128 cudaDev 3 nvmlDev 3 - Init COMPLETE | |
| ip-172-31-33-235:21961:22023 [2] NCCL INFO comm 0x7fb958001cf0 rank 106 nranks 128 cudaDev 2 nvmlDev 2 - Init COMPLETE | |
| ip-172-31-37-141:25825:25894 [1] NCCL INFO comm 0x7f1ac4001cf0 rank 33 nranks 128 cudaDev 1 nvmlDev 1 - Init COMPLETE | |
| ip-172-31-44-142:24935:25000 [6] NCCL INFO comm 0x7facf4001cf0 rank 22 nranks 128 cudaDev 6 nvmlDev 6 - Init COMPLETE | |
| ip-172-31-38-14:25302:25368 [4] NCCL INFO comm 0x7fcf98001cf0 rank 12 nranks 128 cudaDev 4 nvmlDev 4 - Init COMPLETE | |
| ip-172-31-47-188:22013:22084 [5] NCCL INFO comm 0x7f4cc0001cf0 rank 77 nranks 128 cudaDev 5 nvmlDev 5 - Init COMPLETE | |
| ip-172-31-47-240:24693:24767 [4] NCCL INFO comm 0x7f6ac4001cf0 rank 60 nranks 128 cudaDev 4 nvmlDev 4 - Init COMPLETE | |
| ip-172-31-32-86:24696:24767 [5] NCCL INFO comm 0x7f3d5c001cf0 rank 53 nranks 128 cudaDev 5 nvmlDev 5 - Init COMPLETE | |
| ip-172-31-44-230:21992:22064 [5] NCCL INFO comm 0x7f4e18001cf0 rank 101 nranks 128 cudaDev 5 nvmlDev 5 - Init COMPLETE | |
| ip-172-31-37-141:25828:25896 [4] NCCL INFO comm 0x7f5694001cf0 rank 36 nranks 128 cudaDev 4 nvmlDev 4 - Init COMPLETE | |
| ip-172-31-47-188:22015:22086 [7] NCCL INFO comm 0x7f4b14001cf0 rank 79 nranks 128 cudaDev 7 nvmlDev 7 - Init COMPLETE | |
| ip-172-31-33-235:21965:22037 [6] NCCL INFO comm 0x7fb9c8001cf0 rank 110 nranks 128 cudaDev 6 nvmlDev 6 - Init COMPLETE | |
| ip-172-31-36-128:22022:22097 [1] NCCL INFO comm 0x7fc6b0001cf0 rank 113 nranks 128 cudaDev 1 nvmlDev 1 - Init COMPLETE | |
| ip-172-31-38-14:25300:25369 [2] NCCL INFO comm 0x7f82b0001cf0 rank 10 nranks 128 cudaDev 2 nvmlDev 2 - Init COMPLETE | |
| ip-172-31-38-14:25301:25375 [3] NCCL INFO comm 0x7fd9f4001cf0 rank 11 nranks 128 cudaDev 3 nvmlDev 3 - Init COMPLETE | |
| ip-172-31-33-235:21960:22031 [1] NCCL INFO comm 0x7f89c8001cf0 rank 105 nranks 128 cudaDev 1 nvmlDev 1 - Init COMPLETE | |
| ip-172-31-37-141:25829:25898 [5] NCCL INFO comm 0x7fb6e8001cf0 rank 37 nranks 128 cudaDev 5 nvmlDev 5 - Init COMPLETE | |
| ip-172-31-44-230:21994:22063 [7] NCCL INFO comm 0x7f0b98001cf0 rank 103 nranks 128 cudaDev 7 nvmlDev 7 - Init COMPLETE | |
| ip-172-31-33-235:21963:22035 [4] NCCL INFO comm 0x7f8478001cf0 rank 108 nranks 128 cudaDev 4 nvmlDev 4 - Init COMPLETE | |
| ip-172-31-37-141:25831:25892 [7] NCCL INFO comm 0x7fc784001cf0 rank 39 nranks 128 cudaDev 7 nvmlDev 7 - Init COMPLETE | |
| ip-172-31-33-235:21966:22034 [7] NCCL INFO comm 0x7f9550001cf0 rank 111 nranks 128 cudaDev 7 nvmlDev 7 - Init COMPLETE | |
| ip-172-31-38-14:25306:25370 [7] NCCL INFO comm 0x7fb8a0001cf0 rank 15 nranks 128 cudaDev 7 nvmlDev 7 - Init COMPLETE | |
| ip-172-31-38-14:25299:25372 [1] NCCL INFO comm 0x7f5014001cf0 rank 9 nranks 128 cudaDev 1 nvmlDev 1 - Init COMPLETE | |
| ip-172-31-36-128:22030:22096 [7] NCCL INFO comm 0x7faf3c001cf0 rank 119 nranks 128 cudaDev 7 nvmlDev 7 - Init COMPLETE | |
| ip-172-31-36-128:22027:22092 [6] NCCL INFO comm 0x7f74b0001cf0 rank 118 nranks 128 cudaDev 6 nvmlDev 6 - Init COMPLETE | |
| ip-172-31-36-128:22026:22094 [5] NCCL INFO comm 0x7fad80001cf0 rank 117 nranks 128 cudaDev 5 nvmlDev 5 - Init COMPLETE | |
| ip-172-31-33-235:21964:22033 [5] NCCL INFO comm 0x7fb884001cf0 rank 109 nranks 128 cudaDev 5 nvmlDev 5 - Init COMPLETE | |
| ip-172-31-38-14:25303:25373 [5] NCCL INFO comm 0x7feaf0001cf0 rank 13 nranks 128 cudaDev 5 nvmlDev 5 - Init COMPLETE | |
| ip-172-31-36-128:22025:22095 [4] NCCL INFO comm 0x7fc6b4001cf0 rank 116 nranks 128 cudaDev 4 nvmlDev 4 - Init COMPLETE | |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment