Last active
October 27, 2022 02:53
-
-
Save davidberard98/b09c4b3bfc4b9b7a41785c65c612fec8 to your computer and use it in GitHub Desktop.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| /data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and may be removed in the future, please use 'weights' instead. | |
| warnings.warn( | |
| /data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and may be removed in the future. The current behavior is equivalent to passing `weights=ResNet50_Weights.IMAGENET1K_V1`. You can also use `weights=ResNet50_Weights.DEFAULT` to get the most up-to-date weights. | |
| warnings.warn(msg) | |
| /data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and may be removed in the future, please use 'weights' instead. | |
| warnings.warn( | |
| /data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and may be removed in the future. The current behavior is equivalent to passing `weights=ResNet50_Weights.IMAGENET1K_V1`. You can also use `weights=ResNet50_Weights.DEFAULT` to get the most up-to-date weights. | |
| warnings.warn(msg) | |
| [2022-10-26 04:14:27,885] torch._dynamo.variables.torch: [WARNING] Profiler will be ignored | |
| [2022-10-26 04:14:29,204] torch._dynamo.variables.torch: [WARNING] Profiler will be ignored | |
| [2022-10-26 04:21:22,358] torch._dynamo.optimizations.training: [WARNING] Unable to use Aot Autograd because of presence of mutation | |
| [2022-10-26 04:21:22,361] torch._inductor.compile_fx: [WARNING] Aot Autograd is not safe to run, so falling back to eager | |
| [2022-10-26 04:21:22,540] torch._dynamo.variables.torch: [WARNING] Profiler will be ignored | |
| [2022-10-26 04:21:22,761] torch._dynamo.variables.torch: [WARNING] Profiler will be ignored | |
| [2022-10-26 04:21:22,979] torch._dynamo.variables.torch: [WARNING] Profiler will be ignored | |
| [2022-10-26 04:22:40,674] torch._dynamo.optimizations.training: [WARNING] Unable to use Aot Autograd because of presence of mutation | |
| [2022-10-26 04:22:40,675] torch._inductor.compile_fx: [WARNING] Aot Autograd is not safe to run, so falling back to eager | |
| /data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and may be removed in the future, please use 'weights' instead. | |
| warnings.warn( | |
| /data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and may be removed in the future. The current behavior is equivalent to passing `weights=ResNet50_Weights.IMAGENET1K_V1`. You can also use `weights=ResNet50_Weights.DEFAULT` to get the most up-to-date weights. | |
| warnings.warn(msg) | |
| [2022-10-26 04:23:03,878] torch._dynamo.variables.torch: [WARNING] Profiler will be ignored | |
| [2022-10-26 04:23:05,196] torch._dynamo.variables.torch: [WARNING] Profiler will be ignored | |
| [2022-10-26 04:23:17,615] torch._dynamo.optimizations.training: [WARNING] Unable to use Aot Autograd because of presence of mutation | |
| [2022-10-26 04:23:17,615] torch._inductor.compile_fx: [WARNING] Aot Autograd is not safe to run, so falling back to eager | |
| Traceback (most recent call last): | |
| File "/data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/fx/graph_module.py", line 269, in __call__ | |
| return super(self.cls, obj).__call__(*args, **kwargs) # type: ignore[misc] | |
| File "/data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1363, in _call_impl | |
| return forward_call(*input, **kwargs) | |
| File "<eval_with_key>.19", line 7, in forward | |
| self_layer3_0_bn3 += self_layer3_0_downsample_1; iadd = self_layer3_0_bn3; self_layer3_0_bn3 = self_layer3_0_downsample_1 = None | |
| RuntimeError: Output 1 of CompiledFunctionBackward is a view and is being modified inplace. This view is the output of a function that returns multiple views. Such functions do not allow the output views to be modified inplace. You should replace the inplace operation by an out-of-place one. | |
| Exception raised from handle_view_on_rebase at /scratch/dberard/bench-fast/pytorch/torch/csrc/autograd/variable.cpp:819 (most recent call first): | |
| frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x6b (0x7fa6759cc2fb in /data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/lib/libc10.so) | |
| frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 0xce (0x7fa6759c7afe in /data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/lib/libc10.so) | |
| frame #2: torch::autograd::handle_view_on_rebase(torch::autograd::DifferentiableViewMeta*, bool) + 0x37b (0x7fa68ae8f97b in /data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so) | |
| frame #3: <unknown function> + 0x5ec24aa (0x7fa68a3864aa in /data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so) | |
| frame #4: <unknown function> + 0x60ef8ea (0x7fa68a5b38ea in /data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so) | |
| frame #5: at::_ops::add__Tensor::call(at::Tensor&, at::Tensor const&, c10::Scalar const&) + 0x15b (0x7fa688c25ccb in /data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so) | |
| frame #6: <unknown function> + 0x5e58f8 (0x7fa6924eb8f8 in /data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/lib/libtorch_python.so) | |
| frame #7: <unknown function> + 0x5e5d36 (0x7fa6924ebd36 in /data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/lib/libtorch_python.so) | |
| frame #8: <unknown function> + 0x1b9c03 (0x563bbf75ec03 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #9: <unknown function> + 0x194a64 (0x563bbf739a64 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #10: <unknown function> + 0x230eac (0x563bbf7d5eac in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #11: PyNumber_InPlaceAdd + 0x279 (0x563bbf74a0f9 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #12: _PyEval_EvalFrameDefault + 0x16b1 (0x563bbf77d051 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #13: <unknown function> + 0x8932fb (0x7fa6927992fb in /data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/lib/libtorch_python.so) | |
| frame #14: _PyFunction_Vectorcall + 0x1b7 (0x563bbf7727e7 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #15: <unknown function> + 0x166bf8 (0x563bbf70bbf8 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #16: PyObject_Call + 0x7d (0x563bbf6dc20d in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #17: _PyEval_EvalFrameDefault + 0x1f07 (0x563bbf77d8a7 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #18: <unknown function> + 0x8932fb (0x7fa6927992fb in /data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/lib/libtorch_python.so) | |
| frame #19: _PyEval_EvalCodeWithName + 0x260 (0x563bbf771600 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #20: _PyFunction_Vectorcall + 0x594 (0x563bbf772bc4 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #21: <unknown function> + 0x166bf8 (0x563bbf70bbf8 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #22: PyObject_Call + 0x7d (0x563bbf6dc20d in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #23: _PyEval_EvalFrameDefault + 0x1f07 (0x563bbf77d8a7 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #24: <unknown function> + 0x8932fb (0x7fa6927992fb in /data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/lib/libtorch_python.so) | |
| frame #25: _PyEval_EvalCodeWithName + 0x888 (0x563bbf771c28 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #26: _PyFunction_Vectorcall + 0x594 (0x563bbf772bc4 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #27: _PyObject_FastCallDict + 0xe7 (0x563bbf7036b7 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #28: <unknown function> + 0x194d2b (0x563bbf739d2b in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #29: PyObject_Call + 0x45d (0x563bbf6dc5ed in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #30: _PyEval_EvalFrameDefault + 0x1f07 (0x563bbf77d8a7 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #31: <unknown function> + 0x8932fb (0x7fa6927992fb in /data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/lib/libtorch_python.so) | |
| frame #32: _PyEval_EvalCodeWithName + 0x260 (0x563bbf771600 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #33: _PyFunction_Vectorcall + 0x594 (0x563bbf772bc4 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #34: _PyObject_FastCallDict + 0x5f (0x563bbf70362f in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #35: <unknown function> + 0x194d2b (0x563bbf739d2b in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #36: PyObject_Call + 0x45d (0x563bbf6dc5ed in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #37: _PyEval_EvalFrameDefault + 0x1f07 (0x563bbf77d8a7 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #38: <unknown function> + 0x8932fb (0x7fa6927992fb in /data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/lib/libtorch_python.so) | |
| frame #39: _PyEval_EvalCodeWithName + 0x260 (0x563bbf771600 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #40: _PyFunction_Vectorcall + 0x594 (0x563bbf772bc4 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #41: <unknown function> + 0x166bf8 (0x563bbf70bbf8 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #42: PyObject_Call + 0x7d (0x563bbf6dc20d in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #43: _PyEval_EvalFrameDefault + 0x1f07 (0x563bbf77d8a7 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #44: <unknown function> + 0x8932fb (0x7fa6927992fb in /data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/lib/libtorch_python.so) | |
| frame #45: _PyEval_EvalCodeWithName + 0x260 (0x563bbf771600 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #46: _PyFunction_Vectorcall + 0x594 (0x563bbf772bc4 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #47: _PyObject_FastCallDict + 0xe7 (0x563bbf7036b7 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #48: <unknown function> + 0x194d2b (0x563bbf739d2b in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #49: PyObject_Call + 0x45d (0x563bbf6dc5ed in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #50: _PyEval_EvalFrameDefault + 0x1f07 (0x563bbf77d8a7 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #51: <unknown function> + 0x8932fb (0x7fa6927992fb in /data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/lib/libtorch_python.so) | |
| frame #52: _PyFunction_Vectorcall + 0x1b7 (0x563bbf7727e7 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #53: <unknown function> + 0x166b2e (0x563bbf70bb2e in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #54: _PyEval_EvalFrameDefault + 0x71b (0x563bbf77c0bb in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #55: <unknown function> + 0x8932fb (0x7fa6927992fb in /data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/lib/libtorch_python.so) | |
| frame #56: _PyEval_EvalCodeWithName + 0x260 (0x563bbf771600 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #57: _PyFunction_Vectorcall + 0x594 (0x563bbf772bc4 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #58: _PyEval_EvalFrameDefault + 0x4c0 (0x563bbf77be60 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #59: <unknown function> + 0x8932fb (0x7fa6927992fb in /data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/lib/libtorch_python.so) | |
| frame #60: _PyEval_EvalCodeWithName + 0xd5f (0x563bbf7720ff in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #61: _PyFunction_Vectorcall + 0x594 (0x563bbf772bc4 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #62: <unknown function> + 0x166bf8 (0x563bbf70bbf8 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #63: PyObject_Call + 0x7d (0x563bbf6dc20d in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| Call using an FX-traced Module, line 7 of the traced Module's generated forward function: | |
| self_layer3_0_downsample_1 = self.self_layer3_0_downsample_1(self_layer3_0_downsample_0); self_layer3_0_downsample_0 = None | |
| self_layer3_0_bn3 += self_layer3_0_downsample_1; iadd = self_layer3_0_bn3; self_layer3_0_bn3 = self_layer3_0_downsample_1 = None | |
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ <--- HERE | |
| self_layer3_0_relu = self.self_layer3_0_relu(iadd); iadd = None | |
| self_layer3_1_conv1 = self.self_layer3_1_conv1(self_layer3_0_relu) | |
| [2022-10-26 04:23:17,639] torch._dynamo.output_graph: [WARNING] ---------------------------------------- | |
| [2022-10-26 04:23:17,639] torch._dynamo.output_graph: [WARNING] TORCHDYNAMO: backend compiler failed | |
| [2022-10-26 04:23:17,639] torch._dynamo.output_graph: [WARNING] Output 1 of CompiledFunctionBackward is a view and is being modified inplace. This view is the output of a function that returns multiple views. Such functions do not allow the output views to be modified inplace. You should replace the inplace operation by an out-of-place one. | |
| Exception raised from handle_view_on_rebase at /scratch/dberard/bench-fast/pytorch/torch/csrc/autograd/variable.cpp:819 (most recent call first): | |
| frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x6b (0x7fa6759cc2fb in /data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/lib/libc10.so) | |
| frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 0xce (0x7fa6759c7afe in /data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/lib/libc10.so) | |
| frame #2: torch::autograd::handle_view_on_rebase(torch::autograd::DifferentiableViewMeta*, bool) + 0x37b (0x7fa68ae8f97b in /data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so) | |
| frame #3: <unknown function> + 0x5ec24aa (0x7fa68a3864aa in /data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so) | |
| frame #4: <unknown function> + 0x60ef8ea (0x7fa68a5b38ea in /data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so) | |
| frame #5: at::_ops::add__Tensor::call(at::Tensor&, at::Tensor const&, c10::Scalar const&) + 0x15b (0x7fa688c25ccb in /data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so) | |
| frame #6: <unknown function> + 0x5e58f8 (0x7fa6924eb8f8 in /data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/lib/libtorch_python.so) | |
| frame #7: <unknown function> + 0x5e5d36 (0x7fa6924ebd36 in /data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/lib/libtorch_python.so) | |
| frame #8: <unknown function> + 0x1b9c03 (0x563bbf75ec03 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #9: <unknown function> + 0x194a64 (0x563bbf739a64 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #10: <unknown function> + 0x230eac (0x563bbf7d5eac in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #11: PyNumber_InPlaceAdd + 0x279 (0x563bbf74a0f9 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #12: _PyEval_EvalFrameDefault + 0x16b1 (0x563bbf77d051 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #13: <unknown function> + 0x8932fb (0x7fa6927992fb in /data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/lib/libtorch_python.so) | |
| frame #14: _PyFunction_Vectorcall + 0x1b7 (0x563bbf7727e7 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #15: <unknown function> + 0x166bf8 (0x563bbf70bbf8 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #16: PyObject_Call + 0x7d (0x563bbf6dc20d in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #17: _PyEval_EvalFrameDefault + 0x1f07 (0x563bbf77d8a7 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #18: <unknown function> + 0x8932fb (0x7fa6927992fb in /data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/lib/libtorch_python.so) | |
| frame #19: _PyEval_EvalCodeWithName + 0x260 (0x563bbf771600 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #20: _PyFunction_Vectorcall + 0x594 (0x563bbf772bc4 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #21: <unknown function> + 0x166bf8 (0x563bbf70bbf8 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #22: PyObject_Call + 0x7d (0x563bbf6dc20d in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #23: _PyEval_EvalFrameDefault + 0x1f07 (0x563bbf77d8a7 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #24: <unknown function> + 0x8932fb (0x7fa6927992fb in /data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/lib/libtorch_python.so) | |
| frame #25: _PyEval_EvalCodeWithName + 0x888 (0x563bbf771c28 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #26: _PyFunction_Vectorcall + 0x594 (0x563bbf772bc4 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #27: _PyObject_FastCallDict + 0xe7 (0x563bbf7036b7 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #28: <unknown function> + 0x194d2b (0x563bbf739d2b in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #29: PyObject_Call + 0x45d (0x563bbf6dc5ed in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #30: _PyEval_EvalFrameDefault + 0x1f07 (0x563bbf77d8a7 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #31: <unknown function> + 0x8932fb (0x7fa6927992fb in /data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/lib/libtorch_python.so) | |
| frame #32: _PyEval_EvalCodeWithName + 0x260 (0x563bbf771600 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #33: _PyFunction_Vectorcall + 0x594 (0x563bbf772bc4 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #34: _PyObject_FastCallDict + 0x5f (0x563bbf70362f in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #35: <unknown function> + 0x194d2b (0x563bbf739d2b in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #36: PyObject_Call + 0x45d (0x563bbf6dc5ed in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #37: _PyEval_EvalFrameDefault + 0x1f07 (0x563bbf77d8a7 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #38: <unknown function> + 0x8932fb (0x7fa6927992fb in /data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/lib/libtorch_python.so) | |
| frame #39: _PyEval_EvalCodeWithName + 0x260 (0x563bbf771600 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #40: _PyFunction_Vectorcall + 0x594 (0x563bbf772bc4 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #41: <unknown function> + 0x166bf8 (0x563bbf70bbf8 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #42: PyObject_Call + 0x7d (0x563bbf6dc20d in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #43: _PyEval_EvalFrameDefault + 0x1f07 (0x563bbf77d8a7 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #44: <unknown function> + 0x8932fb (0x7fa6927992fb in /data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/lib/libtorch_python.so) | |
| frame #45: _PyEval_EvalCodeWithName + 0x260 (0x563bbf771600 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #46: _PyFunction_Vectorcall + 0x594 (0x563bbf772bc4 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #47: _PyObject_FastCallDict + 0xe7 (0x563bbf7036b7 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #48: <unknown function> + 0x194d2b (0x563bbf739d2b in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #49: PyObject_Call + 0x45d (0x563bbf6dc5ed in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #50: _PyEval_EvalFrameDefault + 0x1f07 (0x563bbf77d8a7 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #51: <unknown function> + 0x8932fb (0x7fa6927992fb in /data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/lib/libtorch_python.so) | |
| frame #52: _PyFunction_Vectorcall + 0x1b7 (0x563bbf7727e7 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #53: <unknown function> + 0x166b2e (0x563bbf70bb2e in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #54: _PyEval_EvalFrameDefault + 0x71b (0x563bbf77c0bb in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #55: <unknown function> + 0x8932fb (0x7fa6927992fb in /data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/lib/libtorch_python.so) | |
| frame #56: _PyEval_EvalCodeWithName + 0x260 (0x563bbf771600 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #57: _PyFunction_Vectorcall + 0x594 (0x563bbf772bc4 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #58: _PyEval_EvalFrameDefault + 0x4c0 (0x563bbf77be60 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #59: <unknown function> + 0x8932fb (0x7fa6927992fb in /data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/lib/libtorch_python.so) | |
| frame #60: _PyEval_EvalCodeWithName + 0xd5f (0x563bbf7720ff in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #61: _PyFunction_Vectorcall + 0x594 (0x563bbf772bc4 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #62: <unknown function> + 0x166bf8 (0x563bbf70bbf8 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #63: PyObject_Call + 0x7d (0x563bbf6dc20d in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| While executing %submod_1 : [#users=2] = call_module[target=compiled_submod_1](args = (%getitem, %getitem_1), kwargs = {}) | |
| Original traceback: | |
| None | |
| Traceback (most recent call last): | |
| File "/data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/_dynamo/output_graph.py", line 428, in call_user_compiler | |
| compiled_fn = self.compiler_fn(gm, self.example_inputs()) | |
| File "/data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/_dynamo/optimizations/distributed.py", line 207, in compile_fn | |
| submod_compiler.run(*example_inputs) | |
| File "/data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/fx/interpreter.py", line 130, in run | |
| self.env[node] = self.run_node(node) | |
| File "/data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/_dynamo/optimizations/distributed.py", line 204, in run_node | |
| return getattr(self, n.op)(n.target, args, kwargs) | |
| File "/data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/fx/interpreter.py", line 288, in call_module | |
| return submod(*args, **kwargs) | |
| File "/data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1363, in _call_impl | |
| return forward_call(*input, **kwargs) | |
| File "/data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/_dynamo/optimizations/distributed.py", line 160, in forward | |
| x = self.compiled_submod(*args) | |
| File "/data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/fx/graph_module.py", line 660, in call_wrapped | |
| return self._wrapped_call(self, *args, **kwargs) | |
| File "/data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/fx/graph_module.py", line 277, in __call__ | |
| raise e.with_traceback(None) | |
| RuntimeError: Output 1 of CompiledFunctionBackward is a view and is being modified inplace. This view is the output of a function that returns multiple views. Such functions do not allow the output views to be modified inplace. You should replace the inplace operation by an out-of-place one. | |
| Exception raised from handle_view_on_rebase at /scratch/dberard/bench-fast/pytorch/torch/csrc/autograd/variable.cpp:819 (most recent call first): | |
| frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x6b (0x7fa6759cc2fb in /data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/lib/libc10.so) | |
| frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 0xce (0x7fa6759c7afe in /data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/lib/libc10.so) | |
| frame #2: torch::autograd::handle_view_on_rebase(torch::autograd::DifferentiableViewMeta*, bool) + 0x37b (0x7fa68ae8f97b in /data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so) | |
| frame #3: <unknown function> + 0x5ec24aa (0x7fa68a3864aa in /data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so) | |
| frame #4: <unknown function> + 0x60ef8ea (0x7fa68a5b38ea in /data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so) | |
| frame #5: at::_ops::add__Tensor::call(at::Tensor&, at::Tensor const&, c10::Scalar const&) + 0x15b (0x7fa688c25ccb in /data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so) | |
| frame #6: <unknown function> + 0x5e58f8 (0x7fa6924eb8f8 in /data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/lib/libtorch_python.so) | |
| frame #7: <unknown function> + 0x5e5d36 (0x7fa6924ebd36 in /data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/lib/libtorch_python.so) | |
| frame #8: <unknown function> + 0x1b9c03 (0x563bbf75ec03 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #9: <unknown function> + 0x194a64 (0x563bbf739a64 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #10: <unknown function> + 0x230eac (0x563bbf7d5eac in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #11: PyNumber_InPlaceAdd + 0x279 (0x563bbf74a0f9 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #12: _PyEval_EvalFrameDefault + 0x16b1 (0x563bbf77d051 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #13: <unknown function> + 0x8932fb (0x7fa6927992fb in /data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/lib/libtorch_python.so) | |
| frame #14: _PyFunction_Vectorcall + 0x1b7 (0x563bbf7727e7 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #15: <unknown function> + 0x166bf8 (0x563bbf70bbf8 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #16: PyObject_Call + 0x7d (0x563bbf6dc20d in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #17: _PyEval_EvalFrameDefault + 0x1f07 (0x563bbf77d8a7 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #18: <unknown function> + 0x8932fb (0x7fa6927992fb in /data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/lib/libtorch_python.so) | |
| frame #19: _PyEval_EvalCodeWithName + 0x260 (0x563bbf771600 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #20: _PyFunction_Vectorcall + 0x594 (0x563bbf772bc4 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #21: <unknown function> + 0x166bf8 (0x563bbf70bbf8 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #22: PyObject_Call + 0x7d (0x563bbf6dc20d in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #23: _PyEval_EvalFrameDefault + 0x1f07 (0x563bbf77d8a7 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #24: <unknown function> + 0x8932fb (0x7fa6927992fb in /data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/lib/libtorch_python.so) | |
| frame #25: _PyEval_EvalCodeWithName + 0x888 (0x563bbf771c28 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #26: _PyFunction_Vectorcall + 0x594 (0x563bbf772bc4 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #27: _PyObject_FastCallDict + 0xe7 (0x563bbf7036b7 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #28: <unknown function> + 0x194d2b (0x563bbf739d2b in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #29: PyObject_Call + 0x45d (0x563bbf6dc5ed in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #30: _PyEval_EvalFrameDefault + 0x1f07 (0x563bbf77d8a7 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #31: <unknown function> + 0x8932fb (0x7fa6927992fb in /data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/lib/libtorch_python.so) | |
| frame #32: _PyEval_EvalCodeWithName + 0x260 (0x563bbf771600 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #33: _PyFunction_Vectorcall + 0x594 (0x563bbf772bc4 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #34: _PyObject_FastCallDict + 0x5f (0x563bbf70362f in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #35: <unknown function> + 0x194d2b (0x563bbf739d2b in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #36: PyObject_Call + 0x45d (0x563bbf6dc5ed in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #37: _PyEval_EvalFrameDefault + 0x1f07 (0x563bbf77d8a7 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #38: <unknown function> + 0x8932fb (0x7fa6927992fb in /data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/lib/libtorch_python.so) | |
| frame #39: _PyEval_EvalCodeWithName + 0x260 (0x563bbf771600 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #40: _PyFunction_Vectorcall + 0x594 (0x563bbf772bc4 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #41: <unknown function> + 0x166bf8 (0x563bbf70bbf8 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #42: PyObject_Call + 0x7d (0x563bbf6dc20d in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #43: _PyEval_EvalFrameDefault + 0x1f07 (0x563bbf77d8a7 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #44: <unknown function> + 0x8932fb (0x7fa6927992fb in /data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/lib/libtorch_python.so) | |
| frame #45: _PyEval_EvalCodeWithName + 0x260 (0x563bbf771600 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #46: _PyFunction_Vectorcall + 0x594 (0x563bbf772bc4 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #47: _PyObject_FastCallDict + 0xe7 (0x563bbf7036b7 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #48: <unknown function> + 0x194d2b (0x563bbf739d2b in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #49: PyObject_Call + 0x45d (0x563bbf6dc5ed in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #50: _PyEval_EvalFrameDefault + 0x1f07 (0x563bbf77d8a7 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #51: <unknown function> + 0x8932fb (0x7fa6927992fb in /data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/lib/libtorch_python.so) | |
| frame #52: _PyFunction_Vectorcall + 0x1b7 (0x563bbf7727e7 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #53: <unknown function> + 0x166b2e (0x563bbf70bb2e in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #54: _PyEval_EvalFrameDefault + 0x71b (0x563bbf77c0bb in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #55: <unknown function> + 0x8932fb (0x7fa6927992fb in /data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/lib/libtorch_python.so) | |
| frame #56: _PyEval_EvalCodeWithName + 0x260 (0x563bbf771600 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #57: _PyFunction_Vectorcall + 0x594 (0x563bbf772bc4 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #58: _PyEval_EvalFrameDefault + 0x4c0 (0x563bbf77be60 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #59: <unknown function> + 0x8932fb (0x7fa6927992fb in /data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/lib/libtorch_python.so) | |
| frame #60: _PyEval_EvalCodeWithName + 0xd5f (0x563bbf7720ff in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #61: _PyFunction_Vectorcall + 0x594 (0x563bbf772bc4 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #62: <unknown function> + 0x166bf8 (0x563bbf70bbf8 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #63: PyObject_Call + 0x7d (0x563bbf6dc20d in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| While executing %submod_1 : [#users=2] = call_module[target=compiled_submod_1](args = (%getitem, %getitem_1), kwargs = {}) | |
| Original traceback: | |
| None | |
| [2022-10-26 04:23:17,647] torch._dynamo.output_graph: [WARNING] ---------------------------------------- | |
| [2022-10-26 04:23:17,648] torch._dynamo.convert_frame: [ERROR] WON'T CONVERT forward /data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torchvision/models/resnet.py line 284 | |
| due to: | |
| Traceback (most recent call last): | |
| File "/data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/fx/graph_module.py", line 277, in __call__ | |
| raise e.with_traceback(None) | |
| RuntimeError: Output 1 of CompiledFunctionBackward is a view and is being modified inplace. This view is the output of a function that returns multiple views. Such functions do not allow the output views to be modified inplace. You should replace the inplace operation by an out-of-place one. | |
| Exception raised from handle_view_on_rebase at /scratch/dberard/bench-fast/pytorch/torch/csrc/autograd/variable.cpp:819 (most recent call first): | |
| frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x6b (0x7fa6759cc2fb in /data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/lib/libc10.so) | |
| frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 0xce (0x7fa6759c7afe in /data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/lib/libc10.so) | |
| frame #2: torch::autograd::handle_view_on_rebase(torch::autograd::DifferentiableViewMeta*, bool) + 0x37b (0x7fa68ae8f97b in /data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so) | |
| frame #3: <unknown function> + 0x5ec24aa (0x7fa68a3864aa in /data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so) | |
| frame #4: <unknown function> + 0x60ef8ea (0x7fa68a5b38ea in /data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so) | |
| frame #5: at::_ops::add__Tensor::call(at::Tensor&, at::Tensor const&, c10::Scalar const&) + 0x15b (0x7fa688c25ccb in /data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so) | |
| frame #6: <unknown function> + 0x5e58f8 (0x7fa6924eb8f8 in /data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/lib/libtorch_python.so) | |
| frame #7: <unknown function> + 0x5e5d36 (0x7fa6924ebd36 in /data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/lib/libtorch_python.so) | |
| frame #8: <unknown function> + 0x1b9c03 (0x563bbf75ec03 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #9: <unknown function> + 0x194a64 (0x563bbf739a64 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #10: <unknown function> + 0x230eac (0x563bbf7d5eac in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #11: PyNumber_InPlaceAdd + 0x279 (0x563bbf74a0f9 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #12: _PyEval_EvalFrameDefault + 0x16b1 (0x563bbf77d051 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #13: <unknown function> + 0x8932fb (0x7fa6927992fb in /data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/lib/libtorch_python.so) | |
| frame #14: _PyFunction_Vectorcall + 0x1b7 (0x563bbf7727e7 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #15: <unknown function> + 0x166bf8 (0x563bbf70bbf8 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #16: PyObject_Call + 0x7d (0x563bbf6dc20d in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #17: _PyEval_EvalFrameDefault + 0x1f07 (0x563bbf77d8a7 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #18: <unknown function> + 0x8932fb (0x7fa6927992fb in /data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/lib/libtorch_python.so) | |
| frame #19: _PyEval_EvalCodeWithName + 0x260 (0x563bbf771600 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #20: _PyFunction_Vectorcall + 0x594 (0x563bbf772bc4 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #21: <unknown function> + 0x166bf8 (0x563bbf70bbf8 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #22: PyObject_Call + 0x7d (0x563bbf6dc20d in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #23: _PyEval_EvalFrameDefault + 0x1f07 (0x563bbf77d8a7 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #24: <unknown function> + 0x8932fb (0x7fa6927992fb in /data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/lib/libtorch_python.so) | |
| frame #25: _PyEval_EvalCodeWithName + 0x888 (0x563bbf771c28 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #26: _PyFunction_Vectorcall + 0x594 (0x563bbf772bc4 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #27: _PyObject_FastCallDict + 0xe7 (0x563bbf7036b7 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #28: <unknown function> + 0x194d2b (0x563bbf739d2b in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #29: PyObject_Call + 0x45d (0x563bbf6dc5ed in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #30: _PyEval_EvalFrameDefault + 0x1f07 (0x563bbf77d8a7 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #31: <unknown function> + 0x8932fb (0x7fa6927992fb in /data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/lib/libtorch_python.so) | |
| frame #32: _PyEval_EvalCodeWithName + 0x260 (0x563bbf771600 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #33: _PyFunction_Vectorcall + 0x594 (0x563bbf772bc4 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #34: _PyObject_FastCallDict + 0x5f (0x563bbf70362f in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #35: <unknown function> + 0x194d2b (0x563bbf739d2b in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #36: PyObject_Call + 0x45d (0x563bbf6dc5ed in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #37: _PyEval_EvalFrameDefault + 0x1f07 (0x563bbf77d8a7 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #38: <unknown function> + 0x8932fb (0x7fa6927992fb in /data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/lib/libtorch_python.so) | |
| frame #39: _PyEval_EvalCodeWithName + 0x260 (0x563bbf771600 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #40: _PyFunction_Vectorcall + 0x594 (0x563bbf772bc4 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #41: <unknown function> + 0x166bf8 (0x563bbf70bbf8 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #42: PyObject_Call + 0x7d (0x563bbf6dc20d in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #43: _PyEval_EvalFrameDefault + 0x1f07 (0x563bbf77d8a7 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #44: <unknown function> + 0x8932fb (0x7fa6927992fb in /data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/lib/libtorch_python.so) | |
| frame #45: _PyEval_EvalCodeWithName + 0x260 (0x563bbf771600 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #46: _PyFunction_Vectorcall + 0x594 (0x563bbf772bc4 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #47: _PyObject_FastCallDict + 0xe7 (0x563bbf7036b7 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #48: <unknown function> + 0x194d2b (0x563bbf739d2b in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #49: PyObject_Call + 0x45d (0x563bbf6dc5ed in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #50: _PyEval_EvalFrameDefault + 0x1f07 (0x563bbf77d8a7 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #51: <unknown function> + 0x8932fb (0x7fa6927992fb in /data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/lib/libtorch_python.so) | |
| frame #52: _PyFunction_Vectorcall + 0x1b7 (0x563bbf7727e7 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #53: <unknown function> + 0x166b2e (0x563bbf70bb2e in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #54: _PyEval_EvalFrameDefault + 0x71b (0x563bbf77c0bb in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #55: <unknown function> + 0x8932fb (0x7fa6927992fb in /data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/lib/libtorch_python.so) | |
| frame #56: _PyEval_EvalCodeWithName + 0x260 (0x563bbf771600 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #57: _PyFunction_Vectorcall + 0x594 (0x563bbf772bc4 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #58: _PyEval_EvalFrameDefault + 0x4c0 (0x563bbf77be60 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #59: <unknown function> + 0x8932fb (0x7fa6927992fb in /data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/lib/libtorch_python.so) | |
| frame #60: _PyEval_EvalCodeWithName + 0xd5f (0x563bbf7720ff in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #61: _PyFunction_Vectorcall + 0x594 (0x563bbf772bc4 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #62: <unknown function> + 0x166bf8 (0x563bbf70bbf8 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #63: PyObject_Call + 0x7d (0x563bbf6dc20d in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| While executing %submod_1 : [#users=2] = call_module[target=compiled_submod_1](args = (%getitem, %getitem_1), kwargs = {}) | |
| Original traceback: | |
| None | |
| The above exception was the direct cause of the following exception: | |
| Traceback (most recent call last): | |
| File "/data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/_dynamo/output_graph.py", line 437, in call_user_compiler | |
| raise BackendCompilerFailed(self.compiler_fn, e) from e | |
| torch._dynamo.exc.BackendCompilerFailed: compile_fn raised RuntimeError: Output 1 of CompiledFunctionBackward is a view and is being modified inplace. This view is the output of a function that returns multiple views. Such functions do not allow the output views to be modified inplace. You should replace the inplace operation by an out-of-place one. | |
| Exception raised from handle_view_on_rebase at /scratch/dberard/bench-fast/pytorch/torch/csrc/autograd/variable.cpp:819 (most recent call first): | |
| frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x6b (0x7fa6759cc2fb in /data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/lib/libc10.so) | |
| frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 0xce (0x7fa6759c7afe in /data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/lib/libc10.so) | |
| frame #2: torch::autograd::handle_view_on_rebase(torch::autograd::DifferentiableViewMeta*, bool) + 0x37b (0x7fa68ae8f97b in /data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so) | |
| frame #3: <unknown function> + 0x5ec24aa (0x7fa68a3864aa in /data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so) | |
| frame #4: <unknown function> + 0x60ef8ea (0x7fa68a5b38ea in /data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so) | |
| frame #5: at::_ops::add__Tensor::call(at::Tensor&, at::Tensor const&, c10::Scalar const&) + 0x15b (0x7fa688c25ccb in /data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so) | |
| frame #6: <unknown function> + 0x5e58f8 (0x7fa6924eb8f8 in /data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/lib/libtorch_python.so) | |
| frame #7: <unknown function> + 0x5e5d36 (0x7fa6924ebd36 in /data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/lib/libtorch_python.so) | |
| frame #8: <unknown function> + 0x1b9c03 (0x563bbf75ec03 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #9: <unknown function> + 0x194a64 (0x563bbf739a64 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #10: <unknown function> + 0x230eac (0x563bbf7d5eac in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #11: PyNumber_InPlaceAdd + 0x279 (0x563bbf74a0f9 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #12: _PyEval_EvalFrameDefault + 0x16b1 (0x563bbf77d051 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #13: <unknown function> + 0x8932fb (0x7fa6927992fb in /data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/lib/libtorch_python.so) | |
| frame #14: _PyFunction_Vectorcall + 0x1b7 (0x563bbf7727e7 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #15: <unknown function> + 0x166bf8 (0x563bbf70bbf8 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #16: PyObject_Call + 0x7d (0x563bbf6dc20d in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #17: _PyEval_EvalFrameDefault + 0x1f07 (0x563bbf77d8a7 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #18: <unknown function> + 0x8932fb (0x7fa6927992fb in /data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/lib/libtorch_python.so) | |
| frame #19: _PyEval_EvalCodeWithName + 0x260 (0x563bbf771600 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #20: _PyFunction_Vectorcall + 0x594 (0x563bbf772bc4 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #21: <unknown function> + 0x166bf8 (0x563bbf70bbf8 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #22: PyObject_Call + 0x7d (0x563bbf6dc20d in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #23: _PyEval_EvalFrameDefault + 0x1f07 (0x563bbf77d8a7 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #24: <unknown function> + 0x8932fb (0x7fa6927992fb in /data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/lib/libtorch_python.so) | |
| frame #25: _PyEval_EvalCodeWithName + 0x888 (0x563bbf771c28 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #26: _PyFunction_Vectorcall + 0x594 (0x563bbf772bc4 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #27: _PyObject_FastCallDict + 0xe7 (0x563bbf7036b7 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #28: <unknown function> + 0x194d2b (0x563bbf739d2b in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #29: PyObject_Call + 0x45d (0x563bbf6dc5ed in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #30: _PyEval_EvalFrameDefault + 0x1f07 (0x563bbf77d8a7 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #31: <unknown function> + 0x8932fb (0x7fa6927992fb in /data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/lib/libtorch_python.so) | |
| frame #32: _PyEval_EvalCodeWithName + 0x260 (0x563bbf771600 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #33: _PyFunction_Vectorcall + 0x594 (0x563bbf772bc4 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #34: _PyObject_FastCallDict + 0x5f (0x563bbf70362f in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #35: <unknown function> + 0x194d2b (0x563bbf739d2b in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #36: PyObject_Call + 0x45d (0x563bbf6dc5ed in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #37: _PyEval_EvalFrameDefault + 0x1f07 (0x563bbf77d8a7 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #38: <unknown function> + 0x8932fb (0x7fa6927992fb in /data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/lib/libtorch_python.so) | |
| frame #39: _PyEval_EvalCodeWithName + 0x260 (0x563bbf771600 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #40: _PyFunction_Vectorcall + 0x594 (0x563bbf772bc4 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #41: <unknown function> + 0x166bf8 (0x563bbf70bbf8 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #42: PyObject_Call + 0x7d (0x563bbf6dc20d in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #43: _PyEval_EvalFrameDefault + 0x1f07 (0x563bbf77d8a7 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #44: <unknown function> + 0x8932fb (0x7fa6927992fb in /data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/lib/libtorch_python.so) | |
| frame #45: _PyEval_EvalCodeWithName + 0x260 (0x563bbf771600 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #46: _PyFunction_Vectorcall + 0x594 (0x563bbf772bc4 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #47: _PyObject_FastCallDict + 0xe7 (0x563bbf7036b7 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #48: <unknown function> + 0x194d2b (0x563bbf739d2b in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #49: PyObject_Call + 0x45d (0x563bbf6dc5ed in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #50: _PyEval_EvalFrameDefault + 0x1f07 (0x563bbf77d8a7 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #51: <unknown function> + 0x8932fb (0x7fa6927992fb in /data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/lib/libtorch_python.so) | |
| frame #52: _PyFunction_Vectorcall + 0x1b7 (0x563bbf7727e7 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #53: <unknown function> + 0x166b2e (0x563bbf70bb2e in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #54: _PyEval_EvalFrameDefault + 0x71b (0x563bbf77c0bb in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #55: <unknown function> + 0x8932fb (0x7fa6927992fb in /data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/lib/libtorch_python.so) | |
| frame #56: _PyEval_EvalCodeWithName + 0x260 (0x563bbf771600 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #57: _PyFunction_Vectorcall + 0x594 (0x563bbf772bc4 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #58: _PyEval_EvalFrameDefault + 0x4c0 (0x563bbf77be60 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #59: <unknown function> + 0x8932fb (0x7fa6927992fb in /data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/lib/libtorch_python.so) | |
| frame #60: _PyEval_EvalCodeWithName + 0xd5f (0x563bbf7720ff in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #61: _PyFunction_Vectorcall + 0x594 (0x563bbf772bc4 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #62: <unknown function> + 0x166bf8 (0x563bbf70bbf8 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #63: PyObject_Call + 0x7d (0x563bbf6dc20d in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| While executing %submod_1 : [#users=2] = call_module[target=compiled_submod_1](args = (%getitem, %getitem_1), kwargs = {}) | |
| Original traceback: | |
| None | |
| You can suppress this exception and fall back to eager by setting: | |
| torchdynamo.config.suppress_errors = True | |
| from user code: | |
| File "/data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torchvision/models/resnet.py", line 285, in forward | |
| return self._forward_impl(x) | |
| Set torch._dynamo.config.verbose=True for more information | |
| ========== | |
| [2022-10-26 04:23:17,649] torch._dynamo.eval_frame: [ERROR] Error while processing frame | |
| Traceback (most recent call last): | |
| File "/data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/_dynamo/output_graph.py", line 428, in call_user_compiler | |
| compiled_fn = self.compiler_fn(gm, self.example_inputs()) | |
| File "/data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/_dynamo/optimizations/distributed.py", line 207, in compile_fn | |
| submod_compiler.run(*example_inputs) | |
| File "/data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/fx/interpreter.py", line 130, in run | |
| self.env[node] = self.run_node(node) | |
| File "/data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/_dynamo/optimizations/distributed.py", line 204, in run_node | |
| return getattr(self, n.op)(n.target, args, kwargs) | |
| File "/data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/fx/interpreter.py", line 288, in call_module | |
| return submod(*args, **kwargs) | |
| File "/data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1363, in _call_impl | |
| return forward_call(*input, **kwargs) | |
| File "/data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/_dynamo/optimizations/distributed.py", line 160, in forward | |
| x = self.compiled_submod(*args) | |
| File "/data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/fx/graph_module.py", line 660, in call_wrapped | |
| return self._wrapped_call(self, *args, **kwargs) | |
| File "/data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/fx/graph_module.py", line 277, in __call__ | |
| raise e.with_traceback(None) | |
| RuntimeError: Output 1 of CompiledFunctionBackward is a view and is being modified inplace. This view is the output of a function that returns multiple views. Such functions do not allow the output views to be modified inplace. You should replace the inplace operation by an out-of-place one. | |
| Exception raised from handle_view_on_rebase at /scratch/dberard/bench-fast/pytorch/torch/csrc/autograd/variable.cpp:819 (most recent call first): | |
| frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x6b (0x7fa6759cc2fb in /data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/lib/libc10.so) | |
| frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 0xce (0x7fa6759c7afe in /data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/lib/libc10.so) | |
| frame #2: torch::autograd::handle_view_on_rebase(torch::autograd::DifferentiableViewMeta*, bool) + 0x37b (0x7fa68ae8f97b in /data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so) | |
| frame #3: <unknown function> + 0x5ec24aa (0x7fa68a3864aa in /data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so) | |
| frame #4: <unknown function> + 0x60ef8ea (0x7fa68a5b38ea in /data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so) | |
| frame #5: at::_ops::add__Tensor::call(at::Tensor&, at::Tensor const&, c10::Scalar const&) + 0x15b (0x7fa688c25ccb in /data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so) | |
| frame #6: <unknown function> + 0x5e58f8 (0x7fa6924eb8f8 in /data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/lib/libtorch_python.so) | |
| frame #7: <unknown function> + 0x5e5d36 (0x7fa6924ebd36 in /data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/lib/libtorch_python.so) | |
| frame #8: <unknown function> + 0x1b9c03 (0x563bbf75ec03 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #9: <unknown function> + 0x194a64 (0x563bbf739a64 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #10: <unknown function> + 0x230eac (0x563bbf7d5eac in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #11: PyNumber_InPlaceAdd + 0x279 (0x563bbf74a0f9 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #12: _PyEval_EvalFrameDefault + 0x16b1 (0x563bbf77d051 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #13: <unknown function> + 0x8932fb (0x7fa6927992fb in /data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/lib/libtorch_python.so) | |
| frame #14: _PyFunction_Vectorcall + 0x1b7 (0x563bbf7727e7 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #15: <unknown function> + 0x166bf8 (0x563bbf70bbf8 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #16: PyObject_Call + 0x7d (0x563bbf6dc20d in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #17: _PyEval_EvalFrameDefault + 0x1f07 (0x563bbf77d8a7 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #18: <unknown function> + 0x8932fb (0x7fa6927992fb in /data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/lib/libtorch_python.so) | |
| frame #19: _PyEval_EvalCodeWithName + 0x260 (0x563bbf771600 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #20: _PyFunction_Vectorcall + 0x594 (0x563bbf772bc4 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #21: <unknown function> + 0x166bf8 (0x563bbf70bbf8 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #22: PyObject_Call + 0x7d (0x563bbf6dc20d in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #23: _PyEval_EvalFrameDefault + 0x1f07 (0x563bbf77d8a7 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #24: <unknown function> + 0x8932fb (0x7fa6927992fb in /data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/lib/libtorch_python.so) | |
| frame #25: _PyEval_EvalCodeWithName + 0x888 (0x563bbf771c28 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #26: _PyFunction_Vectorcall + 0x594 (0x563bbf772bc4 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #27: _PyObject_FastCallDict + 0xe7 (0x563bbf7036b7 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #28: <unknown function> + 0x194d2b (0x563bbf739d2b in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #29: PyObject_Call + 0x45d (0x563bbf6dc5ed in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #30: _PyEval_EvalFrameDefault + 0x1f07 (0x563bbf77d8a7 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #31: <unknown function> + 0x8932fb (0x7fa6927992fb in /data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/lib/libtorch_python.so) | |
| frame #32: _PyEval_EvalCodeWithName + 0x260 (0x563bbf771600 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #33: _PyFunction_Vectorcall + 0x594 (0x563bbf772bc4 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #34: _PyObject_FastCallDict + 0x5f (0x563bbf70362f in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #35: <unknown function> + 0x194d2b (0x563bbf739d2b in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #36: PyObject_Call + 0x45d (0x563bbf6dc5ed in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #37: _PyEval_EvalFrameDefault + 0x1f07 (0x563bbf77d8a7 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #38: <unknown function> + 0x8932fb (0x7fa6927992fb in /data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/lib/libtorch_python.so) | |
| frame #39: _PyEval_EvalCodeWithName + 0x260 (0x563bbf771600 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #40: _PyFunction_Vectorcall + 0x594 (0x563bbf772bc4 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #41: <unknown function> + 0x166bf8 (0x563bbf70bbf8 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #42: PyObject_Call + 0x7d (0x563bbf6dc20d in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #43: _PyEval_EvalFrameDefault + 0x1f07 (0x563bbf77d8a7 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #44: <unknown function> + 0x8932fb (0x7fa6927992fb in /data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/lib/libtorch_python.so) | |
| frame #45: _PyEval_EvalCodeWithName + 0x260 (0x563bbf771600 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #46: _PyFunction_Vectorcall + 0x594 (0x563bbf772bc4 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #47: _PyObject_FastCallDict + 0xe7 (0x563bbf7036b7 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #48: <unknown function> + 0x194d2b (0x563bbf739d2b in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #49: PyObject_Call + 0x45d (0x563bbf6dc5ed in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #50: _PyEval_EvalFrameDefault + 0x1f07 (0x563bbf77d8a7 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #51: <unknown function> + 0x8932fb (0x7fa6927992fb in /data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/lib/libtorch_python.so) | |
| frame #52: _PyFunction_Vectorcall + 0x1b7 (0x563bbf7727e7 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #53: <unknown function> + 0x166b2e (0x563bbf70bb2e in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #54: _PyEval_EvalFrameDefault + 0x71b (0x563bbf77c0bb in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #55: <unknown function> + 0x8932fb (0x7fa6927992fb in /data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/lib/libtorch_python.so) | |
| frame #56: _PyEval_EvalCodeWithName + 0x260 (0x563bbf771600 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #57: _PyFunction_Vectorcall + 0x594 (0x563bbf772bc4 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #58: _PyEval_EvalFrameDefault + 0x4c0 (0x563bbf77be60 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #59: <unknown function> + 0x8932fb (0x7fa6927992fb in /data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/lib/libtorch_python.so) | |
| frame #60: _PyEval_EvalCodeWithName + 0xd5f (0x563bbf7720ff in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #61: _PyFunction_Vectorcall + 0x594 (0x563bbf772bc4 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #62: <unknown function> + 0x166bf8 (0x563bbf70bbf8 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #63: PyObject_Call + 0x7d (0x563bbf6dc20d in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| While executing %submod_1 : [#users=2] = call_module[target=compiled_submod_1](args = (%getitem, %getitem_1), kwargs = {}) | |
| Original traceback: | |
| None | |
| The above exception was the direct cause of the following exception: | |
| Traceback (most recent call last): | |
| File "/data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/_dynamo/eval_frame.py", line 241, in catch_errors | |
| return hijacked_callback(frame, cache_size) | |
| File "/data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 452, in _convert_frame | |
| result = inner_convert(frame, cache_size) | |
| File "/data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 118, in _fn | |
| return fn(*args, **kwargs) | |
| File "/data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 86, in time_wrapper | |
| r = func(*args, **kwargs) | |
| File "/data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 325, in _convert_frame_assert | |
| return _compile( | |
| File "/data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 380, in _compile | |
| out_code = transform_code_object(code, transform) | |
| File "/data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/_dynamo/bytecode_transformation.py", line 341, in transform_code_object | |
| transformations(instructions, code_options) | |
| File "/data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 368, in transform | |
| tracer.run() | |
| File "/data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 1447, in run | |
| super().run() | |
| File "/data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 349, in run | |
| and self.step() | |
| File "/data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 322, in step | |
| getattr(self, inst.opname)(inst) | |
| File "/data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 1509, in RETURN_VALUE | |
| self.output.compile_subgraph(self) | |
| File "/data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/_dynamo/output_graph.py", line 328, in compile_subgraph | |
| self.compile_and_call_fx_graph(tx, list(reversed(stack_values)), root) | |
| File "/data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/_dynamo/output_graph.py", line 397, in compile_and_call_fx_graph | |
| compiled_fn = self.call_user_compiler(gm) | |
| File "/data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/_dynamo/output_graph.py", line 437, in call_user_compiler | |
| raise BackendCompilerFailed(self.compiler_fn, e) from e | |
| torch._dynamo.exc.BackendCompilerFailed: compile_fn raised RuntimeError: Output 1 of CompiledFunctionBackward is a view and is being modified inplace. This view is the output of a function that returns multiple views. Such functions do not allow the output views to be modified inplace. You should replace the inplace operation by an out-of-place one. | |
| Exception raised from handle_view_on_rebase at /scratch/dberard/bench-fast/pytorch/torch/csrc/autograd/variable.cpp:819 (most recent call first): | |
| frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x6b (0x7fa6759cc2fb in /data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/lib/libc10.so) | |
| frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 0xce (0x7fa6759c7afe in /data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/lib/libc10.so) | |
| frame #2: torch::autograd::handle_view_on_rebase(torch::autograd::DifferentiableViewMeta*, bool) + 0x37b (0x7fa68ae8f97b in /data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so) | |
| frame #3: <unknown function> + 0x5ec24aa (0x7fa68a3864aa in /data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so) | |
| frame #4: <unknown function> + 0x60ef8ea (0x7fa68a5b38ea in /data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so) | |
| frame #5: at::_ops::add__Tensor::call(at::Tensor&, at::Tensor const&, c10::Scalar const&) + 0x15b (0x7fa688c25ccb in /data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so) | |
| frame #6: <unknown function> + 0x5e58f8 (0x7fa6924eb8f8 in /data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/lib/libtorch_python.so) | |
| frame #7: <unknown function> + 0x5e5d36 (0x7fa6924ebd36 in /data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/lib/libtorch_python.so) | |
| frame #8: <unknown function> + 0x1b9c03 (0x563bbf75ec03 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #9: <unknown function> + 0x194a64 (0x563bbf739a64 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #10: <unknown function> + 0x230eac (0x563bbf7d5eac in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #11: PyNumber_InPlaceAdd + 0x279 (0x563bbf74a0f9 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #12: _PyEval_EvalFrameDefault + 0x16b1 (0x563bbf77d051 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #13: <unknown function> + 0x8932fb (0x7fa6927992fb in /data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/lib/libtorch_python.so) | |
| frame #14: _PyFunction_Vectorcall + 0x1b7 (0x563bbf7727e7 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #15: <unknown function> + 0x166bf8 (0x563bbf70bbf8 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #16: PyObject_Call + 0x7d (0x563bbf6dc20d in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #17: _PyEval_EvalFrameDefault + 0x1f07 (0x563bbf77d8a7 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #18: <unknown function> + 0x8932fb (0x7fa6927992fb in /data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/lib/libtorch_python.so) | |
| frame #19: _PyEval_EvalCodeWithName + 0x260 (0x563bbf771600 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #20: _PyFunction_Vectorcall + 0x594 (0x563bbf772bc4 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #21: <unknown function> + 0x166bf8 (0x563bbf70bbf8 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #22: PyObject_Call + 0x7d (0x563bbf6dc20d in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #23: _PyEval_EvalFrameDefault + 0x1f07 (0x563bbf77d8a7 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #24: <unknown function> + 0x8932fb (0x7fa6927992fb in /data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/lib/libtorch_python.so) | |
| frame #25: _PyEval_EvalCodeWithName + 0x888 (0x563bbf771c28 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #26: _PyFunction_Vectorcall + 0x594 (0x563bbf772bc4 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #27: _PyObject_FastCallDict + 0xe7 (0x563bbf7036b7 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #28: <unknown function> + 0x194d2b (0x563bbf739d2b in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #29: PyObject_Call + 0x45d (0x563bbf6dc5ed in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #30: _PyEval_EvalFrameDefault + 0x1f07 (0x563bbf77d8a7 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #31: <unknown function> + 0x8932fb (0x7fa6927992fb in /data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/lib/libtorch_python.so) | |
| frame #32: _PyEval_EvalCodeWithName + 0x260 (0x563bbf771600 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #33: _PyFunction_Vectorcall + 0x594 (0x563bbf772bc4 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #34: _PyObject_FastCallDict + 0x5f (0x563bbf70362f in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #35: <unknown function> + 0x194d2b (0x563bbf739d2b in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #36: PyObject_Call + 0x45d (0x563bbf6dc5ed in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #37: _PyEval_EvalFrameDefault + 0x1f07 (0x563bbf77d8a7 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #38: <unknown function> + 0x8932fb (0x7fa6927992fb in /data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/lib/libtorch_python.so) | |
| frame #39: _PyEval_EvalCodeWithName + 0x260 (0x563bbf771600 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #40: _PyFunction_Vectorcall + 0x594 (0x563bbf772bc4 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #41: <unknown function> + 0x166bf8 (0x563bbf70bbf8 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #42: PyObject_Call + 0x7d (0x563bbf6dc20d in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #43: _PyEval_EvalFrameDefault + 0x1f07 (0x563bbf77d8a7 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #44: <unknown function> + 0x8932fb (0x7fa6927992fb in /data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/lib/libtorch_python.so) | |
| frame #45: _PyEval_EvalCodeWithName + 0x260 (0x563bbf771600 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #46: _PyFunction_Vectorcall + 0x594 (0x563bbf772bc4 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #47: _PyObject_FastCallDict + 0xe7 (0x563bbf7036b7 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #48: <unknown function> + 0x194d2b (0x563bbf739d2b in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #49: PyObject_Call + 0x45d (0x563bbf6dc5ed in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #50: _PyEval_EvalFrameDefault + 0x1f07 (0x563bbf77d8a7 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #51: <unknown function> + 0x8932fb (0x7fa6927992fb in /data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/lib/libtorch_python.so) | |
| frame #52: _PyFunction_Vectorcall + 0x1b7 (0x563bbf7727e7 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #53: <unknown function> + 0x166b2e (0x563bbf70bb2e in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #54: _PyEval_EvalFrameDefault + 0x71b (0x563bbf77c0bb in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #55: <unknown function> + 0x8932fb (0x7fa6927992fb in /data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/lib/libtorch_python.so) | |
| frame #56: _PyEval_EvalCodeWithName + 0x260 (0x563bbf771600 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #57: _PyFunction_Vectorcall + 0x594 (0x563bbf772bc4 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #58: _PyEval_EvalFrameDefault + 0x4c0 (0x563bbf77be60 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #59: <unknown function> + 0x8932fb (0x7fa6927992fb in /data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/lib/libtorch_python.so) | |
| frame #60: _PyEval_EvalCodeWithName + 0xd5f (0x563bbf7720ff in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #61: _PyFunction_Vectorcall + 0x594 (0x563bbf772bc4 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #62: <unknown function> + 0x166bf8 (0x563bbf70bbf8 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #63: PyObject_Call + 0x7d (0x563bbf6dc20d in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| While executing %submod_1 : [#users=2] = call_module[target=compiled_submod_1](args = (%getitem, %getitem_1), kwargs = {}) | |
| Original traceback: | |
| None | |
| You can suppress this exception and fall back to eager by setting: | |
| torchdynamo.config.suppress_errors = True | |
| Process Process-3: | |
| Traceback (most recent call last): | |
| File "/data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/_dynamo/output_graph.py", line 428, in call_user_compiler | |
| compiled_fn = self.compiler_fn(gm, self.example_inputs()) | |
| File "/data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/_dynamo/optimizations/distributed.py", line 207, in compile_fn | |
| submod_compiler.run(*example_inputs) | |
| File "/data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/fx/interpreter.py", line 130, in run | |
| self.env[node] = self.run_node(node) | |
| File "/data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/_dynamo/optimizations/distributed.py", line 204, in run_node | |
| return getattr(self, n.op)(n.target, args, kwargs) | |
| File "/data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/fx/interpreter.py", line 288, in call_module | |
| return submod(*args, **kwargs) | |
| File "/data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1363, in _call_impl | |
| return forward_call(*input, **kwargs) | |
| File "/data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/_dynamo/optimizations/distributed.py", line 160, in forward | |
| x = self.compiled_submod(*args) | |
| File "/data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/fx/graph_module.py", line 660, in call_wrapped | |
| return self._wrapped_call(self, *args, **kwargs) | |
| File "/data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/fx/graph_module.py", line 277, in __call__ | |
| raise e.with_traceback(None) | |
| RuntimeError: Output 1 of CompiledFunctionBackward is a view and is being modified inplace. This view is the output of a function that returns multiple views. Such functions do not allow the output views to be modified inplace. You should replace the inplace operation by an out-of-place one. | |
| Exception raised from handle_view_on_rebase at /scratch/dberard/bench-fast/pytorch/torch/csrc/autograd/variable.cpp:819 (most recent call first): | |
| frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x6b (0x7fa6759cc2fb in /data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/lib/libc10.so) | |
| frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 0xce (0x7fa6759c7afe in /data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/lib/libc10.so) | |
| frame #2: torch::autograd::handle_view_on_rebase(torch::autograd::DifferentiableViewMeta*, bool) + 0x37b (0x7fa68ae8f97b in /data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so) | |
| frame #3: <unknown function> + 0x5ec24aa (0x7fa68a3864aa in /data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so) | |
| frame #4: <unknown function> + 0x60ef8ea (0x7fa68a5b38ea in /data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so) | |
| frame #5: at::_ops::add__Tensor::call(at::Tensor&, at::Tensor const&, c10::Scalar const&) + 0x15b (0x7fa688c25ccb in /data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so) | |
| frame #6: <unknown function> + 0x5e58f8 (0x7fa6924eb8f8 in /data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/lib/libtorch_python.so) | |
| frame #7: <unknown function> + 0x5e5d36 (0x7fa6924ebd36 in /data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/lib/libtorch_python.so) | |
| frame #8: <unknown function> + 0x1b9c03 (0x563bbf75ec03 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #9: <unknown function> + 0x194a64 (0x563bbf739a64 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #10: <unknown function> + 0x230eac (0x563bbf7d5eac in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #11: PyNumber_InPlaceAdd + 0x279 (0x563bbf74a0f9 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #12: _PyEval_EvalFrameDefault + 0x16b1 (0x563bbf77d051 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #13: <unknown function> + 0x8932fb (0x7fa6927992fb in /data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/lib/libtorch_python.so) | |
| frame #14: _PyFunction_Vectorcall + 0x1b7 (0x563bbf7727e7 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #15: <unknown function> + 0x166bf8 (0x563bbf70bbf8 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #16: PyObject_Call + 0x7d (0x563bbf6dc20d in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #17: _PyEval_EvalFrameDefault + 0x1f07 (0x563bbf77d8a7 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #18: <unknown function> + 0x8932fb (0x7fa6927992fb in /data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/lib/libtorch_python.so) | |
| frame #19: _PyEval_EvalCodeWithName + 0x260 (0x563bbf771600 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #20: _PyFunction_Vectorcall + 0x594 (0x563bbf772bc4 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #21: <unknown function> + 0x166bf8 (0x563bbf70bbf8 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #22: PyObject_Call + 0x7d (0x563bbf6dc20d in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #23: _PyEval_EvalFrameDefault + 0x1f07 (0x563bbf77d8a7 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #24: <unknown function> + 0x8932fb (0x7fa6927992fb in /data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/lib/libtorch_python.so) | |
| frame #25: _PyEval_EvalCodeWithName + 0x888 (0x563bbf771c28 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #26: _PyFunction_Vectorcall + 0x594 (0x563bbf772bc4 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #27: _PyObject_FastCallDict + 0xe7 (0x563bbf7036b7 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #28: <unknown function> + 0x194d2b (0x563bbf739d2b in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #29: PyObject_Call + 0x45d (0x563bbf6dc5ed in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #30: _PyEval_EvalFrameDefault + 0x1f07 (0x563bbf77d8a7 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #31: <unknown function> + 0x8932fb (0x7fa6927992fb in /data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/lib/libtorch_python.so) | |
| frame #32: _PyEval_EvalCodeWithName + 0x260 (0x563bbf771600 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #33: _PyFunction_Vectorcall + 0x594 (0x563bbf772bc4 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #34: _PyObject_FastCallDict + 0x5f (0x563bbf70362f in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #35: <unknown function> + 0x194d2b (0x563bbf739d2b in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #36: PyObject_Call + 0x45d (0x563bbf6dc5ed in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #37: _PyEval_EvalFrameDefault + 0x1f07 (0x563bbf77d8a7 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #38: <unknown function> + 0x8932fb (0x7fa6927992fb in /data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/lib/libtorch_python.so) | |
| frame #39: _PyEval_EvalCodeWithName + 0x260 (0x563bbf771600 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #40: _PyFunction_Vectorcall + 0x594 (0x563bbf772bc4 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #41: <unknown function> + 0x166bf8 (0x563bbf70bbf8 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #42: PyObject_Call + 0x7d (0x563bbf6dc20d in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #43: _PyEval_EvalFrameDefault + 0x1f07 (0x563bbf77d8a7 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #44: <unknown function> + 0x8932fb (0x7fa6927992fb in /data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/lib/libtorch_python.so) | |
| frame #45: _PyEval_EvalCodeWithName + 0x260 (0x563bbf771600 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #46: _PyFunction_Vectorcall + 0x594 (0x563bbf772bc4 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #47: _PyObject_FastCallDict + 0xe7 (0x563bbf7036b7 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #48: <unknown function> + 0x194d2b (0x563bbf739d2b in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #49: PyObject_Call + 0x45d (0x563bbf6dc5ed in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #50: _PyEval_EvalFrameDefault + 0x1f07 (0x563bbf77d8a7 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #51: <unknown function> + 0x8932fb (0x7fa6927992fb in /data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/lib/libtorch_python.so) | |
| frame #52: _PyFunction_Vectorcall + 0x1b7 (0x563bbf7727e7 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #53: <unknown function> + 0x166b2e (0x563bbf70bb2e in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #54: _PyEval_EvalFrameDefault + 0x71b (0x563bbf77c0bb in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #55: <unknown function> + 0x8932fb (0x7fa6927992fb in /data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/lib/libtorch_python.so) | |
| frame #56: _PyEval_EvalCodeWithName + 0x260 (0x563bbf771600 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #57: _PyFunction_Vectorcall + 0x594 (0x563bbf772bc4 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #58: _PyEval_EvalFrameDefault + 0x4c0 (0x563bbf77be60 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #59: <unknown function> + 0x8932fb (0x7fa6927992fb in /data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/lib/libtorch_python.so) | |
| frame #60: _PyEval_EvalCodeWithName + 0xd5f (0x563bbf7720ff in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #61: _PyFunction_Vectorcall + 0x594 (0x563bbf772bc4 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #62: <unknown function> + 0x166bf8 (0x563bbf70bbf8 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #63: PyObject_Call + 0x7d (0x563bbf6dc20d in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| While executing %submod_1 : [#users=2] = call_module[target=compiled_submod_1](args = (%getitem, %getitem_1), kwargs = {}) | |
| Original traceback: | |
| None | |
| The above exception was the direct cause of the following exception: | |
| Traceback (most recent call last): | |
| File "/data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap | |
| self.run() | |
| File "/data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/multiprocessing/process.py", line 108, in run | |
| self._target(*self._args, **self._kwargs) | |
| File "userbenchmark/ddp_experiments/__init__.py", line 203, in run_once | |
| result = trainer.measure() | |
| File "/fsx/users/dberard/scratch-local/bench-fast/benchmark/torchbenchmark/util/distributed/core_model/trainer.py", line 89, in measure | |
| self.benchmark.invoke() | |
| File "/fsx/users/dberard/scratch-local/bench-fast/benchmark/torchbenchmark/util/model.py", line 243, in invoke | |
| self.train() | |
| File "/data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/_dynamo/eval_frame.py", line 160, in _fn | |
| return fn(*args, **kwargs) | |
| File "/fsx/users/dberard/scratch-local/bench-fast/benchmark/torchbenchmark/util/framework/vision/model_factory.py", line 69, in train | |
| pred = self.model(data) | |
| File "/data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1363, in _call_impl | |
| return forward_call(*input, **kwargs) | |
| File "/data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/nn/parallel/distributed.py", line 1040, in forward | |
| output = self._run_ddp_forward(*inputs, **kwargs) | |
| File "/data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/nn/parallel/distributed.py", line 1000, in _run_ddp_forward | |
| return module_to_run(*inputs[0], **kwargs[0]) | |
| File "/data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1363, in _call_impl | |
| return forward_call(*input, **kwargs) | |
| File "/data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/_dynamo/eval_frame.py", line 241, in catch_errors | |
| return hijacked_callback(frame, cache_size) | |
| File "/data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 452, in _convert_frame | |
| result = inner_convert(frame, cache_size) | |
| File "/data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 118, in _fn | |
| return fn(*args, **kwargs) | |
| File "/data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 86, in time_wrapper | |
| r = func(*args, **kwargs) | |
| File "/data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 325, in _convert_frame_assert | |
| return _compile( | |
| File "/data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 380, in _compile | |
| out_code = transform_code_object(code, transform) | |
| File "/data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/_dynamo/bytecode_transformation.py", line 341, in transform_code_object | |
| transformations(instructions, code_options) | |
| File "/data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 368, in transform | |
| tracer.run() | |
| File "/data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 1447, in run | |
| super().run() | |
| File "/data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 349, in run | |
| and self.step() | |
| File "/data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 322, in step | |
| getattr(self, inst.opname)(inst) | |
| File "/data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 1509, in RETURN_VALUE | |
| self.output.compile_subgraph(self) | |
| File "/data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/_dynamo/output_graph.py", line 328, in compile_subgraph | |
| self.compile_and_call_fx_graph(tx, list(reversed(stack_values)), root) | |
| File "/data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/_dynamo/output_graph.py", line 397, in compile_and_call_fx_graph | |
| compiled_fn = self.call_user_compiler(gm) | |
| File "/data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/_dynamo/output_graph.py", line 437, in call_user_compiler | |
| raise BackendCompilerFailed(self.compiler_fn, e) from e | |
| torch._dynamo.exc.BackendCompilerFailed: compile_fn raised RuntimeError: Output 1 of CompiledFunctionBackward is a view and is being modified inplace. This view is the output of a function that returns multiple views. Such functions do not allow the output views to be modified inplace. You should replace the inplace operation by an out-of-place one. | |
| Exception raised from handle_view_on_rebase at /scratch/dberard/bench-fast/pytorch/torch/csrc/autograd/variable.cpp:819 (most recent call first): | |
| frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x6b (0x7fa6759cc2fb in /data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/lib/libc10.so) | |
| frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 0xce (0x7fa6759c7afe in /data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/lib/libc10.so) | |
| frame #2: torch::autograd::handle_view_on_rebase(torch::autograd::DifferentiableViewMeta*, bool) + 0x37b (0x7fa68ae8f97b in /data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so) | |
| frame #3: <unknown function> + 0x5ec24aa (0x7fa68a3864aa in /data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so) | |
| frame #4: <unknown function> + 0x60ef8ea (0x7fa68a5b38ea in /data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so) | |
| frame #5: at::_ops::add__Tensor::call(at::Tensor&, at::Tensor const&, c10::Scalar const&) + 0x15b (0x7fa688c25ccb in /data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so) | |
| frame #6: <unknown function> + 0x5e58f8 (0x7fa6924eb8f8 in /data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/lib/libtorch_python.so) | |
| frame #7: <unknown function> + 0x5e5d36 (0x7fa6924ebd36 in /data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/lib/libtorch_python.so) | |
| frame #8: <unknown function> + 0x1b9c03 (0x563bbf75ec03 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #9: <unknown function> + 0x194a64 (0x563bbf739a64 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #10: <unknown function> + 0x230eac (0x563bbf7d5eac in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #11: PyNumber_InPlaceAdd + 0x279 (0x563bbf74a0f9 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #12: _PyEval_EvalFrameDefault + 0x16b1 (0x563bbf77d051 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #13: <unknown function> + 0x8932fb (0x7fa6927992fb in /data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/lib/libtorch_python.so) | |
| frame #14: _PyFunction_Vectorcall + 0x1b7 (0x563bbf7727e7 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #15: <unknown function> + 0x166bf8 (0x563bbf70bbf8 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #16: PyObject_Call + 0x7d (0x563bbf6dc20d in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #17: _PyEval_EvalFrameDefault + 0x1f07 (0x563bbf77d8a7 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #18: <unknown function> + 0x8932fb (0x7fa6927992fb in /data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/lib/libtorch_python.so) | |
| frame #19: _PyEval_EvalCodeWithName + 0x260 (0x563bbf771600 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #20: _PyFunction_Vectorcall + 0x594 (0x563bbf772bc4 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #21: <unknown function> + 0x166bf8 (0x563bbf70bbf8 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #22: PyObject_Call + 0x7d (0x563bbf6dc20d in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #23: _PyEval_EvalFrameDefault + 0x1f07 (0x563bbf77d8a7 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #24: <unknown function> + 0x8932fb (0x7fa6927992fb in /data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/lib/libtorch_python.so) | |
| frame #25: _PyEval_EvalCodeWithName + 0x888 (0x563bbf771c28 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #26: _PyFunction_Vectorcall + 0x594 (0x563bbf772bc4 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #27: _PyObject_FastCallDict + 0xe7 (0x563bbf7036b7 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #28: <unknown function> + 0x194d2b (0x563bbf739d2b in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #29: PyObject_Call + 0x45d (0x563bbf6dc5ed in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #30: _PyEval_EvalFrameDefault + 0x1f07 (0x563bbf77d8a7 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #31: <unknown function> + 0x8932fb (0x7fa6927992fb in /data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/lib/libtorch_python.so) | |
| frame #32: _PyEval_EvalCodeWithName + 0x260 (0x563bbf771600 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #33: _PyFunction_Vectorcall + 0x594 (0x563bbf772bc4 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #34: _PyObject_FastCallDict + 0x5f (0x563bbf70362f in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #35: <unknown function> + 0x194d2b (0x563bbf739d2b in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #36: PyObject_Call + 0x45d (0x563bbf6dc5ed in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #37: _PyEval_EvalFrameDefault + 0x1f07 (0x563bbf77d8a7 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #38: <unknown function> + 0x8932fb (0x7fa6927992fb in /data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/lib/libtorch_python.so) | |
| frame #39: _PyEval_EvalCodeWithName + 0x260 (0x563bbf771600 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #40: _PyFunction_Vectorcall + 0x594 (0x563bbf772bc4 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #41: <unknown function> + 0x166bf8 (0x563bbf70bbf8 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #42: PyObject_Call + 0x7d (0x563bbf6dc20d in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #43: _PyEval_EvalFrameDefault + 0x1f07 (0x563bbf77d8a7 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #44: <unknown function> + 0x8932fb (0x7fa6927992fb in /data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/lib/libtorch_python.so) | |
| frame #45: _PyEval_EvalCodeWithName + 0x260 (0x563bbf771600 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #46: _PyFunction_Vectorcall + 0x594 (0x563bbf772bc4 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #47: _PyObject_FastCallDict + 0xe7 (0x563bbf7036b7 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #48: <unknown function> + 0x194d2b (0x563bbf739d2b in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #49: PyObject_Call + 0x45d (0x563bbf6dc5ed in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #50: _PyEval_EvalFrameDefault + 0x1f07 (0x563bbf77d8a7 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #51: <unknown function> + 0x8932fb (0x7fa6927992fb in /data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/lib/libtorch_python.so) | |
| frame #52: _PyFunction_Vectorcall + 0x1b7 (0x563bbf7727e7 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #53: <unknown function> + 0x166b2e (0x563bbf70bb2e in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #54: _PyEval_EvalFrameDefault + 0x71b (0x563bbf77c0bb in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #55: <unknown function> + 0x8932fb (0x7fa6927992fb in /data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/lib/libtorch_python.so) | |
| frame #56: _PyEval_EvalCodeWithName + 0x260 (0x563bbf771600 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #57: _PyFunction_Vectorcall + 0x594 (0x563bbf772bc4 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #58: _PyEval_EvalFrameDefault + 0x4c0 (0x563bbf77be60 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #59: <unknown function> + 0x8932fb (0x7fa6927992fb in /data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/lib/libtorch_python.so) | |
| frame #60: _PyEval_EvalCodeWithName + 0xd5f (0x563bbf7720ff in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #61: _PyFunction_Vectorcall + 0x594 (0x563bbf772bc4 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #62: <unknown function> + 0x166bf8 (0x563bbf70bbf8 in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| frame #63: PyObject_Call + 0x7d (0x563bbf6dc20d in /data/home/dberard/miniconda/envs/bench-fast/bin/python) | |
| While executing %submod_1 : [#users=2] = call_module[target=compiled_submod_1](args = (%getitem, %getitem_1), kwargs = {}) | |
| Original traceback: | |
| None | |
| You can suppress this exception and fall back to eager by setting: | |
| torchdynamo.config.suppress_errors = True |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| submitit INFO (2022-10-26 04:13:31,105) - Starting with JobEnvironment(job_id=74200, hostname=a100-st-p4d24xlarge-47, local_rank=0(8), node=0(1), global_rank=0(8)) | |
| submitit INFO (2022-10-26 04:13:31,106) - Loading pickle: /fsx/users/dberard/scratch-local/bench-fast/benchmark/logs_oct25/74200_submitted.pkl | |
| This is node 0 | |
| run_once | |
| Process group: 8 tasks, rank: 0 | |
| MY HOSTNAME: a100-st-p4d24xlarge-47 | |
| FI_PROVIDER : efa | |
| LD_LIBRARY_PATH : /fsx/users/dberard/scratch-local/bench-fast/aws-ofi-nccl/lib:/opt/amazon/efa/lib:/usr/local/cuda-11.6/lib:/usr/local/cuda-11.6/lib64:/opt/amazon/efa/lib:/opt/amazon/openmpi/lib:/usr/local/cuda/efa/lib:/usr/local/cuda/lib:/usr/local/cuda:/usr/local/cuda/lib64:/usr/local/cuda/extras/CUPTI/lib64:/usr/local/cuda/targets/x86_64-linux/lib:/usr/local/lib:/usr/lib: | |
| NCCL_DEBUG : INFO | |
| FI_EFA_USE_DEVICE_RDMA : 1 | |
| still waiting for process to exit | |
| didn't get the result, queue empty | |
| still waiting for process to exit | |
| a100-st-p4d24xlarge-47:40610:40610 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to ens | |
| a100-st-p4d24xlarge-47:40610:40610 [0] NCCL INFO NCCL_SOCKET_IFNAME set to ens | |
| a100-st-p4d24xlarge-47:40610:40610 [0] NCCL INFO Bootstrap : Using ens32:10.200.88.223<0> | |
| a100-st-p4d24xlarge-47:40610:40610 [0] NCCL INFO NET/Plugin: Failed to find ncclCollNetPlugin_v6 symbol. | |
| a100-st-p4d24xlarge-47:40610:40610 [0] NCCL INFO NET/Plugin: Failed to find ncclCollNetPlugin symbol (v4 or v5). | |
| a100-st-p4d24xlarge-47:40610:40610 [0] NCCL INFO cudaDriverVersion 11060 | |
| NCCL version 2.14.3+cuda11.6 | |
| a100-st-p4d24xlarge-47:40610:41147 [0] NCCL INFO NET/OFI Using aws-ofi-nccl 1.4.0aws | |
| a100-st-p4d24xlarge-47:40610:41147 [0] NCCL INFO NET/OFI Running on p4d.24xlarge platform, Setting NCCL_TOPO_FILE environment variable to /fsx/users/dberard/scratch-local/bench-fast/aws-ofi-nccl/share/aws-ofi-nccl/xml/p4d-24xl-topo.xml | |
| a100-st-p4d24xlarge-47:40610:41147 [0] NCCL INFO NET/OFI Setting FI_EFA_FORK_SAFE environment variable to 1 | |
| a100-st-p4d24xlarge-47:40610:41147 [0] NCCL INFO NET/OFI Selected Provider is efa | |
| a100-st-p4d24xlarge-47:40610:41147 [0] NCCL INFO Using network AWS Libfabric | |
| a100-st-p4d24xlarge-47:40610:41147 [0] NCCL INFO NCCL_TOPO_FILE set by environment to /fsx/users/dberard/scratch-local/bench-fast/aws-ofi-nccl/share/aws-ofi-nccl/xml/p4d-24xl-topo.xml | |
| a100-st-p4d24xlarge-47:40610:41147 [0] NCCL INFO NET/AWS Libfabric : GPU Direct RDMA Enabled for HCA 0 'rdmap16s27' | |
| a100-st-p4d24xlarge-47:40610:41147 [0] NCCL INFO NET/AWS Libfabric : GPU Direct RDMA Enabled for HCA 1 'rdmap32s27' | |
| a100-st-p4d24xlarge-47:40610:41147 [0] NCCL INFO NET/AWS Libfabric : GPU Direct RDMA Enabled for HCA 2 'rdmap144s27' | |
| a100-st-p4d24xlarge-47:40610:41147 [0] NCCL INFO NET/AWS Libfabric : GPU Direct RDMA Enabled for HCA 3 'rdmap160s27' | |
| a100-st-p4d24xlarge-47:40610:41147 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 101c0 / HCA 0 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-47:40610:41147 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 101d0 / HCA 0 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-47:40610:41147 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 201c0 / HCA 0 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-47:40610:41147 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 201d0 / HCA 0 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-47:40610:41147 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 901c0 / HCA 0 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-47:40610:41147 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 901d0 / HCA 0 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-47:40610:41147 [0] NCCL INFO GPU Direct RDMA Enabled for GPU a01c0 / HCA 0 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-47:40610:41147 [0] NCCL INFO GPU Direct RDMA Enabled for GPU a01d0 / HCA 0 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-47:40610:41147 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 101c0 / HCA 1 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-47:40610:41147 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 101d0 / HCA 1 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-47:40610:41147 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 201c0 / HCA 1 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-47:40610:41147 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 201d0 / HCA 1 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-47:40610:41147 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 901c0 / HCA 1 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-47:40610:41147 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 901d0 / HCA 1 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-47:40610:41147 [0] NCCL INFO GPU Direct RDMA Enabled for GPU a01c0 / HCA 1 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-47:40610:41147 [0] NCCL INFO GPU Direct RDMA Enabled for GPU a01d0 / HCA 1 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-47:40610:41147 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 101c0 / HCA 2 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-47:40610:41147 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 101d0 / HCA 2 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-47:40610:41147 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 201c0 / HCA 2 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-47:40610:41147 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 201d0 / HCA 2 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-47:40610:41147 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 901c0 / HCA 2 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-47:40610:41147 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 901d0 / HCA 2 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-47:40610:41147 [0] NCCL INFO GPU Direct RDMA Enabled for GPU a01c0 / HCA 2 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-47:40610:41147 [0] NCCL INFO GPU Direct RDMA Enabled for GPU a01d0 / HCA 2 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-47:40610:41147 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 101c0 / HCA 3 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-47:40610:41147 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 101d0 / HCA 3 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-47:40610:41147 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 201c0 / HCA 3 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-47:40610:41147 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 201d0 / HCA 3 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-47:40610:41147 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 901c0 / HCA 3 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-47:40610:41147 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 901d0 / HCA 3 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-47:40610:41147 [0] NCCL INFO GPU Direct RDMA Enabled for GPU a01c0 / HCA 3 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-47:40610:41147 [0] NCCL INFO GPU Direct RDMA Enabled for GPU a01d0 / HCA 3 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-47:40610:41147 [0] NCCL INFO Setting affinity for GPU 0 to 1f0000,0000001f | |
| a100-st-p4d24xlarge-47:40610:41147 [0] NCCL INFO Channel 00/24 : 0 1 2 3 4 5 6 7 | |
| a100-st-p4d24xlarge-47:40610:41147 [0] NCCL INFO Channel 01/24 : 0 1 2 3 4 5 6 7 | |
| a100-st-p4d24xlarge-47:40610:41147 [0] NCCL INFO Channel 02/24 : 0 1 2 3 4 5 6 7 | |
| a100-st-p4d24xlarge-47:40610:41147 [0] NCCL INFO Channel 03/24 : 0 1 2 3 4 5 6 7 | |
| a100-st-p4d24xlarge-47:40610:41147 [0] NCCL INFO Channel 04/24 : 0 1 2 3 4 5 6 7 | |
| a100-st-p4d24xlarge-47:40610:41147 [0] NCCL INFO Channel 05/24 : 0 1 2 3 4 5 6 7 | |
| a100-st-p4d24xlarge-47:40610:41147 [0] NCCL INFO Channel 06/24 : 0 1 2 3 4 5 6 7 | |
| a100-st-p4d24xlarge-47:40610:41147 [0] NCCL INFO Channel 07/24 : 0 1 2 3 4 5 6 7 | |
| a100-st-p4d24xlarge-47:40610:41147 [0] NCCL INFO Channel 08/24 : 0 1 2 3 4 5 6 7 | |
| a100-st-p4d24xlarge-47:40610:41147 [0] NCCL INFO Channel 09/24 : 0 1 2 3 4 5 6 7 | |
| a100-st-p4d24xlarge-47:40610:41147 [0] NCCL INFO Channel 10/24 : 0 1 2 3 4 5 6 7 | |
| a100-st-p4d24xlarge-47:40610:41147 [0] NCCL INFO Channel 11/24 : 0 1 2 3 4 5 6 7 | |
| a100-st-p4d24xlarge-47:40610:41147 [0] NCCL INFO Channel 12/24 : 0 1 2 3 4 5 6 7 | |
| a100-st-p4d24xlarge-47:40610:41147 [0] NCCL INFO Channel 13/24 : 0 1 2 3 4 5 6 7 | |
| a100-st-p4d24xlarge-47:40610:41147 [0] NCCL INFO Channel 14/24 : 0 1 2 3 4 5 6 7 | |
| a100-st-p4d24xlarge-47:40610:41147 [0] NCCL INFO Channel 15/24 : 0 1 2 3 4 5 6 7 | |
| a100-st-p4d24xlarge-47:40610:41147 [0] NCCL INFO Channel 16/24 : 0 1 2 3 4 5 6 7 | |
| a100-st-p4d24xlarge-47:40610:41147 [0] NCCL INFO Channel 17/24 : 0 1 2 3 4 5 6 7 | |
| a100-st-p4d24xlarge-47:40610:41147 [0] NCCL INFO Channel 18/24 : 0 1 2 3 4 5 6 7 | |
| a100-st-p4d24xlarge-47:40610:41147 [0] NCCL INFO Channel 19/24 : 0 1 2 3 4 5 6 7 | |
| a100-st-p4d24xlarge-47:40610:41147 [0] NCCL INFO Channel 20/24 : 0 1 2 3 4 5 6 7 | |
| a100-st-p4d24xlarge-47:40610:41147 [0] NCCL INFO Channel 21/24 : 0 1 2 3 4 5 6 7 | |
| a100-st-p4d24xlarge-47:40610:41147 [0] NCCL INFO Channel 22/24 : 0 1 2 3 4 5 6 7 | |
| a100-st-p4d24xlarge-47:40610:41147 [0] NCCL INFO Channel 23/24 : 0 1 2 3 4 5 6 7 | |
| a100-st-p4d24xlarge-47:40610:41147 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1 [2] 1/-1/-1->0->-1 [3] 1/-1/-1->0->-1 [4] 1/-1/-1->0->-1 [5] 1/-1/-1->0->-1 [6] 1/-1/-1->0->-1 [7] 1/-1/-1->0->-1 [8] 1/-1/-1->0->-1 [9] 1/-1/-1->0->-1 [10] 1/-1/-1->0->-1 [11] 1/-1/-1->0->-1 [12] 1/-1/-1->0->-1 [13] 1/-1/-1->0->-1 [14] 1/-1/-1->0->-1 [15] 1/-1/-1->0->-1 [16] 1/-1/-1->0->-1 [17] 1/-1/-1->0->-1 [18] 1/-1/-1->0->-1 [19] 1/-1/-1->0->-1 [20] 1/-1/-1->0->-1 [21] 1/-1/-1->0->-1 [22] 1/-1/-1->0->-1 [23] 1/-1/-1->0->-1 | |
| a100-st-p4d24xlarge-47:40610:41147 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2c8002dd0 | |
| a100-st-p4d24xlarge-47:40610:41202 [0] NCCL INFO New proxy recv connection 0 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:40610:41147 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2c8002e10 | |
| a100-st-p4d24xlarge-47:40610:41202 [0] NCCL INFO New proxy recv connection 1 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:40610:41202 [0] NCCL INFO New proxy recv connection 2 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:40610:41147 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2c8002e50 | |
| a100-st-p4d24xlarge-47:40610:41202 [0] NCCL INFO New proxy recv connection 3 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:40610:41147 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2c8002e90 | |
| a100-st-p4d24xlarge-47:40610:41202 [0] NCCL INFO New proxy recv connection 4 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:40610:41147 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2c8002ed0 | |
| a100-st-p4d24xlarge-47:40610:41202 [0] NCCL INFO New proxy recv connection 5 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:40610:41147 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2c8002f10 | |
| a100-st-p4d24xlarge-47:40610:41202 [0] NCCL INFO New proxy recv connection 6 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:40610:41147 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2c8002f50 | |
| a100-st-p4d24xlarge-47:40610:41202 [0] NCCL INFO New proxy recv connection 7 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:40610:41147 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2c8002f90 | |
| a100-st-p4d24xlarge-47:40610:41202 [0] NCCL INFO New proxy recv connection 8 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:40610:41147 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2c8002fd0 | |
| a100-st-p4d24xlarge-47:40610:41202 [0] NCCL INFO New proxy recv connection 9 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:40610:41147 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2c8003010 | |
| a100-st-p4d24xlarge-47:40610:41202 [0] NCCL INFO New proxy recv connection 10 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:40610:41147 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2c8003050 | |
| a100-st-p4d24xlarge-47:40610:41147 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2c8003090 | |
| a100-st-p4d24xlarge-47:40610:41202 [0] NCCL INFO New proxy recv connection 11 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:40610:41147 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2c80030d0 | |
| a100-st-p4d24xlarge-47:40610:41202 [0] NCCL INFO New proxy recv connection 12 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:40610:41147 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2c8003110 | |
| a100-st-p4d24xlarge-47:40610:41202 [0] NCCL INFO New proxy recv connection 13 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:40610:41147 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2c8003150 | |
| a100-st-p4d24xlarge-47:40610:41202 [0] NCCL INFO New proxy recv connection 14 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:40610:41147 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2c8003190 | |
| a100-st-p4d24xlarge-47:40610:41202 [0] NCCL INFO New proxy recv connection 15 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:40610:41147 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2c80031d0 | |
| a100-st-p4d24xlarge-47:40610:41202 [0] NCCL INFO New proxy recv connection 16 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:40610:41147 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2c8003210 | |
| a100-st-p4d24xlarge-47:40610:41202 [0] NCCL INFO New proxy recv connection 17 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:40610:41147 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2c8003250 | |
| a100-st-p4d24xlarge-47:40610:41202 [0] NCCL INFO New proxy recv connection 18 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:40610:41147 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2c8003290 | |
| a100-st-p4d24xlarge-47:40610:41202 [0] NCCL INFO New proxy recv connection 19 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:40610:41147 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2c80032d0 | |
| a100-st-p4d24xlarge-47:40610:41202 [0] NCCL INFO New proxy recv connection 20 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:40610:41147 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2c8003310 | |
| a100-st-p4d24xlarge-47:40610:41202 [0] NCCL INFO New proxy recv connection 21 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:40610:41147 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2c8003350 | |
| a100-st-p4d24xlarge-47:40610:41202 [0] NCCL INFO New proxy recv connection 22 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:40610:41147 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2c8003390 | |
| a100-st-p4d24xlarge-47:40610:41202 [0] NCCL INFO New proxy recv connection 23 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:40610:41147 [0] NCCL INFO Channel 00/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-47:40610:41147 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2c80033d0 | |
| a100-st-p4d24xlarge-47:40610:41202 [0] NCCL INFO New proxy send connection 24 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:40610:41147 [0] NCCL INFO Channel 01/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-47:40610:41147 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2c8003410 | |
| a100-st-p4d24xlarge-47:40610:41202 [0] NCCL INFO New proxy send connection 25 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:40610:41147 [0] NCCL INFO Channel 02/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-47:40610:41147 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2c8003450 | |
| a100-st-p4d24xlarge-47:40610:41202 [0] NCCL INFO New proxy send connection 26 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:40610:41147 [0] NCCL INFO Channel 03/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-47:40610:41147 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2c8003490 | |
| a100-st-p4d24xlarge-47:40610:41202 [0] NCCL INFO New proxy send connection 27 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:40610:41147 [0] NCCL INFO Channel 04/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-47:40610:41147 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2c80034d0 | |
| a100-st-p4d24xlarge-47:40610:41202 [0] NCCL INFO New proxy send connection 28 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:40610:41147 [0] NCCL INFO Channel 05/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-47:40610:41147 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2c8003510 | |
| a100-st-p4d24xlarge-47:40610:41202 [0] NCCL INFO New proxy send connection 29 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:40610:41147 [0] NCCL INFO Channel 06/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-47:40610:41147 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2c8003550 | |
| a100-st-p4d24xlarge-47:40610:41202 [0] NCCL INFO New proxy send connection 30 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:40610:41147 [0] NCCL INFO Channel 07/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-47:40610:41147 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2c8003590 | |
| a100-st-p4d24xlarge-47:40610:41202 [0] NCCL INFO New proxy send connection 31 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:40610:41147 [0] NCCL INFO Channel 08/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-47:40610:41147 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2c80035d0 | |
| a100-st-p4d24xlarge-47:40610:41202 [0] NCCL INFO New proxy send connection 32 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:40610:41147 [0] NCCL INFO Channel 09/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-47:40610:41147 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2c8003610 | |
| a100-st-p4d24xlarge-47:40610:41202 [0] NCCL INFO New proxy send connection 33 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:40610:41147 [0] NCCL INFO Channel 10/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-47:40610:41147 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2c8003650 | |
| a100-st-p4d24xlarge-47:40610:41202 [0] NCCL INFO New proxy send connection 34 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:40610:41147 [0] NCCL INFO Channel 11/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-47:40610:41147 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2c8003690 | |
| a100-st-p4d24xlarge-47:40610:41202 [0] NCCL INFO New proxy send connection 35 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:40610:41147 [0] NCCL INFO Channel 12/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-47:40610:41147 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2c80036d0 | |
| a100-st-p4d24xlarge-47:40610:41202 [0] NCCL INFO New proxy send connection 36 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:40610:41147 [0] NCCL INFO Channel 13/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-47:40610:41147 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2c8003710 | |
| a100-st-p4d24xlarge-47:40610:41202 [0] NCCL INFO New proxy send connection 37 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:40610:41147 [0] NCCL INFO Channel 14/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-47:40610:41147 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2c8003750 | |
| a100-st-p4d24xlarge-47:40610:41202 [0] NCCL INFO New proxy send connection 38 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:40610:41147 [0] NCCL INFO Channel 15/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-47:40610:41147 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2c8003790 | |
| a100-st-p4d24xlarge-47:40610:41202 [0] NCCL INFO New proxy send connection 39 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:40610:41147 [0] NCCL INFO Channel 16/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-47:40610:41147 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2c80037d0 | |
| a100-st-p4d24xlarge-47:40610:41202 [0] NCCL INFO New proxy send connection 40 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:40610:41147 [0] NCCL INFO Channel 17/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-47:40610:41147 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2c8003810 | |
| a100-st-p4d24xlarge-47:40610:41202 [0] NCCL INFO New proxy send connection 41 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:40610:41147 [0] NCCL INFO Channel 18/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-47:40610:41147 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2c8003850 | |
| a100-st-p4d24xlarge-47:40610:41202 [0] NCCL INFO New proxy send connection 42 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:40610:41147 [0] NCCL INFO Channel 19/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-47:40610:41147 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2c8003890 | |
| a100-st-p4d24xlarge-47:40610:41202 [0] NCCL INFO New proxy send connection 43 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:40610:41147 [0] NCCL INFO Channel 20/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-47:40610:41147 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2c80038d0 | |
| a100-st-p4d24xlarge-47:40610:41202 [0] NCCL INFO New proxy send connection 44 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:40610:41147 [0] NCCL INFO Channel 21/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-47:40610:41147 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2c8003910 | |
| a100-st-p4d24xlarge-47:40610:41202 [0] NCCL INFO New proxy send connection 45 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:40610:41147 [0] NCCL INFO Channel 22/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-47:40610:41147 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2c8003950 | |
| a100-st-p4d24xlarge-47:40610:41202 [0] NCCL INFO New proxy send connection 46 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:40610:41147 [0] NCCL INFO Channel 23/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-47:40610:41147 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2c8003990 | |
| a100-st-p4d24xlarge-47:40610:41202 [0] NCCL INFO New proxy send connection 47 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:40610:41147 [0] NCCL INFO Connected all rings | |
| a100-st-p4d24xlarge-47:40610:41147 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2c80039d0 | |
| a100-st-p4d24xlarge-47:40610:41202 [0] NCCL INFO New proxy recv connection 48 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:40610:41147 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2c8003a10 | |
| a100-st-p4d24xlarge-47:40610:41202 [0] NCCL INFO New proxy recv connection 49 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:40610:41202 [0] NCCL INFO New proxy recv connection 50 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:40610:41147 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2c8003a50 | |
| a100-st-p4d24xlarge-47:40610:41147 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2c8003a90 | |
| a100-st-p4d24xlarge-47:40610:41202 [0] NCCL INFO New proxy recv connection 51 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:40610:41147 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2c8003ad0 | |
| a100-st-p4d24xlarge-47:40610:41202 [0] NCCL INFO New proxy recv connection 52 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:40610:41147 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2c8003b10 | |
| a100-st-p4d24xlarge-47:40610:41202 [0] NCCL INFO New proxy recv connection 53 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:40610:41147 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2c8003b50 | |
| a100-st-p4d24xlarge-47:40610:41202 [0] NCCL INFO New proxy recv connection 54 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:40610:41147 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2c8003b90 | |
| a100-st-p4d24xlarge-47:40610:41202 [0] NCCL INFO New proxy recv connection 55 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:40610:41147 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2c8003bd0 | |
| a100-st-p4d24xlarge-47:40610:41202 [0] NCCL INFO New proxy recv connection 56 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:40610:41147 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2c8003c10 | |
| a100-st-p4d24xlarge-47:40610:41202 [0] NCCL INFO New proxy recv connection 57 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:40610:41147 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2c8003c50 | |
| a100-st-p4d24xlarge-47:40610:41202 [0] NCCL INFO New proxy recv connection 58 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:40610:41147 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2c8003c90 | |
| a100-st-p4d24xlarge-47:40610:41202 [0] NCCL INFO New proxy recv connection 59 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:40610:41147 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2c8003cd0 | |
| a100-st-p4d24xlarge-47:40610:41202 [0] NCCL INFO New proxy recv connection 60 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:40610:41147 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2c8003d10 | |
| a100-st-p4d24xlarge-47:40610:41202 [0] NCCL INFO New proxy recv connection 61 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:40610:41147 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2c8003d50 | |
| a100-st-p4d24xlarge-47:40610:41202 [0] NCCL INFO New proxy recv connection 62 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:40610:41147 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2c8003d90 | |
| a100-st-p4d24xlarge-47:40610:41202 [0] NCCL INFO New proxy recv connection 63 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:40610:41147 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2c8003dd0 | |
| a100-st-p4d24xlarge-47:40610:41202 [0] NCCL INFO New proxy recv connection 64 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:40610:41147 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2c8003e10 | |
| a100-st-p4d24xlarge-47:40610:41202 [0] NCCL INFO New proxy recv connection 65 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:40610:41147 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2c8003e50 | |
| a100-st-p4d24xlarge-47:40610:41202 [0] NCCL INFO New proxy recv connection 66 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:40610:41147 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2c8003e90 | |
| a100-st-p4d24xlarge-47:40610:41202 [0] NCCL INFO New proxy recv connection 67 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:40610:41147 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2c8003ed0 | |
| a100-st-p4d24xlarge-47:40610:41202 [0] NCCL INFO New proxy recv connection 68 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:40610:41147 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2c8003f10 | |
| a100-st-p4d24xlarge-47:40610:41202 [0] NCCL INFO New proxy recv connection 69 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:40610:41147 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2c8003f50 | |
| a100-st-p4d24xlarge-47:40610:41202 [0] NCCL INFO New proxy recv connection 70 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:40610:41147 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2c8003f90 | |
| a100-st-p4d24xlarge-47:40610:41202 [0] NCCL INFO New proxy recv connection 71 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:40610:41147 [0] NCCL INFO Connected all trees | |
| a100-st-p4d24xlarge-47:40610:41147 [0] NCCL INFO NCCL_ALGO set by environment to ring | |
| a100-st-p4d24xlarge-47:40610:41147 [0] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512 | |
| a100-st-p4d24xlarge-47:40610:41147 [0] NCCL INFO 24 coll channels, 32 p2p channels, 32 p2p channels per peer | |
| a100-st-p4d24xlarge-47:40610:41202 [0] NCCL INFO New proxy send connection 72 from local rank 0, transport 2 | |
| a100-st-p4d24xlarge-47:40610:41147 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2c8003fd0 | |
| a100-st-p4d24xlarge-47:40610:41147 [0] NCCL INFO comm 0x563c6a5c7760 rank 0 nranks 8 cudaDev 0 busId 101c0 - Init COMPLETE | |
| didn't get the result, queue empty | |
| result {'latency_median': 62.83059310913086, 'latency_stdev': 0.47123895276864264} | |
| has been placed into the queue | |
| got exit 0 | |
| got the result! | |
| exit code: 0 and result: {'nodes': 1, 'model_name': 'torchbenchmark.models.resnet50.Model', 'backend': 'eager', 'has_breaks': False, 'static_graph': False, 'result': {'latency_median': 62.83059310913086, 'latency_stdev': 0.47123895276864264}} | |
| <RESULT>{"nodes": 1, "model_name": "torchbenchmark.models.resnet50.Model", "backend": "eager", "has_breaks": false, "static_graph": false, "result": {"latency_median": 62.83059310913086, "latency_stdev": 0.47123895276864264}}</RESULT> | |
| run_once | |
| Process group: 8 tasks, rank: 0 | |
| MY HOSTNAME: a100-st-p4d24xlarge-47 | |
| FI_PROVIDER : efa | |
| LD_LIBRARY_PATH : /fsx/users/dberard/scratch-local/bench-fast/aws-ofi-nccl/lib:/opt/amazon/efa/lib:/usr/local/cuda-11.6/lib:/usr/local/cuda-11.6/lib64:/opt/amazon/efa/lib:/opt/amazon/openmpi/lib:/usr/local/cuda/efa/lib:/usr/local/cuda/lib:/usr/local/cuda:/usr/local/cuda/lib64:/usr/local/cuda/extras/CUPTI/lib64:/usr/local/cuda/targets/x86_64-linux/lib:/usr/local/lib:/usr/lib: | |
| NCCL_DEBUG : INFO | |
| FI_EFA_USE_DEVICE_RDMA : 1 | |
| still waiting for process to exit | |
| didn't get the result, queue empty | |
| still waiting for process to exit | |
| a100-st-p4d24xlarge-47:41346:41346 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to ens | |
| a100-st-p4d24xlarge-47:41346:41346 [0] NCCL INFO NCCL_SOCKET_IFNAME set to ens | |
| a100-st-p4d24xlarge-47:41346:41346 [0] NCCL INFO Bootstrap : Using ens32:10.200.88.223<0> | |
| a100-st-p4d24xlarge-47:41346:41346 [0] NCCL INFO NET/Plugin: Failed to find ncclCollNetPlugin_v6 symbol. | |
| a100-st-p4d24xlarge-47:41346:41346 [0] NCCL INFO NET/Plugin: Failed to find ncclCollNetPlugin symbol (v4 or v5). | |
| a100-st-p4d24xlarge-47:41346:41346 [0] NCCL INFO cudaDriverVersion 11060 | |
| NCCL version 2.14.3+cuda11.6 | |
| a100-st-p4d24xlarge-47:41346:42475 [0] NCCL INFO NET/OFI Using aws-ofi-nccl 1.4.0aws | |
| a100-st-p4d24xlarge-47:41346:42475 [0] NCCL INFO NET/OFI Running on p4d.24xlarge platform, Setting NCCL_TOPO_FILE environment variable to /fsx/users/dberard/scratch-local/bench-fast/aws-ofi-nccl/share/aws-ofi-nccl/xml/p4d-24xl-topo.xml | |
| a100-st-p4d24xlarge-47:41346:42475 [0] NCCL INFO NET/OFI Setting FI_EFA_FORK_SAFE environment variable to 1 | |
| a100-st-p4d24xlarge-47:41346:42475 [0] NCCL INFO NET/OFI Selected Provider is efa | |
| a100-st-p4d24xlarge-47:41346:42475 [0] NCCL INFO Using network AWS Libfabric | |
| a100-st-p4d24xlarge-47:41346:42475 [0] NCCL INFO NCCL_TOPO_FILE set by environment to /fsx/users/dberard/scratch-local/bench-fast/aws-ofi-nccl/share/aws-ofi-nccl/xml/p4d-24xl-topo.xml | |
| a100-st-p4d24xlarge-47:41346:42475 [0] NCCL INFO NET/AWS Libfabric : GPU Direct RDMA Enabled for HCA 0 'rdmap16s27' | |
| a100-st-p4d24xlarge-47:41346:42475 [0] NCCL INFO NET/AWS Libfabric : GPU Direct RDMA Enabled for HCA 1 'rdmap32s27' | |
| a100-st-p4d24xlarge-47:41346:42475 [0] NCCL INFO NET/AWS Libfabric : GPU Direct RDMA Enabled for HCA 2 'rdmap144s27' | |
| a100-st-p4d24xlarge-47:41346:42475 [0] NCCL INFO NET/AWS Libfabric : GPU Direct RDMA Enabled for HCA 3 'rdmap160s27' | |
| a100-st-p4d24xlarge-47:41346:42475 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 101c0 / HCA 0 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-47:41346:42475 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 101d0 / HCA 0 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-47:41346:42475 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 201c0 / HCA 0 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-47:41346:42475 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 201d0 / HCA 0 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-47:41346:42475 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 901c0 / HCA 0 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-47:41346:42475 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 901d0 / HCA 0 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-47:41346:42475 [0] NCCL INFO GPU Direct RDMA Enabled for GPU a01c0 / HCA 0 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-47:41346:42475 [0] NCCL INFO GPU Direct RDMA Enabled for GPU a01d0 / HCA 0 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-47:41346:42475 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 101c0 / HCA 1 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-47:41346:42475 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 101d0 / HCA 1 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-47:41346:42475 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 201c0 / HCA 1 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-47:41346:42475 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 201d0 / HCA 1 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-47:41346:42475 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 901c0 / HCA 1 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-47:41346:42475 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 901d0 / HCA 1 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-47:41346:42475 [0] NCCL INFO GPU Direct RDMA Enabled for GPU a01c0 / HCA 1 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-47:41346:42475 [0] NCCL INFO GPU Direct RDMA Enabled for GPU a01d0 / HCA 1 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-47:41346:42475 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 101c0 / HCA 2 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-47:41346:42475 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 101d0 / HCA 2 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-47:41346:42475 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 201c0 / HCA 2 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-47:41346:42475 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 201d0 / HCA 2 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-47:41346:42475 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 901c0 / HCA 2 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-47:41346:42475 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 901d0 / HCA 2 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-47:41346:42475 [0] NCCL INFO GPU Direct RDMA Enabled for GPU a01c0 / HCA 2 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-47:41346:42475 [0] NCCL INFO GPU Direct RDMA Enabled for GPU a01d0 / HCA 2 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-47:41346:42475 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 101c0 / HCA 3 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-47:41346:42475 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 101d0 / HCA 3 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-47:41346:42475 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 201c0 / HCA 3 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-47:41346:42475 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 201d0 / HCA 3 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-47:41346:42475 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 901c0 / HCA 3 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-47:41346:42475 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 901d0 / HCA 3 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-47:41346:42475 [0] NCCL INFO GPU Direct RDMA Enabled for GPU a01c0 / HCA 3 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-47:41346:42475 [0] NCCL INFO GPU Direct RDMA Enabled for GPU a01d0 / HCA 3 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-47:41346:42475 [0] NCCL INFO Setting affinity for GPU 0 to 1f0000,0000001f | |
| a100-st-p4d24xlarge-47:41346:42475 [0] NCCL INFO Channel 00/24 : 0 1 2 3 4 5 6 7 | |
| a100-st-p4d24xlarge-47:41346:42475 [0] NCCL INFO Channel 01/24 : 0 1 2 3 4 5 6 7 | |
| a100-st-p4d24xlarge-47:41346:42475 [0] NCCL INFO Channel 02/24 : 0 1 2 3 4 5 6 7 | |
| a100-st-p4d24xlarge-47:41346:42475 [0] NCCL INFO Channel 03/24 : 0 1 2 3 4 5 6 7 | |
| a100-st-p4d24xlarge-47:41346:42475 [0] NCCL INFO Channel 04/24 : 0 1 2 3 4 5 6 7 | |
| a100-st-p4d24xlarge-47:41346:42475 [0] NCCL INFO Channel 05/24 : 0 1 2 3 4 5 6 7 | |
| a100-st-p4d24xlarge-47:41346:42475 [0] NCCL INFO Channel 06/24 : 0 1 2 3 4 5 6 7 | |
| a100-st-p4d24xlarge-47:41346:42475 [0] NCCL INFO Channel 07/24 : 0 1 2 3 4 5 6 7 | |
| a100-st-p4d24xlarge-47:41346:42475 [0] NCCL INFO Channel 08/24 : 0 1 2 3 4 5 6 7 | |
| a100-st-p4d24xlarge-47:41346:42475 [0] NCCL INFO Channel 09/24 : 0 1 2 3 4 5 6 7 | |
| a100-st-p4d24xlarge-47:41346:42475 [0] NCCL INFO Channel 10/24 : 0 1 2 3 4 5 6 7 | |
| a100-st-p4d24xlarge-47:41346:42475 [0] NCCL INFO Channel 11/24 : 0 1 2 3 4 5 6 7 | |
| a100-st-p4d24xlarge-47:41346:42475 [0] NCCL INFO Channel 12/24 : 0 1 2 3 4 5 6 7 | |
| a100-st-p4d24xlarge-47:41346:42475 [0] NCCL INFO Channel 13/24 : 0 1 2 3 4 5 6 7 | |
| a100-st-p4d24xlarge-47:41346:42475 [0] NCCL INFO Channel 14/24 : 0 1 2 3 4 5 6 7 | |
| a100-st-p4d24xlarge-47:41346:42475 [0] NCCL INFO Channel 15/24 : 0 1 2 3 4 5 6 7 | |
| a100-st-p4d24xlarge-47:41346:42475 [0] NCCL INFO Channel 16/24 : 0 1 2 3 4 5 6 7 | |
| a100-st-p4d24xlarge-47:41346:42475 [0] NCCL INFO Channel 17/24 : 0 1 2 3 4 5 6 7 | |
| a100-st-p4d24xlarge-47:41346:42475 [0] NCCL INFO Channel 18/24 : 0 1 2 3 4 5 6 7 | |
| a100-st-p4d24xlarge-47:41346:42475 [0] NCCL INFO Channel 19/24 : 0 1 2 3 4 5 6 7 | |
| a100-st-p4d24xlarge-47:41346:42475 [0] NCCL INFO Channel 20/24 : 0 1 2 3 4 5 6 7 | |
| a100-st-p4d24xlarge-47:41346:42475 [0] NCCL INFO Channel 21/24 : 0 1 2 3 4 5 6 7 | |
| a100-st-p4d24xlarge-47:41346:42475 [0] NCCL INFO Channel 22/24 : 0 1 2 3 4 5 6 7 | |
| a100-st-p4d24xlarge-47:41346:42475 [0] NCCL INFO Channel 23/24 : 0 1 2 3 4 5 6 7 | |
| a100-st-p4d24xlarge-47:41346:42475 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1 [2] 1/-1/-1->0->-1 [3] 1/-1/-1->0->-1 [4] 1/-1/-1->0->-1 [5] 1/-1/-1->0->-1 [6] 1/-1/-1->0->-1 [7] 1/-1/-1->0->-1 [8] 1/-1/-1->0->-1 [9] 1/-1/-1->0->-1 [10] 1/-1/-1->0->-1 [11] 1/-1/-1->0->-1 [12] 1/-1/-1->0->-1 [13] 1/-1/-1->0->-1 [14] 1/-1/-1->0->-1 [15] 1/-1/-1->0->-1 [16] 1/-1/-1->0->-1 [17] 1/-1/-1->0->-1 [18] 1/-1/-1->0->-1 [19] 1/-1/-1->0->-1 [20] 1/-1/-1->0->-1 [21] 1/-1/-1->0->-1 [22] 1/-1/-1->0->-1 [23] 1/-1/-1->0->-1 | |
| a100-st-p4d24xlarge-47:41346:42475 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2c8002dd0 | |
| a100-st-p4d24xlarge-47:41346:42521 [0] NCCL INFO New proxy recv connection 0 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:41346:42475 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2c8002e10 | |
| a100-st-p4d24xlarge-47:41346:42521 [0] NCCL INFO New proxy recv connection 1 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:41346:42521 [0] NCCL INFO New proxy recv connection 2 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:41346:42475 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2c8002e50 | |
| a100-st-p4d24xlarge-47:41346:42521 [0] NCCL INFO New proxy recv connection 3 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:41346:42475 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2c8002e90 | |
| a100-st-p4d24xlarge-47:41346:42521 [0] NCCL INFO New proxy recv connection 4 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:41346:42475 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2c8002ed0 | |
| a100-st-p4d24xlarge-47:41346:42521 [0] NCCL INFO New proxy recv connection 5 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:41346:42475 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2c8002f10 | |
| a100-st-p4d24xlarge-47:41346:42521 [0] NCCL INFO New proxy recv connection 6 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:41346:42475 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2c8002f50 | |
| a100-st-p4d24xlarge-47:41346:42475 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2c8002f90 | |
| a100-st-p4d24xlarge-47:41346:42521 [0] NCCL INFO New proxy recv connection 7 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:41346:42475 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2c8002fd0 | |
| a100-st-p4d24xlarge-47:41346:42521 [0] NCCL INFO New proxy recv connection 8 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:41346:42475 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2c8003010 | |
| a100-st-p4d24xlarge-47:41346:42521 [0] NCCL INFO New proxy recv connection 9 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:41346:42475 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2c8003050 | |
| a100-st-p4d24xlarge-47:41346:42521 [0] NCCL INFO New proxy recv connection 10 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:41346:42475 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2c8003090 | |
| a100-st-p4d24xlarge-47:41346:42521 [0] NCCL INFO New proxy recv connection 11 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:41346:42475 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2c80030d0 | |
| a100-st-p4d24xlarge-47:41346:42521 [0] NCCL INFO New proxy recv connection 12 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:41346:42475 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2c8003110 | |
| a100-st-p4d24xlarge-47:41346:42521 [0] NCCL INFO New proxy recv connection 13 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:41346:42475 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2c8003150 | |
| a100-st-p4d24xlarge-47:41346:42521 [0] NCCL INFO New proxy recv connection 14 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:41346:42475 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2c8003190 | |
| a100-st-p4d24xlarge-47:41346:42521 [0] NCCL INFO New proxy recv connection 15 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:41346:42475 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2c80031d0 | |
| a100-st-p4d24xlarge-47:41346:42521 [0] NCCL INFO New proxy recv connection 16 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:41346:42475 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2c8003210 | |
| a100-st-p4d24xlarge-47:41346:42521 [0] NCCL INFO New proxy recv connection 17 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:41346:42475 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2c8003250 | |
| a100-st-p4d24xlarge-47:41346:42521 [0] NCCL INFO New proxy recv connection 18 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:41346:42475 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2c8003290 | |
| a100-st-p4d24xlarge-47:41346:42521 [0] NCCL INFO New proxy recv connection 19 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:41346:42475 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2c80032d0 | |
| a100-st-p4d24xlarge-47:41346:42521 [0] NCCL INFO New proxy recv connection 20 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:41346:42475 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2c8003310 | |
| a100-st-p4d24xlarge-47:41346:42521 [0] NCCL INFO New proxy recv connection 21 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:41346:42475 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2c8003350 | |
| a100-st-p4d24xlarge-47:41346:42521 [0] NCCL INFO New proxy recv connection 22 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:41346:42475 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2c8003390 | |
| a100-st-p4d24xlarge-47:41346:42521 [0] NCCL INFO New proxy recv connection 23 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:41346:42475 [0] NCCL INFO Channel 00/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-47:41346:42475 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2c80033d0 | |
| a100-st-p4d24xlarge-47:41346:42521 [0] NCCL INFO New proxy send connection 24 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:41346:42475 [0] NCCL INFO Channel 01/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-47:41346:42475 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2c8003410 | |
| a100-st-p4d24xlarge-47:41346:42521 [0] NCCL INFO New proxy send connection 25 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:41346:42475 [0] NCCL INFO Channel 02/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-47:41346:42475 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2c8003450 | |
| a100-st-p4d24xlarge-47:41346:42521 [0] NCCL INFO New proxy send connection 26 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:41346:42475 [0] NCCL INFO Channel 03/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-47:41346:42475 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2c8003490 | |
| a100-st-p4d24xlarge-47:41346:42521 [0] NCCL INFO New proxy send connection 27 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:41346:42475 [0] NCCL INFO Channel 04/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-47:41346:42475 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2c80034d0 | |
| a100-st-p4d24xlarge-47:41346:42521 [0] NCCL INFO New proxy send connection 28 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:41346:42475 [0] NCCL INFO Channel 05/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-47:41346:42475 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2c8003510 | |
| a100-st-p4d24xlarge-47:41346:42521 [0] NCCL INFO New proxy send connection 29 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:41346:42475 [0] NCCL INFO Channel 06/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-47:41346:42475 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2c8003550 | |
| a100-st-p4d24xlarge-47:41346:42521 [0] NCCL INFO New proxy send connection 30 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:41346:42475 [0] NCCL INFO Channel 07/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-47:41346:42475 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2c8003590 | |
| a100-st-p4d24xlarge-47:41346:42521 [0] NCCL INFO New proxy send connection 31 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:41346:42475 [0] NCCL INFO Channel 08/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-47:41346:42475 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2c80035d0 | |
| a100-st-p4d24xlarge-47:41346:42521 [0] NCCL INFO New proxy send connection 32 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:41346:42475 [0] NCCL INFO Channel 09/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-47:41346:42475 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2c8003610 | |
| a100-st-p4d24xlarge-47:41346:42521 [0] NCCL INFO New proxy send connection 33 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:41346:42475 [0] NCCL INFO Channel 10/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-47:41346:42475 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2c8003650 | |
| a100-st-p4d24xlarge-47:41346:42521 [0] NCCL INFO New proxy send connection 34 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:41346:42475 [0] NCCL INFO Channel 11/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-47:41346:42475 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2c8003690 | |
| a100-st-p4d24xlarge-47:41346:42521 [0] NCCL INFO New proxy send connection 35 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:41346:42475 [0] NCCL INFO Channel 12/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-47:41346:42475 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2c80036d0 | |
| a100-st-p4d24xlarge-47:41346:42521 [0] NCCL INFO New proxy send connection 36 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:41346:42475 [0] NCCL INFO Channel 13/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-47:41346:42475 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2c8003710 | |
| a100-st-p4d24xlarge-47:41346:42521 [0] NCCL INFO New proxy send connection 37 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:41346:42475 [0] NCCL INFO Channel 14/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-47:41346:42475 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2c8003750 | |
| a100-st-p4d24xlarge-47:41346:42521 [0] NCCL INFO New proxy send connection 38 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:41346:42475 [0] NCCL INFO Channel 15/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-47:41346:42475 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2c8003790 | |
| a100-st-p4d24xlarge-47:41346:42521 [0] NCCL INFO New proxy send connection 39 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:41346:42475 [0] NCCL INFO Channel 16/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-47:41346:42475 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2c80037d0 | |
| a100-st-p4d24xlarge-47:41346:42521 [0] NCCL INFO New proxy send connection 40 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:41346:42475 [0] NCCL INFO Channel 17/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-47:41346:42475 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2c8003810 | |
| a100-st-p4d24xlarge-47:41346:42521 [0] NCCL INFO New proxy send connection 41 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:41346:42475 [0] NCCL INFO Channel 18/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-47:41346:42475 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2c8003850 | |
| a100-st-p4d24xlarge-47:41346:42521 [0] NCCL INFO New proxy send connection 42 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:41346:42475 [0] NCCL INFO Channel 19/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-47:41346:42475 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2c8003890 | |
| a100-st-p4d24xlarge-47:41346:42521 [0] NCCL INFO New proxy send connection 43 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:41346:42475 [0] NCCL INFO Channel 20/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-47:41346:42475 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2c80038d0 | |
| a100-st-p4d24xlarge-47:41346:42521 [0] NCCL INFO New proxy send connection 44 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:41346:42475 [0] NCCL INFO Channel 21/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-47:41346:42475 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2c8003910 | |
| a100-st-p4d24xlarge-47:41346:42521 [0] NCCL INFO New proxy send connection 45 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:41346:42475 [0] NCCL INFO Channel 22/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-47:41346:42475 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2c8003950 | |
| a100-st-p4d24xlarge-47:41346:42521 [0] NCCL INFO New proxy send connection 46 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:41346:42475 [0] NCCL INFO Channel 23/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-47:41346:42475 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2c8003990 | |
| a100-st-p4d24xlarge-47:41346:42521 [0] NCCL INFO New proxy send connection 47 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:41346:42475 [0] NCCL INFO Connected all rings | |
| a100-st-p4d24xlarge-47:41346:42475 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2c80039d0 | |
| a100-st-p4d24xlarge-47:41346:42521 [0] NCCL INFO New proxy recv connection 48 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:41346:42521 [0] NCCL INFO New proxy recv connection 49 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:41346:42475 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2c8003a10 | |
| a100-st-p4d24xlarge-47:41346:42521 [0] NCCL INFO New proxy recv connection 50 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:41346:42475 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2c8003a50 | |
| a100-st-p4d24xlarge-47:41346:42475 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2c8003a90 | |
| a100-st-p4d24xlarge-47:41346:42521 [0] NCCL INFO New proxy recv connection 51 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:41346:42475 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2c8003ad0 | |
| a100-st-p4d24xlarge-47:41346:42521 [0] NCCL INFO New proxy recv connection 52 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:41346:42475 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2c8003b10 | |
| a100-st-p4d24xlarge-47:41346:42521 [0] NCCL INFO New proxy recv connection 53 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:41346:42475 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2c8003b50 | |
| a100-st-p4d24xlarge-47:41346:42521 [0] NCCL INFO New proxy recv connection 54 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:41346:42475 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2c8003b90 | |
| a100-st-p4d24xlarge-47:41346:42521 [0] NCCL INFO New proxy recv connection 55 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:41346:42475 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2c8003bd0 | |
| a100-st-p4d24xlarge-47:41346:42521 [0] NCCL INFO New proxy recv connection 56 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:41346:42475 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2c8003c10 | |
| a100-st-p4d24xlarge-47:41346:42521 [0] NCCL INFO New proxy recv connection 57 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:41346:42475 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2c8003c50 | |
| a100-st-p4d24xlarge-47:41346:42521 [0] NCCL INFO New proxy recv connection 58 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:41346:42475 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2c8003c90 | |
| a100-st-p4d24xlarge-47:41346:42521 [0] NCCL INFO New proxy recv connection 59 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:41346:42475 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2c8003cd0 | |
| a100-st-p4d24xlarge-47:41346:42521 [0] NCCL INFO New proxy recv connection 60 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:41346:42475 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2c8003d10 | |
| a100-st-p4d24xlarge-47:41346:42521 [0] NCCL INFO New proxy recv connection 61 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:41346:42475 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2c8003d50 | |
| a100-st-p4d24xlarge-47:41346:42521 [0] NCCL INFO New proxy recv connection 62 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:41346:42475 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2c8003d90 | |
| a100-st-p4d24xlarge-47:41346:42521 [0] NCCL INFO New proxy recv connection 63 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:41346:42475 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2c8003dd0 | |
| a100-st-p4d24xlarge-47:41346:42521 [0] NCCL INFO New proxy recv connection 64 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:41346:42475 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2c8003e10 | |
| a100-st-p4d24xlarge-47:41346:42521 [0] NCCL INFO New proxy recv connection 65 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:41346:42475 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2c8003e50 | |
| a100-st-p4d24xlarge-47:41346:42521 [0] NCCL INFO New proxy recv connection 66 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:41346:42475 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2c8003e90 | |
| a100-st-p4d24xlarge-47:41346:42521 [0] NCCL INFO New proxy recv connection 67 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:41346:42475 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2c8003ed0 | |
| a100-st-p4d24xlarge-47:41346:42521 [0] NCCL INFO New proxy recv connection 68 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:41346:42475 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2c8003f10 | |
| a100-st-p4d24xlarge-47:41346:42521 [0] NCCL INFO New proxy recv connection 69 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:41346:42475 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2c8003f50 | |
| a100-st-p4d24xlarge-47:41346:42521 [0] NCCL INFO New proxy recv connection 70 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:41346:42475 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2c8003f90 | |
| a100-st-p4d24xlarge-47:41346:42521 [0] NCCL INFO New proxy recv connection 71 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:41346:42475 [0] NCCL INFO Connected all trees | |
| a100-st-p4d24xlarge-47:41346:42475 [0] NCCL INFO NCCL_ALGO set by environment to ring | |
| a100-st-p4d24xlarge-47:41346:42475 [0] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512 | |
| a100-st-p4d24xlarge-47:41346:42475 [0] NCCL INFO 24 coll channels, 32 p2p channels, 32 p2p channels per peer | |
| a100-st-p4d24xlarge-47:41346:42521 [0] NCCL INFO New proxy send connection 72 from local rank 0, transport 2 | |
| a100-st-p4d24xlarge-47:41346:42475 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2c8003fd0 | |
| a100-st-p4d24xlarge-47:41346:42475 [0] NCCL INFO comm 0x563c6c315c50 rank 0 nranks 8 cudaDev 0 busId 101c0 - Init COMPLETE | |
| didn't get the result, queue empty | |
| still waiting for process to exit | |
| didn't get the result, queue empty | |
| still waiting for process to exit | |
| didn't get the result, queue empty | |
| still waiting for process to exit | |
| didn't get the result, queue empty | |
| still waiting for process to exit | |
| didn't get the result, queue empty | |
| still waiting for process to exit | |
| didn't get the result, queue empty | |
| still waiting for process to exit | |
| didn't get the result, queue empty | |
| still waiting for process to exit | |
| didn't get the result, queue empty | |
| still waiting for process to exit | |
| didn't get the result, queue empty | |
| still waiting for process to exit | |
| didn't get the result, queue empty | |
| still waiting for process to exit | |
| didn't get the result, queue empty | |
| still waiting for process to exit | |
| didn't get the result, queue empty | |
| still waiting for process to exit | |
| didn't get the result, queue empty | |
| still waiting for process to exit | |
| didn't get the result, queue empty | |
| still waiting for process to exit | |
| didn't get the result, queue empty | |
| still waiting for process to exit | |
| didn't get the result, queue empty | |
| still waiting for process to exit | |
| didn't get the result, queue empty | |
| still waiting for process to exit | |
| didn't get the result, queue empty | |
| still waiting for process to exit | |
| didn't get the result, queue empty | |
| still waiting for process to exit | |
| didn't get the result, queue empty | |
| still waiting for process to exit | |
| didn't get the result, queue empty | |
| still waiting for process to exit | |
| didn't get the result, queue empty | |
| still waiting for process to exit | |
| didn't get the result, queue empty | |
| still waiting for process to exit | |
| didn't get the result, queue empty | |
| still waiting for process to exit | |
| didn't get the result, queue empty | |
| still waiting for process to exit | |
| didn't get the result, queue empty | |
| still waiting for process to exit | |
| didn't get the result, queue empty | |
| still waiting for process to exit | |
| didn't get the result, queue empty | |
| still waiting for process to exit | |
| didn't get the result, queue empty | |
| still waiting for process to exit | |
| didn't get the result, queue empty | |
| still waiting for process to exit | |
| didn't get the result, queue empty | |
| still waiting for process to exit | |
| didn't get the result, queue empty | |
| still waiting for process to exit | |
| didn't get the result, queue empty | |
| still waiting for process to exit | |
| didn't get the result, queue empty | |
| still waiting for process to exit | |
| didn't get the result, queue empty | |
| still waiting for process to exit | |
| didn't get the result, queue empty | |
| still waiting for process to exit | |
| didn't get the result, queue empty | |
| still waiting for process to exit | |
| didn't get the result, queue empty | |
| still waiting for process to exit | |
| didn't get the result, queue empty | |
| still waiting for process to exit | |
| didn't get the result, queue empty | |
| still waiting for process to exit | |
| didn't get the result, queue empty | |
| still waiting for process to exit | |
| didn't get the result, queue empty | |
| still waiting for process to exit | |
| didn't get the result, queue empty | |
| still waiting for process to exit | |
| didn't get the result, queue empty | |
| still waiting for process to exit | |
| didn't get the result, queue empty | |
| still waiting for process to exit | |
| didn't get the result, queue empty | |
| still waiting for process to exit | |
| didn't get the result, queue empty | |
| still waiting for process to exit | |
| didn't get the result, queue empty | |
| still waiting for process to exit | |
| didn't get the result, queue empty | |
| still waiting for process to exit | |
| didn't get the result, queue empty | |
| result {'latency_median': 80.85042953491211, 'latency_stdev': 0.2639016516493464} | |
| has been placed into the queue | |
| got exit 0 | |
| got the result! | |
| exit code: 0 and result: {'nodes': 1, 'model_name': 'torchbenchmark.models.resnet50.Model', 'backend': 'torchdynamo_inductor', 'has_breaks': False, 'static_graph': False, 'result': {'latency_median': 80.85042953491211, 'latency_stdev': 0.2639016516493464}} | |
| <RESULT>{"nodes": 1, "model_name": "torchbenchmark.models.resnet50.Model", "backend": "torchdynamo_inductor", "has_breaks": false, "static_graph": false, "result": {"latency_median": 80.85042953491211, "latency_stdev": 0.2639016516493464}}</RESULT> | |
| run_once | |
| Process group: 8 tasks, rank: 0 | |
| MY HOSTNAME: a100-st-p4d24xlarge-47 | |
| FI_PROVIDER : efa | |
| LD_LIBRARY_PATH : /fsx/users/dberard/scratch-local/bench-fast/aws-ofi-nccl/lib:/opt/amazon/efa/lib:/usr/local/cuda-11.6/lib:/usr/local/cuda-11.6/lib64:/opt/amazon/efa/lib:/opt/amazon/openmpi/lib:/usr/local/cuda/efa/lib:/usr/local/cuda/lib:/usr/local/cuda:/usr/local/cuda/lib64:/usr/local/cuda/extras/CUPTI/lib64:/usr/local/cuda/targets/x86_64-linux/lib:/usr/local/lib:/usr/lib: | |
| NCCL_DEBUG : INFO | |
| FI_EFA_USE_DEVICE_RDMA : 1 | |
| still waiting for process to exit | |
| didn't get the result, queue empty | |
| still waiting for process to exit | |
| a100-st-p4d24xlarge-47:87091:87091 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to ens | |
| a100-st-p4d24xlarge-47:87091:87091 [0] NCCL INFO NCCL_SOCKET_IFNAME set to ens | |
| a100-st-p4d24xlarge-47:87091:87091 [0] NCCL INFO Bootstrap : Using ens32:10.200.88.223<0> | |
| a100-st-p4d24xlarge-47:87091:87091 [0] NCCL INFO NET/Plugin: Failed to find ncclCollNetPlugin_v6 symbol. | |
| a100-st-p4d24xlarge-47:87091:87091 [0] NCCL INFO NET/Plugin: Failed to find ncclCollNetPlugin symbol (v4 or v5). | |
| a100-st-p4d24xlarge-47:87091:87091 [0] NCCL INFO cudaDriverVersion 11060 | |
| NCCL version 2.14.3+cuda11.6 | |
| a100-st-p4d24xlarge-47:87091:88189 [0] NCCL INFO NET/OFI Using aws-ofi-nccl 1.4.0aws | |
| a100-st-p4d24xlarge-47:87091:88189 [0] NCCL INFO NET/OFI Running on p4d.24xlarge platform, Setting NCCL_TOPO_FILE environment variable to /fsx/users/dberard/scratch-local/bench-fast/aws-ofi-nccl/share/aws-ofi-nccl/xml/p4d-24xl-topo.xml | |
| a100-st-p4d24xlarge-47:87091:88189 [0] NCCL INFO NET/OFI Setting FI_EFA_FORK_SAFE environment variable to 1 | |
| a100-st-p4d24xlarge-47:87091:88189 [0] NCCL INFO NET/OFI Selected Provider is efa | |
| a100-st-p4d24xlarge-47:87091:88189 [0] NCCL INFO Using network AWS Libfabric | |
| a100-st-p4d24xlarge-47:87091:88189 [0] NCCL INFO NCCL_TOPO_FILE set by environment to /fsx/users/dberard/scratch-local/bench-fast/aws-ofi-nccl/share/aws-ofi-nccl/xml/p4d-24xl-topo.xml | |
| a100-st-p4d24xlarge-47:87091:88189 [0] NCCL INFO NET/AWS Libfabric : GPU Direct RDMA Enabled for HCA 0 'rdmap16s27' | |
| a100-st-p4d24xlarge-47:87091:88189 [0] NCCL INFO NET/AWS Libfabric : GPU Direct RDMA Enabled for HCA 1 'rdmap32s27' | |
| a100-st-p4d24xlarge-47:87091:88189 [0] NCCL INFO NET/AWS Libfabric : GPU Direct RDMA Enabled for HCA 2 'rdmap144s27' | |
| a100-st-p4d24xlarge-47:87091:88189 [0] NCCL INFO NET/AWS Libfabric : GPU Direct RDMA Enabled for HCA 3 'rdmap160s27' | |
| a100-st-p4d24xlarge-47:87091:88189 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 101c0 / HCA 0 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-47:87091:88189 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 101d0 / HCA 0 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-47:87091:88189 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 201c0 / HCA 0 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-47:87091:88189 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 201d0 / HCA 0 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-47:87091:88189 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 901c0 / HCA 0 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-47:87091:88189 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 901d0 / HCA 0 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-47:87091:88189 [0] NCCL INFO GPU Direct RDMA Enabled for GPU a01c0 / HCA 0 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-47:87091:88189 [0] NCCL INFO GPU Direct RDMA Enabled for GPU a01d0 / HCA 0 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-47:87091:88189 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 101c0 / HCA 1 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-47:87091:88189 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 101d0 / HCA 1 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-47:87091:88189 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 201c0 / HCA 1 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-47:87091:88189 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 201d0 / HCA 1 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-47:87091:88189 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 901c0 / HCA 1 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-47:87091:88189 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 901d0 / HCA 1 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-47:87091:88189 [0] NCCL INFO GPU Direct RDMA Enabled for GPU a01c0 / HCA 1 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-47:87091:88189 [0] NCCL INFO GPU Direct RDMA Enabled for GPU a01d0 / HCA 1 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-47:87091:88189 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 101c0 / HCA 2 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-47:87091:88189 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 101d0 / HCA 2 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-47:87091:88189 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 201c0 / HCA 2 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-47:87091:88189 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 201d0 / HCA 2 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-47:87091:88189 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 901c0 / HCA 2 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-47:87091:88189 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 901d0 / HCA 2 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-47:87091:88189 [0] NCCL INFO GPU Direct RDMA Enabled for GPU a01c0 / HCA 2 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-47:87091:88189 [0] NCCL INFO GPU Direct RDMA Enabled for GPU a01d0 / HCA 2 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-47:87091:88189 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 101c0 / HCA 3 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-47:87091:88189 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 101d0 / HCA 3 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-47:87091:88189 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 201c0 / HCA 3 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-47:87091:88189 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 201d0 / HCA 3 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-47:87091:88189 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 901c0 / HCA 3 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-47:87091:88189 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 901d0 / HCA 3 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-47:87091:88189 [0] NCCL INFO GPU Direct RDMA Enabled for GPU a01c0 / HCA 3 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-47:87091:88189 [0] NCCL INFO GPU Direct RDMA Enabled for GPU a01d0 / HCA 3 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-47:87091:88189 [0] NCCL INFO Setting affinity for GPU 0 to 1f0000,0000001f | |
| a100-st-p4d24xlarge-47:87091:88189 [0] NCCL INFO Channel 00/24 : 0 1 2 3 4 5 6 7 | |
| a100-st-p4d24xlarge-47:87091:88189 [0] NCCL INFO Channel 01/24 : 0 1 2 3 4 5 6 7 | |
| a100-st-p4d24xlarge-47:87091:88189 [0] NCCL INFO Channel 02/24 : 0 1 2 3 4 5 6 7 | |
| a100-st-p4d24xlarge-47:87091:88189 [0] NCCL INFO Channel 03/24 : 0 1 2 3 4 5 6 7 | |
| a100-st-p4d24xlarge-47:87091:88189 [0] NCCL INFO Channel 04/24 : 0 1 2 3 4 5 6 7 | |
| a100-st-p4d24xlarge-47:87091:88189 [0] NCCL INFO Channel 05/24 : 0 1 2 3 4 5 6 7 | |
| a100-st-p4d24xlarge-47:87091:88189 [0] NCCL INFO Channel 06/24 : 0 1 2 3 4 5 6 7 | |
| a100-st-p4d24xlarge-47:87091:88189 [0] NCCL INFO Channel 07/24 : 0 1 2 3 4 5 6 7 | |
| a100-st-p4d24xlarge-47:87091:88189 [0] NCCL INFO Channel 08/24 : 0 1 2 3 4 5 6 7 | |
| a100-st-p4d24xlarge-47:87091:88189 [0] NCCL INFO Channel 09/24 : 0 1 2 3 4 5 6 7 | |
| a100-st-p4d24xlarge-47:87091:88189 [0] NCCL INFO Channel 10/24 : 0 1 2 3 4 5 6 7 | |
| a100-st-p4d24xlarge-47:87091:88189 [0] NCCL INFO Channel 11/24 : 0 1 2 3 4 5 6 7 | |
| a100-st-p4d24xlarge-47:87091:88189 [0] NCCL INFO Channel 12/24 : 0 1 2 3 4 5 6 7 | |
| a100-st-p4d24xlarge-47:87091:88189 [0] NCCL INFO Channel 13/24 : 0 1 2 3 4 5 6 7 | |
| a100-st-p4d24xlarge-47:87091:88189 [0] NCCL INFO Channel 14/24 : 0 1 2 3 4 5 6 7 | |
| a100-st-p4d24xlarge-47:87091:88189 [0] NCCL INFO Channel 15/24 : 0 1 2 3 4 5 6 7 | |
| a100-st-p4d24xlarge-47:87091:88189 [0] NCCL INFO Channel 16/24 : 0 1 2 3 4 5 6 7 | |
| a100-st-p4d24xlarge-47:87091:88189 [0] NCCL INFO Channel 17/24 : 0 1 2 3 4 5 6 7 | |
| a100-st-p4d24xlarge-47:87091:88189 [0] NCCL INFO Channel 18/24 : 0 1 2 3 4 5 6 7 | |
| a100-st-p4d24xlarge-47:87091:88189 [0] NCCL INFO Channel 19/24 : 0 1 2 3 4 5 6 7 | |
| a100-st-p4d24xlarge-47:87091:88189 [0] NCCL INFO Channel 20/24 : 0 1 2 3 4 5 6 7 | |
| a100-st-p4d24xlarge-47:87091:88189 [0] NCCL INFO Channel 21/24 : 0 1 2 3 4 5 6 7 | |
| a100-st-p4d24xlarge-47:87091:88189 [0] NCCL INFO Channel 22/24 : 0 1 2 3 4 5 6 7 | |
| a100-st-p4d24xlarge-47:87091:88189 [0] NCCL INFO Channel 23/24 : 0 1 2 3 4 5 6 7 | |
| a100-st-p4d24xlarge-47:87091:88189 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1 [2] 1/-1/-1->0->-1 [3] 1/-1/-1->0->-1 [4] 1/-1/-1->0->-1 [5] 1/-1/-1->0->-1 [6] 1/-1/-1->0->-1 [7] 1/-1/-1->0->-1 [8] 1/-1/-1->0->-1 [9] 1/-1/-1->0->-1 [10] 1/-1/-1->0->-1 [11] 1/-1/-1->0->-1 [12] 1/-1/-1->0->-1 [13] 1/-1/-1->0->-1 [14] 1/-1/-1->0->-1 [15] 1/-1/-1->0->-1 [16] 1/-1/-1->0->-1 [17] 1/-1/-1->0->-1 [18] 1/-1/-1->0->-1 [19] 1/-1/-1->0->-1 [20] 1/-1/-1->0->-1 [21] 1/-1/-1->0->-1 [22] 1/-1/-1->0->-1 [23] 1/-1/-1->0->-1 | |
| a100-st-p4d24xlarge-47:87091:88189 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2bc002dd0 | |
| a100-st-p4d24xlarge-47:87091:88232 [0] NCCL INFO New proxy recv connection 0 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:87091:88189 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2bc002e10 | |
| a100-st-p4d24xlarge-47:87091:88232 [0] NCCL INFO New proxy recv connection 1 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:87091:88232 [0] NCCL INFO New proxy recv connection 2 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:87091:88189 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2bc002e50 | |
| a100-st-p4d24xlarge-47:87091:88232 [0] NCCL INFO New proxy recv connection 3 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:87091:88189 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2bc002e90 | |
| a100-st-p4d24xlarge-47:87091:88232 [0] NCCL INFO New proxy recv connection 4 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:87091:88189 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2bc002ed0 | |
| a100-st-p4d24xlarge-47:87091:88232 [0] NCCL INFO New proxy recv connection 5 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:87091:88189 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2bc002f10 | |
| a100-st-p4d24xlarge-47:87091:88232 [0] NCCL INFO New proxy recv connection 6 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:87091:88189 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2bc002f50 | |
| a100-st-p4d24xlarge-47:87091:88232 [0] NCCL INFO New proxy recv connection 7 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:87091:88189 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2bc002f90 | |
| a100-st-p4d24xlarge-47:87091:88189 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2bc002fd0 | |
| a100-st-p4d24xlarge-47:87091:88232 [0] NCCL INFO New proxy recv connection 8 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:87091:88189 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2bc003010 | |
| a100-st-p4d24xlarge-47:87091:88232 [0] NCCL INFO New proxy recv connection 9 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:87091:88189 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2bc003050 | |
| a100-st-p4d24xlarge-47:87091:88232 [0] NCCL INFO New proxy recv connection 10 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:87091:88189 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2bc003090 | |
| a100-st-p4d24xlarge-47:87091:88232 [0] NCCL INFO New proxy recv connection 11 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:87091:88189 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2bc0030d0 | |
| a100-st-p4d24xlarge-47:87091:88232 [0] NCCL INFO New proxy recv connection 12 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:87091:88189 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2bc003110 | |
| a100-st-p4d24xlarge-47:87091:88232 [0] NCCL INFO New proxy recv connection 13 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:87091:88189 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2bc003150 | |
| a100-st-p4d24xlarge-47:87091:88232 [0] NCCL INFO New proxy recv connection 14 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:87091:88189 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2bc003190 | |
| a100-st-p4d24xlarge-47:87091:88232 [0] NCCL INFO New proxy recv connection 15 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:87091:88189 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2bc0031d0 | |
| a100-st-p4d24xlarge-47:87091:88232 [0] NCCL INFO New proxy recv connection 16 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:87091:88189 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2bc003210 | |
| a100-st-p4d24xlarge-47:87091:88232 [0] NCCL INFO New proxy recv connection 17 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:87091:88189 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2bc003250 | |
| a100-st-p4d24xlarge-47:87091:88232 [0] NCCL INFO New proxy recv connection 18 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:87091:88189 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2bc003290 | |
| a100-st-p4d24xlarge-47:87091:88232 [0] NCCL INFO New proxy recv connection 19 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:87091:88189 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2bc0032d0 | |
| a100-st-p4d24xlarge-47:87091:88232 [0] NCCL INFO New proxy recv connection 20 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:87091:88189 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2bc003310 | |
| a100-st-p4d24xlarge-47:87091:88232 [0] NCCL INFO New proxy recv connection 21 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:87091:88189 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2bc003350 | |
| a100-st-p4d24xlarge-47:87091:88232 [0] NCCL INFO New proxy recv connection 22 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:87091:88189 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2bc003390 | |
| a100-st-p4d24xlarge-47:87091:88232 [0] NCCL INFO New proxy recv connection 23 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:87091:88189 [0] NCCL INFO Channel 00/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-47:87091:88189 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2bc0033d0 | |
| a100-st-p4d24xlarge-47:87091:88232 [0] NCCL INFO New proxy send connection 24 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:87091:88189 [0] NCCL INFO Channel 01/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-47:87091:88189 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2bc003410 | |
| a100-st-p4d24xlarge-47:87091:88232 [0] NCCL INFO New proxy send connection 25 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:87091:88189 [0] NCCL INFO Channel 02/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-47:87091:88189 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2bc003450 | |
| a100-st-p4d24xlarge-47:87091:88232 [0] NCCL INFO New proxy send connection 26 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:87091:88189 [0] NCCL INFO Channel 03/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-47:87091:88189 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2bc003490 | |
| a100-st-p4d24xlarge-47:87091:88232 [0] NCCL INFO New proxy send connection 27 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:87091:88189 [0] NCCL INFO Channel 04/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-47:87091:88189 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2bc0034d0 | |
| a100-st-p4d24xlarge-47:87091:88232 [0] NCCL INFO New proxy send connection 28 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:87091:88189 [0] NCCL INFO Channel 05/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-47:87091:88189 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2bc003510 | |
| a100-st-p4d24xlarge-47:87091:88232 [0] NCCL INFO New proxy send connection 29 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:87091:88189 [0] NCCL INFO Channel 06/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-47:87091:88189 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2bc003550 | |
| a100-st-p4d24xlarge-47:87091:88232 [0] NCCL INFO New proxy send connection 30 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:87091:88189 [0] NCCL INFO Channel 07/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-47:87091:88189 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2bc003590 | |
| a100-st-p4d24xlarge-47:87091:88232 [0] NCCL INFO New proxy send connection 31 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:87091:88189 [0] NCCL INFO Channel 08/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-47:87091:88189 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2bc0035d0 | |
| a100-st-p4d24xlarge-47:87091:88232 [0] NCCL INFO New proxy send connection 32 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:87091:88189 [0] NCCL INFO Channel 09/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-47:87091:88189 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2bc003610 | |
| a100-st-p4d24xlarge-47:87091:88232 [0] NCCL INFO New proxy send connection 33 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:87091:88189 [0] NCCL INFO Channel 10/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-47:87091:88189 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2bc003650 | |
| a100-st-p4d24xlarge-47:87091:88232 [0] NCCL INFO New proxy send connection 34 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:87091:88189 [0] NCCL INFO Channel 11/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-47:87091:88189 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2bc003690 | |
| a100-st-p4d24xlarge-47:87091:88232 [0] NCCL INFO New proxy send connection 35 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:87091:88189 [0] NCCL INFO Channel 12/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-47:87091:88189 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2bc0036d0 | |
| a100-st-p4d24xlarge-47:87091:88232 [0] NCCL INFO New proxy send connection 36 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:87091:88189 [0] NCCL INFO Channel 13/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-47:87091:88189 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2bc003710 | |
| a100-st-p4d24xlarge-47:87091:88232 [0] NCCL INFO New proxy send connection 37 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:87091:88189 [0] NCCL INFO Channel 14/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-47:87091:88189 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2bc003750 | |
| a100-st-p4d24xlarge-47:87091:88232 [0] NCCL INFO New proxy send connection 38 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:87091:88189 [0] NCCL INFO Channel 15/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-47:87091:88189 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2bc003790 | |
| a100-st-p4d24xlarge-47:87091:88232 [0] NCCL INFO New proxy send connection 39 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:87091:88189 [0] NCCL INFO Channel 16/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-47:87091:88189 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2bc0037d0 | |
| a100-st-p4d24xlarge-47:87091:88232 [0] NCCL INFO New proxy send connection 40 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:87091:88189 [0] NCCL INFO Channel 17/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-47:87091:88189 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2bc003810 | |
| a100-st-p4d24xlarge-47:87091:88232 [0] NCCL INFO New proxy send connection 41 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:87091:88189 [0] NCCL INFO Channel 18/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-47:87091:88189 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2bc003850 | |
| a100-st-p4d24xlarge-47:87091:88232 [0] NCCL INFO New proxy send connection 42 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:87091:88189 [0] NCCL INFO Channel 19/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-47:87091:88189 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2bc003890 | |
| a100-st-p4d24xlarge-47:87091:88232 [0] NCCL INFO New proxy send connection 43 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:87091:88189 [0] NCCL INFO Channel 20/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-47:87091:88189 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2bc0038d0 | |
| a100-st-p4d24xlarge-47:87091:88232 [0] NCCL INFO New proxy send connection 44 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:87091:88189 [0] NCCL INFO Channel 21/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-47:87091:88189 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2bc003910 | |
| a100-st-p4d24xlarge-47:87091:88232 [0] NCCL INFO New proxy send connection 45 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:87091:88189 [0] NCCL INFO Channel 22/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-47:87091:88189 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2bc003950 | |
| a100-st-p4d24xlarge-47:87091:88232 [0] NCCL INFO New proxy send connection 46 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:87091:88189 [0] NCCL INFO Channel 23/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-47:87091:88189 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2bc003990 | |
| a100-st-p4d24xlarge-47:87091:88232 [0] NCCL INFO New proxy send connection 47 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:87091:88189 [0] NCCL INFO Connected all rings | |
| a100-st-p4d24xlarge-47:87091:88189 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2bc0039d0 | |
| a100-st-p4d24xlarge-47:87091:88232 [0] NCCL INFO New proxy recv connection 48 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:87091:88232 [0] NCCL INFO New proxy recv connection 49 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:87091:88189 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2bc003a10 | |
| a100-st-p4d24xlarge-47:87091:88189 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2bc003a50 | |
| a100-st-p4d24xlarge-47:87091:88232 [0] NCCL INFO New proxy recv connection 50 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:87091:88189 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2bc003a90 | |
| a100-st-p4d24xlarge-47:87091:88232 [0] NCCL INFO New proxy recv connection 51 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:87091:88189 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2bc003ad0 | |
| a100-st-p4d24xlarge-47:87091:88232 [0] NCCL INFO New proxy recv connection 52 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:87091:88189 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2bc003b10 | |
| a100-st-p4d24xlarge-47:87091:88232 [0] NCCL INFO New proxy recv connection 53 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:87091:88189 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2bc003b50 | |
| a100-st-p4d24xlarge-47:87091:88232 [0] NCCL INFO New proxy recv connection 54 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:87091:88189 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2bc003b90 | |
| a100-st-p4d24xlarge-47:87091:88232 [0] NCCL INFO New proxy recv connection 55 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:87091:88189 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2bc003bd0 | |
| a100-st-p4d24xlarge-47:87091:88232 [0] NCCL INFO New proxy recv connection 56 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:87091:88189 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2bc003c10 | |
| a100-st-p4d24xlarge-47:87091:88232 [0] NCCL INFO New proxy recv connection 57 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:87091:88189 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2bc003c50 | |
| a100-st-p4d24xlarge-47:87091:88232 [0] NCCL INFO New proxy recv connection 58 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:87091:88189 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2bc003c90 | |
| a100-st-p4d24xlarge-47:87091:88232 [0] NCCL INFO New proxy recv connection 59 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:87091:88189 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2bc003cd0 | |
| a100-st-p4d24xlarge-47:87091:88232 [0] NCCL INFO New proxy recv connection 60 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:87091:88189 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2bc003d10 | |
| a100-st-p4d24xlarge-47:87091:88232 [0] NCCL INFO New proxy recv connection 61 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:87091:88189 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2bc003d50 | |
| a100-st-p4d24xlarge-47:87091:88232 [0] NCCL INFO New proxy recv connection 62 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:87091:88189 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2bc003d90 | |
| a100-st-p4d24xlarge-47:87091:88232 [0] NCCL INFO New proxy recv connection 63 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:87091:88189 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2bc003dd0 | |
| a100-st-p4d24xlarge-47:87091:88232 [0] NCCL INFO New proxy recv connection 64 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:87091:88189 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2bc003e10 | |
| a100-st-p4d24xlarge-47:87091:88232 [0] NCCL INFO New proxy recv connection 65 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:87091:88189 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2bc003e50 | |
| a100-st-p4d24xlarge-47:87091:88232 [0] NCCL INFO New proxy recv connection 66 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:87091:88189 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2bc003e90 | |
| a100-st-p4d24xlarge-47:87091:88232 [0] NCCL INFO New proxy recv connection 67 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:87091:88189 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2bc003ed0 | |
| a100-st-p4d24xlarge-47:87091:88232 [0] NCCL INFO New proxy recv connection 68 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:87091:88189 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2bc003f10 | |
| a100-st-p4d24xlarge-47:87091:88232 [0] NCCL INFO New proxy recv connection 69 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:87091:88189 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2bc003f50 | |
| a100-st-p4d24xlarge-47:87091:88232 [0] NCCL INFO New proxy recv connection 70 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:87091:88189 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2bc003f90 | |
| a100-st-p4d24xlarge-47:87091:88232 [0] NCCL INFO New proxy recv connection 71 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-47:87091:88189 [0] NCCL INFO Connected all trees | |
| a100-st-p4d24xlarge-47:87091:88189 [0] NCCL INFO NCCL_ALGO set by environment to ring | |
| a100-st-p4d24xlarge-47:87091:88189 [0] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512 | |
| a100-st-p4d24xlarge-47:87091:88189 [0] NCCL INFO 24 coll channels, 32 p2p channels, 32 p2p channels per peer | |
| a100-st-p4d24xlarge-47:87091:88232 [0] NCCL INFO New proxy send connection 72 from local rank 0, transport 2 | |
| a100-st-p4d24xlarge-47:87091:88189 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7fa2bc003fd0 | |
| a100-st-p4d24xlarge-47:87091:88189 [0] NCCL INFO comm 0x563c6b00afa0 rank 0 nranks 8 cudaDev 0 busId 101c0 - Init COMPLETE | |
| didn't get the result, queue empty | |
| DDPOptimizer used bucket cap 26214400 and produced the following buckets: | |
| Index Size (b) Param Names | |
| ------- ---------- --------------------------------- | |
| 0 8196000 self_fc_weight | |
| self_fc_bias | |
| 1 31502336 self_layer4_2_bn3_weight | |
| self_layer4_2_bn3_bias | |
| self_layer4_2_conv3_weight | |
| self_layer4_2_bn2_weight | |
| self_layer4_2_bn2_bias | |
| self_layer4_2_conv2_weight | |
| self_layer4_2_bn1_weight | |
| self_layer4_2_bn1_bias | |
| self_layer4_2_conv1_weight | |
| self_layer4_1_bn3_weight | |
| self_layer4_1_bn3_bias | |
| self_layer4_1_conv3_weight | |
| self_layer4_1_bn2_weight | |
| self_layer4_1_bn2_bias | |
| self_layer4_1_conv2_weight | |
| 2 26255360 self_layer4_1_bn1_weight | |
| self_layer4_1_bn1_bias | |
| self_layer4_1_conv1_weight | |
| self_layer4_0_downsample_1_weight | |
| self_layer4_0_downsample_1_bias | |
| self_layer4_0_downsample_0_weight | |
| self_layer4_0_bn3_weight | |
| self_layer4_0_bn3_bias | |
| self_layer4_0_conv3_weight | |
| self_layer4_0_bn2_weight | |
| self_layer4_0_bn2_bias | |
| self_layer4_0_conv2_weight | |
| 3 26550272 self_layer4_0_bn1_weight | |
| self_layer4_0_bn1_bias | |
| self_layer4_0_conv1_weight | |
| self_layer3_5_bn3_weight | |
| self_layer3_5_bn3_bias | |
| self_layer3_5_conv3_weight | |
| self_layer3_5_bn2_weight | |
| self_layer3_5_bn2_bias | |
| self_layer3_5_conv2_weight | |
| self_layer3_5_bn1_weight | |
| self_layer3_5_bn1_bias | |
| self_layer3_5_conv1_weight | |
| self_layer3_4_bn3_weight | |
| self_layer3_4_bn3_bias | |
| self_layer3_4_conv3_weight | |
| self_layer3_4_bn2_weight | |
| self_layer3_4_bn2_bias | |
| self_layer3_4_conv2_weight | |
| self_layer3_4_bn1_weight | |
| self_layer3_4_bn1_bias | |
| self_layer3_4_conv1_weight | |
| self_layer3_3_bn3_weight | |
| self_layer3_3_bn3_bias | |
| self_layer3_3_conv3_weight | |
| self_layer3_3_bn2_weight | |
| self_layer3_3_bn2_bias | |
| self_layer3_3_conv2_weight | |
| self_layer3_3_bn1_weight | |
| self_layer3_3_bn1_bias | |
| self_layer3_3_conv1_weight | |
| self_layer3_2_bn3_weight | |
| self_layer3_2_bn3_bias | |
| self_layer3_2_conv3_weight | |
| self_layer3_2_bn2_weight | |
| self_layer3_2_bn2_bias | |
| self_layer3_2_conv2_weight | |
| self_layer3_2_bn1_weight | |
| self_layer3_2_bn1_bias | |
| self_layer3_2_conv1_weight | |
| self_layer3_1_bn3_weight | |
| self_layer3_1_bn3_bias | |
| self_layer3_1_conv3_weight | |
| self_layer3_1_bn2_weight | |
| self_layer3_1_bn2_bias | |
| self_layer3_1_conv2_weight | |
| self_layer3_1_bn1_weight | |
| self_layer3_1_bn1_bias | |
| self_layer3_1_conv1_weight | |
| self_layer3_0_downsample_1_weight | |
| self_layer3_0_downsample_1_bias | |
| self_layer3_0_downsample_0_weight | |
| 4 9724160 self_layer3_0_bn3_weight | |
| self_layer3_0_bn3_bias | |
| self_layer3_0_conv3_weight | |
| self_layer3_0_bn2_weight | |
| self_layer3_0_bn2_bias | |
| self_layer3_0_conv2_weight | |
| self_layer3_0_bn1_weight | |
| self_layer3_0_bn1_bias | |
| self_layer3_0_conv1_weight | |
| self_layer2_3_bn3_weight | |
| self_layer2_3_bn3_bias | |
| self_layer2_3_conv3_weight | |
| self_layer2_3_bn2_weight | |
| self_layer2_3_bn2_bias | |
| self_layer2_3_conv2_weight | |
| self_layer2_3_bn1_weight | |
| self_layer2_3_bn1_bias | |
| self_layer2_3_conv1_weight | |
| self_layer2_2_bn3_weight | |
| self_layer2_2_bn3_bias | |
| self_layer2_2_conv3_weight | |
| self_layer2_2_bn2_weight | |
| self_layer2_2_bn2_bias | |
| self_layer2_2_conv2_weight | |
| self_layer2_2_bn1_weight | |
| self_layer2_2_bn1_bias | |
| self_layer2_2_conv1_weight | |
| self_layer2_1_bn3_weight | |
| self_layer2_1_bn3_bias | |
| self_layer2_1_conv3_weight | |
| self_layer2_1_bn2_weight | |
| self_layer2_1_bn2_bias | |
| self_layer2_1_conv2_weight | |
| self_layer2_1_bn1_weight | |
| self_layer2_1_bn1_bias | |
| self_layer2_1_conv1_weight | |
| self_layer2_0_downsample_1_weight | |
| self_layer2_0_downsample_1_bias | |
| self_layer2_0_downsample_0_weight | |
| self_layer2_0_bn3_weight | |
| self_layer2_0_bn3_bias | |
| self_layer2_0_conv3_weight | |
| self_layer2_0_bn2_weight | |
| self_layer2_0_bn2_bias | |
| self_layer2_0_conv2_weight | |
| self_layer2_0_bn1_weight | |
| self_layer2_0_bn1_bias | |
| self_layer2_0_conv1_weight | |
| self_layer1_2_bn3_weight | |
| self_layer1_2_bn3_bias | |
| self_layer1_2_conv3_weight | |
| self_layer1_2_bn2_weight | |
| self_layer1_2_bn2_bias | |
| self_layer1_2_conv2_weight | |
| self_layer1_2_bn1_weight | |
| self_layer1_2_bn1_bias | |
| self_layer1_2_conv1_weight | |
| self_layer1_1_bn3_weight | |
| self_layer1_1_bn3_bias | |
| self_layer1_1_conv3_weight | |
| self_layer1_1_bn2_weight | |
| self_layer1_1_bn2_bias | |
| self_layer1_1_conv2_weight | |
| self_layer1_1_bn1_weight | |
| self_layer1_1_bn1_bias | |
| self_layer1_1_conv1_weight | |
| self_layer1_0_downsample_1_weight | |
| self_layer1_0_downsample_1_bias | |
| self_layer1_0_downsample_0_weight | |
| self_layer1_0_bn3_weight | |
| self_layer1_0_bn3_bias | |
| self_layer1_0_conv3_weight | |
| self_layer1_0_bn2_weight | |
| self_layer1_0_bn2_bias | |
| self_layer1_0_conv2_weight | |
| self_layer1_0_bn1_weight | |
| self_layer1_0_bn1_bias | |
| self_layer1_0_conv1_weight | |
| self_bn1_weight | |
| self_bn1_bias | |
| self_conv1_weight | |
| ---orig graph--- | |
| graph(): | |
| %x : torch.Tensor [#users=1] = placeholder[target=x] | |
| %self_conv1 : [#users=1] = call_module[target=self_conv1](args = (%x,), kwargs = {}) | |
| %self_bn1 : [#users=1] = call_module[target=self_bn1](args = (%self_conv1,), kwargs = {}) | |
| %self_relu : [#users=1] = call_module[target=self_relu](args = (%self_bn1,), kwargs = {}) | |
| %self_maxpool : [#users=2] = call_module[target=self_maxpool](args = (%self_relu,), kwargs = {}) | |
| %self_layer1_0_conv1 : [#users=1] = call_module[target=self_layer1_0_conv1](args = (%self_maxpool,), kwargs = {}) | |
| %self_layer1_0_bn1 : [#users=1] = call_module[target=self_layer1_0_bn1](args = (%self_layer1_0_conv1,), kwargs = {}) | |
| %self_layer1_0_relu : [#users=1] = call_module[target=self_layer1_0_relu](args = (%self_layer1_0_bn1,), kwargs = {}) | |
| %self_layer1_0_conv2 : [#users=1] = call_module[target=self_layer1_0_conv2](args = (%self_layer1_0_relu,), kwargs = {}) | |
| %self_layer1_0_bn2 : [#users=1] = call_module[target=self_layer1_0_bn2](args = (%self_layer1_0_conv2,), kwargs = {}) | |
| %self_layer1_0_relu_1 : [#users=1] = call_module[target=self_layer1_0_relu](args = (%self_layer1_0_bn2,), kwargs = {}) | |
| %self_layer1_0_conv3 : [#users=1] = call_module[target=self_layer1_0_conv3](args = (%self_layer1_0_relu_1,), kwargs = {}) | |
| %self_layer1_0_bn3 : [#users=1] = call_module[target=self_layer1_0_bn3](args = (%self_layer1_0_conv3,), kwargs = {}) | |
| %self_layer1_0_downsample_0 : [#users=1] = call_module[target=self_layer1_0_downsample_0](args = (%self_maxpool,), kwargs = {}) | |
| %self_layer1_0_downsample_1 : [#users=1] = call_module[target=self_layer1_0_downsample_1](args = (%self_layer1_0_downsample_0,), kwargs = {}) | |
| %iadd : [#users=1] = call_function[target=operator.iadd](args = (%self_layer1_0_bn3, %self_layer1_0_downsample_1), kwargs = {}) | |
| %self_layer1_0_relu_2 : [#users=2] = call_module[target=self_layer1_0_relu](args = (%iadd,), kwargs = {}) | |
| %self_layer1_1_conv1 : [#users=1] = call_module[target=self_layer1_1_conv1](args = (%self_layer1_0_relu_2,), kwargs = {}) | |
| %self_layer1_1_bn1 : [#users=1] = call_module[target=self_layer1_1_bn1](args = (%self_layer1_1_conv1,), kwargs = {}) | |
| %self_layer1_1_relu : [#users=1] = call_module[target=self_layer1_1_relu](args = (%self_layer1_1_bn1,), kwargs = {}) | |
| %self_layer1_1_conv2 : [#users=1] = call_module[target=self_layer1_1_conv2](args = (%self_layer1_1_relu,), kwargs = {}) | |
| %self_layer1_1_bn2 : [#users=1] = call_module[target=self_layer1_1_bn2](args = (%self_layer1_1_conv2,), kwargs = {}) | |
| %self_layer1_1_relu_1 : [#users=1] = call_module[target=self_layer1_1_relu](args = (%self_layer1_1_bn2,), kwargs = {}) | |
| %self_layer1_1_conv3 : [#users=1] = call_module[target=self_layer1_1_conv3](args = (%self_layer1_1_relu_1,), kwargs = {}) | |
| %self_layer1_1_bn3 : [#users=1] = call_module[target=self_layer1_1_bn3](args = (%self_layer1_1_conv3,), kwargs = {}) | |
| %iadd_1 : [#users=1] = call_function[target=operator.iadd](args = (%self_layer1_1_bn3, %self_layer1_0_relu_2), kwargs = {}) | |
| %self_layer1_1_relu_2 : [#users=2] = call_module[target=self_layer1_1_relu](args = (%iadd_1,), kwargs = {}) | |
| %self_layer1_2_conv1 : [#users=1] = call_module[target=self_layer1_2_conv1](args = (%self_layer1_1_relu_2,), kwargs = {}) | |
| %self_layer1_2_bn1 : [#users=1] = call_module[target=self_layer1_2_bn1](args = (%self_layer1_2_conv1,), kwargs = {}) | |
| %self_layer1_2_relu : [#users=1] = call_module[target=self_layer1_2_relu](args = (%self_layer1_2_bn1,), kwargs = {}) | |
| %self_layer1_2_conv2 : [#users=1] = call_module[target=self_layer1_2_conv2](args = (%self_layer1_2_relu,), kwargs = {}) | |
| %self_layer1_2_bn2 : [#users=1] = call_module[target=self_layer1_2_bn2](args = (%self_layer1_2_conv2,), kwargs = {}) | |
| %self_layer1_2_relu_1 : [#users=1] = call_module[target=self_layer1_2_relu](args = (%self_layer1_2_bn2,), kwargs = {}) | |
| %self_layer1_2_conv3 : [#users=1] = call_module[target=self_layer1_2_conv3](args = (%self_layer1_2_relu_1,), kwargs = {}) | |
| %self_layer1_2_bn3 : [#users=1] = call_module[target=self_layer1_2_bn3](args = (%self_layer1_2_conv3,), kwargs = {}) | |
| %iadd_2 : [#users=1] = call_function[target=operator.iadd](args = (%self_layer1_2_bn3, %self_layer1_1_relu_2), kwargs = {}) | |
| %self_layer1_2_relu_2 : [#users=2] = call_module[target=self_layer1_2_relu](args = (%iadd_2,), kwargs = {}) | |
| %self_layer2_0_conv1 : [#users=1] = call_module[target=self_layer2_0_conv1](args = (%self_layer1_2_relu_2,), kwargs = {}) | |
| %self_layer2_0_bn1 : [#users=1] = call_module[target=self_layer2_0_bn1](args = (%self_layer2_0_conv1,), kwargs = {}) | |
| %self_layer2_0_relu : [#users=1] = call_module[target=self_layer2_0_relu](args = (%self_layer2_0_bn1,), kwargs = {}) | |
| %self_layer2_0_conv2 : [#users=1] = call_module[target=self_layer2_0_conv2](args = (%self_layer2_0_relu,), kwargs = {}) | |
| %self_layer2_0_bn2 : [#users=1] = call_module[target=self_layer2_0_bn2](args = (%self_layer2_0_conv2,), kwargs = {}) | |
| %self_layer2_0_relu_1 : [#users=1] = call_module[target=self_layer2_0_relu](args = (%self_layer2_0_bn2,), kwargs = {}) | |
| %self_layer2_0_conv3 : [#users=1] = call_module[target=self_layer2_0_conv3](args = (%self_layer2_0_relu_1,), kwargs = {}) | |
| %self_layer2_0_bn3 : [#users=1] = call_module[target=self_layer2_0_bn3](args = (%self_layer2_0_conv3,), kwargs = {}) | |
| %self_layer2_0_downsample_0 : [#users=1] = call_module[target=self_layer2_0_downsample_0](args = (%self_layer1_2_relu_2,), kwargs = {}) | |
| %self_layer2_0_downsample_1 : [#users=1] = call_module[target=self_layer2_0_downsample_1](args = (%self_layer2_0_downsample_0,), kwargs = {}) | |
| %iadd_3 : [#users=1] = call_function[target=operator.iadd](args = (%self_layer2_0_bn3, %self_layer2_0_downsample_1), kwargs = {}) | |
| %self_layer2_0_relu_2 : [#users=2] = call_module[target=self_layer2_0_relu](args = (%iadd_3,), kwargs = {}) | |
| %self_layer2_1_conv1 : [#users=1] = call_module[target=self_layer2_1_conv1](args = (%self_layer2_0_relu_2,), kwargs = {}) | |
| %self_layer2_1_bn1 : [#users=1] = call_module[target=self_layer2_1_bn1](args = (%self_layer2_1_conv1,), kwargs = {}) | |
| %self_layer2_1_relu : [#users=1] = call_module[target=self_layer2_1_relu](args = (%self_layer2_1_bn1,), kwargs = {}) | |
| %self_layer2_1_conv2 : [#users=1] = call_module[target=self_layer2_1_conv2](args = (%self_layer2_1_relu,), kwargs = {}) | |
| %self_layer2_1_bn2 : [#users=1] = call_module[target=self_layer2_1_bn2](args = (%self_layer2_1_conv2,), kwargs = {}) | |
| %self_layer2_1_relu_1 : [#users=1] = call_module[target=self_layer2_1_relu](args = (%self_layer2_1_bn2,), kwargs = {}) | |
| %self_layer2_1_conv3 : [#users=1] = call_module[target=self_layer2_1_conv3](args = (%self_layer2_1_relu_1,), kwargs = {}) | |
| %self_layer2_1_bn3 : [#users=1] = call_module[target=self_layer2_1_bn3](args = (%self_layer2_1_conv3,), kwargs = {}) | |
| %iadd_4 : [#users=1] = call_function[target=operator.iadd](args = (%self_layer2_1_bn3, %self_layer2_0_relu_2), kwargs = {}) | |
| %self_layer2_1_relu_2 : [#users=2] = call_module[target=self_layer2_1_relu](args = (%iadd_4,), kwargs = {}) | |
| %self_layer2_2_conv1 : [#users=1] = call_module[target=self_layer2_2_conv1](args = (%self_layer2_1_relu_2,), kwargs = {}) | |
| %self_layer2_2_bn1 : [#users=1] = call_module[target=self_layer2_2_bn1](args = (%self_layer2_2_conv1,), kwargs = {}) | |
| %self_layer2_2_relu : [#users=1] = call_module[target=self_layer2_2_relu](args = (%self_layer2_2_bn1,), kwargs = {}) | |
| %self_layer2_2_conv2 : [#users=1] = call_module[target=self_layer2_2_conv2](args = (%self_layer2_2_relu,), kwargs = {}) | |
| %self_layer2_2_bn2 : [#users=1] = call_module[target=self_layer2_2_bn2](args = (%self_layer2_2_conv2,), kwargs = {}) | |
| %self_layer2_2_relu_1 : [#users=1] = call_module[target=self_layer2_2_relu](args = (%self_layer2_2_bn2,), kwargs = {}) | |
| %self_layer2_2_conv3 : [#users=1] = call_module[target=self_layer2_2_conv3](args = (%self_layer2_2_relu_1,), kwargs = {}) | |
| %self_layer2_2_bn3 : [#users=1] = call_module[target=self_layer2_2_bn3](args = (%self_layer2_2_conv3,), kwargs = {}) | |
| %iadd_5 : [#users=1] = call_function[target=operator.iadd](args = (%self_layer2_2_bn3, %self_layer2_1_relu_2), kwargs = {}) | |
| %self_layer2_2_relu_2 : [#users=2] = call_module[target=self_layer2_2_relu](args = (%iadd_5,), kwargs = {}) | |
| %self_layer2_3_conv1 : [#users=1] = call_module[target=self_layer2_3_conv1](args = (%self_layer2_2_relu_2,), kwargs = {}) | |
| %self_layer2_3_bn1 : [#users=1] = call_module[target=self_layer2_3_bn1](args = (%self_layer2_3_conv1,), kwargs = {}) | |
| %self_layer2_3_relu : [#users=1] = call_module[target=self_layer2_3_relu](args = (%self_layer2_3_bn1,), kwargs = {}) | |
| %self_layer2_3_conv2 : [#users=1] = call_module[target=self_layer2_3_conv2](args = (%self_layer2_3_relu,), kwargs = {}) | |
| %self_layer2_3_bn2 : [#users=1] = call_module[target=self_layer2_3_bn2](args = (%self_layer2_3_conv2,), kwargs = {}) | |
| %self_layer2_3_relu_1 : [#users=1] = call_module[target=self_layer2_3_relu](args = (%self_layer2_3_bn2,), kwargs = {}) | |
| %self_layer2_3_conv3 : [#users=1] = call_module[target=self_layer2_3_conv3](args = (%self_layer2_3_relu_1,), kwargs = {}) | |
| %self_layer2_3_bn3 : [#users=1] = call_module[target=self_layer2_3_bn3](args = (%self_layer2_3_conv3,), kwargs = {}) | |
| %iadd_6 : [#users=1] = call_function[target=operator.iadd](args = (%self_layer2_3_bn3, %self_layer2_2_relu_2), kwargs = {}) | |
| %self_layer2_3_relu_2 : [#users=2] = call_module[target=self_layer2_3_relu](args = (%iadd_6,), kwargs = {}) | |
| %self_layer3_0_conv1 : [#users=1] = call_module[target=self_layer3_0_conv1](args = (%self_layer2_3_relu_2,), kwargs = {}) | |
| %self_layer3_0_bn1 : [#users=1] = call_module[target=self_layer3_0_bn1](args = (%self_layer3_0_conv1,), kwargs = {}) | |
| %self_layer3_0_relu : [#users=1] = call_module[target=self_layer3_0_relu](args = (%self_layer3_0_bn1,), kwargs = {}) | |
| %self_layer3_0_conv2 : [#users=1] = call_module[target=self_layer3_0_conv2](args = (%self_layer3_0_relu,), kwargs = {}) | |
| %self_layer3_0_bn2 : [#users=1] = call_module[target=self_layer3_0_bn2](args = (%self_layer3_0_conv2,), kwargs = {}) | |
| %self_layer3_0_relu_1 : [#users=1] = call_module[target=self_layer3_0_relu](args = (%self_layer3_0_bn2,), kwargs = {}) | |
| %self_layer3_0_conv3 : [#users=1] = call_module[target=self_layer3_0_conv3](args = (%self_layer3_0_relu_1,), kwargs = {}) | |
| %self_layer3_0_bn3 : [#users=1] = call_module[target=self_layer3_0_bn3](args = (%self_layer3_0_conv3,), kwargs = {}) | |
| %self_layer3_0_downsample_0 : [#users=1] = call_module[target=self_layer3_0_downsample_0](args = (%self_layer2_3_relu_2,), kwargs = {}) | |
| %self_layer3_0_downsample_1 : [#users=1] = call_module[target=self_layer3_0_downsample_1](args = (%self_layer3_0_downsample_0,), kwargs = {}) | |
| %iadd_7 : [#users=1] = call_function[target=operator.iadd](args = (%self_layer3_0_bn3, %self_layer3_0_downsample_1), kwargs = {}) | |
| %self_layer3_0_relu_2 : [#users=2] = call_module[target=self_layer3_0_relu](args = (%iadd_7,), kwargs = {}) | |
| %self_layer3_1_conv1 : [#users=1] = call_module[target=self_layer3_1_conv1](args = (%self_layer3_0_relu_2,), kwargs = {}) | |
| %self_layer3_1_bn1 : [#users=1] = call_module[target=self_layer3_1_bn1](args = (%self_layer3_1_conv1,), kwargs = {}) | |
| %self_layer3_1_relu : [#users=1] = call_module[target=self_layer3_1_relu](args = (%self_layer3_1_bn1,), kwargs = {}) | |
| %self_layer3_1_conv2 : [#users=1] = call_module[target=self_layer3_1_conv2](args = (%self_layer3_1_relu,), kwargs = {}) | |
| %self_layer3_1_bn2 : [#users=1] = call_module[target=self_layer3_1_bn2](args = (%self_layer3_1_conv2,), kwargs = {}) | |
| %self_layer3_1_relu_1 : [#users=1] = call_module[target=self_layer3_1_relu](args = (%self_layer3_1_bn2,), kwargs = {}) | |
| %self_layer3_1_conv3 : [#users=1] = call_module[target=self_layer3_1_conv3](args = (%self_layer3_1_relu_1,), kwargs = {}) | |
| %self_layer3_1_bn3 : [#users=1] = call_module[target=self_layer3_1_bn3](args = (%self_layer3_1_conv3,), kwargs = {}) | |
| %iadd_8 : [#users=1] = call_function[target=operator.iadd](args = (%self_layer3_1_bn3, %self_layer3_0_relu_2), kwargs = {}) | |
| %self_layer3_1_relu_2 : [#users=2] = call_module[target=self_layer3_1_relu](args = (%iadd_8,), kwargs = {}) | |
| %self_layer3_2_conv1 : [#users=1] = call_module[target=self_layer3_2_conv1](args = (%self_layer3_1_relu_2,), kwargs = {}) | |
| %self_layer3_2_bn1 : [#users=1] = call_module[target=self_layer3_2_bn1](args = (%self_layer3_2_conv1,), kwargs = {}) | |
| %self_layer3_2_relu : [#users=1] = call_module[target=self_layer3_2_relu](args = (%self_layer3_2_bn1,), kwargs = {}) | |
| %self_layer3_2_conv2 : [#users=1] = call_module[target=self_layer3_2_conv2](args = (%self_layer3_2_relu,), kwargs = {}) | |
| %self_layer3_2_bn2 : [#users=1] = call_module[target=self_layer3_2_bn2](args = (%self_layer3_2_conv2,), kwargs = {}) | |
| %self_layer3_2_relu_1 : [#users=1] = call_module[target=self_layer3_2_relu](args = (%self_layer3_2_bn2,), kwargs = {}) | |
| %self_layer3_2_conv3 : [#users=1] = call_module[target=self_layer3_2_conv3](args = (%self_layer3_2_relu_1,), kwargs = {}) | |
| %self_layer3_2_bn3 : [#users=1] = call_module[target=self_layer3_2_bn3](args = (%self_layer3_2_conv3,), kwargs = {}) | |
| %iadd_9 : [#users=1] = call_function[target=operator.iadd](args = (%self_layer3_2_bn3, %self_layer3_1_relu_2), kwargs = {}) | |
| %self_layer3_2_relu_2 : [#users=2] = call_module[target=self_layer3_2_relu](args = (%iadd_9,), kwargs = {}) | |
| %self_layer3_3_conv1 : [#users=1] = call_module[target=self_layer3_3_conv1](args = (%self_layer3_2_relu_2,), kwargs = {}) | |
| %self_layer3_3_bn1 : [#users=1] = call_module[target=self_layer3_3_bn1](args = (%self_layer3_3_conv1,), kwargs = {}) | |
| %self_layer3_3_relu : [#users=1] = call_module[target=self_layer3_3_relu](args = (%self_layer3_3_bn1,), kwargs = {}) | |
| %self_layer3_3_conv2 : [#users=1] = call_module[target=self_layer3_3_conv2](args = (%self_layer3_3_relu,), kwargs = {}) | |
| %self_layer3_3_bn2 : [#users=1] = call_module[target=self_layer3_3_bn2](args = (%self_layer3_3_conv2,), kwargs = {}) | |
| %self_layer3_3_relu_1 : [#users=1] = call_module[target=self_layer3_3_relu](args = (%self_layer3_3_bn2,), kwargs = {}) | |
| %self_layer3_3_conv3 : [#users=1] = call_module[target=self_layer3_3_conv3](args = (%self_layer3_3_relu_1,), kwargs = {}) | |
| %self_layer3_3_bn3 : [#users=1] = call_module[target=self_layer3_3_bn3](args = (%self_layer3_3_conv3,), kwargs = {}) | |
| %iadd_10 : [#users=1] = call_function[target=operator.iadd](args = (%self_layer3_3_bn3, %self_layer3_2_relu_2), kwargs = {}) | |
| %self_layer3_3_relu_2 : [#users=2] = call_module[target=self_layer3_3_relu](args = (%iadd_10,), kwargs = {}) | |
| %self_layer3_4_conv1 : [#users=1] = call_module[target=self_layer3_4_conv1](args = (%self_layer3_3_relu_2,), kwargs = {}) | |
| %self_layer3_4_bn1 : [#users=1] = call_module[target=self_layer3_4_bn1](args = (%self_layer3_4_conv1,), kwargs = {}) | |
| %self_layer3_4_relu : [#users=1] = call_module[target=self_layer3_4_relu](args = (%self_layer3_4_bn1,), kwargs = {}) | |
| %self_layer3_4_conv2 : [#users=1] = call_module[target=self_layer3_4_conv2](args = (%self_layer3_4_relu,), kwargs = {}) | |
| %self_layer3_4_bn2 : [#users=1] = call_module[target=self_layer3_4_bn2](args = (%self_layer3_4_conv2,), kwargs = {}) | |
| %self_layer3_4_relu_1 : [#users=1] = call_module[target=self_layer3_4_relu](args = (%self_layer3_4_bn2,), kwargs = {}) | |
| %self_layer3_4_conv3 : [#users=1] = call_module[target=self_layer3_4_conv3](args = (%self_layer3_4_relu_1,), kwargs = {}) | |
| %self_layer3_4_bn3 : [#users=1] = call_module[target=self_layer3_4_bn3](args = (%self_layer3_4_conv3,), kwargs = {}) | |
| %iadd_11 : [#users=1] = call_function[target=operator.iadd](args = (%self_layer3_4_bn3, %self_layer3_3_relu_2), kwargs = {}) | |
| %self_layer3_4_relu_2 : [#users=2] = call_module[target=self_layer3_4_relu](args = (%iadd_11,), kwargs = {}) | |
| %self_layer3_5_conv1 : [#users=1] = call_module[target=self_layer3_5_conv1](args = (%self_layer3_4_relu_2,), kwargs = {}) | |
| %self_layer3_5_bn1 : [#users=1] = call_module[target=self_layer3_5_bn1](args = (%self_layer3_5_conv1,), kwargs = {}) | |
| %self_layer3_5_relu : [#users=1] = call_module[target=self_layer3_5_relu](args = (%self_layer3_5_bn1,), kwargs = {}) | |
| %self_layer3_5_conv2 : [#users=1] = call_module[target=self_layer3_5_conv2](args = (%self_layer3_5_relu,), kwargs = {}) | |
| %self_layer3_5_bn2 : [#users=1] = call_module[target=self_layer3_5_bn2](args = (%self_layer3_5_conv2,), kwargs = {}) | |
| %self_layer3_5_relu_1 : [#users=1] = call_module[target=self_layer3_5_relu](args = (%self_layer3_5_bn2,), kwargs = {}) | |
| %self_layer3_5_conv3 : [#users=1] = call_module[target=self_layer3_5_conv3](args = (%self_layer3_5_relu_1,), kwargs = {}) | |
| %self_layer3_5_bn3 : [#users=1] = call_module[target=self_layer3_5_bn3](args = (%self_layer3_5_conv3,), kwargs = {}) | |
| %iadd_12 : [#users=1] = call_function[target=operator.iadd](args = (%self_layer3_5_bn3, %self_layer3_4_relu_2), kwargs = {}) | |
| %self_layer3_5_relu_2 : [#users=2] = call_module[target=self_layer3_5_relu](args = (%iadd_12,), kwargs = {}) | |
| %self_layer4_0_conv1 : [#users=1] = call_module[target=self_layer4_0_conv1](args = (%self_layer3_5_relu_2,), kwargs = {}) | |
| %self_layer4_0_bn1 : [#users=1] = call_module[target=self_layer4_0_bn1](args = (%self_layer4_0_conv1,), kwargs = {}) | |
| %self_layer4_0_relu : [#users=1] = call_module[target=self_layer4_0_relu](args = (%self_layer4_0_bn1,), kwargs = {}) | |
| %self_layer4_0_conv2 : [#users=1] = call_module[target=self_layer4_0_conv2](args = (%self_layer4_0_relu,), kwargs = {}) | |
| %self_layer4_0_bn2 : [#users=1] = call_module[target=self_layer4_0_bn2](args = (%self_layer4_0_conv2,), kwargs = {}) | |
| %self_layer4_0_relu_1 : [#users=1] = call_module[target=self_layer4_0_relu](args = (%self_layer4_0_bn2,), kwargs = {}) | |
| %self_layer4_0_conv3 : [#users=1] = call_module[target=self_layer4_0_conv3](args = (%self_layer4_0_relu_1,), kwargs = {}) | |
| %self_layer4_0_bn3 : [#users=1] = call_module[target=self_layer4_0_bn3](args = (%self_layer4_0_conv3,), kwargs = {}) | |
| %self_layer4_0_downsample_0 : [#users=1] = call_module[target=self_layer4_0_downsample_0](args = (%self_layer3_5_relu_2,), kwargs = {}) | |
| %self_layer4_0_downsample_1 : [#users=1] = call_module[target=self_layer4_0_downsample_1](args = (%self_layer4_0_downsample_0,), kwargs = {}) | |
| %iadd_13 : [#users=1] = call_function[target=operator.iadd](args = (%self_layer4_0_bn3, %self_layer4_0_downsample_1), kwargs = {}) | |
| %self_layer4_0_relu_2 : [#users=2] = call_module[target=self_layer4_0_relu](args = (%iadd_13,), kwargs = {}) | |
| %self_layer4_1_conv1 : [#users=1] = call_module[target=self_layer4_1_conv1](args = (%self_layer4_0_relu_2,), kwargs = {}) | |
| %self_layer4_1_bn1 : [#users=1] = call_module[target=self_layer4_1_bn1](args = (%self_layer4_1_conv1,), kwargs = {}) | |
| %self_layer4_1_relu : [#users=1] = call_module[target=self_layer4_1_relu](args = (%self_layer4_1_bn1,), kwargs = {}) | |
| %self_layer4_1_conv2 : [#users=1] = call_module[target=self_layer4_1_conv2](args = (%self_layer4_1_relu,), kwargs = {}) | |
| %self_layer4_1_bn2 : [#users=1] = call_module[target=self_layer4_1_bn2](args = (%self_layer4_1_conv2,), kwargs = {}) | |
| %self_layer4_1_relu_1 : [#users=1] = call_module[target=self_layer4_1_relu](args = (%self_layer4_1_bn2,), kwargs = {}) | |
| %self_layer4_1_conv3 : [#users=1] = call_module[target=self_layer4_1_conv3](args = (%self_layer4_1_relu_1,), kwargs = {}) | |
| %self_layer4_1_bn3 : [#users=1] = call_module[target=self_layer4_1_bn3](args = (%self_layer4_1_conv3,), kwargs = {}) | |
| %iadd_14 : [#users=1] = call_function[target=operator.iadd](args = (%self_layer4_1_bn3, %self_layer4_0_relu_2), kwargs = {}) | |
| %self_layer4_1_relu_2 : [#users=2] = call_module[target=self_layer4_1_relu](args = (%iadd_14,), kwargs = {}) | |
| %self_layer4_2_conv1 : [#users=1] = call_module[target=self_layer4_2_conv1](args = (%self_layer4_1_relu_2,), kwargs = {}) | |
| %self_layer4_2_bn1 : [#users=1] = call_module[target=self_layer4_2_bn1](args = (%self_layer4_2_conv1,), kwargs = {}) | |
| %self_layer4_2_relu : [#users=1] = call_module[target=self_layer4_2_relu](args = (%self_layer4_2_bn1,), kwargs = {}) | |
| %self_layer4_2_conv2 : [#users=1] = call_module[target=self_layer4_2_conv2](args = (%self_layer4_2_relu,), kwargs = {}) | |
| %self_layer4_2_bn2 : [#users=1] = call_module[target=self_layer4_2_bn2](args = (%self_layer4_2_conv2,), kwargs = {}) | |
| %self_layer4_2_relu_1 : [#users=1] = call_module[target=self_layer4_2_relu](args = (%self_layer4_2_bn2,), kwargs = {}) | |
| %self_layer4_2_conv3 : [#users=1] = call_module[target=self_layer4_2_conv3](args = (%self_layer4_2_relu_1,), kwargs = {}) | |
| %self_layer4_2_bn3 : [#users=1] = call_module[target=self_layer4_2_bn3](args = (%self_layer4_2_conv3,), kwargs = {}) | |
| %iadd_15 : [#users=1] = call_function[target=operator.iadd](args = (%self_layer4_2_bn3, %self_layer4_1_relu_2), kwargs = {}) | |
| %self_layer4_2_relu_2 : [#users=1] = call_module[target=self_layer4_2_relu](args = (%iadd_15,), kwargs = {}) | |
| %self_avgpool : [#users=1] = call_module[target=self_avgpool](args = (%self_layer4_2_relu_2,), kwargs = {}) | |
| %flatten : [#users=1] = call_function[target=torch.flatten](args = (%self_avgpool, 1), kwargs = {}) | |
| %self_fc : [#users=1] = call_module[target=self_fc](args = (%flatten,), kwargs = {}) | |
| return (self_fc,) | |
| ---split graph--- | |
| graph(): | |
| %x : torch.Tensor [#users=1] = placeholder[target=x] | |
| %submod_0 : [#users=2] = call_module[target=submod_0](args = (%x,), kwargs = {}) | |
| %getitem : [#users=1] = call_function[target=operator.getitem](args = (%submod_0, 0), kwargs = {}) | |
| %getitem_1 : [#users=1] = call_function[target=operator.getitem](args = (%submod_0, 1), kwargs = {}) | |
| %submod_1 : [#users=2] = call_module[target=submod_1](args = (%getitem, %getitem_1), kwargs = {}) | |
| %getitem_2 : [#users=1] = call_function[target=operator.getitem](args = (%submod_1, 0), kwargs = {}) | |
| %getitem_3 : [#users=1] = call_function[target=operator.getitem](args = (%submod_1, 1), kwargs = {}) | |
| %submod_2 : [#users=2] = call_module[target=submod_2](args = (%getitem_2, %getitem_3), kwargs = {}) | |
| %getitem_4 : [#users=1] = call_function[target=operator.getitem](args = (%submod_2, 0), kwargs = {}) | |
| %getitem_5 : [#users=1] = call_function[target=operator.getitem](args = (%submod_2, 1), kwargs = {}) | |
| %submod_3 : [#users=1] = call_module[target=submod_3](args = (%getitem_4, %getitem_5), kwargs = {}) | |
| %submod_4 : [#users=1] = call_module[target=submod_4](args = (%submod_3,), kwargs = {}) | |
| return (submod_4,) | |
| --- graph--- | |
| graph(): | |
| %x : torch.Tensor [#users=1] = placeholder[target=x] | |
| %submod_0 : [#users=2] = call_module[target=submod_0](args = (%x,), kwargs = {}) | |
| %getitem : [#users=1] = call_function[target=operator.getitem](args = (%submod_0, 0), kwargs = {}) | |
| %getitem_1 : [#users=1] = call_function[target=operator.getitem](args = (%submod_0, 1), kwargs = {}) | |
| %submod_1 : [#users=2] = call_module[target=submod_1](args = (%getitem, %getitem_1), kwargs = {}) | |
| %getitem_2 : [#users=1] = call_function[target=operator.getitem](args = (%submod_1, 0), kwargs = {}) | |
| %getitem_3 : [#users=1] = call_function[target=operator.getitem](args = (%submod_1, 1), kwargs = {}) | |
| %submod_2 : [#users=2] = call_module[target=submod_2](args = (%getitem_2, %getitem_3), kwargs = {}) | |
| %getitem_4 : [#users=1] = call_function[target=operator.getitem](args = (%submod_2, 0), kwargs = {}) | |
| %getitem_5 : [#users=1] = call_function[target=operator.getitem](args = (%submod_2, 1), kwargs = {}) | |
| %submod_3 : [#users=1] = call_module[target=submod_3](args = (%getitem_4, %getitem_5), kwargs = {}) | |
| %submod_4 : [#users=1] = call_module[target=submod_4](args = (%submod_3,), kwargs = {}) | |
| return (submod_4,) | |
| ---submod_0 graph--- | |
| graph(): | |
| %x : [#users=1] = placeholder[target=x] | |
| %self_conv1 : [#users=1] = call_module[target=self_conv1](args = (%x,), kwargs = {}) | |
| %self_bn1 : [#users=1] = call_module[target=self_bn1](args = (%self_conv1,), kwargs = {}) | |
| %self_relu : [#users=1] = call_module[target=self_relu](args = (%self_bn1,), kwargs = {}) | |
| %self_maxpool : [#users=2] = call_module[target=self_maxpool](args = (%self_relu,), kwargs = {}) | |
| %self_layer1_0_conv1 : [#users=1] = call_module[target=self_layer1_0_conv1](args = (%self_maxpool,), kwargs = {}) | |
| %self_layer1_0_bn1 : [#users=1] = call_module[target=self_layer1_0_bn1](args = (%self_layer1_0_conv1,), kwargs = {}) | |
| %self_layer1_0_relu : [#users=1] = call_module[target=self_layer1_0_relu](args = (%self_layer1_0_bn1,), kwargs = {}) | |
| %self_layer1_0_conv2 : [#users=1] = call_module[target=self_layer1_0_conv2](args = (%self_layer1_0_relu,), kwargs = {}) | |
| %self_layer1_0_bn2 : [#users=1] = call_module[target=self_layer1_0_bn2](args = (%self_layer1_0_conv2,), kwargs = {}) | |
| %self_layer1_0_relu_1 : [#users=1] = call_module[target=self_layer1_0_relu](args = (%self_layer1_0_bn2,), kwargs = {}) | |
| %self_layer1_0_conv3 : [#users=1] = call_module[target=self_layer1_0_conv3](args = (%self_layer1_0_relu_1,), kwargs = {}) | |
| %self_layer1_0_bn3 : [#users=1] = call_module[target=self_layer1_0_bn3](args = (%self_layer1_0_conv3,), kwargs = {}) | |
| %self_layer1_0_downsample_0 : [#users=1] = call_module[target=self_layer1_0_downsample_0](args = (%self_maxpool,), kwargs = {}) | |
| %self_layer1_0_downsample_1 : [#users=1] = call_module[target=self_layer1_0_downsample_1](args = (%self_layer1_0_downsample_0,), kwargs = {}) | |
| %iadd : [#users=1] = call_function[target=operator.iadd](args = (%self_layer1_0_bn3, %self_layer1_0_downsample_1), kwargs = {}) | |
| %self_layer1_0_relu_2 : [#users=2] = call_module[target=self_layer1_0_relu](args = (%iadd,), kwargs = {}) | |
| %self_layer1_1_conv1 : [#users=1] = call_module[target=self_layer1_1_conv1](args = (%self_layer1_0_relu_2,), kwargs = {}) | |
| %self_layer1_1_bn1 : [#users=1] = call_module[target=self_layer1_1_bn1](args = (%self_layer1_1_conv1,), kwargs = {}) | |
| %self_layer1_1_relu : [#users=1] = call_module[target=self_layer1_1_relu](args = (%self_layer1_1_bn1,), kwargs = {}) | |
| %self_layer1_1_conv2 : [#users=1] = call_module[target=self_layer1_1_conv2](args = (%self_layer1_1_relu,), kwargs = {}) | |
| %self_layer1_1_bn2 : [#users=1] = call_module[target=self_layer1_1_bn2](args = (%self_layer1_1_conv2,), kwargs = {}) | |
| %self_layer1_1_relu_1 : [#users=1] = call_module[target=self_layer1_1_relu](args = (%self_layer1_1_bn2,), kwargs = {}) | |
| %self_layer1_1_conv3 : [#users=1] = call_module[target=self_layer1_1_conv3](args = (%self_layer1_1_relu_1,), kwargs = {}) | |
| %self_layer1_1_bn3 : [#users=1] = call_module[target=self_layer1_1_bn3](args = (%self_layer1_1_conv3,), kwargs = {}) | |
| %iadd_1 : [#users=1] = call_function[target=operator.iadd](args = (%self_layer1_1_bn3, %self_layer1_0_relu_2), kwargs = {}) | |
| %self_layer1_1_relu_2 : [#users=2] = call_module[target=self_layer1_1_relu](args = (%iadd_1,), kwargs = {}) | |
| %self_layer1_2_conv1 : [#users=1] = call_module[target=self_layer1_2_conv1](args = (%self_layer1_1_relu_2,), kwargs = {}) | |
| %self_layer1_2_bn1 : [#users=1] = call_module[target=self_layer1_2_bn1](args = (%self_layer1_2_conv1,), kwargs = {}) | |
| %self_layer1_2_relu : [#users=1] = call_module[target=self_layer1_2_relu](args = (%self_layer1_2_bn1,), kwargs = {}) | |
| %self_layer1_2_conv2 : [#users=1] = call_module[target=self_layer1_2_conv2](args = (%self_layer1_2_relu,), kwargs = {}) | |
| %self_layer1_2_bn2 : [#users=1] = call_module[target=self_layer1_2_bn2](args = (%self_layer1_2_conv2,), kwargs = {}) | |
| %self_layer1_2_relu_1 : [#users=1] = call_module[target=self_layer1_2_relu](args = (%self_layer1_2_bn2,), kwargs = {}) | |
| %self_layer1_2_conv3 : [#users=1] = call_module[target=self_layer1_2_conv3](args = (%self_layer1_2_relu_1,), kwargs = {}) | |
| %self_layer1_2_bn3 : [#users=1] = call_module[target=self_layer1_2_bn3](args = (%self_layer1_2_conv3,), kwargs = {}) | |
| %iadd_2 : [#users=1] = call_function[target=operator.iadd](args = (%self_layer1_2_bn3, %self_layer1_1_relu_2), kwargs = {}) | |
| %self_layer1_2_relu_2 : [#users=2] = call_module[target=self_layer1_2_relu](args = (%iadd_2,), kwargs = {}) | |
| %self_layer2_0_conv1 : [#users=1] = call_module[target=self_layer2_0_conv1](args = (%self_layer1_2_relu_2,), kwargs = {}) | |
| %self_layer2_0_bn1 : [#users=1] = call_module[target=self_layer2_0_bn1](args = (%self_layer2_0_conv1,), kwargs = {}) | |
| %self_layer2_0_relu : [#users=1] = call_module[target=self_layer2_0_relu](args = (%self_layer2_0_bn1,), kwargs = {}) | |
| %self_layer2_0_conv2 : [#users=1] = call_module[target=self_layer2_0_conv2](args = (%self_layer2_0_relu,), kwargs = {}) | |
| %self_layer2_0_bn2 : [#users=1] = call_module[target=self_layer2_0_bn2](args = (%self_layer2_0_conv2,), kwargs = {}) | |
| %self_layer2_0_relu_1 : [#users=1] = call_module[target=self_layer2_0_relu](args = (%self_layer2_0_bn2,), kwargs = {}) | |
| %self_layer2_0_conv3 : [#users=1] = call_module[target=self_layer2_0_conv3](args = (%self_layer2_0_relu_1,), kwargs = {}) | |
| %self_layer2_0_bn3 : [#users=1] = call_module[target=self_layer2_0_bn3](args = (%self_layer2_0_conv3,), kwargs = {}) | |
| %self_layer2_0_downsample_0 : [#users=1] = call_module[target=self_layer2_0_downsample_0](args = (%self_layer1_2_relu_2,), kwargs = {}) | |
| %self_layer2_0_downsample_1 : [#users=1] = call_module[target=self_layer2_0_downsample_1](args = (%self_layer2_0_downsample_0,), kwargs = {}) | |
| %iadd_3 : [#users=1] = call_function[target=operator.iadd](args = (%self_layer2_0_bn3, %self_layer2_0_downsample_1), kwargs = {}) | |
| %self_layer2_0_relu_2 : [#users=2] = call_module[target=self_layer2_0_relu](args = (%iadd_3,), kwargs = {}) | |
| %self_layer2_1_conv1 : [#users=1] = call_module[target=self_layer2_1_conv1](args = (%self_layer2_0_relu_2,), kwargs = {}) | |
| %self_layer2_1_bn1 : [#users=1] = call_module[target=self_layer2_1_bn1](args = (%self_layer2_1_conv1,), kwargs = {}) | |
| %self_layer2_1_relu : [#users=1] = call_module[target=self_layer2_1_relu](args = (%self_layer2_1_bn1,), kwargs = {}) | |
| %self_layer2_1_conv2 : [#users=1] = call_module[target=self_layer2_1_conv2](args = (%self_layer2_1_relu,), kwargs = {}) | |
| %self_layer2_1_bn2 : [#users=1] = call_module[target=self_layer2_1_bn2](args = (%self_layer2_1_conv2,), kwargs = {}) | |
| %self_layer2_1_relu_1 : [#users=1] = call_module[target=self_layer2_1_relu](args = (%self_layer2_1_bn2,), kwargs = {}) | |
| %self_layer2_1_conv3 : [#users=1] = call_module[target=self_layer2_1_conv3](args = (%self_layer2_1_relu_1,), kwargs = {}) | |
| %self_layer2_1_bn3 : [#users=1] = call_module[target=self_layer2_1_bn3](args = (%self_layer2_1_conv3,), kwargs = {}) | |
| %iadd_4 : [#users=1] = call_function[target=operator.iadd](args = (%self_layer2_1_bn3, %self_layer2_0_relu_2), kwargs = {}) | |
| %self_layer2_1_relu_2 : [#users=2] = call_module[target=self_layer2_1_relu](args = (%iadd_4,), kwargs = {}) | |
| %self_layer2_2_conv1 : [#users=1] = call_module[target=self_layer2_2_conv1](args = (%self_layer2_1_relu_2,), kwargs = {}) | |
| %self_layer2_2_bn1 : [#users=1] = call_module[target=self_layer2_2_bn1](args = (%self_layer2_2_conv1,), kwargs = {}) | |
| %self_layer2_2_relu : [#users=1] = call_module[target=self_layer2_2_relu](args = (%self_layer2_2_bn1,), kwargs = {}) | |
| %self_layer2_2_conv2 : [#users=1] = call_module[target=self_layer2_2_conv2](args = (%self_layer2_2_relu,), kwargs = {}) | |
| %self_layer2_2_bn2 : [#users=1] = call_module[target=self_layer2_2_bn2](args = (%self_layer2_2_conv2,), kwargs = {}) | |
| %self_layer2_2_relu_1 : [#users=1] = call_module[target=self_layer2_2_relu](args = (%self_layer2_2_bn2,), kwargs = {}) | |
| %self_layer2_2_conv3 : [#users=1] = call_module[target=self_layer2_2_conv3](args = (%self_layer2_2_relu_1,), kwargs = {}) | |
| %self_layer2_2_bn3 : [#users=1] = call_module[target=self_layer2_2_bn3](args = (%self_layer2_2_conv3,), kwargs = {}) | |
| %iadd_5 : [#users=1] = call_function[target=operator.iadd](args = (%self_layer2_2_bn3, %self_layer2_1_relu_2), kwargs = {}) | |
| %self_layer2_2_relu_2 : [#users=2] = call_module[target=self_layer2_2_relu](args = (%iadd_5,), kwargs = {}) | |
| %self_layer2_3_conv1 : [#users=1] = call_module[target=self_layer2_3_conv1](args = (%self_layer2_2_relu_2,), kwargs = {}) | |
| %self_layer2_3_bn1 : [#users=1] = call_module[target=self_layer2_3_bn1](args = (%self_layer2_3_conv1,), kwargs = {}) | |
| %self_layer2_3_relu : [#users=1] = call_module[target=self_layer2_3_relu](args = (%self_layer2_3_bn1,), kwargs = {}) | |
| %self_layer2_3_conv2 : [#users=1] = call_module[target=self_layer2_3_conv2](args = (%self_layer2_3_relu,), kwargs = {}) | |
| %self_layer2_3_bn2 : [#users=1] = call_module[target=self_layer2_3_bn2](args = (%self_layer2_3_conv2,), kwargs = {}) | |
| %self_layer2_3_relu_1 : [#users=1] = call_module[target=self_layer2_3_relu](args = (%self_layer2_3_bn2,), kwargs = {}) | |
| %self_layer2_3_conv3 : [#users=1] = call_module[target=self_layer2_3_conv3](args = (%self_layer2_3_relu_1,), kwargs = {}) | |
| %self_layer2_3_bn3 : [#users=1] = call_module[target=self_layer2_3_bn3](args = (%self_layer2_3_conv3,), kwargs = {}) | |
| %iadd_6 : [#users=1] = call_function[target=operator.iadd](args = (%self_layer2_3_bn3, %self_layer2_2_relu_2), kwargs = {}) | |
| %self_layer2_3_relu_2 : [#users=2] = call_module[target=self_layer2_3_relu](args = (%iadd_6,), kwargs = {}) | |
| %self_layer3_0_conv1 : [#users=1] = call_module[target=self_layer3_0_conv1](args = (%self_layer2_3_relu_2,), kwargs = {}) | |
| %self_layer3_0_bn1 : [#users=1] = call_module[target=self_layer3_0_bn1](args = (%self_layer3_0_conv1,), kwargs = {}) | |
| %self_layer3_0_relu : [#users=1] = call_module[target=self_layer3_0_relu](args = (%self_layer3_0_bn1,), kwargs = {}) | |
| %self_layer3_0_conv2 : [#users=1] = call_module[target=self_layer3_0_conv2](args = (%self_layer3_0_relu,), kwargs = {}) | |
| %self_layer3_0_bn2 : [#users=1] = call_module[target=self_layer3_0_bn2](args = (%self_layer3_0_conv2,), kwargs = {}) | |
| %self_layer3_0_relu_1 : [#users=1] = call_module[target=self_layer3_0_relu](args = (%self_layer3_0_bn2,), kwargs = {}) | |
| %self_layer3_0_conv3 : [#users=1] = call_module[target=self_layer3_0_conv3](args = (%self_layer3_0_relu_1,), kwargs = {}) | |
| %self_layer3_0_bn3 : [#users=1] = call_module[target=self_layer3_0_bn3](args = (%self_layer3_0_conv3,), kwargs = {}) | |
| return (self_layer2_3_relu_2, self_layer3_0_bn3) | |
| ---submod_1 graph--- | |
| graph(): | |
| %self_layer2_3_relu_2 : [#users=1] = placeholder[target=self_layer2_3_relu_2] | |
| %self_layer3_0_bn3 : [#users=1] = placeholder[target=self_layer3_0_bn3] | |
| %self_layer3_0_downsample_0 : [#users=1] = call_module[target=self_layer3_0_downsample_0](args = (%self_layer2_3_relu_2,), kwargs = {}) | |
| %self_layer3_0_downsample_1 : [#users=1] = call_module[target=self_layer3_0_downsample_1](args = (%self_layer3_0_downsample_0,), kwargs = {}) | |
| %iadd : [#users=1] = call_function[target=operator.iadd](args = (%self_layer3_0_bn3, %self_layer3_0_downsample_1), kwargs = {}) | |
| %self_layer3_0_relu : [#users=2] = call_module[target=self_layer3_0_relu](args = (%iadd,), kwargs = {}) | |
| %self_layer3_1_conv1 : [#users=1] = call_module[target=self_layer3_1_conv1](args = (%self_layer3_0_relu,), kwargs = {}) | |
| %self_layer3_1_bn1 : [#users=1] = call_module[target=self_layer3_1_bn1](args = (%self_layer3_1_conv1,), kwargs = {}) | |
| %self_layer3_1_relu : [#users=1] = call_module[target=self_layer3_1_relu](args = (%self_layer3_1_bn1,), kwargs = {}) | |
| %self_layer3_1_conv2 : [#users=1] = call_module[target=self_layer3_1_conv2](args = (%self_layer3_1_relu,), kwargs = {}) | |
| %self_layer3_1_bn2 : [#users=1] = call_module[target=self_layer3_1_bn2](args = (%self_layer3_1_conv2,), kwargs = {}) | |
| %self_layer3_1_relu_1 : [#users=1] = call_module[target=self_layer3_1_relu](args = (%self_layer3_1_bn2,), kwargs = {}) | |
| %self_layer3_1_conv3 : [#users=1] = call_module[target=self_layer3_1_conv3](args = (%self_layer3_1_relu_1,), kwargs = {}) | |
| %self_layer3_1_bn3 : [#users=1] = call_module[target=self_layer3_1_bn3](args = (%self_layer3_1_conv3,), kwargs = {}) | |
| %iadd_1 : [#users=1] = call_function[target=operator.iadd](args = (%self_layer3_1_bn3, %self_layer3_0_relu), kwargs = {}) | |
| %self_layer3_1_relu_2 : [#users=2] = call_module[target=self_layer3_1_relu](args = (%iadd_1,), kwargs = {}) | |
| %self_layer3_2_conv1 : [#users=1] = call_module[target=self_layer3_2_conv1](args = (%self_layer3_1_relu_2,), kwargs = {}) | |
| %self_layer3_2_bn1 : [#users=1] = call_module[target=self_layer3_2_bn1](args = (%self_layer3_2_conv1,), kwargs = {}) | |
| %self_layer3_2_relu : [#users=1] = call_module[target=self_layer3_2_relu](args = (%self_layer3_2_bn1,), kwargs = {}) | |
| %self_layer3_2_conv2 : [#users=1] = call_module[target=self_layer3_2_conv2](args = (%self_layer3_2_relu,), kwargs = {}) | |
| %self_layer3_2_bn2 : [#users=1] = call_module[target=self_layer3_2_bn2](args = (%self_layer3_2_conv2,), kwargs = {}) | |
| %self_layer3_2_relu_1 : [#users=1] = call_module[target=self_layer3_2_relu](args = (%self_layer3_2_bn2,), kwargs = {}) | |
| %self_layer3_2_conv3 : [#users=1] = call_module[target=self_layer3_2_conv3](args = (%self_layer3_2_relu_1,), kwargs = {}) | |
| %self_layer3_2_bn3 : [#users=1] = call_module[target=self_layer3_2_bn3](args = (%self_layer3_2_conv3,), kwargs = {}) | |
| %iadd_2 : [#users=1] = call_function[target=operator.iadd](args = (%self_layer3_2_bn3, %self_layer3_1_relu_2), kwargs = {}) | |
| %self_layer3_2_relu_2 : [#users=2] = call_module[target=self_layer3_2_relu](args = (%iadd_2,), kwargs = {}) | |
| %self_layer3_3_conv1 : [#users=1] = call_module[target=self_layer3_3_conv1](args = (%self_layer3_2_relu_2,), kwargs = {}) | |
| %self_layer3_3_bn1 : [#users=1] = call_module[target=self_layer3_3_bn1](args = (%self_layer3_3_conv1,), kwargs = {}) | |
| %self_layer3_3_relu : [#users=1] = call_module[target=self_layer3_3_relu](args = (%self_layer3_3_bn1,), kwargs = {}) | |
| %self_layer3_3_conv2 : [#users=1] = call_module[target=self_layer3_3_conv2](args = (%self_layer3_3_relu,), kwargs = {}) | |
| %self_layer3_3_bn2 : [#users=1] = call_module[target=self_layer3_3_bn2](args = (%self_layer3_3_conv2,), kwargs = {}) | |
| %self_layer3_3_relu_1 : [#users=1] = call_module[target=self_layer3_3_relu](args = (%self_layer3_3_bn2,), kwargs = {}) | |
| %self_layer3_3_conv3 : [#users=1] = call_module[target=self_layer3_3_conv3](args = (%self_layer3_3_relu_1,), kwargs = {}) | |
| %self_layer3_3_bn3 : [#users=1] = call_module[target=self_layer3_3_bn3](args = (%self_layer3_3_conv3,), kwargs = {}) | |
| %iadd_3 : [#users=1] = call_function[target=operator.iadd](args = (%self_layer3_3_bn3, %self_layer3_2_relu_2), kwargs = {}) | |
| %self_layer3_3_relu_2 : [#users=2] = call_module[target=self_layer3_3_relu](args = (%iadd_3,), kwargs = {}) | |
| %self_layer3_4_conv1 : [#users=1] = call_module[target=self_layer3_4_conv1](args = (%self_layer3_3_relu_2,), kwargs = {}) | |
| %self_layer3_4_bn1 : [#users=1] = call_module[target=self_layer3_4_bn1](args = (%self_layer3_4_conv1,), kwargs = {}) | |
| %self_layer3_4_relu : [#users=1] = call_module[target=self_layer3_4_relu](args = (%self_layer3_4_bn1,), kwargs = {}) | |
| %self_layer3_4_conv2 : [#users=1] = call_module[target=self_layer3_4_conv2](args = (%self_layer3_4_relu,), kwargs = {}) | |
| %self_layer3_4_bn2 : [#users=1] = call_module[target=self_layer3_4_bn2](args = (%self_layer3_4_conv2,), kwargs = {}) | |
| %self_layer3_4_relu_1 : [#users=1] = call_module[target=self_layer3_4_relu](args = (%self_layer3_4_bn2,), kwargs = {}) | |
| %self_layer3_4_conv3 : [#users=1] = call_module[target=self_layer3_4_conv3](args = (%self_layer3_4_relu_1,), kwargs = {}) | |
| %self_layer3_4_bn3 : [#users=1] = call_module[target=self_layer3_4_bn3](args = (%self_layer3_4_conv3,), kwargs = {}) | |
| %iadd_4 : [#users=1] = call_function[target=operator.iadd](args = (%self_layer3_4_bn3, %self_layer3_3_relu_2), kwargs = {}) | |
| %self_layer3_4_relu_2 : [#users=2] = call_module[target=self_layer3_4_relu](args = (%iadd_4,), kwargs = {}) | |
| %self_layer3_5_conv1 : [#users=1] = call_module[target=self_layer3_5_conv1](args = (%self_layer3_4_relu_2,), kwargs = {}) | |
| %self_layer3_5_bn1 : [#users=1] = call_module[target=self_layer3_5_bn1](args = (%self_layer3_5_conv1,), kwargs = {}) | |
| %self_layer3_5_relu : [#users=1] = call_module[target=self_layer3_5_relu](args = (%self_layer3_5_bn1,), kwargs = {}) | |
| %self_layer3_5_conv2 : [#users=1] = call_module[target=self_layer3_5_conv2](args = (%self_layer3_5_relu,), kwargs = {}) | |
| %self_layer3_5_bn2 : [#users=1] = call_module[target=self_layer3_5_bn2](args = (%self_layer3_5_conv2,), kwargs = {}) | |
| %self_layer3_5_relu_1 : [#users=1] = call_module[target=self_layer3_5_relu](args = (%self_layer3_5_bn2,), kwargs = {}) | |
| %self_layer3_5_conv3 : [#users=1] = call_module[target=self_layer3_5_conv3](args = (%self_layer3_5_relu_1,), kwargs = {}) | |
| %self_layer3_5_bn3 : [#users=1] = call_module[target=self_layer3_5_bn3](args = (%self_layer3_5_conv3,), kwargs = {}) | |
| %iadd_5 : [#users=1] = call_function[target=operator.iadd](args = (%self_layer3_5_bn3, %self_layer3_4_relu_2), kwargs = {}) | |
| %self_layer3_5_relu_2 : [#users=2] = call_module[target=self_layer3_5_relu](args = (%iadd_5,), kwargs = {}) | |
| %self_layer4_0_conv1 : [#users=1] = call_module[target=self_layer4_0_conv1](args = (%self_layer3_5_relu_2,), kwargs = {}) | |
| %self_layer4_0_bn1 : [#users=1] = call_module[target=self_layer4_0_bn1](args = (%self_layer4_0_conv1,), kwargs = {}) | |
| %self_layer4_0_relu : [#users=1] = call_module[target=self_layer4_0_relu](args = (%self_layer4_0_bn1,), kwargs = {}) | |
| return (self_layer4_0_relu, self_layer3_5_relu_2) | |
| ---submod_2 graph--- | |
| graph(): | |
| %self_layer4_0_relu : [#users=1] = placeholder[target=self_layer4_0_relu] | |
| %self_layer3_5_relu_2 : [#users=1] = placeholder[target=self_layer3_5_relu_2] | |
| %self_layer4_0_conv2 : [#users=1] = call_module[target=self_layer4_0_conv2](args = (%self_layer4_0_relu,), kwargs = {}) | |
| %self_layer4_0_bn2 : [#users=1] = call_module[target=self_layer4_0_bn2](args = (%self_layer4_0_conv2,), kwargs = {}) | |
| %self_layer4_0_relu_1 : [#users=1] = call_module[target=self_layer4_0_relu](args = (%self_layer4_0_bn2,), kwargs = {}) | |
| %self_layer4_0_conv3 : [#users=1] = call_module[target=self_layer4_0_conv3](args = (%self_layer4_0_relu_1,), kwargs = {}) | |
| %self_layer4_0_bn3 : [#users=1] = call_module[target=self_layer4_0_bn3](args = (%self_layer4_0_conv3,), kwargs = {}) | |
| %self_layer4_0_downsample_0 : [#users=1] = call_module[target=self_layer4_0_downsample_0](args = (%self_layer3_5_relu_2,), kwargs = {}) | |
| %self_layer4_0_downsample_1 : [#users=1] = call_module[target=self_layer4_0_downsample_1](args = (%self_layer4_0_downsample_0,), kwargs = {}) | |
| %iadd : [#users=1] = call_function[target=operator.iadd](args = (%self_layer4_0_bn3, %self_layer4_0_downsample_1), kwargs = {}) | |
| %self_layer4_0_relu_2 : [#users=2] = call_module[target=self_layer4_0_relu](args = (%iadd,), kwargs = {}) | |
| %self_layer4_1_conv1 : [#users=1] = call_module[target=self_layer4_1_conv1](args = (%self_layer4_0_relu_2,), kwargs = {}) | |
| %self_layer4_1_bn1 : [#users=1] = call_module[target=self_layer4_1_bn1](args = (%self_layer4_1_conv1,), kwargs = {}) | |
| %self_layer4_1_relu : [#users=1] = call_module[target=self_layer4_1_relu](args = (%self_layer4_1_bn1,), kwargs = {}) | |
| return (self_layer4_1_relu, self_layer4_0_relu_2) | |
| ---submod_3 graph--- | |
| graph(): | |
| %self_layer4_1_relu : [#users=1] = placeholder[target=self_layer4_1_relu] | |
| %self_layer4_0_relu_2 : [#users=1] = placeholder[target=self_layer4_0_relu_2] | |
| %self_layer4_1_conv2 : [#users=1] = call_module[target=self_layer4_1_conv2](args = (%self_layer4_1_relu,), kwargs = {}) | |
| %self_layer4_1_bn2 : [#users=1] = call_module[target=self_layer4_1_bn2](args = (%self_layer4_1_conv2,), kwargs = {}) | |
| %self_layer4_1_relu_1 : [#users=1] = call_module[target=self_layer4_1_relu](args = (%self_layer4_1_bn2,), kwargs = {}) | |
| %self_layer4_1_conv3 : [#users=1] = call_module[target=self_layer4_1_conv3](args = (%self_layer4_1_relu_1,), kwargs = {}) | |
| %self_layer4_1_bn3 : [#users=1] = call_module[target=self_layer4_1_bn3](args = (%self_layer4_1_conv3,), kwargs = {}) | |
| %iadd : [#users=1] = call_function[target=operator.iadd](args = (%self_layer4_1_bn3, %self_layer4_0_relu_2), kwargs = {}) | |
| %self_layer4_1_relu_2 : [#users=2] = call_module[target=self_layer4_1_relu](args = (%iadd,), kwargs = {}) | |
| %self_layer4_2_conv1 : [#users=1] = call_module[target=self_layer4_2_conv1](args = (%self_layer4_1_relu_2,), kwargs = {}) | |
| %self_layer4_2_bn1 : [#users=1] = call_module[target=self_layer4_2_bn1](args = (%self_layer4_2_conv1,), kwargs = {}) | |
| %self_layer4_2_relu : [#users=1] = call_module[target=self_layer4_2_relu](args = (%self_layer4_2_bn1,), kwargs = {}) | |
| %self_layer4_2_conv2 : [#users=1] = call_module[target=self_layer4_2_conv2](args = (%self_layer4_2_relu,), kwargs = {}) | |
| %self_layer4_2_bn2 : [#users=1] = call_module[target=self_layer4_2_bn2](args = (%self_layer4_2_conv2,), kwargs = {}) | |
| %self_layer4_2_relu_1 : [#users=1] = call_module[target=self_layer4_2_relu](args = (%self_layer4_2_bn2,), kwargs = {}) | |
| %self_layer4_2_conv3 : [#users=1] = call_module[target=self_layer4_2_conv3](args = (%self_layer4_2_relu_1,), kwargs = {}) | |
| %self_layer4_2_bn3 : [#users=1] = call_module[target=self_layer4_2_bn3](args = (%self_layer4_2_conv3,), kwargs = {}) | |
| %iadd_1 : [#users=1] = call_function[target=operator.iadd](args = (%self_layer4_2_bn3, %self_layer4_1_relu_2), kwargs = {}) | |
| %self_layer4_2_relu_2 : [#users=1] = call_module[target=self_layer4_2_relu](args = (%iadd_1,), kwargs = {}) | |
| %self_avgpool : [#users=1] = call_module[target=self_avgpool](args = (%self_layer4_2_relu_2,), kwargs = {}) | |
| %flatten : [#users=1] = call_function[target=torch.flatten](args = (%self_avgpool, 1), kwargs = {}) | |
| return flatten | |
| ---submod_4 graph--- | |
| graph(): | |
| %flatten : [#users=1] = placeholder[target=flatten] | |
| %self_fc : [#users=1] = call_module[target=self_fc](args = (%flatten,), kwargs = {}) | |
| return self_fc | |
| --------------- | |
| run_node placeholder, x got args tuple() | |
| run_node call_module, submod_0 got args tuple(T[torch.Size([32, 3, 224, 224])]) | |
| still waiting for process to exit | |
| didn't get the result, queue empty | |
| run_node call_function, <built-in function getitem> got args tuple(tuple(T[torch.Size([32, 512, 28, 28])], T[torch.Size([32, 1024, 14, 14])]), 0) | |
| run_node call_function, <built-in function getitem> got args tuple(tuple(T[torch.Size([32, 512, 28, 28])], T[torch.Size([32, 1024, 14, 14])]), 1) | |
| run_node call_module, submod_1 got args tuple(T[torch.Size([32, 512, 28, 28])], T[torch.Size([32, 1024, 14, 14])]) | |
| got exit 1 | |
| didn't get the result, queue empty | |
| exit code: 1 and result: {'nodes': 1, 'model_name': 'torchbenchmark.models.resnet50.Model', 'backend': 'torchdynamo_inductor', 'has_breaks': True, 'static_graph': False, 'result': None} | |
| <RESULT>{"nodes": 1, "model_name": "torchbenchmark.models.resnet50.Model", "backend": "torchdynamo_inductor", "has_breaks": true, "static_graph": false, "result": null}</RESULT> | |
| submitit INFO (2022-10-26 04:23:25,105) - Job completed successfully |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| submitit INFO (2022-10-26 02:47:41,285) - Starting with JobEnvironment(job_id=74173, hostname=a100-st-p4d24xlarge-24, local_rank=0(8), node=0(2), global_rank=0(16)) | |
| submitit INFO (2022-10-26 02:47:41,286) - Loading pickle: /fsx/users/dberard/scratch-local/bench-fast/benchmark/logs_oct25/74173_submitted.pkl | |
| This is node 0 | |
| run_once | |
| Process group: 16 tasks, rank: 0 | |
| MY HOSTNAME: a100-st-p4d24xlarge-24 | |
| FI_PROVIDER : efa | |
| LD_LIBRARY_PATH : /fsx/users/dberard/scratch-local/bench-fast/aws-ofi-nccl/lib:/opt/amazon/efa/lib:/usr/local/cuda-11.6/lib:/usr/local/cuda-11.6/lib64:/opt/amazon/efa/lib:/opt/amazon/openmpi/lib:/usr/local/cuda/efa/lib:/usr/local/cuda/lib:/usr/local/cuda:/usr/local/cuda/lib64:/usr/local/cuda/extras/CUPTI/lib64:/usr/local/cuda/targets/x86_64-linux/lib:/usr/local/lib:/usr/lib: | |
| NCCL_DEBUG : INFO | |
| FI_EFA_USE_DEVICE_RDMA : 1 | |
| still waiting for process to exit | |
| didn't get the result, queue empty | |
| a100-st-p4d24xlarge-24:97518:97518 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to ens | |
| a100-st-p4d24xlarge-24:97518:97518 [0] NCCL INFO NCCL_SOCKET_IFNAME set to ens | |
| a100-st-p4d24xlarge-24:97518:97518 [0] NCCL INFO Bootstrap : Using ens32:10.200.74.168<0> | |
| a100-st-p4d24xlarge-24:97518:97518 [0] NCCL INFO NET/Plugin: Failed to find ncclCollNetPlugin_v6 symbol. | |
| a100-st-p4d24xlarge-24:97518:97518 [0] NCCL INFO NET/Plugin: Failed to find ncclCollNetPlugin symbol (v4 or v5). | |
| a100-st-p4d24xlarge-24:97518:97518 [0] NCCL INFO cudaDriverVersion 11060 | |
| NCCL version 2.14.3+cuda11.6 | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO NET/OFI Using aws-ofi-nccl 1.4.0aws | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO NET/OFI Running on p4d.24xlarge platform, Setting NCCL_TOPO_FILE environment variable to /fsx/users/dberard/scratch-local/bench-fast/aws-ofi-nccl/share/aws-ofi-nccl/xml/p4d-24xl-topo.xml | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO NET/OFI Setting FI_EFA_FORK_SAFE environment variable to 1 | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO NET/OFI Selected Provider is efa | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO Using network AWS Libfabric | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO NCCL_TOPO_FILE set by environment to /fsx/users/dberard/scratch-local/bench-fast/aws-ofi-nccl/share/aws-ofi-nccl/xml/p4d-24xl-topo.xml | |
| still waiting for process to exit | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO NET/AWS Libfabric : GPU Direct RDMA Enabled for HCA 0 'rdmap16s27' | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO NET/AWS Libfabric : GPU Direct RDMA Enabled for HCA 1 'rdmap32s27' | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO NET/AWS Libfabric : GPU Direct RDMA Enabled for HCA 2 'rdmap144s27' | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO NET/AWS Libfabric : GPU Direct RDMA Enabled for HCA 3 'rdmap160s27' | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 101c0 / HCA 0 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 101d0 / HCA 0 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 201c0 / HCA 0 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 201d0 / HCA 0 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 901c0 / HCA 0 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 901d0 / HCA 0 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO GPU Direct RDMA Enabled for GPU a01c0 / HCA 0 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO GPU Direct RDMA Enabled for GPU a01d0 / HCA 0 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 101c0 / HCA 1 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 101d0 / HCA 1 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 201c0 / HCA 1 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 201d0 / HCA 1 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 901c0 / HCA 1 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 901d0 / HCA 1 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO GPU Direct RDMA Enabled for GPU a01c0 / HCA 1 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO GPU Direct RDMA Enabled for GPU a01d0 / HCA 1 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 101c0 / HCA 2 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 101d0 / HCA 2 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 201c0 / HCA 2 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 201d0 / HCA 2 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 901c0 / HCA 2 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 901d0 / HCA 2 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO GPU Direct RDMA Enabled for GPU a01c0 / HCA 2 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO GPU Direct RDMA Enabled for GPU a01d0 / HCA 2 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 101c0 / HCA 3 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 101d0 / HCA 3 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 201c0 / HCA 3 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 201d0 / HCA 3 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 901c0 / HCA 3 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 901d0 / HCA 3 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO GPU Direct RDMA Enabled for GPU a01c0 / HCA 3 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO GPU Direct RDMA Enabled for GPU a01d0 / HCA 3 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 101c0 / HCA 0 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 101d0 / HCA 0 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 201c0 / HCA 0 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 201d0 / HCA 0 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 901c0 / HCA 0 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 901d0 / HCA 0 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO GPU Direct RDMA Enabled for GPU a01c0 / HCA 0 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO GPU Direct RDMA Enabled for GPU a01d0 / HCA 0 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 101c0 / HCA 1 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 101d0 / HCA 1 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 201c0 / HCA 1 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 201d0 / HCA 1 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 901c0 / HCA 1 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 901d0 / HCA 1 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO GPU Direct RDMA Enabled for GPU a01c0 / HCA 1 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO GPU Direct RDMA Enabled for GPU a01d0 / HCA 1 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 101c0 / HCA 2 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 101d0 / HCA 2 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 201c0 / HCA 2 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 201d0 / HCA 2 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 901c0 / HCA 2 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 901d0 / HCA 2 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO GPU Direct RDMA Enabled for GPU a01c0 / HCA 2 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO GPU Direct RDMA Enabled for GPU a01d0 / HCA 2 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 101c0 / HCA 3 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 101d0 / HCA 3 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 201c0 / HCA 3 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 201d0 / HCA 3 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 901c0 / HCA 3 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 901d0 / HCA 3 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO GPU Direct RDMA Enabled for GPU a01c0 / HCA 3 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO GPU Direct RDMA Enabled for GPU a01d0 / HCA 3 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO Setting affinity for GPU 0 to 1f0000,0000001f | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO Channel 00/08 : 0 7 6 5 4 3 2 1 8 15 14 13 12 11 10 9 | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO Channel 01/08 : 0 3 10 15 14 13 12 9 8 11 2 7 6 5 4 1 | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO Channel 02/08 : 0 7 6 5 12 11 10 9 8 15 14 13 4 3 2 1 | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO Channel 03/08 : 0 5 4 7 14 11 10 9 8 13 12 15 6 3 2 1 | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO Channel 04/08 : 0 7 6 5 4 3 2 1 8 15 14 13 12 11 10 9 | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO Channel 05/08 : 0 3 10 15 14 13 12 9 8 11 2 7 6 5 4 1 | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO Channel 06/08 : 0 7 6 5 12 11 10 9 8 15 14 13 4 3 2 1 | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO Channel 07/08 : 0 5 4 7 14 11 10 9 8 13 12 15 6 3 2 1 | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO Trees [0] 1/8/-1->0->-1 [1] 1/-1/-1->0->7 [2] 1/-1/-1->0->7 [3] 1/-1/-1->0->7 [4] 1/-1/-1->0->8 [5] 1/-1/-1->0->7 [6] 1/-1/-1->0->7 [7] 1/-1/-1->0->7 | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO Channel 01/0 : 0[101c0] -> 3[201d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f3538002dd0 | |
| a100-st-p4d24xlarge-24:97518:98071 [0] NCCL INFO New proxy send connection 0 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO Channel 05/0 : 0[101c0] -> 3[201d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f3538002e10 | |
| a100-st-p4d24xlarge-24:97518:98071 [0] NCCL INFO New proxy send connection 1 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO Channel 03/0 : 0[101c0] -> 5[901d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f3538002e50 | |
| a100-st-p4d24xlarge-24:97518:98071 [0] NCCL INFO New proxy send connection 2 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO Channel 07/0 : 0[101c0] -> 5[901d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-24:97518:98071 [0] NCCL INFO New proxy send connection 3 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f3538002e90 | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 101c0 / HCA 0 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:97518:98071 [0] NCCL INFO New proxy recv connection 4 from local rank 0, transport 2 | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f3538002ed0 | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO Channel 00/0 : 9[101d0] -> 0[101c0] [receive] via NET/AWS Libfabric/0/GDRDMA | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 101c0 / HCA 0 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f3538002f10 | |
| a100-st-p4d24xlarge-24:97518:98071 [0] NCCL INFO New proxy recv connection 5 from local rank 0, transport 2 | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO Channel 04/0 : 9[101d0] -> 0[101c0] [receive] via NET/AWS Libfabric/0/GDRDMA | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO Channel 00/0 : 0[101c0] -> 7[a01d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-24:97518:98071 [0] NCCL INFO New proxy send connection 6 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f3538002f50 | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO Channel 02/0 : 0[101c0] -> 7[a01d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-24:97518:98071 [0] NCCL INFO New proxy send connection 7 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f3538002f90 | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO Channel 04/0 : 0[101c0] -> 7[a01d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-24:97518:98071 [0] NCCL INFO New proxy send connection 8 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f3538002fd0 | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO Channel 06/0 : 0[101c0] -> 7[a01d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f3538003010 | |
| a100-st-p4d24xlarge-24:97518:98071 [0] NCCL INFO New proxy send connection 9 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f3538003050 | |
| a100-st-p4d24xlarge-24:97518:98071 [0] NCCL INFO New proxy recv connection 10 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f3538003090 | |
| a100-st-p4d24xlarge-24:97518:98071 [0] NCCL INFO New proxy recv connection 11 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f35380030d0 | |
| a100-st-p4d24xlarge-24:97518:98071 [0] NCCL INFO New proxy recv connection 12 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f3538003110 | |
| a100-st-p4d24xlarge-24:97518:98071 [0] NCCL INFO New proxy recv connection 13 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f3538003150 | |
| a100-st-p4d24xlarge-24:97518:98071 [0] NCCL INFO New proxy recv connection 14 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f3538003190 | |
| a100-st-p4d24xlarge-24:97518:98071 [0] NCCL INFO New proxy recv connection 15 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO Connected all rings | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO Channel 00/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f35380031d0 | |
| a100-st-p4d24xlarge-24:97518:98071 [0] NCCL INFO New proxy send connection 16 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO Channel 01/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-24:97518:98071 [0] NCCL INFO New proxy send connection 17 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f3538003210 | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO Channel 02/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-24:97518:98071 [0] NCCL INFO New proxy send connection 18 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f3538003250 | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO Channel 03/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-24:97518:98071 [0] NCCL INFO New proxy send connection 19 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f3538003290 | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO Channel 04/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-24:97518:98071 [0] NCCL INFO New proxy send connection 20 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f35380032d0 | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO Channel 05/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-24:97518:98071 [0] NCCL INFO New proxy send connection 21 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f3538003310 | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO Channel 06/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-24:97518:98071 [0] NCCL INFO New proxy send connection 22 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f3538003350 | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO Channel 07/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-24:97518:98071 [0] NCCL INFO New proxy send connection 23 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f3538003390 | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO Channel 01/0 : 0[101c0] -> 7[a01d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-24:97518:98071 [0] NCCL INFO New proxy send connection 24 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f35380033d0 | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO Channel 03/0 : 0[101c0] -> 7[a01d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-24:97518:98071 [0] NCCL INFO New proxy send connection 25 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f3538003410 | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO Channel 05/0 : 0[101c0] -> 7[a01d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-24:97518:98071 [0] NCCL INFO New proxy send connection 26 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f3538003450 | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO Channel 07/0 : 0[101c0] -> 7[a01d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-24:97518:98071 [0] NCCL INFO New proxy send connection 27 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f3538003490 | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 101c0 / HCA 0 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f35380034d0 | |
| a100-st-p4d24xlarge-24:97518:98071 [0] NCCL INFO New proxy recv connection 28 from local rank 0, transport 2 | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO Channel 00/0 : 8[101c0] -> 0[101c0] [receive] via NET/AWS Libfabric/0/GDRDMA | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 101c0 / HCA 0 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:97518:98071 [0] NCCL INFO New proxy recv connection 29 from local rank 0, transport 2 | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f3538003510 | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO Channel 04/0 : 8[101c0] -> 0[101c0] [receive] via NET/AWS Libfabric/0/GDRDMA | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 101c0 / HCA 0 (distance 3 <= 4), read 1 | |
| a100-st-p4d24xlarge-24:97518:98071 [0] NCCL INFO New proxy send connection 30 from local rank 0, transport 2 | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f3538003550 | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO Channel 00/0 : 0[101c0] -> 8[101c0] [send] via NET/AWS Libfabric/0/GDRDMA | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 101c0 / HCA 0 (distance 3 <= 4), read 1 | |
| a100-st-p4d24xlarge-24:97518:98071 [0] NCCL INFO New proxy send connection 31 from local rank 0, transport 2 | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f3538003590 | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO Channel 04/0 : 0[101c0] -> 8[101c0] [send] via NET/AWS Libfabric/0/GDRDMA | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f35380035d0 | |
| a100-st-p4d24xlarge-24:97518:98071 [0] NCCL INFO New proxy recv connection 32 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f3538003610 | |
| a100-st-p4d24xlarge-24:97518:98071 [0] NCCL INFO New proxy recv connection 33 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f3538003650 | |
| a100-st-p4d24xlarge-24:97518:98071 [0] NCCL INFO New proxy recv connection 34 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f3538003690 | |
| a100-st-p4d24xlarge-24:97518:98071 [0] NCCL INFO New proxy recv connection 35 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f35380036d0 | |
| a100-st-p4d24xlarge-24:97518:98071 [0] NCCL INFO New proxy recv connection 36 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f3538003710 | |
| a100-st-p4d24xlarge-24:97518:98071 [0] NCCL INFO New proxy recv connection 37 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f3538003750 | |
| a100-st-p4d24xlarge-24:97518:98071 [0] NCCL INFO New proxy recv connection 38 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f3538003790 | |
| a100-st-p4d24xlarge-24:97518:98071 [0] NCCL INFO New proxy recv connection 39 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO Connected all trees | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO NCCL_ALGO set by environment to ring | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 512 | 512 | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO 8 coll channels, 8 p2p channels, 2 p2p channels per peer | |
| a100-st-p4d24xlarge-24:97518:98071 [0] NCCL INFO New proxy send connection 40 from local rank 0, transport 2 | |
| a100-st-p4d24xlarge-24:97518:98071 [0] NCCL INFO New proxy send connection 41 from local rank 1, transport 2 | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f35380037d0 | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 101c0 / HCA 1 (distance 3 <= 4), read 1 | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 101c0 / HCA 1 (distance 3 <= 4), read 1 | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 101c0 / HCA 2 (distance 3 <= 4), read 1 | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 101c0 / HCA 2 (distance 3 <= 4), read 1 | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 101c0 / HCA 3 (distance 3 <= 4), read 1 | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 101c0 / HCA 3 (distance 3 <= 4), read 1 | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 101c0 / HCA 1 (distance 3 <= 4), read 1 | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 101c0 / HCA 1 (distance 3 <= 4), read 1 | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 101c0 / HCA 2 (distance 3 <= 4), read 1 | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 101c0 / HCA 2 (distance 3 <= 4), read 1 | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 101c0 / HCA 3 (distance 3 <= 4), read 1 | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 101c0 / HCA 3 (distance 3 <= 4), read 1 | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO Connection to proxy localRank 2 -> connection 0x7f60dc003690 | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO Connection to proxy localRank 4 -> connection 0x7f4b6c003690 | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO Connection to proxy localRank 6 -> connection 0x7f7c6c003690 | |
| a100-st-p4d24xlarge-24:97518:98071 [0] NCCL INFO New proxy send connection 42 from local rank 7, transport 2 | |
| a100-st-p4d24xlarge-24:97518:98071 [0] NCCL INFO New proxy send connection 43 from local rank 6, transport 2 | |
| a100-st-p4d24xlarge-24:97518:98071 [0] NCCL INFO New proxy send connection 44 from local rank 2, transport 2 | |
| a100-st-p4d24xlarge-24:97518:98071 [0] NCCL INFO New proxy send connection 45 from local rank 5, transport 2 | |
| a100-st-p4d24xlarge-24:97518:98071 [0] NCCL INFO New proxy send connection 46 from local rank 3, transport 2 | |
| a100-st-p4d24xlarge-24:97518:98071 [0] NCCL INFO New proxy send connection 47 from local rank 4, transport 2 | |
| a100-st-p4d24xlarge-24:97518:98032 [0] NCCL INFO comm 0x563e7e05bc30 rank 0 nranks 16 cudaDev 0 busId 101c0 - Init COMPLETE | |
| didn't get the result, queue empty | |
| still waiting for process to exit | |
| didn't get the result, queue empty | |
| still waiting for process to exit | |
| didn't get the result, queue empty | |
| result {'latency_median': 76.04889678955078, 'latency_stdev': 1.490786349319905} | |
| has been placed into the queue | |
| got exit 0 | |
| got the result! | |
| exit code: 0 and result: {'nodes': 2, 'model_name': 'torchbenchmark.models.resnet50.Model', 'backend': 'eager', 'has_breaks': False, 'static_graph': False, 'result': {'latency_median': 76.04889678955078, 'latency_stdev': 1.490786349319905}} | |
| <RESULT>{"nodes": 2, "model_name": "torchbenchmark.models.resnet50.Model", "backend": "eager", "has_breaks": false, "static_graph": false, "result": {"latency_median": 76.04889678955078, "latency_stdev": 1.490786349319905}}</RESULT> | |
| run_once | |
| Process group: 16 tasks, rank: 0 | |
| MY HOSTNAME: a100-st-p4d24xlarge-24 | |
| FI_PROVIDER : efa | |
| LD_LIBRARY_PATH : /fsx/users/dberard/scratch-local/bench-fast/aws-ofi-nccl/lib:/opt/amazon/efa/lib:/usr/local/cuda-11.6/lib:/usr/local/cuda-11.6/lib64:/opt/amazon/efa/lib:/opt/amazon/openmpi/lib:/usr/local/cuda/efa/lib:/usr/local/cuda/lib:/usr/local/cuda:/usr/local/cuda/lib64:/usr/local/cuda/extras/CUPTI/lib64:/usr/local/cuda/targets/x86_64-linux/lib:/usr/local/lib:/usr/lib: | |
| NCCL_DEBUG : INFO | |
| FI_EFA_USE_DEVICE_RDMA : 1 | |
| still waiting for process to exit | |
| didn't get the result, queue empty | |
| a100-st-p4d24xlarge-24:917:917 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to ens | |
| a100-st-p4d24xlarge-24:917:917 [0] NCCL INFO NCCL_SOCKET_IFNAME set to ens | |
| a100-st-p4d24xlarge-24:917:917 [0] NCCL INFO Bootstrap : Using ens32:10.200.74.168<0> | |
| a100-st-p4d24xlarge-24:917:917 [0] NCCL INFO NET/Plugin: Failed to find ncclCollNetPlugin_v6 symbol. | |
| a100-st-p4d24xlarge-24:917:917 [0] NCCL INFO NET/Plugin: Failed to find ncclCollNetPlugin symbol (v4 or v5). | |
| a100-st-p4d24xlarge-24:917:917 [0] NCCL INFO cudaDriverVersion 11060 | |
| NCCL version 2.14.3+cuda11.6 | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO NET/OFI Using aws-ofi-nccl 1.4.0aws | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO NET/OFI Running on p4d.24xlarge platform, Setting NCCL_TOPO_FILE environment variable to /fsx/users/dberard/scratch-local/bench-fast/aws-ofi-nccl/share/aws-ofi-nccl/xml/p4d-24xl-topo.xml | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO NET/OFI Setting FI_EFA_FORK_SAFE environment variable to 1 | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO NET/OFI Selected Provider is efa | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO Using network AWS Libfabric | |
| still waiting for process to exit | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO NCCL_TOPO_FILE set by environment to /fsx/users/dberard/scratch-local/bench-fast/aws-ofi-nccl/share/aws-ofi-nccl/xml/p4d-24xl-topo.xml | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO NET/AWS Libfabric : GPU Direct RDMA Enabled for HCA 0 'rdmap16s27' | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO NET/AWS Libfabric : GPU Direct RDMA Enabled for HCA 1 'rdmap32s27' | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO NET/AWS Libfabric : GPU Direct RDMA Enabled for HCA 2 'rdmap144s27' | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO NET/AWS Libfabric : GPU Direct RDMA Enabled for HCA 3 'rdmap160s27' | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 101c0 / HCA 0 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 101d0 / HCA 0 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 201c0 / HCA 0 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 201d0 / HCA 0 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 901c0 / HCA 0 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 901d0 / HCA 0 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO GPU Direct RDMA Enabled for GPU a01c0 / HCA 0 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO GPU Direct RDMA Enabled for GPU a01d0 / HCA 0 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 101c0 / HCA 1 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 101d0 / HCA 1 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 201c0 / HCA 1 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 201d0 / HCA 1 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 901c0 / HCA 1 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 901d0 / HCA 1 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO GPU Direct RDMA Enabled for GPU a01c0 / HCA 1 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO GPU Direct RDMA Enabled for GPU a01d0 / HCA 1 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 101c0 / HCA 2 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 101d0 / HCA 2 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 201c0 / HCA 2 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 201d0 / HCA 2 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 901c0 / HCA 2 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 901d0 / HCA 2 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO GPU Direct RDMA Enabled for GPU a01c0 / HCA 2 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO GPU Direct RDMA Enabled for GPU a01d0 / HCA 2 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 101c0 / HCA 3 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 101d0 / HCA 3 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 201c0 / HCA 3 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 201d0 / HCA 3 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 901c0 / HCA 3 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 901d0 / HCA 3 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO GPU Direct RDMA Enabled for GPU a01c0 / HCA 3 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO GPU Direct RDMA Enabled for GPU a01d0 / HCA 3 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 101c0 / HCA 0 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 101d0 / HCA 0 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 201c0 / HCA 0 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 201d0 / HCA 0 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 901c0 / HCA 0 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 901d0 / HCA 0 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO GPU Direct RDMA Enabled for GPU a01c0 / HCA 0 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO GPU Direct RDMA Enabled for GPU a01d0 / HCA 0 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 101c0 / HCA 1 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 101d0 / HCA 1 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 201c0 / HCA 1 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 201d0 / HCA 1 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 901c0 / HCA 1 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 901d0 / HCA 1 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO GPU Direct RDMA Enabled for GPU a01c0 / HCA 1 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO GPU Direct RDMA Enabled for GPU a01d0 / HCA 1 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 101c0 / HCA 2 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 101d0 / HCA 2 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 201c0 / HCA 2 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 201d0 / HCA 2 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 901c0 / HCA 2 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 901d0 / HCA 2 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO GPU Direct RDMA Enabled for GPU a01c0 / HCA 2 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO GPU Direct RDMA Enabled for GPU a01d0 / HCA 2 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 101c0 / HCA 3 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 101d0 / HCA 3 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 201c0 / HCA 3 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 201d0 / HCA 3 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 901c0 / HCA 3 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 901d0 / HCA 3 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO GPU Direct RDMA Enabled for GPU a01c0 / HCA 3 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO GPU Direct RDMA Enabled for GPU a01d0 / HCA 3 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO Setting affinity for GPU 0 to 1f0000,0000001f | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO Channel 00/08 : 0 7 6 5 4 3 2 1 8 15 14 13 12 11 10 9 | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO Channel 01/08 : 0 3 10 15 14 13 12 9 8 11 2 7 6 5 4 1 | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO Channel 02/08 : 0 7 6 5 12 11 10 9 8 15 14 13 4 3 2 1 | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO Channel 03/08 : 0 5 4 7 14 11 10 9 8 13 12 15 6 3 2 1 | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO Channel 04/08 : 0 7 6 5 4 3 2 1 8 15 14 13 12 11 10 9 | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO Channel 05/08 : 0 3 10 15 14 13 12 9 8 11 2 7 6 5 4 1 | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO Channel 06/08 : 0 7 6 5 12 11 10 9 8 15 14 13 4 3 2 1 | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO Channel 07/08 : 0 5 4 7 14 11 10 9 8 13 12 15 6 3 2 1 | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO Trees [0] 1/8/-1->0->-1 [1] 1/-1/-1->0->7 [2] 1/-1/-1->0->7 [3] 1/-1/-1->0->7 [4] 1/-1/-1->0->8 [5] 1/-1/-1->0->7 [6] 1/-1/-1->0->7 [7] 1/-1/-1->0->7 | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO Channel 01/0 : 0[101c0] -> 3[201d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f3538002dd0 | |
| a100-st-p4d24xlarge-24:917:2158 [0] NCCL INFO New proxy send connection 0 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO Channel 05/0 : 0[101c0] -> 3[201d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f3538002e10 | |
| a100-st-p4d24xlarge-24:917:2158 [0] NCCL INFO New proxy send connection 1 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO Channel 03/0 : 0[101c0] -> 5[901d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-24:917:2158 [0] NCCL INFO New proxy send connection 2 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f3538002e50 | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO Channel 07/0 : 0[101c0] -> 5[901d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-24:917:2158 [0] NCCL INFO New proxy send connection 3 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f3538002e90 | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 101c0 / HCA 0 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:917:2158 [0] NCCL INFO New proxy recv connection 4 from local rank 0, transport 2 | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f3538002ed0 | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO Channel 00/0 : 9[101d0] -> 0[101c0] [receive] via NET/AWS Libfabric/0/GDRDMA | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 101c0 / HCA 0 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:917:2158 [0] NCCL INFO New proxy recv connection 5 from local rank 0, transport 2 | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f3538002f10 | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO Channel 04/0 : 9[101d0] -> 0[101c0] [receive] via NET/AWS Libfabric/0/GDRDMA | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO Channel 00/0 : 0[101c0] -> 7[a01d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-24:917:2158 [0] NCCL INFO New proxy send connection 6 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f3538002f50 | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO Channel 02/0 : 0[101c0] -> 7[a01d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-24:917:2158 [0] NCCL INFO New proxy send connection 7 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f3538002f90 | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO Channel 04/0 : 0[101c0] -> 7[a01d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-24:917:2158 [0] NCCL INFO New proxy send connection 8 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f3538002fd0 | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO Channel 06/0 : 0[101c0] -> 7[a01d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-24:917:2158 [0] NCCL INFO New proxy send connection 9 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f3538003010 | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f3538003050 | |
| a100-st-p4d24xlarge-24:917:2158 [0] NCCL INFO New proxy recv connection 10 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f3538003090 | |
| a100-st-p4d24xlarge-24:917:2158 [0] NCCL INFO New proxy recv connection 11 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f35380030d0 | |
| a100-st-p4d24xlarge-24:917:2158 [0] NCCL INFO New proxy recv connection 12 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f3538003110 | |
| a100-st-p4d24xlarge-24:917:2158 [0] NCCL INFO New proxy recv connection 13 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f3538003150 | |
| a100-st-p4d24xlarge-24:917:2158 [0] NCCL INFO New proxy recv connection 14 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f3538003190 | |
| a100-st-p4d24xlarge-24:917:2158 [0] NCCL INFO New proxy recv connection 15 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO Connected all rings | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO Channel 00/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f35380031d0 | |
| a100-st-p4d24xlarge-24:917:2158 [0] NCCL INFO New proxy send connection 16 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO Channel 01/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-24:917:2158 [0] NCCL INFO New proxy send connection 17 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f3538003210 | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO Channel 02/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-24:917:2158 [0] NCCL INFO New proxy send connection 18 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f3538003250 | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO Channel 03/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f3538003290 | |
| a100-st-p4d24xlarge-24:917:2158 [0] NCCL INFO New proxy send connection 19 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO Channel 04/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f35380032d0 | |
| a100-st-p4d24xlarge-24:917:2158 [0] NCCL INFO New proxy send connection 20 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO Channel 05/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f3538003310 | |
| a100-st-p4d24xlarge-24:917:2158 [0] NCCL INFO New proxy send connection 21 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO Channel 06/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f3538003350 | |
| a100-st-p4d24xlarge-24:917:2158 [0] NCCL INFO New proxy send connection 22 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO Channel 07/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f3538003390 | |
| a100-st-p4d24xlarge-24:917:2158 [0] NCCL INFO New proxy send connection 23 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO Channel 01/0 : 0[101c0] -> 7[a01d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f35380033d0 | |
| a100-st-p4d24xlarge-24:917:2158 [0] NCCL INFO New proxy send connection 24 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO Channel 03/0 : 0[101c0] -> 7[a01d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f3538003410 | |
| a100-st-p4d24xlarge-24:917:2158 [0] NCCL INFO New proxy send connection 25 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO Channel 05/0 : 0[101c0] -> 7[a01d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f3538003450 | |
| a100-st-p4d24xlarge-24:917:2158 [0] NCCL INFO New proxy send connection 26 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO Channel 07/0 : 0[101c0] -> 7[a01d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f3538003490 | |
| a100-st-p4d24xlarge-24:917:2158 [0] NCCL INFO New proxy send connection 27 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 101c0 / HCA 0 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:917:2158 [0] NCCL INFO New proxy recv connection 28 from local rank 0, transport 2 | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f35380034d0 | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO Channel 00/0 : 8[101c0] -> 0[101c0] [receive] via NET/AWS Libfabric/0/GDRDMA | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 101c0 / HCA 0 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:917:2158 [0] NCCL INFO New proxy recv connection 29 from local rank 0, transport 2 | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f3538003510 | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO Channel 04/0 : 8[101c0] -> 0[101c0] [receive] via NET/AWS Libfabric/0/GDRDMA | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 101c0 / HCA 0 (distance 3 <= 4), read 1 | |
| a100-st-p4d24xlarge-24:917:2158 [0] NCCL INFO New proxy send connection 30 from local rank 0, transport 2 | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f3538003550 | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO Channel 00/0 : 0[101c0] -> 8[101c0] [send] via NET/AWS Libfabric/0/GDRDMA | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 101c0 / HCA 0 (distance 3 <= 4), read 1 | |
| a100-st-p4d24xlarge-24:917:2158 [0] NCCL INFO New proxy send connection 31 from local rank 0, transport 2 | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f3538003590 | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO Channel 04/0 : 0[101c0] -> 8[101c0] [send] via NET/AWS Libfabric/0/GDRDMA | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f35380035d0 | |
| a100-st-p4d24xlarge-24:917:2158 [0] NCCL INFO New proxy recv connection 32 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f3538003610 | |
| a100-st-p4d24xlarge-24:917:2158 [0] NCCL INFO New proxy recv connection 33 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f3538003650 | |
| a100-st-p4d24xlarge-24:917:2158 [0] NCCL INFO New proxy recv connection 34 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f3538003690 | |
| a100-st-p4d24xlarge-24:917:2158 [0] NCCL INFO New proxy recv connection 35 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f35380036d0 | |
| a100-st-p4d24xlarge-24:917:2158 [0] NCCL INFO New proxy recv connection 36 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f3538003710 | |
| a100-st-p4d24xlarge-24:917:2158 [0] NCCL INFO New proxy recv connection 37 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-24:917:2158 [0] NCCL INFO New proxy recv connection 38 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f3538003750 | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f3538003790 | |
| a100-st-p4d24xlarge-24:917:2158 [0] NCCL INFO New proxy recv connection 39 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO Connected all trees | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO NCCL_ALGO set by environment to ring | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 512 | 512 | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO 8 coll channels, 8 p2p channels, 2 p2p channels per peer | |
| a100-st-p4d24xlarge-24:917:2158 [0] NCCL INFO New proxy send connection 40 from local rank 0, transport 2 | |
| a100-st-p4d24xlarge-24:917:2158 [0] NCCL INFO New proxy send connection 41 from local rank 1, transport 2 | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f35380037d0 | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 101c0 / HCA 1 (distance 3 <= 4), read 1 | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 101c0 / HCA 1 (distance 3 <= 4), read 1 | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 101c0 / HCA 2 (distance 3 <= 4), read 1 | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 101c0 / HCA 2 (distance 3 <= 4), read 1 | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 101c0 / HCA 3 (distance 3 <= 4), read 1 | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 101c0 / HCA 3 (distance 3 <= 4), read 1 | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 101c0 / HCA 1 (distance 3 <= 4), read 1 | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 101c0 / HCA 1 (distance 3 <= 4), read 1 | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 101c0 / HCA 2 (distance 3 <= 4), read 1 | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 101c0 / HCA 2 (distance 3 <= 4), read 1 | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 101c0 / HCA 3 (distance 3 <= 4), read 1 | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 101c0 / HCA 3 (distance 3 <= 4), read 1 | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO Connection to proxy localRank 2 -> connection 0x7f60e0003690 | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO Connection to proxy localRank 4 -> connection 0x7f4b70003690 | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO Connection to proxy localRank 6 -> connection 0x7f7c70003690 | |
| a100-st-p4d24xlarge-24:917:2158 [0] NCCL INFO New proxy send connection 42 from local rank 7, transport 2 | |
| a100-st-p4d24xlarge-24:917:2158 [0] NCCL INFO New proxy send connection 43 from local rank 2, transport 2 | |
| a100-st-p4d24xlarge-24:917:2158 [0] NCCL INFO New proxy send connection 44 from local rank 6, transport 2 | |
| didn't get the result, queue empty | |
| a100-st-p4d24xlarge-24:917:2158 [0] NCCL INFO New proxy send connection 45 from local rank 5, transport 2 | |
| a100-st-p4d24xlarge-24:917:2158 [0] NCCL INFO New proxy send connection 46 from local rank 3, transport 2 | |
| a100-st-p4d24xlarge-24:917:2158 [0] NCCL INFO New proxy send connection 47 from local rank 4, transport 2 | |
| a100-st-p4d24xlarge-24:917:2014 [0] NCCL INFO comm 0x563e7ef92c90 rank 0 nranks 16 cudaDev 0 busId 101c0 - Init COMPLETE | |
| still waiting for process to exit | |
| didn't get the result, queue empty | |
| still waiting for process to exit | |
| didn't get the result, queue empty | |
| still waiting for process to exit | |
| didn't get the result, queue empty | |
| still waiting for process to exit | |
| didn't get the result, queue empty | |
| still waiting for process to exit | |
| didn't get the result, queue empty | |
| still waiting for process to exit | |
| didn't get the result, queue empty | |
| still waiting for process to exit | |
| didn't get the result, queue empty | |
| still waiting for process to exit | |
| didn't get the result, queue empty | |
| still waiting for process to exit | |
| didn't get the result, queue empty | |
| still waiting for process to exit | |
| didn't get the result, queue empty | |
| still waiting for process to exit | |
| didn't get the result, queue empty | |
| still waiting for process to exit | |
| didn't get the result, queue empty | |
| still waiting for process to exit | |
| didn't get the result, queue empty | |
| still waiting for process to exit | |
| didn't get the result, queue empty | |
| still waiting for process to exit | |
| didn't get the result, queue empty | |
| still waiting for process to exit | |
| didn't get the result, queue empty | |
| still waiting for process to exit | |
| didn't get the result, queue empty | |
| still waiting for process to exit | |
| didn't get the result, queue empty | |
| still waiting for process to exit | |
| didn't get the result, queue empty | |
| still waiting for process to exit | |
| didn't get the result, queue empty | |
| still waiting for process to exit | |
| result {'latency_median': 93.18195343017578, 'latency_stdev': 0.26483998867502734} | |
| has been placed into the queue | |
| got the result! | |
| got exit 0 | |
| (already got results) | |
| exit code: 0 and result: {'nodes': 2, 'model_name': 'torchbenchmark.models.resnet50.Model', 'backend': 'torchdynamo_inductor', 'has_breaks': False, 'static_graph': False, 'result': {'latency_median': 93.18195343017578, 'latency_stdev': 0.26483998867502734}} | |
| <RESULT>{"nodes": 2, "model_name": "torchbenchmark.models.resnet50.Model", "backend": "torchdynamo_inductor", "has_breaks": false, "static_graph": false, "result": {"latency_median": 93.18195343017578, "latency_stdev": 0.26483998867502734}}</RESULT> | |
| run_once | |
| Process group: 16 tasks, rank: 0 | |
| MY HOSTNAME: a100-st-p4d24xlarge-24 | |
| FI_PROVIDER : efa | |
| LD_LIBRARY_PATH : /fsx/users/dberard/scratch-local/bench-fast/aws-ofi-nccl/lib:/opt/amazon/efa/lib:/usr/local/cuda-11.6/lib:/usr/local/cuda-11.6/lib64:/opt/amazon/efa/lib:/opt/amazon/openmpi/lib:/usr/local/cuda/efa/lib:/usr/local/cuda/lib:/usr/local/cuda:/usr/local/cuda/lib64:/usr/local/cuda/extras/CUPTI/lib64:/usr/local/cuda/targets/x86_64-linux/lib:/usr/local/lib:/usr/lib: | |
| NCCL_DEBUG : INFO | |
| FI_EFA_USE_DEVICE_RDMA : 1 | |
| still waiting for process to exit | |
| didn't get the result, queue empty | |
| still waiting for process to exit | |
| a100-st-p4d24xlarge-24:6123:6123 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to ens | |
| a100-st-p4d24xlarge-24:6123:6123 [0] NCCL INFO NCCL_SOCKET_IFNAME set to ens | |
| a100-st-p4d24xlarge-24:6123:6123 [0] NCCL INFO Bootstrap : Using ens32:10.200.74.168<0> | |
| a100-st-p4d24xlarge-24:6123:6123 [0] NCCL INFO NET/Plugin: Failed to find ncclCollNetPlugin_v6 symbol. | |
| a100-st-p4d24xlarge-24:6123:6123 [0] NCCL INFO NET/Plugin: Failed to find ncclCollNetPlugin symbol (v4 or v5). | |
| a100-st-p4d24xlarge-24:6123:6123 [0] NCCL INFO cudaDriverVersion 11060 | |
| NCCL version 2.14.3+cuda11.6 | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO NET/OFI Using aws-ofi-nccl 1.4.0aws | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO NET/OFI Running on p4d.24xlarge platform, Setting NCCL_TOPO_FILE environment variable to /fsx/users/dberard/scratch-local/bench-fast/aws-ofi-nccl/share/aws-ofi-nccl/xml/p4d-24xl-topo.xml | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO NET/OFI Setting FI_EFA_FORK_SAFE environment variable to 1 | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO NET/OFI Selected Provider is efa | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO Using network AWS Libfabric | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO NCCL_TOPO_FILE set by environment to /fsx/users/dberard/scratch-local/bench-fast/aws-ofi-nccl/share/aws-ofi-nccl/xml/p4d-24xl-topo.xml | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO NET/AWS Libfabric : GPU Direct RDMA Enabled for HCA 0 'rdmap16s27' | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO NET/AWS Libfabric : GPU Direct RDMA Enabled for HCA 1 'rdmap32s27' | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO NET/AWS Libfabric : GPU Direct RDMA Enabled for HCA 2 'rdmap144s27' | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO NET/AWS Libfabric : GPU Direct RDMA Enabled for HCA 3 'rdmap160s27' | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 101c0 / HCA 0 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 101d0 / HCA 0 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 201c0 / HCA 0 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 201d0 / HCA 0 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 901c0 / HCA 0 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 901d0 / HCA 0 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO GPU Direct RDMA Enabled for GPU a01c0 / HCA 0 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO GPU Direct RDMA Enabled for GPU a01d0 / HCA 0 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 101c0 / HCA 1 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 101d0 / HCA 1 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 201c0 / HCA 1 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 201d0 / HCA 1 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 901c0 / HCA 1 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 901d0 / HCA 1 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO GPU Direct RDMA Enabled for GPU a01c0 / HCA 1 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO GPU Direct RDMA Enabled for GPU a01d0 / HCA 1 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 101c0 / HCA 2 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 101d0 / HCA 2 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 201c0 / HCA 2 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 201d0 / HCA 2 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 901c0 / HCA 2 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 901d0 / HCA 2 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO GPU Direct RDMA Enabled for GPU a01c0 / HCA 2 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO GPU Direct RDMA Enabled for GPU a01d0 / HCA 2 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 101c0 / HCA 3 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 101d0 / HCA 3 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 201c0 / HCA 3 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 201d0 / HCA 3 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 901c0 / HCA 3 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 901d0 / HCA 3 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO GPU Direct RDMA Enabled for GPU a01c0 / HCA 3 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO GPU Direct RDMA Enabled for GPU a01d0 / HCA 3 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 101c0 / HCA 0 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 101d0 / HCA 0 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 201c0 / HCA 0 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 201d0 / HCA 0 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 901c0 / HCA 0 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 901d0 / HCA 0 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO GPU Direct RDMA Enabled for GPU a01c0 / HCA 0 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO GPU Direct RDMA Enabled for GPU a01d0 / HCA 0 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 101c0 / HCA 1 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 101d0 / HCA 1 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 201c0 / HCA 1 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 201d0 / HCA 1 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 901c0 / HCA 1 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 901d0 / HCA 1 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO GPU Direct RDMA Enabled for GPU a01c0 / HCA 1 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO GPU Direct RDMA Enabled for GPU a01d0 / HCA 1 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 101c0 / HCA 2 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 101d0 / HCA 2 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 201c0 / HCA 2 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 201d0 / HCA 2 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 901c0 / HCA 2 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 901d0 / HCA 2 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO GPU Direct RDMA Enabled for GPU a01c0 / HCA 2 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO GPU Direct RDMA Enabled for GPU a01d0 / HCA 2 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 101c0 / HCA 3 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 101d0 / HCA 3 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 201c0 / HCA 3 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 201d0 / HCA 3 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 901c0 / HCA 3 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 901d0 / HCA 3 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO GPU Direct RDMA Enabled for GPU a01c0 / HCA 3 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO GPU Direct RDMA Enabled for GPU a01d0 / HCA 3 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO Setting affinity for GPU 0 to 1f0000,0000001f | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO Channel 00/08 : 0 7 6 5 4 3 2 1 8 15 14 13 12 11 10 9 | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO Channel 01/08 : 0 3 10 15 14 13 12 9 8 11 2 7 6 5 4 1 | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO Channel 02/08 : 0 7 6 5 12 11 10 9 8 15 14 13 4 3 2 1 | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO Channel 03/08 : 0 5 4 7 14 11 10 9 8 13 12 15 6 3 2 1 | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO Channel 04/08 : 0 7 6 5 4 3 2 1 8 15 14 13 12 11 10 9 | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO Channel 05/08 : 0 3 10 15 14 13 12 9 8 11 2 7 6 5 4 1 | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO Channel 06/08 : 0 7 6 5 12 11 10 9 8 15 14 13 4 3 2 1 | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO Channel 07/08 : 0 5 4 7 14 11 10 9 8 13 12 15 6 3 2 1 | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO Trees [0] 1/8/-1->0->-1 [1] 1/-1/-1->0->7 [2] 1/-1/-1->0->7 [3] 1/-1/-1->0->7 [4] 1/-1/-1->0->8 [5] 1/-1/-1->0->7 [6] 1/-1/-1->0->7 [7] 1/-1/-1->0->7 | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO Channel 01/0 : 0[101c0] -> 3[201d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f3548002dd0 | |
| a100-st-p4d24xlarge-24:6123:7261 [0] NCCL INFO New proxy send connection 0 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO Channel 05/0 : 0[101c0] -> 3[201d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f3548002e10 | |
| a100-st-p4d24xlarge-24:6123:7261 [0] NCCL INFO New proxy send connection 1 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO Channel 03/0 : 0[101c0] -> 5[901d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-24:6123:7261 [0] NCCL INFO New proxy send connection 2 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f3548002e50 | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO Channel 07/0 : 0[101c0] -> 5[901d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-24:6123:7261 [0] NCCL INFO New proxy send connection 3 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f3548002e90 | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 101c0 / HCA 0 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:6123:7261 [0] NCCL INFO New proxy recv connection 4 from local rank 0, transport 2 | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f3548002ed0 | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO Channel 00/0 : 9[101d0] -> 0[101c0] [receive] via NET/AWS Libfabric/0/GDRDMA | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 101c0 / HCA 0 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:6123:7261 [0] NCCL INFO New proxy recv connection 5 from local rank 0, transport 2 | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f3548002f10 | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO Channel 04/0 : 9[101d0] -> 0[101c0] [receive] via NET/AWS Libfabric/0/GDRDMA | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO Channel 00/0 : 0[101c0] -> 7[a01d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-24:6123:7261 [0] NCCL INFO New proxy send connection 6 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f3548002f50 | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO Channel 02/0 : 0[101c0] -> 7[a01d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-24:6123:7261 [0] NCCL INFO New proxy send connection 7 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f3548002f90 | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO Channel 04/0 : 0[101c0] -> 7[a01d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-24:6123:7261 [0] NCCL INFO New proxy send connection 8 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f3548002fd0 | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO Channel 06/0 : 0[101c0] -> 7[a01d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-24:6123:7261 [0] NCCL INFO New proxy send connection 9 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f3548003010 | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f3548003050 | |
| a100-st-p4d24xlarge-24:6123:7261 [0] NCCL INFO New proxy recv connection 10 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f3548003090 | |
| a100-st-p4d24xlarge-24:6123:7261 [0] NCCL INFO New proxy recv connection 11 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f35480030d0 | |
| a100-st-p4d24xlarge-24:6123:7261 [0] NCCL INFO New proxy recv connection 12 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f3548003110 | |
| a100-st-p4d24xlarge-24:6123:7261 [0] NCCL INFO New proxy recv connection 13 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f3548003150 | |
| a100-st-p4d24xlarge-24:6123:7261 [0] NCCL INFO New proxy recv connection 14 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f3548003190 | |
| a100-st-p4d24xlarge-24:6123:7261 [0] NCCL INFO New proxy recv connection 15 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO Connected all rings | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO Channel 00/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f35480031d0 | |
| a100-st-p4d24xlarge-24:6123:7261 [0] NCCL INFO New proxy send connection 16 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO Channel 01/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-24:6123:7261 [0] NCCL INFO New proxy send connection 17 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f3548003210 | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO Channel 02/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-24:6123:7261 [0] NCCL INFO New proxy send connection 18 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f3548003250 | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO Channel 03/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-24:6123:7261 [0] NCCL INFO New proxy send connection 19 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f3548003290 | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO Channel 04/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-24:6123:7261 [0] NCCL INFO New proxy send connection 20 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f35480032d0 | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO Channel 05/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-24:6123:7261 [0] NCCL INFO New proxy send connection 21 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f3548003310 | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO Channel 06/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-24:6123:7261 [0] NCCL INFO New proxy send connection 22 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f3548003350 | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO Channel 07/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-24:6123:7261 [0] NCCL INFO New proxy send connection 23 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f3548003390 | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO Channel 01/0 : 0[101c0] -> 7[a01d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f35480033d0 | |
| a100-st-p4d24xlarge-24:6123:7261 [0] NCCL INFO New proxy send connection 24 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO Channel 03/0 : 0[101c0] -> 7[a01d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-24:6123:7261 [0] NCCL INFO New proxy send connection 25 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f3548003410 | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO Channel 05/0 : 0[101c0] -> 7[a01d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-24:6123:7261 [0] NCCL INFO New proxy send connection 26 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f3548003450 | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO Channel 07/0 : 0[101c0] -> 7[a01d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f3548003490 | |
| a100-st-p4d24xlarge-24:6123:7261 [0] NCCL INFO New proxy send connection 27 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 101c0 / HCA 0 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:6123:7261 [0] NCCL INFO New proxy recv connection 28 from local rank 0, transport 2 | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f35480034d0 | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO Channel 00/0 : 8[101c0] -> 0[101c0] [receive] via NET/AWS Libfabric/0/GDRDMA | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 101c0 / HCA 0 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-24:6123:7261 [0] NCCL INFO New proxy recv connection 29 from local rank 0, transport 2 | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f3548003510 | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO Channel 04/0 : 8[101c0] -> 0[101c0] [receive] via NET/AWS Libfabric/0/GDRDMA | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 101c0 / HCA 0 (distance 3 <= 4), read 1 | |
| a100-st-p4d24xlarge-24:6123:7261 [0] NCCL INFO New proxy send connection 30 from local rank 0, transport 2 | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f3548003550 | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO Channel 00/0 : 0[101c0] -> 8[101c0] [send] via NET/AWS Libfabric/0/GDRDMA | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 101c0 / HCA 0 (distance 3 <= 4), read 1 | |
| a100-st-p4d24xlarge-24:6123:7261 [0] NCCL INFO New proxy send connection 31 from local rank 0, transport 2 | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f3548003590 | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO Channel 04/0 : 0[101c0] -> 8[101c0] [send] via NET/AWS Libfabric/0/GDRDMA | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f35480035d0 | |
| a100-st-p4d24xlarge-24:6123:7261 [0] NCCL INFO New proxy recv connection 32 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f3548003610 | |
| a100-st-p4d24xlarge-24:6123:7261 [0] NCCL INFO New proxy recv connection 33 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f3548003650 | |
| a100-st-p4d24xlarge-24:6123:7261 [0] NCCL INFO New proxy recv connection 34 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f3548003690 | |
| a100-st-p4d24xlarge-24:6123:7261 [0] NCCL INFO New proxy recv connection 35 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f35480036d0 | |
| a100-st-p4d24xlarge-24:6123:7261 [0] NCCL INFO New proxy recv connection 36 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f3548003710 | |
| a100-st-p4d24xlarge-24:6123:7261 [0] NCCL INFO New proxy recv connection 37 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f3548003750 | |
| a100-st-p4d24xlarge-24:6123:7261 [0] NCCL INFO New proxy recv connection 38 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f3548003790 | |
| a100-st-p4d24xlarge-24:6123:7261 [0] NCCL INFO New proxy recv connection 39 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO Connected all trees | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO NCCL_ALGO set by environment to ring | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 512 | 512 | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO 8 coll channels, 8 p2p channels, 2 p2p channels per peer | |
| a100-st-p4d24xlarge-24:6123:7261 [0] NCCL INFO New proxy send connection 40 from local rank 0, transport 2 | |
| a100-st-p4d24xlarge-24:6123:7261 [0] NCCL INFO New proxy send connection 41 from local rank 1, transport 2 | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f35480037d0 | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 101c0 / HCA 1 (distance 3 <= 4), read 1 | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 101c0 / HCA 1 (distance 3 <= 4), read 1 | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 101c0 / HCA 2 (distance 3 <= 4), read 1 | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 101c0 / HCA 2 (distance 3 <= 4), read 1 | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 101c0 / HCA 3 (distance 3 <= 4), read 1 | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 101c0 / HCA 3 (distance 3 <= 4), read 1 | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 101c0 / HCA 1 (distance 3 <= 4), read 1 | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 101c0 / HCA 1 (distance 3 <= 4), read 1 | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 101c0 / HCA 2 (distance 3 <= 4), read 1 | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 101c0 / HCA 2 (distance 3 <= 4), read 1 | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 101c0 / HCA 3 (distance 3 <= 4), read 1 | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 101c0 / HCA 3 (distance 3 <= 4), read 1 | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO Connection to proxy localRank 2 -> connection 0x7f60d8003590 | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO Connection to proxy localRank 4 -> connection 0x7f4b70003110 | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO Connection to proxy localRank 6 -> connection 0x7f7c6c003590 | |
| a100-st-p4d24xlarge-24:6123:7261 [0] NCCL INFO New proxy send connection 42 from local rank 7, transport 2 | |
| a100-st-p4d24xlarge-24:6123:7261 [0] NCCL INFO New proxy send connection 43 from local rank 6, transport 2 | |
| didn't get the result, queue empty | |
| a100-st-p4d24xlarge-24:6123:7261 [0] NCCL INFO New proxy send connection 44 from local rank 2, transport 2 | |
| a100-st-p4d24xlarge-24:6123:7261 [0] NCCL INFO New proxy send connection 45 from local rank 5, transport 2 | |
| a100-st-p4d24xlarge-24:6123:7261 [0] NCCL INFO New proxy send connection 46 from local rank 3, transport 2 | |
| a100-st-p4d24xlarge-24:6123:7261 [0] NCCL INFO New proxy send connection 47 from local rank 4, transport 2 | |
| a100-st-p4d24xlarge-24:6123:7127 [0] NCCL INFO comm 0x563e7e6ef4b0 rank 0 nranks 16 cudaDev 0 busId 101c0 - Init COMPLETE | |
| DDPOptimizer used bucket cap 26214400 and produced the following buckets: | |
| Index Size (b) Param Names | |
| ------- ---------- --------------------------------- | |
| 0 30257056 self_fc_weight | |
| self_fc_bias | |
| self_layer4_2_bn3_weight | |
| self_layer4_2_bn3_bias | |
| self_layer4_2_conv3_weight | |
| self_layer4_2_bn2_weight | |
| self_layer4_2_bn2_bias | |
| self_layer4_2_conv2_weight | |
| self_layer4_2_bn1_weight | |
| self_layer4_2_bn1_bias | |
| self_layer4_2_conv1_weight | |
| self_layer4_1_bn3_weight | |
| self_layer4_1_bn3_bias | |
| self_layer4_1_conv3_weight | |
| 1 26255360 self_layer4_1_bn2_weight | |
| self_layer4_1_bn2_bias | |
| self_layer4_1_conv2_weight | |
| self_layer4_1_bn1_weight | |
| self_layer4_1_bn1_bias | |
| self_layer4_1_conv1_weight | |
| self_layer4_0_downsample_1_weight | |
| self_layer4_0_downsample_1_bias | |
| self_layer4_0_downsample_0_weight | |
| self_layer4_0_bn3_weight | |
| self_layer4_0_bn3_bias | |
| self_layer4_0_conv3_weight | |
| 2 28366848 self_layer4_0_bn2_weight | |
| self_layer4_0_bn2_bias | |
| self_layer4_0_conv2_weight | |
| self_layer4_0_bn1_weight | |
| self_layer4_0_bn1_bias | |
| self_layer4_0_conv1_weight | |
| self_layer3_5_bn3_weight | |
| self_layer3_5_bn3_bias | |
| self_layer3_5_conv3_weight | |
| self_layer3_5_bn2_weight | |
| self_layer3_5_bn2_bias | |
| self_layer3_5_conv2_weight | |
| self_layer3_5_bn1_weight | |
| self_layer3_5_bn1_bias | |
| self_layer3_5_conv1_weight | |
| self_layer3_4_bn3_weight | |
| self_layer3_4_bn3_bias | |
| self_layer3_4_conv3_weight | |
| self_layer3_4_bn2_weight | |
| self_layer3_4_bn2_bias | |
| self_layer3_4_conv2_weight | |
| self_layer3_4_bn1_weight | |
| self_layer3_4_bn1_bias | |
| self_layer3_4_conv1_weight | |
| self_layer3_3_bn3_weight | |
| self_layer3_3_bn3_bias | |
| self_layer3_3_conv3_weight | |
| self_layer3_3_bn2_weight | |
| self_layer3_3_bn2_bias | |
| self_layer3_3_conv2_weight | |
| self_layer3_3_bn1_weight | |
| self_layer3_3_bn1_bias | |
| self_layer3_3_conv1_weight | |
| self_layer3_2_bn3_weight | |
| self_layer3_2_bn3_bias | |
| self_layer3_2_conv3_weight | |
| self_layer3_2_bn2_weight | |
| self_layer3_2_bn2_bias | |
| self_layer3_2_conv2_weight | |
| 3 17348864 self_layer3_2_bn1_weight | |
| self_layer3_2_bn1_bias | |
| self_layer3_2_conv1_weight | |
| self_layer3_1_bn3_weight | |
| self_layer3_1_bn3_bias | |
| self_layer3_1_conv3_weight | |
| self_layer3_1_bn2_weight | |
| self_layer3_1_bn2_bias | |
| self_layer3_1_conv2_weight | |
| self_layer3_1_bn1_weight | |
| self_layer3_1_bn1_bias | |
| self_layer3_1_conv1_weight | |
| self_layer3_0_downsample_1_weight | |
| self_layer3_0_downsample_1_bias | |
| self_layer3_0_downsample_0_weight | |
| self_layer3_0_bn3_weight | |
| self_layer3_0_bn3_bias | |
| self_layer3_0_conv3_weight | |
| self_layer3_0_bn2_weight | |
| self_layer3_0_bn2_bias | |
| self_layer3_0_conv2_weight | |
| self_layer3_0_bn1_weight | |
| self_layer3_0_bn1_bias | |
| self_layer3_0_conv1_weight | |
| self_layer2_3_bn3_weight | |
| self_layer2_3_bn3_bias | |
| self_layer2_3_conv3_weight | |
| self_layer2_3_bn2_weight | |
| self_layer2_3_bn2_bias | |
| self_layer2_3_conv2_weight | |
| self_layer2_3_bn1_weight | |
| self_layer2_3_bn1_bias | |
| self_layer2_3_conv1_weight | |
| self_layer2_2_bn3_weight | |
| self_layer2_2_bn3_bias | |
| self_layer2_2_conv3_weight | |
| self_layer2_2_bn2_weight | |
| self_layer2_2_bn2_bias | |
| self_layer2_2_conv2_weight | |
| self_layer2_2_bn1_weight | |
| self_layer2_2_bn1_bias | |
| self_layer2_2_conv1_weight | |
| self_layer2_1_bn3_weight | |
| self_layer2_1_bn3_bias | |
| self_layer2_1_conv3_weight | |
| self_layer2_1_bn2_weight | |
| self_layer2_1_bn2_bias | |
| self_layer2_1_conv2_weight | |
| self_layer2_1_bn1_weight | |
| self_layer2_1_bn1_bias | |
| self_layer2_1_conv1_weight | |
| self_layer2_0_downsample_1_weight | |
| self_layer2_0_downsample_1_bias | |
| self_layer2_0_downsample_0_weight | |
| self_layer2_0_bn3_weight | |
| self_layer2_0_bn3_bias | |
| self_layer2_0_conv3_weight | |
| self_layer2_0_bn2_weight | |
| self_layer2_0_bn2_bias | |
| self_layer2_0_conv2_weight | |
| self_layer2_0_bn1_weight | |
| self_layer2_0_bn1_bias | |
| self_layer2_0_conv1_weight | |
| self_layer1_2_bn3_weight | |
| self_layer1_2_bn3_bias | |
| self_layer1_2_conv3_weight | |
| self_layer1_2_bn2_weight | |
| self_layer1_2_bn2_bias | |
| self_layer1_2_conv2_weight | |
| self_layer1_2_bn1_weight | |
| self_layer1_2_bn1_bias | |
| self_layer1_2_conv1_weight | |
| self_layer1_1_bn3_weight | |
| self_layer1_1_bn3_bias | |
| self_layer1_1_conv3_weight | |
| self_layer1_1_bn2_weight | |
| self_layer1_1_bn2_bias | |
| self_layer1_1_conv2_weight | |
| self_layer1_1_bn1_weight | |
| self_layer1_1_bn1_bias | |
| self_layer1_1_conv1_weight | |
| self_layer1_0_downsample_1_weight | |
| self_layer1_0_downsample_1_bias | |
| self_layer1_0_downsample_0_weight | |
| self_layer1_0_bn3_weight | |
| self_layer1_0_bn3_bias | |
| self_layer1_0_conv3_weight | |
| self_layer1_0_bn2_weight | |
| self_layer1_0_bn2_bias | |
| self_layer1_0_conv2_weight | |
| self_layer1_0_bn1_weight | |
| self_layer1_0_bn1_bias | |
| self_layer1_0_conv1_weight | |
| self_bn1_weight | |
| self_bn1_bias | |
| self_conv1_weight | |
| ---orig graph--- | |
| graph(): | |
| %x : torch.Tensor [#users=1] = placeholder[target=x] | |
| %self_conv1 : [#users=1] = call_module[target=self_conv1](args = (%x,), kwargs = {}) | |
| %self_bn1 : [#users=1] = call_module[target=self_bn1](args = (%self_conv1,), kwargs = {}) | |
| %self_relu : [#users=1] = call_module[target=self_relu](args = (%self_bn1,), kwargs = {}) | |
| %self_maxpool : [#users=2] = call_module[target=self_maxpool](args = (%self_relu,), kwargs = {}) | |
| %self_layer1_0_conv1 : [#users=1] = call_module[target=self_layer1_0_conv1](args = (%self_maxpool,), kwargs = {}) | |
| %self_layer1_0_bn1 : [#users=1] = call_module[target=self_layer1_0_bn1](args = (%self_layer1_0_conv1,), kwargs = {}) | |
| %self_layer1_0_relu : [#users=1] = call_module[target=self_layer1_0_relu](args = (%self_layer1_0_bn1,), kwargs = {}) | |
| %self_layer1_0_conv2 : [#users=1] = call_module[target=self_layer1_0_conv2](args = (%self_layer1_0_relu,), kwargs = {}) | |
| %self_layer1_0_bn2 : [#users=1] = call_module[target=self_layer1_0_bn2](args = (%self_layer1_0_conv2,), kwargs = {}) | |
| %self_layer1_0_relu_1 : [#users=1] = call_module[target=self_layer1_0_relu](args = (%self_layer1_0_bn2,), kwargs = {}) | |
| %self_layer1_0_conv3 : [#users=1] = call_module[target=self_layer1_0_conv3](args = (%self_layer1_0_relu_1,), kwargs = {}) | |
| %self_layer1_0_bn3 : [#users=1] = call_module[target=self_layer1_0_bn3](args = (%self_layer1_0_conv3,), kwargs = {}) | |
| %self_layer1_0_downsample_0 : [#users=1] = call_module[target=self_layer1_0_downsample_0](args = (%self_maxpool,), kwargs = {}) | |
| %self_layer1_0_downsample_1 : [#users=1] = call_module[target=self_layer1_0_downsample_1](args = (%self_layer1_0_downsample_0,), kwargs = {}) | |
| %iadd : [#users=1] = call_function[target=operator.iadd](args = (%self_layer1_0_bn3, %self_layer1_0_downsample_1), kwargs = {}) | |
| %self_layer1_0_relu_2 : [#users=2] = call_module[target=self_layer1_0_relu](args = (%iadd,), kwargs = {}) | |
| %self_layer1_1_conv1 : [#users=1] = call_module[target=self_layer1_1_conv1](args = (%self_layer1_0_relu_2,), kwargs = {}) | |
| %self_layer1_1_bn1 : [#users=1] = call_module[target=self_layer1_1_bn1](args = (%self_layer1_1_conv1,), kwargs = {}) | |
| %self_layer1_1_relu : [#users=1] = call_module[target=self_layer1_1_relu](args = (%self_layer1_1_bn1,), kwargs = {}) | |
| %self_layer1_1_conv2 : [#users=1] = call_module[target=self_layer1_1_conv2](args = (%self_layer1_1_relu,), kwargs = {}) | |
| %self_layer1_1_bn2 : [#users=1] = call_module[target=self_layer1_1_bn2](args = (%self_layer1_1_conv2,), kwargs = {}) | |
| %self_layer1_1_relu_1 : [#users=1] = call_module[target=self_layer1_1_relu](args = (%self_layer1_1_bn2,), kwargs = {}) | |
| %self_layer1_1_conv3 : [#users=1] = call_module[target=self_layer1_1_conv3](args = (%self_layer1_1_relu_1,), kwargs = {}) | |
| %self_layer1_1_bn3 : [#users=1] = call_module[target=self_layer1_1_bn3](args = (%self_layer1_1_conv3,), kwargs = {}) | |
| %iadd_1 : [#users=1] = call_function[target=operator.iadd](args = (%self_layer1_1_bn3, %self_layer1_0_relu_2), kwargs = {}) | |
| %self_layer1_1_relu_2 : [#users=2] = call_module[target=self_layer1_1_relu](args = (%iadd_1,), kwargs = {}) | |
| %self_layer1_2_conv1 : [#users=1] = call_module[target=self_layer1_2_conv1](args = (%self_layer1_1_relu_2,), kwargs = {}) | |
| %self_layer1_2_bn1 : [#users=1] = call_module[target=self_layer1_2_bn1](args = (%self_layer1_2_conv1,), kwargs = {}) | |
| %self_layer1_2_relu : [#users=1] = call_module[target=self_layer1_2_relu](args = (%self_layer1_2_bn1,), kwargs = {}) | |
| %self_layer1_2_conv2 : [#users=1] = call_module[target=self_layer1_2_conv2](args = (%self_layer1_2_relu,), kwargs = {}) | |
| %self_layer1_2_bn2 : [#users=1] = call_module[target=self_layer1_2_bn2](args = (%self_layer1_2_conv2,), kwargs = {}) | |
| %self_layer1_2_relu_1 : [#users=1] = call_module[target=self_layer1_2_relu](args = (%self_layer1_2_bn2,), kwargs = {}) | |
| %self_layer1_2_conv3 : [#users=1] = call_module[target=self_layer1_2_conv3](args = (%self_layer1_2_relu_1,), kwargs = {}) | |
| %self_layer1_2_bn3 : [#users=1] = call_module[target=self_layer1_2_bn3](args = (%self_layer1_2_conv3,), kwargs = {}) | |
| %iadd_2 : [#users=1] = call_function[target=operator.iadd](args = (%self_layer1_2_bn3, %self_layer1_1_relu_2), kwargs = {}) | |
| %self_layer1_2_relu_2 : [#users=2] = call_module[target=self_layer1_2_relu](args = (%iadd_2,), kwargs = {}) | |
| %self_layer2_0_conv1 : [#users=1] = call_module[target=self_layer2_0_conv1](args = (%self_layer1_2_relu_2,), kwargs = {}) | |
| %self_layer2_0_bn1 : [#users=1] = call_module[target=self_layer2_0_bn1](args = (%self_layer2_0_conv1,), kwargs = {}) | |
| %self_layer2_0_relu : [#users=1] = call_module[target=self_layer2_0_relu](args = (%self_layer2_0_bn1,), kwargs = {}) | |
| %self_layer2_0_conv2 : [#users=1] = call_module[target=self_layer2_0_conv2](args = (%self_layer2_0_relu,), kwargs = {}) | |
| %self_layer2_0_bn2 : [#users=1] = call_module[target=self_layer2_0_bn2](args = (%self_layer2_0_conv2,), kwargs = {}) | |
| %self_layer2_0_relu_1 : [#users=1] = call_module[target=self_layer2_0_relu](args = (%self_layer2_0_bn2,), kwargs = {}) | |
| %self_layer2_0_conv3 : [#users=1] = call_module[target=self_layer2_0_conv3](args = (%self_layer2_0_relu_1,), kwargs = {}) | |
| %self_layer2_0_bn3 : [#users=1] = call_module[target=self_layer2_0_bn3](args = (%self_layer2_0_conv3,), kwargs = {}) | |
| %self_layer2_0_downsample_0 : [#users=1] = call_module[target=self_layer2_0_downsample_0](args = (%self_layer1_2_relu_2,), kwargs = {}) | |
| %self_layer2_0_downsample_1 : [#users=1] = call_module[target=self_layer2_0_downsample_1](args = (%self_layer2_0_downsample_0,), kwargs = {}) | |
| %iadd_3 : [#users=1] = call_function[target=operator.iadd](args = (%self_layer2_0_bn3, %self_layer2_0_downsample_1), kwargs = {}) | |
| %self_layer2_0_relu_2 : [#users=2] = call_module[target=self_layer2_0_relu](args = (%iadd_3,), kwargs = {}) | |
| %self_layer2_1_conv1 : [#users=1] = call_module[target=self_layer2_1_conv1](args = (%self_layer2_0_relu_2,), kwargs = {}) | |
| %self_layer2_1_bn1 : [#users=1] = call_module[target=self_layer2_1_bn1](args = (%self_layer2_1_conv1,), kwargs = {}) | |
| %self_layer2_1_relu : [#users=1] = call_module[target=self_layer2_1_relu](args = (%self_layer2_1_bn1,), kwargs = {}) | |
| %self_layer2_1_conv2 : [#users=1] = call_module[target=self_layer2_1_conv2](args = (%self_layer2_1_relu,), kwargs = {}) | |
| %self_layer2_1_bn2 : [#users=1] = call_module[target=self_layer2_1_bn2](args = (%self_layer2_1_conv2,), kwargs = {}) | |
| %self_layer2_1_relu_1 : [#users=1] = call_module[target=self_layer2_1_relu](args = (%self_layer2_1_bn2,), kwargs = {}) | |
| %self_layer2_1_conv3 : [#users=1] = call_module[target=self_layer2_1_conv3](args = (%self_layer2_1_relu_1,), kwargs = {}) | |
| %self_layer2_1_bn3 : [#users=1] = call_module[target=self_layer2_1_bn3](args = (%self_layer2_1_conv3,), kwargs = {}) | |
| %iadd_4 : [#users=1] = call_function[target=operator.iadd](args = (%self_layer2_1_bn3, %self_layer2_0_relu_2), kwargs = {}) | |
| %self_layer2_1_relu_2 : [#users=2] = call_module[target=self_layer2_1_relu](args = (%iadd_4,), kwargs = {}) | |
| %self_layer2_2_conv1 : [#users=1] = call_module[target=self_layer2_2_conv1](args = (%self_layer2_1_relu_2,), kwargs = {}) | |
| %self_layer2_2_bn1 : [#users=1] = call_module[target=self_layer2_2_bn1](args = (%self_layer2_2_conv1,), kwargs = {}) | |
| %self_layer2_2_relu : [#users=1] = call_module[target=self_layer2_2_relu](args = (%self_layer2_2_bn1,), kwargs = {}) | |
| %self_layer2_2_conv2 : [#users=1] = call_module[target=self_layer2_2_conv2](args = (%self_layer2_2_relu,), kwargs = {}) | |
| %self_layer2_2_bn2 : [#users=1] = call_module[target=self_layer2_2_bn2](args = (%self_layer2_2_conv2,), kwargs = {}) | |
| %self_layer2_2_relu_1 : [#users=1] = call_module[target=self_layer2_2_relu](args = (%self_layer2_2_bn2,), kwargs = {}) | |
| %self_layer2_2_conv3 : [#users=1] = call_module[target=self_layer2_2_conv3](args = (%self_layer2_2_relu_1,), kwargs = {}) | |
| %self_layer2_2_bn3 : [#users=1] = call_module[target=self_layer2_2_bn3](args = (%self_layer2_2_conv3,), kwargs = {}) | |
| %iadd_5 : [#users=1] = call_function[target=operator.iadd](args = (%self_layer2_2_bn3, %self_layer2_1_relu_2), kwargs = {}) | |
| %self_layer2_2_relu_2 : [#users=2] = call_module[target=self_layer2_2_relu](args = (%iadd_5,), kwargs = {}) | |
| %self_layer2_3_conv1 : [#users=1] = call_module[target=self_layer2_3_conv1](args = (%self_layer2_2_relu_2,), kwargs = {}) | |
| %self_layer2_3_bn1 : [#users=1] = call_module[target=self_layer2_3_bn1](args = (%self_layer2_3_conv1,), kwargs = {}) | |
| %self_layer2_3_relu : [#users=1] = call_module[target=self_layer2_3_relu](args = (%self_layer2_3_bn1,), kwargs = {}) | |
| %self_layer2_3_conv2 : [#users=1] = call_module[target=self_layer2_3_conv2](args = (%self_layer2_3_relu,), kwargs = {}) | |
| %self_layer2_3_bn2 : [#users=1] = call_module[target=self_layer2_3_bn2](args = (%self_layer2_3_conv2,), kwargs = {}) | |
| %self_layer2_3_relu_1 : [#users=1] = call_module[target=self_layer2_3_relu](args = (%self_layer2_3_bn2,), kwargs = {}) | |
| %self_layer2_3_conv3 : [#users=1] = call_module[target=self_layer2_3_conv3](args = (%self_layer2_3_relu_1,), kwargs = {}) | |
| %self_layer2_3_bn3 : [#users=1] = call_module[target=self_layer2_3_bn3](args = (%self_layer2_3_conv3,), kwargs = {}) | |
| %iadd_6 : [#users=1] = call_function[target=operator.iadd](args = (%self_layer2_3_bn3, %self_layer2_2_relu_2), kwargs = {}) | |
| %self_layer2_3_relu_2 : [#users=2] = call_module[target=self_layer2_3_relu](args = (%iadd_6,), kwargs = {}) | |
| %self_layer3_0_conv1 : [#users=1] = call_module[target=self_layer3_0_conv1](args = (%self_layer2_3_relu_2,), kwargs = {}) | |
| %self_layer3_0_bn1 : [#users=1] = call_module[target=self_layer3_0_bn1](args = (%self_layer3_0_conv1,), kwargs = {}) | |
| %self_layer3_0_relu : [#users=1] = call_module[target=self_layer3_0_relu](args = (%self_layer3_0_bn1,), kwargs = {}) | |
| %self_layer3_0_conv2 : [#users=1] = call_module[target=self_layer3_0_conv2](args = (%self_layer3_0_relu,), kwargs = {}) | |
| %self_layer3_0_bn2 : [#users=1] = call_module[target=self_layer3_0_bn2](args = (%self_layer3_0_conv2,), kwargs = {}) | |
| %self_layer3_0_relu_1 : [#users=1] = call_module[target=self_layer3_0_relu](args = (%self_layer3_0_bn2,), kwargs = {}) | |
| %self_layer3_0_conv3 : [#users=1] = call_module[target=self_layer3_0_conv3](args = (%self_layer3_0_relu_1,), kwargs = {}) | |
| %self_layer3_0_bn3 : [#users=1] = call_module[target=self_layer3_0_bn3](args = (%self_layer3_0_conv3,), kwargs = {}) | |
| %self_layer3_0_downsample_0 : [#users=1] = call_module[target=self_layer3_0_downsample_0](args = (%self_layer2_3_relu_2,), kwargs = {}) | |
| %self_layer3_0_downsample_1 : [#users=1] = call_module[target=self_layer3_0_downsample_1](args = (%self_layer3_0_downsample_0,), kwargs = {}) | |
| %iadd_7 : [#users=1] = call_function[target=operator.iadd](args = (%self_layer3_0_bn3, %self_layer3_0_downsample_1), kwargs = {}) | |
| %self_layer3_0_relu_2 : [#users=2] = call_module[target=self_layer3_0_relu](args = (%iadd_7,), kwargs = {}) | |
| %self_layer3_1_conv1 : [#users=1] = call_module[target=self_layer3_1_conv1](args = (%self_layer3_0_relu_2,), kwargs = {}) | |
| %self_layer3_1_bn1 : [#users=1] = call_module[target=self_layer3_1_bn1](args = (%self_layer3_1_conv1,), kwargs = {}) | |
| %self_layer3_1_relu : [#users=1] = call_module[target=self_layer3_1_relu](args = (%self_layer3_1_bn1,), kwargs = {}) | |
| %self_layer3_1_conv2 : [#users=1] = call_module[target=self_layer3_1_conv2](args = (%self_layer3_1_relu,), kwargs = {}) | |
| %self_layer3_1_bn2 : [#users=1] = call_module[target=self_layer3_1_bn2](args = (%self_layer3_1_conv2,), kwargs = {}) | |
| %self_layer3_1_relu_1 : [#users=1] = call_module[target=self_layer3_1_relu](args = (%self_layer3_1_bn2,), kwargs = {}) | |
| %self_layer3_1_conv3 : [#users=1] = call_module[target=self_layer3_1_conv3](args = (%self_layer3_1_relu_1,), kwargs = {}) | |
| %self_layer3_1_bn3 : [#users=1] = call_module[target=self_layer3_1_bn3](args = (%self_layer3_1_conv3,), kwargs = {}) | |
| %iadd_8 : [#users=1] = call_function[target=operator.iadd](args = (%self_layer3_1_bn3, %self_layer3_0_relu_2), kwargs = {}) | |
| %self_layer3_1_relu_2 : [#users=2] = call_module[target=self_layer3_1_relu](args = (%iadd_8,), kwargs = {}) | |
| %self_layer3_2_conv1 : [#users=1] = call_module[target=self_layer3_2_conv1](args = (%self_layer3_1_relu_2,), kwargs = {}) | |
| %self_layer3_2_bn1 : [#users=1] = call_module[target=self_layer3_2_bn1](args = (%self_layer3_2_conv1,), kwargs = {}) | |
| %self_layer3_2_relu : [#users=1] = call_module[target=self_layer3_2_relu](args = (%self_layer3_2_bn1,), kwargs = {}) | |
| %self_layer3_2_conv2 : [#users=1] = call_module[target=self_layer3_2_conv2](args = (%self_layer3_2_relu,), kwargs = {}) | |
| %self_layer3_2_bn2 : [#users=1] = call_module[target=self_layer3_2_bn2](args = (%self_layer3_2_conv2,), kwargs = {}) | |
| %self_layer3_2_relu_1 : [#users=1] = call_module[target=self_layer3_2_relu](args = (%self_layer3_2_bn2,), kwargs = {}) | |
| %self_layer3_2_conv3 : [#users=1] = call_module[target=self_layer3_2_conv3](args = (%self_layer3_2_relu_1,), kwargs = {}) | |
| %self_layer3_2_bn3 : [#users=1] = call_module[target=self_layer3_2_bn3](args = (%self_layer3_2_conv3,), kwargs = {}) | |
| %iadd_9 : [#users=1] = call_function[target=operator.iadd](args = (%self_layer3_2_bn3, %self_layer3_1_relu_2), kwargs = {}) | |
| %self_layer3_2_relu_2 : [#users=2] = call_module[target=self_layer3_2_relu](args = (%iadd_9,), kwargs = {}) | |
| %self_layer3_3_conv1 : [#users=1] = call_module[target=self_layer3_3_conv1](args = (%self_layer3_2_relu_2,), kwargs = {}) | |
| %self_layer3_3_bn1 : [#users=1] = call_module[target=self_layer3_3_bn1](args = (%self_layer3_3_conv1,), kwargs = {}) | |
| %self_layer3_3_relu : [#users=1] = call_module[target=self_layer3_3_relu](args = (%self_layer3_3_bn1,), kwargs = {}) | |
| %self_layer3_3_conv2 : [#users=1] = call_module[target=self_layer3_3_conv2](args = (%self_layer3_3_relu,), kwargs = {}) | |
| %self_layer3_3_bn2 : [#users=1] = call_module[target=self_layer3_3_bn2](args = (%self_layer3_3_conv2,), kwargs = {}) | |
| %self_layer3_3_relu_1 : [#users=1] = call_module[target=self_layer3_3_relu](args = (%self_layer3_3_bn2,), kwargs = {}) | |
| %self_layer3_3_conv3 : [#users=1] = call_module[target=self_layer3_3_conv3](args = (%self_layer3_3_relu_1,), kwargs = {}) | |
| %self_layer3_3_bn3 : [#users=1] = call_module[target=self_layer3_3_bn3](args = (%self_layer3_3_conv3,), kwargs = {}) | |
| %iadd_10 : [#users=1] = call_function[target=operator.iadd](args = (%self_layer3_3_bn3, %self_layer3_2_relu_2), kwargs = {}) | |
| %self_layer3_3_relu_2 : [#users=2] = call_module[target=self_layer3_3_relu](args = (%iadd_10,), kwargs = {}) | |
| %self_layer3_4_conv1 : [#users=1] = call_module[target=self_layer3_4_conv1](args = (%self_layer3_3_relu_2,), kwargs = {}) | |
| %self_layer3_4_bn1 : [#users=1] = call_module[target=self_layer3_4_bn1](args = (%self_layer3_4_conv1,), kwargs = {}) | |
| %self_layer3_4_relu : [#users=1] = call_module[target=self_layer3_4_relu](args = (%self_layer3_4_bn1,), kwargs = {}) | |
| %self_layer3_4_conv2 : [#users=1] = call_module[target=self_layer3_4_conv2](args = (%self_layer3_4_relu,), kwargs = {}) | |
| %self_layer3_4_bn2 : [#users=1] = call_module[target=self_layer3_4_bn2](args = (%self_layer3_4_conv2,), kwargs = {}) | |
| %self_layer3_4_relu_1 : [#users=1] = call_module[target=self_layer3_4_relu](args = (%self_layer3_4_bn2,), kwargs = {}) | |
| %self_layer3_4_conv3 : [#users=1] = call_module[target=self_layer3_4_conv3](args = (%self_layer3_4_relu_1,), kwargs = {}) | |
| %self_layer3_4_bn3 : [#users=1] = call_module[target=self_layer3_4_bn3](args = (%self_layer3_4_conv3,), kwargs = {}) | |
| %iadd_11 : [#users=1] = call_function[target=operator.iadd](args = (%self_layer3_4_bn3, %self_layer3_3_relu_2), kwargs = {}) | |
| %self_layer3_4_relu_2 : [#users=2] = call_module[target=self_layer3_4_relu](args = (%iadd_11,), kwargs = {}) | |
| %self_layer3_5_conv1 : [#users=1] = call_module[target=self_layer3_5_conv1](args = (%self_layer3_4_relu_2,), kwargs = {}) | |
| %self_layer3_5_bn1 : [#users=1] = call_module[target=self_layer3_5_bn1](args = (%self_layer3_5_conv1,), kwargs = {}) | |
| %self_layer3_5_relu : [#users=1] = call_module[target=self_layer3_5_relu](args = (%self_layer3_5_bn1,), kwargs = {}) | |
| %self_layer3_5_conv2 : [#users=1] = call_module[target=self_layer3_5_conv2](args = (%self_layer3_5_relu,), kwargs = {}) | |
| %self_layer3_5_bn2 : [#users=1] = call_module[target=self_layer3_5_bn2](args = (%self_layer3_5_conv2,), kwargs = {}) | |
| %self_layer3_5_relu_1 : [#users=1] = call_module[target=self_layer3_5_relu](args = (%self_layer3_5_bn2,), kwargs = {}) | |
| %self_layer3_5_conv3 : [#users=1] = call_module[target=self_layer3_5_conv3](args = (%self_layer3_5_relu_1,), kwargs = {}) | |
| %self_layer3_5_bn3 : [#users=1] = call_module[target=self_layer3_5_bn3](args = (%self_layer3_5_conv3,), kwargs = {}) | |
| %iadd_12 : [#users=1] = call_function[target=operator.iadd](args = (%self_layer3_5_bn3, %self_layer3_4_relu_2), kwargs = {}) | |
| %self_layer3_5_relu_2 : [#users=2] = call_module[target=self_layer3_5_relu](args = (%iadd_12,), kwargs = {}) | |
| %self_layer4_0_conv1 : [#users=1] = call_module[target=self_layer4_0_conv1](args = (%self_layer3_5_relu_2,), kwargs = {}) | |
| %self_layer4_0_bn1 : [#users=1] = call_module[target=self_layer4_0_bn1](args = (%self_layer4_0_conv1,), kwargs = {}) | |
| %self_layer4_0_relu : [#users=1] = call_module[target=self_layer4_0_relu](args = (%self_layer4_0_bn1,), kwargs = {}) | |
| %self_layer4_0_conv2 : [#users=1] = call_module[target=self_layer4_0_conv2](args = (%self_layer4_0_relu,), kwargs = {}) | |
| %self_layer4_0_bn2 : [#users=1] = call_module[target=self_layer4_0_bn2](args = (%self_layer4_0_conv2,), kwargs = {}) | |
| %self_layer4_0_relu_1 : [#users=1] = call_module[target=self_layer4_0_relu](args = (%self_layer4_0_bn2,), kwargs = {}) | |
| %self_layer4_0_conv3 : [#users=1] = call_module[target=self_layer4_0_conv3](args = (%self_layer4_0_relu_1,), kwargs = {}) | |
| %self_layer4_0_bn3 : [#users=1] = call_module[target=self_layer4_0_bn3](args = (%self_layer4_0_conv3,), kwargs = {}) | |
| %self_layer4_0_downsample_0 : [#users=1] = call_module[target=self_layer4_0_downsample_0](args = (%self_layer3_5_relu_2,), kwargs = {}) | |
| %self_layer4_0_downsample_1 : [#users=1] = call_module[target=self_layer4_0_downsample_1](args = (%self_layer4_0_downsample_0,), kwargs = {}) | |
| %iadd_13 : [#users=1] = call_function[target=operator.iadd](args = (%self_layer4_0_bn3, %self_layer4_0_downsample_1), kwargs = {}) | |
| %self_layer4_0_relu_2 : [#users=2] = call_module[target=self_layer4_0_relu](args = (%iadd_13,), kwargs = {}) | |
| %self_layer4_1_conv1 : [#users=1] = call_module[target=self_layer4_1_conv1](args = (%self_layer4_0_relu_2,), kwargs = {}) | |
| %self_layer4_1_bn1 : [#users=1] = call_module[target=self_layer4_1_bn1](args = (%self_layer4_1_conv1,), kwargs = {}) | |
| %self_layer4_1_relu : [#users=1] = call_module[target=self_layer4_1_relu](args = (%self_layer4_1_bn1,), kwargs = {}) | |
| %self_layer4_1_conv2 : [#users=1] = call_module[target=self_layer4_1_conv2](args = (%self_layer4_1_relu,), kwargs = {}) | |
| %self_layer4_1_bn2 : [#users=1] = call_module[target=self_layer4_1_bn2](args = (%self_layer4_1_conv2,), kwargs = {}) | |
| %self_layer4_1_relu_1 : [#users=1] = call_module[target=self_layer4_1_relu](args = (%self_layer4_1_bn2,), kwargs = {}) | |
| %self_layer4_1_conv3 : [#users=1] = call_module[target=self_layer4_1_conv3](args = (%self_layer4_1_relu_1,), kwargs = {}) | |
| %self_layer4_1_bn3 : [#users=1] = call_module[target=self_layer4_1_bn3](args = (%self_layer4_1_conv3,), kwargs = {}) | |
| %iadd_14 : [#users=1] = call_function[target=operator.iadd](args = (%self_layer4_1_bn3, %self_layer4_0_relu_2), kwargs = {}) | |
| %self_layer4_1_relu_2 : [#users=2] = call_module[target=self_layer4_1_relu](args = (%iadd_14,), kwargs = {}) | |
| %self_layer4_2_conv1 : [#users=1] = call_module[target=self_layer4_2_conv1](args = (%self_layer4_1_relu_2,), kwargs = {}) | |
| %self_layer4_2_bn1 : [#users=1] = call_module[target=self_layer4_2_bn1](args = (%self_layer4_2_conv1,), kwargs = {}) | |
| %self_layer4_2_relu : [#users=1] = call_module[target=self_layer4_2_relu](args = (%self_layer4_2_bn1,), kwargs = {}) | |
| %self_layer4_2_conv2 : [#users=1] = call_module[target=self_layer4_2_conv2](args = (%self_layer4_2_relu,), kwargs = {}) | |
| %self_layer4_2_bn2 : [#users=1] = call_module[target=self_layer4_2_bn2](args = (%self_layer4_2_conv2,), kwargs = {}) | |
| %self_layer4_2_relu_1 : [#users=1] = call_module[target=self_layer4_2_relu](args = (%self_layer4_2_bn2,), kwargs = {}) | |
| %self_layer4_2_conv3 : [#users=1] = call_module[target=self_layer4_2_conv3](args = (%self_layer4_2_relu_1,), kwargs = {}) | |
| %self_layer4_2_bn3 : [#users=1] = call_module[target=self_layer4_2_bn3](args = (%self_layer4_2_conv3,), kwargs = {}) | |
| %iadd_15 : [#users=1] = call_function[target=operator.iadd](args = (%self_layer4_2_bn3, %self_layer4_1_relu_2), kwargs = {}) | |
| %self_layer4_2_relu_2 : [#users=1] = call_module[target=self_layer4_2_relu](args = (%iadd_15,), kwargs = {}) | |
| %self_avgpool : [#users=1] = call_module[target=self_avgpool](args = (%self_layer4_2_relu_2,), kwargs = {}) | |
| %flatten : [#users=1] = call_function[target=torch.flatten](args = (%self_avgpool, 1), kwargs = {}) | |
| %self_fc : [#users=1] = call_module[target=self_fc](args = (%flatten,), kwargs = {}) | |
| return (self_fc,) | |
| ---split graph--- | |
| graph(): | |
| %x : torch.Tensor [#users=1] = placeholder[target=x] | |
| %submod_0 : [#users=2] = call_module[target=submod_0](args = (%x,), kwargs = {}) | |
| %getitem : [#users=1] = call_function[target=operator.getitem](args = (%submod_0, 0), kwargs = {}) | |
| %getitem_1 : [#users=1] = call_function[target=operator.getitem](args = (%submod_0, 1), kwargs = {}) | |
| %submod_1 : [#users=2] = call_module[target=submod_1](args = (%getitem, %getitem_1), kwargs = {}) | |
| %getitem_2 : [#users=1] = call_function[target=operator.getitem](args = (%submod_1, 0), kwargs = {}) | |
| %getitem_3 : [#users=1] = call_function[target=operator.getitem](args = (%submod_1, 1), kwargs = {}) | |
| %submod_2 : [#users=2] = call_module[target=submod_2](args = (%getitem_2, %getitem_3), kwargs = {}) | |
| %getitem_4 : [#users=1] = call_function[target=operator.getitem](args = (%submod_2, 0), kwargs = {}) | |
| %getitem_5 : [#users=1] = call_function[target=operator.getitem](args = (%submod_2, 1), kwargs = {}) | |
| %submod_3 : [#users=1] = call_module[target=submod_3](args = (%getitem_4, %getitem_5), kwargs = {}) | |
| return (submod_3,) | |
| --- graph--- | |
| graph(): | |
| %x : torch.Tensor [#users=1] = placeholder[target=x] | |
| %submod_0 : [#users=2] = call_module[target=submod_0](args = (%x,), kwargs = {}) | |
| %getitem : [#users=1] = call_function[target=operator.getitem](args = (%submod_0, 0), kwargs = {}) | |
| %getitem_1 : [#users=1] = call_function[target=operator.getitem](args = (%submod_0, 1), kwargs = {}) | |
| %submod_1 : [#users=2] = call_module[target=submod_1](args = (%getitem, %getitem_1), kwargs = {}) | |
| %getitem_2 : [#users=1] = call_function[target=operator.getitem](args = (%submod_1, 0), kwargs = {}) | |
| %getitem_3 : [#users=1] = call_function[target=operator.getitem](args = (%submod_1, 1), kwargs = {}) | |
| %submod_2 : [#users=2] = call_module[target=submod_2](args = (%getitem_2, %getitem_3), kwargs = {}) | |
| %getitem_4 : [#users=1] = call_function[target=operator.getitem](args = (%submod_2, 0), kwargs = {}) | |
| %getitem_5 : [#users=1] = call_function[target=operator.getitem](args = (%submod_2, 1), kwargs = {}) | |
| %submod_3 : [#users=1] = call_module[target=submod_3](args = (%getitem_4, %getitem_5), kwargs = {}) | |
| return (submod_3,) | |
| ---submod_0 graph--- | |
| graph(): | |
| %x : [#users=1] = placeholder[target=x] | |
| %self_conv1 : [#users=1] = call_module[target=self_conv1](args = (%x,), kwargs = {}) | |
| %self_bn1 : [#users=1] = call_module[target=self_bn1](args = (%self_conv1,), kwargs = {}) | |
| %self_relu : [#users=1] = call_module[target=self_relu](args = (%self_bn1,), kwargs = {}) | |
| %self_maxpool : [#users=2] = call_module[target=self_maxpool](args = (%self_relu,), kwargs = {}) | |
| %self_layer1_0_conv1 : [#users=1] = call_module[target=self_layer1_0_conv1](args = (%self_maxpool,), kwargs = {}) | |
| %self_layer1_0_bn1 : [#users=1] = call_module[target=self_layer1_0_bn1](args = (%self_layer1_0_conv1,), kwargs = {}) | |
| %self_layer1_0_relu : [#users=1] = call_module[target=self_layer1_0_relu](args = (%self_layer1_0_bn1,), kwargs = {}) | |
| %self_layer1_0_conv2 : [#users=1] = call_module[target=self_layer1_0_conv2](args = (%self_layer1_0_relu,), kwargs = {}) | |
| %self_layer1_0_bn2 : [#users=1] = call_module[target=self_layer1_0_bn2](args = (%self_layer1_0_conv2,), kwargs = {}) | |
| %self_layer1_0_relu_1 : [#users=1] = call_module[target=self_layer1_0_relu](args = (%self_layer1_0_bn2,), kwargs = {}) | |
| %self_layer1_0_conv3 : [#users=1] = call_module[target=self_layer1_0_conv3](args = (%self_layer1_0_relu_1,), kwargs = {}) | |
| %self_layer1_0_bn3 : [#users=1] = call_module[target=self_layer1_0_bn3](args = (%self_layer1_0_conv3,), kwargs = {}) | |
| %self_layer1_0_downsample_0 : [#users=1] = call_module[target=self_layer1_0_downsample_0](args = (%self_maxpool,), kwargs = {}) | |
| %self_layer1_0_downsample_1 : [#users=1] = call_module[target=self_layer1_0_downsample_1](args = (%self_layer1_0_downsample_0,), kwargs = {}) | |
| %iadd : [#users=1] = call_function[target=operator.iadd](args = (%self_layer1_0_bn3, %self_layer1_0_downsample_1), kwargs = {}) | |
| %self_layer1_0_relu_2 : [#users=2] = call_module[target=self_layer1_0_relu](args = (%iadd,), kwargs = {}) | |
| %self_layer1_1_conv1 : [#users=1] = call_module[target=self_layer1_1_conv1](args = (%self_layer1_0_relu_2,), kwargs = {}) | |
| %self_layer1_1_bn1 : [#users=1] = call_module[target=self_layer1_1_bn1](args = (%self_layer1_1_conv1,), kwargs = {}) | |
| %self_layer1_1_relu : [#users=1] = call_module[target=self_layer1_1_relu](args = (%self_layer1_1_bn1,), kwargs = {}) | |
| %self_layer1_1_conv2 : [#users=1] = call_module[target=self_layer1_1_conv2](args = (%self_layer1_1_relu,), kwargs = {}) | |
| %self_layer1_1_bn2 : [#users=1] = call_module[target=self_layer1_1_bn2](args = (%self_layer1_1_conv2,), kwargs = {}) | |
| %self_layer1_1_relu_1 : [#users=1] = call_module[target=self_layer1_1_relu](args = (%self_layer1_1_bn2,), kwargs = {}) | |
| %self_layer1_1_conv3 : [#users=1] = call_module[target=self_layer1_1_conv3](args = (%self_layer1_1_relu_1,), kwargs = {}) | |
| %self_layer1_1_bn3 : [#users=1] = call_module[target=self_layer1_1_bn3](args = (%self_layer1_1_conv3,), kwargs = {}) | |
| %iadd_1 : [#users=1] = call_function[target=operator.iadd](args = (%self_layer1_1_bn3, %self_layer1_0_relu_2), kwargs = {}) | |
| %self_layer1_1_relu_2 : [#users=2] = call_module[target=self_layer1_1_relu](args = (%iadd_1,), kwargs = {}) | |
| %self_layer1_2_conv1 : [#users=1] = call_module[target=self_layer1_2_conv1](args = (%self_layer1_1_relu_2,), kwargs = {}) | |
| %self_layer1_2_bn1 : [#users=1] = call_module[target=self_layer1_2_bn1](args = (%self_layer1_2_conv1,), kwargs = {}) | |
| %self_layer1_2_relu : [#users=1] = call_module[target=self_layer1_2_relu](args = (%self_layer1_2_bn1,), kwargs = {}) | |
| %self_layer1_2_conv2 : [#users=1] = call_module[target=self_layer1_2_conv2](args = (%self_layer1_2_relu,), kwargs = {}) | |
| %self_layer1_2_bn2 : [#users=1] = call_module[target=self_layer1_2_bn2](args = (%self_layer1_2_conv2,), kwargs = {}) | |
| %self_layer1_2_relu_1 : [#users=1] = call_module[target=self_layer1_2_relu](args = (%self_layer1_2_bn2,), kwargs = {}) | |
| %self_layer1_2_conv3 : [#users=1] = call_module[target=self_layer1_2_conv3](args = (%self_layer1_2_relu_1,), kwargs = {}) | |
| %self_layer1_2_bn3 : [#users=1] = call_module[target=self_layer1_2_bn3](args = (%self_layer1_2_conv3,), kwargs = {}) | |
| %iadd_2 : [#users=1] = call_function[target=operator.iadd](args = (%self_layer1_2_bn3, %self_layer1_1_relu_2), kwargs = {}) | |
| %self_layer1_2_relu_2 : [#users=2] = call_module[target=self_layer1_2_relu](args = (%iadd_2,), kwargs = {}) | |
| %self_layer2_0_conv1 : [#users=1] = call_module[target=self_layer2_0_conv1](args = (%self_layer1_2_relu_2,), kwargs = {}) | |
| %self_layer2_0_bn1 : [#users=1] = call_module[target=self_layer2_0_bn1](args = (%self_layer2_0_conv1,), kwargs = {}) | |
| %self_layer2_0_relu : [#users=1] = call_module[target=self_layer2_0_relu](args = (%self_layer2_0_bn1,), kwargs = {}) | |
| %self_layer2_0_conv2 : [#users=1] = call_module[target=self_layer2_0_conv2](args = (%self_layer2_0_relu,), kwargs = {}) | |
| %self_layer2_0_bn2 : [#users=1] = call_module[target=self_layer2_0_bn2](args = (%self_layer2_0_conv2,), kwargs = {}) | |
| %self_layer2_0_relu_1 : [#users=1] = call_module[target=self_layer2_0_relu](args = (%self_layer2_0_bn2,), kwargs = {}) | |
| %self_layer2_0_conv3 : [#users=1] = call_module[target=self_layer2_0_conv3](args = (%self_layer2_0_relu_1,), kwargs = {}) | |
| %self_layer2_0_bn3 : [#users=1] = call_module[target=self_layer2_0_bn3](args = (%self_layer2_0_conv3,), kwargs = {}) | |
| %self_layer2_0_downsample_0 : [#users=1] = call_module[target=self_layer2_0_downsample_0](args = (%self_layer1_2_relu_2,), kwargs = {}) | |
| %self_layer2_0_downsample_1 : [#users=1] = call_module[target=self_layer2_0_downsample_1](args = (%self_layer2_0_downsample_0,), kwargs = {}) | |
| %iadd_3 : [#users=1] = call_function[target=operator.iadd](args = (%self_layer2_0_bn3, %self_layer2_0_downsample_1), kwargs = {}) | |
| %self_layer2_0_relu_2 : [#users=2] = call_module[target=self_layer2_0_relu](args = (%iadd_3,), kwargs = {}) | |
| %self_layer2_1_conv1 : [#users=1] = call_module[target=self_layer2_1_conv1](args = (%self_layer2_0_relu_2,), kwargs = {}) | |
| %self_layer2_1_bn1 : [#users=1] = call_module[target=self_layer2_1_bn1](args = (%self_layer2_1_conv1,), kwargs = {}) | |
| %self_layer2_1_relu : [#users=1] = call_module[target=self_layer2_1_relu](args = (%self_layer2_1_bn1,), kwargs = {}) | |
| %self_layer2_1_conv2 : [#users=1] = call_module[target=self_layer2_1_conv2](args = (%self_layer2_1_relu,), kwargs = {}) | |
| %self_layer2_1_bn2 : [#users=1] = call_module[target=self_layer2_1_bn2](args = (%self_layer2_1_conv2,), kwargs = {}) | |
| %self_layer2_1_relu_1 : [#users=1] = call_module[target=self_layer2_1_relu](args = (%self_layer2_1_bn2,), kwargs = {}) | |
| %self_layer2_1_conv3 : [#users=1] = call_module[target=self_layer2_1_conv3](args = (%self_layer2_1_relu_1,), kwargs = {}) | |
| %self_layer2_1_bn3 : [#users=1] = call_module[target=self_layer2_1_bn3](args = (%self_layer2_1_conv3,), kwargs = {}) | |
| %iadd_4 : [#users=1] = call_function[target=operator.iadd](args = (%self_layer2_1_bn3, %self_layer2_0_relu_2), kwargs = {}) | |
| %self_layer2_1_relu_2 : [#users=2] = call_module[target=self_layer2_1_relu](args = (%iadd_4,), kwargs = {}) | |
| %self_layer2_2_conv1 : [#users=1] = call_module[target=self_layer2_2_conv1](args = (%self_layer2_1_relu_2,), kwargs = {}) | |
| %self_layer2_2_bn1 : [#users=1] = call_module[target=self_layer2_2_bn1](args = (%self_layer2_2_conv1,), kwargs = {}) | |
| %self_layer2_2_relu : [#users=1] = call_module[target=self_layer2_2_relu](args = (%self_layer2_2_bn1,), kwargs = {}) | |
| %self_layer2_2_conv2 : [#users=1] = call_module[target=self_layer2_2_conv2](args = (%self_layer2_2_relu,), kwargs = {}) | |
| %self_layer2_2_bn2 : [#users=1] = call_module[target=self_layer2_2_bn2](args = (%self_layer2_2_conv2,), kwargs = {}) | |
| %self_layer2_2_relu_1 : [#users=1] = call_module[target=self_layer2_2_relu](args = (%self_layer2_2_bn2,), kwargs = {}) | |
| %self_layer2_2_conv3 : [#users=1] = call_module[target=self_layer2_2_conv3](args = (%self_layer2_2_relu_1,), kwargs = {}) | |
| %self_layer2_2_bn3 : [#users=1] = call_module[target=self_layer2_2_bn3](args = (%self_layer2_2_conv3,), kwargs = {}) | |
| %iadd_5 : [#users=1] = call_function[target=operator.iadd](args = (%self_layer2_2_bn3, %self_layer2_1_relu_2), kwargs = {}) | |
| %self_layer2_2_relu_2 : [#users=2] = call_module[target=self_layer2_2_relu](args = (%iadd_5,), kwargs = {}) | |
| %self_layer2_3_conv1 : [#users=1] = call_module[target=self_layer2_3_conv1](args = (%self_layer2_2_relu_2,), kwargs = {}) | |
| %self_layer2_3_bn1 : [#users=1] = call_module[target=self_layer2_3_bn1](args = (%self_layer2_3_conv1,), kwargs = {}) | |
| %self_layer2_3_relu : [#users=1] = call_module[target=self_layer2_3_relu](args = (%self_layer2_3_bn1,), kwargs = {}) | |
| %self_layer2_3_conv2 : [#users=1] = call_module[target=self_layer2_3_conv2](args = (%self_layer2_3_relu,), kwargs = {}) | |
| %self_layer2_3_bn2 : [#users=1] = call_module[target=self_layer2_3_bn2](args = (%self_layer2_3_conv2,), kwargs = {}) | |
| %self_layer2_3_relu_1 : [#users=1] = call_module[target=self_layer2_3_relu](args = (%self_layer2_3_bn2,), kwargs = {}) | |
| %self_layer2_3_conv3 : [#users=1] = call_module[target=self_layer2_3_conv3](args = (%self_layer2_3_relu_1,), kwargs = {}) | |
| %self_layer2_3_bn3 : [#users=1] = call_module[target=self_layer2_3_bn3](args = (%self_layer2_3_conv3,), kwargs = {}) | |
| %iadd_6 : [#users=1] = call_function[target=operator.iadd](args = (%self_layer2_3_bn3, %self_layer2_2_relu_2), kwargs = {}) | |
| %self_layer2_3_relu_2 : [#users=2] = call_module[target=self_layer2_3_relu](args = (%iadd_6,), kwargs = {}) | |
| %self_layer3_0_conv1 : [#users=1] = call_module[target=self_layer3_0_conv1](args = (%self_layer2_3_relu_2,), kwargs = {}) | |
| %self_layer3_0_bn1 : [#users=1] = call_module[target=self_layer3_0_bn1](args = (%self_layer3_0_conv1,), kwargs = {}) | |
| %self_layer3_0_relu : [#users=1] = call_module[target=self_layer3_0_relu](args = (%self_layer3_0_bn1,), kwargs = {}) | |
| %self_layer3_0_conv2 : [#users=1] = call_module[target=self_layer3_0_conv2](args = (%self_layer3_0_relu,), kwargs = {}) | |
| %self_layer3_0_bn2 : [#users=1] = call_module[target=self_layer3_0_bn2](args = (%self_layer3_0_conv2,), kwargs = {}) | |
| %self_layer3_0_relu_1 : [#users=1] = call_module[target=self_layer3_0_relu](args = (%self_layer3_0_bn2,), kwargs = {}) | |
| %self_layer3_0_conv3 : [#users=1] = call_module[target=self_layer3_0_conv3](args = (%self_layer3_0_relu_1,), kwargs = {}) | |
| %self_layer3_0_bn3 : [#users=1] = call_module[target=self_layer3_0_bn3](args = (%self_layer3_0_conv3,), kwargs = {}) | |
| %self_layer3_0_downsample_0 : [#users=1] = call_module[target=self_layer3_0_downsample_0](args = (%self_layer2_3_relu_2,), kwargs = {}) | |
| %self_layer3_0_downsample_1 : [#users=1] = call_module[target=self_layer3_0_downsample_1](args = (%self_layer3_0_downsample_0,), kwargs = {}) | |
| %iadd_7 : [#users=1] = call_function[target=operator.iadd](args = (%self_layer3_0_bn3, %self_layer3_0_downsample_1), kwargs = {}) | |
| %self_layer3_0_relu_2 : [#users=2] = call_module[target=self_layer3_0_relu](args = (%iadd_7,), kwargs = {}) | |
| %self_layer3_1_conv1 : [#users=1] = call_module[target=self_layer3_1_conv1](args = (%self_layer3_0_relu_2,), kwargs = {}) | |
| %self_layer3_1_bn1 : [#users=1] = call_module[target=self_layer3_1_bn1](args = (%self_layer3_1_conv1,), kwargs = {}) | |
| %self_layer3_1_relu : [#users=1] = call_module[target=self_layer3_1_relu](args = (%self_layer3_1_bn1,), kwargs = {}) | |
| %self_layer3_1_conv2 : [#users=1] = call_module[target=self_layer3_1_conv2](args = (%self_layer3_1_relu,), kwargs = {}) | |
| %self_layer3_1_bn2 : [#users=1] = call_module[target=self_layer3_1_bn2](args = (%self_layer3_1_conv2,), kwargs = {}) | |
| %self_layer3_1_relu_1 : [#users=1] = call_module[target=self_layer3_1_relu](args = (%self_layer3_1_bn2,), kwargs = {}) | |
| %self_layer3_1_conv3 : [#users=1] = call_module[target=self_layer3_1_conv3](args = (%self_layer3_1_relu_1,), kwargs = {}) | |
| %self_layer3_1_bn3 : [#users=1] = call_module[target=self_layer3_1_bn3](args = (%self_layer3_1_conv3,), kwargs = {}) | |
| %iadd_8 : [#users=1] = call_function[target=operator.iadd](args = (%self_layer3_1_bn3, %self_layer3_0_relu_2), kwargs = {}) | |
| %self_layer3_1_relu_2 : [#users=2] = call_module[target=self_layer3_1_relu](args = (%iadd_8,), kwargs = {}) | |
| %self_layer3_2_conv1 : [#users=1] = call_module[target=self_layer3_2_conv1](args = (%self_layer3_1_relu_2,), kwargs = {}) | |
| %self_layer3_2_bn1 : [#users=1] = call_module[target=self_layer3_2_bn1](args = (%self_layer3_2_conv1,), kwargs = {}) | |
| %self_layer3_2_relu : [#users=1] = call_module[target=self_layer3_2_relu](args = (%self_layer3_2_bn1,), kwargs = {}) | |
| return (self_layer3_2_relu, self_layer3_1_relu_2) | |
| ---submod_1 graph--- | |
| graph(): | |
| %self_layer3_2_relu : [#users=1] = placeholder[target=self_layer3_2_relu] | |
| %self_layer3_1_relu_2 : [#users=1] = placeholder[target=self_layer3_1_relu_2] | |
| %self_layer3_2_conv2 : [#users=1] = call_module[target=self_layer3_2_conv2](args = (%self_layer3_2_relu,), kwargs = {}) | |
| %self_layer3_2_bn2 : [#users=1] = call_module[target=self_layer3_2_bn2](args = (%self_layer3_2_conv2,), kwargs = {}) | |
| %self_layer3_2_relu_1 : [#users=1] = call_module[target=self_layer3_2_relu](args = (%self_layer3_2_bn2,), kwargs = {}) | |
| %self_layer3_2_conv3 : [#users=1] = call_module[target=self_layer3_2_conv3](args = (%self_layer3_2_relu_1,), kwargs = {}) | |
| %self_layer3_2_bn3 : [#users=1] = call_module[target=self_layer3_2_bn3](args = (%self_layer3_2_conv3,), kwargs = {}) | |
| %iadd : [#users=1] = call_function[target=operator.iadd](args = (%self_layer3_2_bn3, %self_layer3_1_relu_2), kwargs = {}) | |
| %self_layer3_2_relu_2 : [#users=2] = call_module[target=self_layer3_2_relu](args = (%iadd,), kwargs = {}) | |
| %self_layer3_3_conv1 : [#users=1] = call_module[target=self_layer3_3_conv1](args = (%self_layer3_2_relu_2,), kwargs = {}) | |
| %self_layer3_3_bn1 : [#users=1] = call_module[target=self_layer3_3_bn1](args = (%self_layer3_3_conv1,), kwargs = {}) | |
| %self_layer3_3_relu : [#users=1] = call_module[target=self_layer3_3_relu](args = (%self_layer3_3_bn1,), kwargs = {}) | |
| %self_layer3_3_conv2 : [#users=1] = call_module[target=self_layer3_3_conv2](args = (%self_layer3_3_relu,), kwargs = {}) | |
| %self_layer3_3_bn2 : [#users=1] = call_module[target=self_layer3_3_bn2](args = (%self_layer3_3_conv2,), kwargs = {}) | |
| %self_layer3_3_relu_1 : [#users=1] = call_module[target=self_layer3_3_relu](args = (%self_layer3_3_bn2,), kwargs = {}) | |
| %self_layer3_3_conv3 : [#users=1] = call_module[target=self_layer3_3_conv3](args = (%self_layer3_3_relu_1,), kwargs = {}) | |
| %self_layer3_3_bn3 : [#users=1] = call_module[target=self_layer3_3_bn3](args = (%self_layer3_3_conv3,), kwargs = {}) | |
| %iadd_1 : [#users=1] = call_function[target=operator.iadd](args = (%self_layer3_3_bn3, %self_layer3_2_relu_2), kwargs = {}) | |
| %self_layer3_3_relu_2 : [#users=2] = call_module[target=self_layer3_3_relu](args = (%iadd_1,), kwargs = {}) | |
| %self_layer3_4_conv1 : [#users=1] = call_module[target=self_layer3_4_conv1](args = (%self_layer3_3_relu_2,), kwargs = {}) | |
| %self_layer3_4_bn1 : [#users=1] = call_module[target=self_layer3_4_bn1](args = (%self_layer3_4_conv1,), kwargs = {}) | |
| %self_layer3_4_relu : [#users=1] = call_module[target=self_layer3_4_relu](args = (%self_layer3_4_bn1,), kwargs = {}) | |
| %self_layer3_4_conv2 : [#users=1] = call_module[target=self_layer3_4_conv2](args = (%self_layer3_4_relu,), kwargs = {}) | |
| %self_layer3_4_bn2 : [#users=1] = call_module[target=self_layer3_4_bn2](args = (%self_layer3_4_conv2,), kwargs = {}) | |
| %self_layer3_4_relu_1 : [#users=1] = call_module[target=self_layer3_4_relu](args = (%self_layer3_4_bn2,), kwargs = {}) | |
| %self_layer3_4_conv3 : [#users=1] = call_module[target=self_layer3_4_conv3](args = (%self_layer3_4_relu_1,), kwargs = {}) | |
| %self_layer3_4_bn3 : [#users=1] = call_module[target=self_layer3_4_bn3](args = (%self_layer3_4_conv3,), kwargs = {}) | |
| %iadd_2 : [#users=1] = call_function[target=operator.iadd](args = (%self_layer3_4_bn3, %self_layer3_3_relu_2), kwargs = {}) | |
| %self_layer3_4_relu_2 : [#users=2] = call_module[target=self_layer3_4_relu](args = (%iadd_2,), kwargs = {}) | |
| %self_layer3_5_conv1 : [#users=1] = call_module[target=self_layer3_5_conv1](args = (%self_layer3_4_relu_2,), kwargs = {}) | |
| %self_layer3_5_bn1 : [#users=1] = call_module[target=self_layer3_5_bn1](args = (%self_layer3_5_conv1,), kwargs = {}) | |
| %self_layer3_5_relu : [#users=1] = call_module[target=self_layer3_5_relu](args = (%self_layer3_5_bn1,), kwargs = {}) | |
| %self_layer3_5_conv2 : [#users=1] = call_module[target=self_layer3_5_conv2](args = (%self_layer3_5_relu,), kwargs = {}) | |
| %self_layer3_5_bn2 : [#users=1] = call_module[target=self_layer3_5_bn2](args = (%self_layer3_5_conv2,), kwargs = {}) | |
| %self_layer3_5_relu_1 : [#users=1] = call_module[target=self_layer3_5_relu](args = (%self_layer3_5_bn2,), kwargs = {}) | |
| %self_layer3_5_conv3 : [#users=1] = call_module[target=self_layer3_5_conv3](args = (%self_layer3_5_relu_1,), kwargs = {}) | |
| %self_layer3_5_bn3 : [#users=1] = call_module[target=self_layer3_5_bn3](args = (%self_layer3_5_conv3,), kwargs = {}) | |
| %iadd_3 : [#users=1] = call_function[target=operator.iadd](args = (%self_layer3_5_bn3, %self_layer3_4_relu_2), kwargs = {}) | |
| %self_layer3_5_relu_2 : [#users=2] = call_module[target=self_layer3_5_relu](args = (%iadd_3,), kwargs = {}) | |
| %self_layer4_0_conv1 : [#users=1] = call_module[target=self_layer4_0_conv1](args = (%self_layer3_5_relu_2,), kwargs = {}) | |
| %self_layer4_0_bn1 : [#users=1] = call_module[target=self_layer4_0_bn1](args = (%self_layer4_0_conv1,), kwargs = {}) | |
| %self_layer4_0_relu : [#users=1] = call_module[target=self_layer4_0_relu](args = (%self_layer4_0_bn1,), kwargs = {}) | |
| %self_layer4_0_conv2 : [#users=1] = call_module[target=self_layer4_0_conv2](args = (%self_layer4_0_relu,), kwargs = {}) | |
| %self_layer4_0_bn2 : [#users=1] = call_module[target=self_layer4_0_bn2](args = (%self_layer4_0_conv2,), kwargs = {}) | |
| %self_layer4_0_relu_1 : [#users=1] = call_module[target=self_layer4_0_relu](args = (%self_layer4_0_bn2,), kwargs = {}) | |
| return (self_layer4_0_relu_1, self_layer3_5_relu_2) | |
| ---submod_2 graph--- | |
| graph(): | |
| %self_layer4_0_relu_1 : [#users=1] = placeholder[target=self_layer4_0_relu_1] | |
| %self_layer3_5_relu_2 : [#users=1] = placeholder[target=self_layer3_5_relu_2] | |
| %self_layer4_0_conv3 : [#users=1] = call_module[target=self_layer4_0_conv3](args = (%self_layer4_0_relu_1,), kwargs = {}) | |
| %self_layer4_0_bn3 : [#users=1] = call_module[target=self_layer4_0_bn3](args = (%self_layer4_0_conv3,), kwargs = {}) | |
| %self_layer4_0_downsample_0 : [#users=1] = call_module[target=self_layer4_0_downsample_0](args = (%self_layer3_5_relu_2,), kwargs = {}) | |
| %self_layer4_0_downsample_1 : [#users=1] = call_module[target=self_layer4_0_downsample_1](args = (%self_layer4_0_downsample_0,), kwargs = {}) | |
| %iadd : [#users=1] = call_function[target=operator.iadd](args = (%self_layer4_0_bn3, %self_layer4_0_downsample_1), kwargs = {}) | |
| %self_layer4_0_relu : [#users=2] = call_module[target=self_layer4_0_relu](args = (%iadd,), kwargs = {}) | |
| %self_layer4_1_conv1 : [#users=1] = call_module[target=self_layer4_1_conv1](args = (%self_layer4_0_relu,), kwargs = {}) | |
| %self_layer4_1_bn1 : [#users=1] = call_module[target=self_layer4_1_bn1](args = (%self_layer4_1_conv1,), kwargs = {}) | |
| %self_layer4_1_relu : [#users=1] = call_module[target=self_layer4_1_relu](args = (%self_layer4_1_bn1,), kwargs = {}) | |
| %self_layer4_1_conv2 : [#users=1] = call_module[target=self_layer4_1_conv2](args = (%self_layer4_1_relu,), kwargs = {}) | |
| %self_layer4_1_bn2 : [#users=1] = call_module[target=self_layer4_1_bn2](args = (%self_layer4_1_conv2,), kwargs = {}) | |
| %self_layer4_1_relu_1 : [#users=1] = call_module[target=self_layer4_1_relu](args = (%self_layer4_1_bn2,), kwargs = {}) | |
| return (self_layer4_1_relu_1, self_layer4_0_relu) | |
| ---submod_3 graph--- | |
| graph(): | |
| %self_layer4_1_relu_1 : [#users=1] = placeholder[target=self_layer4_1_relu_1] | |
| %self_layer4_0_relu_2 : [#users=1] = placeholder[target=self_layer4_0_relu_2] | |
| %self_layer4_1_conv3 : [#users=1] = call_module[target=self_layer4_1_conv3](args = (%self_layer4_1_relu_1,), kwargs = {}) | |
| %self_layer4_1_bn3 : [#users=1] = call_module[target=self_layer4_1_bn3](args = (%self_layer4_1_conv3,), kwargs = {}) | |
| %iadd : [#users=1] = call_function[target=operator.iadd](args = (%self_layer4_1_bn3, %self_layer4_0_relu_2), kwargs = {}) | |
| %self_layer4_1_relu : [#users=2] = call_module[target=self_layer4_1_relu](args = (%iadd,), kwargs = {}) | |
| %self_layer4_2_conv1 : [#users=1] = call_module[target=self_layer4_2_conv1](args = (%self_layer4_1_relu,), kwargs = {}) | |
| %self_layer4_2_bn1 : [#users=1] = call_module[target=self_layer4_2_bn1](args = (%self_layer4_2_conv1,), kwargs = {}) | |
| %self_layer4_2_relu : [#users=1] = call_module[target=self_layer4_2_relu](args = (%self_layer4_2_bn1,), kwargs = {}) | |
| %self_layer4_2_conv2 : [#users=1] = call_module[target=self_layer4_2_conv2](args = (%self_layer4_2_relu,), kwargs = {}) | |
| %self_layer4_2_bn2 : [#users=1] = call_module[target=self_layer4_2_bn2](args = (%self_layer4_2_conv2,), kwargs = {}) | |
| %self_layer4_2_relu_1 : [#users=1] = call_module[target=self_layer4_2_relu](args = (%self_layer4_2_bn2,), kwargs = {}) | |
| %self_layer4_2_conv3 : [#users=1] = call_module[target=self_layer4_2_conv3](args = (%self_layer4_2_relu_1,), kwargs = {}) | |
| %self_layer4_2_bn3 : [#users=1] = call_module[target=self_layer4_2_bn3](args = (%self_layer4_2_conv3,), kwargs = {}) | |
| %iadd_1 : [#users=1] = call_function[target=operator.iadd](args = (%self_layer4_2_bn3, %self_layer4_1_relu), kwargs = {}) | |
| %self_layer4_2_relu_2 : [#users=1] = call_module[target=self_layer4_2_relu](args = (%iadd_1,), kwargs = {}) | |
| %self_avgpool : [#users=1] = call_module[target=self_avgpool](args = (%self_layer4_2_relu_2,), kwargs = {}) | |
| %flatten : [#users=1] = call_function[target=torch.flatten](args = (%self_avgpool, 1), kwargs = {}) | |
| %self_fc : [#users=1] = call_module[target=self_fc](args = (%flatten,), kwargs = {}) | |
| return self_fc | |
| --------------- | |
| run_node placeholder, x got args tuple() | |
| run_node call_module, submod_0 got args tuple(T[torch.Size([32, 3, 224, 224])]) | |
| still waiting for process to exit | |
| didn't get the result, queue empty | |
| still waiting for process to exit | |
| didn't get the result, queue empty | |
| still waiting for process to exit | |
| didn't get the result, queue empty | |
| run_node call_function, <built-in function getitem> got args tuple(tuple(T[torch.Size([32, 256, 14, 14])], T[torch.Size([32, 1024, 14, 14])]), 0) | |
| run_node call_function, <built-in function getitem> got args tuple(tuple(T[torch.Size([32, 256, 14, 14])], T[torch.Size([32, 1024, 14, 14])]), 1) | |
| run_node call_module, submod_1 got args tuple(T[torch.Size([32, 256, 14, 14])], T[torch.Size([32, 1024, 14, 14])]) | |
| still waiting for process to exit | |
| run_node call_function, <built-in function getitem> got args tuple(tuple(T[torch.Size([32, 512, 7, 7])], T[torch.Size([32, 1024, 14, 14])]), 0) | |
| run_node call_function, <built-in function getitem> got args tuple(tuple(T[torch.Size([32, 512, 7, 7])], T[torch.Size([32, 1024, 14, 14])]), 1) | |
| run_node call_module, submod_2 got args tuple(T[torch.Size([32, 512, 7, 7])], T[torch.Size([32, 1024, 14, 14])]) | |
| didn't get the result, queue empty | |
| still waiting for process to exit | |
| didn't get the result, queue empty | |
| run_node call_function, <built-in function getitem> got args tuple(tuple(T[torch.Size([32, 512, 7, 7])], T[torch.Size([32, 2048, 7, 7])]), 0) | |
| run_node call_function, <built-in function getitem> got args tuple(tuple(T[torch.Size([32, 512, 7, 7])], T[torch.Size([32, 2048, 7, 7])]), 1) | |
| run_node call_module, submod_3 got args tuple(T[torch.Size([32, 512, 7, 7])], T[torch.Size([32, 2048, 7, 7])]) | |
| run_node output, output got args tuple(tuple(T[torch.Size([32, 1000])])) | |
| ---final graph--- | |
| graph(): | |
| %x : torch.Tensor [#users=1] = placeholder[target=x] | |
| %submod_0 : [#users=2] = call_module[target=compiled_submod_0](args = (%x,), kwargs = {}) | |
| %getitem : [#users=1] = call_function[target=operator.getitem](args = (%submod_0, 0), kwargs = {}) | |
| %getitem_1 : [#users=1] = call_function[target=operator.getitem](args = (%submod_0, 1), kwargs = {}) | |
| %submod_1 : [#users=2] = call_module[target=compiled_submod_1](args = (%getitem, %getitem_1), kwargs = {}) | |
| %getitem_2 : [#users=1] = call_function[target=operator.getitem](args = (%submod_1, 0), kwargs = {}) | |
| %getitem_3 : [#users=1] = call_function[target=operator.getitem](args = (%submod_1, 1), kwargs = {}) | |
| %submod_2 : [#users=2] = call_module[target=compiled_submod_2](args = (%getitem_2, %getitem_3), kwargs = {}) | |
| %getitem_4 : [#users=1] = call_function[target=operator.getitem](args = (%submod_2, 0), kwargs = {}) | |
| %getitem_5 : [#users=1] = call_function[target=operator.getitem](args = (%submod_2, 1), kwargs = {}) | |
| %submod_3 : [#users=1] = call_module[target=compiled_submod_3](args = (%getitem_4, %getitem_5), kwargs = {}) | |
| return (submod_3,) | |
| --------------- | |
| still waiting for process to exit | |
| didn't get the result, queue empty | |
| still waiting for process to exit | |
| didn't get the result, queue empty | |
| still waiting for process to exit | |
| didn't get the result, queue empty | |
| still waiting for process to exit | |
| didn't get the result, queue empty | |
| still waiting for process to exit | |
| didn't get the result, queue empty | |
| still waiting for process to exit | |
| didn't get the result, queue empty | |
| still waiting for process to exit | |
| didn't get the result, queue empty | |
| still waiting for process to exit | |
| didn't get the result, queue empty | |
| still waiting for process to exit | |
| didn't get the result, queue empty | |
| still waiting for process to exit | |
| didn't get the result, queue empty | |
| still waiting for process to exit | |
| didn't get the result, queue empty | |
| still waiting for process to exit | |
| didn't get the result, queue empty | |
| still waiting for process to exit | |
| didn't get the result, queue empty | |
| still waiting for process to exit | |
| didn't get the result, queue empty | |
| still waiting for process to exit | |
| didn't get the result, queue empty | |
| still waiting for process to exit | |
| didn't get the result, queue empty | |
| still waiting for process to exit | |
| didn't get the result, queue empty | |
| still waiting for process to exit | |
| didn't get the result, queue empty | |
| still waiting for process to exit | |
| didn't get the result, queue empty | |
| still waiting for process to exit | |
| didn't get the result, queue empty | |
| still waiting for process to exit | |
| didn't get the result, queue empty | |
| still waiting for process to exit | |
| result {'latency_median': 88.8248291015625, 'latency_stdev': 0.22256090897985134} | |
| has been placed into the queue | |
| got the result! | |
| got exit 0 | |
| (already got results) | |
| exit code: 0 and result: {'nodes': 2, 'model_name': 'torchbenchmark.models.resnet50.Model', 'backend': 'torchdynamo_inductor', 'has_breaks': True, 'static_graph': False, 'result': {'latency_median': 88.8248291015625, 'latency_stdev': 0.22256090897985134}} | |
| <RESULT>{"nodes": 2, "model_name": "torchbenchmark.models.resnet50.Model", "backend": "torchdynamo_inductor", "has_breaks": true, "static_graph": false, "result": {"latency_median": 88.8248291015625, "latency_stdev": 0.22256090897985134}}</RESULT> | |
| submitit INFO (2022-10-26 02:57:12,347) - Job completed successfully |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| /data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and may be removed in the future, please use 'weights' instead. | |
| warnings.warn( | |
| /data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and may be removed in the future. The current behavior is equivalent to passing `weights=ResNet50_Weights.IMAGENET1K_V1`. You can also use `weights=ResNet50_Weights.DEFAULT` to get the most up-to-date weights. | |
| warnings.warn(msg) | |
| /data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and may be removed in the future, please use 'weights' instead. | |
| warnings.warn( | |
| /data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and may be removed in the future. The current behavior is equivalent to passing `weights=ResNet50_Weights.IMAGENET1K_V1`. You can also use `weights=ResNet50_Weights.DEFAULT` to get the most up-to-date weights. | |
| warnings.warn(msg) | |
| [2022-10-26 23:11:52,922] torch._dynamo.variables.torch: [WARNING] Profiler will be ignored | |
| [2022-10-26 23:11:54,279] torch._dynamo.variables.torch: [WARNING] Profiler will be ignored | |
| [2022-10-26 23:13:29,380] torch._dynamo.optimizations.training: [WARNING] Unable to use Aot Autograd because of presence of mutation | |
| [2022-10-26 23:13:29,380] torch._inductor.compile_fx: [WARNING] Aot Autograd is not safe to run, so falling back to eager | |
| [2022-10-26 23:13:29,554] torch._dynamo.variables.torch: [WARNING] Profiler will be ignored | |
| [2022-10-26 23:13:29,806] torch._dynamo.variables.torch: [WARNING] Profiler will be ignored | |
| [2022-10-26 23:13:30,055] torch._dynamo.variables.torch: [WARNING] Profiler will be ignored | |
| [2022-10-26 23:14:47,694] torch._dynamo.optimizations.training: [WARNING] Unable to use Aot Autograd because of presence of mutation | |
| [2022-10-26 23:14:47,695] torch._inductor.compile_fx: [WARNING] Aot Autograd is not safe to run, so falling back to eager | |
| /data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and may be removed in the future, please use 'weights' instead. | |
| warnings.warn( | |
| /data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and may be removed in the future. The current behavior is equivalent to passing `weights=ResNet50_Weights.IMAGENET1K_V1`. You can also use `weights=ResNet50_Weights.DEFAULT` to get the most up-to-date weights. | |
| warnings.warn(msg) | |
| [2022-10-26 23:15:10,239] torch._dynamo.variables.torch: [WARNING] Profiler will be ignored | |
| [2022-10-26 23:15:11,591] torch._dynamo.variables.torch: [WARNING] Profiler will be ignored | |
| [2022-10-26 23:15:22,586] torch._dynamo.optimizations.training: [WARNING] Unable to use Aot Autograd because of presence of mutation | |
| [2022-10-26 23:15:22,586] torch._inductor.compile_fx: [WARNING] Aot Autograd is not safe to run, so falling back to eager | |
| [2022-10-26 23:16:43,761] torch._dynamo.optimizations.training: [WARNING] Unable to use Aot Autograd because of presence of mutation | |
| [2022-10-26 23:16:43,761] torch._inductor.compile_fx: [WARNING] Aot Autograd is not safe to run, so falling back to eager | |
| [2022-10-26 23:16:43,938] torch._dynamo.variables.torch: [WARNING] Profiler will be ignored | |
| [2022-10-26 23:16:44,190] torch._dynamo.variables.torch: [WARNING] Profiler will be ignored | |
| [2022-10-26 23:16:44,437] torch._dynamo.variables.torch: [WARNING] Profiler will be ignored | |
| [2022-10-26 23:18:00,742] torch._dynamo.optimizations.training: [WARNING] Unable to use Aot Autograd because of presence of mutation | |
| [2022-10-26 23:18:00,743] torch._inductor.compile_fx: [WARNING] Aot Autograd is not safe to run, so falling back to eager |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| submitit INFO (2022-10-26 23:11:13,091) - Starting with JobEnvironment(job_id=74383, hostname=a100-st-p4d24xlarge-35, local_rank=0(8), node=0(1), global_rank=0(8)) | |
| submitit INFO (2022-10-26 23:11:13,092) - Loading pickle: /fsx/users/dberard/scratch-local/bench-fast/benchmark/logs_oct25/74383_submitted.pkl | |
| This is node 0 | |
| run_once | |
| Process group: 8 tasks, rank: 0 | |
| MY HOSTNAME: a100-st-p4d24xlarge-35 | |
| FI_PROVIDER : efa | |
| LD_LIBRARY_PATH : /fsx/users/dberard/scratch-local/bench-fast/aws-ofi-nccl/lib:/opt/amazon/efa/lib:/usr/local/cuda-11.6/lib:/usr/local/cuda-11.6/lib64:/opt/amazon/efa/lib:/opt/amazon/openmpi/lib:/usr/local/cuda/efa/lib:/usr/local/cuda/lib:/usr/local/cuda:/usr/local/cuda/lib64:/usr/local/cuda/extras/CUPTI/lib64:/usr/local/cuda/targets/x86_64-linux/lib:/usr/local/lib:/usr/lib: | |
| NCCL_DEBUG : INFO | |
| FI_EFA_USE_DEVICE_RDMA : 1 | |
| still waiting for process to exit | |
| didn't get the result, queue empty | |
| a100-st-p4d24xlarge-35:49529:49529 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to ens | |
| a100-st-p4d24xlarge-35:49529:49529 [0] NCCL INFO NCCL_SOCKET_IFNAME set to ens | |
| a100-st-p4d24xlarge-35:49529:49529 [0] NCCL INFO Bootstrap : Using ens32:10.200.70.12<0> | |
| a100-st-p4d24xlarge-35:49529:49529 [0] NCCL INFO NET/Plugin: Failed to find ncclCollNetPlugin_v6 symbol. | |
| a100-st-p4d24xlarge-35:49529:49529 [0] NCCL INFO NET/Plugin: Failed to find ncclCollNetPlugin symbol (v4 or v5). | |
| a100-st-p4d24xlarge-35:49529:49529 [0] NCCL INFO cudaDriverVersion 11060 | |
| NCCL version 2.14.3+cuda11.6 | |
| a100-st-p4d24xlarge-35:49529:50028 [0] NCCL INFO NET/OFI Using aws-ofi-nccl 1.4.0aws | |
| a100-st-p4d24xlarge-35:49529:50028 [0] NCCL INFO NET/OFI Running on p4d.24xlarge platform, Setting NCCL_TOPO_FILE environment variable to /fsx/users/dberard/scratch-local/bench-fast/aws-ofi-nccl/share/aws-ofi-nccl/xml/p4d-24xl-topo.xml | |
| a100-st-p4d24xlarge-35:49529:50028 [0] NCCL INFO NET/OFI Setting FI_EFA_FORK_SAFE environment variable to 1 | |
| a100-st-p4d24xlarge-35:49529:50028 [0] NCCL INFO NET/OFI Selected Provider is efa | |
| a100-st-p4d24xlarge-35:49529:50028 [0] NCCL INFO Using network AWS Libfabric | |
| a100-st-p4d24xlarge-35:49529:50028 [0] NCCL INFO NCCL_TOPO_FILE set by environment to /fsx/users/dberard/scratch-local/bench-fast/aws-ofi-nccl/share/aws-ofi-nccl/xml/p4d-24xl-topo.xml | |
| a100-st-p4d24xlarge-35:49529:50028 [0] NCCL INFO NET/AWS Libfabric : GPU Direct RDMA Enabled for HCA 0 'rdmap16s27' | |
| a100-st-p4d24xlarge-35:49529:50028 [0] NCCL INFO NET/AWS Libfabric : GPU Direct RDMA Enabled for HCA 1 'rdmap32s27' | |
| a100-st-p4d24xlarge-35:49529:50028 [0] NCCL INFO NET/AWS Libfabric : GPU Direct RDMA Enabled for HCA 2 'rdmap144s27' | |
| a100-st-p4d24xlarge-35:49529:50028 [0] NCCL INFO NET/AWS Libfabric : GPU Direct RDMA Enabled for HCA 3 'rdmap160s27' | |
| a100-st-p4d24xlarge-35:49529:50028 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 101c0 / HCA 0 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-35:49529:50028 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 101d0 / HCA 0 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-35:49529:50028 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 201c0 / HCA 0 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-35:49529:50028 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 201d0 / HCA 0 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-35:49529:50028 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 901c0 / HCA 0 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-35:49529:50028 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 901d0 / HCA 0 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-35:49529:50028 [0] NCCL INFO GPU Direct RDMA Enabled for GPU a01c0 / HCA 0 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-35:49529:50028 [0] NCCL INFO GPU Direct RDMA Enabled for GPU a01d0 / HCA 0 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-35:49529:50028 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 101c0 / HCA 1 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-35:49529:50028 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 101d0 / HCA 1 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-35:49529:50028 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 201c0 / HCA 1 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-35:49529:50028 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 201d0 / HCA 1 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-35:49529:50028 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 901c0 / HCA 1 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-35:49529:50028 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 901d0 / HCA 1 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-35:49529:50028 [0] NCCL INFO GPU Direct RDMA Enabled for GPU a01c0 / HCA 1 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-35:49529:50028 [0] NCCL INFO GPU Direct RDMA Enabled for GPU a01d0 / HCA 1 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-35:49529:50028 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 101c0 / HCA 2 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-35:49529:50028 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 101d0 / HCA 2 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-35:49529:50028 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 201c0 / HCA 2 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-35:49529:50028 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 201d0 / HCA 2 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-35:49529:50028 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 901c0 / HCA 2 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-35:49529:50028 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 901d0 / HCA 2 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-35:49529:50028 [0] NCCL INFO GPU Direct RDMA Enabled for GPU a01c0 / HCA 2 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-35:49529:50028 [0] NCCL INFO GPU Direct RDMA Enabled for GPU a01d0 / HCA 2 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-35:49529:50028 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 101c0 / HCA 3 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-35:49529:50028 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 101d0 / HCA 3 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-35:49529:50028 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 201c0 / HCA 3 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-35:49529:50028 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 201d0 / HCA 3 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-35:49529:50028 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 901c0 / HCA 3 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-35:49529:50028 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 901d0 / HCA 3 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-35:49529:50028 [0] NCCL INFO GPU Direct RDMA Enabled for GPU a01c0 / HCA 3 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-35:49529:50028 [0] NCCL INFO GPU Direct RDMA Enabled for GPU a01d0 / HCA 3 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-35:49529:50028 [0] NCCL INFO Setting affinity for GPU 0 to 1f0000,0000001f | |
| a100-st-p4d24xlarge-35:49529:50028 [0] NCCL INFO Channel 00/24 : 0 1 2 3 4 5 6 7 | |
| a100-st-p4d24xlarge-35:49529:50028 [0] NCCL INFO Channel 01/24 : 0 1 2 3 4 5 6 7 | |
| a100-st-p4d24xlarge-35:49529:50028 [0] NCCL INFO Channel 02/24 : 0 1 2 3 4 5 6 7 | |
| a100-st-p4d24xlarge-35:49529:50028 [0] NCCL INFO Channel 03/24 : 0 1 2 3 4 5 6 7 | |
| a100-st-p4d24xlarge-35:49529:50028 [0] NCCL INFO Channel 04/24 : 0 1 2 3 4 5 6 7 | |
| a100-st-p4d24xlarge-35:49529:50028 [0] NCCL INFO Channel 05/24 : 0 1 2 3 4 5 6 7 | |
| a100-st-p4d24xlarge-35:49529:50028 [0] NCCL INFO Channel 06/24 : 0 1 2 3 4 5 6 7 | |
| a100-st-p4d24xlarge-35:49529:50028 [0] NCCL INFO Channel 07/24 : 0 1 2 3 4 5 6 7 | |
| a100-st-p4d24xlarge-35:49529:50028 [0] NCCL INFO Channel 08/24 : 0 1 2 3 4 5 6 7 | |
| a100-st-p4d24xlarge-35:49529:50028 [0] NCCL INFO Channel 09/24 : 0 1 2 3 4 5 6 7 | |
| a100-st-p4d24xlarge-35:49529:50028 [0] NCCL INFO Channel 10/24 : 0 1 2 3 4 5 6 7 | |
| a100-st-p4d24xlarge-35:49529:50028 [0] NCCL INFO Channel 11/24 : 0 1 2 3 4 5 6 7 | |
| a100-st-p4d24xlarge-35:49529:50028 [0] NCCL INFO Channel 12/24 : 0 1 2 3 4 5 6 7 | |
| a100-st-p4d24xlarge-35:49529:50028 [0] NCCL INFO Channel 13/24 : 0 1 2 3 4 5 6 7 | |
| a100-st-p4d24xlarge-35:49529:50028 [0] NCCL INFO Channel 14/24 : 0 1 2 3 4 5 6 7 | |
| a100-st-p4d24xlarge-35:49529:50028 [0] NCCL INFO Channel 15/24 : 0 1 2 3 4 5 6 7 | |
| a100-st-p4d24xlarge-35:49529:50028 [0] NCCL INFO Channel 16/24 : 0 1 2 3 4 5 6 7 | |
| a100-st-p4d24xlarge-35:49529:50028 [0] NCCL INFO Channel 17/24 : 0 1 2 3 4 5 6 7 | |
| a100-st-p4d24xlarge-35:49529:50028 [0] NCCL INFO Channel 18/24 : 0 1 2 3 4 5 6 7 | |
| a100-st-p4d24xlarge-35:49529:50028 [0] NCCL INFO Channel 19/24 : 0 1 2 3 4 5 6 7 | |
| a100-st-p4d24xlarge-35:49529:50028 [0] NCCL INFO Channel 20/24 : 0 1 2 3 4 5 6 7 | |
| a100-st-p4d24xlarge-35:49529:50028 [0] NCCL INFO Channel 21/24 : 0 1 2 3 4 5 6 7 | |
| a100-st-p4d24xlarge-35:49529:50028 [0] NCCL INFO Channel 22/24 : 0 1 2 3 4 5 6 7 | |
| a100-st-p4d24xlarge-35:49529:50028 [0] NCCL INFO Channel 23/24 : 0 1 2 3 4 5 6 7 | |
| a100-st-p4d24xlarge-35:49529:50028 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1 [2] 1/-1/-1->0->-1 [3] 1/-1/-1->0->-1 [4] 1/-1/-1->0->-1 [5] 1/-1/-1->0->-1 [6] 1/-1/-1->0->-1 [7] 1/-1/-1->0->-1 [8] 1/-1/-1->0->-1 [9] 1/-1/-1->0->-1 [10] 1/-1/-1->0->-1 [11] 1/-1/-1->0->-1 [12] 1/-1/-1->0->-1 [13] 1/-1/-1->0->-1 [14] 1/-1/-1->0->-1 [15] 1/-1/-1->0->-1 [16] 1/-1/-1->0->-1 [17] 1/-1/-1->0->-1 [18] 1/-1/-1->0->-1 [19] 1/-1/-1->0->-1 [20] 1/-1/-1->0->-1 [21] 1/-1/-1->0->-1 [22] 1/-1/-1->0->-1 [23] 1/-1/-1->0->-1 | |
| a100-st-p4d24xlarge-35:49529:50028 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af8002dd0 | |
| a100-st-p4d24xlarge-35:49529:50077 [0] NCCL INFO New proxy recv connection 0 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:49529:50028 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af8002e10 | |
| a100-st-p4d24xlarge-35:49529:50077 [0] NCCL INFO New proxy recv connection 1 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:49529:50077 [0] NCCL INFO New proxy recv connection 2 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:49529:50028 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af8002e50 | |
| a100-st-p4d24xlarge-35:49529:50028 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af8002e90 | |
| a100-st-p4d24xlarge-35:49529:50077 [0] NCCL INFO New proxy recv connection 3 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:49529:50077 [0] NCCL INFO New proxy recv connection 4 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:49529:50028 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af8002ed0 | |
| a100-st-p4d24xlarge-35:49529:50077 [0] NCCL INFO New proxy recv connection 5 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:49529:50028 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af8002f10 | |
| a100-st-p4d24xlarge-35:49529:50077 [0] NCCL INFO New proxy recv connection 6 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:49529:50028 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af8002f50 | |
| a100-st-p4d24xlarge-35:49529:50077 [0] NCCL INFO New proxy recv connection 7 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:49529:50028 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af8002f90 | |
| a100-st-p4d24xlarge-35:49529:50028 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af8002fd0 | |
| a100-st-p4d24xlarge-35:49529:50077 [0] NCCL INFO New proxy recv connection 8 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:49529:50028 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af8003010 | |
| a100-st-p4d24xlarge-35:49529:50077 [0] NCCL INFO New proxy recv connection 9 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:49529:50028 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af8003050 | |
| a100-st-p4d24xlarge-35:49529:50077 [0] NCCL INFO New proxy recv connection 10 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:49529:50028 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af8003090 | |
| a100-st-p4d24xlarge-35:49529:50077 [0] NCCL INFO New proxy recv connection 11 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:49529:50028 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af80030d0 | |
| a100-st-p4d24xlarge-35:49529:50077 [0] NCCL INFO New proxy recv connection 12 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:49529:50028 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af8003110 | |
| a100-st-p4d24xlarge-35:49529:50077 [0] NCCL INFO New proxy recv connection 13 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:49529:50028 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af8003150 | |
| a100-st-p4d24xlarge-35:49529:50077 [0] NCCL INFO New proxy recv connection 14 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:49529:50028 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af8003190 | |
| a100-st-p4d24xlarge-35:49529:50077 [0] NCCL INFO New proxy recv connection 15 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:49529:50028 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af80031d0 | |
| a100-st-p4d24xlarge-35:49529:50077 [0] NCCL INFO New proxy recv connection 16 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:49529:50028 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af8003210 | |
| a100-st-p4d24xlarge-35:49529:50077 [0] NCCL INFO New proxy recv connection 17 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:49529:50028 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af8003250 | |
| a100-st-p4d24xlarge-35:49529:50077 [0] NCCL INFO New proxy recv connection 18 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:49529:50028 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af8003290 | |
| a100-st-p4d24xlarge-35:49529:50077 [0] NCCL INFO New proxy recv connection 19 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:49529:50028 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af80032d0 | |
| a100-st-p4d24xlarge-35:49529:50077 [0] NCCL INFO New proxy recv connection 20 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:49529:50028 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af8003310 | |
| a100-st-p4d24xlarge-35:49529:50077 [0] NCCL INFO New proxy recv connection 21 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:49529:50028 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af8003350 | |
| a100-st-p4d24xlarge-35:49529:50077 [0] NCCL INFO New proxy recv connection 22 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:49529:50028 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af8003390 | |
| a100-st-p4d24xlarge-35:49529:50077 [0] NCCL INFO New proxy recv connection 23 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:49529:50028 [0] NCCL INFO Channel 00/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-35:49529:50028 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af80033d0 | |
| a100-st-p4d24xlarge-35:49529:50077 [0] NCCL INFO New proxy send connection 24 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:49529:50028 [0] NCCL INFO Channel 01/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-35:49529:50028 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af8003410 | |
| a100-st-p4d24xlarge-35:49529:50077 [0] NCCL INFO New proxy send connection 25 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:49529:50028 [0] NCCL INFO Channel 02/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-35:49529:50028 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af8003450 | |
| a100-st-p4d24xlarge-35:49529:50077 [0] NCCL INFO New proxy send connection 26 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:49529:50028 [0] NCCL INFO Channel 03/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-35:49529:50028 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af8003490 | |
| a100-st-p4d24xlarge-35:49529:50077 [0] NCCL INFO New proxy send connection 27 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:49529:50028 [0] NCCL INFO Channel 04/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-35:49529:50028 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af80034d0 | |
| a100-st-p4d24xlarge-35:49529:50077 [0] NCCL INFO New proxy send connection 28 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:49529:50028 [0] NCCL INFO Channel 05/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-35:49529:50028 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af8003510 | |
| a100-st-p4d24xlarge-35:49529:50077 [0] NCCL INFO New proxy send connection 29 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:49529:50028 [0] NCCL INFO Channel 06/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-35:49529:50028 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af8003550 | |
| a100-st-p4d24xlarge-35:49529:50077 [0] NCCL INFO New proxy send connection 30 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:49529:50028 [0] NCCL INFO Channel 07/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-35:49529:50028 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af8003590 | |
| a100-st-p4d24xlarge-35:49529:50077 [0] NCCL INFO New proxy send connection 31 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:49529:50028 [0] NCCL INFO Channel 08/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-35:49529:50028 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af80035d0 | |
| a100-st-p4d24xlarge-35:49529:50077 [0] NCCL INFO New proxy send connection 32 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:49529:50028 [0] NCCL INFO Channel 09/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-35:49529:50028 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af8003610 | |
| a100-st-p4d24xlarge-35:49529:50077 [0] NCCL INFO New proxy send connection 33 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:49529:50028 [0] NCCL INFO Channel 10/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-35:49529:50028 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af8003650 | |
| a100-st-p4d24xlarge-35:49529:50077 [0] NCCL INFO New proxy send connection 34 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:49529:50028 [0] NCCL INFO Channel 11/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-35:49529:50028 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af8003690 | |
| a100-st-p4d24xlarge-35:49529:50077 [0] NCCL INFO New proxy send connection 35 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:49529:50028 [0] NCCL INFO Channel 12/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-35:49529:50028 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af80036d0 | |
| a100-st-p4d24xlarge-35:49529:50077 [0] NCCL INFO New proxy send connection 36 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:49529:50028 [0] NCCL INFO Channel 13/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-35:49529:50028 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af8003710 | |
| a100-st-p4d24xlarge-35:49529:50077 [0] NCCL INFO New proxy send connection 37 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:49529:50028 [0] NCCL INFO Channel 14/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-35:49529:50028 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af8003750 | |
| a100-st-p4d24xlarge-35:49529:50077 [0] NCCL INFO New proxy send connection 38 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:49529:50028 [0] NCCL INFO Channel 15/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-35:49529:50028 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af8003790 | |
| a100-st-p4d24xlarge-35:49529:50077 [0] NCCL INFO New proxy send connection 39 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:49529:50028 [0] NCCL INFO Channel 16/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-35:49529:50028 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af80037d0 | |
| a100-st-p4d24xlarge-35:49529:50077 [0] NCCL INFO New proxy send connection 40 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:49529:50028 [0] NCCL INFO Channel 17/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-35:49529:50028 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af8003810 | |
| a100-st-p4d24xlarge-35:49529:50077 [0] NCCL INFO New proxy send connection 41 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:49529:50028 [0] NCCL INFO Channel 18/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-35:49529:50028 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af8003850 | |
| a100-st-p4d24xlarge-35:49529:50077 [0] NCCL INFO New proxy send connection 42 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:49529:50028 [0] NCCL INFO Channel 19/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-35:49529:50028 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af8003890 | |
| a100-st-p4d24xlarge-35:49529:50077 [0] NCCL INFO New proxy send connection 43 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:49529:50028 [0] NCCL INFO Channel 20/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-35:49529:50028 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af80038d0 | |
| a100-st-p4d24xlarge-35:49529:50077 [0] NCCL INFO New proxy send connection 44 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:49529:50028 [0] NCCL INFO Channel 21/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-35:49529:50028 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af8003910 | |
| a100-st-p4d24xlarge-35:49529:50077 [0] NCCL INFO New proxy send connection 45 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:49529:50028 [0] NCCL INFO Channel 22/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-35:49529:50028 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af8003950 | |
| a100-st-p4d24xlarge-35:49529:50077 [0] NCCL INFO New proxy send connection 46 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:49529:50028 [0] NCCL INFO Channel 23/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-35:49529:50028 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af8003990 | |
| a100-st-p4d24xlarge-35:49529:50077 [0] NCCL INFO New proxy send connection 47 from local rank 0, transport 0 | |
| still waiting for process to exit | |
| a100-st-p4d24xlarge-35:49529:50028 [0] NCCL INFO Connected all rings | |
| a100-st-p4d24xlarge-35:49529:50028 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af80039d0 | |
| a100-st-p4d24xlarge-35:49529:50077 [0] NCCL INFO New proxy recv connection 48 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:49529:50077 [0] NCCL INFO New proxy recv connection 49 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:49529:50028 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af8003a10 | |
| a100-st-p4d24xlarge-35:49529:50028 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af8003a50 | |
| a100-st-p4d24xlarge-35:49529:50077 [0] NCCL INFO New proxy recv connection 50 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:49529:50028 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af8003a90 | |
| a100-st-p4d24xlarge-35:49529:50077 [0] NCCL INFO New proxy recv connection 51 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:49529:50028 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af8003ad0 | |
| a100-st-p4d24xlarge-35:49529:50077 [0] NCCL INFO New proxy recv connection 52 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:49529:50028 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af8003b10 | |
| a100-st-p4d24xlarge-35:49529:50077 [0] NCCL INFO New proxy recv connection 53 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:49529:50028 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af8003b50 | |
| a100-st-p4d24xlarge-35:49529:50077 [0] NCCL INFO New proxy recv connection 54 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:49529:50028 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af8003b90 | |
| a100-st-p4d24xlarge-35:49529:50077 [0] NCCL INFO New proxy recv connection 55 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:49529:50028 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af8003bd0 | |
| a100-st-p4d24xlarge-35:49529:50077 [0] NCCL INFO New proxy recv connection 56 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:49529:50028 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af8003c10 | |
| a100-st-p4d24xlarge-35:49529:50077 [0] NCCL INFO New proxy recv connection 57 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:49529:50028 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af8003c50 | |
| a100-st-p4d24xlarge-35:49529:50077 [0] NCCL INFO New proxy recv connection 58 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:49529:50028 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af8003c90 | |
| a100-st-p4d24xlarge-35:49529:50077 [0] NCCL INFO New proxy recv connection 59 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:49529:50028 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af8003cd0 | |
| a100-st-p4d24xlarge-35:49529:50077 [0] NCCL INFO New proxy recv connection 60 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:49529:50028 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af8003d10 | |
| a100-st-p4d24xlarge-35:49529:50077 [0] NCCL INFO New proxy recv connection 61 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:49529:50028 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af8003d50 | |
| a100-st-p4d24xlarge-35:49529:50077 [0] NCCL INFO New proxy recv connection 62 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:49529:50028 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af8003d90 | |
| a100-st-p4d24xlarge-35:49529:50077 [0] NCCL INFO New proxy recv connection 63 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:49529:50028 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af8003dd0 | |
| a100-st-p4d24xlarge-35:49529:50077 [0] NCCL INFO New proxy recv connection 64 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:49529:50028 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af8003e10 | |
| a100-st-p4d24xlarge-35:49529:50077 [0] NCCL INFO New proxy recv connection 65 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:49529:50028 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af8003e50 | |
| a100-st-p4d24xlarge-35:49529:50077 [0] NCCL INFO New proxy recv connection 66 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:49529:50028 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af8003e90 | |
| a100-st-p4d24xlarge-35:49529:50077 [0] NCCL INFO New proxy recv connection 67 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:49529:50028 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af8003ed0 | |
| a100-st-p4d24xlarge-35:49529:50077 [0] NCCL INFO New proxy recv connection 68 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:49529:50028 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af8003f10 | |
| a100-st-p4d24xlarge-35:49529:50077 [0] NCCL INFO New proxy recv connection 69 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:49529:50028 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af8003f50 | |
| a100-st-p4d24xlarge-35:49529:50077 [0] NCCL INFO New proxy recv connection 70 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:49529:50028 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af8003f90 | |
| a100-st-p4d24xlarge-35:49529:50077 [0] NCCL INFO New proxy recv connection 71 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:49529:50028 [0] NCCL INFO Connected all trees | |
| a100-st-p4d24xlarge-35:49529:50028 [0] NCCL INFO NCCL_ALGO set by environment to ring | |
| a100-st-p4d24xlarge-35:49529:50028 [0] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512 | |
| a100-st-p4d24xlarge-35:49529:50028 [0] NCCL INFO 24 coll channels, 32 p2p channels, 32 p2p channels per peer | |
| a100-st-p4d24xlarge-35:49529:50077 [0] NCCL INFO New proxy send connection 72 from local rank 0, transport 2 | |
| a100-st-p4d24xlarge-35:49529:50028 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af8003fd0 | |
| a100-st-p4d24xlarge-35:49529:50028 [0] NCCL INFO comm 0x5562b12335c0 rank 0 nranks 8 cudaDev 0 busId 101c0 - Init COMPLETE | |
| result {'latency_median': 63.02003288269043, 'latency_stdev': 0.2317810499888692} | |
| has been placed into the queue | |
| got the result! | |
| got exit 0 | |
| (already got results) | |
| exit code: 0 and result: {'nodes': 1, 'model_name': 'torchbenchmark.models.resnet50.Model', 'backend': 'eager', 'has_breaks': False, 'static_graph': False, 'result': {'latency_median': 63.02003288269043, 'latency_stdev': 0.2317810499888692}} | |
| <RESULT>{"nodes": 1, "model_name": "torchbenchmark.models.resnet50.Model", "backend": "eager", "has_breaks": false, "static_graph": false, "result": {"latency_median": 63.02003288269043, "latency_stdev": 0.2317810499888692}}</RESULT> | |
| run_once | |
| Process group: 8 tasks, rank: 0 | |
| MY HOSTNAME: a100-st-p4d24xlarge-35 | |
| FI_PROVIDER : efa | |
| LD_LIBRARY_PATH : /fsx/users/dberard/scratch-local/bench-fast/aws-ofi-nccl/lib:/opt/amazon/efa/lib:/usr/local/cuda-11.6/lib:/usr/local/cuda-11.6/lib64:/opt/amazon/efa/lib:/opt/amazon/openmpi/lib:/usr/local/cuda/efa/lib:/usr/local/cuda/lib:/usr/local/cuda:/usr/local/cuda/lib64:/usr/local/cuda/extras/CUPTI/lib64:/usr/local/cuda/targets/x86_64-linux/lib:/usr/local/lib:/usr/lib: | |
| NCCL_DEBUG : INFO | |
| FI_EFA_USE_DEVICE_RDMA : 1 | |
| still waiting for process to exit | |
| didn't get the result, queue empty | |
| a100-st-p4d24xlarge-35:50204:50204 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to ens | |
| a100-st-p4d24xlarge-35:50204:50204 [0] NCCL INFO NCCL_SOCKET_IFNAME set to ens | |
| a100-st-p4d24xlarge-35:50204:50204 [0] NCCL INFO Bootstrap : Using ens32:10.200.70.12<0> | |
| a100-st-p4d24xlarge-35:50204:50204 [0] NCCL INFO NET/Plugin: Failed to find ncclCollNetPlugin_v6 symbol. | |
| a100-st-p4d24xlarge-35:50204:50204 [0] NCCL INFO NET/Plugin: Failed to find ncclCollNetPlugin symbol (v4 or v5). | |
| a100-st-p4d24xlarge-35:50204:50204 [0] NCCL INFO cudaDriverVersion 11060 | |
| NCCL version 2.14.3+cuda11.6 | |
| a100-st-p4d24xlarge-35:50204:51295 [0] NCCL INFO NET/OFI Using aws-ofi-nccl 1.4.0aws | |
| a100-st-p4d24xlarge-35:50204:51295 [0] NCCL INFO NET/OFI Running on p4d.24xlarge platform, Setting NCCL_TOPO_FILE environment variable to /fsx/users/dberard/scratch-local/bench-fast/aws-ofi-nccl/share/aws-ofi-nccl/xml/p4d-24xl-topo.xml | |
| a100-st-p4d24xlarge-35:50204:51295 [0] NCCL INFO NET/OFI Setting FI_EFA_FORK_SAFE environment variable to 1 | |
| a100-st-p4d24xlarge-35:50204:51295 [0] NCCL INFO NET/OFI Selected Provider is efa | |
| a100-st-p4d24xlarge-35:50204:51295 [0] NCCL INFO Using network AWS Libfabric | |
| a100-st-p4d24xlarge-35:50204:51295 [0] NCCL INFO NCCL_TOPO_FILE set by environment to /fsx/users/dberard/scratch-local/bench-fast/aws-ofi-nccl/share/aws-ofi-nccl/xml/p4d-24xl-topo.xml | |
| still waiting for process to exit | |
| a100-st-p4d24xlarge-35:50204:51295 [0] NCCL INFO NET/AWS Libfabric : GPU Direct RDMA Enabled for HCA 0 'rdmap16s27' | |
| a100-st-p4d24xlarge-35:50204:51295 [0] NCCL INFO NET/AWS Libfabric : GPU Direct RDMA Enabled for HCA 1 'rdmap32s27' | |
| a100-st-p4d24xlarge-35:50204:51295 [0] NCCL INFO NET/AWS Libfabric : GPU Direct RDMA Enabled for HCA 2 'rdmap144s27' | |
| a100-st-p4d24xlarge-35:50204:51295 [0] NCCL INFO NET/AWS Libfabric : GPU Direct RDMA Enabled for HCA 3 'rdmap160s27' | |
| a100-st-p4d24xlarge-35:50204:51295 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 101c0 / HCA 0 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-35:50204:51295 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 101d0 / HCA 0 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-35:50204:51295 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 201c0 / HCA 0 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-35:50204:51295 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 201d0 / HCA 0 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-35:50204:51295 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 901c0 / HCA 0 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-35:50204:51295 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 901d0 / HCA 0 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-35:50204:51295 [0] NCCL INFO GPU Direct RDMA Enabled for GPU a01c0 / HCA 0 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-35:50204:51295 [0] NCCL INFO GPU Direct RDMA Enabled for GPU a01d0 / HCA 0 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-35:50204:51295 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 101c0 / HCA 1 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-35:50204:51295 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 101d0 / HCA 1 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-35:50204:51295 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 201c0 / HCA 1 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-35:50204:51295 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 201d0 / HCA 1 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-35:50204:51295 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 901c0 / HCA 1 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-35:50204:51295 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 901d0 / HCA 1 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-35:50204:51295 [0] NCCL INFO GPU Direct RDMA Enabled for GPU a01c0 / HCA 1 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-35:50204:51295 [0] NCCL INFO GPU Direct RDMA Enabled for GPU a01d0 / HCA 1 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-35:50204:51295 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 101c0 / HCA 2 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-35:50204:51295 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 101d0 / HCA 2 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-35:50204:51295 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 201c0 / HCA 2 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-35:50204:51295 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 201d0 / HCA 2 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-35:50204:51295 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 901c0 / HCA 2 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-35:50204:51295 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 901d0 / HCA 2 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-35:50204:51295 [0] NCCL INFO GPU Direct RDMA Enabled for GPU a01c0 / HCA 2 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-35:50204:51295 [0] NCCL INFO GPU Direct RDMA Enabled for GPU a01d0 / HCA 2 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-35:50204:51295 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 101c0 / HCA 3 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-35:50204:51295 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 101d0 / HCA 3 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-35:50204:51295 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 201c0 / HCA 3 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-35:50204:51295 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 201d0 / HCA 3 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-35:50204:51295 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 901c0 / HCA 3 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-35:50204:51295 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 901d0 / HCA 3 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-35:50204:51295 [0] NCCL INFO GPU Direct RDMA Enabled for GPU a01c0 / HCA 3 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-35:50204:51295 [0] NCCL INFO GPU Direct RDMA Enabled for GPU a01d0 / HCA 3 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-35:50204:51295 [0] NCCL INFO Setting affinity for GPU 0 to 1f0000,0000001f | |
| a100-st-p4d24xlarge-35:50204:51295 [0] NCCL INFO Channel 00/24 : 0 1 2 3 4 5 6 7 | |
| a100-st-p4d24xlarge-35:50204:51295 [0] NCCL INFO Channel 01/24 : 0 1 2 3 4 5 6 7 | |
| a100-st-p4d24xlarge-35:50204:51295 [0] NCCL INFO Channel 02/24 : 0 1 2 3 4 5 6 7 | |
| a100-st-p4d24xlarge-35:50204:51295 [0] NCCL INFO Channel 03/24 : 0 1 2 3 4 5 6 7 | |
| a100-st-p4d24xlarge-35:50204:51295 [0] NCCL INFO Channel 04/24 : 0 1 2 3 4 5 6 7 | |
| a100-st-p4d24xlarge-35:50204:51295 [0] NCCL INFO Channel 05/24 : 0 1 2 3 4 5 6 7 | |
| a100-st-p4d24xlarge-35:50204:51295 [0] NCCL INFO Channel 06/24 : 0 1 2 3 4 5 6 7 | |
| a100-st-p4d24xlarge-35:50204:51295 [0] NCCL INFO Channel 07/24 : 0 1 2 3 4 5 6 7 | |
| a100-st-p4d24xlarge-35:50204:51295 [0] NCCL INFO Channel 08/24 : 0 1 2 3 4 5 6 7 | |
| a100-st-p4d24xlarge-35:50204:51295 [0] NCCL INFO Channel 09/24 : 0 1 2 3 4 5 6 7 | |
| a100-st-p4d24xlarge-35:50204:51295 [0] NCCL INFO Channel 10/24 : 0 1 2 3 4 5 6 7 | |
| a100-st-p4d24xlarge-35:50204:51295 [0] NCCL INFO Channel 11/24 : 0 1 2 3 4 5 6 7 | |
| a100-st-p4d24xlarge-35:50204:51295 [0] NCCL INFO Channel 12/24 : 0 1 2 3 4 5 6 7 | |
| a100-st-p4d24xlarge-35:50204:51295 [0] NCCL INFO Channel 13/24 : 0 1 2 3 4 5 6 7 | |
| a100-st-p4d24xlarge-35:50204:51295 [0] NCCL INFO Channel 14/24 : 0 1 2 3 4 5 6 7 | |
| a100-st-p4d24xlarge-35:50204:51295 [0] NCCL INFO Channel 15/24 : 0 1 2 3 4 5 6 7 | |
| a100-st-p4d24xlarge-35:50204:51295 [0] NCCL INFO Channel 16/24 : 0 1 2 3 4 5 6 7 | |
| a100-st-p4d24xlarge-35:50204:51295 [0] NCCL INFO Channel 17/24 : 0 1 2 3 4 5 6 7 | |
| a100-st-p4d24xlarge-35:50204:51295 [0] NCCL INFO Channel 18/24 : 0 1 2 3 4 5 6 7 | |
| a100-st-p4d24xlarge-35:50204:51295 [0] NCCL INFO Channel 19/24 : 0 1 2 3 4 5 6 7 | |
| a100-st-p4d24xlarge-35:50204:51295 [0] NCCL INFO Channel 20/24 : 0 1 2 3 4 5 6 7 | |
| a100-st-p4d24xlarge-35:50204:51295 [0] NCCL INFO Channel 21/24 : 0 1 2 3 4 5 6 7 | |
| a100-st-p4d24xlarge-35:50204:51295 [0] NCCL INFO Channel 22/24 : 0 1 2 3 4 5 6 7 | |
| a100-st-p4d24xlarge-35:50204:51295 [0] NCCL INFO Channel 23/24 : 0 1 2 3 4 5 6 7 | |
| a100-st-p4d24xlarge-35:50204:51295 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1 [2] 1/-1/-1->0->-1 [3] 1/-1/-1->0->-1 [4] 1/-1/-1->0->-1 [5] 1/-1/-1->0->-1 [6] 1/-1/-1->0->-1 [7] 1/-1/-1->0->-1 [8] 1/-1/-1->0->-1 [9] 1/-1/-1->0->-1 [10] 1/-1/-1->0->-1 [11] 1/-1/-1->0->-1 [12] 1/-1/-1->0->-1 [13] 1/-1/-1->0->-1 [14] 1/-1/-1->0->-1 [15] 1/-1/-1->0->-1 [16] 1/-1/-1->0->-1 [17] 1/-1/-1->0->-1 [18] 1/-1/-1->0->-1 [19] 1/-1/-1->0->-1 [20] 1/-1/-1->0->-1 [21] 1/-1/-1->0->-1 [22] 1/-1/-1->0->-1 [23] 1/-1/-1->0->-1 | |
| a100-st-p4d24xlarge-35:50204:51295 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af0002dd0 | |
| a100-st-p4d24xlarge-35:50204:51338 [0] NCCL INFO New proxy recv connection 0 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:50204:51295 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af0002e10 | |
| a100-st-p4d24xlarge-35:50204:51338 [0] NCCL INFO New proxy recv connection 1 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:50204:51338 [0] NCCL INFO New proxy recv connection 2 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:50204:51295 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af0002e50 | |
| a100-st-p4d24xlarge-35:50204:51338 [0] NCCL INFO New proxy recv connection 3 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:50204:51295 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af0002e90 | |
| a100-st-p4d24xlarge-35:50204:51338 [0] NCCL INFO New proxy recv connection 4 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:50204:51295 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af0002ed0 | |
| a100-st-p4d24xlarge-35:50204:51338 [0] NCCL INFO New proxy recv connection 5 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:50204:51295 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af0002f10 | |
| a100-st-p4d24xlarge-35:50204:51338 [0] NCCL INFO New proxy recv connection 6 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:50204:51295 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af0002f50 | |
| a100-st-p4d24xlarge-35:50204:51338 [0] NCCL INFO New proxy recv connection 7 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:50204:51295 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af0002f90 | |
| a100-st-p4d24xlarge-35:50204:51295 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af0002fd0 | |
| a100-st-p4d24xlarge-35:50204:51338 [0] NCCL INFO New proxy recv connection 8 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:50204:51295 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af0003010 | |
| a100-st-p4d24xlarge-35:50204:51338 [0] NCCL INFO New proxy recv connection 9 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:50204:51295 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af0003050 | |
| a100-st-p4d24xlarge-35:50204:51338 [0] NCCL INFO New proxy recv connection 10 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:50204:51295 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af0003090 | |
| a100-st-p4d24xlarge-35:50204:51338 [0] NCCL INFO New proxy recv connection 11 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:50204:51295 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af00030d0 | |
| a100-st-p4d24xlarge-35:50204:51338 [0] NCCL INFO New proxy recv connection 12 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:50204:51295 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af0003110 | |
| a100-st-p4d24xlarge-35:50204:51338 [0] NCCL INFO New proxy recv connection 13 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:50204:51295 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af0003150 | |
| a100-st-p4d24xlarge-35:50204:51338 [0] NCCL INFO New proxy recv connection 14 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:50204:51295 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af0003190 | |
| a100-st-p4d24xlarge-35:50204:51338 [0] NCCL INFO New proxy recv connection 15 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:50204:51295 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af00031d0 | |
| a100-st-p4d24xlarge-35:50204:51338 [0] NCCL INFO New proxy recv connection 16 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:50204:51295 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af0003210 | |
| a100-st-p4d24xlarge-35:50204:51338 [0] NCCL INFO New proxy recv connection 17 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:50204:51295 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af0003250 | |
| a100-st-p4d24xlarge-35:50204:51338 [0] NCCL INFO New proxy recv connection 18 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:50204:51295 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af0003290 | |
| a100-st-p4d24xlarge-35:50204:51338 [0] NCCL INFO New proxy recv connection 19 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:50204:51295 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af00032d0 | |
| a100-st-p4d24xlarge-35:50204:51338 [0] NCCL INFO New proxy recv connection 20 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:50204:51295 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af0003310 | |
| a100-st-p4d24xlarge-35:50204:51338 [0] NCCL INFO New proxy recv connection 21 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:50204:51295 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af0003350 | |
| a100-st-p4d24xlarge-35:50204:51338 [0] NCCL INFO New proxy recv connection 22 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:50204:51295 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af0003390 | |
| a100-st-p4d24xlarge-35:50204:51338 [0] NCCL INFO New proxy recv connection 23 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:50204:51295 [0] NCCL INFO Channel 00/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-35:50204:51295 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af00033d0 | |
| a100-st-p4d24xlarge-35:50204:51338 [0] NCCL INFO New proxy send connection 24 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:50204:51295 [0] NCCL INFO Channel 01/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-35:50204:51295 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af0003410 | |
| a100-st-p4d24xlarge-35:50204:51338 [0] NCCL INFO New proxy send connection 25 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:50204:51295 [0] NCCL INFO Channel 02/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-35:50204:51295 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af0003450 | |
| a100-st-p4d24xlarge-35:50204:51338 [0] NCCL INFO New proxy send connection 26 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:50204:51295 [0] NCCL INFO Channel 03/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-35:50204:51295 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af0003490 | |
| a100-st-p4d24xlarge-35:50204:51338 [0] NCCL INFO New proxy send connection 27 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:50204:51295 [0] NCCL INFO Channel 04/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-35:50204:51295 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af00034d0 | |
| a100-st-p4d24xlarge-35:50204:51338 [0] NCCL INFO New proxy send connection 28 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:50204:51295 [0] NCCL INFO Channel 05/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-35:50204:51295 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af0003510 | |
| a100-st-p4d24xlarge-35:50204:51338 [0] NCCL INFO New proxy send connection 29 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:50204:51295 [0] NCCL INFO Channel 06/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-35:50204:51295 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af0003550 | |
| a100-st-p4d24xlarge-35:50204:51338 [0] NCCL INFO New proxy send connection 30 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:50204:51295 [0] NCCL INFO Channel 07/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-35:50204:51295 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af0003590 | |
| a100-st-p4d24xlarge-35:50204:51338 [0] NCCL INFO New proxy send connection 31 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:50204:51295 [0] NCCL INFO Channel 08/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-35:50204:51295 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af00035d0 | |
| a100-st-p4d24xlarge-35:50204:51338 [0] NCCL INFO New proxy send connection 32 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:50204:51295 [0] NCCL INFO Channel 09/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-35:50204:51295 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af0003610 | |
| a100-st-p4d24xlarge-35:50204:51338 [0] NCCL INFO New proxy send connection 33 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:50204:51295 [0] NCCL INFO Channel 10/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-35:50204:51295 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af0003650 | |
| a100-st-p4d24xlarge-35:50204:51338 [0] NCCL INFO New proxy send connection 34 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:50204:51295 [0] NCCL INFO Channel 11/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-35:50204:51295 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af0003690 | |
| a100-st-p4d24xlarge-35:50204:51338 [0] NCCL INFO New proxy send connection 35 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:50204:51295 [0] NCCL INFO Channel 12/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-35:50204:51295 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af00036d0 | |
| a100-st-p4d24xlarge-35:50204:51338 [0] NCCL INFO New proxy send connection 36 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:50204:51295 [0] NCCL INFO Channel 13/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-35:50204:51295 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af0003710 | |
| a100-st-p4d24xlarge-35:50204:51338 [0] NCCL INFO New proxy send connection 37 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:50204:51295 [0] NCCL INFO Channel 14/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-35:50204:51295 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af0003750 | |
| a100-st-p4d24xlarge-35:50204:51338 [0] NCCL INFO New proxy send connection 38 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:50204:51295 [0] NCCL INFO Channel 15/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-35:50204:51295 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af0003790 | |
| a100-st-p4d24xlarge-35:50204:51338 [0] NCCL INFO New proxy send connection 39 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:50204:51295 [0] NCCL INFO Channel 16/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-35:50204:51295 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af00037d0 | |
| a100-st-p4d24xlarge-35:50204:51338 [0] NCCL INFO New proxy send connection 40 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:50204:51295 [0] NCCL INFO Channel 17/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-35:50204:51295 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af0003810 | |
| a100-st-p4d24xlarge-35:50204:51338 [0] NCCL INFO New proxy send connection 41 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:50204:51295 [0] NCCL INFO Channel 18/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-35:50204:51295 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af0003850 | |
| a100-st-p4d24xlarge-35:50204:51338 [0] NCCL INFO New proxy send connection 42 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:50204:51295 [0] NCCL INFO Channel 19/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-35:50204:51295 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af0003890 | |
| a100-st-p4d24xlarge-35:50204:51338 [0] NCCL INFO New proxy send connection 43 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:50204:51295 [0] NCCL INFO Channel 20/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-35:50204:51295 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af00038d0 | |
| a100-st-p4d24xlarge-35:50204:51338 [0] NCCL INFO New proxy send connection 44 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:50204:51295 [0] NCCL INFO Channel 21/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-35:50204:51295 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af0003910 | |
| a100-st-p4d24xlarge-35:50204:51338 [0] NCCL INFO New proxy send connection 45 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:50204:51295 [0] NCCL INFO Channel 22/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-35:50204:51295 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af0003950 | |
| a100-st-p4d24xlarge-35:50204:51338 [0] NCCL INFO New proxy send connection 46 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:50204:51295 [0] NCCL INFO Channel 23/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-35:50204:51295 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af0003990 | |
| a100-st-p4d24xlarge-35:50204:51338 [0] NCCL INFO New proxy send connection 47 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:50204:51295 [0] NCCL INFO Connected all rings | |
| a100-st-p4d24xlarge-35:50204:51295 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af00039d0 | |
| a100-st-p4d24xlarge-35:50204:51338 [0] NCCL INFO New proxy recv connection 48 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:50204:51338 [0] NCCL INFO New proxy recv connection 49 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:50204:51295 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af0003a10 | |
| a100-st-p4d24xlarge-35:50204:51338 [0] NCCL INFO New proxy recv connection 50 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:50204:51295 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af0003a50 | |
| a100-st-p4d24xlarge-35:50204:51295 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af0003a90 | |
| a100-st-p4d24xlarge-35:50204:51338 [0] NCCL INFO New proxy recv connection 51 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:50204:51295 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af0003ad0 | |
| a100-st-p4d24xlarge-35:50204:51338 [0] NCCL INFO New proxy recv connection 52 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:50204:51295 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af0003b10 | |
| a100-st-p4d24xlarge-35:50204:51338 [0] NCCL INFO New proxy recv connection 53 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:50204:51295 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af0003b50 | |
| a100-st-p4d24xlarge-35:50204:51338 [0] NCCL INFO New proxy recv connection 54 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:50204:51295 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af0003b90 | |
| a100-st-p4d24xlarge-35:50204:51338 [0] NCCL INFO New proxy recv connection 55 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:50204:51295 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af0003bd0 | |
| a100-st-p4d24xlarge-35:50204:51338 [0] NCCL INFO New proxy recv connection 56 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:50204:51295 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af0003c10 | |
| a100-st-p4d24xlarge-35:50204:51338 [0] NCCL INFO New proxy recv connection 57 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:50204:51295 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af0003c50 | |
| a100-st-p4d24xlarge-35:50204:51338 [0] NCCL INFO New proxy recv connection 58 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:50204:51295 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af0003c90 | |
| a100-st-p4d24xlarge-35:50204:51338 [0] NCCL INFO New proxy recv connection 59 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:50204:51295 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af0003cd0 | |
| a100-st-p4d24xlarge-35:50204:51338 [0] NCCL INFO New proxy recv connection 60 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:50204:51295 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af0003d10 | |
| a100-st-p4d24xlarge-35:50204:51338 [0] NCCL INFO New proxy recv connection 61 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:50204:51295 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af0003d50 | |
| a100-st-p4d24xlarge-35:50204:51338 [0] NCCL INFO New proxy recv connection 62 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:50204:51295 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af0003d90 | |
| a100-st-p4d24xlarge-35:50204:51338 [0] NCCL INFO New proxy recv connection 63 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:50204:51295 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af0003dd0 | |
| a100-st-p4d24xlarge-35:50204:51338 [0] NCCL INFO New proxy recv connection 64 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:50204:51295 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af0003e10 | |
| a100-st-p4d24xlarge-35:50204:51338 [0] NCCL INFO New proxy recv connection 65 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:50204:51295 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af0003e50 | |
| a100-st-p4d24xlarge-35:50204:51338 [0] NCCL INFO New proxy recv connection 66 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:50204:51295 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af0003e90 | |
| a100-st-p4d24xlarge-35:50204:51338 [0] NCCL INFO New proxy recv connection 67 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:50204:51295 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af0003ed0 | |
| a100-st-p4d24xlarge-35:50204:51338 [0] NCCL INFO New proxy recv connection 68 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:50204:51295 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af0003f10 | |
| a100-st-p4d24xlarge-35:50204:51338 [0] NCCL INFO New proxy recv connection 69 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:50204:51295 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af0003f50 | |
| a100-st-p4d24xlarge-35:50204:51338 [0] NCCL INFO New proxy recv connection 70 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:50204:51295 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af0003f90 | |
| a100-st-p4d24xlarge-35:50204:51338 [0] NCCL INFO New proxy recv connection 71 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:50204:51295 [0] NCCL INFO Connected all trees | |
| a100-st-p4d24xlarge-35:50204:51295 [0] NCCL INFO NCCL_ALGO set by environment to ring | |
| a100-st-p4d24xlarge-35:50204:51295 [0] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512 | |
| a100-st-p4d24xlarge-35:50204:51295 [0] NCCL INFO 24 coll channels, 32 p2p channels, 32 p2p channels per peer | |
| a100-st-p4d24xlarge-35:50204:51338 [0] NCCL INFO New proxy send connection 72 from local rank 0, transport 2 | |
| a100-st-p4d24xlarge-35:50204:51295 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af0003fd0 | |
| a100-st-p4d24xlarge-35:50204:51295 [0] NCCL INFO comm 0x5562b16bb6f0 rank 0 nranks 8 cudaDev 0 busId 101c0 - Init COMPLETE | |
| didn't get the result, queue empty | |
| still waiting for process to exit | |
| didn't get the result, queue empty | |
| still waiting for process to exit | |
| didn't get the result, queue empty | |
| still waiting for process to exit | |
| didn't get the result, queue empty | |
| still waiting for process to exit | |
| didn't get the result, queue empty | |
| still waiting for process to exit | |
| didn't get the result, queue empty | |
| still waiting for process to exit | |
| didn't get the result, queue empty | |
| still waiting for process to exit | |
| didn't get the result, queue empty | |
| still waiting for process to exit | |
| didn't get the result, queue empty | |
| still waiting for process to exit | |
| didn't get the result, queue empty | |
| still waiting for process to exit | |
| didn't get the result, queue empty | |
| still waiting for process to exit | |
| didn't get the result, queue empty | |
| still waiting for process to exit | |
| didn't get the result, queue empty | |
| still waiting for process to exit | |
| didn't get the result, queue empty | |
| still waiting for process to exit | |
| didn't get the result, queue empty | |
| still waiting for process to exit | |
| didn't get the result, queue empty | |
| still waiting for process to exit | |
| didn't get the result, queue empty | |
| still waiting for process to exit | |
| didn't get the result, queue empty | |
| still waiting for process to exit | |
| result {'latency_median': 73.09516906738281, 'latency_stdev': 0.6081925671589652} | |
| has been placed into the queue | |
| got the result! | |
| got exit 0 | |
| (already got results) | |
| exit code: 0 and result: {'nodes': 1, 'model_name': 'torchbenchmark.models.resnet50.Model', 'backend': 'torchdynamo_inductor', 'has_breaks': False, 'static_graph': False, 'result': {'latency_median': 73.09516906738281, 'latency_stdev': 0.6081925671589652}} | |
| <RESULT>{"nodes": 1, "model_name": "torchbenchmark.models.resnet50.Model", "backend": "torchdynamo_inductor", "has_breaks": false, "static_graph": false, "result": {"latency_median": 73.09516906738281, "latency_stdev": 0.6081925671589652}}</RESULT> | |
| run_once | |
| Process group: 8 tasks, rank: 0 | |
| MY HOSTNAME: a100-st-p4d24xlarge-35 | |
| FI_PROVIDER : efa | |
| LD_LIBRARY_PATH : /fsx/users/dberard/scratch-local/bench-fast/aws-ofi-nccl/lib:/opt/amazon/efa/lib:/usr/local/cuda-11.6/lib:/usr/local/cuda-11.6/lib64:/opt/amazon/efa/lib:/opt/amazon/openmpi/lib:/usr/local/cuda/efa/lib:/usr/local/cuda/lib:/usr/local/cuda:/usr/local/cuda/lib64:/usr/local/cuda/extras/CUPTI/lib64:/usr/local/cuda/targets/x86_64-linux/lib:/usr/local/lib:/usr/lib: | |
| NCCL_DEBUG : INFO | |
| FI_EFA_USE_DEVICE_RDMA : 1 | |
| still waiting for process to exit | |
| didn't get the result, queue empty | |
| a100-st-p4d24xlarge-35:54314:54314 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to ens | |
| a100-st-p4d24xlarge-35:54314:54314 [0] NCCL INFO NCCL_SOCKET_IFNAME set to ens | |
| a100-st-p4d24xlarge-35:54314:54314 [0] NCCL INFO Bootstrap : Using ens32:10.200.70.12<0> | |
| a100-st-p4d24xlarge-35:54314:54314 [0] NCCL INFO NET/Plugin: Failed to find ncclCollNetPlugin_v6 symbol. | |
| a100-st-p4d24xlarge-35:54314:54314 [0] NCCL INFO NET/Plugin: Failed to find ncclCollNetPlugin symbol (v4 or v5). | |
| a100-st-p4d24xlarge-35:54314:54314 [0] NCCL INFO cudaDriverVersion 11060 | |
| NCCL version 2.14.3+cuda11.6 | |
| a100-st-p4d24xlarge-35:54314:55382 [0] NCCL INFO NET/OFI Using aws-ofi-nccl 1.4.0aws | |
| a100-st-p4d24xlarge-35:54314:55382 [0] NCCL INFO NET/OFI Running on p4d.24xlarge platform, Setting NCCL_TOPO_FILE environment variable to /fsx/users/dberard/scratch-local/bench-fast/aws-ofi-nccl/share/aws-ofi-nccl/xml/p4d-24xl-topo.xml | |
| a100-st-p4d24xlarge-35:54314:55382 [0] NCCL INFO NET/OFI Setting FI_EFA_FORK_SAFE environment variable to 1 | |
| a100-st-p4d24xlarge-35:54314:55382 [0] NCCL INFO NET/OFI Selected Provider is efa | |
| a100-st-p4d24xlarge-35:54314:55382 [0] NCCL INFO Using network AWS Libfabric | |
| a100-st-p4d24xlarge-35:54314:55382 [0] NCCL INFO NCCL_TOPO_FILE set by environment to /fsx/users/dberard/scratch-local/bench-fast/aws-ofi-nccl/share/aws-ofi-nccl/xml/p4d-24xl-topo.xml | |
| still waiting for process to exit | |
| a100-st-p4d24xlarge-35:54314:55382 [0] NCCL INFO NET/AWS Libfabric : GPU Direct RDMA Enabled for HCA 0 'rdmap16s27' | |
| a100-st-p4d24xlarge-35:54314:55382 [0] NCCL INFO NET/AWS Libfabric : GPU Direct RDMA Enabled for HCA 1 'rdmap32s27' | |
| a100-st-p4d24xlarge-35:54314:55382 [0] NCCL INFO NET/AWS Libfabric : GPU Direct RDMA Enabled for HCA 2 'rdmap144s27' | |
| a100-st-p4d24xlarge-35:54314:55382 [0] NCCL INFO NET/AWS Libfabric : GPU Direct RDMA Enabled for HCA 3 'rdmap160s27' | |
| a100-st-p4d24xlarge-35:54314:55382 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 101c0 / HCA 0 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-35:54314:55382 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 101d0 / HCA 0 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-35:54314:55382 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 201c0 / HCA 0 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-35:54314:55382 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 201d0 / HCA 0 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-35:54314:55382 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 901c0 / HCA 0 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-35:54314:55382 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 901d0 / HCA 0 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-35:54314:55382 [0] NCCL INFO GPU Direct RDMA Enabled for GPU a01c0 / HCA 0 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-35:54314:55382 [0] NCCL INFO GPU Direct RDMA Enabled for GPU a01d0 / HCA 0 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-35:54314:55382 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 101c0 / HCA 1 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-35:54314:55382 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 101d0 / HCA 1 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-35:54314:55382 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 201c0 / HCA 1 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-35:54314:55382 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 201d0 / HCA 1 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-35:54314:55382 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 901c0 / HCA 1 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-35:54314:55382 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 901d0 / HCA 1 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-35:54314:55382 [0] NCCL INFO GPU Direct RDMA Enabled for GPU a01c0 / HCA 1 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-35:54314:55382 [0] NCCL INFO GPU Direct RDMA Enabled for GPU a01d0 / HCA 1 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-35:54314:55382 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 101c0 / HCA 2 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-35:54314:55382 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 101d0 / HCA 2 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-35:54314:55382 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 201c0 / HCA 2 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-35:54314:55382 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 201d0 / HCA 2 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-35:54314:55382 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 901c0 / HCA 2 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-35:54314:55382 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 901d0 / HCA 2 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-35:54314:55382 [0] NCCL INFO GPU Direct RDMA Enabled for GPU a01c0 / HCA 2 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-35:54314:55382 [0] NCCL INFO GPU Direct RDMA Enabled for GPU a01d0 / HCA 2 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-35:54314:55382 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 101c0 / HCA 3 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-35:54314:55382 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 101d0 / HCA 3 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-35:54314:55382 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 201c0 / HCA 3 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-35:54314:55382 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 201d0 / HCA 3 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-35:54314:55382 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 901c0 / HCA 3 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-35:54314:55382 [0] NCCL INFO GPU Direct RDMA Enabled for GPU 901d0 / HCA 3 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-35:54314:55382 [0] NCCL INFO GPU Direct RDMA Enabled for GPU a01c0 / HCA 3 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-35:54314:55382 [0] NCCL INFO GPU Direct RDMA Enabled for GPU a01d0 / HCA 3 (distance 3 <= 4), read 0 | |
| a100-st-p4d24xlarge-35:54314:55382 [0] NCCL INFO Setting affinity for GPU 0 to 1f0000,0000001f | |
| a100-st-p4d24xlarge-35:54314:55382 [0] NCCL INFO Channel 00/24 : 0 1 2 3 4 5 6 7 | |
| a100-st-p4d24xlarge-35:54314:55382 [0] NCCL INFO Channel 01/24 : 0 1 2 3 4 5 6 7 | |
| a100-st-p4d24xlarge-35:54314:55382 [0] NCCL INFO Channel 02/24 : 0 1 2 3 4 5 6 7 | |
| a100-st-p4d24xlarge-35:54314:55382 [0] NCCL INFO Channel 03/24 : 0 1 2 3 4 5 6 7 | |
| a100-st-p4d24xlarge-35:54314:55382 [0] NCCL INFO Channel 04/24 : 0 1 2 3 4 5 6 7 | |
| a100-st-p4d24xlarge-35:54314:55382 [0] NCCL INFO Channel 05/24 : 0 1 2 3 4 5 6 7 | |
| a100-st-p4d24xlarge-35:54314:55382 [0] NCCL INFO Channel 06/24 : 0 1 2 3 4 5 6 7 | |
| a100-st-p4d24xlarge-35:54314:55382 [0] NCCL INFO Channel 07/24 : 0 1 2 3 4 5 6 7 | |
| a100-st-p4d24xlarge-35:54314:55382 [0] NCCL INFO Channel 08/24 : 0 1 2 3 4 5 6 7 | |
| a100-st-p4d24xlarge-35:54314:55382 [0] NCCL INFO Channel 09/24 : 0 1 2 3 4 5 6 7 | |
| a100-st-p4d24xlarge-35:54314:55382 [0] NCCL INFO Channel 10/24 : 0 1 2 3 4 5 6 7 | |
| a100-st-p4d24xlarge-35:54314:55382 [0] NCCL INFO Channel 11/24 : 0 1 2 3 4 5 6 7 | |
| a100-st-p4d24xlarge-35:54314:55382 [0] NCCL INFO Channel 12/24 : 0 1 2 3 4 5 6 7 | |
| a100-st-p4d24xlarge-35:54314:55382 [0] NCCL INFO Channel 13/24 : 0 1 2 3 4 5 6 7 | |
| a100-st-p4d24xlarge-35:54314:55382 [0] NCCL INFO Channel 14/24 : 0 1 2 3 4 5 6 7 | |
| a100-st-p4d24xlarge-35:54314:55382 [0] NCCL INFO Channel 15/24 : 0 1 2 3 4 5 6 7 | |
| a100-st-p4d24xlarge-35:54314:55382 [0] NCCL INFO Channel 16/24 : 0 1 2 3 4 5 6 7 | |
| a100-st-p4d24xlarge-35:54314:55382 [0] NCCL INFO Channel 17/24 : 0 1 2 3 4 5 6 7 | |
| a100-st-p4d24xlarge-35:54314:55382 [0] NCCL INFO Channel 18/24 : 0 1 2 3 4 5 6 7 | |
| a100-st-p4d24xlarge-35:54314:55382 [0] NCCL INFO Channel 19/24 : 0 1 2 3 4 5 6 7 | |
| a100-st-p4d24xlarge-35:54314:55382 [0] NCCL INFO Channel 20/24 : 0 1 2 3 4 5 6 7 | |
| a100-st-p4d24xlarge-35:54314:55382 [0] NCCL INFO Channel 21/24 : 0 1 2 3 4 5 6 7 | |
| a100-st-p4d24xlarge-35:54314:55382 [0] NCCL INFO Channel 22/24 : 0 1 2 3 4 5 6 7 | |
| a100-st-p4d24xlarge-35:54314:55382 [0] NCCL INFO Channel 23/24 : 0 1 2 3 4 5 6 7 | |
| a100-st-p4d24xlarge-35:54314:55382 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1 [2] 1/-1/-1->0->-1 [3] 1/-1/-1->0->-1 [4] 1/-1/-1->0->-1 [5] 1/-1/-1->0->-1 [6] 1/-1/-1->0->-1 [7] 1/-1/-1->0->-1 [8] 1/-1/-1->0->-1 [9] 1/-1/-1->0->-1 [10] 1/-1/-1->0->-1 [11] 1/-1/-1->0->-1 [12] 1/-1/-1->0->-1 [13] 1/-1/-1->0->-1 [14] 1/-1/-1->0->-1 [15] 1/-1/-1->0->-1 [16] 1/-1/-1->0->-1 [17] 1/-1/-1->0->-1 [18] 1/-1/-1->0->-1 [19] 1/-1/-1->0->-1 [20] 1/-1/-1->0->-1 [21] 1/-1/-1->0->-1 [22] 1/-1/-1->0->-1 [23] 1/-1/-1->0->-1 | |
| a100-st-p4d24xlarge-35:54314:55382 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af0002dd0 | |
| a100-st-p4d24xlarge-35:54314:55424 [0] NCCL INFO New proxy recv connection 0 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:54314:55382 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af0002e10 | |
| a100-st-p4d24xlarge-35:54314:55424 [0] NCCL INFO New proxy recv connection 1 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:54314:55424 [0] NCCL INFO New proxy recv connection 2 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:54314:55382 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af0002e50 | |
| a100-st-p4d24xlarge-35:54314:55424 [0] NCCL INFO New proxy recv connection 3 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:54314:55382 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af0002e90 | |
| a100-st-p4d24xlarge-35:54314:55424 [0] NCCL INFO New proxy recv connection 4 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:54314:55382 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af0002ed0 | |
| a100-st-p4d24xlarge-35:54314:55424 [0] NCCL INFO New proxy recv connection 5 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:54314:55382 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af0002f10 | |
| a100-st-p4d24xlarge-35:54314:55424 [0] NCCL INFO New proxy recv connection 6 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:54314:55382 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af0002f50 | |
| a100-st-p4d24xlarge-35:54314:55424 [0] NCCL INFO New proxy recv connection 7 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:54314:55382 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af0002f90 | |
| a100-st-p4d24xlarge-35:54314:55424 [0] NCCL INFO New proxy recv connection 8 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:54314:55382 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af0002fd0 | |
| a100-st-p4d24xlarge-35:54314:55382 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af0003010 | |
| a100-st-p4d24xlarge-35:54314:55424 [0] NCCL INFO New proxy recv connection 9 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:54314:55382 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af0003050 | |
| a100-st-p4d24xlarge-35:54314:55424 [0] NCCL INFO New proxy recv connection 10 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:54314:55382 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af0003090 | |
| a100-st-p4d24xlarge-35:54314:55424 [0] NCCL INFO New proxy recv connection 11 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:54314:55382 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af00030d0 | |
| a100-st-p4d24xlarge-35:54314:55424 [0] NCCL INFO New proxy recv connection 12 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:54314:55382 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af0003110 | |
| a100-st-p4d24xlarge-35:54314:55424 [0] NCCL INFO New proxy recv connection 13 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:54314:55382 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af0003150 | |
| a100-st-p4d24xlarge-35:54314:55424 [0] NCCL INFO New proxy recv connection 14 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:54314:55382 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af0003190 | |
| a100-st-p4d24xlarge-35:54314:55424 [0] NCCL INFO New proxy recv connection 15 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:54314:55382 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af00031d0 | |
| a100-st-p4d24xlarge-35:54314:55424 [0] NCCL INFO New proxy recv connection 16 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:54314:55382 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af0003210 | |
| a100-st-p4d24xlarge-35:54314:55424 [0] NCCL INFO New proxy recv connection 17 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:54314:55382 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af0003250 | |
| a100-st-p4d24xlarge-35:54314:55424 [0] NCCL INFO New proxy recv connection 18 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:54314:55382 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af0003290 | |
| a100-st-p4d24xlarge-35:54314:55424 [0] NCCL INFO New proxy recv connection 19 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:54314:55382 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af00032d0 | |
| a100-st-p4d24xlarge-35:54314:55424 [0] NCCL INFO New proxy recv connection 20 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:54314:55382 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af0003310 | |
| a100-st-p4d24xlarge-35:54314:55424 [0] NCCL INFO New proxy recv connection 21 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:54314:55382 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af0003350 | |
| a100-st-p4d24xlarge-35:54314:55424 [0] NCCL INFO New proxy recv connection 22 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:54314:55382 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af0003390 | |
| a100-st-p4d24xlarge-35:54314:55424 [0] NCCL INFO New proxy recv connection 23 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:54314:55382 [0] NCCL INFO Channel 00/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-35:54314:55382 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af00033d0 | |
| a100-st-p4d24xlarge-35:54314:55424 [0] NCCL INFO New proxy send connection 24 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:54314:55382 [0] NCCL INFO Channel 01/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-35:54314:55382 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af0003410 | |
| a100-st-p4d24xlarge-35:54314:55424 [0] NCCL INFO New proxy send connection 25 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:54314:55382 [0] NCCL INFO Channel 02/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-35:54314:55382 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af0003450 | |
| a100-st-p4d24xlarge-35:54314:55424 [0] NCCL INFO New proxy send connection 26 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:54314:55382 [0] NCCL INFO Channel 03/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-35:54314:55382 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af0003490 | |
| a100-st-p4d24xlarge-35:54314:55424 [0] NCCL INFO New proxy send connection 27 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:54314:55382 [0] NCCL INFO Channel 04/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-35:54314:55382 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af00034d0 | |
| a100-st-p4d24xlarge-35:54314:55424 [0] NCCL INFO New proxy send connection 28 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:54314:55382 [0] NCCL INFO Channel 05/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-35:54314:55382 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af0003510 | |
| a100-st-p4d24xlarge-35:54314:55424 [0] NCCL INFO New proxy send connection 29 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:54314:55382 [0] NCCL INFO Channel 06/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-35:54314:55382 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af0003550 | |
| a100-st-p4d24xlarge-35:54314:55424 [0] NCCL INFO New proxy send connection 30 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:54314:55382 [0] NCCL INFO Channel 07/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-35:54314:55382 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af0003590 | |
| a100-st-p4d24xlarge-35:54314:55424 [0] NCCL INFO New proxy send connection 31 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:54314:55382 [0] NCCL INFO Channel 08/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-35:54314:55382 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af00035d0 | |
| a100-st-p4d24xlarge-35:54314:55424 [0] NCCL INFO New proxy send connection 32 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:54314:55382 [0] NCCL INFO Channel 09/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-35:54314:55382 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af0003610 | |
| a100-st-p4d24xlarge-35:54314:55424 [0] NCCL INFO New proxy send connection 33 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:54314:55382 [0] NCCL INFO Channel 10/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-35:54314:55382 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af0003650 | |
| a100-st-p4d24xlarge-35:54314:55424 [0] NCCL INFO New proxy send connection 34 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:54314:55382 [0] NCCL INFO Channel 11/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-35:54314:55382 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af0003690 | |
| a100-st-p4d24xlarge-35:54314:55424 [0] NCCL INFO New proxy send connection 35 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:54314:55382 [0] NCCL INFO Channel 12/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-35:54314:55382 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af00036d0 | |
| a100-st-p4d24xlarge-35:54314:55424 [0] NCCL INFO New proxy send connection 36 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:54314:55382 [0] NCCL INFO Channel 13/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-35:54314:55382 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af0003710 | |
| a100-st-p4d24xlarge-35:54314:55424 [0] NCCL INFO New proxy send connection 37 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:54314:55382 [0] NCCL INFO Channel 14/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-35:54314:55382 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af0003750 | |
| a100-st-p4d24xlarge-35:54314:55424 [0] NCCL INFO New proxy send connection 38 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:54314:55382 [0] NCCL INFO Channel 15/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-35:54314:55382 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af0003790 | |
| a100-st-p4d24xlarge-35:54314:55424 [0] NCCL INFO New proxy send connection 39 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:54314:55382 [0] NCCL INFO Channel 16/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-35:54314:55382 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af00037d0 | |
| a100-st-p4d24xlarge-35:54314:55424 [0] NCCL INFO New proxy send connection 40 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:54314:55382 [0] NCCL INFO Channel 17/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-35:54314:55382 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af0003810 | |
| a100-st-p4d24xlarge-35:54314:55424 [0] NCCL INFO New proxy send connection 41 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:54314:55382 [0] NCCL INFO Channel 18/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-35:54314:55382 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af0003850 | |
| a100-st-p4d24xlarge-35:54314:55424 [0] NCCL INFO New proxy send connection 42 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:54314:55382 [0] NCCL INFO Channel 19/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-35:54314:55382 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af0003890 | |
| a100-st-p4d24xlarge-35:54314:55424 [0] NCCL INFO New proxy send connection 43 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:54314:55382 [0] NCCL INFO Channel 20/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-35:54314:55382 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af00038d0 | |
| a100-st-p4d24xlarge-35:54314:55424 [0] NCCL INFO New proxy send connection 44 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:54314:55382 [0] NCCL INFO Channel 21/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-35:54314:55382 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af0003910 | |
| a100-st-p4d24xlarge-35:54314:55424 [0] NCCL INFO New proxy send connection 45 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:54314:55382 [0] NCCL INFO Channel 22/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-35:54314:55382 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af0003950 | |
| a100-st-p4d24xlarge-35:54314:55424 [0] NCCL INFO New proxy send connection 46 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:54314:55382 [0] NCCL INFO Channel 23/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read | |
| a100-st-p4d24xlarge-35:54314:55382 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af0003990 | |
| a100-st-p4d24xlarge-35:54314:55424 [0] NCCL INFO New proxy send connection 47 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:54314:55382 [0] NCCL INFO Connected all rings | |
| a100-st-p4d24xlarge-35:54314:55382 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af00039d0 | |
| a100-st-p4d24xlarge-35:54314:55424 [0] NCCL INFO New proxy recv connection 48 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:54314:55382 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af0003a10 | |
| a100-st-p4d24xlarge-35:54314:55424 [0] NCCL INFO New proxy recv connection 49 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:54314:55424 [0] NCCL INFO New proxy recv connection 50 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:54314:55382 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af0003a50 | |
| a100-st-p4d24xlarge-35:54314:55382 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af0003a90 | |
| a100-st-p4d24xlarge-35:54314:55424 [0] NCCL INFO New proxy recv connection 51 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:54314:55382 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af0003ad0 | |
| a100-st-p4d24xlarge-35:54314:55424 [0] NCCL INFO New proxy recv connection 52 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:54314:55382 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af0003b10 | |
| a100-st-p4d24xlarge-35:54314:55424 [0] NCCL INFO New proxy recv connection 53 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:54314:55382 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af0003b50 | |
| a100-st-p4d24xlarge-35:54314:55424 [0] NCCL INFO New proxy recv connection 54 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:54314:55382 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af0003b90 | |
| a100-st-p4d24xlarge-35:54314:55424 [0] NCCL INFO New proxy recv connection 55 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:54314:55382 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af0003bd0 | |
| a100-st-p4d24xlarge-35:54314:55424 [0] NCCL INFO New proxy recv connection 56 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:54314:55382 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af0003c10 | |
| a100-st-p4d24xlarge-35:54314:55424 [0] NCCL INFO New proxy recv connection 57 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:54314:55382 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af0003c50 | |
| a100-st-p4d24xlarge-35:54314:55424 [0] NCCL INFO New proxy recv connection 58 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:54314:55382 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af0003c90 | |
| a100-st-p4d24xlarge-35:54314:55424 [0] NCCL INFO New proxy recv connection 59 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:54314:55382 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af0003cd0 | |
| a100-st-p4d24xlarge-35:54314:55424 [0] NCCL INFO New proxy recv connection 60 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:54314:55382 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af0003d10 | |
| a100-st-p4d24xlarge-35:54314:55424 [0] NCCL INFO New proxy recv connection 61 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:54314:55382 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af0003d50 | |
| a100-st-p4d24xlarge-35:54314:55424 [0] NCCL INFO New proxy recv connection 62 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:54314:55382 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af0003d90 | |
| a100-st-p4d24xlarge-35:54314:55424 [0] NCCL INFO New proxy recv connection 63 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:54314:55382 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af0003dd0 | |
| a100-st-p4d24xlarge-35:54314:55424 [0] NCCL INFO New proxy recv connection 64 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:54314:55382 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af0003e10 | |
| a100-st-p4d24xlarge-35:54314:55424 [0] NCCL INFO New proxy recv connection 65 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:54314:55382 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af0003e50 | |
| a100-st-p4d24xlarge-35:54314:55424 [0] NCCL INFO New proxy recv connection 66 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:54314:55382 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af0003e90 | |
| a100-st-p4d24xlarge-35:54314:55424 [0] NCCL INFO New proxy recv connection 67 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:54314:55382 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af0003ed0 | |
| a100-st-p4d24xlarge-35:54314:55424 [0] NCCL INFO New proxy recv connection 68 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:54314:55382 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af0003f10 | |
| a100-st-p4d24xlarge-35:54314:55424 [0] NCCL INFO New proxy recv connection 69 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:54314:55382 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af0003f50 | |
| a100-st-p4d24xlarge-35:54314:55424 [0] NCCL INFO New proxy recv connection 70 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:54314:55382 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af0003f90 | |
| a100-st-p4d24xlarge-35:54314:55424 [0] NCCL INFO New proxy recv connection 71 from local rank 0, transport 0 | |
| a100-st-p4d24xlarge-35:54314:55382 [0] NCCL INFO Connected all trees | |
| a100-st-p4d24xlarge-35:54314:55382 [0] NCCL INFO NCCL_ALGO set by environment to ring | |
| a100-st-p4d24xlarge-35:54314:55382 [0] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512 | |
| a100-st-p4d24xlarge-35:54314:55382 [0] NCCL INFO 24 coll channels, 32 p2p channels, 32 p2p channels per peer | |
| a100-st-p4d24xlarge-35:54314:55424 [0] NCCL INFO New proxy send connection 72 from local rank 0, transport 2 | |
| a100-st-p4d24xlarge-35:54314:55382 [0] NCCL INFO Connection to proxy localRank 0 -> connection 0x7f4af0003fd0 | |
| a100-st-p4d24xlarge-35:54314:55382 [0] NCCL INFO comm 0x5562b1c1a0a0 rank 0 nranks 8 cudaDev 0 busId 101c0 - Init COMPLETE | |
| didn't get the result, queue empty | |
| DDPOptimizer used bucket cap 26214400 and produced the following buckets: | |
| Index Size (b) Param Names | |
| ------- ---------- --------------------------------- | |
| 0 8196000 self_fc_weight | |
| self_fc_bias | |
| 1 31502336 self_layer4_2_bn3_weight | |
| self_layer4_2_bn3_bias | |
| self_layer4_2_conv3_weight | |
| self_layer4_2_bn2_weight | |
| self_layer4_2_bn2_bias | |
| self_layer4_2_conv2_weight | |
| self_layer4_2_bn1_weight | |
| self_layer4_2_bn1_bias | |
| self_layer4_2_conv1_weight | |
| self_layer4_1_bn3_weight | |
| self_layer4_1_bn3_bias | |
| self_layer4_1_conv3_weight | |
| self_layer4_1_bn2_weight | |
| self_layer4_1_bn2_bias | |
| self_layer4_1_conv2_weight | |
| 2 26255360 self_layer4_1_bn1_weight | |
| self_layer4_1_bn1_bias | |
| self_layer4_1_conv1_weight | |
| self_layer4_0_downsample_1_weight | |
| self_layer4_0_downsample_1_bias | |
| self_layer4_0_downsample_0_weight | |
| self_layer4_0_bn3_weight | |
| self_layer4_0_bn3_bias | |
| self_layer4_0_conv3_weight | |
| self_layer4_0_bn2_weight | |
| self_layer4_0_bn2_bias | |
| self_layer4_0_conv2_weight | |
| 3 26550272 self_layer4_0_bn1_weight | |
| self_layer4_0_bn1_bias | |
| self_layer4_0_conv1_weight | |
| self_layer3_5_bn3_weight | |
| self_layer3_5_bn3_bias | |
| self_layer3_5_conv3_weight | |
| self_layer3_5_bn2_weight | |
| self_layer3_5_bn2_bias | |
| self_layer3_5_conv2_weight | |
| self_layer3_5_bn1_weight | |
| self_layer3_5_bn1_bias | |
| self_layer3_5_conv1_weight | |
| self_layer3_4_bn3_weight | |
| self_layer3_4_bn3_bias | |
| self_layer3_4_conv3_weight | |
| self_layer3_4_bn2_weight | |
| self_layer3_4_bn2_bias | |
| self_layer3_4_conv2_weight | |
| self_layer3_4_bn1_weight | |
| self_layer3_4_bn1_bias | |
| self_layer3_4_conv1_weight | |
| self_layer3_3_bn3_weight | |
| self_layer3_3_bn3_bias | |
| self_layer3_3_conv3_weight | |
| self_layer3_3_bn2_weight | |
| self_layer3_3_bn2_bias | |
| self_layer3_3_conv2_weight | |
| self_layer3_3_bn1_weight | |
| self_layer3_3_bn1_bias | |
| self_layer3_3_conv1_weight | |
| self_layer3_2_bn3_weight | |
| self_layer3_2_bn3_bias | |
| self_layer3_2_conv3_weight | |
| self_layer3_2_bn2_weight | |
| self_layer3_2_bn2_bias | |
| self_layer3_2_conv2_weight | |
| self_layer3_2_bn1_weight | |
| self_layer3_2_bn1_bias | |
| self_layer3_2_conv1_weight | |
| self_layer3_1_bn3_weight | |
| self_layer3_1_bn3_bias | |
| self_layer3_1_conv3_weight | |
| self_layer3_1_bn2_weight | |
| self_layer3_1_bn2_bias | |
| self_layer3_1_conv2_weight | |
| self_layer3_1_bn1_weight | |
| self_layer3_1_bn1_bias | |
| self_layer3_1_conv1_weight | |
| self_layer3_0_downsample_1_weight | |
| self_layer3_0_downsample_1_bias | |
| self_layer3_0_downsample_0_weight | |
| 4 9724160 self_layer3_0_bn3_weight | |
| self_layer3_0_bn3_bias | |
| self_layer3_0_conv3_weight | |
| self_layer3_0_bn2_weight | |
| self_layer3_0_bn2_bias | |
| self_layer3_0_conv2_weight | |
| self_layer3_0_bn1_weight | |
| self_layer3_0_bn1_bias | |
| self_layer3_0_conv1_weight | |
| self_layer2_3_bn3_weight | |
| self_layer2_3_bn3_bias | |
| self_layer2_3_conv3_weight | |
| self_layer2_3_bn2_weight | |
| self_layer2_3_bn2_bias | |
| self_layer2_3_conv2_weight | |
| self_layer2_3_bn1_weight | |
| self_layer2_3_bn1_bias | |
| self_layer2_3_conv1_weight | |
| self_layer2_2_bn3_weight | |
| self_layer2_2_bn3_bias | |
| self_layer2_2_conv3_weight | |
| self_layer2_2_bn2_weight | |
| self_layer2_2_bn2_bias | |
| self_layer2_2_conv2_weight | |
| self_layer2_2_bn1_weight | |
| self_layer2_2_bn1_bias | |
| self_layer2_2_conv1_weight | |
| self_layer2_1_bn3_weight | |
| self_layer2_1_bn3_bias | |
| self_layer2_1_conv3_weight | |
| self_layer2_1_bn2_weight | |
| self_layer2_1_bn2_bias | |
| self_layer2_1_conv2_weight | |
| self_layer2_1_bn1_weight | |
| self_layer2_1_bn1_bias | |
| self_layer2_1_conv1_weight | |
| self_layer2_0_downsample_1_weight | |
| self_layer2_0_downsample_1_bias | |
| self_layer2_0_downsample_0_weight | |
| self_layer2_0_bn3_weight | |
| self_layer2_0_bn3_bias | |
| self_layer2_0_conv3_weight | |
| self_layer2_0_bn2_weight | |
| self_layer2_0_bn2_bias | |
| self_layer2_0_conv2_weight | |
| self_layer2_0_bn1_weight | |
| self_layer2_0_bn1_bias | |
| self_layer2_0_conv1_weight | |
| self_layer1_2_bn3_weight | |
| self_layer1_2_bn3_bias | |
| self_layer1_2_conv3_weight | |
| self_layer1_2_bn2_weight | |
| self_layer1_2_bn2_bias | |
| self_layer1_2_conv2_weight | |
| self_layer1_2_bn1_weight | |
| self_layer1_2_bn1_bias | |
| self_layer1_2_conv1_weight | |
| self_layer1_1_bn3_weight | |
| self_layer1_1_bn3_bias | |
| self_layer1_1_conv3_weight | |
| self_layer1_1_bn2_weight | |
| self_layer1_1_bn2_bias | |
| self_layer1_1_conv2_weight | |
| self_layer1_1_bn1_weight | |
| self_layer1_1_bn1_bias | |
| self_layer1_1_conv1_weight | |
| self_layer1_0_downsample_1_weight | |
| self_layer1_0_downsample_1_bias | |
| self_layer1_0_downsample_0_weight | |
| self_layer1_0_bn3_weight | |
| self_layer1_0_bn3_bias | |
| self_layer1_0_conv3_weight | |
| self_layer1_0_bn2_weight | |
| self_layer1_0_bn2_bias | |
| self_layer1_0_conv2_weight | |
| self_layer1_0_bn1_weight | |
| self_layer1_0_bn1_bias | |
| self_layer1_0_conv1_weight | |
| self_bn1_weight | |
| self_bn1_bias | |
| self_conv1_weight | |
| ---orig graph--- | |
| graph(): | |
| %x : torch.Tensor [#users=1] = placeholder[target=x] | |
| %self_conv1 : [#users=1] = call_module[target=self_conv1](args = (%x,), kwargs = {}) | |
| %self_bn1 : [#users=1] = call_module[target=self_bn1](args = (%self_conv1,), kwargs = {}) | |
| %self_relu : [#users=1] = call_module[target=self_relu](args = (%self_bn1,), kwargs = {}) | |
| %self_maxpool : [#users=2] = call_module[target=self_maxpool](args = (%self_relu,), kwargs = {}) | |
| %self_layer1_0_conv1 : [#users=1] = call_module[target=self_layer1_0_conv1](args = (%self_maxpool,), kwargs = {}) | |
| %self_layer1_0_bn1 : [#users=1] = call_module[target=self_layer1_0_bn1](args = (%self_layer1_0_conv1,), kwargs = {}) | |
| %self_layer1_0_relu : [#users=1] = call_module[target=self_layer1_0_relu](args = (%self_layer1_0_bn1,), kwargs = {}) | |
| %self_layer1_0_conv2 : [#users=1] = call_module[target=self_layer1_0_conv2](args = (%self_layer1_0_relu,), kwargs = {}) | |
| %self_layer1_0_bn2 : [#users=1] = call_module[target=self_layer1_0_bn2](args = (%self_layer1_0_conv2,), kwargs = {}) | |
| %self_layer1_0_relu_1 : [#users=1] = call_module[target=self_layer1_0_relu](args = (%self_layer1_0_bn2,), kwargs = {}) | |
| %self_layer1_0_conv3 : [#users=1] = call_module[target=self_layer1_0_conv3](args = (%self_layer1_0_relu_1,), kwargs = {}) | |
| %self_layer1_0_bn3 : [#users=1] = call_module[target=self_layer1_0_bn3](args = (%self_layer1_0_conv3,), kwargs = {}) | |
| %self_layer1_0_downsample_0 : [#users=1] = call_module[target=self_layer1_0_downsample_0](args = (%self_maxpool,), kwargs = {}) | |
| %self_layer1_0_downsample_1 : [#users=1] = call_module[target=self_layer1_0_downsample_1](args = (%self_layer1_0_downsample_0,), kwargs = {}) | |
| %iadd : [#users=1] = call_function[target=operator.iadd](args = (%self_layer1_0_bn3, %self_layer1_0_downsample_1), kwargs = {}) | |
| %self_layer1_0_relu_2 : [#users=2] = call_module[target=self_layer1_0_relu](args = (%iadd,), kwargs = {}) | |
| %self_layer1_1_conv1 : [#users=1] = call_module[target=self_layer1_1_conv1](args = (%self_layer1_0_relu_2,), kwargs = {}) | |
| %self_layer1_1_bn1 : [#users=1] = call_module[target=self_layer1_1_bn1](args = (%self_layer1_1_conv1,), kwargs = {}) | |
| %self_layer1_1_relu : [#users=1] = call_module[target=self_layer1_1_relu](args = (%self_layer1_1_bn1,), kwargs = {}) | |
| %self_layer1_1_conv2 : [#users=1] = call_module[target=self_layer1_1_conv2](args = (%self_layer1_1_relu,), kwargs = {}) | |
| %self_layer1_1_bn2 : [#users=1] = call_module[target=self_layer1_1_bn2](args = (%self_layer1_1_conv2,), kwargs = {}) | |
| %self_layer1_1_relu_1 : [#users=1] = call_module[target=self_layer1_1_relu](args = (%self_layer1_1_bn2,), kwargs = {}) | |
| %self_layer1_1_conv3 : [#users=1] = call_module[target=self_layer1_1_conv3](args = (%self_layer1_1_relu_1,), kwargs = {}) | |
| %self_layer1_1_bn3 : [#users=1] = call_module[target=self_layer1_1_bn3](args = (%self_layer1_1_conv3,), kwargs = {}) | |
| %iadd_1 : [#users=1] = call_function[target=operator.iadd](args = (%self_layer1_1_bn3, %self_layer1_0_relu_2), kwargs = {}) | |
| %self_layer1_1_relu_2 : [#users=2] = call_module[target=self_layer1_1_relu](args = (%iadd_1,), kwargs = {}) | |
| %self_layer1_2_conv1 : [#users=1] = call_module[target=self_layer1_2_conv1](args = (%self_layer1_1_relu_2,), kwargs = {}) | |
| %self_layer1_2_bn1 : [#users=1] = call_module[target=self_layer1_2_bn1](args = (%self_layer1_2_conv1,), kwargs = {}) | |
| %self_layer1_2_relu : [#users=1] = call_module[target=self_layer1_2_relu](args = (%self_layer1_2_bn1,), kwargs = {}) | |
| %self_layer1_2_conv2 : [#users=1] = call_module[target=self_layer1_2_conv2](args = (%self_layer1_2_relu,), kwargs = {}) | |
| %self_layer1_2_bn2 : [#users=1] = call_module[target=self_layer1_2_bn2](args = (%self_layer1_2_conv2,), kwargs = {}) | |
| %self_layer1_2_relu_1 : [#users=1] = call_module[target=self_layer1_2_relu](args = (%self_layer1_2_bn2,), kwargs = {}) | |
| %self_layer1_2_conv3 : [#users=1] = call_module[target=self_layer1_2_conv3](args = (%self_layer1_2_relu_1,), kwargs = {}) | |
| %self_layer1_2_bn3 : [#users=1] = call_module[target=self_layer1_2_bn3](args = (%self_layer1_2_conv3,), kwargs = {}) | |
| %iadd_2 : [#users=1] = call_function[target=operator.iadd](args = (%self_layer1_2_bn3, %self_layer1_1_relu_2), kwargs = {}) | |
| %self_layer1_2_relu_2 : [#users=2] = call_module[target=self_layer1_2_relu](args = (%iadd_2,), kwargs = {}) | |
| %self_layer2_0_conv1 : [#users=1] = call_module[target=self_layer2_0_conv1](args = (%self_layer1_2_relu_2,), kwargs = {}) | |
| %self_layer2_0_bn1 : [#users=1] = call_module[target=self_layer2_0_bn1](args = (%self_layer2_0_conv1,), kwargs = {}) | |
| %self_layer2_0_relu : [#users=1] = call_module[target=self_layer2_0_relu](args = (%self_layer2_0_bn1,), kwargs = {}) | |
| %self_layer2_0_conv2 : [#users=1] = call_module[target=self_layer2_0_conv2](args = (%self_layer2_0_relu,), kwargs = {}) | |
| %self_layer2_0_bn2 : [#users=1] = call_module[target=self_layer2_0_bn2](args = (%self_layer2_0_conv2,), kwargs = {}) | |
| %self_layer2_0_relu_1 : [#users=1] = call_module[target=self_layer2_0_relu](args = (%self_layer2_0_bn2,), kwargs = {}) | |
| %self_layer2_0_conv3 : [#users=1] = call_module[target=self_layer2_0_conv3](args = (%self_layer2_0_relu_1,), kwargs = {}) | |
| %self_layer2_0_bn3 : [#users=1] = call_module[target=self_layer2_0_bn3](args = (%self_layer2_0_conv3,), kwargs = {}) | |
| %self_layer2_0_downsample_0 : [#users=1] = call_module[target=self_layer2_0_downsample_0](args = (%self_layer1_2_relu_2,), kwargs = {}) | |
| %self_layer2_0_downsample_1 : [#users=1] = call_module[target=self_layer2_0_downsample_1](args = (%self_layer2_0_downsample_0,), kwargs = {}) | |
| %iadd_3 : [#users=1] = call_function[target=operator.iadd](args = (%self_layer2_0_bn3, %self_layer2_0_downsample_1), kwargs = {}) | |
| %self_layer2_0_relu_2 : [#users=2] = call_module[target=self_layer2_0_relu](args = (%iadd_3,), kwargs = {}) | |
| %self_layer2_1_conv1 : [#users=1] = call_module[target=self_layer2_1_conv1](args = (%self_layer2_0_relu_2,), kwargs = {}) | |
| %self_layer2_1_bn1 : [#users=1] = call_module[target=self_layer2_1_bn1](args = (%self_layer2_1_conv1,), kwargs = {}) | |
| %self_layer2_1_relu : [#users=1] = call_module[target=self_layer2_1_relu](args = (%self_layer2_1_bn1,), kwargs = {}) | |
| %self_layer2_1_conv2 : [#users=1] = call_module[target=self_layer2_1_conv2](args = (%self_layer2_1_relu,), kwargs = {}) | |
| %self_layer2_1_bn2 : [#users=1] = call_module[target=self_layer2_1_bn2](args = (%self_layer2_1_conv2,), kwargs = {}) | |
| %self_layer2_1_relu_1 : [#users=1] = call_module[target=self_layer2_1_relu](args = (%self_layer2_1_bn2,), kwargs = {}) | |
| %self_layer2_1_conv3 : [#users=1] = call_module[target=self_layer2_1_conv3](args = (%self_layer2_1_relu_1,), kwargs = {}) | |
| %self_layer2_1_bn3 : [#users=1] = call_module[target=self_layer2_1_bn3](args = (%self_layer2_1_conv3,), kwargs = {}) | |
| %iadd_4 : [#users=1] = call_function[target=operator.iadd](args = (%self_layer2_1_bn3, %self_layer2_0_relu_2), kwargs = {}) | |
| %self_layer2_1_relu_2 : [#users=2] = call_module[target=self_layer2_1_relu](args = (%iadd_4,), kwargs = {}) | |
| %self_layer2_2_conv1 : [#users=1] = call_module[target=self_layer2_2_conv1](args = (%self_layer2_1_relu_2,), kwargs = {}) | |
| %self_layer2_2_bn1 : [#users=1] = call_module[target=self_layer2_2_bn1](args = (%self_layer2_2_conv1,), kwargs = {}) | |
| %self_layer2_2_relu : [#users=1] = call_module[target=self_layer2_2_relu](args = (%self_layer2_2_bn1,), kwargs = {}) | |
| %self_layer2_2_conv2 : [#users=1] = call_module[target=self_layer2_2_conv2](args = (%self_layer2_2_relu,), kwargs = {}) | |
| %self_layer2_2_bn2 : [#users=1] = call_module[target=self_layer2_2_bn2](args = (%self_layer2_2_conv2,), kwargs = {}) | |
| %self_layer2_2_relu_1 : [#users=1] = call_module[target=self_layer2_2_relu](args = (%self_layer2_2_bn2,), kwargs = {}) | |
| %self_layer2_2_conv3 : [#users=1] = call_module[target=self_layer2_2_conv3](args = (%self_layer2_2_relu_1,), kwargs = {}) | |
| %self_layer2_2_bn3 : [#users=1] = call_module[target=self_layer2_2_bn3](args = (%self_layer2_2_conv3,), kwargs = {}) | |
| %iadd_5 : [#users=1] = call_function[target=operator.iadd](args = (%self_layer2_2_bn3, %self_layer2_1_relu_2), kwargs = {}) | |
| %self_layer2_2_relu_2 : [#users=2] = call_module[target=self_layer2_2_relu](args = (%iadd_5,), kwargs = {}) | |
| %self_layer2_3_conv1 : [#users=1] = call_module[target=self_layer2_3_conv1](args = (%self_layer2_2_relu_2,), kwargs = {}) | |
| %self_layer2_3_bn1 : [#users=1] = call_module[target=self_layer2_3_bn1](args = (%self_layer2_3_conv1,), kwargs = {}) | |
| %self_layer2_3_relu : [#users=1] = call_module[target=self_layer2_3_relu](args = (%self_layer2_3_bn1,), kwargs = {}) | |
| %self_layer2_3_conv2 : [#users=1] = call_module[target=self_layer2_3_conv2](args = (%self_layer2_3_relu,), kwargs = {}) | |
| %self_layer2_3_bn2 : [#users=1] = call_module[target=self_layer2_3_bn2](args = (%self_layer2_3_conv2,), kwargs = {}) | |
| %self_layer2_3_relu_1 : [#users=1] = call_module[target=self_layer2_3_relu](args = (%self_layer2_3_bn2,), kwargs = {}) | |
| %self_layer2_3_conv3 : [#users=1] = call_module[target=self_layer2_3_conv3](args = (%self_layer2_3_relu_1,), kwargs = {}) | |
| %self_layer2_3_bn3 : [#users=1] = call_module[target=self_layer2_3_bn3](args = (%self_layer2_3_conv3,), kwargs = {}) | |
| %iadd_6 : [#users=1] = call_function[target=operator.iadd](args = (%self_layer2_3_bn3, %self_layer2_2_relu_2), kwargs = {}) | |
| %self_layer2_3_relu_2 : [#users=2] = call_module[target=self_layer2_3_relu](args = (%iadd_6,), kwargs = {}) | |
| %self_layer3_0_conv1 : [#users=1] = call_module[target=self_layer3_0_conv1](args = (%self_layer2_3_relu_2,), kwargs = {}) | |
| %self_layer3_0_bn1 : [#users=1] = call_module[target=self_layer3_0_bn1](args = (%self_layer3_0_conv1,), kwargs = {}) | |
| %self_layer3_0_relu : [#users=1] = call_module[target=self_layer3_0_relu](args = (%self_layer3_0_bn1,), kwargs = {}) | |
| %self_layer3_0_conv2 : [#users=1] = call_module[target=self_layer3_0_conv2](args = (%self_layer3_0_relu,), kwargs = {}) | |
| %self_layer3_0_bn2 : [#users=1] = call_module[target=self_layer3_0_bn2](args = (%self_layer3_0_conv2,), kwargs = {}) | |
| %self_layer3_0_relu_1 : [#users=1] = call_module[target=self_layer3_0_relu](args = (%self_layer3_0_bn2,), kwargs = {}) | |
| %self_layer3_0_conv3 : [#users=1] = call_module[target=self_layer3_0_conv3](args = (%self_layer3_0_relu_1,), kwargs = {}) | |
| %self_layer3_0_bn3 : [#users=1] = call_module[target=self_layer3_0_bn3](args = (%self_layer3_0_conv3,), kwargs = {}) | |
| %self_layer3_0_downsample_0 : [#users=1] = call_module[target=self_layer3_0_downsample_0](args = (%self_layer2_3_relu_2,), kwargs = {}) | |
| %self_layer3_0_downsample_1 : [#users=1] = call_module[target=self_layer3_0_downsample_1](args = (%self_layer3_0_downsample_0,), kwargs = {}) | |
| %iadd_7 : [#users=1] = call_function[target=operator.iadd](args = (%self_layer3_0_bn3, %self_layer3_0_downsample_1), kwargs = {}) | |
| %self_layer3_0_relu_2 : [#users=2] = call_module[target=self_layer3_0_relu](args = (%iadd_7,), kwargs = {}) | |
| %self_layer3_1_conv1 : [#users=1] = call_module[target=self_layer3_1_conv1](args = (%self_layer3_0_relu_2,), kwargs = {}) | |
| %self_layer3_1_bn1 : [#users=1] = call_module[target=self_layer3_1_bn1](args = (%self_layer3_1_conv1,), kwargs = {}) | |
| %self_layer3_1_relu : [#users=1] = call_module[target=self_layer3_1_relu](args = (%self_layer3_1_bn1,), kwargs = {}) | |
| %self_layer3_1_conv2 : [#users=1] = call_module[target=self_layer3_1_conv2](args = (%self_layer3_1_relu,), kwargs = {}) | |
| %self_layer3_1_bn2 : [#users=1] = call_module[target=self_layer3_1_bn2](args = (%self_layer3_1_conv2,), kwargs = {}) | |
| %self_layer3_1_relu_1 : [#users=1] = call_module[target=self_layer3_1_relu](args = (%self_layer3_1_bn2,), kwargs = {}) | |
| %self_layer3_1_conv3 : [#users=1] = call_module[target=self_layer3_1_conv3](args = (%self_layer3_1_relu_1,), kwargs = {}) | |
| %self_layer3_1_bn3 : [#users=1] = call_module[target=self_layer3_1_bn3](args = (%self_layer3_1_conv3,), kwargs = {}) | |
| %iadd_8 : [#users=1] = call_function[target=operator.iadd](args = (%self_layer3_1_bn3, %self_layer3_0_relu_2), kwargs = {}) | |
| %self_layer3_1_relu_2 : [#users=2] = call_module[target=self_layer3_1_relu](args = (%iadd_8,), kwargs = {}) | |
| %self_layer3_2_conv1 : [#users=1] = call_module[target=self_layer3_2_conv1](args = (%self_layer3_1_relu_2,), kwargs = {}) | |
| %self_layer3_2_bn1 : [#users=1] = call_module[target=self_layer3_2_bn1](args = (%self_layer3_2_conv1,), kwargs = {}) | |
| %self_layer3_2_relu : [#users=1] = call_module[target=self_layer3_2_relu](args = (%self_layer3_2_bn1,), kwargs = {}) | |
| %self_layer3_2_conv2 : [#users=1] = call_module[target=self_layer3_2_conv2](args = (%self_layer3_2_relu,), kwargs = {}) | |
| %self_layer3_2_bn2 : [#users=1] = call_module[target=self_layer3_2_bn2](args = (%self_layer3_2_conv2,), kwargs = {}) | |
| %self_layer3_2_relu_1 : [#users=1] = call_module[target=self_layer3_2_relu](args = (%self_layer3_2_bn2,), kwargs = {}) | |
| %self_layer3_2_conv3 : [#users=1] = call_module[target=self_layer3_2_conv3](args = (%self_layer3_2_relu_1,), kwargs = {}) | |
| %self_layer3_2_bn3 : [#users=1] = call_module[target=self_layer3_2_bn3](args = (%self_layer3_2_conv3,), kwargs = {}) | |
| %iadd_9 : [#users=1] = call_function[target=operator.iadd](args = (%self_layer3_2_bn3, %self_layer3_1_relu_2), kwargs = {}) | |
| %self_layer3_2_relu_2 : [#users=2] = call_module[target=self_layer3_2_relu](args = (%iadd_9,), kwargs = {}) | |
| %self_layer3_3_conv1 : [#users=1] = call_module[target=self_layer3_3_conv1](args = (%self_layer3_2_relu_2,), kwargs = {}) | |
| %self_layer3_3_bn1 : [#users=1] = call_module[target=self_layer3_3_bn1](args = (%self_layer3_3_conv1,), kwargs = {}) | |
| %self_layer3_3_relu : [#users=1] = call_module[target=self_layer3_3_relu](args = (%self_layer3_3_bn1,), kwargs = {}) | |
| %self_layer3_3_conv2 : [#users=1] = call_module[target=self_layer3_3_conv2](args = (%self_layer3_3_relu,), kwargs = {}) | |
| %self_layer3_3_bn2 : [#users=1] = call_module[target=self_layer3_3_bn2](args = (%self_layer3_3_conv2,), kwargs = {}) | |
| %self_layer3_3_relu_1 : [#users=1] = call_module[target=self_layer3_3_relu](args = (%self_layer3_3_bn2,), kwargs = {}) | |
| %self_layer3_3_conv3 : [#users=1] = call_module[target=self_layer3_3_conv3](args = (%self_layer3_3_relu_1,), kwargs = {}) | |
| %self_layer3_3_bn3 : [#users=1] = call_module[target=self_layer3_3_bn3](args = (%self_layer3_3_conv3,), kwargs = {}) | |
| %iadd_10 : [#users=1] = call_function[target=operator.iadd](args = (%self_layer3_3_bn3, %self_layer3_2_relu_2), kwargs = {}) | |
| %self_layer3_3_relu_2 : [#users=2] = call_module[target=self_layer3_3_relu](args = (%iadd_10,), kwargs = {}) | |
| %self_layer3_4_conv1 : [#users=1] = call_module[target=self_layer3_4_conv1](args = (%self_layer3_3_relu_2,), kwargs = {}) | |
| %self_layer3_4_bn1 : [#users=1] = call_module[target=self_layer3_4_bn1](args = (%self_layer3_4_conv1,), kwargs = {}) | |
| %self_layer3_4_relu : [#users=1] = call_module[target=self_layer3_4_relu](args = (%self_layer3_4_bn1,), kwargs = {}) | |
| %self_layer3_4_conv2 : [#users=1] = call_module[target=self_layer3_4_conv2](args = (%self_layer3_4_relu,), kwargs = {}) | |
| %self_layer3_4_bn2 : [#users=1] = call_module[target=self_layer3_4_bn2](args = (%self_layer3_4_conv2,), kwargs = {}) | |
| %self_layer3_4_relu_1 : [#users=1] = call_module[target=self_layer3_4_relu](args = (%self_layer3_4_bn2,), kwargs = {}) | |
| %self_layer3_4_conv3 : [#users=1] = call_module[target=self_layer3_4_conv3](args = (%self_layer3_4_relu_1,), kwargs = {}) | |
| %self_layer3_4_bn3 : [#users=1] = call_module[target=self_layer3_4_bn3](args = (%self_layer3_4_conv3,), kwargs = {}) | |
| %iadd_11 : [#users=1] = call_function[target=operator.iadd](args = (%self_layer3_4_bn3, %self_layer3_3_relu_2), kwargs = {}) | |
| %self_layer3_4_relu_2 : [#users=2] = call_module[target=self_layer3_4_relu](args = (%iadd_11,), kwargs = {}) | |
| %self_layer3_5_conv1 : [#users=1] = call_module[target=self_layer3_5_conv1](args = (%self_layer3_4_relu_2,), kwargs = {}) | |
| %self_layer3_5_bn1 : [#users=1] = call_module[target=self_layer3_5_bn1](args = (%self_layer3_5_conv1,), kwargs = {}) | |
| %self_layer3_5_relu : [#users=1] = call_module[target=self_layer3_5_relu](args = (%self_layer3_5_bn1,), kwargs = {}) | |
| %self_layer3_5_conv2 : [#users=1] = call_module[target=self_layer3_5_conv2](args = (%self_layer3_5_relu,), kwargs = {}) | |
| %self_layer3_5_bn2 : [#users=1] = call_module[target=self_layer3_5_bn2](args = (%self_layer3_5_conv2,), kwargs = {}) | |
| %self_layer3_5_relu_1 : [#users=1] = call_module[target=self_layer3_5_relu](args = (%self_layer3_5_bn2,), kwargs = {}) | |
| %self_layer3_5_conv3 : [#users=1] = call_module[target=self_layer3_5_conv3](args = (%self_layer3_5_relu_1,), kwargs = {}) | |
| %self_layer3_5_bn3 : [#users=1] = call_module[target=self_layer3_5_bn3](args = (%self_layer3_5_conv3,), kwargs = {}) | |
| %iadd_12 : [#users=1] = call_function[target=operator.iadd](args = (%self_layer3_5_bn3, %self_layer3_4_relu_2), kwargs = {}) | |
| %self_layer3_5_relu_2 : [#users=2] = call_module[target=self_layer3_5_relu](args = (%iadd_12,), kwargs = {}) | |
| %self_layer4_0_conv1 : [#users=1] = call_module[target=self_layer4_0_conv1](args = (%self_layer3_5_relu_2,), kwargs = {}) | |
| %self_layer4_0_bn1 : [#users=1] = call_module[target=self_layer4_0_bn1](args = (%self_layer4_0_conv1,), kwargs = {}) | |
| %self_layer4_0_relu : [#users=1] = call_module[target=self_layer4_0_relu](args = (%self_layer4_0_bn1,), kwargs = {}) | |
| %self_layer4_0_conv2 : [#users=1] = call_module[target=self_layer4_0_conv2](args = (%self_layer4_0_relu,), kwargs = {}) | |
| %self_layer4_0_bn2 : [#users=1] = call_module[target=self_layer4_0_bn2](args = (%self_layer4_0_conv2,), kwargs = {}) | |
| %self_layer4_0_relu_1 : [#users=1] = call_module[target=self_layer4_0_relu](args = (%self_layer4_0_bn2,), kwargs = {}) | |
| %self_layer4_0_conv3 : [#users=1] = call_module[target=self_layer4_0_conv3](args = (%self_layer4_0_relu_1,), kwargs = {}) | |
| %self_layer4_0_bn3 : [#users=1] = call_module[target=self_layer4_0_bn3](args = (%self_layer4_0_conv3,), kwargs = {}) | |
| %self_layer4_0_downsample_0 : [#users=1] = call_module[target=self_layer4_0_downsample_0](args = (%self_layer3_5_relu_2,), kwargs = {}) | |
| %self_layer4_0_downsample_1 : [#users=1] = call_module[target=self_layer4_0_downsample_1](args = (%self_layer4_0_downsample_0,), kwargs = {}) | |
| %iadd_13 : [#users=1] = call_function[target=operator.iadd](args = (%self_layer4_0_bn3, %self_layer4_0_downsample_1), kwargs = {}) | |
| %self_layer4_0_relu_2 : [#users=2] = call_module[target=self_layer4_0_relu](args = (%iadd_13,), kwargs = {}) | |
| %self_layer4_1_conv1 : [#users=1] = call_module[target=self_layer4_1_conv1](args = (%self_layer4_0_relu_2,), kwargs = {}) | |
| %self_layer4_1_bn1 : [#users=1] = call_module[target=self_layer4_1_bn1](args = (%self_layer4_1_conv1,), kwargs = {}) | |
| %self_layer4_1_relu : [#users=1] = call_module[target=self_layer4_1_relu](args = (%self_layer4_1_bn1,), kwargs = {}) | |
| %self_layer4_1_conv2 : [#users=1] = call_module[target=self_layer4_1_conv2](args = (%self_layer4_1_relu,), kwargs = {}) | |
| %self_layer4_1_bn2 : [#users=1] = call_module[target=self_layer4_1_bn2](args = (%self_layer4_1_conv2,), kwargs = {}) | |
| %self_layer4_1_relu_1 : [#users=1] = call_module[target=self_layer4_1_relu](args = (%self_layer4_1_bn2,), kwargs = {}) | |
| %self_layer4_1_conv3 : [#users=1] = call_module[target=self_layer4_1_conv3](args = (%self_layer4_1_relu_1,), kwargs = {}) | |
| %self_layer4_1_bn3 : [#users=1] = call_module[target=self_layer4_1_bn3](args = (%self_layer4_1_conv3,), kwargs = {}) | |
| %iadd_14 : [#users=1] = call_function[target=operator.iadd](args = (%self_layer4_1_bn3, %self_layer4_0_relu_2), kwargs = {}) | |
| %self_layer4_1_relu_2 : [#users=2] = call_module[target=self_layer4_1_relu](args = (%iadd_14,), kwargs = {}) | |
| %self_layer4_2_conv1 : [#users=1] = call_module[target=self_layer4_2_conv1](args = (%self_layer4_1_relu_2,), kwargs = {}) | |
| %self_layer4_2_bn1 : [#users=1] = call_module[target=self_layer4_2_bn1](args = (%self_layer4_2_conv1,), kwargs = {}) | |
| %self_layer4_2_relu : [#users=1] = call_module[target=self_layer4_2_relu](args = (%self_layer4_2_bn1,), kwargs = {}) | |
| %self_layer4_2_conv2 : [#users=1] = call_module[target=self_layer4_2_conv2](args = (%self_layer4_2_relu,), kwargs = {}) | |
| %self_layer4_2_bn2 : [#users=1] = call_module[target=self_layer4_2_bn2](args = (%self_layer4_2_conv2,), kwargs = {}) | |
| %self_layer4_2_relu_1 : [#users=1] = call_module[target=self_layer4_2_relu](args = (%self_layer4_2_bn2,), kwargs = {}) | |
| %self_layer4_2_conv3 : [#users=1] = call_module[target=self_layer4_2_conv3](args = (%self_layer4_2_relu_1,), kwargs = {}) | |
| %self_layer4_2_bn3 : [#users=1] = call_module[target=self_layer4_2_bn3](args = (%self_layer4_2_conv3,), kwargs = {}) | |
| %iadd_15 : [#users=1] = call_function[target=operator.iadd](args = (%self_layer4_2_bn3, %self_layer4_1_relu_2), kwargs = {}) | |
| %self_layer4_2_relu_2 : [#users=1] = call_module[target=self_layer4_2_relu](args = (%iadd_15,), kwargs = {}) | |
| %self_avgpool : [#users=1] = call_module[target=self_avgpool](args = (%self_layer4_2_relu_2,), kwargs = {}) | |
| %flatten : [#users=1] = call_function[target=torch.flatten](args = (%self_avgpool, 1), kwargs = {}) | |
| %self_fc : [#users=1] = call_module[target=self_fc](args = (%flatten,), kwargs = {}) | |
| return (self_fc,) | |
| ---split graph--- | |
| graph(): | |
| %x : torch.Tensor [#users=1] = placeholder[target=x] | |
| %submod_0 : [#users=2] = call_module[target=submod_0](args = (%x,), kwargs = {}) | |
| %getitem : [#users=1] = call_function[target=operator.getitem](args = (%submod_0, 0), kwargs = {}) | |
| %getitem_1 : [#users=1] = call_function[target=operator.getitem](args = (%submod_0, 1), kwargs = {}) | |
| %submod_1 : [#users=2] = call_module[target=submod_1](args = (%getitem, %getitem_1), kwargs = {}) | |
| %getitem_2 : [#users=1] = call_function[target=operator.getitem](args = (%submod_1, 0), kwargs = {}) | |
| %getitem_3 : [#users=1] = call_function[target=operator.getitem](args = (%submod_1, 1), kwargs = {}) | |
| %submod_2 : [#users=2] = call_module[target=submod_2](args = (%getitem_2, %getitem_3), kwargs = {}) | |
| %getitem_4 : [#users=1] = call_function[target=operator.getitem](args = (%submod_2, 0), kwargs = {}) | |
| %getitem_5 : [#users=1] = call_function[target=operator.getitem](args = (%submod_2, 1), kwargs = {}) | |
| %submod_3 : [#users=1] = call_module[target=submod_3](args = (%getitem_4, %getitem_5), kwargs = {}) | |
| %submod_4 : [#users=1] = call_module[target=submod_4](args = (%submod_3,), kwargs = {}) | |
| return (submod_4,) | |
| --- graph--- | |
| graph(): | |
| %x : torch.Tensor [#users=1] = placeholder[target=x] | |
| %submod_0 : [#users=2] = call_module[target=submod_0](args = (%x,), kwargs = {}) | |
| %getitem : [#users=1] = call_function[target=operator.getitem](args = (%submod_0, 0), kwargs = {}) | |
| %getitem_1 : [#users=1] = call_function[target=operator.getitem](args = (%submod_0, 1), kwargs = {}) | |
| %submod_1 : [#users=2] = call_module[target=submod_1](args = (%getitem, %getitem_1), kwargs = {}) | |
| %getitem_2 : [#users=1] = call_function[target=operator.getitem](args = (%submod_1, 0), kwargs = {}) | |
| %getitem_3 : [#users=1] = call_function[target=operator.getitem](args = (%submod_1, 1), kwargs = {}) | |
| %submod_2 : [#users=2] = call_module[target=submod_2](args = (%getitem_2, %getitem_3), kwargs = {}) | |
| %getitem_4 : [#users=1] = call_function[target=operator.getitem](args = (%submod_2, 0), kwargs = {}) | |
| %getitem_5 : [#users=1] = call_function[target=operator.getitem](args = (%submod_2, 1), kwargs = {}) | |
| %submod_3 : [#users=1] = call_module[target=submod_3](args = (%getitem_4, %getitem_5), kwargs = {}) | |
| %submod_4 : [#users=1] = call_module[target=submod_4](args = (%submod_3,), kwargs = {}) | |
| return (submod_4,) | |
| ---submod_0 graph--- | |
| graph(): | |
| %x : [#users=1] = placeholder[target=x] | |
| %self_conv1 : [#users=1] = call_module[target=self_conv1](args = (%x,), kwargs = {}) | |
| %self_bn1 : [#users=1] = call_module[target=self_bn1](args = (%self_conv1,), kwargs = {}) | |
| %self_relu : [#users=1] = call_module[target=self_relu](args = (%self_bn1,), kwargs = {}) | |
| %self_maxpool : [#users=2] = call_module[target=self_maxpool](args = (%self_relu,), kwargs = {}) | |
| %self_layer1_0_conv1 : [#users=1] = call_module[target=self_layer1_0_conv1](args = (%self_maxpool,), kwargs = {}) | |
| %self_layer1_0_bn1 : [#users=1] = call_module[target=self_layer1_0_bn1](args = (%self_layer1_0_conv1,), kwargs = {}) | |
| %self_layer1_0_relu : [#users=1] = call_module[target=self_layer1_0_relu](args = (%self_layer1_0_bn1,), kwargs = {}) | |
| %self_layer1_0_conv2 : [#users=1] = call_module[target=self_layer1_0_conv2](args = (%self_layer1_0_relu,), kwargs = {}) | |
| %self_layer1_0_bn2 : [#users=1] = call_module[target=self_layer1_0_bn2](args = (%self_layer1_0_conv2,), kwargs = {}) | |
| %self_layer1_0_relu_1 : [#users=1] = call_module[target=self_layer1_0_relu](args = (%self_layer1_0_bn2,), kwargs = {}) | |
| %self_layer1_0_conv3 : [#users=1] = call_module[target=self_layer1_0_conv3](args = (%self_layer1_0_relu_1,), kwargs = {}) | |
| %self_layer1_0_bn3 : [#users=1] = call_module[target=self_layer1_0_bn3](args = (%self_layer1_0_conv3,), kwargs = {}) | |
| %self_layer1_0_downsample_0 : [#users=1] = call_module[target=self_layer1_0_downsample_0](args = (%self_maxpool,), kwargs = {}) | |
| %self_layer1_0_downsample_1 : [#users=1] = call_module[target=self_layer1_0_downsample_1](args = (%self_layer1_0_downsample_0,), kwargs = {}) | |
| %iadd : [#users=1] = call_function[target=operator.iadd](args = (%self_layer1_0_bn3, %self_layer1_0_downsample_1), kwargs = {}) | |
| %self_layer1_0_relu_2 : [#users=2] = call_module[target=self_layer1_0_relu](args = (%iadd,), kwargs = {}) | |
| %self_layer1_1_conv1 : [#users=1] = call_module[target=self_layer1_1_conv1](args = (%self_layer1_0_relu_2,), kwargs = {}) | |
| %self_layer1_1_bn1 : [#users=1] = call_module[target=self_layer1_1_bn1](args = (%self_layer1_1_conv1,), kwargs = {}) | |
| %self_layer1_1_relu : [#users=1] = call_module[target=self_layer1_1_relu](args = (%self_layer1_1_bn1,), kwargs = {}) | |
| %self_layer1_1_conv2 : [#users=1] = call_module[target=self_layer1_1_conv2](args = (%self_layer1_1_relu,), kwargs = {}) | |
| %self_layer1_1_bn2 : [#users=1] = call_module[target=self_layer1_1_bn2](args = (%self_layer1_1_conv2,), kwargs = {}) | |
| %self_layer1_1_relu_1 : [#users=1] = call_module[target=self_layer1_1_relu](args = (%self_layer1_1_bn2,), kwargs = {}) | |
| %self_layer1_1_conv3 : [#users=1] = call_module[target=self_layer1_1_conv3](args = (%self_layer1_1_relu_1,), kwargs = {}) | |
| %self_layer1_1_bn3 : [#users=1] = call_module[target=self_layer1_1_bn3](args = (%self_layer1_1_conv3,), kwargs = {}) | |
| %iadd_1 : [#users=1] = call_function[target=operator.iadd](args = (%self_layer1_1_bn3, %self_layer1_0_relu_2), kwargs = {}) | |
| %self_layer1_1_relu_2 : [#users=2] = call_module[target=self_layer1_1_relu](args = (%iadd_1,), kwargs = {}) | |
| %self_layer1_2_conv1 : [#users=1] = call_module[target=self_layer1_2_conv1](args = (%self_layer1_1_relu_2,), kwargs = {}) | |
| %self_layer1_2_bn1 : [#users=1] = call_module[target=self_layer1_2_bn1](args = (%self_layer1_2_conv1,), kwargs = {}) | |
| %self_layer1_2_relu : [#users=1] = call_module[target=self_layer1_2_relu](args = (%self_layer1_2_bn1,), kwargs = {}) | |
| %self_layer1_2_conv2 : [#users=1] = call_module[target=self_layer1_2_conv2](args = (%self_layer1_2_relu,), kwargs = {}) | |
| %self_layer1_2_bn2 : [#users=1] = call_module[target=self_layer1_2_bn2](args = (%self_layer1_2_conv2,), kwargs = {}) | |
| %self_layer1_2_relu_1 : [#users=1] = call_module[target=self_layer1_2_relu](args = (%self_layer1_2_bn2,), kwargs = {}) | |
| %self_layer1_2_conv3 : [#users=1] = call_module[target=self_layer1_2_conv3](args = (%self_layer1_2_relu_1,), kwargs = {}) | |
| %self_layer1_2_bn3 : [#users=1] = call_module[target=self_layer1_2_bn3](args = (%self_layer1_2_conv3,), kwargs = {}) | |
| %iadd_2 : [#users=1] = call_function[target=operator.iadd](args = (%self_layer1_2_bn3, %self_layer1_1_relu_2), kwargs = {}) | |
| %self_layer1_2_relu_2 : [#users=2] = call_module[target=self_layer1_2_relu](args = (%iadd_2,), kwargs = {}) | |
| %self_layer2_0_conv1 : [#users=1] = call_module[target=self_layer2_0_conv1](args = (%self_layer1_2_relu_2,), kwargs = {}) | |
| %self_layer2_0_bn1 : [#users=1] = call_module[target=self_layer2_0_bn1](args = (%self_layer2_0_conv1,), kwargs = {}) | |
| %self_layer2_0_relu : [#users=1] = call_module[target=self_layer2_0_relu](args = (%self_layer2_0_bn1,), kwargs = {}) | |
| %self_layer2_0_conv2 : [#users=1] = call_module[target=self_layer2_0_conv2](args = (%self_layer2_0_relu,), kwargs = {}) | |
| %self_layer2_0_bn2 : [#users=1] = call_module[target=self_layer2_0_bn2](args = (%self_layer2_0_conv2,), kwargs = {}) | |
| %self_layer2_0_relu_1 : [#users=1] = call_module[target=self_layer2_0_relu](args = (%self_layer2_0_bn2,), kwargs = {}) | |
| %self_layer2_0_conv3 : [#users=1] = call_module[target=self_layer2_0_conv3](args = (%self_layer2_0_relu_1,), kwargs = {}) | |
| %self_layer2_0_bn3 : [#users=1] = call_module[target=self_layer2_0_bn3](args = (%self_layer2_0_conv3,), kwargs = {}) | |
| %self_layer2_0_downsample_0 : [#users=1] = call_module[target=self_layer2_0_downsample_0](args = (%self_layer1_2_relu_2,), kwargs = {}) | |
| %self_layer2_0_downsample_1 : [#users=1] = call_module[target=self_layer2_0_downsample_1](args = (%self_layer2_0_downsample_0,), kwargs = {}) | |
| %iadd_3 : [#users=1] = call_function[target=operator.iadd](args = (%self_layer2_0_bn3, %self_layer2_0_downsample_1), kwargs = {}) | |
| %self_layer2_0_relu_2 : [#users=2] = call_module[target=self_layer2_0_relu](args = (%iadd_3,), kwargs = {}) | |
| %self_layer2_1_conv1 : [#users=1] = call_module[target=self_layer2_1_conv1](args = (%self_layer2_0_relu_2,), kwargs = {}) | |
| %self_layer2_1_bn1 : [#users=1] = call_module[target=self_layer2_1_bn1](args = (%self_layer2_1_conv1,), kwargs = {}) | |
| %self_layer2_1_relu : [#users=1] = call_module[target=self_layer2_1_relu](args = (%self_layer2_1_bn1,), kwargs = {}) | |
| %self_layer2_1_conv2 : [#users=1] = call_module[target=self_layer2_1_conv2](args = (%self_layer2_1_relu,), kwargs = {}) | |
| %self_layer2_1_bn2 : [#users=1] = call_module[target=self_layer2_1_bn2](args = (%self_layer2_1_conv2,), kwargs = {}) | |
| %self_layer2_1_relu_1 : [#users=1] = call_module[target=self_layer2_1_relu](args = (%self_layer2_1_bn2,), kwargs = {}) | |
| %self_layer2_1_conv3 : [#users=1] = call_module[target=self_layer2_1_conv3](args = (%self_layer2_1_relu_1,), kwargs = {}) | |
| %self_layer2_1_bn3 : [#users=1] = call_module[target=self_layer2_1_bn3](args = (%self_layer2_1_conv3,), kwargs = {}) | |
| %iadd_4 : [#users=1] = call_function[target=operator.iadd](args = (%self_layer2_1_bn3, %self_layer2_0_relu_2), kwargs = {}) | |
| %self_layer2_1_relu_2 : [#users=2] = call_module[target=self_layer2_1_relu](args = (%iadd_4,), kwargs = {}) | |
| %self_layer2_2_conv1 : [#users=1] = call_module[target=self_layer2_2_conv1](args = (%self_layer2_1_relu_2,), kwargs = {}) | |
| %self_layer2_2_bn1 : [#users=1] = call_module[target=self_layer2_2_bn1](args = (%self_layer2_2_conv1,), kwargs = {}) | |
| %self_layer2_2_relu : [#users=1] = call_module[target=self_layer2_2_relu](args = (%self_layer2_2_bn1,), kwargs = {}) | |
| %self_layer2_2_conv2 : [#users=1] = call_module[target=self_layer2_2_conv2](args = (%self_layer2_2_relu,), kwargs = {}) | |
| %self_layer2_2_bn2 : [#users=1] = call_module[target=self_layer2_2_bn2](args = (%self_layer2_2_conv2,), kwargs = {}) | |
| %self_layer2_2_relu_1 : [#users=1] = call_module[target=self_layer2_2_relu](args = (%self_layer2_2_bn2,), kwargs = {}) | |
| %self_layer2_2_conv3 : [#users=1] = call_module[target=self_layer2_2_conv3](args = (%self_layer2_2_relu_1,), kwargs = {}) | |
| %self_layer2_2_bn3 : [#users=1] = call_module[target=self_layer2_2_bn3](args = (%self_layer2_2_conv3,), kwargs = {}) | |
| %iadd_5 : [#users=1] = call_function[target=operator.iadd](args = (%self_layer2_2_bn3, %self_layer2_1_relu_2), kwargs = {}) | |
| %self_layer2_2_relu_2 : [#users=2] = call_module[target=self_layer2_2_relu](args = (%iadd_5,), kwargs = {}) | |
| %self_layer2_3_conv1 : [#users=1] = call_module[target=self_layer2_3_conv1](args = (%self_layer2_2_relu_2,), kwargs = {}) | |
| %self_layer2_3_bn1 : [#users=1] = call_module[target=self_layer2_3_bn1](args = (%self_layer2_3_conv1,), kwargs = {}) | |
| %self_layer2_3_relu : [#users=1] = call_module[target=self_layer2_3_relu](args = (%self_layer2_3_bn1,), kwargs = {}) | |
| %self_layer2_3_conv2 : [#users=1] = call_module[target=self_layer2_3_conv2](args = (%self_layer2_3_relu,), kwargs = {}) | |
| %self_layer2_3_bn2 : [#users=1] = call_module[target=self_layer2_3_bn2](args = (%self_layer2_3_conv2,), kwargs = {}) | |
| %self_layer2_3_relu_1 : [#users=1] = call_module[target=self_layer2_3_relu](args = (%self_layer2_3_bn2,), kwargs = {}) | |
| %self_layer2_3_conv3 : [#users=1] = call_module[target=self_layer2_3_conv3](args = (%self_layer2_3_relu_1,), kwargs = {}) | |
| %self_layer2_3_bn3 : [#users=1] = call_module[target=self_layer2_3_bn3](args = (%self_layer2_3_conv3,), kwargs = {}) | |
| %iadd_6 : [#users=1] = call_function[target=operator.iadd](args = (%self_layer2_3_bn3, %self_layer2_2_relu_2), kwargs = {}) | |
| %self_layer2_3_relu_2 : [#users=2] = call_module[target=self_layer2_3_relu](args = (%iadd_6,), kwargs = {}) | |
| %self_layer3_0_conv1 : [#users=1] = call_module[target=self_layer3_0_conv1](args = (%self_layer2_3_relu_2,), kwargs = {}) | |
| %self_layer3_0_bn1 : [#users=1] = call_module[target=self_layer3_0_bn1](args = (%self_layer3_0_conv1,), kwargs = {}) | |
| %self_layer3_0_relu : [#users=1] = call_module[target=self_layer3_0_relu](args = (%self_layer3_0_bn1,), kwargs = {}) | |
| %self_layer3_0_conv2 : [#users=1] = call_module[target=self_layer3_0_conv2](args = (%self_layer3_0_relu,), kwargs = {}) | |
| %self_layer3_0_bn2 : [#users=1] = call_module[target=self_layer3_0_bn2](args = (%self_layer3_0_conv2,), kwargs = {}) | |
| %self_layer3_0_relu_1 : [#users=1] = call_module[target=self_layer3_0_relu](args = (%self_layer3_0_bn2,), kwargs = {}) | |
| %self_layer3_0_conv3 : [#users=1] = call_module[target=self_layer3_0_conv3](args = (%self_layer3_0_relu_1,), kwargs = {}) | |
| %self_layer3_0_bn3 : [#users=1] = call_module[target=self_layer3_0_bn3](args = (%self_layer3_0_conv3,), kwargs = {}) | |
| return (self_layer2_3_relu_2, self_layer3_0_bn3) | |
| ---submod_1 graph--- | |
| graph(): | |
| %self_layer2_3_relu_2 : [#users=1] = placeholder[target=self_layer2_3_relu_2] | |
| %self_layer3_0_bn3 : [#users=1] = placeholder[target=self_layer3_0_bn3] | |
| %self_layer3_0_downsample_0 : [#users=1] = call_module[target=self_layer3_0_downsample_0](args = (%self_layer2_3_relu_2,), kwargs = {}) | |
| %self_layer3_0_downsample_1 : [#users=1] = call_module[target=self_layer3_0_downsample_1](args = (%self_layer3_0_downsample_0,), kwargs = {}) | |
| %iadd : [#users=1] = call_function[target=operator.iadd](args = (%self_layer3_0_bn3, %self_layer3_0_downsample_1), kwargs = {}) | |
| %self_layer3_0_relu : [#users=2] = call_module[target=self_layer3_0_relu](args = (%iadd,), kwargs = {}) | |
| %self_layer3_1_conv1 : [#users=1] = call_module[target=self_layer3_1_conv1](args = (%self_layer3_0_relu,), kwargs = {}) | |
| %self_layer3_1_bn1 : [#users=1] = call_module[target=self_layer3_1_bn1](args = (%self_layer3_1_conv1,), kwargs = {}) | |
| %self_layer3_1_relu : [#users=1] = call_module[target=self_layer3_1_relu](args = (%self_layer3_1_bn1,), kwargs = {}) | |
| %self_layer3_1_conv2 : [#users=1] = call_module[target=self_layer3_1_conv2](args = (%self_layer3_1_relu,), kwargs = {}) | |
| %self_layer3_1_bn2 : [#users=1] = call_module[target=self_layer3_1_bn2](args = (%self_layer3_1_conv2,), kwargs = {}) | |
| %self_layer3_1_relu_1 : [#users=1] = call_module[target=self_layer3_1_relu](args = (%self_layer3_1_bn2,), kwargs = {}) | |
| %self_layer3_1_conv3 : [#users=1] = call_module[target=self_layer3_1_conv3](args = (%self_layer3_1_relu_1,), kwargs = {}) | |
| %self_layer3_1_bn3 : [#users=1] = call_module[target=self_layer3_1_bn3](args = (%self_layer3_1_conv3,), kwargs = {}) | |
| %iadd_1 : [#users=1] = call_function[target=operator.iadd](args = (%self_layer3_1_bn3, %self_layer3_0_relu), kwargs = {}) | |
| %self_layer3_1_relu_2 : [#users=2] = call_module[target=self_layer3_1_relu](args = (%iadd_1,), kwargs = {}) | |
| %self_layer3_2_conv1 : [#users=1] = call_module[target=self_layer3_2_conv1](args = (%self_layer3_1_relu_2,), kwargs = {}) | |
| %self_layer3_2_bn1 : [#users=1] = call_module[target=self_layer3_2_bn1](args = (%self_layer3_2_conv1,), kwargs = {}) | |
| %self_layer3_2_relu : [#users=1] = call_module[target=self_layer3_2_relu](args = (%self_layer3_2_bn1,), kwargs = {}) | |
| %self_layer3_2_conv2 : [#users=1] = call_module[target=self_layer3_2_conv2](args = (%self_layer3_2_relu,), kwargs = {}) | |
| %self_layer3_2_bn2 : [#users=1] = call_module[target=self_layer3_2_bn2](args = (%self_layer3_2_conv2,), kwargs = {}) | |
| %self_layer3_2_relu_1 : [#users=1] = call_module[target=self_layer3_2_relu](args = (%self_layer3_2_bn2,), kwargs = {}) | |
| %self_layer3_2_conv3 : [#users=1] = call_module[target=self_layer3_2_conv3](args = (%self_layer3_2_relu_1,), kwargs = {}) | |
| %self_layer3_2_bn3 : [#users=1] = call_module[target=self_layer3_2_bn3](args = (%self_layer3_2_conv3,), kwargs = {}) | |
| %iadd_2 : [#users=1] = call_function[target=operator.iadd](args = (%self_layer3_2_bn3, %self_layer3_1_relu_2), kwargs = {}) | |
| %self_layer3_2_relu_2 : [#users=2] = call_module[target=self_layer3_2_relu](args = (%iadd_2,), kwargs = {}) | |
| %self_layer3_3_conv1 : [#users=1] = call_module[target=self_layer3_3_conv1](args = (%self_layer3_2_relu_2,), kwargs = {}) | |
| %self_layer3_3_bn1 : [#users=1] = call_module[target=self_layer3_3_bn1](args = (%self_layer3_3_conv1,), kwargs = {}) | |
| %self_layer3_3_relu : [#users=1] = call_module[target=self_layer3_3_relu](args = (%self_layer3_3_bn1,), kwargs = {}) | |
| %self_layer3_3_conv2 : [#users=1] = call_module[target=self_layer3_3_conv2](args = (%self_layer3_3_relu,), kwargs = {}) | |
| %self_layer3_3_bn2 : [#users=1] = call_module[target=self_layer3_3_bn2](args = (%self_layer3_3_conv2,), kwargs = {}) | |
| %self_layer3_3_relu_1 : [#users=1] = call_module[target=self_layer3_3_relu](args = (%self_layer3_3_bn2,), kwargs = {}) | |
| %self_layer3_3_conv3 : [#users=1] = call_module[target=self_layer3_3_conv3](args = (%self_layer3_3_relu_1,), kwargs = {}) | |
| %self_layer3_3_bn3 : [#users=1] = call_module[target=self_layer3_3_bn3](args = (%self_layer3_3_conv3,), kwargs = {}) | |
| %iadd_3 : [#users=1] = call_function[target=operator.iadd](args = (%self_layer3_3_bn3, %self_layer3_2_relu_2), kwargs = {}) | |
| %self_layer3_3_relu_2 : [#users=2] = call_module[target=self_layer3_3_relu](args = (%iadd_3,), kwargs = {}) | |
| %self_layer3_4_conv1 : [#users=1] = call_module[target=self_layer3_4_conv1](args = (%self_layer3_3_relu_2,), kwargs = {}) | |
| %self_layer3_4_bn1 : [#users=1] = call_module[target=self_layer3_4_bn1](args = (%self_layer3_4_conv1,), kwargs = {}) | |
| %self_layer3_4_relu : [#users=1] = call_module[target=self_layer3_4_relu](args = (%self_layer3_4_bn1,), kwargs = {}) | |
| %self_layer3_4_conv2 : [#users=1] = call_module[target=self_layer3_4_conv2](args = (%self_layer3_4_relu,), kwargs = {}) | |
| %self_layer3_4_bn2 : [#users=1] = call_module[target=self_layer3_4_bn2](args = (%self_layer3_4_conv2,), kwargs = {}) | |
| %self_layer3_4_relu_1 : [#users=1] = call_module[target=self_layer3_4_relu](args = (%self_layer3_4_bn2,), kwargs = {}) | |
| %self_layer3_4_conv3 : [#users=1] = call_module[target=self_layer3_4_conv3](args = (%self_layer3_4_relu_1,), kwargs = {}) | |
| %self_layer3_4_bn3 : [#users=1] = call_module[target=self_layer3_4_bn3](args = (%self_layer3_4_conv3,), kwargs = {}) | |
| %iadd_4 : [#users=1] = call_function[target=operator.iadd](args = (%self_layer3_4_bn3, %self_layer3_3_relu_2), kwargs = {}) | |
| %self_layer3_4_relu_2 : [#users=2] = call_module[target=self_layer3_4_relu](args = (%iadd_4,), kwargs = {}) | |
| %self_layer3_5_conv1 : [#users=1] = call_module[target=self_layer3_5_conv1](args = (%self_layer3_4_relu_2,), kwargs = {}) | |
| %self_layer3_5_bn1 : [#users=1] = call_module[target=self_layer3_5_bn1](args = (%self_layer3_5_conv1,), kwargs = {}) | |
| %self_layer3_5_relu : [#users=1] = call_module[target=self_layer3_5_relu](args = (%self_layer3_5_bn1,), kwargs = {}) | |
| %self_layer3_5_conv2 : [#users=1] = call_module[target=self_layer3_5_conv2](args = (%self_layer3_5_relu,), kwargs = {}) | |
| %self_layer3_5_bn2 : [#users=1] = call_module[target=self_layer3_5_bn2](args = (%self_layer3_5_conv2,), kwargs = {}) | |
| %self_layer3_5_relu_1 : [#users=1] = call_module[target=self_layer3_5_relu](args = (%self_layer3_5_bn2,), kwargs = {}) | |
| %self_layer3_5_conv3 : [#users=1] = call_module[target=self_layer3_5_conv3](args = (%self_layer3_5_relu_1,), kwargs = {}) | |
| %self_layer3_5_bn3 : [#users=1] = call_module[target=self_layer3_5_bn3](args = (%self_layer3_5_conv3,), kwargs = {}) | |
| %iadd_5 : [#users=1] = call_function[target=operator.iadd](args = (%self_layer3_5_bn3, %self_layer3_4_relu_2), kwargs = {}) | |
| %self_layer3_5_relu_2 : [#users=2] = call_module[target=self_layer3_5_relu](args = (%iadd_5,), kwargs = {}) | |
| %self_layer4_0_conv1 : [#users=1] = call_module[target=self_layer4_0_conv1](args = (%self_layer3_5_relu_2,), kwargs = {}) | |
| %self_layer4_0_bn1 : [#users=1] = call_module[target=self_layer4_0_bn1](args = (%self_layer4_0_conv1,), kwargs = {}) | |
| %self_layer4_0_relu : [#users=1] = call_module[target=self_layer4_0_relu](args = (%self_layer4_0_bn1,), kwargs = {}) | |
| return (self_layer4_0_relu, self_layer3_5_relu_2) | |
| ---submod_2 graph--- | |
| graph(): | |
| %self_layer4_0_relu : [#users=1] = placeholder[target=self_layer4_0_relu] | |
| %self_layer3_5_relu_2 : [#users=1] = placeholder[target=self_layer3_5_relu_2] | |
| %self_layer4_0_conv2 : [#users=1] = call_module[target=self_layer4_0_conv2](args = (%self_layer4_0_relu,), kwargs = {}) | |
| %self_layer4_0_bn2 : [#users=1] = call_module[target=self_layer4_0_bn2](args = (%self_layer4_0_conv2,), kwargs = {}) | |
| %self_layer4_0_relu_1 : [#users=1] = call_module[target=self_layer4_0_relu](args = (%self_layer4_0_bn2,), kwargs = {}) | |
| %self_layer4_0_conv3 : [#users=1] = call_module[target=self_layer4_0_conv3](args = (%self_layer4_0_relu_1,), kwargs = {}) | |
| %self_layer4_0_bn3 : [#users=1] = call_module[target=self_layer4_0_bn3](args = (%self_layer4_0_conv3,), kwargs = {}) | |
| %self_layer4_0_downsample_0 : [#users=1] = call_module[target=self_layer4_0_downsample_0](args = (%self_layer3_5_relu_2,), kwargs = {}) | |
| %self_layer4_0_downsample_1 : [#users=1] = call_module[target=self_layer4_0_downsample_1](args = (%self_layer4_0_downsample_0,), kwargs = {}) | |
| %iadd : [#users=1] = call_function[target=operator.iadd](args = (%self_layer4_0_bn3, %self_layer4_0_downsample_1), kwargs = {}) | |
| %self_layer4_0_relu_2 : [#users=2] = call_module[target=self_layer4_0_relu](args = (%iadd,), kwargs = {}) | |
| %self_layer4_1_conv1 : [#users=1] = call_module[target=self_layer4_1_conv1](args = (%self_layer4_0_relu_2,), kwargs = {}) | |
| %self_layer4_1_bn1 : [#users=1] = call_module[target=self_layer4_1_bn1](args = (%self_layer4_1_conv1,), kwargs = {}) | |
| %self_layer4_1_relu : [#users=1] = call_module[target=self_layer4_1_relu](args = (%self_layer4_1_bn1,), kwargs = {}) | |
| return (self_layer4_1_relu, self_layer4_0_relu_2) | |
| ---submod_3 graph--- | |
| graph(): | |
| %self_layer4_1_relu : [#users=1] = placeholder[target=self_layer4_1_relu] | |
| %self_layer4_0_relu_2 : [#users=1] = placeholder[target=self_layer4_0_relu_2] | |
| %self_layer4_1_conv2 : [#users=1] = call_module[target=self_layer4_1_conv2](args = (%self_layer4_1_relu,), kwargs = {}) | |
| %self_layer4_1_bn2 : [#users=1] = call_module[target=self_layer4_1_bn2](args = (%self_layer4_1_conv2,), kwargs = {}) | |
| %self_layer4_1_relu_1 : [#users=1] = call_module[target=self_layer4_1_relu](args = (%self_layer4_1_bn2,), kwargs = {}) | |
| %self_layer4_1_conv3 : [#users=1] = call_module[target=self_layer4_1_conv3](args = (%self_layer4_1_relu_1,), kwargs = {}) | |
| %self_layer4_1_bn3 : [#users=1] = call_module[target=self_layer4_1_bn3](args = (%self_layer4_1_conv3,), kwargs = {}) | |
| %iadd : [#users=1] = call_function[target=operator.iadd](args = (%self_layer4_1_bn3, %self_layer4_0_relu_2), kwargs = {}) | |
| %self_layer4_1_relu_2 : [#users=2] = call_module[target=self_layer4_1_relu](args = (%iadd,), kwargs = {}) | |
| %self_layer4_2_conv1 : [#users=1] = call_module[target=self_layer4_2_conv1](args = (%self_layer4_1_relu_2,), kwargs = {}) | |
| %self_layer4_2_bn1 : [#users=1] = call_module[target=self_layer4_2_bn1](args = (%self_layer4_2_conv1,), kwargs = {}) | |
| %self_layer4_2_relu : [#users=1] = call_module[target=self_layer4_2_relu](args = (%self_layer4_2_bn1,), kwargs = {}) | |
| %self_layer4_2_conv2 : [#users=1] = call_module[target=self_layer4_2_conv2](args = (%self_layer4_2_relu,), kwargs = {}) | |
| %self_layer4_2_bn2 : [#users=1] = call_module[target=self_layer4_2_bn2](args = (%self_layer4_2_conv2,), kwargs = {}) | |
| %self_layer4_2_relu_1 : [#users=1] = call_module[target=self_layer4_2_relu](args = (%self_layer4_2_bn2,), kwargs = {}) | |
| %self_layer4_2_conv3 : [#users=1] = call_module[target=self_layer4_2_conv3](args = (%self_layer4_2_relu_1,), kwargs = {}) | |
| %self_layer4_2_bn3 : [#users=1] = call_module[target=self_layer4_2_bn3](args = (%self_layer4_2_conv3,), kwargs = {}) | |
| %iadd_1 : [#users=1] = call_function[target=operator.iadd](args = (%self_layer4_2_bn3, %self_layer4_1_relu_2), kwargs = {}) | |
| %self_layer4_2_relu_2 : [#users=1] = call_module[target=self_layer4_2_relu](args = (%iadd_1,), kwargs = {}) | |
| %self_avgpool : [#users=1] = call_module[target=self_avgpool](args = (%self_layer4_2_relu_2,), kwargs = {}) | |
| %flatten : [#users=1] = call_function[target=torch.flatten](args = (%self_avgpool, 1), kwargs = {}) | |
| return flatten | |
| ---submod_4 graph--- | |
| graph(): | |
| %flatten : [#users=1] = placeholder[target=flatten] | |
| %self_fc : [#users=1] = call_module[target=self_fc](args = (%flatten,), kwargs = {}) | |
| return self_fc | |
| --------------- | |
| run_node placeholder, x got args tuple() | |
| run_node call_module, submod_0 got args tuple(T[torch.Size([32, 3, 224, 224])]) | |
| still waiting for process to exit | |
| run_node call_function, <built-in function getitem> got args tuple(tuple(T[torch.Size([32, 512, 28, 28])], T[torch.Size([32, 1024, 14, 14])]), 0) | |
| run_node call_function, <built-in function getitem> got args tuple(tuple(T[torch.Size([32, 512, 28, 28])], T[torch.Size([32, 1024, 14, 14])]), 1) | |
| run_node call_module, submod_1 got args tuple(T[torch.Size([32, 512, 28, 28])], T[torch.Size([32, 1024, 14, 14])]) | |
| run_node call_function, <built-in function getitem> got args tuple(tuple(T[torch.Size([32, 512, 14, 14])], T[torch.Size([32, 1024, 14, 14])]), 0) | |
| run_node call_function, <built-in function getitem> got args tuple(tuple(T[torch.Size([32, 512, 14, 14])], T[torch.Size([32, 1024, 14, 14])]), 1) | |
| run_node call_module, submod_2 got args tuple(T[torch.Size([32, 512, 14, 14])], T[torch.Size([32, 1024, 14, 14])]) | |
| didn't get the result, queue empty | |
| run_node call_function, <built-in function getitem> got args tuple(tuple(T[torch.Size([32, 512, 7, 7])], T[torch.Size([32, 2048, 7, 7])]), 0) | |
| run_node call_function, <built-in function getitem> got args tuple(tuple(T[torch.Size([32, 512, 7, 7])], T[torch.Size([32, 2048, 7, 7])]), 1) | |
| run_node call_module, submod_3 got args tuple(T[torch.Size([32, 512, 7, 7])], T[torch.Size([32, 2048, 7, 7])]) | |
| run_node call_module, submod_4 got args tuple(T[torch.Size([32, 2048])]) | |
| run_node output, output got args tuple(tuple(T[torch.Size([32, 1000])])) | |
| ---final graph--- | |
| graph(): | |
| %x : torch.Tensor [#users=1] = placeholder[target=x] | |
| %submod_0 : [#users=2] = call_module[target=compiled_submod_0](args = (%x,), kwargs = {}) | |
| %getitem : [#users=1] = call_function[target=operator.getitem](args = (%submod_0, 0), kwargs = {}) | |
| %getitem_1 : [#users=1] = call_function[target=operator.getitem](args = (%submod_0, 1), kwargs = {}) | |
| %submod_1 : [#users=2] = call_module[target=compiled_submod_1](args = (%getitem, %getitem_1), kwargs = {}) | |
| %getitem_2 : [#users=1] = call_function[target=operator.getitem](args = (%submod_1, 0), kwargs = {}) | |
| %getitem_3 : [#users=1] = call_function[target=operator.getitem](args = (%submod_1, 1), kwargs = {}) | |
| %submod_2 : [#users=2] = call_module[target=compiled_submod_2](args = (%getitem_2, %getitem_3), kwargs = {}) | |
| %getitem_4 : [#users=1] = call_function[target=operator.getitem](args = (%submod_2, 0), kwargs = {}) | |
| %getitem_5 : [#users=1] = call_function[target=operator.getitem](args = (%submod_2, 1), kwargs = {}) | |
| %submod_3 : [#users=1] = call_module[target=compiled_submod_3](args = (%getitem_4, %getitem_5), kwargs = {}) | |
| %submod_4 : [#users=1] = call_module[target=compiled_submod_4](args = (%submod_3,), kwargs = {}) | |
| return (submod_4,) | |
| --------------- | |
| still waiting for process to exit | |
| didn't get the result, queue empty | |
| still waiting for process to exit | |
| didn't get the result, queue empty | |
| still waiting for process to exit | |
| didn't get the result, queue empty | |
| still waiting for process to exit | |
| didn't get the result, queue empty | |
| still waiting for process to exit | |
| didn't get the result, queue empty | |
| still waiting for process to exit | |
| didn't get the result, queue empty | |
| still waiting for process to exit | |
| didn't get the result, queue empty | |
| still waiting for process to exit | |
| didn't get the result, queue empty | |
| still waiting for process to exit | |
| didn't get the result, queue empty | |
| still waiting for process to exit | |
| didn't get the result, queue empty | |
| still waiting for process to exit | |
| didn't get the result, queue empty | |
| still waiting for process to exit | |
| didn't get the result, queue empty | |
| still waiting for process to exit | |
| didn't get the result, queue empty | |
| still waiting for process to exit | |
| didn't get the result, queue empty | |
| still waiting for process to exit | |
| didn't get the result, queue empty | |
| still waiting for process to exit | |
| didn't get the result, queue empty | |
| result {'latency_median': 73.5912971496582, 'latency_stdev': 0.5808285154412532} | |
| has been placed into the queue | |
| got exit 0 | |
| got the result! | |
| exit code: 0 and result: {'nodes': 1, 'model_name': 'torchbenchmark.models.resnet50.Model', 'backend': 'torchdynamo_inductor', 'has_breaks': True, 'static_graph': False, 'result': {'latency_median': 73.5912971496582, 'latency_stdev': 0.5808285154412532}} | |
| <RESULT>{"nodes": 1, "model_name": "torchbenchmark.models.resnet50.Model", "backend": "torchdynamo_inductor", "has_breaks": true, "static_graph": false, "result": {"latency_median": 73.5912971496582, "latency_stdev": 0.5808285154412532}}</RESULT> | |
| submitit INFO (2022-10-26 23:18:06,460) - Job completed successfully |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
These are resnet50 results.
The failing logs come from https://github.com/pytorch/pytorch/tree/davidberard98/repro-dynamo-ddp-resnet-issue
The success logs come from viable/strict as of Oct 26, 2022 afternoon.
The logs contain results from 3 runs:
The logs contain DDPOptimizer(debug=True) i.e. debug turned on for DDPOptimizer. In the failing logs, we get an error about mutated views coming from inductor.
But we can see by comparing the fail and success logs that the graph splits are the same (you can check with vimdiff)