James Reed jamesr66a
jamesr66a
/ standalone_tracing.patch
Last active
March 24, 2022 03:29
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| diff --git a/test/t5_test.py b/test/t5_test.py | |
| index e6b36b8..b72108f 100644 | |
| --- a/test/t5_test.py | |
| +++ b/test/t5_test.py | |
| @@ -34,4 +34,13 @@ t5_input = torch.zeros(bs, seq_length, dtype=torch.long).random_(t5.config.vocab | |
| decoder_input_ids = torch.zeros(bs, seq_length, dtype=torch.long).random_(t5.config.vocab_size) | |
| t5_output = t5(input_ids=t5_input, decoder_attention_mask=None, decoder_input_ids=decoder_input_ids) | |
| -t5_pipe_output = t5_pipe(input_ids=t5_input, decoder_attention_mask=None, decoder_input_ids=decoder_input_ids) | |
| + |
jamesr66a
/ meta_tracer.py
Last active
March 16, 2022 18:10
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| import torch | |
| import torch.fx | |
| class TestModuleWithShapeControlFlow(torch.nn.Module): | |
| def forward(self, x): | |
| # with normal symtracing, the `x.dim()` accesses would fail | |
| if x.dim() == 3: | |
| y = x[0, :, :] | |
| elif x.dim() == 4: | |
| y = x[0, :, :, :] |
jamesr66a
/ dynamic_len_patch.py
Last active
March 10, 2022 23:55
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| # a.py | |
| import torch | |
| class Foo(torch.nn.Module): | |
| def forward(self, x): | |
| return x + len(x) | |
| # b.py | |
| import torch |
jamesr66a
/ fix_hf_fx_serialization.patch
Created
March 8, 2022 00:31
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| commit b0703a2d968ccc91760ad738e9a50b3a913969a9 | |
| Author: James Reed <[email protected]> | |
| Date: Wed Mar 2 01:03:05 2022 +0000 | |
| Serialization fixes for HF tracer | |
| diff --git a/src/transformers/utils/fx.py b/src/transformers/utils/fx.py | |
| index b88ae4ae7..aeb345ad3 100644 | |
| --- a/src/transformers/utils/fx.py | |
| +++ b/src/transformers/utils/fx.py |
jamesr66a
/ deser_log.txt
Created
February 23, 2022 23:46
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| submod_2 UserRRef(RRefId = GloballyUniqueId(created_on=0, local_id=30), ForkId = GloballyUniqueId(created_on=0, local_id=31)) | |
| (46501) ^^^^ Scenario 1 (created_on=0, local_id=30) | |
| (46501) Instantiating OwnerRRef GloballyUniqueId(created_on=0, local_id=30) with future 0x7f6020007e80 | |
| (46501) Deserializing PyRRef parent: 0 rref ID: GloballyUniqueId(created_on=0, local_id=30) fork ID: GloballyUniqueId(created_on=0, local_id=33) | |
| (46501) ^^^^ Scenario 2 (created_on=0, local_id=30) | |
| (46501) Deserializing PyRRef parent: 0 rref ID: GloballyUniqueId(created_on=0, local_id=30) fork ID: GloballyUniqueId(created_on=0, local_id=38) | |
| getitem_12 (46501) ../torch/csrc/distributed/rpc/rref_context.cpp:769 GloballyUniqueId(created_on=0, local_id=30) GloballyUniqueId(created_on=2, local_id=2) | |
| (46501) ../torch/csrc/distributed/rpc/rref_context.cpp:474 GloballyUniqueId(created_on=0, local_id=30) | |
| (46501) ../torch/csrc/distributed/rpc/rref_context.cpp:769 GloballyUniqueId(created_on=0, local_id=30) GloballyUniqueId(created_on=2, local_i |
jamesr66a
/ rref_destruction_stack.txt
Created
February 23, 2022 23:20
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| #0 0x00007f10250bbd1d in __cxa_throw () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6 | |
| #1 0x00007f101047603d in torch::distributed::rpc::RRefContext::delForkOfOwner(torch::distributed::rpc::GloballyUniqueId const&, torch::distributed::rpc::GloballyUniqueId const&) () from /fsx/users/jamesreed/pytorch/torch/lib/libtorch_cpu.so | |
| #2 0x00007f1010477dc2 in torch::distributed::rpc::RRefContext::notifyOwnerAndParentOfFork(torch::distributed::rpc::GloballyUniqueId const&, short, c10::intrusive_ptr<torch::distributed::rpc::RRef, c10::detail::intrusive_target_default_null_type<torch::distributed::rpc::RRef> > const&) () from /fsx/users/jamesreed/pytorch/torch/lib/libtorch_cpu.so | |
| #3 0x00007f10166da01a in torch::distributed::rpc::PyRRef::unpickle(pybind11::tuple const&) () | |
| from /fsx/users/jamesreed/pytorch/torch/lib/libtorch_python.so | |
| #4 0x00007f10166d1ef7 in void pybind11::cpp_function::initialize<torch::distributed::rpc::PyRRef (*&)(pybind11::tuple const&), torch::distributed::rpc::PyRRef, pybind11::tuple const&, |
jamesr66a
/ keep_rref_alive.txt
Created
February 23, 2022 23:13
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| (16399) ^^^^ Scenario 1 (created_on=0, local_id=30) | |
| (16399) Instantiating OwnerRRef GloballyUniqueId(created_on=0, local_id=30) with future 0x7f19c4007910 | |
| (16399) ../torch/csrc/distributed/rpc/rref_context.cpp:769 GloballyUniqueId(created_on=0, local_id=30) GloballyUniqueId(created_on=2, local_id=2) | |
| (16399) ../torch/csrc/distributed/rpc/rref_context.cpp:474 GloballyUniqueId(created_on=0, local_id=30) | |
| (16399) ^^^^ Scenario 2 (created_on=0, local_id=30) | |
| ^^^^^ (16399) ../torch/csrc/distributed/rpc/rref_context.cpp:769 GloballyUniqueId(created_on=0, local_id=30) GloballyUniqueId(created_on=2, local_id=8) | |
| (16399) ../torch/csrc/distributed/rpc/rref_context.cpp:474 GloballyUniqueId(created_on=0, local_id=30) | |
| (16399) ^^^^ Scenario 2 (created_on=0, local_id=30) | |
| (16399) ^^^^ Scenario 2 (created_on=0, local_id=30) | |
| (16399) ../torch/csrc/distributed/rpc/rref_context.cpp:814 GloballyUniqueId(created_on=0, local_id=30) GloballyUniqueId(created_on=2, local_id=2) |
jamesr66a
/ rref_context_log.txt
Created
February 23, 2022 21:54
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| WARNING:torch.distributed.run: | |
| ***************************************** | |
| Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. | |
| ***************************************** | |
| REPLICATE config: False -> MultiUseParameterConfig.TRANSMIT | |
| GraphModule( | |
| (submod_0): GraphModule() | |
| (submod_1): GraphModule() | |
| (submod_2): GraphModule() | |
| (_loss): MSELoss() |
jamesr66a
/ rref_context_edits.patch
Created
February 23, 2022 21:53
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| diff --git a/torch/csrc/distributed/rpc/rref_context.cpp b/torch/csrc/distributed/rpc/rref_context.cpp | |
| index 004e9422be..28b2bbc5c2 100644 | |
| --- a/torch/csrc/distributed/rpc/rref_context.cpp | |
| +++ b/torch/csrc/distributed/rpc/rref_context.cpp | |
| @@ -4,6 +4,9 @@ | |
| #include <sstream> | |
| +#include <iostream> | |
| +#include <unistd.h> |
jamesr66a
/ rref_logging.txt
Created
February 23, 2022 21:26
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| WARNING:torch.distributed.run: | |
| ***************************************** | |
| Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. | |
| ***************************************** | |
| [W socket.cpp:701] The server socket on [ip-10-200-31-5.ec2.internal]:46879 is not yet listening (errno: 111 - Connection refused), will retry. | |
| [W socket.cpp:701] The server socket on [ip-10-200-31-5.ec2.internal]:46879 is not yet listening (errno: 111 - Connection refused), will retry. | |
| [W socket.cpp:701] The server socket on [ip-10-200-31-5.ec2.internal]:46879 is not yet listening (errno: 111 - Connection refused), will retry. | |
| [W socket.cpp:701] The server socket on [ip-10-200-31-5.ec2.internal]:46879 is not yet listening (errno: 111 - Connection refused), will retry. | |
| REPLICATE config: False -> MultiUseParameterConfig.TRANSMIT | |
| GraphModule( |