activation checkpoint error when debug=True with python stacktrace only

gistfile1.txt

torch.utils.checkpoint.CheckpointError: torch.utils.checkpoint: Recomputed values for the following tensors have different metadata than during the forward pass.
tensor at position 1:
saved metadata: {'shape': torch.Size([1]), 'dtype': torch.float32, 'device': device(type='cpu')}
recomputed metadata: {'shape': torch.Size([2]), 'dtype': torch.float32, 'device': device(type='cpu')}


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/local/pytorch1/test/test_autograd.py", line 5692, in test_checkpoint_detects_non_determinism
    out.backward()
  File "/local/pytorch1/torch/_tensor.py", line 488, in backward
    torch.autograd.backward(
  File "/local/pytorch1/torch/autograd/__init__.py", line 204, in backward
    Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
  File "/local/pytorch1/torch/utils/checkpoint.py", line 1065, in unpack_hook_with_error_cb
    frame.unpack_error_cb(e)
  File "/local/pytorch1/torch/utils/checkpoint.py", line 936, in unpack_error_cb
    raise CheckpointError(
torch.utils.checkpoint.CheckpointError:  An error happened while unpacking tensors; dumping logs of latest computation
because you passed `debug=True` to `torch.utils.checkpoint.checkpoint()`.
Scroll all the way down for guidance on how to navigate these logs.

+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~+
|        1. Stack traces of the operators that ran in the original forward     |
+------------------------------------------------------------------------------+

$1: f32[1] = torch._ops.aten.sin.default($0)   (1 of 2 in original)

/local/pytorch1/test/test_autograd.py:5648:save_2_tensors
/local/pytorch1/test/test_autograd.py:5658:fn
/local/pytorch1/torch/utils/checkpoint.py:1177:_checkpoint_without_reentrant
/local/pytorch1/torch/utils/checkpoint.py:431:checkpoint
/local/pytorch1/test/test_autograd.py:5691:test_checkpoint_detects_non_determinism
/opt/miniconda3/envs/pytorch1/lib/python3.9/unittest/case.py:550:_callTestMethod
/opt/miniconda3/envs/pytorch1/lib/python3.9/unittest/case.py:592:run
/local/pytorch1/torch/testing/_internal/common_utils.py:2248:_run_with_retry
/local/pytorch1/torch/testing/_internal/common_utils.py:2319:run
/opt/miniconda3/envs/pytorch1/lib/python3.9/unittest/case.py:651:__call__
/opt/miniconda3/envs/pytorch1/lib/python3.9/unittest/suite.py:122:run
/opt/miniconda3/envs/pytorch1/lib/python3.9/unittest/suite.py:84:__call__
/opt/miniconda3/envs/pytorch1/lib/python3.9/unittest/suite.py:122:run
/opt/miniconda3/envs/pytorch1/lib/python3.9/unittest/suite.py:84:__call__
/opt/miniconda3/envs/pytorch1/lib/python3.9/unittest/runner.py:184:run
/opt/miniconda3/envs/pytorch1/lib/python3.9/unittest/main.py:271:runTests
/opt/miniconda3/envs/pytorch1/lib/python3.9/unittest/main.py:101:__init__
/local/pytorch1/torch/testing/_internal/common_utils.py:892:run_tests
/local/pytorch1/test/test_autograd.py:11248:<module>

$2: f32[1] = torch._ops.aten.exp.default($1)   (2 of 2 in original)

/local/pytorch1/test/test_autograd.py:5648:save_2_tensors
/local/pytorch1/test/test_autograd.py:5658:fn
/local/pytorch1/torch/utils/checkpoint.py:1177:_checkpoint_without_reentrant
/local/pytorch1/torch/utils/checkpoint.py:431:checkpoint
/local/pytorch1/test/test_autograd.py:5691:test_checkpoint_detects_non_determinism
/opt/miniconda3/envs/pytorch1/lib/python3.9/unittest/case.py:550:_callTestMethod
/opt/miniconda3/envs/pytorch1/lib/python3.9/unittest/case.py:592:run
/local/pytorch1/torch/testing/_internal/common_utils.py:2248:_run_with_retry
/local/pytorch1/torch/testing/_internal/common_utils.py:2319:run
/opt/miniconda3/envs/pytorch1/lib/python3.9/unittest/case.py:651:__call__
/opt/miniconda3/envs/pytorch1/lib/python3.9/unittest/suite.py:122:run
/opt/miniconda3/envs/pytorch1/lib/python3.9/unittest/suite.py:84:__call__
/opt/miniconda3/envs/pytorch1/lib/python3.9/unittest/suite.py:122:run
/opt/miniconda3/envs/pytorch1/lib/python3.9/unittest/suite.py:84:__call__
/opt/miniconda3/envs/pytorch1/lib/python3.9/unittest/runner.py:184:run
/opt/miniconda3/envs/pytorch1/lib/python3.9/unittest/main.py:271:runTests
/opt/miniconda3/envs/pytorch1/lib/python3.9/unittest/main.py:101:__init__
/local/pytorch1/torch/testing/_internal/common_utils.py:892:run_tests
/local/pytorch1/test/test_autograd.py:11248:<module>


+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~+
|        2. Stack traces of the operators that ran during recomputation        |
+------------------------------------------------------------------------------+

$1: f32[1] = torch._ops.aten.detach.default($0)   (1 of 8 in recompute)

/local/pytorch1/torch/utils/checkpoint.py:998:pack_hook
/local/pytorch1/test/test_autograd.py:5651:save_2_tensors_alt
/local/pytorch1/test/test_autograd.py:5660:fn
/local/pytorch1/torch/utils/checkpoint.py:1161:recompute_fn
/local/pytorch1/torch/utils/checkpoint.py:1041:unpack_hook
/local/pytorch1/torch/utils/checkpoint.py:1063:unpack_hook_with_error_cb
/local/pytorch1/torch/autograd/__init__.py:204:backward
/local/pytorch1/torch/_tensor.py:488:backward
/local/pytorch1/test/test_autograd.py:5692:test_checkpoint_detects_non_determinism
/opt/miniconda3/envs/pytorch1/lib/python3.9/unittest/case.py:550:_callTestMethod
/opt/miniconda3/envs/pytorch1/lib/python3.9/unittest/case.py:592:run
/local/pytorch1/torch/testing/_internal/common_utils.py:2248:_run_with_retry
/local/pytorch1/torch/testing/_internal/common_utils.py:2319:run
/opt/miniconda3/envs/pytorch1/lib/python3.9/unittest/case.py:651:__call__
/opt/miniconda3/envs/pytorch1/lib/python3.9/unittest/suite.py:122:run
/opt/miniconda3/envs/pytorch1/lib/python3.9/unittest/suite.py:84:__call__
/opt/miniconda3/envs/pytorch1/lib/python3.9/unittest/suite.py:122:run
/opt/miniconda3/envs/pytorch1/lib/python3.9/unittest/suite.py:84:__call__
/opt/miniconda3/envs/pytorch1/lib/python3.9/unittest/runner.py:184:run
/opt/miniconda3/envs/pytorch1/lib/python3.9/unittest/main.py:271:runTests
/opt/miniconda3/envs/pytorch1/lib/python3.9/unittest/main.py:101:__init__
/local/pytorch1/torch/testing/_internal/common_utils.py:892:run_tests
/local/pytorch1/test/test_autograd.py:11248:<module>

$2: f32[1] = torch._ops.aten.detach.default($1)   (2 of 8 in recompute)

/local/pytorch1/torch/utils/checkpoint.py:998:pack_hook
/local/pytorch1/test/test_autograd.py:5651:save_2_tensors_alt
/local/pytorch1/test/test_autograd.py:5660:fn
/local/pytorch1/torch/utils/checkpoint.py:1161:recompute_fn
/local/pytorch1/torch/utils/checkpoint.py:1041:unpack_hook
/local/pytorch1/torch/utils/checkpoint.py:1063:unpack_hook_with_error_cb
/local/pytorch1/torch/autograd/__init__.py:204:backward
/local/pytorch1/torch/_tensor.py:488:backward
/local/pytorch1/test/test_autograd.py:5692:test_checkpoint_detects_non_determinism
/opt/miniconda3/envs/pytorch1/lib/python3.9/unittest/case.py:550:_callTestMethod
/opt/miniconda3/envs/pytorch1/lib/python3.9/unittest/case.py:592:run
/local/pytorch1/torch/testing/_internal/common_utils.py:2248:_run_with_retry
/local/pytorch1/torch/testing/_internal/common_utils.py:2319:run
/opt/miniconda3/envs/pytorch1/lib/python3.9/unittest/case.py:651:__call__
/opt/miniconda3/envs/pytorch1/lib/python3.9/unittest/suite.py:122:run
/opt/miniconda3/envs/pytorch1/lib/python3.9/unittest/suite.py:84:__call__
/opt/miniconda3/envs/pytorch1/lib/python3.9/unittest/suite.py:122:run
/opt/miniconda3/envs/pytorch1/lib/python3.9/unittest/suite.py:84:__call__
/opt/miniconda3/envs/pytorch1/lib/python3.9/unittest/runner.py:184:run
/opt/miniconda3/envs/pytorch1/lib/python3.9/unittest/main.py:271:runTests
/opt/miniconda3/envs/pytorch1/lib/python3.9/unittest/main.py:101:__init__
/local/pytorch1/torch/testing/_internal/common_utils.py:892:run_tests
/local/pytorch1/test/test_autograd.py:11248:<module>

$3: f32[1] = torch._ops.aten.detach.default($0)   (3 of 8 in recompute)

/local/pytorch1/torch/utils/checkpoint.py:1005:pack_hook
/local/pytorch1/test/test_autograd.py:5651:save_2_tensors_alt
/local/pytorch1/test/test_autograd.py:5660:fn
/local/pytorch1/torch/utils/checkpoint.py:1161:recompute_fn
/local/pytorch1/torch/utils/checkpoint.py:1041:unpack_hook
/local/pytorch1/torch/utils/checkpoint.py:1063:unpack_hook_with_error_cb
/local/pytorch1/torch/autograd/__init__.py:204:backward
/local/pytorch1/torch/_tensor.py:488:backward
/local/pytorch1/test/test_autograd.py:5692:test_checkpoint_detects_non_determinism
/opt/miniconda3/envs/pytorch1/lib/python3.9/unittest/case.py:550:_callTestMethod
/opt/miniconda3/envs/pytorch1/lib/python3.9/unittest/case.py:592:run
/local/pytorch1/torch/testing/_internal/common_utils.py:2248:_run_with_retry
/local/pytorch1/torch/testing/_internal/common_utils.py:2319:run
/opt/miniconda3/envs/pytorch1/lib/python3.9/unittest/case.py:651:__call__
/opt/miniconda3/envs/pytorch1/lib/python3.9/unittest/suite.py:122:run
/opt/miniconda3/envs/pytorch1/lib/python3.9/unittest/suite.py:84:__call__
/opt/miniconda3/envs/pytorch1/lib/python3.9/unittest/suite.py:122:run
/opt/miniconda3/envs/pytorch1/lib/python3.9/unittest/suite.py:84:__call__
/opt/miniconda3/envs/pytorch1/lib/python3.9/unittest/runner.py:184:run
/opt/miniconda3/envs/pytorch1/lib/python3.9/unittest/main.py:271:runTests
/opt/miniconda3/envs/pytorch1/lib/python3.9/unittest/main.py:101:__init__
/local/pytorch1/torch/testing/_internal/common_utils.py:892:run_tests
/local/pytorch1/test/test_autograd.py:11248:<module>

$4: f32[1] = torch._ops.aten.detach.default($3)   (4 of 8 in recompute)

/local/pytorch1/torch/utils/checkpoint.py:1005:pack_hook
/local/pytorch1/test/test_autograd.py:5651:save_2_tensors_alt
/local/pytorch1/test/test_autograd.py:5660:fn
/local/pytorch1/torch/utils/checkpoint.py:1161:recompute_fn
/local/pytorch1/torch/utils/checkpoint.py:1041:unpack_hook
/local/pytorch1/torch/utils/checkpoint.py:1063:unpack_hook_with_error_cb
/local/pytorch1/torch/autograd/__init__.py:204:backward
/local/pytorch1/torch/_tensor.py:488:backward
/local/pytorch1/test/test_autograd.py:5692:test_checkpoint_detects_non_determinism
/opt/miniconda3/envs/pytorch1/lib/python3.9/unittest/case.py:550:_callTestMethod
/opt/miniconda3/envs/pytorch1/lib/python3.9/unittest/case.py:592:run
/local/pytorch1/torch/testing/_internal/common_utils.py:2248:_run_with_retry
/local/pytorch1/torch/testing/_internal/common_utils.py:2319:run
/opt/miniconda3/envs/pytorch1/lib/python3.9/unittest/case.py:651:__call__
/opt/miniconda3/envs/pytorch1/lib/python3.9/unittest/suite.py:122:run
/opt/miniconda3/envs/pytorch1/lib/python3.9/unittest/suite.py:84:__call__
/opt/miniconda3/envs/pytorch1/lib/python3.9/unittest/suite.py:122:run
/opt/miniconda3/envs/pytorch1/lib/python3.9/unittest/suite.py:84:__call__
/opt/miniconda3/envs/pytorch1/lib/python3.9/unittest/runner.py:184:run
/opt/miniconda3/envs/pytorch1/lib/python3.9/unittest/main.py:271:runTests
/opt/miniconda3/envs/pytorch1/lib/python3.9/unittest/main.py:101:__init__
/local/pytorch1/torch/testing/_internal/common_utils.py:892:run_tests
/local/pytorch1/test/test_autograd.py:11248:<module>

$5: f32[1] = torch._ops.aten.sin.default($0)   (5 of 8 in recompute)

/local/pytorch1/test/test_autograd.py:5651:save_2_tensors_alt
/local/pytorch1/test/test_autograd.py:5660:fn
/local/pytorch1/torch/utils/checkpoint.py:1161:recompute_fn
/local/pytorch1/torch/utils/checkpoint.py:1041:unpack_hook
/local/pytorch1/torch/utils/checkpoint.py:1063:unpack_hook_with_error_cb
/local/pytorch1/torch/autograd/__init__.py:204:backward
/local/pytorch1/torch/_tensor.py:488:backward
/local/pytorch1/test/test_autograd.py:5692:test_checkpoint_detects_non_determinism
/opt/miniconda3/envs/pytorch1/lib/python3.9/unittest/case.py:550:_callTestMethod
/opt/miniconda3/envs/pytorch1/lib/python3.9/unittest/case.py:592:run
/local/pytorch1/torch/testing/_internal/common_utils.py:2248:_run_with_retry
/local/pytorch1/torch/testing/_internal/common_utils.py:2319:run
/opt/miniconda3/envs/pytorch1/lib/python3.9/unittest/case.py:651:__call__
/opt/miniconda3/envs/pytorch1/lib/python3.9/unittest/suite.py:122:run
/opt/miniconda3/envs/pytorch1/lib/python3.9/unittest/suite.py:84:__call__
/opt/miniconda3/envs/pytorch1/lib/python3.9/unittest/suite.py:122:run
/opt/miniconda3/envs/pytorch1/lib/python3.9/unittest/suite.py:84:__call__
/opt/miniconda3/envs/pytorch1/lib/python3.9/unittest/runner.py:184:run
/opt/miniconda3/envs/pytorch1/lib/python3.9/unittest/main.py:271:runTests
/opt/miniconda3/envs/pytorch1/lib/python3.9/unittest/main.py:101:__init__
/local/pytorch1/torch/testing/_internal/common_utils.py:892:run_tests
/local/pytorch1/test/test_autograd.py:11248:<module>

$6: f32[2] = torch._ops.aten.lift_fresh.default($6)   (6 of 8 in recompute)

/local/pytorch1/test/test_autograd.py:5651:save_2_tensors_alt
/local/pytorch1/test/test_autograd.py:5660:fn
/local/pytorch1/torch/utils/checkpoint.py:1161:recompute_fn
/local/pytorch1/torch/utils/checkpoint.py:1041:unpack_hook
/local/pytorch1/torch/utils/checkpoint.py:1063:unpack_hook_with_error_cb
/local/pytorch1/torch/autograd/__init__.py:204:backward
/local/pytorch1/torch/_tensor.py:488:backward
/local/pytorch1/test/test_autograd.py:5692:test_checkpoint_detects_non_determinism
/opt/miniconda3/envs/pytorch1/lib/python3.9/unittest/case.py:550:_callTestMethod
/opt/miniconda3/envs/pytorch1/lib/python3.9/unittest/case.py:592:run
/local/pytorch1/torch/testing/_internal/common_utils.py:2248:_run_with_retry
/local/pytorch1/torch/testing/_internal/common_utils.py:2319:run
/opt/miniconda3/envs/pytorch1/lib/python3.9/unittest/case.py:651:__call__
/opt/miniconda3/envs/pytorch1/lib/python3.9/unittest/suite.py:122:run
/opt/miniconda3/envs/pytorch1/lib/python3.9/unittest/suite.py:84:__call__
/opt/miniconda3/envs/pytorch1/lib/python3.9/unittest/suite.py:122:run
/opt/miniconda3/envs/pytorch1/lib/python3.9/unittest/suite.py:84:__call__
/opt/miniconda3/envs/pytorch1/lib/python3.9/unittest/runner.py:184:run
/opt/miniconda3/envs/pytorch1/lib/python3.9/unittest/main.py:271:runTests
/opt/miniconda3/envs/pytorch1/lib/python3.9/unittest/main.py:101:__init__
/local/pytorch1/torch/testing/_internal/common_utils.py:892:run_tests
/local/pytorch1/test/test_autograd.py:11248:<module>

$7: f32[2] = torch._ops.aten.detach.default($6)   (7 of 8 in recompute)

/local/pytorch1/torch/utils/checkpoint.py:998:pack_hook
/local/pytorch1/test/test_autograd.py:5651:save_2_tensors_alt
/local/pytorch1/test/test_autograd.py:5660:fn
/local/pytorch1/torch/utils/checkpoint.py:1161:recompute_fn
/local/pytorch1/torch/utils/checkpoint.py:1041:unpack_hook
/local/pytorch1/torch/utils/checkpoint.py:1063:unpack_hook_with_error_cb
/local/pytorch1/torch/autograd/__init__.py:204:backward
/local/pytorch1/torch/_tensor.py:488:backward
/local/pytorch1/test/test_autograd.py:5692:test_checkpoint_detects_non_determinism
/opt/miniconda3/envs/pytorch1/lib/python3.9/unittest/case.py:550:_callTestMethod
/opt/miniconda3/envs/pytorch1/lib/python3.9/unittest/case.py:592:run
/local/pytorch1/torch/testing/_internal/common_utils.py:2248:_run_with_retry
/local/pytorch1/torch/testing/_internal/common_utils.py:2319:run
/opt/miniconda3/envs/pytorch1/lib/python3.9/unittest/case.py:651:__call__
/opt/miniconda3/envs/pytorch1/lib/python3.9/unittest/suite.py:122:run
/opt/miniconda3/envs/pytorch1/lib/python3.9/unittest/suite.py:84:__call__
/opt/miniconda3/envs/pytorch1/lib/python3.9/unittest/suite.py:122:run
/opt/miniconda3/envs/pytorch1/lib/python3.9/unittest/suite.py:84:__call__
/opt/miniconda3/envs/pytorch1/lib/python3.9/unittest/runner.py:184:run
/opt/miniconda3/envs/pytorch1/lib/python3.9/unittest/main.py:271:runTests
/opt/miniconda3/envs/pytorch1/lib/python3.9/unittest/main.py:101:__init__
/local/pytorch1/torch/testing/_internal/common_utils.py:892:run_tests
/local/pytorch1/test/test_autograd.py:11248:<module>

$8: f32[2] = torch._ops.aten.detach.default($7)   (8 of 8 in recompute)

/local/pytorch1/torch/utils/checkpoint.py:998:pack_hook
/local/pytorch1/test/test_autograd.py:5651:save_2_tensors_alt
/local/pytorch1/test/test_autograd.py:5660:fn
/local/pytorch1/torch/utils/checkpoint.py:1161:recompute_fn
/local/pytorch1/torch/utils/checkpoint.py:1041:unpack_hook
/local/pytorch1/torch/utils/checkpoint.py:1063:unpack_hook_with_error_cb
/local/pytorch1/torch/autograd/__init__.py:204:backward
/local/pytorch1/torch/_tensor.py:488:backward
/local/pytorch1/test/test_autograd.py:5692:test_checkpoint_detects_non_determinism
/opt/miniconda3/envs/pytorch1/lib/python3.9/unittest/case.py:550:_callTestMethod
/opt/miniconda3/envs/pytorch1/lib/python3.9/unittest/case.py:592:run
/local/pytorch1/torch/testing/_internal/common_utils.py:2248:_run_with_retry
/local/pytorch1/torch/testing/_internal/common_utils.py:2319:run
/opt/miniconda3/envs/pytorch1/lib/python3.9/unittest/case.py:651:__call__
/opt/miniconda3/envs/pytorch1/lib/python3.9/unittest/suite.py:122:run
/opt/miniconda3/envs/pytorch1/lib/python3.9/unittest/suite.py:84:__call__
/opt/miniconda3/envs/pytorch1/lib/python3.9/unittest/suite.py:122:run
/opt/miniconda3/envs/pytorch1/lib/python3.9/unittest/suite.py:84:__call__
/opt/miniconda3/envs/pytorch1/lib/python3.9/unittest/runner.py:184:run
/opt/miniconda3/envs/pytorch1/lib/python3.9/unittest/main.py:271:runTests
/opt/miniconda3/envs/pytorch1/lib/python3.9/unittest/main.py:101:__init__
/local/pytorch1/torch/testing/_internal/common_utils.py:892:run_tests
/local/pytorch1/test/test_autograd.py:11248:<module>


+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~+
|       3. Log of operators in the original forward and recomputation          |
+------------------------------------------------------------------------------+
(Scroll up to correlate stack traces with each operation listed below. This
 helps identify their source in the code.)

IMPORTANT: Differences in "detach" calls between the original forward and the
           recomputation are expected. They are introduced by the checkpointing
           mechanism and can be ignored.

Operations executed during the original forward:

$1: f32[1] = torch._ops.aten.sin.default($0)
$2: f32[1] = torch._ops.aten.exp.default($1)

Operations executed during recomputation:

$1: f32[1] = torch._ops.aten.detach.default($0)
$2: f32[1] = torch._ops.aten.detach.default($1)
$3: f32[1] = torch._ops.aten.detach.default($0)
$4: f32[1] = torch._ops.aten.detach.default($3)
$5: f32[1] = torch._ops.aten.sin.default($0)
$6: f32[2] = torch._ops.aten.lift_fresh.default($6)
$7: f32[2] = torch._ops.aten.detach.default($6)
$8: f32[2] = torch._ops.aten.detach.default($7)

+------------------------------------------------------------------------------+
 ERROR: Detected non-determinism while running activation checkpointing

 You are seeing this error because you passed `debug=True` to checkpoint and
 tensors to be saved during the original forward and differ between those saved
 during recomputation. This can happen if different operators were ran in the
 original forward and in the recomputation.

 To identify where the mismatch may be coming from, you can do the following:

 1) Compare the operators ran during original forward and recomputation to
    see where they differ. These operators are printed above in the order they
    were executed.

 2) Review the stack trace for each operator to locate its invocation source.
    Each operator's stack trace is printed in their execution order.

 Note that the logs can be quite long. Here's how they are structured:
 (Tip: you can Ctrl-f for these headers)

 1. Stack traces of the operators that ran in the original forward
 2. Stack traces of the operators that ran during recomputation
 3. Log of operators in the original forward and recomputation
 4. Error message                                             <--- You are here
--------------------------------------------------------------------------------