gistfile1.txt

W1202 13:59:56.994000 2658530 site-packages/torch/_logging/_internal.py:1084] [1/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored                            13:59:58 [117/1719]
Traceback (most recent call last):
  File "/data/users/jerryzh/ao/examples/sam2_amg_server/server.py", line 709, in <module>
    fire.Fire(main)
  File "/home/jerryzh/.conda/envs/ao/lib/python3.10/site-packages/fire/core.py", line 135, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
  File "/home/jerryzh/.conda/envs/ao/lib/python3.10/site-packages/fire/core.py", line 468, in _Fire
    component, remaining_args = _CallAndUpdateTrace(
  File "/home/jerryzh/.conda/envs/ao/lib/python3.10/site-packages/fire/core.py", line 684, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
  File "/data/users/jerryzh/ao/examples/sam2_amg_server/server.py", line 612, in main
    masks = image_tensor_to_masks(image_tensor, mask_generator)
  File "/data/users/jerryzh/ao/examples/sam2_amg_server/server.py", line 191, in image_tensor_to_masks
    masks = mask_generator.generate(example_image)
  File "/home/jerryzh/.conda/envs/ao/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
    return func(*args, **kwargs)
  File "/data/users/jerryzh/ao/torchao/_models/sam2/automatic_mask_generator.py", line 199, in generate
    mask_data = self._generate_masks(image)
  File "/data/users/jerryzh/ao/torchao/_models/sam2/automatic_mask_generator.py", line 245, in _generate_masks
    crop_data = self._process_crop(image, crop_box, layer_idx, orig_size)
  File "/data/users/jerryzh/ao/torchao/_models/sam2/automatic_mask_generator.py", line 300, in _process_crop
    return self._process_crop_points(cropped_im_size, crop_layer_idx, crop_box, orig_size)
  File "/data/users/jerryzh/ao/torchao/_models/sam2/automatic_mask_generator.py", line 315, in _process_crop_points
    batch_data = self._process_batch(
  File "/data/users/jerryzh/ao/torchao/_models/sam2/automatic_mask_generator.py", line 605, in _process_batch
    data = self._process_batch_fullgraph(points, im_size, crop_box, crop_box_torch, orig_size, normalize, orig_box_torch)
  File "/data/users/jerryzh/ao/torchao/_models/sam2/automatic_mask_generator.py", line 489, in _process_batch_fullgraph
    low_res_masks, iou_preds = self.predictor._predict_masks(
  File "/home/jerryzh/.conda/envs/ao/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 573, in _fn
    return fn(*args, **kwargs)
  File "/home/jerryzh/.conda/envs/ao/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 1379, in __call__
    return self._torchdynamo_orig_callable(
  File "/home/jerryzh/.conda/envs/ao/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 547, in __call__
    return _compile(
  File "/home/jerryzh/.conda/envs/ao/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 986, in _compile
    guarded_code = compile_inner(code, one_graph, hooks, transform)
  File "/home/jerryzh/.conda/envs/ao/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 715, in compile_inner
    return _compile_inner(code, one_graph, hooks, transform)
  File "/home/jerryzh/.conda/envs/ao/lib/python3.10/site-packages/torch/_utils_internal.py", line 95, in wrapper_function
    return function(*args, **kwargs)
  File "/home/jerryzh/.conda/envs/ao/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 750, in _compile_inner
    out_code = transform_code_object(code, transform)
  File "/home/jerryzh/.conda/envs/ao/lib/python3.10/site-packages/torch/_dynamo/bytecode_transformation.py", line 1349, in transform_code_object
    transformations(instructions, code_options)
  File "/home/jerryzh/.conda/envs/ao/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 231, in _fn
    return fn(*args, **kwargs)
  File "/home/jerryzh/.conda/envs/ao/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 662, in transform
      tracer.run()                                                                                                                                                                                    13:59:58 [70/1719]
  File "/home/jerryzh/.conda/envs/ao/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2865, in run
    super().run()
  File "/home/jerryzh/.conda/envs/ao/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1054, in run
    while self.step():
  File "/home/jerryzh/.conda/envs/ao/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 964, in step
    self.dispatch_table[inst.opcode](self, inst)
  File "/home/jerryzh/.conda/envs/ao/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 661, in wrapper
    return inner_fn(self, inst)
  File "/home/jerryzh/.conda/envs/ao/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1750, in CALL_FUNCTION_KW
    self.call_function(fn, args, kwargs)
  File "/home/jerryzh/.conda/envs/ao/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 899, in call_function
    self.push(fn.call_function(self, args, kwargs))  # type: ignore[arg-type]
  File "/home/jerryzh/.conda/envs/ao/lib/python3.10/site-packages/torch/_dynamo/variables/lazy.py", line 170, in realize_and_forward
    return getattr(self.realize(), name)(*args, **kwargs)
  File "/home/jerryzh/.conda/envs/ao/lib/python3.10/site-packages/torch/_dynamo/variables/nn_module.py", line 914, in call_function
    return variables.UserFunctionVariable(fn, source=source).call_function(
  File "/home/jerryzh/.conda/envs/ao/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 328, in call_function
    return super().call_function(tx, args, kwargs)
  File "/home/jerryzh/.conda/envs/ao/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 129, in call_function
    return tx.inline_user_function_return(self, [*self.self_args(), *args], kwargs)
  File "/home/jerryzh/.conda/envs/ao/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 905, in inline_user_function_return
    return InliningInstructionTranslator.inline_call(self, fn, args, kwargs)
  File "/home/jerryzh/.conda/envs/ao/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 3069, in inline_call
    return cls.inline_call_(parent, func, args, kwargs)
  File "/home/jerryzh/.conda/envs/ao/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 3195, in inline_call_
    tracer.run()
  File "/home/jerryzh/.conda/envs/ao/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1054, in run
    while self.step():
  File "/home/jerryzh/.conda/envs/ao/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 964, in step
    self.dispatch_table[inst.opcode](self, inst)
  File "/home/jerryzh/.conda/envs/ao/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 661, in wrapper
    return inner_fn(self, inst)
  File "/home/jerryzh/.conda/envs/ao/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1660, in CALL_FUNCTION
    self.call_function(fn, args, {})
  File "/home/jerryzh/.conda/envs/ao/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 899, in call_function
    self.push(fn.call_function(self, args, kwargs))  # type: ignore[arg-type]
  File "/home/jerryzh/.conda/envs/ao/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 129, in call_function
    return tx.inline_user_function_return(self, [*self.self_args(), *args], kwargs)
  File "/home/jerryzh/.conda/envs/ao/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 905, in inline_user_function_return
    return InliningInstructionTranslator.inline_call(self, fn, args, kwargs)
  File "/home/jerryzh/.conda/envs/ao/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 3069, in inline_call
    return cls.inline_call_(parent, func, args, kwargs)
  File "/home/jerryzh/.conda/envs/ao/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 3195, in inline_call_
    tracer.run()
  File "/home/jerryzh/.conda/envs/ao/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1054, in run
    while self.step():
      File "/home/jerryzh/.conda/envs/ao/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 964, in step                                                                             13:59:58 [23/1719]
    self.dispatch_table[inst.opcode](self, inst)
  File "/home/jerryzh/.conda/envs/ao/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 661, in wrapper
    return inner_fn(self, inst)
  File "/home/jerryzh/.conda/envs/ao/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1660, in CALL_FUNCTION
    self.call_function(fn, args, {})
  File "/home/jerryzh/.conda/envs/ao/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 899, in call_function
    self.push(fn.call_function(self, args, kwargs))  # type: ignore[arg-type]
  File "/home/jerryzh/.conda/envs/ao/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 328, in call_function
    return super().call_function(tx, args, kwargs)
  File "/home/jerryzh/.conda/envs/ao/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 129, in call_function
    return tx.inline_user_function_return(self, [*self.self_args(), *args], kwargs)
  File "/home/jerryzh/.conda/envs/ao/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 905, in inline_user_function_return
    return InliningInstructionTranslator.inline_call(self, fn, args, kwargs)
  File "/home/jerryzh/.conda/envs/ao/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 3069, in inline_call
    return cls.inline_call_(parent, func, args, kwargs)
  File "/home/jerryzh/.conda/envs/ao/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 3195, in inline_call_
    tracer.run()
  File "/home/jerryzh/.conda/envs/ao/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1054, in run
    while self.step():
  File "/home/jerryzh/.conda/envs/ao/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 964, in step
    self.dispatch_table[inst.opcode](self, inst)
  File "/home/jerryzh/.conda/envs/ao/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 661, in wrapper
    return inner_fn(self, inst)
  File "/home/jerryzh/.conda/envs/ao/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1660, in CALL_FUNCTION
    self.call_function(fn, args, {})
  File "/home/jerryzh/.conda/envs/ao/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 899, in call_function
    self.push(fn.call_function(self, args, kwargs))  # type: ignore[arg-type]
  File "/home/jerryzh/.conda/envs/ao/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 328, in call_function
    return super().call_function(tx, args, kwargs)
  File "/home/jerryzh/.conda/envs/ao/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 129, in call_function
    return tx.inline_user_function_return(self, [*self.self_args(), *args], kwargs)
  File "/home/jerryzh/.conda/envs/ao/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 905, in inline_user_function_return
    return InliningInstructionTranslator.inline_call(self, fn, args, kwargs)
  File "/home/jerryzh/.conda/envs/ao/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 3069, in inline_call
    return cls.inline_call_(parent, func, args, kwargs)
  File "/home/jerryzh/.conda/envs/ao/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 3116, in inline_call_
    sub_locals = func.bind_args(parent, args, kwargs)
  File "/home/jerryzh/.conda/envs/ao/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 266, in bind_args
    contents_var = VariableTracker.build(
  File "/home/jerryzh/.conda/envs/ao/lib/python3.10/site-packages/torch/_dynamo/variables/base.py", line 467, in build
    return builder.VariableBuilder(tx, source)(value)
  File "/home/jerryzh/.conda/envs/ao/lib/python3.10/site-packages/torch/_dynamo/variables/builder.py", line 386, in __call__
    vt = self._wrap(value)
  File "/home/jerryzh/.conda/envs/ao/lib/python3.10/site-packages/torch/_dynamo/variables/builder.py", line 570, in _wrap
    return type_dispatch(self, value)
  File "/home/jerryzh/.conda/envs/ao/lib/python3.10/site-packages/torch/_dynamo/variables/builder.py", line 495, in wrap_removable_handle
      unimplemented("unregistered hook removable handle")
  File "/home/jerryzh/.conda/envs/ao/lib/python3.10/site-packages/torch/_dynamo/exc.py", line 317, in unimplemented
    raise Unsupported(msg, case_name=case_name)
torch._dynamo.exc.Unsupported: unregistered hook removable handle

from user code:
   File "/data/users/jerryzh/ao/torchao/_models/sam2/sam2_image_predictor.py", line 450, in _predict_masks
    low_res_masks, iou_predictions, _, _ = self.model.sam_mask_decoder(
  File "/home/jerryzh/.conda/envs/ao/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1844, in _call_impl
    return inner()
  File "/home/jerryzh/.conda/envs/ao/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1772, in inner
    args_kwargs_result = hook(self, args, kwargs)  # type: ignore[misc]
  File "/data/users/jerryzh/ao/torchao/quantization/autoquant.py", line 1137, in autoquant_prehook
    module.finalize_autoquant()

Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information


You can suppress this exception and fall back to eager by setting:
    import torch._dynamo
    torch._dynamo.config.suppress_errors = True