ray_data_torchvision_variable_len.log

2025-02-04 13:43:35,467 INFO streaming_executor.py:109 -- Execution plan of Dataset: InputDataBuffer[Input] -> TaskPoolMapOperator[ReadTorch->Map(extract_and_process_image)] -> LimitOperator[limit=1]
Running Dataset. Active & requested resources: 1/12 CPU, 256.0MB/1.0GB object store: : 0.00 row [00:01, ? row/s]2025-02-04 13:43:36,763 ERROR streaming_executor_state.py:485 -- An exception was raised from a task of operator "ReadTorch->Map(extract_and_process_image)". Dataset execution will now abort. To ignore this exception and continue, set DataContext.max_errored_blocks.
⚠️  Dataset execution failed: : 0.00 row [00:01, ? row/s]                                                       
- ReadTorch->Map(extract_and_process_image): Tasks: 1; Queued blocks: 0; Resources: 1.0 CPU, 256.0MB object store: : 0.00 row [00:01, ? row/s]
- limit=1: Tasks: 0; Queued blocks: 0; Resources: 0.0 CPU, 0.0B object store: : 0.00 row [00:01, ? row/s]                                     
2025-02-04 13:43:36,781 ERROR exceptions.py:73 -- Exception occurred in Ray Data or Ray Core internal code. If you continue to see this error, please open an issue on the Ray project GitHub page with the full stack trace below: https://github.com/ray-project/ray/issues/new/choose                                              
2025-02-04 13:43:36,781 ERROR exceptions.py:81 -- Full stack trace:
Traceback (most recent call last):
  File "/Users/rdecal/src/ANYSCALE/tutorials/ray_data_map_reduce/.conda/lib/python3.11/site-packages/ray/data/exceptions.py", line 49, in handle_trace
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/Users/rdecal/src/ANYSCALE/tutorials/ray_data_map_reduce/.conda/lib/python3.11/site-packages/ray/data/_internal/plan.py", line 429, in execute_to_iterator
    bundle_iter = itertools.chain([next(gen)], gen)
                                   ^^^^^^^^^
  File "/Users/rdecal/src/ANYSCALE/tutorials/ray_data_map_reduce/.conda/lib/python3.11/site-packages/ray/data/_internal/execution/interfaces/executor.py", line 37, in __next__
    return self.get_next()
           ^^^^^^^^^^^^^^^
  File "/Users/rdecal/src/ANYSCALE/tutorials/ray_data_map_reduce/.conda/lib/python3.11/site-packages/ray/data/_internal/execution/legacy_compat.py", line 76, in get_next
    bundle = self._base_iterator.get_next(output_split_idx)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/rdecal/src/ANYSCALE/tutorials/ray_data_map_reduce/.conda/lib/python3.11/site-packages/ray/data/_internal/execution/streaming_executor.py", line 157, in get_next
    item = self._outer._output_node.get_output_blocking(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/rdecal/src/ANYSCALE/tutorials/ray_data_map_reduce/.conda/lib/python3.11/site-packages/ray/data/_internal/execution/streaming_executor_state.py", line 312, in get_output_blocking
    raise self._exception
  File "/Users/rdecal/src/ANYSCALE/tutorials/ray_data_map_reduce/.conda/lib/python3.11/site-packages/ray/data/_internal/execution/streaming_executor.py", line 233, in run
    continue_sched = self._scheduling_loop_step(self._topology)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/rdecal/src/ANYSCALE/tutorials/ray_data_map_reduce/.conda/lib/python3.11/site-packages/ray/data/_internal/execution/streaming_executor.py", line 292, in _scheduling_loop_step
    num_errored_blocks = process_completed_tasks(
                         ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/rdecal/src/ANYSCALE/tutorials/ray_data_map_reduce/.conda/lib/python3.11/site-packages/ray/data/_internal/execution/streaming_executor_state.py", line 486, in process_completed_tasks
    raise e from None
  File "/Users/rdecal/src/ANYSCALE/tutorials/ray_data_map_reduce/.conda/lib/python3.11/site-packages/ray/data/_internal/execution/streaming_executor_state.py", line 453, in process_completed_tasks
    bytes_read = task.on_data_ready(
                 ^^^^^^^^^^^^^^^^^^^
  File "/Users/rdecal/src/ANYSCALE/tutorials/ray_data_map_reduce/.conda/lib/python3.11/site-packages/ray/data/_internal/execution/interfaces/physical_operator.py", line 105, in on_data_ready
    raise ex from None
  File "/Users/rdecal/src/ANYSCALE/tutorials/ray_data_map_reduce/.conda/lib/python3.11/site-packages/ray/data/_internal/execution/interfaces/physical_operator.py", line 101, in on_data_ready
    ray.get(block_ref)
  File "/Users/rdecal/src/ANYSCALE/tutorials/ray_data_map_reduce/.conda/lib/python3.11/site-packages/ray/_private/auto_init_hook.py", line 21, in auto_init_wrapper
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/Users/rdecal/src/ANYSCALE/tutorials/ray_data_map_reduce/.conda/lib/python3.11/site-packages/ray/_private/client_mode_hook.py", line 103, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/rdecal/src/ANYSCALE/tutorials/ray_data_map_reduce/.conda/lib/python3.11/site-packages/ray/_private/worker.py", line 2772, in get
    values, debugger_breakpoint = worker.get_objects(object_refs, timeout=timeout)
                                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/rdecal/src/ANYSCALE/tutorials/ray_data_map_reduce/.conda/lib/python3.11/site-packages/ray/_private/worker.py", line 919, in get_objects
    raise value.as_instanceof_cause()
ray.exceptions.RayTaskError(ArrowConversionError): ray::ReadTorch->Map(extract_and_process_image)() (pid=59772, ip=127.0.0.1)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/rdecal/src/ANYSCALE/tutorials/ray_data_map_reduce/.conda/lib/python3.11/site-packages/ray/data/_internal/arrow_ops/transform_pyarrow.py", line 125, in unify_schemas
    raise ValueError(
ValueError: Found columns with both objects and tensors: {'image'}

The above exception was the direct cause of the following exception:

ray::ReadTorch->Map(extract_and_process_image)() (pid=59772, ip=127.0.0.1)
  File "/Users/rdecal/src/ANYSCALE/tutorials/ray_data_map_reduce/.conda/lib/python3.11/site-packages/ray/data/_internal/execution/operators/map_operator.py", line 507, in _map_task
    for b_out in map_transformer.apply_transform(iter(blocks), ctx):
  File "/Users/rdecal/src/ANYSCALE/tutorials/ray_data_map_reduce/.conda/lib/python3.11/site-packages/ray/data/_internal/execution/operators/map_transformer.py", line 395, in __call__
    yield output_buffer.next()
          ^^^^^^^^^^^^^^^^^^^^
  File "/Users/rdecal/src/ANYSCALE/tutorials/ray_data_map_reduce/.conda/lib/python3.11/site-packages/ray/data/_internal/output_buffer.py", line 73, in next
    block_to_yield = self._buffer.build()
                     ^^^^^^^^^^^^^^^^^^^^
  File "/Users/rdecal/src/ANYSCALE/tutorials/ray_data_map_reduce/.conda/lib/python3.11/site-packages/ray/data/_internal/delegating_block_builder.py", line 68, in build
    return self._builder.build()
           ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/rdecal/src/ANYSCALE/tutorials/ray_data_map_reduce/.conda/lib/python3.11/site-packages/ray/data/_internal/table_block.py", line 133, in build
    return self._concat_tables(tables)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/rdecal/src/ANYSCALE/tutorials/ray_data_map_reduce/.conda/lib/python3.11/site-packages/ray/data/_internal/arrow_block.py", line 149, in _concat_tables
    return transform_pyarrow.concat(tables)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/rdecal/src/ANYSCALE/tutorials/ray_data_map_reduce/.conda/lib/python3.11/site-packages/ray/data/_internal/arrow_ops/transform_pyarrow.py", line 256, in concat
    raise ArrowConversionError(str(blocks)) from e
ray.air.util.tensor_extensions.arrow.ArrowConversionError: Error converting data to Arrow: [pyarrow.Table
image: extension<ray.data.arrow_variable_shaped_tensor<ArrowVariableShapedTensorType>>
----
image: [  -- is_valid: all not null
  -- child 0 type: large_list<item: uint8>
[[255,255,255,...
ray.data.exceptions.SystemException

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/rdecal/src/ANYSCALE/tutorials/ray_data_map_reduce/test_fromtorch.py", line 21, in <module>
    print(ds.take(1))
          ^^^^^^^^^^
  File "/Users/rdecal/src/ANYSCALE/tutorials/ray_data_map_reduce/.conda/lib/python3.11/site-packages/ray/data/dataset.py", line 2711, in take
    for row in limited_ds.iter_rows():
  File "/Users/rdecal/src/ANYSCALE/tutorials/ray_data_map_reduce/.conda/lib/python3.11/site-packages/ray/data/iterator.py", line 214, in _wrapped_iterator
    for batch in batch_iterable:
  File "/Users/rdecal/src/ANYSCALE/tutorials/ray_data_map_reduce/.conda/lib/python3.11/site-packages/ray/data/iterator.py", line 154, in _create_iterator
    ) = self._to_ref_bundle_iterator()
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/rdecal/src/ANYSCALE/tutorials/ray_data_map_reduce/.conda/lib/python3.11/site-packages/ray/data/_internal/iterator/iterator_impl.py", line 28, in _to_ref_bundle_iterator
    ref_bundles_iterator, stats, executor = ds._plan.execute_to_iterator()
                                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/rdecal/src/ANYSCALE/tutorials/ray_data_map_reduce/.conda/lib/python3.11/site-packages/ray/data/exceptions.py", line 89, in handle_trace
    raise e.with_traceback(None) from SystemException()
ray.exceptions.RayTaskError(ArrowConversionError): ray::ReadTorch->Map(extract_and_process_image)() (pid=59772, ip=127.0.0.1)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/rdecal/src/ANYSCALE/tutorials/ray_data_map_reduce/.conda/lib/python3.11/site-packages/ray/data/_internal/arrow_ops/transform_pyarrow.py", line 125, in unify_schemas
    raise ValueError(
ValueError: Found columns with both objects and tensors: {'image'}

The above exception was the direct cause of the following exception:

ray::ReadTorch->Map(extract_and_process_image)() (pid=59772, ip=127.0.0.1)
  File "/Users/rdecal/src/ANYSCALE/tutorials/ray_data_map_reduce/.conda/lib/python3.11/site-packages/ray/data/_internal/execution/operators/map_operator.py", line 507, in _map_task
    for b_out in map_transformer.apply_transform(iter(blocks), ctx):
  File "/Users/rdecal/src/ANYSCALE/tutorials/ray_data_map_reduce/.conda/lib/python3.11/site-packages/ray/data/_internal/execution/operators/map_transformer.py", line 395, in __call__
    yield output_buffer.next()
          ^^^^^^^^^^^^^^^^^^^^
  File "/Users/rdecal/src/ANYSCALE/tutorials/ray_data_map_reduce/.conda/lib/python3.11/site-packages/ray/data/_internal/output_buffer.py", line 73, in next
    block_to_yield = self._buffer.build()
                     ^^^^^^^^^^^^^^^^^^^^
  File "/Users/rdecal/src/ANYSCALE/tutorials/ray_data_map_reduce/.conda/lib/python3.11/site-packages/ray/data/_internal/delegating_block_builder.py", line 68, in build
    return self._builder.build()
           ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/rdecal/src/ANYSCALE/tutorials/ray_data_map_reduce/.conda/lib/python3.11/site-packages/ray/data/_internal/table_block.py", line 133, in build
    return self._concat_tables(tables)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/rdecal/src/ANYSCALE/tutorials/ray_data_map_reduce/.conda/lib/python3.11/site-packages/ray/data/_internal/arrow_block.py", line 149, in _concat_tables
    return transform_pyarrow.concat(tables)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/rdecal/src/ANYSCALE/tutorials/ray_data_map_reduce/.conda/lib/python3.11/site-packages/ray/data/_internal/arrow_ops/transform_pyarrow.py", line 256, in concat
    raise ArrowConversionError(str(blocks)) from e
ray.air.util.tensor_extensions.arrow.ArrowConversionError: Error converting data to Arrow: [pyarrow.Table
image: extension<ray.data.arrow_variable_shaped_tensor<ArrowVariableShapedTensorType>>
----
image: [  -- is_valid: all not null
  -- child 0 type: large_list<item: uint8>
[[255,255,255,...