[Data] Restoring `ray.air.util.tensor_extensions.arrow` class aliases to fix deserialization of existing datasets by alexeykudinkin · Pull Request #59818 · ray-project/ray

alexeykudinkin · 2026-01-02T20:54:59Z

Description

Context: Slack

#59420 moved Ray Data's Arrow tensor extensions from ray.air.util.tensor_extensions to ray.data._internal.tensor_extensions.

That actually broke deserialization of the datasets written with older Ray Data implementation of these extensions inheriting from pyarrow.PyExtensionType:

PyEtensionType pickles class-ref into the metadata when writing the data (in that case it's ray.air.util.tensor_extensions.arrow.ArrowTensorType for ex)
Upon reading the data it tries to unpickle it and now fails b/c these classes were moved.

Related issues

Link related issues: "Fixes #1234", "Closes #1234", or "Related to #1234".

Additional information

Optional: Add implementation details, API changes, usage examples, screenshots, etc.

Signed-off-by: Alexey Kudinkin <ak@anyscale.com>

gemini-code-assist

Code Review

This pull request introduces a backward compatibility fix for deserializing older Ray datasets by restoring class aliases for tensor extensions. The changes are well-structured, including an improved error message for easier debugging and dedicated CI tests to validate the fix. My review includes a suggestion to refactor the new test configuration in release/release_data_tests.yaml to improve maintainability by reducing code duplication.

gemini-code-assist · 2026-01-02T20:55:56Z

release/release_data_tests.yaml

+    byod:
+      # NOTE: Image classification have to pin Pyarrow to 19.0 due to dataset using
+      #       previous tensor extension type inheriting from ``pyarrow.PyExtensionType``
+      #       that is removed in Pyarrow 21.0
+      python_depset: image_classification_py3.10.lock


To improve maintainability and avoid duplicating this configuration, you can define this byod block as a YAML anchor. You can then reuse this definition in the image_classification_chaos test configuration (lines 529-533) with an alias like byod: *pyarrow_pin.

byod: &pyarrow_pin # NOTE: Image classification have to pin Pyarrow to 19.0 due to dataset using # previous tensor extension type inheriting from ``pyarrow.PyExtensionType`` # that is removed in Pyarrow 21.0 python_depset: image_classification_py3.10.lock

gemini-code-assist · 2026-01-02T20:55:56Z

release/release_data_tests.yaml

+    byod:
+      # NOTE: Image classification have to pin Pyarrow to 19.0 due to dataset using
+      #       previous tensor extension type inheriting from ``pyarrow.PyExtensionType``
+      #       that is removed in Pyarrow 21.0
+      python_depset: image_classification_py3.10.lock


To avoid configuration duplication, you can use a YAML alias here to reference the byod block defined in the image_classification_{{scaling}} test above. This makes the file more maintainable.

byod: *pyarrow_pin

cursor · 2026-01-02T20:56:02Z

python/ray/data/_internal/datasource/parquet_datasource.py

+                f"to read data written with an older version of Ray. Reading data "
+                f"written with older versions of Ray might expose you to arbitrary code "
+                f"execution. To try reading the data anyway, "
+                f"preset `RAY_DATA_AUTOLOAD_PYEXTENSIONTYPE=1` on *all* nodes."


Missing space between concatenated string literals in error message

The error message has adjacent string literals where line 167 ends with "on *all* nodes." and line 168 starts with "To learn more...". When concatenated, this produces "...on *all* nodes.To learn more..." without a space between the sentences, making the error message harder to read.

Signed-off-by: Alexey Kudinkin <ak@anyscale.com>

iamjustinhsu

👍

iamjustinhsu · 2026-01-02T21:14:06Z

release/release_data_tests.yaml

  group: data-batch-inference

  cluster:
+    byod:


wait is this change orthogonal because we were always using pyarrow 21.0.0 before? or were we previously always using 19.0.0?

goutamvenkat-anyscale · 2026-01-02T21:30:41Z

release/release_multimodal_inference_benchmarks_tests.yaml

 - name: DEFAULTS
  python: "3.10"
-  group: multimodal-inference-benchmarks
+  group: data-multimodal-inference-benchmarks


Was this a drive by fix?

… to fix deserialization of existing datasets (#59828) ## Description Follow-up for #59818 1. Fixing serde for `ArrowPythonObjectType` 2. Missing `__init__.py` files making packages omitted at build time ## Related issues > Link related issues: "Fixes #1234", "Closes #1234", or "Related to #1234". ## Additional information > Optional: Add implementation details, API changes, usage examples, screenshots, etc. --------- Signed-off-by: Alexey Kudinkin <ak@anyscale.com>

… to fix deserialization of existing datasets (ray-project#59818) ## Description Context: [Slack](https://anyscaleteam.slack.com/archives/C04FMM4NPQ9/p1767322231131189) ray-project#59420 moved Ray Data's Arrow tensor extensions from `ray.air.util.tensor_extensions` to `ray.data._internal.tensor_extensions`. That actually broke deserialization of the datasets written with older Ray Data implementation of these extensions inheriting from `pyarrow.PyExtensionType`: 1. `PyEtensionType` pickles class-ref into the metadata when writing the data (in that case it's `ray.air.util.tensor_extensions.arrow.ArrowTensorType` for ex) 2. Upon reading the data it tries to unpickle it and now fails b/c these classes were moved. ## Related issues > Link related issues: "Fixes ray-project#1234", "Closes ray-project#1234", or "Related to ray-project#1234". ## Additional information > Optional: Add implementation details, API changes, usage examples, screenshots, etc. --------- Signed-off-by: Alexey Kudinkin <ak@anyscale.com> Signed-off-by: jasonwrwang <jasonwrwang@tencent.com>

… to fix deserialization of existing datasets (ray-project#59828) ## Description Follow-up for ray-project#59818 1. Fixing serde for `ArrowPythonObjectType` 2. Missing `__init__.py` files making packages omitted at build time ## Related issues > Link related issues: "Fixes ray-project#1234", "Closes ray-project#1234", or "Related to ray-project#1234". ## Additional information > Optional: Add implementation details, API changes, usage examples, screenshots, etc. --------- Signed-off-by: Alexey Kudinkin <ak@anyscale.com> Signed-off-by: jasonwrwang <jasonwrwang@tencent.com>

… to fix deserialization of existing datasets (ray-project#59818) ## Description Context: [Slack](https://anyscaleteam.slack.com/archives/C04FMM4NPQ9/p1767322231131189) ray-project#59420 moved Ray Data's Arrow tensor extensions from `ray.air.util.tensor_extensions` to `ray.data._internal.tensor_extensions`. That actually broke deserialization of the datasets written with older Ray Data implementation of these extensions inheriting from `pyarrow.PyExtensionType`: 1. `PyEtensionType` pickles class-ref into the metadata when writing the data (in that case it's `ray.air.util.tensor_extensions.arrow.ArrowTensorType` for ex) 2. Upon reading the data it tries to unpickle it and now fails b/c these classes were moved. ## Related issues > Link related issues: "Fixes ray-project#1234", "Closes ray-project#1234", or "Related to ray-project#1234". ## Additional information > Optional: Add implementation details, API changes, usage examples, screenshots, etc. --------- Signed-off-by: Alexey Kudinkin <ak@anyscale.com>

… to fix deserialization of existing datasets (ray-project#59828) ## Description Follow-up for ray-project#59818 1. Fixing serde for `ArrowPythonObjectType` 2. Missing `__init__.py` files making packages omitted at build time ## Related issues > Link related issues: "Fixes ray-project#1234", "Closes ray-project#1234", or "Related to ray-project#1234". ## Additional information > Optional: Add implementation details, API changes, usage examples, screenshots, etc. --------- Signed-off-by: Alexey Kudinkin <ak@anyscale.com>

… to fix deserialization of existing datasets (ray-project#59818) ## Description Context: [Slack](https://anyscaleteam.slack.com/archives/C04FMM4NPQ9/p1767322231131189) ray-project#59420 moved Ray Data's Arrow tensor extensions from `ray.air.util.tensor_extensions` to `ray.data._internal.tensor_extensions`. That actually broke deserialization of the datasets written with older Ray Data implementation of these extensions inheriting from `pyarrow.PyExtensionType`: 1. `PyEtensionType` pickles class-ref into the metadata when writing the data (in that case it's `ray.air.util.tensor_extensions.arrow.ArrowTensorType` for ex) 2. Upon reading the data it tries to unpickle it and now fails b/c these classes were moved. ## Related issues > Link related issues: "Fixes ray-project#1234", "Closes ray-project#1234", or "Related to ray-project#1234". ## Additional information > Optional: Add implementation details, API changes, usage examples, screenshots, etc. --------- Signed-off-by: Alexey Kudinkin <ak@anyscale.com>

… to fix deserialization of existing datasets (ray-project#59828) ## Description Follow-up for ray-project#59818 1. Fixing serde for `ArrowPythonObjectType` 2. Missing `__init__.py` files making packages omitted at build time ## Related issues > Link related issues: "Fixes ray-project#1234", "Closes ray-project#1234", or "Related to ray-project#1234". ## Additional information > Optional: Add implementation details, API changes, usage examples, screenshots, etc. --------- Signed-off-by: Alexey Kudinkin <ak@anyscale.com>

… to fix deserialization of existing datasets (ray-project#59818) ## Description Context: [Slack](https://anyscaleteam.slack.com/archives/C04FMM4NPQ9/p1767322231131189) ray-project#59420 moved Ray Data's Arrow tensor extensions from `ray.air.util.tensor_extensions` to `ray.data._internal.tensor_extensions`. That actually broke deserialization of the datasets written with older Ray Data implementation of these extensions inheriting from `pyarrow.PyExtensionType`: 1. `PyEtensionType` pickles class-ref into the metadata when writing the data (in that case it's `ray.air.util.tensor_extensions.arrow.ArrowTensorType` for ex) 2. Upon reading the data it tries to unpickle it and now fails b/c these classes were moved. ## Related issues > Link related issues: "Fixes ray-project#1234", "Closes ray-project#1234", or "Related to ray-project#1234". ## Additional information > Optional: Add implementation details, API changes, usage examples, screenshots, etc. --------- Signed-off-by: Alexey Kudinkin <ak@anyscale.com> Signed-off-by: peterxcli <peterxcli@gmail.com>

… to fix deserialization of existing datasets (ray-project#59828) ## Description Follow-up for ray-project#59818 1. Fixing serde for `ArrowPythonObjectType` 2. Missing `__init__.py` files making packages omitted at build time ## Related issues > Link related issues: "Fixes ray-project#1234", "Closes ray-project#1234", or "Related to ray-project#1234". ## Additional information > Optional: Add implementation details, API changes, usage examples, screenshots, etc. --------- Signed-off-by: Alexey Kudinkin <ak@anyscale.com> Signed-off-by: peterxcli <peterxcli@gmail.com>

… to fix deserialization of existing datasets (ray-project#59818) ## Description Context: [Slack](https://anyscaleteam.slack.com/archives/C04FMM4NPQ9/p1767322231131189) ray-project#59420 moved Ray Data's Arrow tensor extensions from `ray.air.util.tensor_extensions` to `ray.data._internal.tensor_extensions`. That actually broke deserialization of the datasets written with older Ray Data implementation of these extensions inheriting from `pyarrow.PyExtensionType`: 1. `PyEtensionType` pickles class-ref into the metadata when writing the data (in that case it's `ray.air.util.tensor_extensions.arrow.ArrowTensorType` for ex) 2. Upon reading the data it tries to unpickle it and now fails b/c these classes were moved. ## Related issues > Link related issues: "Fixes ray-project#1234", "Closes ray-project#1234", or "Related to ray-project#1234". ## Additional information > Optional: Add implementation details, API changes, usage examples, screenshots, etc. --------- Signed-off-by: Alexey Kudinkin <ak@anyscale.com> Signed-off-by: peterxcli <peterxcli@gmail.com>

… to fix deserialization of existing datasets (ray-project#59828) ## Description Follow-up for ray-project#59818 1. Fixing serde for `ArrowPythonObjectType` 2. Missing `__init__.py` files making packages omitted at build time ## Related issues > Link related issues: "Fixes ray-project#1234", "Closes ray-project#1234", or "Related to ray-project#1234". ## Additional information > Optional: Add implementation details, API changes, usage examples, screenshots, etc. --------- Signed-off-by: Alexey Kudinkin <ak@anyscale.com> Signed-off-by: peterxcli <peterxcli@gmail.com>

alexeykudinkin added 4 commits January 2, 2026 12:48

Restored aliases to ray.air.util.tensor_extensions.arrow

fd1680a

Signed-off-by: Alexey Kudinkin <ak@anyscale.com>

Updated error message to properly log pickled class ref

92f648a

Signed-off-by: Alexey Kudinkin <ak@anyscale.com>

Added depset lock for batch inference tests

3b78e6c

Signed-off-by: Alexey Kudinkin <ak@anyscale.com>

Prefixing multimodal benchmarks release tests with "data-"

ee4e766

Signed-off-by: Alexey Kudinkin <ak@anyscale.com>

alexeykudinkin requested review from a team as code owners January 2, 2026 20:55

alexeykudinkin added the go add ONLY when ready to merge, run all tests label Jan 2, 2026

gemini-code-assist bot reviewed Jan 2, 2026

View reviewed changes

cursor bot reviewed Jan 2, 2026

View reviewed changes

lint

93258ff

Signed-off-by: Alexey Kudinkin <ak@anyscale.com>

alexeykudinkin force-pushed the ak/btch-inf-rel-tst-fix branch from c936a4f to 93258ff Compare January 2, 2026 21:13

iamjustinhsu approved these changes Jan 2, 2026

View reviewed changes

bveeramani approved these changes Jan 2, 2026

View reviewed changes

bveeramani enabled auto-merge (squash) January 2, 2026 21:29

goutamvenkat-anyscale reviewed Jan 2, 2026

View reviewed changes

goutamvenkat-anyscale approved these changes Jan 2, 2026

View reviewed changes

matthewdeng approved these changes Jan 2, 2026

View reviewed changes

bveeramani merged commit b46b7fd into master Jan 2, 2026
6 of 7 checks passed

bveeramani deleted the ak/btch-inf-rel-tst-fix branch January 2, 2026 22:23

alexeykudinkin mentioned this pull request Jan 3, 2026

[Data] Restoring ray.air.util.object_extensions.arrow class aliases to fix deserialization of existing datasets #59828

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Data] Restoring `ray.air.util.tensor_extensions.arrow` class aliases to fix deserialization of existing datasets#59818

[Data] Restoring `ray.air.util.tensor_extensions.arrow` class aliases to fix deserialization of existing datasets#59818
bveeramani merged 5 commits intomasterfrom
ak/btch-inf-rel-tst-fix

alexeykudinkin commented Jan 2, 2026 •

edited

Loading

Uh oh!

gemini-code-assist bot left a comment

Uh oh!

gemini-code-assist bot Jan 2, 2026

Uh oh!

gemini-code-assist bot Jan 2, 2026

Uh oh!

cursor bot Jan 2, 2026

Uh oh!

iamjustinhsu left a comment

Uh oh!

iamjustinhsu Jan 2, 2026

Uh oh!

goutamvenkat-anyscale Jan 2, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Conversation

alexeykudinkin commented Jan 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Related issues

Additional information

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

gemini-code-assist bot Jan 2, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist bot Jan 2, 2026

Choose a reason for hiding this comment

Uh oh!

cursor bot Jan 2, 2026

Choose a reason for hiding this comment

Missing space between concatenated string literals in error message

Uh oh!

iamjustinhsu left a comment

Choose a reason for hiding this comment

Uh oh!

iamjustinhsu Jan 2, 2026

Choose a reason for hiding this comment

Uh oh!

goutamvenkat-anyscale Jan 2, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

alexeykudinkin commented Jan 2, 2026 •

edited

Loading