Skip to content

[data] dedupe schemas#55854

Merged
alexeykudinkin merged 15 commits intoray-project:masterfrom
iamjustinhsu:jhsu/dedupe-schemas
Aug 27, 2025
Merged

[data] dedupe schemas#55854
alexeykudinkin merged 15 commits intoray-project:masterfrom
iamjustinhsu:jhsu/dedupe-schemas

Conversation

@iamjustinhsu
Copy link
Contributor

@iamjustinhsu iamjustinhsu commented Aug 22, 2025 •
edited
Loading

Why are these changes needed?

results: TBD

  • 2180 columns, 1000 schemas deduping is 1 second

Related issue number

Checks

  • I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
  • I've run scripts/format.sh to lint the changes in this PR.
  • I've included any doc changes needed for https://docs.ray.io/en/master/.
    • I've added any new APIs to the API Reference. For example, if I added a
      method in Tune, I've added it in doc/source/tune/api/ under the
      corresponding .rst file.
  • I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
  • Testing Strategy
    • Unit tests
    • Release tests
    • This PR is not tested :(

@iamjustinhsu
[data] dedupe schemas
f5be8bc 
Signed-off-by: iamjustinhsu <jhsu@anyscale.com>
@iamjustinhsu
comment
0904bd6 
Signed-off-by: iamjustinhsu <jhsu@anyscale.com>
alexeykudinkin
alexeykudinkin reviewed Aug 22, 2025
View reviewed changes
@iamjustinhsu
hash ArrowPythonObjecType
bbd1243 
Signed-off-by: iamjustinhsu <jhsu@anyscale.com>
@iamjustinhsu
done
635647a 
Signed-off-by: iamjustinhsu <jhsu@anyscale.com>
@iamjustinhsu
done
5d05869 
Signed-off-by: iamjustinhsu <jhsu@anyscale.com>
@iamjustinhsu iamjustinhsu added the go add ONLY when ready to merge, run all tests label Aug 25, 2025
@iamjustinhsu iamjustinhsu marked this pull request as ready for review August 25, 2025 22:24
@iamjustinhsu iamjustinhsu requested review from a team as code owners August 25, 2025 22:24
@iamjustinhsu
lint
62ac12f 
Signed-off-by: iamjustinhsu <jhsu@anyscale.com>
@ray-gardener ray-gardener bot added the data Ray Data-related issues label Aug 26, 2025
@iamjustinhsu
generalize
2bc7092 
Signed-off-by: iamjustinhsu <jhsu@anyscale.com>
iamjustinhsu
iamjustinhsu commented Aug 26, 2025
View reviewed changes
alexeykudinkin
alexeykudinkin reviewed Aug 26, 2025
View reviewed changes
python/ray/data/_internal/arrow_ops/transform_pyarrow.py Outdated
Comment on lines +186 to +188
except Exception:
# Unsure if there are cases where schemas are NOT hashable
pass
Copy link
Contributor

@alexeykudinkin alexeykudinkin Aug 26, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Don't swallow exceptions

@iamjustinhsu
add warning + remove_metadata in RefBundle check
b84121d 
Signed-off-by: iamjustinhsu <jhsu@anyscale.com>
@iamjustinhsu
remove test
71d6eaa 
Signed-off-by: iamjustinhsu <jhsu@anyscale.com>
alexeykudinkin
alexeykudinkin approved these changes Aug 27, 2025
View reviewed changes
@iamjustinhsu
print exception too
3a2b67c 
Signed-off-by: iamjustinhsu <jhsu@anyscale.com>
alexeykudinkin
alexeykudinkin reviewed Aug 27, 2025
View reviewed changes
python/ray/data/_internal/execution/interfaces/ref_bundle.py
Comment on lines +68 to +69
if isinstance(self.schema, pa.lib.Schema):
self.schema = self.schema.remove_metadata()
Copy link
Contributor

@alexeykudinkin alexeykudinkin Aug 27, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@iamjustinhsu on a second thought: why don't we do this when we convert from Pandas to Pyarrow, since we only do that in normalize_block_types essentially

Copy link
Contributor Author

@iamjustinhsu iamjustinhsu Aug 27, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i've seen it occur elsewhere as well: #55854 (comment). With the patch release not sure if would should try to find every area where this occurs

Copy link
Contributor

@alexeykudinkin alexeykudinkin Aug 27, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ack. Add a comment and elaborate on why we're doing this

alexeykudinkin and others added 5 commits August 27, 2025 12:49
@alexeykudinkin
Tidying up
f42fb25 
Signed-off-by: Alexey Kudinkin <alexey.kudinkin@gmail.com>
@iamjustinhsu
lint
de0db90 
Signed-off-by: iamjustinhsu <jhsu@anyscale.com>
@iamjustinhsu
comment
5debf3b 
Signed-off-by: iamjustinhsu <jhsu@anyscale.com>
@iamjustinhsu
fix
1f7e0f4 
Signed-off-by: iamjustinhsu <jhsu@anyscale.com>
@iamjustinhsu
name
9439b0c 
Signed-off-by: iamjustinhsu <jhsu@anyscale.com>
@alexeykudinkin alexeykudinkin enabled auto-merge (squash) August 27, 2025 21:24
dayshah
dayshah approved these changes Aug 27, 2025
View reviewed changes
Copy link
Contributor

@dayshah dayshah left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you update so core isn't a code owner on any of this @alexeykudinkin
trusting your approval here lol

@alexeykudinkin alexeykudinkin merged commit 4a6c264 into ray-project:master Aug 27, 2025
6 checks passed
omatthew98 pushed a commit to omatthew98/ray that referenced this pull request Aug 28, 2025
@iamjustinhsu @alexeykudinkin @omatthew98
[data] dedupe schemas (ray-project#55854)
fd26490 
<!-- Thank you for your contribution! Please review
https://github.com/ray-project/ray/blob/master/CONTRIBUTING.rst before
opening a pull request. -->

<!-- Please add a reviewer to the assignee section when you create a PR.
If you don't have the access to it, we will shortly find a reviewer and
assign them to your PR. -->

## Why are these changes needed?
results: TBD
<!-- Please give a short summary of the change and the problem this
solves. -->
- 2180 columns, 1000 schemas deduping is 1 second

## Related issue number

<!-- For example: "Closes ray-project#1234" -->

## Checks

- [ ] I've signed off every commit(by using the -s flag, i.e., `git
commit -s`) in this PR.
- [ ] I've run `scripts/format.sh` to lint the changes in this PR.
- [ ] I've included any doc changes needed for
https://docs.ray.io/en/master/.
- [ ] I've added any new APIs to the API Reference. For example, if I
added a
method in Tune, I've added it in `doc/source/tune/api/` under the
           corresponding `.rst` file.
- [ ] I've made sure the tests are passing. Note that there might be a
few flaky tests, see the recent failures at https://flakey-tests.ray.io/
- Testing Strategy
   - [ ] Unit tests
   - [ ] Release tests
   - [ ] This PR is not tested :(

---------

Signed-off-by: iamjustinhsu <jhsu@anyscale.com>
Signed-off-by: Alexey Kudinkin <alexey.kudinkin@gmail.com>
Co-authored-by: Alexey Kudinkin <alexey.kudinkin@gmail.com>
@omatthew98 omatthew98 mentioned this pull request Aug 28, 2025
Merged
8 tasks
@iamjustinhsu iamjustinhsu mentioned this pull request Aug 28, 2025
Closed
8 tasks
omatthew98 pushed a commit to omatthew98/ray that referenced this pull request Aug 28, 2025
@iamjustinhsu @alexeykudinkin @omatthew98
[data] dedupe schemas (ray-project#55854)
81ff179 
<!-- Thank you for your contribution! Please review
https://github.com/ray-project/ray/blob/master/CONTRIBUTING.rst before
opening a pull request. -->

<!-- Please add a reviewer to the assignee section when you create a PR.
If you don't have the access to it, we will shortly find a reviewer and
assign them to your PR. -->

## Why are these changes needed?
results: TBD
<!-- Please give a short summary of the change and the problem this
solves. -->
- 2180 columns, 1000 schemas deduping is 1 second

## Related issue number

<!-- For example: "Closes ray-project#1234" -->

## Checks

- [ ] I've signed off every commit(by using the -s flag, i.e., `git
commit -s`) in this PR.
- [ ] I've run `scripts/format.sh` to lint the changes in this PR.
- [ ] I've included any doc changes needed for
https://docs.ray.io/en/master/.
- [ ] I've added any new APIs to the API Reference. For example, if I
added a
method in Tune, I've added it in `doc/source/tune/api/` under the
           corresponding `.rst` file.
- [ ] I've made sure the tests are passing. Note that there might be a
few flaky tests, see the recent failures at https://flakey-tests.ray.io/
- Testing Strategy
   - [ ] Unit tests
   - [ ] Release tests
   - [ ] This PR is not tested :(

---------

Signed-off-by: iamjustinhsu <jhsu@anyscale.com>
Signed-off-by: Alexey Kudinkin <alexey.kudinkin@gmail.com>
Co-authored-by: Alexey Kudinkin <alexey.kudinkin@gmail.com>
Signed-off-by: Matthew Owen <mowen@anyscale.com>
aslonnie pushed a commit that referenced this pull request Aug 28, 2025
@omatthew98 @iamjustinhsu @alexeykudinkin
[data] Cherry pick data fixes for 2.49.1 (#56058)
c057f1e 
Cherry pick two fixes for ray data (from
#55854 and
#55926).

---------

Signed-off-by: iamjustinhsu <jhsu@anyscale.com>
Signed-off-by: Alexey Kudinkin <alexey.kudinkin@gmail.com>
Signed-off-by: Matthew Owen <mowen@anyscale.com>
Co-authored-by: iamjustinhsu <140442892+iamjustinhsu@users.noreply.github.com>
Co-authored-by: Alexey Kudinkin <alexey.kudinkin@gmail.com>
@iamjustinhsu iamjustinhsu deleted the jhsu/dedupe-schemas branch August 29, 2025 04:55
tohtana pushed a commit to tohtana/ray that referenced this pull request Aug 29, 2025
@iamjustinhsu @alexeykudinkin @tohtana
[data] dedupe schemas (ray-project#55854)
b98f3f7 
<!-- Thank you for your contribution! Please review
https://github.com/ray-project/ray/blob/master/CONTRIBUTING.rst before
opening a pull request. -->

<!-- Please add a reviewer to the assignee section when you create a PR.
If you don't have the access to it, we will shortly find a reviewer and
assign them to your PR. -->

## Why are these changes needed?
results: TBD
<!-- Please give a short summary of the change and the problem this
solves. -->
- 2180 columns, 1000 schemas deduping is 1 second

## Related issue number

<!-- For example: "Closes ray-project#1234" -->

## Checks

- [ ] I've signed off every commit(by using the -s flag, i.e., `git
commit -s`) in this PR.
- [ ] I've run `scripts/format.sh` to lint the changes in this PR.
- [ ] I've included any doc changes needed for
https://docs.ray.io/en/master/.
- [ ] I've added any new APIs to the API Reference. For example, if I
added a
method in Tune, I've added it in `doc/source/tune/api/` under the
           corresponding `.rst` file.
- [ ] I've made sure the tests are passing. Note that there might be a
few flaky tests, see the recent failures at https://flakey-tests.ray.io/
- Testing Strategy
   - [ ] Unit tests
   - [ ] Release tests
   - [ ] This PR is not tested :(

---------

Signed-off-by: iamjustinhsu <jhsu@anyscale.com>
Signed-off-by: Alexey Kudinkin <alexey.kudinkin@gmail.com>
Co-authored-by: Alexey Kudinkin <alexey.kudinkin@gmail.com>
Signed-off-by: Masahiro Tanaka <mtanaka@anyscale.com>
tohtana pushed a commit to tohtana/ray that referenced this pull request Aug 29, 2025
@iamjustinhsu @alexeykudinkin @tohtana
[data] dedupe schemas (ray-project#55854)
2e6f894 
<!-- Thank you for your contribution! Please review
https://github.com/ray-project/ray/blob/master/CONTRIBUTING.rst before
opening a pull request. -->

<!-- Please add a reviewer to the assignee section when you create a PR.
If you don't have the access to it, we will shortly find a reviewer and
assign them to your PR. -->

## Why are these changes needed?
results: TBD
<!-- Please give a short summary of the change and the problem this
solves. -->
- 2180 columns, 1000 schemas deduping is 1 second

## Related issue number

<!-- For example: "Closes ray-project#1234" -->

## Checks

- [ ] I've signed off every commit(by using the -s flag, i.e., `git
commit -s`) in this PR.
- [ ] I've run `scripts/format.sh` to lint the changes in this PR.
- [ ] I've included any doc changes needed for
https://docs.ray.io/en/master/.
- [ ] I've added any new APIs to the API Reference. For example, if I
added a
method in Tune, I've added it in `doc/source/tune/api/` under the
           corresponding `.rst` file.
- [ ] I've made sure the tests are passing. Note that there might be a
few flaky tests, see the recent failures at https://flakey-tests.ray.io/
- Testing Strategy
   - [ ] Unit tests
   - [ ] Release tests
   - [ ] This PR is not tested :(

---------

Signed-off-by: iamjustinhsu <jhsu@anyscale.com>
Signed-off-by: Alexey Kudinkin <alexey.kudinkin@gmail.com>
Co-authored-by: Alexey Kudinkin <alexey.kudinkin@gmail.com>
Signed-off-by: Masahiro Tanaka <mtanaka@anyscale.com>
gangsf pushed a commit to gangsf/ray that referenced this pull request Sep 2, 2025
@iamjustinhsu @alexeykudinkin
[data] dedupe schemas (ray-project#55854)
b8e9fd6 
<!-- Thank you for your contribution! Please review
https://github.com/ray-project/ray/blob/master/CONTRIBUTING.rst before
opening a pull request. -->

<!-- Please add a reviewer to the assignee section when you create a PR.
If you don't have the access to it, we will shortly find a reviewer and
assign them to your PR. -->

## Why are these changes needed?
results: TBD
<!-- Please give a short summary of the change and the problem this
solves. -->
- 2180 columns, 1000 schemas deduping is 1 second

## Related issue number

<!-- For example: "Closes ray-project#1234" -->

## Checks

- [ ] I've signed off every commit(by using the -s flag, i.e., `git
commit -s`) in this PR.
- [ ] I've run `scripts/format.sh` to lint the changes in this PR.
- [ ] I've included any doc changes needed for
https://docs.ray.io/en/master/.
- [ ] I've added any new APIs to the API Reference. For example, if I
added a
method in Tune, I've added it in `doc/source/tune/api/` under the
           corresponding `.rst` file.
- [ ] I've made sure the tests are passing. Note that there might be a
few flaky tests, see the recent failures at https://flakey-tests.ray.io/
- Testing Strategy
   - [ ] Unit tests
   - [ ] Release tests
   - [ ] This PR is not tested :(

---------

Signed-off-by: iamjustinhsu <jhsu@anyscale.com>
Signed-off-by: Alexey Kudinkin <alexey.kudinkin@gmail.com>
Co-authored-by: Alexey Kudinkin <alexey.kudinkin@gmail.com>
Signed-off-by: Gang Zhao <gang@gang-JQ62HD2C37.local>
sampan-s-nayak pushed a commit to sampan-s-nayak/ray that referenced this pull request Sep 8, 2025
@iamjustinhsu @alexeykudinkin
[data] dedupe schemas (ray-project#55854)
5c80019 
<!-- Thank you for your contribution! Please review
https://github.com/ray-project/ray/blob/master/CONTRIBUTING.rst before
opening a pull request. -->

<!-- Please add a reviewer to the assignee section when you create a PR.
If you don't have the access to it, we will shortly find a reviewer and
assign them to your PR. -->

## Why are these changes needed?
results: TBD
<!-- Please give a short summary of the change and the problem this
solves. -->
- 2180 columns, 1000 schemas deduping is 1 second

## Related issue number

<!-- For example: "Closes ray-project#1234" -->

## Checks

- [ ] I've signed off every commit(by using the -s flag, i.e., `git
commit -s`) in this PR.
- [ ] I've run `scripts/format.sh` to lint the changes in this PR.
- [ ] I've included any doc changes needed for
https://docs.ray.io/en/master/.
- [ ] I've added any new APIs to the API Reference. For example, if I
added a
method in Tune, I've added it in `doc/source/tune/api/` under the
           corresponding `.rst` file.
- [ ] I've made sure the tests are passing. Note that there might be a
few flaky tests, see the recent failures at https://flakey-tests.ray.io/
- Testing Strategy
   - [ ] Unit tests
   - [ ] Release tests
   - [ ] This PR is not tested :(

---------

Signed-off-by: iamjustinhsu <jhsu@anyscale.com>
Signed-off-by: Alexey Kudinkin <alexey.kudinkin@gmail.com>
Co-authored-by: Alexey Kudinkin <alexey.kudinkin@gmail.com>
Signed-off-by: sampan <sampan@anyscale.com>
jugalshah291 pushed a commit to jugalshah291/ray_fork that referenced this pull request Sep 11, 2025
@iamjustinhsu @alexeykudinkin @jugalshah291
[data] dedupe schemas (ray-project#55854)
3ac38da 
<!-- Thank you for your contribution! Please review
https://github.com/ray-project/ray/blob/master/CONTRIBUTING.rst before
opening a pull request. -->

<!-- Please add a reviewer to the assignee section when you create a PR.
If you don't have the access to it, we will shortly find a reviewer and
assign them to your PR. -->

## Why are these changes needed?
results: TBD
<!-- Please give a short summary of the change and the problem this
solves. -->
- 2180 columns, 1000 schemas deduping is 1 second

## Related issue number

<!-- For example: "Closes ray-project#1234" -->

## Checks

- [ ] I've signed off every commit(by using the -s flag, i.e., `git
commit -s`) in this PR.
- [ ] I've run `scripts/format.sh` to lint the changes in this PR.
- [ ] I've included any doc changes needed for
https://docs.ray.io/en/master/.
- [ ] I've added any new APIs to the API Reference. For example, if I
added a
method in Tune, I've added it in `doc/source/tune/api/` under the
           corresponding `.rst` file.
- [ ] I've made sure the tests are passing. Note that there might be a
few flaky tests, see the recent failures at https://flakey-tests.ray.io/
- Testing Strategy
   - [ ] Unit tests
   - [ ] Release tests
   - [ ] This PR is not tested :(

---------

Signed-off-by: iamjustinhsu <jhsu@anyscale.com>
Signed-off-by: Alexey Kudinkin <alexey.kudinkin@gmail.com>
Co-authored-by: Alexey Kudinkin <alexey.kudinkin@gmail.com>
Signed-off-by: jugalshah291 <shah.jugal291@gmail.com>
dstrodtman pushed a commit to dstrodtman/ray that referenced this pull request Oct 6, 2025
@iamjustinhsu @alexeykudinkin @dstrodtman
[data] dedupe schemas (ray-project#55854)
271b116 
<!-- Thank you for your contribution! Please review
https://github.com/ray-project/ray/blob/master/CONTRIBUTING.rst before
opening a pull request. -->

<!-- Please add a reviewer to the assignee section when you create a PR.
If you don't have the access to it, we will shortly find a reviewer and
assign them to your PR. -->

## Why are these changes needed?
results: TBD
<!-- Please give a short summary of the change and the problem this
solves. -->
- 2180 columns, 1000 schemas deduping is 1 second

## Related issue number

<!-- For example: "Closes ray-project#1234" -->

## Checks

- [ ] I've signed off every commit(by using the -s flag, i.e., `git
commit -s`) in this PR.
- [ ] I've run `scripts/format.sh` to lint the changes in this PR.
- [ ] I've included any doc changes needed for
https://docs.ray.io/en/master/.
- [ ] I've added any new APIs to the API Reference. For example, if I
added a
method in Tune, I've added it in `doc/source/tune/api/` under the
           corresponding `.rst` file.
- [ ] I've made sure the tests are passing. Note that there might be a
few flaky tests, see the recent failures at https://flakey-tests.ray.io/
- Testing Strategy
   - [ ] Unit tests
   - [ ] Release tests
   - [ ] This PR is not tested :(

---------

Signed-off-by: iamjustinhsu <jhsu@anyscale.com>
Signed-off-by: Alexey Kudinkin <alexey.kudinkin@gmail.com>
Co-authored-by: Alexey Kudinkin <alexey.kudinkin@gmail.com>
Signed-off-by: Douglas Strodtman <douglas@anyscale.com>
landscapepainter pushed a commit to landscapepainter/ray that referenced this pull request Nov 17, 2025
@iamjustinhsu @alexeykudinkin @landscapepainter
[data] dedupe schemas (ray-project#55854)
831a21a 
<!-- Thank you for your contribution! Please review
https://github.com/ray-project/ray/blob/master/CONTRIBUTING.rst before
opening a pull request. -->

<!-- Please add a reviewer to the assignee section when you create a PR.
If you don't have the access to it, we will shortly find a reviewer and
assign them to your PR. -->

## Why are these changes needed?
results: TBD
<!-- Please give a short summary of the change and the problem this
solves. -->
- 2180 columns, 1000 schemas deduping is 1 second

## Related issue number

<!-- For example: "Closes ray-project#1234" -->

## Checks

- [ ] I've signed off every commit(by using the -s flag, i.e., `git
commit -s`) in this PR.
- [ ] I've run `scripts/format.sh` to lint the changes in this PR.
- [ ] I've included any doc changes needed for
https://docs.ray.io/en/master/.
- [ ] I've added any new APIs to the API Reference. For example, if I
added a
method in Tune, I've added it in `doc/source/tune/api/` under the
           corresponding `.rst` file.
- [ ] I've made sure the tests are passing. Note that there might be a
few flaky tests, see the recent failures at https://flakey-tests.ray.io/
- Testing Strategy
   - [ ] Unit tests
   - [ ] Release tests
   - [ ] This PR is not tested :(

---------

Signed-off-by: iamjustinhsu <jhsu@anyscale.com>
Signed-off-by: Alexey Kudinkin <alexey.kudinkin@gmail.com>
Co-authored-by: Alexey Kudinkin <alexey.kudinkin@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@alexeykudinkin alexeykudinkin alexeykudinkin approved these changes

@dayshah dayshah dayshah approved these changes

Assignees

No one assigned

Labels

data Ray Data-related issues go add ONLY when ready to merge, run all tests

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@iamjustinhsu @alexeykudinkin @dayshah