[Data] [1/n] Predicate Expression Support by goutamvenkat-anyscale · Pull Request #56313 · ray-project/ray

goutamvenkat-anyscale · 2025-09-07T18:43:51Z

Why are these changes needed?

Adds support for predicate expressions in Ray Data's Expression System.
Involves the following:

Support for unary operations (NOT, IS_NULL(), IN and their inverses)
Add PredicateExpr to expression evaluator

Related issue number

Checks

I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
I've run scripts/format.sh to lint the changes in this PR.
I've included any doc changes needed for https://docs.ray.io/en/master/.
- I've added any new APIs to the API Reference. For example, if I added a
  method in Tune, I've added it in doc/source/tune/api/ under the
  corresponding .rst file.
I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
Testing Strategy
- Unit tests
- Release tests
- This PR is not tested :(

gemini-code-assist

Code Review

This pull request introduces valuable support for predicate expressions in Ray Data, including new unary and binary operations, the UnaryExpr and PredicateExpr classes, and a where() function. The implementation is well-structured and the new features are integrated cleanly into the existing expression system. The test coverage is extensive, which is great. However, I've identified a few test cases where the expected outcomes for expressions involving NULL values seem to be incorrect due to a misunderstanding of three-valued logic. I've left specific comments on these tests. Addressing these will ensure the test suite is robust and accurately reflects the expression engine's behavior.

python/ray/data/tests/test_map.py

python/ray/data/expressions.py

Signed-off-by: Goutam V. <goutam@anyscale.com>

alexeykudinkin · 2025-09-16T00:39:14Z

python/ray/data/_expression_evaluator.py

+    if isinstance(expr, UnaryExpr):
+        return ops[expr.op](_eval_expr_recursive(expr.operand, batch, ops))


In a follow-up:

Let's create a visitor to work with exprs

Let's make ops part of the visitor state (to avoid passing it as args)

Just so that I'm clear, are you referring to an ADT?

python/ray/data/expressions.py

Signed-off-by: Goutam V. <goutam@anyscale.com>

iamjustinhsu · 2025-09-16T18:59:52Z

python/ray/data/_expression_evaluator.py

            _eval_expr_recursive(expr.right, batch, ops),
        )
+    if isinstance(expr, UnaryExpr):
+        # TODO: Use Visitor pattern here and store ops in shared state.


wdym by storing ops in a shared state?

In reference to this comment: #56313 (comment)

https://en.wikipedia.org/wiki/Visitor_pattern

python/ray/data/_internal/pandas_block.py

iamjustinhsu · 2025-09-16T19:03:17Z

python/ray/data/expressions.py

        SUB: Subtraction operation (-)
        MUL: Multiplication operation (*)
        DIV: Division operation (/)
+        FLOORDIV: Floor division operation (//)


is there modulo?

Pyarrow doesn't have a native mod kernel. Could be a good follow up

https://stackoverflow.com/questions/75009169/how-to-implement-modulo-operation-using-pyarrow-expression-api-so-that-i-can-use

Signed-off-by: Goutam V. <goutam@anyscale.com>

python/ray/data/expressions.py

python/ray/data/tests/test_expressions.py

python/ray/data/expressions.py

python/ray/data/tests/test_expressions.py

alexeykudinkin · 2025-09-17T23:58:42Z

python/ray/data/tests/test_map.py

+    result_df = result_ds.to_pandas()
+
+    expected_df = pd.DataFrame(
+        {
+            "product": ["A", "B", "C", "D", "E"],
+            "quantity": [10, 5, 20, 15, 3],
+            "price": [100, 200, 50, 75, 300],
+            "region": ["North", "South", "North", "East", "West"],
+            "revenue": [1000, 1000, 1000, 1125, 900],
+            "is_high_value": [True, True, True, True, False],
+            "is_bulk_order": [True, False, True, True, False],
+            "is_premium": [True, True, False, False, True],
+            "needs_special_handling": [True, True, True, True, False],
+            "is_north_region": [True, False, True, False, False],
+        }
+    )
+
+    pd.testing.assert_frame_equal(result_df, expected_df, check_dtype=False)


There's no filtering in this test

Filtering is a separate PR: #56365

Signed-off-by: Goutam V. <goutam@anyscale.com>

alexeykudinkin

LGTM

## Why are these changes needed? Adds support for predicate expressions in Ray Data's Expression System. Involves the following: 1. Support for unary operations (NOT, IS_NULL(), IN and their inverses) 2. Add `PredicateExpr` to expression evaluator ## Related issue number ## Checks - [x] I've signed off every commit(by using the -s flag, i.e., `git commit -s`) in this PR. - [x] I've run `scripts/format.sh` to lint the changes in this PR. - [ ] I've included any doc changes needed for https://docs.ray.io/en/master/. - [ ] I've added any new APIs to the API Reference. For example, if I added a method in Tune, I've added it in `doc/source/tune/api/` under the corresponding `.rst` file. - [ ] I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/ - Testing Strategy - [x] Unit tests - [ ] Release tests - [ ] This PR is not tested :( --------- Signed-off-by: Goutam V. <goutam@anyscale.com> Signed-off-by: Zhiqiang Ma <zhiqiang.ma@intel.com>

## Why are these changes needed? Adds support for predicate expressions in Ray Data's Expression System. Involves the following: 1. Support for unary operations (NOT, IS_NULL(), IN and their inverses) 2. Add `PredicateExpr` to expression evaluator ## Related issue number ## Checks - [x] I've signed off every commit(by using the -s flag, i.e., `git commit -s`) in this PR. - [x] I've run `scripts/format.sh` to lint the changes in this PR. - [ ] I've included any doc changes needed for https://docs.ray.io/en/master/. - [ ] I've added any new APIs to the API Reference. For example, if I added a method in Tune, I've added it in `doc/source/tune/api/` under the corresponding `.rst` file. - [ ] I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/ - Testing Strategy - [x] Unit tests - [ ] Release tests - [ ] This PR is not tested :( --------- Signed-off-by: Goutam V. <goutam@anyscale.com> Signed-off-by: zac <zac@anyscale.com>

## Why are these changes needed? Adds support for predicate expressions in Ray Data's Expression System. Involves the following: 1. Support for unary operations (NOT, IS_NULL(), IN and their inverses) 2. Add `PredicateExpr` to expression evaluator ## Related issue number ## Checks - [x] I've signed off every commit(by using the -s flag, i.e., `git commit -s`) in this PR. - [x] I've run `scripts/format.sh` to lint the changes in this PR. - [ ] I've included any doc changes needed for https://docs.ray.io/en/master/. - [ ] I've added any new APIs to the API Reference. For example, if I added a method in Tune, I've added it in `doc/source/tune/api/` under the corresponding `.rst` file. - [ ] I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/ - Testing Strategy - [x] Unit tests - [ ] Release tests - [ ] This PR is not tested :( --------- Signed-off-by: Goutam V. <goutam@anyscale.com> Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>

## Why are these changes needed? Adds support for predicate expressions in Ray Data's Expression System. Involves the following: 1. Support for unary operations (NOT, IS_NULL(), IN and their inverses) 2. Add `PredicateExpr` to expression evaluator ## Related issue number ## Checks - [x] I've signed off every commit(by using the -s flag, i.e., `git commit -s`) in this PR. - [x] I've run `scripts/format.sh` to lint the changes in this PR. - [ ] I've included any doc changes needed for https://docs.ray.io/en/master/. - [ ] I've added any new APIs to the API Reference. For example, if I added a method in Tune, I've added it in `doc/source/tune/api/` under the corresponding `.rst` file. - [ ] I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/ - Testing Strategy - [x] Unit tests - [ ] Release tests - [ ] This PR is not tested :( --------- Signed-off-by: Goutam V. <goutam@anyscale.com> Signed-off-by: Marco Stephan <marco@magic.dev>

## Why are these changes needed? Adds support for predicate expressions in Ray Data's Expression System. Involves the following: 1. Support for unary operations (NOT, IS_NULL(), IN and their inverses) 2. Add `PredicateExpr` to expression evaluator ## Related issue number ## Checks - [x] I've signed off every commit(by using the -s flag, i.e., `git commit -s`) in this PR. - [x] I've run `scripts/format.sh` to lint the changes in this PR. - [ ] I've included any doc changes needed for https://docs.ray.io/en/master/. - [ ] I've added any new APIs to the API Reference. For example, if I added a method in Tune, I've added it in `doc/source/tune/api/` under the corresponding `.rst` file. - [ ] I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/ - Testing Strategy - [x] Unit tests - [ ] Release tests - [ ] This PR is not tested :( --------- Signed-off-by: Goutam V. <goutam@anyscale.com> Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>

## Why are these changes needed? Adds support for predicate expressions in Ray Data's Expression System. Involves the following: 1. Support for unary operations (NOT, IS_NULL(), IN and their inverses) 2. Add `PredicateExpr` to expression evaluator ## Related issue number ## Checks - [x] I've signed off every commit(by using the -s flag, i.e., `git commit -s`) in this PR. - [x] I've run `scripts/format.sh` to lint the changes in this PR. - [ ] I've included any doc changes needed for https://docs.ray.io/en/master/. - [ ] I've added any new APIs to the API Reference. For example, if I added a method in Tune, I've added it in `doc/source/tune/api/` under the corresponding `.rst` file. - [ ] I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/ - Testing Strategy - [x] Unit tests - [ ] Release tests - [ ] This PR is not tested :( --------- Signed-off-by: Goutam V. <goutam@anyscale.com> Signed-off-by: Douglas Strodtman <douglas@anyscale.com>

## Why are these changes needed? Adds support for predicate expressions in Ray Data's Expression System. Involves the following: 1. Support for unary operations (NOT, IS_NULL(), IN and their inverses) 2. Add `PredicateExpr` to expression evaluator ## Related issue number ## Checks - [x] I've signed off every commit(by using the -s flag, i.e., `git commit -s`) in this PR. - [x] I've run `scripts/format.sh` to lint the changes in this PR. - [ ] I've included any doc changes needed for https://docs.ray.io/en/master/. - [ ] I've added any new APIs to the API Reference. For example, if I added a method in Tune, I've added it in `doc/source/tune/api/` under the corresponding `.rst` file. - [ ] I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/ - Testing Strategy - [x] Unit tests - [ ] Release tests - [ ] This PR is not tested :( --------- Signed-off-by: Goutam V. <goutam@anyscale.com>

Original PR #56313 by goutamvenkat-anyscale Original: ray-project/ray#56313

Merged from original PR #56313 Original: ray-project/ray#56313

## Why are these changes needed? Adds support for predicate expressions in Ray Data's Expression System. Involves the following: 1. Support for unary operations (NOT, IS_NULL(), IN and their inverses) 2. Add `PredicateExpr` to expression evaluator ## Related issue number ## Checks - [x] I've signed off every commit(by using the -s flag, i.e., `git commit -s`) in this PR. - [x] I've run `scripts/format.sh` to lint the changes in this PR. - [ ] I've included any doc changes needed for https://docs.ray.io/en/master/. - [ ] I've added any new APIs to the API Reference. For example, if I added a method in Tune, I've added it in `doc/source/tune/api/` under the corresponding `.rst` file. - [ ] I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/ - Testing Strategy - [x] Unit tests - [ ] Release tests - [ ] This PR is not tested :( --------- Signed-off-by: Goutam V. <goutam@anyscale.com>

## Why are these changes needed? Adds support for predicate expressions in Ray Data's Expression System. Involves the following: 1. Support for unary operations (NOT, IS_NULL(), IN and their inverses) 2. Add `PredicateExpr` to expression evaluator ## Related issue number ## Checks - [x] I've signed off every commit(by using the -s flag, i.e., `git commit -s`) in this PR. - [x] I've run `scripts/format.sh` to lint the changes in this PR. - [ ] I've included any doc changes needed for https://docs.ray.io/en/master/. - [ ] I've added any new APIs to the API Reference. For example, if I added a method in Tune, I've added it in `doc/source/tune/api/` under the corresponding `.rst` file. - [ ] I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/ - Testing Strategy - [x] Unit tests - [ ] Release tests - [ ] This PR is not tested :( --------- Signed-off-by: Goutam V. <goutam@anyscale.com> Signed-off-by: Future-Outlier <eric901201@gmail.com>

goutamvenkat-anyscale requested a review from a team as a code owner September 7, 2025 18:43

gemini-code-assist bot reviewed Sep 7, 2025

View reviewed changes

python/ray/data/tests/test_map.py Outdated Show resolved Hide resolved

python/ray/data/tests/test_map.py Show resolved Hide resolved

python/ray/data/tests/test_map.py Outdated Show resolved Hide resolved

ray-gardener bot added the data Ray Data-related issues label Sep 7, 2025

goutamvenkat-anyscale added the go add ONLY when ready to merge, run all tests label Sep 7, 2025

alexeykudinkin reviewed Sep 8, 2025

View reviewed changes

python/ray/data/expressions.py Outdated Show resolved Hide resolved

goutamvenkat-anyscale added 4 commits September 8, 2025 18:18

[Data] [1/n] Filter Expression Support

9f67d72

Signed-off-by: Goutam V. <goutam@anyscale.com>

clean up

81224e6

Signed-off-by: Goutam V. <goutam@anyscale.com>

Remove where

e2c5a15

Signed-off-by: Goutam V. <goutam@anyscale.com>

rst fix

c806c0c

Signed-off-by: Goutam V. <goutam@anyscale.com>

goutamvenkat-anyscale force-pushed the goutam/filter_expr branch from 7692898 to c806c0c Compare September 9, 2025 01:30

goutamvenkat-anyscale mentioned this pull request Sep 9, 2025

[Data] [2/n] - Add predicate expressions to filter api #56365

Closed

8 tasks

alexeykudinkin reviewed Sep 16, 2025

View reviewed changes

goutamvenkat-anyscale added 4 commits September 15, 2025 20:12

Merge branch 'master' into goutam/filter_expr

3cced46

Remove predicateExpr

ec18b73

Signed-off-by: Goutam V. <goutam@anyscale.com>

Remove old doc

6907774

Signed-off-by: Goutam V. <goutam@anyscale.com>

Add todo comment

dee95ac

Signed-off-by: Goutam V. <goutam@anyscale.com>

iamjustinhsu reviewed Sep 16, 2025

View reviewed changes

Change fn name

2483f60

Signed-off-by: Goutam V. <goutam@anyscale.com>

alexeykudinkin reviewed Sep 17, 2025

View reviewed changes

Address comments

24e1591

Signed-off-by: Goutam V. <goutam@anyscale.com>

alexeykudinkin approved these changes Sep 18, 2025

View reviewed changes

alexeykudinkin merged commit ad2362e into ray-project:master Sep 18, 2025
5 checks passed

goutamvenkat-anyscale mentioned this pull request Sep 19, 2025

[Data] [2/n] - Add predicate expression support for dataset.filter #56716

Merged

8 tasks

snorkelopsstgtesting1-spec mentioned this pull request Oct 22, 2025

[Data] [1/n] Predicate Expression Support snorkel-marlin-repos/ray-project_ray_pr_56313_9837c7ba-a573-444a-9456-2fe69b8d152a#1

Merged

		if isinstance(expr, UnaryExpr):
		return ops[expr.op](_eval_expr_recursive(expr.operand, batch, ops))

Conversation

goutamvenkat-anyscale commented Sep 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why are these changes needed?

Related issue number

Checks

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

alexeykudinkin Sep 16, 2025

Choose a reason for hiding this comment

Uh oh!

goutamvenkat-anyscale Sep 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

iamjustinhsu Sep 16, 2025

Choose a reason for hiding this comment

Uh oh!

goutamvenkat-anyscale Sep 17, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

iamjustinhsu Sep 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

goutamvenkat-anyscale Sep 17, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

alexeykudinkin Sep 17, 2025

Choose a reason for hiding this comment

Uh oh!

goutamvenkat-anyscale Sep 18, 2025

Choose a reason for hiding this comment

Uh oh!

alexeykudinkin left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

goutamvenkat-anyscale commented Sep 7, 2025 •

edited

Loading

goutamvenkat-anyscale Sep 16, 2025 •

edited

Loading

iamjustinhsu Sep 16, 2025 •

edited

Loading