[BUG] Integration tests failing in join_test.py

[nartal1]

Nightly integration tests (Left semi join tests) are failing in join_test.py due to mismatch between CPU and GPU results:

[2024-09-19T13:45:37.529Z] E           AssertionError: CPU and GPU list have different lengths at [] CPU: 260 GPU: 28

[2024-09-19T13:45:37.529Z] 

[2024-09-19T13:45:37.529Z] ../../src/main/python/asserts.py:41: AssertionError

[2024-09-19T13:45:37.529Z] ---------------------------- Captured stderr setup -----------------------------

[2024-09-19T13:45:37.529Z] 2024-09-19 13:00:13 INFO     Running test 'src/main/python/join_test.py::test_sortmerge_join_with_condition_ast[LeftAnti-Timestamp][DATAGEN_SEED=1726749170, TZ=UTC, INJECT_OOM, IGNORE_ORDER({'local': True})]'

[2024-09-19T13:45:37.529Z] ------------------------------ Captured log setup ------------------------------

[2024-09-19T13:45:37.529Z] INFO     __pytest_worker_logger__:spark_init_internal.py:256 Running test 'src/main/python/join_test.py::test_sortmerge_join_with_condition_ast[LeftAnti-Timestamp][DATAGEN_SEED=1726749170, TZ=UTC, INJECT_OOM, IGNORE_ORDER({'local': True})]'

[2024-09-19T13:45:37.529Z] ----------------------------- 

All the failing tests:

[2024-09-19T13:45:37.543Z] =========================== short test summary info ============================
[2024-09-19T13:45:37.543Z] FAILED ../../src/main/python/join_test.py::test_broadcast_join_with_conditionals[LeftSemi-Boolean][DATAGEN_SEED=1726749170, TZ=UTC, IGNORE_ORDER({'local': True})] - AssertionError: CPU and GPU list have different lengths at [] CPU: 457 GPU:...
[2024-09-19T13:45:37.543Z] FAILED ../../src/main/python/join_test.py::test_broadcast_join_with_conditionals[LeftSemi-Byte][DATAGEN_SEED=1726749170, TZ=UTC, IGNORE_ORDER({'local': True})] - AssertionError: CPU and GPU list have different lengths at [] CPU: 298 GPU: 87
[2024-09-19T13:45:37.543Z] FAILED ../../src/main/python/join_test.py::test_broadcast_join_with_conditionals[LeftSemi-Short][DATAGEN_SEED=1726749170, TZ=UTC, INJECT_OOM, IGNORE_ORDER({'local': True})] - AssertionError: CPU and GPU list have different lengths at [] CPU: 239 GPU: 68
[2024-09-19T13:45:37.543Z] FAILED ../../src/main/python/join_test.py::test_broadcast_join_with_conditionals[LeftSemi-Integer][DATAGEN_SEED=1726749170, TZ=UTC, IGNORE_ORDER({'local': True})] - AssertionError: CPU and GPU list have different lengths at [] CPU: 236 GPU: 66
[2024-09-19T13:45:37.543Z] Starting with datagen test seed: 1726749170 (Automatically set). Set env variable DATAGEN_SEED to override.
[2024-09-19T13:45:37.543Z] Starting with OOM injection seed: 1726749170. Set env variable SPARK_RAPIDS_TEST_INJECT_OOM_SEED to override.
[2024-09-19T13:45:37.543Z] 2024-09-19 12:32:51 INFO     Executing global initialization tasks before test launches
[2024-09-19T13:45:37.543Z] 2024-09-19 12:32:51 INFO     Creating directory /home/ubuntu/spark-rapids/integration_tests/target/run_dir-20240919123250-U9Li/hive with permissions 0o777
[2024-09-19T13:45:37.543Z] 2024-09-19 12:32:51 INFO     Skipping findspark init because on xdist master
[2024-09-19T13:45:37.543Z] FAILED ../../src/main/python/join_test.py::test_broadcast_join_with_conditionals[LeftSemi-Long][DATAGEN_SEED=1726749170, TZ=UTC, IGNORE_ORDER({'local': True})] - AssertionError: CPU and GPU list have different lengths at [] CPU: 235 GPU:...
[2024-09-19T13:45:37.543Z] FAILED ../../src/main/python/join_test.py::test_broadcast_join_with_conditionals[LeftSemi-Date][DATAGEN_SEED=1726749170, TZ=UTC, INJECT_OOM, IGNORE_ORDER({'local': True})] - AssertionError: CPU and GPU list have different lengths at [] CPU: 253 GPU:...
[2024-09-19T13:45:37.543Z] FAILED ../../src/main/python/join_test.py::test_broadcast_join_with_conditionals[LeftSemi-Timestamp][DATAGEN_SEED=1726749170, TZ=UTC, INJECT_OOM, IGNORE_ORDER({'local': True})] - AssertionError: CPU and GPU list have different lengths at [] CPU: 240 GPU:...
[2024-09-19T13:45:37.543Z] FAILED ../../src/main/python/join_test.py::test_broadcast_join_with_conditionals[LeftSemi-String][DATAGEN_SEED=1726749170, TZ=UTC, INJECT_OOM, IGNORE_ORDER({'local': True})] - AssertionError: CPU and GPU list have different lengths at [] CPU: 230 GPU:...
[2024-09-19T13:45:37.543Z] FAILED ../../src/main/python/join_test.py::test_broadcast_join_with_conditionals[LeftAnti-Boolean][DATAGEN_SEED=1726749170, TZ=UTC, IGNORE_ORDER({'local': True})] - AssertionError: CPU and GPU list have different lengths at [] CPU: 43 GPU: 239
[2024-09-19T13:45:37.543Z] FAILED ../../src/main/python/join_test.py::test_broadcast_join_with_conditionals[LeftAnti-Byte][DATAGEN_SEED=1726749170, TZ=UTC, IGNORE_ORDER({'local': True})] - AssertionError: CPU and GPU list have different lengths at [] CPU: 202 GPU:...
[2024-09-19T13:45:37.543Z] FAILED ../../src/main/python/join_test.py::test_broadcast_join_with_conditionals[LeftAnti-Short][DATAGEN_SEED=1726749170, TZ=UTC, INJECT_OOM, IGNORE_ORDER({'local': True})] - AssertionError: CPU and GPU list have different lengths at [] CPU: 261 GPU:...
[2024-09-19T13:45:37.543Z] FAILED ../../src/main/python/join_test.py::test_broadcast_join_with_conditionals[LeftAnti-Integer][DATAGEN_SEED=1726749170, TZ=UTC, INJECT_OOM, IGNORE_ORDER({'local': True})] - AssertionError: CPU and GPU list have different lengths at [] CPU: 264 GPU:...
[2024-09-19T13:45:37.543Z] FAILED ../../src/main/python/join_test.py::test_broadcast_join_with_conditionals[LeftAnti-Long][DATAGEN_SEED=1726749170, TZ=UTC, IGNORE_ORDER({'local': True})] - AssertionError: CPU and GPU list have different lengths at [] CPU: 265 GPU:...
[2024-09-19T13:45:37.543Z] FAILED ../../src/main/python/join_test.py::test_broadcast_join_with_conditionals[LeftAnti-Date][DATAGEN_SEED=1726749170, TZ=UTC, IGNORE_ORDER({'local': True})] - AssertionError: CPU and GPU list have different lengths at [] CPU: 247 GPU:...
[2024-09-19T13:45:37.543Z] FAILED ../../src/main/python/join_test.py::test_broadcast_join_with_conditionals[LeftAnti-Timestamp][DATAGEN_SEED=1726749170, TZ=UTC, INJECT_OOM, IGNORE_ORDER({'local': True})] - AssertionError: CPU and GPU list have different lengths at [] CPU: 260 GPU:...
[2024-09-19T13:45:37.543Z] FAILED ../../src/main/python/join_test.py::test_broadcast_join_with_conditionals[LeftAnti-String][DATAGEN_SEED=1726749170, TZ=UTC, INJECT_OOM, IGNORE_ORDER({'local': True})] - AssertionError: CPU and GPU list have different lengths at [] CPU: 270 GPU:...
[2024-09-19T13:45:37.543Z] FAILED ../../src/main/python/join_test.py::test_sortmerge_join_with_condition_ast[LeftSemi-Boolean][DATAGEN_SEED=1726749170, TZ=UTC, IGNORE_ORDER({'local': True})] - AssertionError: CPU and GPU list have different lengths at [] CPU: 457 GPU:...
[2024-09-19T13:45:37.544Z] FAILED ../../src/main/python/join_test.py::test_sortmerge_join_with_condition_ast[LeftSemi-Byte][DATAGEN_SEED=1726749170, TZ=UTC, INJECT_OOM, IGNORE_ORDER({'local': True})] - AssertionError: CPU and GPU list have different lengths at [] CPU: 298 GPU:...
[2024-09-19T13:45:37.544Z] FAILED ../../src/main/python/join_test.py::test_sortmerge_join_with_condition_ast[LeftSemi-Short][DATAGEN_SEED=1726749170, TZ=UTC, INJECT_OOM, IGNORE_ORDER({'local': True})] - AssertionError: CPU and GPU list have different lengths at [] CPU: 239 GPU:...
[2024-09-19T13:45:37.544Z] FAILED ../../src/main/python/join_test.py::test_sortmerge_join_with_condition_ast[LeftSemi-Integer][DATAGEN_SEED=1726749170, TZ=UTC, INJECT_OOM, IGNORE_ORDER({'local': True})] - AssertionError: CPU and GPU list have different lengths at [] CPU: 236 GPU:...
[2024-09-19T13:45:37.544Z] FAILED ../../src/main/python/join_test.py::test_sortmerge_join_with_condition_ast[LeftSemi-Long][DATAGEN_SEED=1726749170, TZ=UTC, INJECT_OOM, IGNORE_ORDER({'local': True})] - AssertionError: CPU and GPU list have different lengths at [] CPU: 235 GPU:...
[2024-09-19T13:45:37.544Z] FAILED ../../src/main/python/join_test.py::test_sortmerge_join_with_condition_ast[LeftSemi-Date][DATAGEN_SEED=1726749170, TZ=UTC, IGNORE_ORDER({'local': True})] - AssertionError: CPU and GPU list have different lengths at [] CPU: 253 GPU:...
[2024-09-19T13:45:37.544Z] FAILED ../../src/main/python/join_test.py::test_sortmerge_join_with_condition_ast[LeftSemi-Timestamp][DATAGEN_SEED=1726749170, TZ=UTC, INJECT_OOM, IGNORE_ORDER({'local': True})] - AssertionError: CPU and GPU list have different lengths at [] CPU: 240 GPU:...
[2024-09-19T13:45:37.544Z] FAILED ../../src/main/python/join_test.py::test_sortmerge_join_with_condition_ast[LeftSemi-String][DATAGEN_SEED=1726749170, TZ=UTC, IGNORE_ORDER({'local': True})] - AssertionError: CPU and GPU list have different lengths at [] CPU: 230 GPU:...
[2024-09-19T13:45:37.544Z] FAILED ../../src/main/python/join_test.py::test_sortmerge_join_with_condition_ast[LeftAnti-Boolean][DATAGEN_SEED=1726749170, TZ=UTC, IGNORE_ORDER({'local': True})] - AssertionError: CPU and GPU list have different lengths at [] CPU: 43 GPU: 296
[2024-09-19T13:45:37.544Z] FAILED ../../src/main/python/join_test.py::test_sortmerge_join_with_condition_ast[LeftAnti-Byte][DATAGEN_SEED=1726749170, TZ=UTC, IGNORE_ORDER({'local': True})] - AssertionError: CPU and GPU list have different lengths at [] CPU: 202 GPU:...
[2024-09-19T13:45:37.544Z] FAILED ../../src/main/python/join_test.py::test_sortmerge_join_with_condition_ast[LeftAnti-Short][DATAGEN_SEED=1726749170, TZ=UTC, INJECT_OOM, IGNORE_ORDER({'local': True})] - AssertionError: CPU and GPU list have different lengths at [] CPU: 261 GPU:...
[2024-09-19T13:45:37.544Z] FAILED ../../src/main/python/join_test.py::test_sortmerge_join_with_condition_ast[LeftAnti-Integer][DATAGEN_SEED=1726749170, TZ=UTC, IGNORE_ORDER({'local': True})] - AssertionError: CPU and GPU list have different lengths at [] CPU: 264 GPU:...
[2024-09-19T13:45:37.544Z] FAILED ../../src/main/python/join_test.py::test_sortmerge_join_with_condition_ast[LeftAnti-Long][DATAGEN_SEED=1726749170, TZ=UTC, IGNORE_ORDER({'local': True})] - AssertionError: CPU and GPU list have different lengths at [] CPU: 265 GPU:...
[2024-09-19T13:45:37.544Z] FAILED ../../src/main/python/join_test.py::test_sortmerge_join_with_condition_ast[LeftAnti-Date][DATAGEN_SEED=1726749170, TZ=UTC, INJECT_OOM, IGNORE_ORDER({'local': True})] - AssertionError: CPU and GPU list have different lengths at [] CPU: 247 GPU:...
[2024-09-19T13:45:37.544Z] FAILED ../../src/main/python/join_test.py::test_sortmerge_join_with_condition_ast[LeftAnti-Timestamp][DATAGEN_SEED=1726749170, TZ=UTC, INJECT_OOM, IGNORE_ORDER({'local': True})] - AssertionError: CPU and GPU list have different lengths at [] CPU: 260 GPU: 28
[2024-09-19T13:45:37.544Z] FAILED ../../src/main/python/join_test.py::test_sortmerge_join_with_condition_ast[LeftAnti-String][DATAGEN_SEED=1726749170, TZ=UTC, IGNORE_ORDER({'local': True})] - AssertionError: CPU and GPU list have different lengths at [] CPU: 270 GPU:...
[2024-09-19T13:45:37.544Z] = 32 failed, 30698 passed, 1099 skipped, 1091 xfailed, 723 xpassed, 14293 warnings in 4366.24s (1:12:46) =

[abellina]

I believe this is because of rapidsai/cudf#16230.

[sameerz]

Waiting for a revert PR rapidsai/cudf#16855 for a short term fix.

[pxLi]

rapidsai/cudf#16855 is merged. Triggering submodule syncup, and will re-build JNI later

[pxLi]

A snapshot JNI with the revert commit is available. lets see if this resolves the issue.