feat: add support for Python 3.14#21540
Conversation
|
Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually. Contributors can view more details about this message here. |
|
/ok to test |
xref #21540 xref rapidsai/build-planning#205 xref rapidsai/dask-cuda#1629 We _should_ be able to merge in Python 3.14 support PRs without admin-merges because of limited APIs support, but the `numba` dependency here needs to be bumped for Python 3.14 testing in `dask-cuda` and others.
d401554 to
9239d76
Compare
|
/ok to test |
42b1f7a to
8ffa4c1
Compare
|
/ok to test |
1 similar comment
|
/ok to test |
ad9c93e to
37117a7
Compare
|
/ok to test |
7d1898e to
d9d0d47
Compare
|
/ok to test |
ccb38b7 to
b94b323
Compare
|
/ok to test |
| version_lte() { | ||
| [ "$1" = "$(echo -e "$1\n$2" | sort -V | head -n1)" ] | ||
| } | ||
|
|
||
| if version_lte "${RAPIDS_PY_VERSION}" "3.13"; then | ||
| for version in "${versions[@]}"; do | ||
| echo "Installing pandas version: ${version}" | ||
| # This loop tests cudf.pandas compatibility with older pandas-numpy versions, | ||
| # requiring numpy<2. cupy>=14 dropped support for numpy<2, so we explicitly | ||
| # downgrade cupy here to avoid an import failure when cupy tries | ||
| # to load against the older numpy. |
There was a problem hiding this comment.
Python 3.14 requires numpy>=2 so skipping this check for Python >3.13
There was a problem hiding this comment.
In the "else" block please print something like "Python ${RAPIDS_PY_VERSION} detected, which is newer than 3.13. Skipping cudf.pandas compatibility tests with numpy < 2."
There was a problem hiding this comment.
Agree that a print would be helpful, the change is fine though. If this winds up being the only change request feel free to add the print in a follow-up PR fixing the pr.yaml changed files.
| # in versions < 2.12, and trigger a pyparsing DeprecationWarning during collection. | ||
| # See https://github.com/pola-rs/polars/pull/25854 | ||
| sed -i 's/^pydantic>=2.0.0.*/pydantic>=2.0.0,<2.12.0/' polars/py-polars/requirements-dev.txt | ||
| sed -i 's/^pydantic>=2.0.0.*/pydantic>=2.0.0,<2.13.0/' polars/py-polars/requirements-dev.txt |
There was a problem hiding this comment.
I think it was pydantic that caused failures from pyo3 when compiling with Python 3.14, so bumping that bound as 2.12.0 apparently has 3.14 support
There was a problem hiding this comment.
I'm a bit confused here. The comment above specifically calls out versions < 2.12 so I am hesitant to change this unless we have established this is safe to change. Probably requires tracing through the linked PR and figure out what versions of what packages are affected.
There was a problem hiding this comment.
Ok, this is a hard blocker for Python 3.14 support
There was a problem hiding this comment.
I'm going to defer to @Matt711 for this question since he would have added this exact filter.
There was a problem hiding this comment.
Okay a couple of things for more context:
- There was a regression in pydantic 2.12.0 (Accelerate stable-segmented-sort with CUB segmented sort #12347) that broke pyiceberg tests in polars. This was
- Also there's a deprecation warning:
PydanticDeprecatedSince212(doesn't exist in < 2.12) that the pyiceberg tests we're using to filter with.
For 1., this was fixed in pydantic 2.12.1 which we would pick up in this new pinning (so that's 👍🏾 )
For 2., we don't need filter the warning out in https://github.com/rapidsai/cudf/pull/21540/changes#diff-7132576ea4ccfd76284de2d8aced9ed5a3e4f2d997c27b7b5ecd2447702361d8R45-R46 anymore now that we'd pick up 2.12.1.
There was a problem hiding this comment.
This shouldn't block this PR, but ideally the pydantic pinning would be >=2.12.1,<2.13.0 and we can remove the code that filters the PydanticDeprecatedSince212 in this script.
There was a problem hiding this comment.
Or actually the pinning would be >=2.12.1 and with the upper bound removed entirely
There was a problem hiding this comment.
Ok -- I'll merge this in so we get the test filters fixed in CI, then I can put up a follow-up to handle the warning removal, bump the pydantic pinning, and fix up the changed-files filters
| version_gte() { | ||
| [ "$2" = "$(echo -e "$2\n$1" | sort -V | head -n1)" ] | ||
| } | ||
|
|
||
| # If the RAPIDS_PY_VERSION is 3.13, set CUDF_TEST_COPY_ON_WRITE to '1' to enable copy-on-write tests. | ||
| if [[ "${RAPIDS_PY_VERSION}" == "3.13" ]]; then | ||
| if version_gte "${RAPIDS_PY_VERSION}" "3.13"; then |
There was a problem hiding this comment.
Assuming we want to run CoW tests for >=Python 3.13, not just 3.13
There was a problem hiding this comment.
I am guessing this was just meant to be "latest" for partial coverage. @vyasr?
There was a problem hiding this comment.
It was just meant for partial coverage, yes. There's no harm in making this a gte, though. It's fine to change what percentage of the matrix runs with CoW on or off as long as we have some coverage in both cases. This branching will be removed before we start to support the next Python version (3.15) since by then cudf will require pandas 3.0 where copy-on-write is the only supported behavior.
| - 'python/cudf_polars/**' | ||
| dask_cudf_only: | ||
| - 'python/dask_cudf/**' | ||
| # cpp_only: |
There was a problem hiding this comment.
Reminder to revert this and any other temporary changes.
There was a problem hiding this comment.
I don't think these should be reverted. The filters are wrong and are skipping portions of the test suite when they shouldn't.
There was a problem hiding this comment.
Yeah we discussed this on Slack, we're skipping tests that we shouldn't. We should be either removing these filters or correcting them.
There was a problem hiding this comment.
This is frustrating. Can I at least add the cpp_only one back?
There was a problem hiding this comment.
It was added back in a working form in #21622
| version_lte() { | ||
| [ "$1" = "$(echo -e "$1\n$2" | sort -V | head -n1)" ] | ||
| } | ||
|
|
||
| if version_lte "${RAPIDS_PY_VERSION}" "3.13"; then | ||
| for version in "${versions[@]}"; do | ||
| echo "Installing pandas version: ${version}" | ||
| # This loop tests cudf.pandas compatibility with older pandas-numpy versions, | ||
| # requiring numpy<2. cupy>=14 dropped support for numpy<2, so we explicitly | ||
| # downgrade cupy here to avoid an import failure when cupy tries | ||
| # to load against the older numpy. |
There was a problem hiding this comment.
In the "else" block please print something like "Python ${RAPIDS_PY_VERSION} detected, which is newer than 3.13. Skipping cudf.pandas compatibility tests with numpy < 2."
| # in versions < 2.12, and trigger a pyparsing DeprecationWarning during collection. | ||
| # See https://github.com/pola-rs/polars/pull/25854 | ||
| sed -i 's/^pydantic>=2.0.0.*/pydantic>=2.0.0,<2.12.0/' polars/py-polars/requirements-dev.txt | ||
| sed -i 's/^pydantic>=2.0.0.*/pydantic>=2.0.0,<2.13.0/' polars/py-polars/requirements-dev.txt |
There was a problem hiding this comment.
I'm a bit confused here. The comment above specifically calls out versions < 2.12 so I am hesitant to change this unless we have established this is safe to change. Probably requires tracing through the linked PR and figure out what versions of what packages are affected.
| version_gte() { | ||
| [ "$2" = "$(echo -e "$2\n$1" | sort -V | head -n1)" ] | ||
| } | ||
|
|
||
| # If the RAPIDS_PY_VERSION is 3.13, set CUDF_TEST_COPY_ON_WRITE to '1' to enable copy-on-write tests. | ||
| if [[ "${RAPIDS_PY_VERSION}" == "3.13" ]]; then | ||
| if version_gte "${RAPIDS_PY_VERSION}" "3.13"; then |
There was a problem hiding this comment.
I am guessing this was just meant to be "latest" for partial coverage. @vyasr?
| - 'python/cudf_polars/**' | ||
| dask_cudf_only: | ||
| - 'python/dask_cudf/**' | ||
| # cpp_only: |
There was a problem hiding this comment.
Yeah we discussed this on Slack, we're skipping tests that we shouldn't. We should be either removing these filters or correcting them.
| version_lte() { | ||
| [ "$1" = "$(echo -e "$1\n$2" | sort -V | head -n1)" ] | ||
| } | ||
|
|
||
| if version_lte "${RAPIDS_PY_VERSION}" "3.13"; then | ||
| for version in "${versions[@]}"; do | ||
| echo "Installing pandas version: ${version}" | ||
| # This loop tests cudf.pandas compatibility with older pandas-numpy versions, | ||
| # requiring numpy<2. cupy>=14 dropped support for numpy<2, so we explicitly | ||
| # downgrade cupy here to avoid an import failure when cupy tries | ||
| # to load against the older numpy. |
There was a problem hiding this comment.
Agree that a print would be helpful, the change is fine though. If this winds up being the only change request feel free to add the print in a follow-up PR fixing the pr.yaml changed files.
| version_gte() { | ||
| [ "$2" = "$(echo -e "$2\n$1" | sort -V | head -n1)" ] | ||
| } | ||
|
|
||
| # If the RAPIDS_PY_VERSION is 3.13, set CUDF_TEST_COPY_ON_WRITE to '1' to enable copy-on-write tests. | ||
| if [[ "${RAPIDS_PY_VERSION}" == "3.13" ]]; then | ||
| if version_gte "${RAPIDS_PY_VERSION}" "3.13"; then |
There was a problem hiding this comment.
It was just meant for partial coverage, yes. There's no harm in making this a gte, though. It's fine to change what percentage of the matrix runs with CoW on or off as long as we have some coverage in both cases. This branching will be removed before we start to support the next Python version (3.15) since by then cudf will require pandas 3.0 where copy-on-write is the only supported behavior.
| # in versions < 2.12, and trigger a pyparsing DeprecationWarning during collection. | ||
| # See https://github.com/pola-rs/polars/pull/25854 | ||
| sed -i 's/^pydantic>=2.0.0.*/pydantic>=2.0.0,<2.12.0/' polars/py-polars/requirements-dev.txt | ||
| sed -i 's/^pydantic>=2.0.0.*/pydantic>=2.0.0,<2.13.0/' polars/py-polars/requirements-dev.txt |
There was a problem hiding this comment.
I'm going to defer to @Matt711 for this question since he would have added this exact filter.
|
/merge |
This reverts commit edf65a0.
- **fix(test_cudf_polars): set new pydantic lower-bound, remove warning filtering** - **chore(cudf-pandas): print message when skipping pandas.cudf compat tests** - **refactor(changed-files): create negation groups for job skipping** Cleanup and followup of some changes made in #21540 Authors: - Gil Forsyth (https://github.com/gforsyth) - Matthew Murray (https://github.com/Matt711) Approvers: - Kyle Edwards (https://github.com/KyleFromNVIDIA) - James Lamb (https://github.com/jameslamb) URL: #21622
## Description Scanning https://anaconda.org/channels/conda-forge/packages/librdkafka, it appears that 2.11.1 is the first version to support Python 3.14. I think this is the cause of the devcontainer conda failures e.g. https://github.com/rapidsai/cudf/actions/runs/23554430068/job/68577154587?pr=21921 (below). So this PR bumps `librdkafka` and follows our similar, narrow pinning so we support Python 3.14 as well. However, the [python-confluent-kafka recipe](https://github.com/conda-forge/python-confluent-kafka-feedstock/blob/main/recipe/meta.yaml#L38) specifies `- librdkafka >={{ version }}` so we're effectively also pinning `python-confluent-kafka` even though it was unpinned on our side in #21540 ``` The following packages are incompatible ├─ librdkafka >=2.8.0,<2.9.0 * is requested and can be installed; ├─ python-confluent-kafka =* * is installable with the potential options │ ├─ python-confluent-kafka 2.1.0 would require │ │ └─ librdkafka >=2.1.0,<2.2.0a0 *, which conflicts with any installable versions previously reported; │ ├─ python-confluent-kafka 2.1.1 would require │ │ └─ librdkafka >=2.1.1,<2.2.0a0 *, which conflicts with any installable versions previously reported; │ ├─ python-confluent-kafka 2.10.0 would require │ │ └─ librdkafka >=2.10.0,<2.11.0a0 *, which conflicts with any installable versions previously reported; │ ├─ python-confluent-kafka 2.10.1 would require │ │ └─ librdkafka >=2.10.1,<2.11.0a0 *, which conflicts with any installable versions previously reported; │ ├─ python-confluent-kafka 2.11.0 would require │ │ └─ librdkafka >=2.11.0,<2.12.0a0 *, which conflicts with any installable versions previously reported; │ ├─ python-confluent-kafka 2.11.1 would require │ │ └─ librdkafka >=2.11.1,<2.12.0a0 *, which conflicts with any installable versions previously reported; │ ├─ python-confluent-kafka 2.12.2 would require │ │ └─ librdkafka >=2.12.1,<2.13.0a0 *, which conflicts with any installable versions previously reported; │ ├─ python-confluent-kafka 2.13.0 would require │ │ └─ librdkafka >=2.13.0 *, which conflicts with any installable versions previously reported; │ ├─ python-confluent-kafka 2.13.2 would require │ │ └─ librdkafka >=2.13.2 *, which conflicts with any installable versions previously reported; │ ├─ python-confluent-kafka 2.2.0 would require │ │ └─ librdkafka >=2.2.0,<2.3.0a0 *, which conflicts with any installable versions previously reported; │ ├─ python-confluent-kafka 2.3.0 would require │ │ └─ librdkafka >=2.3.0,<2.4.0a0 *, which conflicts with any installable versions previously reported; │ ├─ python-confluent-kafka 2.4.0 would require │ │ └─ librdkafka >=2.4.0,<2.5.0a0 *, which conflicts with any installable versions previously reported; │ ├─ python-confluent-kafka 2.5.0 would require │ │ └─ librdkafka >=2.5.0,<2.6.0a0 *, which conflicts with any installable versions previously reported; │ ├─ python-confluent-kafka 2.5.3 would require │ │ └─ librdkafka >=2.5.3,<2.6.0a0 *, which conflicts with any installable versions previously reported; │ ├─ python-confluent-kafka 2.6.0 would require │ │ └─ librdkafka >=2.6.0,<2.7.0a0 *, which conflicts with any installable versions previously reported; │ ├─ python-confluent-kafka 2.6.1 would require │ │ └─ librdkafka >=2.6.1,<2.7.0a0 *, which conflicts with any installable versions previously reported; │ ├─ python-confluent-kafka 2.8.0 would require │ │ └─ python_abi =3.10 *_cp310, which can be installed; │ ├─ python-confluent-kafka 2.8.0 would require │ │ └─ python_abi =3.11 *_cp311, which can be installed; │ ├─ python-confluent-kafka 2.8.0 would require │ │ └─ python_abi =3.12 *_cp312, which can be installed; │ ├─ python-confluent-kafka 2.8.0 would require │ │ └─ python_abi =3.13 *_cp313, which can be installed; │ ├─ python-confluent-kafka 2.8.0 would require │ │ └─ python_abi =3.9 *_cp39, which can be installed; │ ├─ python-confluent-kafka [0.11.0|0.11.4|0.9.4] would require │ │ └─ librdkafka [==0.9.4 *|>=0.9.4,<0.9.5.0a0 *], which conflicts with any installable versions previously reported; │ ├─ python-confluent-kafka 0.11.4 would require │ │ └─ librdkafka >=0.11.0,<0.12 *, which conflicts with any installable versions previously reported; │ ├─ python-confluent-kafka 0.11.4 would require │ │ └─ librdkafka >=0.11.5,<0.11.6.0a0 *, which conflicts with any installable versions previously reported; │ ├─ python-confluent-kafka 1.0.1 would require │ │ └─ librdkafka >=1.0.1,<1.0.2.0a0 *, which conflicts with any installable versions previously reported; │ ├─ python-confluent-kafka [1.1.0|1.3.0] would require │ │ └─ python_abi =* *_cp27mu, which can be installed; │ ├─ python-confluent-kafka [1.1.0|1.3.0] would require │ │ └─ python_abi =* *_cp36m, which can be installed; │ ├─ python-confluent-kafka [1.1.0|1.3.0] would require │ │ └─ python_abi =* *_cp37m, which can be installed; │ ├─ python-confluent-kafka [1.1.0|1.3.0] would require │ │ └─ python_abi =* *_cp38, which can be installed; │ ├─ python-confluent-kafka 1.3.0 would require │ │ └─ librdkafka >=1.4.0,<1.5.0a0 *, which conflicts with any installable versions previously reported; │ ├─ python-confluent-kafka [1.3.0|1.5.0] would require │ │ └─ librdkafka >=1.5.0,<1.6.0a0 *, which conflicts with any installable versions previously reported; │ ├─ python-confluent-kafka [1.5.0|1.6.0] would require │ │ └─ librdkafka >=1.6.0,<1.7.0a0 *, which conflicts with any installable versions previously reported; │ ├─ python-confluent-kafka 1.7.0 would require │ │ └─ librdkafka >=1.7.0,<1.8.0a0 *, which conflicts with any installable versions previously reported; │ ├─ python-confluent-kafka 1.8.2 would require │ │ └─ librdkafka >=1.8.2.0inf.2,<1.9.0a0 *, which conflicts with any installable versions previously reported; │ ├─ python-confluent-kafka 1.9.0 would require │ │ └─ librdkafka >=1.9.1,<1.10.0a0 *, which conflicts with any installable versions previously reported; │ └─ python-confluent-kafka [1.9.0|1.9.2] would require │ └─ librdkafka >=1.9.2,<1.10.0a0 *, which conflicts with any installable versions previously reported; └─ python =3.14 * is not installable because there are no viable options ├─ python [3.14.0|3.14.1|3.14.2|3.14.3] would require │ └─ python_abi =3.14 *_cp314, which conflicts with any installable versions previously reported; ├─ python [3.14.0|3.14.1|3.14.2|3.14.3] would require │ └─ python_abi =3.14 *_cp314t, which conflicts with any installable versions previously reported; └─ python [3.14.0rc1|3.14.0rc2|3.14.0rc3] would require └─ _python_rc =* *, which does not exist (perhaps a missing channel). ``` ## Checklist - [ ] I am familiar with the [Contributing Guidelines](https://github.com/rapidsai/cudf/blob/HEAD/CONTRIBUTING.md). - [ ] New or existing tests cover these changes. - [ ] The documentation is up to date with these changes. Admin merged per eng request
Description
Contributes to rapidsai/build-planning#205
This PR adds support for Python 3.14.
Notes for Reviewers
This is part of ongoing work to add Python 3.14 support across RAPIDS.
It temporarily introduces a build/test matrix including Python 3.14, from rapidsai/shared-workflows#508.
A follow-up PR will revert back to pointing at the
mainbranch ofshared-workflowsonce allRAPIDS repos have added Python 3.14 support.
This will fail until all dependencies have been updated to Python 3.14
CI here is expected to fail until all of this project's upstream dependencies support Python 3.14.
This can be merged whenever all CI jobs are passing.