Upper constraint for shapely needs to be relaxed for e3sm-unified

[andrewdnolan]

Hi there! I'm working with @xylar on configuring and testing the new release of e3sm-unified. We are hitting issue with the current configuration because of pcmdi_metrics, specifically due to your upper constraint on shapely:

pcmdi_metrics/conda-env/dev.yml

Line 26 in 0860eb5

- shapely >=2.0.1,<=2.0.6

We are using proj 9.6.2 and pyproj 3.7.2 in this releasee3sm-unified. (The issue actually has to do with the version of geos used by rasterio needed to support proj 9.6.2 being incompatible with the geos version used with upper constraint of shapely). You can reproduce the error we see by trying to create a conda environment with:

conda create -y -n dry-run --dry-run "python=3.13" "proj 9.6.2" "pyproj 3.7.2" "rasterio" "shapely >=2.0.1,<=2.0.6"

The issue is resolved by removing the upper constraint on shapely (i.e):

conda create -y -n dry-run --dry-run "python=3.13" "proj 9.6.2" "pyproj 3.7.2" "rasterio" "shapely >=2.0.1"

From looking through the git blame and PR history, it looks like the shapely constraint was added to suppress some kind of warning message.

What was the warning? Are you OK with easing that upper shapely constraint? If there's any development that's needed to support the newer versions of shapely, @xylar and I are ready and willing to help with that.

CC @chengzhuzhang @tomvothecoder

[xylar]

I did my best to address this (raise the upper bound) in #1331.

[forsyth2]

Also regarding pcmdi_metrics versions: E3SM-Project/zppy-interfaces#25 (comment). I'm not sure how relevant that is here, but we did see a warning during testing that originated from pcmdi_metrics: The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.

[xylar]

We should fix that separately. Thanks @forsyth2.

[xylar]

@forsyth2, addressed in #1332

[forsyth2]

@xylar Great, thanks for the quick fix!!

[forsyth2]

@xylar @andrewdnolan I was doing unit testing testing on zppy-interfaces and got the following warning (not an error though).

I'm running pcmdi_metrics v4.0.1:

conda list | grep pcmdi
# pcmdi_metrics             4.0.1              pyhd8ed1ab_0    conda-forge

It appears #1332 resolved one warning out of the original two I was getting:

../../miniforge3/envs/zi-pcmdi-diags-20250930/lib/python3.12/site-packages/pcmdi_metrics/resources.py:3
  /gpfs/fs1/home/ac.forsyth2/miniforge3/envs/zi-pcmdi-diags-20250930/lib/python3.12/site-packages/pcmdi_metrics/resources.py:3: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
    import pkg_resources

../../miniforge3/envs/zi-pcmdi-diags-20250930/lib/python3.12/site-packages/pkg_resources/__init__.py:3146
  /gpfs/fs1/home/ac.forsyth2/miniforge3/envs/zi-pcmdi-diags-20250930/lib/python3.12/site-packages/pkg_resources/__init__.py:3146: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('sphinxcontrib')`.
  Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages
    declare_namespace(pkg)

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html

We still have one warning though:

../../miniforge3/envs/zi-main-20251016/lib/python3.13/site-packages/pyogrio/__init__.py:7
  /gpfs/fs1/home/ac.forsyth2/miniforge3/envs/zi-main-20251016/lib/python3.13/site-packages/pyogrio/__init__.py:7: DeprecationWarning: The 'shapely.geos' module is deprecated, and will be removed in a future version. All attributes of 'shapely.geos' are available directly from the top-level 'shapely' namespace (since shapely 2.0.0).
    import shapely.geos  # noqa: F401

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html

This is actually a new warning. It seems like this PR should have resolved this warning though.

In any case, "will be removed in a future version" implies this will work ok for this Unified release, but will need to be addressed before the next one.

[xylar]

@forsyth2 , that warning is in the pyogrio - package and isn't our problem unless those developers don't fix it before something breaks.

[forsyth2]

@xylar Ah I see, sounds good. Thanks!

[lee1043]

Agree to @xylar, and PMP does not directly use pyogrio (but maybe some of PMP's dependency does? Needs to be checked.) Anyway, it looks like pyogrio imports shapely.geos only when shapely version is below 2.0:

https://github.com/geopandas/pyogrio/blob/56a5961e32fa37bc72def28a654c6bfec0285c27/pyogrio/__init__.py#L8-L9

I am curios what shapely version being used in @forsyth2's env.

[lee1043]

@xylar @forsyth2 It looks like pyogrio got the deprecation warning fixed recently: geopandas/pyogrio#542

[forsyth2]

PMP does not directly use pyogrio (but maybe some of PMP's dependency does?

@lee1043 After some iterating with Claude, it comes up with:

Your code → GeoPandas → pyogrio → (deprecated shapely.geos import)

I am curios what shapely version being used in @forsyth2's env.

> conda list --show-channel-urls | grep pyogrio
pyogrio                   0.11.0          py313hab4ff3b_0    conda-forge
> conda list --explicit | grep pyogrio
https://conda.anaconda.org/conda-forge/linux-64/pyogrio-0.11.0-py313hab4ff3b_0.conda
> conda list | grep geo
geometric_features        1.6.1              pyhd8ed1ab_0    conda-forge
geopandas                 1.1.1              pyhd8ed1ab_1    conda-forge
geopandas-base            1.1.1              pyha770c72_1    conda-forge
geos                      3.14.0               h480dda7_0    conda-forge
geotiff                   1.7.4                h1000f5c_4    conda-forge

[lee1043]

@forsyth2 thanks for taking a look. While PMP does not list geopandas as its direct dependency, I found that it is one of dependencies of regionmask that PMP uses.

I think updating pyogrio to v0.11.1 in the env will mute the warning.