Unified rc12 testing: `np.int` error on Perlmutter

[forsyth2]

Running into the following on global_time_series on Perlmutter:

Traceback (most recent call last):
  File "/global/cfs/cdirs/e3sm/forsyth/zppy_test_complete_run_output/v2.LR.historical_0201/post/scripts/global_time_series_18\
50-1860_dir/coupled_global.py", line 603, in <module>
    run(sys.argv)
  File "/global/cfs/cdirs/e3sm/forsyth/zppy_test_complete_run_output/v2.LR.historical_0201/post/scripts/global_time_series_18\
50-1860_dir/coupled_global.py", line 587, in run
    PLOT_DICT[plot_list[i]](ax, xlim, exps)
  File "/global/cfs/cdirs/e3sm/forsyth/zppy_test_complete_run_output/v2.LR.historical_0201/post/scripts/global_time_series_18\
50-1860_dir/coupled_global.py", line 295, in plot_max_moc
    plot(ax, xlim, exps, param_dict)
  File "/global/cfs/cdirs/e3sm/forsyth/zppy_test_complete_run_output/v2.LR.historical_0201/post/scripts/global_time_series_18\
50-1860_dir/coupled_global.py", line 374, in plot
    [year, var] = getmoc(exp["moc"])
  File "/global/cfs/cdirs/e3sm/forsyth/zppy_test_complete_run_output/v2.LR.historical_0201/post/scripts/global_time_series_18\
50-1860_dir/coupled_global.py", line 29, in getmoc
    for iyear in range(np.int(time0[0]), np.int(time0[-1]) + 1):
  File "/global/common/software/e3sm/anaconda_envs/base/envs/e3sm_unified_1.9.0rc12_pm-cpu/lib/python3.10/site-packages/numpy\
/__init__.py", line 305, in __getattr__
    raise AttributeError(__former_attrs__[attr])
AttributeError: module 'numpy' has no attribute 'int'.
`np.int` was a deprecated alias for the builtin `int`. To avoid this error in existing code, use `int` by itself. Doing this \
will not modify any behavior and is safe. When replacing `np.int`, you may wish to use e.g. `np.int64` or `np.int32` to speci\
fy the precision. If you wish to review your current use, check the release note link for additional information.
The aliases was originally deprecated in NumPy 1.20; for more details and guidance see the original release note at:
    https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations. Did you mean: 'inf'?

This np.int issue was resolved by https://github.com/E3SM-Project/zppy/pull/466/files#diff-a10da0e74d3b76ccfa4c61b3666ceb36b458e72e82820d85c357507070820badL29, so it shouldn't be an issue in Unified rc12...

[forsyth2]

/global/common/software/e3sm/anaconda_envs/base/envs/e3sm_unified_1.9.0rc12_login/lib/python3.10/site-packages/zppy/templates/coupled_global.py:
for iyear in range(int(time0[0]), int(time0[-1]) + 1):

/global/cfs/cdirs/e3sm/forsyth/zppy_test_complete_run_output/v2.LR.historical_0201/post/scripts/global_time_series_1850-1860_dir/coupled_global.py:
for iyear in range(np.int(time0[0]), np.int(time0[-1]) + 1):

[forsyth2]

Compare to Compy:

/compyfs/fors729/zppy_test_complete_run_output/v2.LR.historical_0201/post/scripts/global_time_series_1850-1860_dir/coupled_global.py:
for iyear in range(int(time0[0]), int(time0[-1]) + 1):

[forsyth2]

$ conda list zppy
# packages in environment at /global/common/software/e3sm/anaconda_envs/base/envs/e3sm_unified_1.9.0rc12_login:
#
# Name                    Version                   Build  Channel
zppy                      2.3.0rc5           pyh51c0ceb_0    conda-forge/label/zppy_dev

[chengzhuzhang]

I'm looking at the source code in rc12.

/global/common/software/e3sm/anaconda_envs/base/envs/e3sm_unified_1.9.0rc12_pm-cpu/lib/python3.10/site-packages/zppy/templates/coupled_global.py

The script is up-to-date.

It seem like we can rule out that:

• zppy is executed in an old e3sm-unified rc
• The coupled_global.py script in run directory is not newly generated.
  Then I'm out of ideas.

I guess, it is worthwhile to try create an result empty directory and re-test the run.

[forsyth2]

Thanks @chengzhuzhang. Yeah, I'll try re-running.

[xylar]

@forsyth2, please keep me posted on this. I can't find anywhere in e3sm_diags or zppy that np.int is mentioned. This is the last issue I know of before we can release E3SM-Unified.

[forsyth2]

I moved the post directory from the previous run and reran. Still running into the same issue.

$ cd /global/cfs/cdirs/e3sm/forsyth/zppy_test_complete_run_output/v2.LR.historical_0201/post/scripts
$ grep -v "OK" *status
global_time_series_1850-1860.status:ERROR (5)
$ tail -n 17 global_time_series_1850-1860.o14019357 
Traceback (most recent call last):
  File "/global/cfs/cdirs/e3sm/forsyth/zppy_test_complete_run_output/v2.LR.historical_0201/post/scripts/global_time_series_1850-1860_dir/coupled_global.py", line 603, in <module>
    run(sys.argv)
  File "/global/cfs/cdirs/e3sm/forsyth/zppy_test_complete_run_output/v2.LR.historical_0201/post/scripts/global_time_series_1850-1860_dir/coupled_global.py", line 587, in run
    PLOT_DICT[plot_list[i]](ax, xlim, exps)
  File "/global/cfs/cdirs/e3sm/forsyth/zppy_test_complete_run_output/v2.LR.historical_0201/post/scripts/global_time_series_1850-1860_dir/coupled_global.py", line 295, in plot_max_moc
    plot(ax, xlim, exps, param_dict)
  File "/global/cfs/cdirs/e3sm/forsyth/zppy_test_complete_run_output/v2.LR.historical_0201/post/scripts/global_time_series_1850-1860_dir/coupled_global.py", line 374, in plot
    [year, var] = getmoc(exp["moc"])
  File "/global/cfs/cdirs/e3sm/forsyth/zppy_test_complete_run_output/v2.LR.historical_0201/post/scripts/global_time_series_1850-1860_dir/coupled_global.py", line 29, in getmoc
    for iyear in range(np.int(time0[0]), np.int(time0[-1]) + 1):
  File "/global/common/software/e3sm/anaconda_envs/base/envs/e3sm_unified_1.9.0rc12_pm-cpu/lib/python3.10/site-packages/numpy/__init__.py", line 305, in __getattr__
    raise AttributeError(__former_attrs__[attr])
AttributeError: module 'numpy' has no attribute 'int'.
`np.int` was a deprecated alias for the builtin `int`. To avoid this error in existing code, use `int` by itself. Doing this will not modify any behavior and is safe. When replacing `np.int`, you may wish to use e.g. `np.int64` or `np.int32` to specify the precision. If you wish to review your current use, check the release note link for additional information.
The aliases was originally deprecated in NumPy 1.20; for more details and guidance see the original release note at:
    https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations. Did you mean: 'inf'?

[xylar]

@forsyth2, can you post your exact set of commands that is causing this error? Starting with cloning the zppy repo and creating a conda environment if that is part of the workflow? It feels to me like the testing process itself might be messing things up and bringing in the incorrect template from somewhere else.

[forsyth2]

@xylar I'm following the directions at https://github.com/E3SM-Project/zppy/pull/440/files#diff-b84f28870c999d293db6137dfb571c070d7e3b9ccdb1b8263e32a3c3e45f8dd6 (docs/source/dev_guide/pre_release_testing.rst):

# Step 1
# Log into Perlmutter

# Step 2
$ cd /global/homes/f/forsyth/zppy # Repo already exists

# Step 3
$ git fetch upstream main
$ git checkout -b test_unified_rc12 upstream/main
$ git log # check the commits match https://github.com/E3SM-Project/zppy/commits/main
# Most recent: https://github.com/E3SM-Project/zppy/commit/49933f97cd80ab81c436a16bdf85fa9d114f9690 (bump to rc5)

# Step 4
$ cd ../e3sm_diags
$ git checkout main
$ git fetch upstream
$ git reset --hard upstream/main
$ git log # Should match https://github.com/E3SM-Project/e3sm_diags/commits/main
# Most recent: https://github.com/E3SM-Project/e3sm_diags/commit/6aaa7c27dcf65393dd07e258ef2e7da21a660e87 (bump to rc3)
$ mamba clean --all
$ mamba env create -f conda-env/dev.yml -n e3sm_diags_20230816
$ conda activate e3sm_diags_20230816
$ pip install .

# Step 5
$ cd ../zppy
$ emacs tests/integration/utils.py
# 1. Confirm last line includes `generate_cfgs(unified_testing=True)`
# Under `def get_perlmutter_expansions(config):`, set:
# 2. "diags_environment_commands": "source /global/homes/f/forsyth/miniconda3/etc/profile.d/conda.sh; conda activate e3sm_diags_20230816"
# 3. "environment_commands_test": "source /global/common/software/e3sm/anaconda_envs/test_e3sm_unified_1.9.0rc12_pm-cpu.sh",

# Step 6
# Use Unified rc12 rather than zppy dev environment:
$ source /global/common/software/e3sm/anaconda_envs/test_e3sm_unified_1.9.0rc12_pm-cpu.sh

$ Step 7
$ python -u -m unittest tests/test_*.py
# Unit tests pass

$ Step 8 (this is the step I repeat to re-test)
# Follow directions at https://github.com/E3SM-Project/zppy/blob/main/tests/integration/generated/directions_pm-cpu.md:
$ rm -rf /global/cfs/cdirs/e3sm/www/forsyth/zppy_test_complete_run_www/v2.LR.historical_0201 # Or move somewhere else
$ rm -rf /global/cfs/cdirs/e3sm/forsyth/zppy_test_complete_run_output/v2.LR.historical_0201/post # Or move somewhere else
$ python tests/integration/utils.py # only did the first time. Have a hard-coded time increase for MPAS-Analysis at the moment
$ zppy -c tests/integration/generated/test_complete_run_pm-cpu.cfg
# [wait for this to run]
$ cd /global/cfs/cdirs/e3sm/forsyth/zppy_test_complete_run_output/v2.LR.historical_0201/post/scripts
$ grep -v "OK" *status
global_time_series_1850-1860.status:ERROR (5)
$ tail -n 17 global_time_series_1850-1860.o14019357 
Traceback (most recent call last):
  File "/global/cfs/cdirs/e3sm/forsyth/zppy_test_complete_run_output/v2.LR.historical_0201/post/scripts/global_time_series_1850-1860_dir/coupled_global.py", line 603, in <module>
    run(sys.argv)
  File "/global/cfs/cdirs/e3sm/forsyth/zppy_test_complete_run_output/v2.LR.historical_0201/post/scripts/global_time_series_1850-1860_dir/coupled_global.py", line 587, in run
    PLOT_DICT[plot_list[i]](ax, xlim, exps)
  File "/global/cfs/cdirs/e3sm/forsyth/zppy_test_complete_run_output/v2.LR.historical_0201/post/scripts/global_time_series_1850-1860_dir/coupled_global.py", line 295, in plot_max_moc
    plot(ax, xlim, exps, param_dict)
  File "/global/cfs/cdirs/e3sm/forsyth/zppy_test_complete_run_output/v2.LR.historical_0201/post/scripts/global_time_series_1850-1860_dir/coupled_global.py", line 374, in plot
    [year, var] = getmoc(exp["moc"])
  File "/global/cfs/cdirs/e3sm/forsyth/zppy_test_complete_run_output/v2.LR.historical_0201/post/scripts/global_time_series_1850-1860_dir/coupled_global.py", line 29, in getmoc
    for iyear in range(np.int(time0[0]), np.int(time0[-1]) + 1):
  File "/global/common/software/e3sm/anaconda_envs/base/envs/e3sm_unified_1.9.0rc12_pm-cpu/lib/python3.10/site-packages/numpy/__init__.py", line 305, in __getattr__
    raise AttributeError(__former_attrs__[attr])
AttributeError: module 'numpy' has no attribute 'int'.
`np.int` was a deprecated alias for the builtin `int`. To avoid this error in existing code, use `int` by itself. Doing this will not modify any behavior and is safe. When replacing `np.int`, you may wish to use e.g. `np.int64` or `np.int32` to specify the precision. If you wish to review your current use, check the release note link for additional information.
The aliases was originally deprecated in NumPy 1.20; for more details and guidance see the original release note at:
    https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations. Did you mean: 'inf'?

[chengzhuzhang]

Thanks @xylar for the suggestion for @forsyth2 to share the testing steps. I'm trying to locate the central piece, which is configuration file tests/integration/generated/test_complete_run_pm-cpu.cfg (link)on the branchtest_unified_rc12.

It has environment_commands = "source /lcrc/soft/climate/e3sm-unified/test_e3sm_unified_1.9.0rc9_chrysalis.sh" in [default] session. I'm wondering if this is why coupled_global.py had numpy.int which was only in older rcs of zppy.