zppy-pcmdi run issues on NERC Perlmutter

[zhangshixuan1987]

Question criteria

• I added a descriptive title here.
• I searched the zppy GitHub Discussions to find a similar question and didn't find it.
• I searched the zppy documentation.

What is the deadline?

This is not urgent, but I would greatly appreciate a response when convenient.

Describe your question

I’m seeing two issues when running the zppy workflow on NERSC Perlmutter using E3SM_Unified:

(1) e3sm_to_cmip does not perform vertical interpolation for 3D variables (e.g., U, V, T, Q). It appears that the vertical interpolation step/job is never activated or submitted. As a result, only the 2D variables are converted successfully, while the 3D fields are not produced on the target pressure levels.

(2) The PCMDI step is not triggered. Even with the partially completed e3sm_to_cmip results, PCMDI diagnostics (e.g., modes variability) would normally be expected to run, but no job is created for the PCMDI diagnostics step.

Are there are any possible answers you came across?

One possibility is that the E3SM_Unified environment on Perlmutter may not include a complete installation of the components required for e3sm_to_cmip vertical interpolation and/or zppy’s PCMDI modules.

I mention this because I transferred the same data and configuration to Chrysalis and reran the workflow there. On Chrysalis, all jobs related to e3sm_to_cmip vertical interpolation and zppy-pcmdi were at least triggered as expected, whereas on Perlmutter these jobs are not submitted.

What machine were you running on?

NERSC Perlmutter

Environment

On both Perlmutter and Chrysalis, I activated the same conda environment using:
conda activate e3sm_unified_latest

What command did you run?

The command I used was like below:

cd /pscratch/sd/z/zhan391/EAMxx/ne256pg2_ne256pg2.F20TR-SCREAMv1.July-1.spanc800.2xauto.acc150.n0032.test2.1

zppy -c post_pcmdi.cfg

Copy your cfg file

[default]
input = /pscratch/sd/z/zhan391/EAMxx/ne256pg2_ne256pg2.F20TR-SCREAMv1.July-1.spanc800.2xauto.acc150.n0032.test2.1
output = /pscratch/sd/z/zhan391/EAMxx/ne256pg2_ne256pg2.F20TR-SCREAMv1.July-1.spanc800.2xauto.acc150.n0032.test2.1
case = ne256pg2_ne256pg2.F20TR-SCREAMv1.July-1.spanc800.2xauto.acc150.n0032.test2.1
www = /global/cfs/cdirs/e3sm/www/zhan391/eamxx-pcmdi

partition = debug
#account = priority
environment_commands = "source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_pm-cpu.sh"

[ts]
active = True
walltime = "00:10:00"
years = "1995:2004:1"

  [[ atm_monthly_180x360_aave ]]
  input_subdir = "archive/atm/hist"
  input_files = "eam.h0"
  frequency = "monthly"
  mapping_file = /home/ac.zender/data/maps/map_ne30pg2_to_cmip6_180x360_aave.20200201.nc
  # list for pcmdi diag, note: PHIS,hyam,hybm,hyai,hybi need to be included to process the 3D fields
  vars = "ICEFRAC,LANDFRAC,OCNFRAC,PSL,FSNTC,FSNTOAC,SWCF,LWCF,FLUT,FSNT,FSNTOA,FLNT,FLNTC,FSNS,FLNS,FSNS,SHFLX,QFLX,LHFLX,TAUX,TAUY,PRECC,PRECL,PRECSC,PRECSL,TS,TREFHT,U10,QREFHT,TMQ,CLDTOT,CLDHGH,CLDMED,CLDLOW,FLDS,FSDS,TGCLDIWP,TGCLDCWP,TGCLDLWP,FLNSC,FLUTC,FSDSC,SOLIN,FSNSC,AODABS,AODVIS,AODDUST,AREL,TREFMNAV,TREFMXAV,PS,PHIS,U,V,T,Z3"


[e3sm_to_cmip]
active = True
years = "1995:2004:1"
walltime = "00:50:00"
frequency = "monthly"
grid = "180x360_aave"
ts_grid = "180x360_aave"
ts_num_years=1

  [[ atm_monthly_180x360_aave ]]
  # list of variables to be interpolated to pressure level; see e3sm_to_cmip/vrt_remap_plev19.nc
  #cmip_plevdata = "/lcrc/group/e3sm/ac.szhang/acme_scratch/e3sm_project/test_zppy_pmp/zppy/zppy/templates/inclusions/e3sm_to_cmip/vrt_remap_plev19.nc"
  input_files = "eam.h0"
  interp_vars = "U,V,T,Q,Z3"
  ts_subsection = "atm_monthly_180x360_aave"
  #vars="PS,PSL,TREFHT,QREFHT,U10,WIND10,T,U,V,Z3,OMEGA,RELHUM,PMID,OMEGA500,OMEGA700,OMEGA850,T700,Z700,LHFLX,SHFLX,QFLX,SOLIN,FSUT,FLUT,FSUTC,FLUTC,FSDS,FSUS,FLDS,FLUS,FSDSC,FSUSC,FLDSC,CLDTOT,SWCF,LWCF,IWP,PRECLIQ,PRECICE"

[pcmdi_diags]
active = True
#environment_commands_secondary = "source /gpfs/fs1/home/ac.forsyth2/miniforge3/etc/profile.d/conda.sh; conda activate zi-pcmdi-diags-20251007-test1"
#environment_commands = "source /lcrc/soft/climate/e3sm-unified/load_latest_e3sm_unified_chrysalis.sh; conda activate pcmdi_metrics_master" 
walltime = "00:30:00"
# Create rules to construct cmip-like datasets with a string constructed as
# "mip.exp.product.realization", where "."is used as the delimiter (mandatory)
model_name = 'e3sm.amip.EAMXX.test2.1'
model_tableID = 'Amon'
multiprocessing = True
num_workers = 24
#observational data 
obs_ts = '/global/cfs/cdirs/e3sm/e3sm_diags/obs_for_e3sm_diags/time-series'
#obs_ts = '/lcrc/group/e3sm/diagnostics/observations/obs4MIPs_PCMDI_monthly'
pcmdi_debug = False
ts_years = "1995-2004",
ts_num_years=1

  [[ mean_climate ]]
  active = True
  # Default for all variables or specify source for each variable separately.
  obs_sets = "default"
  ref_final_yr = 2018
  ref_start_yr = 1979
  ref_years = "1995-2004",
  target_grid = "1x1"
  target_grid_string = "1px1p"
  # Variables in current pcmdi-cmip list
  # rltcre and rstcre are derived variables as they are not included in cmip_ts
  # define a new variable name to be flexible
  clim_vars = "pr,prw,psl,rlds,rldscs,rltcre,rstcre,rsus,rsuscs,rlus,rlut,rlutcs,rsds,rsdscs,rsdt,rsut,rsutcs,rtmt,sfcWind,tas,tauu,tauv,ts,ta-200,ta-850,ua-200,ua-850,va-200,va-850,zg-500"
  # See zppy/templates/pcmdi_diags/regions_specs.json for options of all regions
  # Below is default setup for global/regional mean climate metrics
  clim_regions = "global,ocean,land,Tropics,NHEX,SHEX"

  [[ variability_modes_cpl ]]
  active = True
  # See zppy/templates/pcmdi_diags/reference_alias.json for options
  obs_sets = "alternate1" 
  ref_final_yr = 2019
  ref_start_yr = 1870
  ref_years = "1995-2004",
  frequency = 'mo'
  landmask = True
  ModUnitsAdjust = '(True,"subtract",273.15)'
  ObsUnitsAdjust = '(True,"subtract",273.15)'
  seasons = "yearly,monthly"
  movc_vars = "ts"
  # Name of modes varibility
  movc_modes = "PDO,NPGO,AMO"
  [[ variability_modes_atm ]]
  active = True
  # See zppy/templates/pcmdi_diags/reference_alias.json for options
  # Default for all variables
  obs_sets = "alternate2" # "default"
  ref_final_yr = 2015
  ref_start_yr = 1853
  ref_years = "1995-2004",
  frequency = 'mo'
  landmask = False
  ModUnitsAdjust = '(True,"divide",100.0)'
  ObsUnitsAdjust = '(True,"divide",100.0)'
  seasons = "DJF,MAM,JJA,SON,yearly,monthly"
  mova_vars = "psl"
  # Varibility modes
  mova_modes = "NAM,NAO,PNA,NPO,SAM,PSA1,PSA2"

  [[ enso ]]
  active = False
  # For enso, the obs_sets is explicitely set for each variable to ensure that
  # they are from the same source, below are setup to used ERA5 reanalysis.
  obs_sets = "default,alternate2,default,default,default,default,default,default,default,alternate2,alternate2,alternate2,alternate2,alternate2"
  ref_final_yr = 2018
  ref_start_yr = 1979
  ref_years = "1995-2004",
  # See zppy/templates/pcmdi_diags/reference_alias.json for options
  enso_groups = "ENSO_perf,ENSO_proc,ENSO_tel"
  enso_vars = "psl,pr,prsn,ts,tas,tauu,tauv,hfls,hfss,rlds,rsds,rlus,rlut,rsdt"

  [[ synthetic_plots ]]
  active = True
  # use subset variables for viewer
  clim_vars = "pr,prw,psl,rlds,rldscs,ua-200,zg-500"
  # use subset regions for viewer
  clim_regions = "global,ocean,land"
  # use subset modes for viewer
  mova_modes = "NAM,NAO,PNA,NPO,SAM"
  # use subset modes for viewer
  movc_modes  = "PDO,NPGO"
  # mean climate
  clim_viewer = True
  # period to be plotted and shown on webpage
  # note that the previous sections allows multiple ts_years periods for processing, while
  # the viewer page focused on period customized by user
  clim_years = "2005-2014"
  # modes variability (atmosphere)
  mova_viewer = True
  # period to be plotted and shown on webpage
  # note that the previous sections allows multiple ts_years periods for processing, while
  # the viewer page focused on period customized by user
  mova_years = "2005-2014"
  # modes variability (coupled)
  movc_viewer = True
  # period to be plotted and shown on webpage
  # note that the previous sections allows multiple ts_years periods for processing, while
  # the viewer page focused on period customized by user
  movc_years = "2005-2014"
  # enso
  enso_viewer = False
  enso_years = ""
  # Location: /lcrc/group/e3sm/diagnostics/pcmdi_data/metrics_data/mean_climate/cmip6/amip
  cmip_clim_set = "cmip6.amip.v20250702"
  # Location: /lcrc/group/e3sm/diagnostics/pcmdi_data/metrics_data/variability_modes/cmip6/amip
  cmip_movs_set = "cmip6.amip.v20210119"
  # Location: /lcrc/group/e3sm/diagnostics/pcmdi_data/metrics_data/enso_metric/cmip6/historical
  # There is no enso metric for CMIP6 AMIP, so use historical instead
  cmip_enso_set = "cmip6.historical.v20210620"

What jobs are failing?

No response

What stack trace are you encountering?

No response

[chengzhuzhang]

@zhangshixuan1987 I tried to look at post/scripts/ sub folder and it looks like you are trying something and creating new atm time-series files. I think we should be able to leverage the new logs and troubleshooting further.

[forsyth2]

Reviewing messages from Slack

cd /pscratch/sd/z/zhan391/EAMxx/ne256pg2_ne256pg2.F20TR-SCREAMv1.July-1.spanc800.2xauto.acc150.n0032.test2.1/post/scripts

ls *.bash
# e3sm_to_cmip_atm_monthly_180x360_aave_1995-1995-0001.bash  e3sm_to_cmip_atm_monthly_180x360_aave_2000-2000-0001.bash
# e3sm_to_cmip_atm_monthly_180x360_aave_1996-1996-0001.bash  e3sm_to_cmip_atm_monthly_180x360_aave_2001-2001-0001.bash
# e3sm_to_cmip_atm_monthly_180x360_aave_1997-1997-0001.bash  e3sm_to_cmip_atm_monthly_180x360_aave_2002-2002-0001.bash
# e3sm_to_cmip_atm_monthly_180x360_aave_1998-1998-0001.bash  e3sm_to_cmip_atm_monthly_180x360_aave_2003-2003-0001.bash
# e3sm_to_cmip_atm_monthly_180x360_aave_1999-1999-0001.bash  e3sm_to_cmip_atm_monthly_180x360_aave_2004-2004-0001.bash

ls *.status
# e3sm_to_cmip_atm_monthly_180x360_aave_1995-1995-0001.status  ts_atm_monthly_180x360_aave_1995-2004.status
# e3sm_to_cmip_atm_monthly_180x360_aave_1996-1996-0001.status  ts_atm_monthly_180x360_aave_1996-1996-0001.status
# e3sm_to_cmip_atm_monthly_180x360_aave_1997-1997-0001.status  ts_atm_monthly_180x360_aave_1997-1997-0001.status
# e3sm_to_cmip_atm_monthly_180x360_aave_1998-1998-0001.status  ts_atm_monthly_180x360_aave_1998-1998-0001.status
# e3sm_to_cmip_atm_monthly_180x360_aave_1999-1999-0001.status  ts_atm_monthly_180x360_aave_1999-1999-0001.status
# e3sm_to_cmip_atm_monthly_180x360_aave_2000-2000-0001.status  ts_atm_monthly_180x360_aave_2000-2000-0001.status
# e3sm_to_cmip_atm_monthly_180x360_aave_2001-2001-0001.status  ts_atm_monthly_180x360_aave_2001-2001-0001.status
# e3sm_to_cmip_atm_monthly_180x360_aave_2002-2002-0001.status  ts_atm_monthly_180x360_aave_2002-2002-0001.status
# e3sm_to_cmip_atm_monthly_180x360_aave_2003-2003-0001.status  ts_atm_monthly_180x360_aave_2003-2003-0001.status
# e3sm_to_cmip_atm_monthly_180x360_aave_2004-2004-0001.status  ts_atm_monthly_180x360_aave_2004-2004-0001.status
# ts_atm_monthly_180x360_aave_1995-1995-0001.status

# So ts has status files but no bash files...

grep -v "OK" *status
# grep: ts_atm_monthly_180x360_aave_1995-2004.status: Permission denied

# So no errors in status files, and one status file without permissions set up.

grep -A 1 "environment" e3sm_to_cmip_atm_monthly_180x360_aave_2004-2004-0001.settings
#   'environment_commands': 'source '
#                           '/global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_pm-cpu.sh',

grep "www" e3sm_to_cmip_atm_monthly_180x360_aave_2004-2004-0001.settings
#   'web_portal_base_path': '/global/cfs/cdirs/e3sm/www',
#   'www': '/global/cfs/cdirs/e3sm/www/zhan391/eamxx-pcmdi',
ls /global/cfs/cdirs/e3sm/www/zhan391/eamxx-pcmdi
# Empty

ls provenance.*.cfg
# ls: cannot access 'provenance.*.cfg': No such file or directory

# So, provenance cfg wasn't created

We would expect a provenance cfg to at least exist, since Unified includes #713. In any case, Shixuan provided the cfg:

Provided cfg

[default]
input = /pscratch/sd/z/zhan391/EAMxx/ne256pg2_ne256pg2.F20TR-SCREAMv1.July-1.spanc800.2xauto.acc150.n0032.test2.1
output = /pscratch/sd/z/zhan391/EAMxx/ne256pg2_ne256pg2.F20TR-SCREAMv1.July-1.spanc800.2xauto.acc150.n0032.test2.1
case = ne256pg2_ne256pg2.F20TR-SCREAMv1.July-1.spanc800.2xauto.acc150.n0032.test2.1
www = /global/cfs/cdirs/e3sm/www/zhan391/eamxx-pcmdi

partition = debug
#account = priority
environment_commands = "source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_pm-cpu.sh"

[ts]
active = True
walltime = "00:10:00"
years = "1995:2004:1"

  [[ atm_monthly_180x360_aave ]]
  input_subdir = "archive/atm/hist"
  input_files = "eam.h0"
  frequency = "monthly"
  mapping_file = /home/ac.zender/data/maps/map_ne30pg2_to_cmip6_180x360_aave.20200201.nc
  # list for pcmdi diag, note: PHIS,hyam,hybm,hyai,hybi need to be included to process the 3D fields
  vars = "ICEFRAC,LANDFRAC,OCNFRAC,PSL,FSNTC,FSNTOAC,SWCF,LWCF,FLUT,FSNT,FSNTOA,FLNT,FLNTC,FSNS,FLNS,FSNS,SHFLX,QFLX,LHFLX,TAUX,TAUY,PRECC,PRECL,PRECSC,PRECSL,TS,TREFHT,U10,QREFHT,TMQ,CLDTOT,CLDHGH,CLDMED,CLDLOW,FLDS,FSDS,TGCLDIWP,TGCLDCWP,TGCLDLWP,FLNSC,FLUTC,FSDSC,SOLIN,FSNSC,AODABS,AODVIS,AODDUST,AREL,TREFMNAV,TREFMXAV,PS,PHIS,U,V,T,Z3"

[e3sm_to_cmip]
active = True
years = "1995:2004:1"
walltime = "00:50:00"
frequency = "monthly"
grid = "180x360_aave"
ts_grid = "180x360_aave"
ts_num_years=1

  [[ atm_monthly_180x360_aave ]]
  # list of variables to be interpolated to pressure level; see e3sm_to_cmip/vrt_remap_plev19.nc
  #cmip_plevdata = "/lcrc/group/e3sm/ac.szhang/acme_scratch/e3sm_project/test_zppy_pmp/zppy/zppy/templates/inclusions/e3sm_to_cmip/vrt_remap_plev19.nc"
  input_files = "eam.h0"
  interp_vars = "U,V,T,Z3"
  ts_subsection = "atm_monthly_180x360_aave"
  #vars="PS,PSL,TREFHT,QREFHT,U10,WIND10,T,U,V,Z3,OMEGA,RELHUM,PMID,OMEGA500,OMEGA700,OMEGA850,T700,Z700,LHFLX,SHFLX,QFLX,SOLIN,FSUT,FLUT,FSUTC,FLUTC,FSDS,FSUS,FLDS,FLUS,FSDSC,FSUSC,FLDSC,CLDTOT,SWCF,LWCF,IWP,PRECLIQ,PRECICE"

[pcmdi_diags]
active = True
#environment_commands_secondary = "source /gpfs/fs1/home/ac.forsyth2/miniforge3/etc/profile.d/conda.sh; conda activate zi-pcmdi-diags-20251007-test1"
#environment_commands = "source /lcrc/soft/climate/e3sm-unified/load_latest_e3sm_unified_chrysalis.sh; conda activate pcmdi_metrics_master" 
walltime = "00:30:00"
# Create rules to construct cmip-like datasets with a string constructed as
# "mip.exp.product.realization", where "."is used as the delimiter (mandatory)
model_name = 'e3sm.amip.EAMXX.test2.1'
model_tableID = 'Amon'
multiprocessing = True
num_workers = 24
#observational data 
obs_ts = '/global/cfs/cdirs/e3sm/e3sm_diags/obs_for_e3sm_diags/time-series'
#obs_ts = '/lcrc/group/e3sm/diagnostics/observations/obs4MIPs_PCMDI_monthly'
pcmdi_debug = False
ts_years = "1995-2004",

  [[ mean_climate ]]
  active = True
  # Default for all variables or specify source for each variable separately.
  obs_sets = "default"
  ref_final_yr = 2018
  ref_start_yr = 1979
  ref_years = "1995-2004",
  target_grid = "1x1"
  target_grid_string = "1px1p"
  # Variables in current pcmdi-cmip list
  # rltcre and rstcre are derived variables as they are not included in cmip_ts
  # define a new variable name to be flexible
  clim_vars = "pr,prw,psl,rlds,rldscs,rltcre,rstcre,rsus,rsuscs,rlus,rlut,rlutcs,rsds,rsdscs,rsdt,rsut,rsutcs,rtmt,sfcWind,tas,tauu,tauv,ts,ta-200,ta-850,ua-200,ua-850,va-200,va-850,zg-500"
  # See zppy/templates/pcmdi_diags/regions_specs.json for options of all regions
  # Below is default setup for global/regional mean climate metrics
  clim_regions = "global,ocean,land,Tropics,NHEX,SHEX"

  [[ variability_modes_cpl ]]
  active = True
  # See zppy/templates/pcmdi_diags/reference_alias.json for options
  obs_sets = "alternate1" 
  ref_final_yr = 2019
  ref_start_yr = 1870
  ref_years = "1995-2004",
  frequency = 'mo'
  landmask = True
  ModUnitsAdjust = '(True,"subtract",273.15)'
  ObsUnitsAdjust = '(True,"subtract",273.15)'
  seasons = "yearly,monthly"
  movc_vars = "ts"
  # Name of modes varibility
  movc_modes = "PDO,NPGO,AMO"

  [[ variability_modes_atm ]]
  active = True
  # See zppy/templates/pcmdi_diags/reference_alias.json for options
  # Default for all variables
  obs_sets = "alternate2" # "default"
  ref_final_yr = 2015
  ref_start_yr = 1853
  ref_years = "1995-2004",
  frequency = 'mo'
  landmask = False
  ModUnitsAdjust = '(True,"divide",100.0)'
  ObsUnitsAdjust = '(True,"divide",100.0)'
  seasons = "DJF,MAM,JJA,SON,yearly,monthly"
  mova_vars = "psl"
  # Varibility modes
  mova_modes = "NAM,NAO,PNA,NPO,SAM,PSA1,PSA2"

  [[ enso ]]
  active = False
  # For enso, the obs_sets is explicitely set for each variable to ensure that
  # they are from the same source, below are setup to used ERA5 reanalysis.
  obs_sets = "default,alternate2,default,default,default,default,default,default,default,alternate2,alternate2,alternate2,alternate2,alternate2"
  ref_final_yr = 2018
  ref_start_yr = 1979
  ref_years = "1995-2004",
  # See zppy/templates/pcmdi_diags/reference_alias.json for options
  enso_groups = "ENSO_perf,ENSO_proc,ENSO_tel"
  enso_vars = "psl,pr,prsn,ts,tas,tauu,tauv,hfls,hfss,rlds,rsds,rlus,rlut,rsdt"

  [[ synthetic_plots ]]
  active = True
  # use subset variables for viewer
  clim_vars = "pr,prw,psl,rlds,rldscs,ua-200,zg-500"
  # use subset regions for viewer
  clim_regions = "global,ocean,land"
  # use subset modes for viewer
  mova_modes = "NAM,NAO,PNA,NPO,SAM"
  # use subset modes for viewer
  movc_modes  = "PDO,NPGO"
  # mean climate
  clim_viewer = True
  # period to be plotted and shown on webpage
  # note that the previous sections allows multiple ts_years periods for processing, while
  # the viewer page focused on period customized by user
  clim_years = "2005-2014"
  # modes variability (atmosphere)
  mova_viewer = True
  # period to be plotted and shown on webpage
  # note that the previous sections allows multiple ts_years periods for processing, while
  # the viewer page focused on period customized by user
  mova_years = "2005-2014"
  # modes variability (coupled)
  movc_viewer = True
  # period to be plotted and shown on webpage
  # note that the previous sections allows multiple ts_years periods for processing, while
  # the viewer page focused on period customized by user
  movc_years = "2005-2014"
  # enso
  enso_viewer = False
  enso_years = ""
  # Location: /lcrc/group/e3sm/diagnostics/pcmdi_data/metrics_data/mean_climate/cmip6/amip
  cmip_clim_set = "cmip6.amip.v20250702"
  # Location: /lcrc/group/e3sm/diagnostics/pcmdi_data/metrics_data/variability_modes/cmip6/amip
  cmip_movs_set = "cmip6.amip.v20210119"
  # Location: /lcrc/group/e3sm/diagnostics/pcmdi_data/metrics_data/enso_metric/cmip6/historical
  # There is no enso metric for CMIP6 AMIP, so use historical instead
  cmip_enso_set = "cmip6.historical.v20210620"

The pcmdi_diags task/subtasks are indeed set to active = True, so it is strange they wouldn't even create a bash file.

Reviewing this discussion

e3sm_to_cmip does not perform vertical interpolation for 3D variables (e.g., U, V, T, Q).

Based on #717, NCO will now automatically exclude 3D variables. So maybe that could be propagating to e3sm_to_cmip? But then I'm not sure why it would be a problem on Perlmutter and not Chrysalis.

The PCMDI step is not triggered.

Yes, it is strange they don't even create bash files, let alone not run.

One possibility is that the E3SM_Unified environment on Perlmutter may not include a complete installation of the components required for e3sm_to_cmip vertical interpolation and/or zppy’s PCMDI modules.

I think this is unlikely since Unified undergoes extensive multi-machine testing. Then again, it's not promising that it works on Chrysalis but not Perlmutter. Maybe a data availability issue?

Looking at the cfg posted above, we do have active = True set correctly.

Looking at:

output = /pscratch/sd/z/zhan391/EAMxx/ne256pg2_ne256pg2.F20TR-SCREAMv1.July-1.spanc800.2xauto.acc150.n0032.test2.1

we find that's the same output dir as mentioned earlier in my comment.

Based on Jill's comment

it looks like you are trying something and creating new atm time-series files.

we may need to wait for any additional work in this directory to be done.

[zhangshixuan1987]

@zhangshixuan1987 I tried to look at post/scripts/ sub folder and it looks like you are trying something and creating new atm time-series files. I think we should be able to leverage the new logs and troubleshooting further.

@chengzhuzhang @forsyth2 : Hi Jill and Ryan, I wanted to describe in detail how I’m running zppy-pcmdi for my EAMxx case, and where I’m currently running into issues.

Here is the workflow I’m using:

1. I manually prepared the monthly time series from EAMxx model output using a bash script with NCO, extracting the monthly variables listed in the .cfg file I shared earlier.

2. I then renamed the EAMxx variables to follow EAM conventions, so the resulting ts files are equivalent (in naming and structure) to those produced directly by EAM.

3. With these prepared time series, I started zppy from the e3sm_to_cmip step. This step normally requires the ts_atm_monthly_180x360_aave jobs to have completed successfully. Since I had already generated these files myself, I manually marked the corresponding status files as OK so that zppy would proceed and trigger e3sm_to_cmip. In this step, I noticed that e3sm_to_cmip only converted 2D variables, while 3D variables requiring vertical interpolation (model levels → pressure levels) were ignored.

4. Even though some variables were not converted successfully, e3sm_to_cmip was triggered as expected. Based on that, I expected to be able to continue and at least trigger the zppy-pcmdi workflow. However, the main issue I’m seeing now is that no job bash scripts are being generated at all for zppy-pcmdi, so the workflow never actually launches.

Note that: I repeat the same procedure in the Chrysalis, and it indeed successfully processed the pcmdi diagnostics expected webpage (https://web.lcrc.anl.gov/public/e3sm/diagnostic_output/ac.szhang/e3sm-pcmdi/ne256pg2_ne256pg2.F20TR-SCREAMv1.July-1.spanc800.2xauto.acc150.n0032.test2.1/pcmdi_diags/model_vs_obs/viewer/).

[forsyth2]

Thank you @andrewdnolan!

[zhangshixuan1987]

@forsyth2 @andrewdnolan Thanks again for the help! I confirmed that "source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_pm-cpu.sh" resolves the issue for me.

What’s a bit puzzling is that on Chrysalis both conda activate e3sm_unified and the load_latest_e3sm_unified_pm-cpu.sh script seem to work equivalently, but on Perlmutter only the source ...load_latest... approach behaves as expected.
Never mind, I will remember to use the correct command from now on.

[forsyth2]

@zhangshixuan1987 Fantastic, glad we got that figured out!

[zhangshixuan1987]

@forsyth2 @chengzhuzhang : Hi Ryan and Jill, I am sorry to bother you again, I am now hit a new error during the "e3sm_to_cmip_stage", see error below:

`2026-02-27 09:31:03.473388 [INFO]: runner.py(init:181) >> * Frequency: mon
2026-02-27 09:31:03.473416 [INFO]: runner.py(init:181) >> * Realm: atm
2026-02-27 09:31:09.503525 [ERROR]: main.py(main:31) >> An unexpected error occurred: Failed to decode variable 'time': unable to decode time units 'days since 1994-10-01 00:00:00' with "calendar 'noleap'". Try opening your dataset with decode_times=False or installing cftime if it is not installed.
Traceback (most recent call last):
File "/global/common/software/e3sm/anaconda_envs/e3smu_1_12_0/pm-cpu/conda/envs/e3sm_unified_1.12.0_compute/lib/python3.13/site-packages/xarray/coding/times.py", line 545, in decode_cf_datetime
dates = _decode_datetime_with_pandas(
flat_num_dates, units, calendar, time_unit
)
File "/global/common/software/e3sm/anaconda_envs/e3smu_1_12_0/pm-cpu/conda/envs/e3sm_unified_1.12.0_compute/lib/python3.13/site-packages/xarray/coding/times.py", line 461, in _decode_datetime_with_pandas
raise OutOfBoundsDatetime(
...<2 lines>...
)
pandas._libs.tslibs.np_datetime.OutOfBoundsDatetime: Cannot decode times from a non-standard calendar, 'noleap', using pandas.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/global/common/software/e3sm/anaconda_envs/e3smu_1_12_0/pm-cpu/conda/envs/e3sm_unified_1.12.0_compute/lib/python3.13/site-packages/xarray/coding/times.py", line 337, in _decode_cf_datetime_dtype
result = decode_cf_datetime(
example_value, units, calendar, use_cftime, time_unit
)
File "/global/common/software/e3sm/anaconda_envs/e3smu_1_12_0/pm-cpu/conda/envs/e3sm_unified_1.12.0_compute/lib/python3.13/site-packages/xarray/coding/times.py", line 549, in decode_cf_datetime
dates = _decode_datetime_with_cftime(
flat_num_dates.astype(float), units, calendar
)
File "/global/common/software/e3sm/anaconda_envs/e3smu_1_12_0/pm-cpu/conda/envs/e3sm_unified_1.12.0_compute/lib/python3.13/site-packages/xarray/coding/times.py", line 365, in _decode_datetime_with_cftime
cftime.num2date(num_dates, units, calendar, only_use_cftime_datetimes=True)
~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "src/cftime/_cftime.pyx", line 627, in cftime._cftime.num2date
File "src/cftime/_cftime.pyx", line 421, in cftime._cftime.cast_to_int
OverflowError: time values outside range of 64 bit signed integers

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/global/common/software/e3sm/anaconda_envs/e3smu_1_12_0/pm-cpu/conda/envs/e3sm_unified_1.12.0_compute/lib/python3.13/site-packages/xarray/conventions.py", line 412, in decode_cf_variables
new_vars[k] = decode_cf_variable(
~~~~~~~~~~~~~~~~~~^
k,
^^
...<8 lines>...
decode_timedelta=_item_or_default(decode_timedelta, k, None),
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
)
^
File "/global/common/software/e3sm/anaconda_envs/e3smu_1_12_0/pm-cpu/conda/envs/e3sm_unified_1.12.0_compute/lib/python3.13/site-packages/xarray/conventions.py", line 239, in decode_cf_variable
var = decode_times.decode(var, name=name)
File "/global/common/software/e3sm/anaconda_envs/e3smu_1_12_0/pm-cpu/conda/envs/e3sm_unified_1.12.0_compute/lib/python3.13/site-packages/xarray/coding/times.py", line 1391, in decode
dtype = _decode_cf_datetime_dtype(
data, units, calendar, self.use_cftime, self.time_unit
)
File "/global/common/software/e3sm/anaconda_envs/e3smu_1_12_0/pm-cpu/conda/envs/e3sm_unified_1.12.0_compute/lib/python3.13/site-packages/xarray/coding/times.py", line 349, in _decode_cf_datetime_dtype
raise ValueError(msg) from err
ValueError: unable to decode time units 'days since 1994-10-01 00:00:00' with "calendar 'noleap'". Try opening your dataset with decode_times=False or installing cftime if it is not installed.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/global/common/software/e3sm/anaconda_envs/e3smu_1_12_0/pm-cpu/conda/envs/e3sm_unified_1.12.0_compute/lib/python3.13/site-packages/e3sm_to_cmip/main.py", line 25, in main
app = E3SMtoCMIP(parsed_args)
File "/global/common/software/e3sm/anaconda_envs/e3smu_1_12_0/pm-cpu/conda/envs/e3sm_unified_1.12.0_compute/lib/python3.13/site-packages/e3sm_to_cmip/runner.py", line 184, in init
self.handlers = self._get_handlers()
~~~~~~~~~~~~~~~~~~^^
File "/global/common/software/e3sm/anaconda_envs/e3smu_1_12_0/pm-cpu/conda/envs/e3sm_unified_1.12.0_compute/lib/python3.13/site-packages/e3sm_to_cmip/runner.py", line 257, in _get_handlers
e3sm_vars = self._get_e3sm_vars(self.input_path)
File "/global/common/software/e3sm/anaconda_envs/e3smu_1_12_0/pm-cpu/conda/envs/e3sm_unified_1.12.0_compute/lib/python3.13/site-packages/e3sm_to_cmip/runner.py", line 325, in get_e3sm_vars
ds = xr.open_dataset(path, decode_timedelta=True)
File "/global/common/software/e3sm/anaconda_envs/e3smu_1_12_0/pm-cpu/conda/envs/e3sm_unified_1.12.0_compute/lib/python3.13/site-packages/xarray/backends/api.py", line 760, in open_dataset
backend_ds = backend.open_dataset(
filename_or_obj,
...<2 lines>...
**kwargs,
)
File "/global/common/software/e3sm/anaconda_envs/e3smu_1_12_0/pm-cpu/conda/envs/e3sm_unified_1.12.0_compute/lib/python3.13/site-packages/xarray/backends/netCDF4.py", line 697, in open_dataset
ds = store_entrypoint.open_dataset(
store,
...<6 lines>...
decode_timedelta=decode_timedelta,
)
File "/global/common/software/e3sm/anaconda_envs/e3smu_1_12_0/pm-cpu/conda/envs/e3sm_unified_1.12.0_compute/lib/python3.13/site-packages/xarray/backends/store.py", line 45, in open_dataset
vars, attrs, coord_names = conventions.decode_cf_variables(
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
vars,
^^^^^
...<7 lines>...
decode_timedelta=decode_timedelta,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
)
^
File "/global/common/software/e3sm/anaconda_envs/e3smu_1_12_0/pm-cpu/conda/envs/e3sm_unified_1.12.0_compute/lib/python3.13/site-packages/xarray/conventions.py", line 425, in decode_cf_variables
raise type(e)(f"Failed to decode variable {k!r}: {e}") from e
ValueError: Failed to decode variable 'time': unable to decode time units 'days since 1994-10-01 00:00:00' with "calendar 'noleap'". Try opening your dataset with decode_times=False or installing cftime if it is not installed.
srun: error: nid004235: task 0: Exited with exit code 1
srun: Terminating StepId=49453578.19`

I am asking again because this is still a library issue, and I would like to confirm with you. The new log files can be found at the following directory:
/pscratch/sd/z/zhan391/EAMxx/ne256pg2_ne256pg2.F20TR-SCREAMv1.July-1.spanc800.2xauto.acc150.n0032.test2.1/post/scripts/

The good thing is that the vertical interpolation is completed successfully, and the failure is pointing to the time modification at the end of the e3sm_to_cmip stage.

[chengzhuzhang]

@zhangshixuan1987 I don't see this error in my recent test with e3sm_to_cmip with EAMxx variables...I tried to look into /pscratch/sd/z/zhan391/EAMxx/ne256pg2_ne256pg2.F20TR-SCREAMv1.July-1.spanc800.2xauto.acc150.n0032.test2.1/post/scripts/ but don't see .o** log output files. Are they purged, if that case, could you restore them?