[Bug]: Failure to generate global time series plot if not starting from year 1

[wlin7]

What happened?

The following error occur when trying to generate global time series plot that does not start from year 1 of a simulation (v3.LR.piControl-spinup). This happens after e3sm_diags and mpas-analysis for the covered period have been completed properly. Same errors also occur for other subperiods, such as global_time_series_0901-1100, global_time_series_1001-1400, regardless of the length of the time series. Does it mean the plotting codes are currently intended only for time series from the beginning of a simulation? Is it easy to make it work for any selected sub-period?

Update time series figures
/lcrc/group/e3sm2/ac.wlin/E3SMv3/20231209.v3.LR.piControl-spinup.chrysalis/post/atm/glb/ts/monthly/10yr/glb.xml
RESTOM
RESSURF
TREFHT
FSNTOA
FLUT
PRECC
PRECL
QFLX
Traceback (most recent call last):
  File "/lcrc/group/e3sm2/ac.wlin/E3SMv3/20231209.v3.LR.piControl-spinup.chrysalis/post/scripts/global_time_series_0101-1400_dir/coupled_global.py", line 651, in <module>
    run_by_region(sys.argv)
  File "/lcrc/group/e3sm2/ac.wlin/E3SMv3/20231209.v3.LR.piControl-spinup.chrysalis/post/scripts/global_time_series_0101-1400_dir/coupled_global.py", line 647, in run_by_region
    run(parameters, rgn)
  File "/lcrc/group/e3sm2/ac.wlin/E3SMv3/20231209.v3.LR.piControl-spinup.chrysalis/post/scripts/global_time_series_0101-1400_dir/coupled_global.py", line 588, in run
    exp["annual"]["ohc"] = ts.globalAnnual("ohc")
  File "/lcrc/group/e3sm2/ac.wlin/E3SMv3/20231209.v3.LR.piControl-spinup.chrysalis/post/scripts/global_time_series_0101-1400_dir/readTS.py", line 65, in globalAnnual
    v = cdutil.YEAR(v)
  File "/lcrc/soft/climate/e3sm-unified/base/envs/e3sm_unified_1.9.2_chrysalis/lib/python3.10/site-packages/cdutil-8.2.1-py3.11.egg/cdutil/times.py", line 1416, in get
  File "/lcrc/soft/climate/e3sm-unified/base/envs/e3sm_unified_1.9.2_chrysalis/lib/python3.10/site-packages/cdutil-8.2.1-py3.11.egg/cdutil/times.py", line 477, in get
  File "/lcrc/soft/climate/e3sm-unified/base/envs/e3sm_unified_1.9.2_chrysalis/lib/python3.10/site-packages/cdutil-8.2.1-py3.11.egg/cdutil/times.py", line 751, in monthBasedSlicer
TypeError: 'NoneType' object is not subscriptable

Examples can be found under /lcrc/group/e3sm2/ac.wlin/E3SMv3/20231209.v3.LR.piControl-spinup.chrysalis/post/scripts

What machine were you running on?

chrysalis

Environment

v2.3.1rc1

What command did you run?

zppy -c /home/ac.wlin/E3SM/utils/post.20231209.v3.LR.piControl-spinup.chrysalis.cfg.1a

Copy your cfg file

[default]
input = /lcrc/group/e3sm2/ac.wlin/E3SMv3/20231209.v3.LR.piControl-spinup.chrysalis
output = /lcrc/group/e3sm2/ac.wlin/E3SMv3/20231209.v3.LR.piControl-spinup.chrysalis
case = 20231209.v3.LR.piControl-spinup.chrysalis
www = /lcrc/group/e3sm/public_html/diagnostic_output/ac.wlin/E3SMv3
partition = compute
environment_commands = "source /lcrc/soft/climate/e3sm-unified/load_latest_e3sm_unified_chrysalis.sh"
campaign = "water_cycle"

[climo]
active = True
years = "0001:1450:50", "0001:1400:100"
walltime = "1:00:00"

  [[ atm_monthly_180x360_aave ]]
  input_subdir = "archive/atm/hist"
  mapping_file = /home/ac.zender/data/maps/map_ne30pg2_to_cmip6_180x360_aave.20200201.nc
  frequency = "monthly"

  [[ atm_monthly_diurnal_8xdaily_180x360_aave ]]
  input_subdir = "archive/atm/hist"
  input_files = "eam.h3"
  mapping_file = /home/ac.zender/data/maps/map_ne30pg2_to_cmip6_180x360_aave.20200201.nc
  vars = "PRECT"
  frequency = "diurnal_8xdaily"

[ts]
active = True
years = "0001:1450:10",
walltime = "00:50:00"

  [[ atm_monthly_180x360_aave ]]
  input_subdir = "archive/atm/hist"
  input_files = "eam.h0"
  frequency = "monthly"
  mapping_file = /home/ac.zender/data/maps/map_ne30pg2_to_cmip6_180x360_aave.20200201.nc
  vars = "FSNTOA,FLUT,FSNT,FLNT,FSNS,FLNS,SHFLX,QFLX,TAUX,TAUY,PRECC,PRECL,PRECSC,PRECSL,TS,TREFHT,CLDTOT,CLDHGH,CLDMED,CLDLOW,U,ICEFRAC,LANDFRAC,OCNFRAC,PS"

  [[ atm_monthly_glb ]]
  input_subdir = "archive/atm/hist"
  input_files = "eam.h0"
  frequency = "monthly"
  mapping_file = "glb"

  [[ land_monthly ]]
  input_subdir = "archive/lnd/hist"
  input_files = "elm.h0"
  mapping_file = ""
  frequency = "monthly"
  vars = "FSH,RH2M"
  extra_vars = "landfrac"

  [[ rof_monthly ]]
  input_subdir = "archive/rof/hist"
  input_files = "mosart.h0"
  mapping_file = ""
  frequency = "monthly"
  vars = "RIVER_DISCHARGE_OVER_LAND_LIQ"
  extra_vars = 'areatotal2'

[e3sm_diags]
active = True
walltime = "4:00:00"
years = "0001:1450:50", "0001:1400:100"
ts_num_years = 10
ref_start_yr = 1979
ref_final_yr = 2016
multiprocessing = True
num_workers = 8

  [[ atm_monthly_180x360_aave ]]
  short_name = '20231209.v3.LR.piControl-spinup'
  grid = '180x360_aave'
  reference_data_path = '/lcrc/soft/climate/e3sm_diags_data/obs_for_e3sm_diags/climatology'
  obs_ts = '/lcrc/soft/climate/e3sm_diags_data/obs_for_e3sm_diags/time-series'
  dc_obs_climo = '/lcrc/group/e3sm/public_html/e3sm_diags_test_data/unit_test_complete_run/obs/climatology'
  climo_diurnal_subsection = "atm_monthly_diurnal_8xdaily_180x360_aave"
  climo_diurnal_frequency = "diurnal_8xdaily"
  sets="lat_lon","zonal_mean_xy","zonal_mean_2d","polar","cosp_histogram","meridional_mean_2d","annual_cycle_zonal_mean","qbo","diurnal_cycle","zonal_mean_2d_stratosphere","aerosol_budget"
[mpas_analysis]
active = True
walltime = "6:00:00"
parallelTaskCount = 6
anomalyRefYear = 101
ts_years = "101-1400", "101-1450",
enso_years = "101-1400","101-1450",
climo_years = "1351-1400","1401-1450",
mesh = "IcoswISC30E3r5"
shortTermArchive = True

[global_time_series]
active = True
years = "101-1400"
ts_num_years = 10
figstr= "20231209.v3.LR.piControl-spinup"
moc_file = "mocTimeSeries_0101-1400.nc"
experiment_name = "20231209.v3.LR.piControl-spinup"
ts_years = "101-1400",
climo_years = "1351-1400",

What jobs are failing?

global_time_series

What stack trace are you encountering?

Traceback (most recent call last):
  File "/lcrc/group/e3sm2/ac.wlin/E3SMv3/20231209.v3.LR.piControl-spinup.chrysalis/post/scripts/global_time_series_0101-1400_dir/coupled_global.py", line 651, in <module>
    run_by_region(sys.argv)
  File "/lcrc/group/e3sm2/ac.wlin/E3SMv3/20231209.v3.LR.piControl-spinup.chrysalis/post/scripts/global_time_series_0101-1400_dir/coupled_global.py", line 647, in run_by_region
    run(parameters, rgn)
  File "/lcrc/group/e3sm2/ac.wlin/E3SMv3/20231209.v3.LR.piControl-spinup.chrysalis/post/scripts/global_time_series_0101-1400_dir/coupled_global.py", line 588, in run
    exp["annual"]["ohc"] = ts.globalAnnual("ohc")
  File "/lcrc/group/e3sm2/ac.wlin/E3SMv3/20231209.v3.LR.piControl-spinup.chrysalis/post/scripts/global_time_series_0101-1400_dir/readTS.py", line 65, in globalAnnual
    v = cdutil.YEAR(v)
  File "/lcrc/soft/climate/e3sm-unified/base/envs/e3sm_unified_1.9.2_chrysalis/lib/python3.10/site-packages/cdutil-8.2.1-py3.11.egg/cdutil/times.py", line 1416, in get
  File "/lcrc/soft/climate/e3sm-unified/base/envs/e3sm_unified_1.9.2_chrysalis/lib/python3.10/site-packages/cdutil-8.2.1-py3.11.egg/cdutil/times.py", line 477, in get
  File "/lcrc/soft/climate/e3sm-unified/base/envs/e3sm_unified_1.9.2_chrysalis/lib/python3.10/site-packages/cdutil-8.2.1-py3.11.egg/cdutil/times.py", line 751, in monthBasedSlicer
TypeError: 'NoneType' object is not subscriptable

[forsyth2]

Does it mean the plotting codes are currently intended only for time series from the beginning of a simulation? Is it easy to make it work for any selected sub-period?

No, that works. The cfg I use for testing starts at 1850 (https://github.com/E3SM-Project/zppy/blob/main/tests/integration/generated/test_complete_run_chrysalis.cfg). But I wasn't sure if that was the simulation start date. So, I ran a cfg starting at 1860 instead and it generated plots ok: https://web.lcrc.anl.gov/public/e3sm/diagnostic_output/ac.forsyth2/zppy_test_debug_www/546-change-start-yr/v2.LR.historical_0201/global_time_series/global_time_series_1860-1870_results/v2_historical_0201_glb.pdf

cfg:

[default]
case = v2.LR.historical_0201
constraint = ""
dry_run = "False"
environment_commands = ""
input = "/lcrc/group/e3sm/ac.forsyth2//E3SMv2/v2.LR.historical_0201"
input_subdir = archive/atm/hist
mapping_file = "map_ne30pg2_to_cmip6_180x360_aave.20200201.nc"
# To run this test, edit `output` and `www` in this file, along with `actual_images_dir` in test_complete_run.py
output = "/lcrc/group/e3sm/ac.forsyth2/zppy_test_debug_output/546-change-start-yr/v2.LR.historical_0201"
partition = "debug"
qos = "regular"
www = "/lcrc/group/e3sm/public_html/diagnostic_output/ac.forsyth2/zppy_test_debug_www/546-change-start-yr"

[ts]
active = True
e3sm_to_cmip_environment_commands = "source /home/ac.forsyth2/miniconda3/etc/profile.d/conda.sh; conda activate e3sm_to_cmip_20240229"
walltime = "00:30:00"
years = "1860:1864:2",

  [[ atm_monthly_glb ]]
  # Note global average won't work for 3D variables.
  frequency = "monthly"
  input_files = "eam.h0"
  input_subdir = "archive/atm/hist"
  mapping_file = "glb"
  years = "1860:1870:5",

  [[ land_monthly_glb ]]
  frequency = "monthly"
  input_files = "eam.h0"
  input_subdir = "archive/atm/hist"
  mapping_file = "glb"


[mpas_analysis]
active = True
anomalyRefYear = 1860
climo_years ="1860-1864", "1865-1870",
enso_years = "1860-1864", "1865-1870",
mesh = "EC30to60E2r2"
parallelTaskCount = 6
partition = "compute"
qos = "regular"
ts_years = "1860-1864", "1860-1870",
walltime = "00:30:00"

[global_time_series]
active = True
climo_years ="1860-1864", "1865-1870",
experiment_name = "v2.LR.historical_0201"
figstr = "v2_historical_0201"
moc_file=mocTimeSeries_1860-1870.nc
ts_num_years = 5
ts_years = "1860-1864", "1860-1870",
walltime = "00:30:00"
years = "1860-1870",

[forsyth2]

I'm having a hard time figuring out what's causing this error. TypeError: 'NoneType' object is not subscriptable is not particularly helpful.

Also worth noting is that the error is coming from cdutil which needs to be removed/replaced anyway as part of the CDAT migration (#346/#519)

[forsyth2]

So something is happening in this block, specifically at exp["annual"]["ohc"] = ts.globalAnnual("ohc") :

zppy/zppy/templates/coupled_global.py

Lines 585 to 590 in 4db3e24

	# Optionally read ohc
	if exp["ocean"] is not None:
	ts = TS(exp["ocean"])
	exp["annual"]["ohc"] = ts.globalAnnual("ohc")
	# annomalies with respect to first year
	exp["annual"]["ohc"][:] = exp["annual"]["ohc"][:] - exp["annual"]["ohc"][0]

But I'm not sure why that's failing.

$ cd /lcrc/group/e3sm2/ac.wlin/E3SMv3/20231209.v3.LR.piControl-spinup.chrysalis/post/scripts
$ grep -v "OK" *status
global_time_series_0051-1400.status:WAITING 465551
global_time_series_0101-1400.status:ERROR (5)
global_time_series_0101-1450.status:WAITING 465658
global_time_series_0901-1100.status:ERROR (5)
global_time_series_0901-1150.status:ERROR (3)
global_time_series_1001-1150.status:ERROR (3)
global_time_series_1001-1200.status:ERROR (5)
global_time_series_1001-1300.status:ERROR (5)
global_time_series_1001-1350.status:ERROR (5)
global_time_series_1001-1400.status:ERROR (5)
global_time_series_1001-1500.status:ERROR (5)
global_time_series_1001-1550.status:ERROR (5)
global_time_series_1001-1700.status:ERROR (5)
global_time_series_1001-1900.status:ERROR (5)
global_time_series_1001-1950.status:ERROR (5)
global_time_series_1001-2000.status:ERROR (5)
global_time_series_1501-1800.status:ERROR (5)
global_time_series_1501-1900.status:ERROR (5)
global_time_series_1501-1950.status:ERROR (3)
global_time_series_1501-2000.status:ERROR (5)
mpas_analysis_ts_0001-1400_climo_1351-1400.status:ERROR (1)
mpas_analysis_ts_0001-1600_climo_1551-1600.status:ERROR (1)
mpas_analysis_ts_0051-1400_climo_1351-1400.status:ERROR (1)
mpas_analysis_ts_0101-1450_climo_1401-1450.status:ERROR (1)
$ grep "OK" global_time_series_*.status
global_time_series_0001-0300.status:OK
global_time_series_0001-0400.status:OK
global_time_series_0001-0450.status:OK
global_time_series_0001-0500.status:OK
global_time_series_0001-0600.status:OK
global_time_series_0001-0650.status:OK
global_time_series_0001-0700.status:OK
global_time_series_0001-0750.status:OK
global_time_series_0001-0800.status:OK
global_time_series_0001-0850.status:OK
global_time_series_0001-0900.status:OK
global_time_series_0001-0950.status:OK
global_time_series_0001-1000.status:OK
global_time_series_0001-1050.status:OK
global_time_series_0001-1100.status:OK
global_time_series_0001-1150.status:OK
global_time_series_0001-1200.status:OK
global_time_series_0001-1250.status:OK
global_time_series_0001-1300.status:OK
global_time_series_0001-1350.status:OK

It does look like only the global_time_series jobs beginning at 0001 ran successfully.

There are two types of errors:
ERROR (5) is what you describe here.

Traceback (most recent call last):
  File "/lcrc/group/e3sm2/ac.wlin/E3SMv3/20231209.v3.LR.piControl-spinup.chrysalis/post/scripts/global_time_ser\
ies_0901-1100_dir/coupled_global.py", line 651, in <module>
    run_by_region(sys.argv)
  File "/lcrc/group/e3sm2/ac.wlin/E3SMv3/20231209.v3.LR.piControl-spinup.chrysalis/post/scripts/global_time_ser\
ies_0901-1100_dir/coupled_global.py", line 647, in run_by_region
    run(parameters, rgn)
  File "/lcrc/group/e3sm2/ac.wlin/E3SMv3/20231209.v3.LR.piControl-spinup.chrysalis/post/scripts/global_time_ser\
ies_0901-1100_dir/coupled_global.py", line 588, in run
    exp["annual"]["ohc"] = ts.globalAnnual("ohc")
  File "/lcrc/group/e3sm2/ac.wlin/E3SMv3/20231209.v3.LR.piControl-spinup.chrysalis/post/scripts/global_time_ser\
ies_0901-1100_dir/readTS.py", line 65, in globalAnnual
    v = cdutil.YEAR(v)
  File "/lcrc/soft/climate/e3sm-unified/base/envs/e3sm_unified_1.9.2_chrysalis/lib/python3.10/site-packages/cdu\
til-8.2.1-py3.11.egg/cdutil/times.py", line 1416, in get
  File "/lcrc/soft/climate/e3sm-unified/base/envs/e3sm_unified_1.9.2_chrysalis/lib/python3.10/site-packages/cdu\
til-8.2.1-py3.11.egg/cdutil/times.py", line 477, in get
  File "/lcrc/soft/climate/e3sm-unified/base/envs/e3sm_unified_1.9.2_chrysalis/lib/python3.10/site-packages/cdu\
til-8.2.1-py3.11.egg/cdutil/times.py", line 751, in monthBasedSlicer
TypeError: 'NoneType' object is not subscriptable

Corresponds to

zppy/zppy/templates/global_time_series.bash

Lines 88 to 95 in 4db3e24

	echo 'Update time series figures'
	cd ${global_ts_dir}
	python coupled_global.py ${case_dir} ${experiment_name} ${figstr} ${start_yr} ${end_yr} {{ color }} ${ts_num_years} ${atmosphere_only} "{{ plot_names }}" {{ regions }}
	if [ $? != 0 ]; then
	cd {{ scriptDir }}
	echo 'ERROR (5)' > {{ prefix }}.status
	exit 5
	fi

$ grep "ERROR (5)" global_time_series_*.status
global_time_series_0101-1400.status:ERROR (5)
global_time_series_0901-1100.status:ERROR (5)
global_time_series_1001-1200.status:ERROR (5)
global_time_series_1001-1300.status:ERROR (5)
global_time_series_1001-1350.status:ERROR (5)
global_time_series_1001-1400.status:ERROR (5)
global_time_series_1001-1500.status:ERROR (5)
global_time_series_1001-1550.status:ERROR (5)
global_time_series_1001-1700.status:ERROR (5)
global_time_series_1001-1900.status:ERROR (5)
global_time_series_1001-1950.status:ERROR (5)
global_time_series_1001-2000.status:ERROR (5)
global_time_series_1501-1800.status:ERROR (5)
global_time_series_1501-1900.status:ERROR (5)
global_time_series_1501-2000.status:ERROR (5)

ERROR (3)

Traceback (most recent call last):
  File "/lcrc/soft/climate/e3sm-unified/base/envs/e3sm_unified_1.9.2_chrysalis/bin/cdscan", line 1843, in <modu\
le>
    main(sys.argv)
  File "/lcrc/soft/climate/e3sm-unified/base/envs/e3sm_unified_1.9.2_chrysalis/bin/cdscan", line 1309, in main
    vartime[0],
  File "/lcrc/soft/climate/e3sm-unified/base/envs/e3sm_unified_1.9.2_chrysalis/lib/python3.10/site-packages/cdm\
s2/axis.py", line 2212, in __getitem__
    raise IndexError('index out of bounds')
IndexError: index out of bounds

Corresponds to

zppy/zppy/templates/global_time_series.bash

Lines 67 to 75 in 4db3e24

	echo 'Create xml for for ocn'
	export CDMS_NO_MPI=true
	cd ${case_dir}/post/ocn/glb/ts/monthly/${ts_num_years}yr
	cdscan -x glb.xml mpaso.glb*.nc
	if [ $? != 0 ]; then
	cd {{ scriptDir }}
	echo 'ERROR (3)' > {{ prefix }}.status
	exit 3
	fi

$ grep "ERROR (3)" global_time_series_*.status
global_time_series_0901-1150.status:ERROR (3)
global_time_series_1001-1150.status:ERROR (3)
global_time_series_1501-1950.status:ERROR (3)