Potential inconsistency of diagnostics generation after update DATA and version

[chengzhuzhang]

In the tests, ILAMB version is updated to v2.7 and DATA directory is updated with ilamb-fetch. This update does resolved #83, but I also noticed some inconsistencies.
Results:
Before updates
After updates

It looks like only a subset of Biomass results (2 out of 5) are showing up. And runoff, permafrost diagnostics are not shown.
And I did see name changes in directory structure in DATA, for instance: runoff directory is renamed as mrro, and sub folders of biomass are renamed, ect. Not sure if that is the cause. Pinging @nocollier , any insights are appreciated!

[nocollier]

So I am looking at your "After updates" and I am noticing some strange things in the logfile:

https://web.lcrc.anl.gov/public/e3sm/diagnostic_output/ac.zhang40/tests/E3SMv3_dev/ilamb_bgc_eu1.9.2rc2/20231012.v3alpha04_trigrid_bgc.piControl.chrysalis/ilamb/land_monthly_0051-0100/ILAMB01.log

If you poke through that, you will see that the only biomass variables that run are NBCD2000 and Thurner. If one of the others failed, it would have an entry in the logfile, like the NOAA emulated CO2 one, with details of the failure. In fact, most of your grey boxes don't seem to have run at all.

This could mean that they are failing in a way that I do not log. Did you happen to capture the screen output?

If none of these comments are help, let's jump on zoom tomorrow and figure this out quickly.

[chengzhuzhang]

Thank you for troubleshooting! Attached is the logfile i captured during the run.
ilamb_0051-0100.o454516.log

Most of the other grey boxes are caused by missing variables, I believe. The biomass case is particularly interesting. I notice that the two successful runs (NBCD2000 and Thurner) have the directory names unchanged between DATA updates. Not sure if it helps, but below are the before and after directory names of DATA.

(base) [ac.zhang40@chrlogin2 DATA]$ cd ../DATA_2.6/
(base) [ac.zhang40@chrlogin2 DATA_2.6]$ cd biomass/
(base) [ac.zhang40@chrlogin2 biomass]$ ll
total 6
drwxrwxr-x 2 ac.xasay-davis cels 4096 Apr 23  2022 GEOCARBON
drwxrwxr-x 2 ac.xasay-davis cels 4096 Apr 23  2022 GLOBAL.CARBON
drwxrwxr-x 2 ac.xasay-davis cels 4096 Apr 23  2022 NBCD2000
drwxrwxr-x 2 ac.xasay-davis cels 4096 Apr 23  2022 Thurner
drwxrwxr-x 2 ac.xasay-davis cels 4096 Apr 23  2022 Tropical
drwxrwxr-x 2 ac.xasay-davis cels 4096 Apr 23  2022 US.FOREST
(base) [ac.zhang40@chrlogin2 biomass]$ cd ../../DATA/biomass
(base) [ac.zhang40@chrlogin2 biomass]$ ll
total 7
drwxrwxr-x 2 ac.zhang40 cels 4096 Jan  9 18:44 ESACCI
drwxrwxr-x 2 ac.zhang40 cels 4096 Jan  9 18:44 GEOCARBON
drwxrwxr-x 2 ac.zhang40 cels 4096 Jan  9 18:44 NBCD2000
drwxrwxr-x 2 ac.zhang40 cels 4096 Jan  9 18:44 Saatchi2011
drwxrwxr-x 2 ac.zhang40 cels 4096 Jan  9 18:44 Thurner
drwxrwxr-x 2 ac.zhang40 cels 4096 Jan  9 18:44 USForest
drwxrwxr-x 2 ac.zhang40 cels 4096 Jan  9 18:44 XuSaatchi2021

Happy to schedule a zoom meeting if none of above is useful..