Enable switching off CDDS extract

[alistairsellar]

Enable switching off CDDS extract when running CMEW and instead provide a path to data on disk. This will be useful when running CMEW for testing purposes (since we don't have to rely on the availability of MASS to ensure CMEW runs).

To switch off CDDS extract, skip_extract = True (from the CDDS documentation) will need to be added to the [conversion] section in the request configuration file created by CMEW/app/configure_standardise/bin/create_request_file.py.

The user must be able to set this value via the rose-suite.conf file.

Acceptance Criteria 1

• Given a working version of CMEW
• When EXTRACT=True (the default) is specified in the rose-suite.conf file
• Then standardise_model_data runs CDDS extract and CDDS convert

Acceptance Criteria 2

• Given a directory containing model output files
• When EXTRACT=False and the full path to the directory are specified in the rose-suite.conf file
• Then standardise_model_data does not run CDDS extract and CDDS convert instead uses the data provided by the full path to the directory

Acceptance Criteria 3

• Given a working version of CMEW
• When EXTRACT=False is specified in the rose-suite.conf file
• Then CMEW fails, since the full path to a directory containing model output files has not been provided

Details

• Extract should be true by default.
• If the flag is false, a path to previously extracted data should be provided.
• Use the flow chart in the Rose requirements in the Developer Guide to determine where the flag should be defined.

[mo-nikosbaltas]

Here I tried to map the Acceptance Criteria to the existing conf and scripts.
Acceptance Criteria mapping
AC1
Given EXTRACT=True (default)
• rose-suite.conf default EXTRACT=true
• create_request_file.py does NOT write skip_extract
• standardise_model_data runs cdds_convert normally → extract + convert.
(to be met by implementation described below)
AC2
Given EXTRACT=False + <path>
• request.cfg includes skip_extract=True
• root_data_dir overridden to <user path>
• standardise_model_data still runs cdds_convert, but CDDS skips extract and converts from given data.
(to be met by implementation described below)
AC3
Given EXTRACT=False without path
• create_request_file.py raises ValueError
• configure_standardise task fails → workflow fails early.
(to be met by implementation described below)

Proposed Design / Implementation Plan
1. Add Rose template variables
- EXTRACT=true (default)
-EXTRACT_DATA_PATH="" (default empty)
in CMEW/rose-suite.conf.

2. Expose vars to configure_standardise tasks
In CMEW/flow.cylc under [[configure_standardise<assessment_area>]], add:
- EXTRACT = {{ EXTRACT }}
-EXTRACT_DATA_PATH = {{ EXTRACT_DATA_PATH }}

3. Modify request.cfg generation
In app/configure_standardise/bin/create_request_file.py:
Parse env:

• extract_flag = EXTRACT.lower() == "true"` (default true if unset)
• extract_data_path = EXTRACT_DATA_PATH.strip()

If extract_flag is False and extract_data_path == "", raise ValueError to fail CMEW (AC3).
Build [conversion]:

• add skip_extract="True" only when EXTRACT=False.

When EXTRACT=False, override root_data_dir in [common] to EXTRACT_DATA_PATH so CDDS converts from pre-extracted data on disk (AC2).

4. Update unit tests
Extend test_create_request_file.py to cover:

• EXTRACT default true → no skip_extract key (AC1)
• EXTRACT false + path → skip_extract present and root_data_dir set to path (AC2)
• EXTRACT false + no path → ValueError raised (AC3)

Notes
• User-provided EXTRACT_DATA_PATH must match CDDS-expected extracted layout. Document required layout in Developer Guide.
• No change required to standardise_model_data app commands; CDDS handles skip_extract internally.

[mo-nikosbaltas]

Changes made fulfilled all three Acceptance Criteria. CMEW run locally successfully.
In the process writing 3 unit tests.

[mo-nikosbaltas]

All three unit tests were written and run successfully using -O metoffice -O unittest.

[mo-nikosbaltas]

To run the second case and fulfill AC2 (EXTRACT=False + <path>), certain changes were made:

1. Created a new template variable in rose-suite. conf, named CLEANUP.
2. Changed flow.cylc to make housekeeping optional.
3. Run once with:
   - EXTRACT=true
   - CLEANUP=false
4. After this run, CDDS data will remain in cylc-run/<workflow>/...../share/data/cdds/cdds_data/
5. Copy the CDDS data from the above location to a directory that we can use for AC2:

• mkdir -p /my/path/to/CDDS
• cp -r cylc-run/<workflow>/...../share/data/cdds/cdds_data/* /my/path/to/CDDS/
• EXTRACT=false
• EXTRACT_DATA_PATH=/my/path/to/CDDS
• CLEANUP=true

[mo-nikosbaltas]

For validation of AC2, the EXTRACT_DATA_PATH was set to a path where the CDDS files can be transferred. Initially the EXTRACT_DATA_PATH was set to the filesystem where my home directory is; as a result the run was failing.
In practice, there is an extra constraint coming from CDDS:
The directory pointed to by EXTRACT_DATA_PATH must be on the same filesystem as the Cylc workflow's share/work/ directory (i.e. where standardised data are restructured to).

If they are not on the same filesystem, restructure_for_cmip6 will not move any files, and standardise_model_data will fail, as I have already experienced.

CDDS's restructure_for_cmip6 cannot currently restructure from a directory on a different filesystem than share/work; it aborts with:

output is not on same filesystem:

Therefore, for AC2 to hold in all cases, either:

we must 'explain' in the CMEW documentation that EXTRACT_DATA_PATH must be on the same filesystem as CYLC_WORKFLOW_SHARE_DIR, or

CDDS needs an option (eg. -C) to allow copy-across-filesystems, and CMEW's restructure_dirs.sh would need to pass that option.

[alistairsellar]

For validation of AC2, the EXTRACT_DATA_PATH was set to a path where the CDDS files can be transferred. Initially the EXTRACT_DATA_PATH was set to the filesystem where my home directory is; as a result the run was failing. In practice, there is an extra constraint coming from CDDS: The directory pointed to by EXTRACT_DATA_PATH must be on the same filesystem as the Cylc workflow's share/work/ directory (i.e. where standardised data are restructured to).

If they are not on the same filesystem, restructure_for_cmip6 will not move any files, and standardise_model_data will fail, as I have already experienced.

CDDS's restructure_for_cmip6 cannot currently restructure from a directory on a different filesystem than share/work; it aborts with:

output is not on same filesystem:

Thanks @mo-nikosbaltas, I'd like to better understand the filesystem constraint before we proceed further with this. Let's discuss with @mo-jareddrayton.

In the meantime, is there anything in the CDDS documentation that explains the constraint?

Conceptually, I think there are three directories we are interested in under AC2:

1. Where CDDS looks for pre-extracted raw model data
2. Where the standardised, but not restructured, data is written
3. Where the restructured data is written

Does CDDS allow separate control of any of these three locations?

For now, I've changed the PR #300 to "draft" and moved it from review-in-progress to development-in-progress in our project board.