CMIP Repo naming convention.

[wolfiex]

As part of the naming conventions for items within the CVs it is suggested that names do not contain underscores, and are rather hyphenated as not to break any file path / DRS parsing.

I suggest extending this to repository labels, for instance replacing CMIP6Plus_CVs with CMIP6Plus-CVs to match the format of WCRP-CMIP and the newly created WCRP-universe.

[durack1]

Seems ok to me. This will not be rolled backward in time to the https://github.com/WCRP-CMIP/CMIP6_CVs/ (I'm not interested in doing the work to change it), but as we're trying to push hard to get all these templates finalized, standardize away!

ping @taylor13 @ltroussellier @matthew-mizielinski any comments?

[taylor13]

I'm fine with replacing underscores.

[ltroussellier]

i have done a basic review :

no problem with removing underscore "" :_

• DRS separator :
  filename "_"
  directory "/"
  dataset "."

have to fix this :

• mix underscore "_" and hyphenated "-" in license
• a dot "." in 0.5km resolution
• a model component contains a "," !

what to do with :

• "-" hyphenated separator for future branded label suffix (i.e temporal_label-vertical_label-horizontal_label-area_label)
  here it is tricky => the DD/collection underscore can be replaced with "-" BUT "-" must NOT appears in terms
• "_" underscore separator for future branded variable (i.e variable_suffix)

caution :

• time-range separator for filename is already a "-" (i.e YYYYMM-YYYYMM)
  it can works well if there is no "-" in parts separated by the "-"

conclusion : seems OK to remove the underscores as long as time-range doesnt contain a "-" and some others cautions

[wolfiex]

Seems ok to me. This will not be rolled backward in time to the https://github.com/WCRP-CMIP/CMIP6_CVs/ (I'm not interested in doing the work to change it), but as we're trying to push hard to get all these templates finalized, standardize away!

ping @taylor13 @ltroussellier @matthew-mizielinski any comments?

As a note: I believe GitHub preserves previous links and definitely does not cause breaking changes to already downloaded repositories.

Current repositories that need updating: CMIP6Plus-CVs and CMIP7-CVs

[taylor13]

Apologies if the terms "name" and "value" aren't the correct terms, but I hope my definitions below are clear.

We need to distinguish between the name of a variable from its value, which must be drawn from a CV. In CMIP, we have names like variable_id, frequency, and source_id. For each of these we have a limited number of values they might be assigned. For example, variable_id might be assigned the value "HadGEM3-GC31-LL"

Also, I think we need to distinguish between CVs that contain DRS items like source_id (i.e., the ones used in uniquely identifying files and atomic datasets), and the ones used for other purposes (e.g., nominal_resolution and standard_name) where they will not appear in strings that need parsing.

In past phases for the CMIP data archive, we adopted the following rules:

• We placed few restrictions on the variable names, which might routinely include an underscore "_".
• We restricted values used to construct the DRS unique identifiers (e.g., in the DRS) that would need parsing to only certain characters. Forbidden characters included underscore "_", period ".", and "/", but hyphens were permitted. I think our rule has been to limit these strings to only alphanumeric characters plus "-", and nothing else.
• For CV values that are not part of the DRS, we allowed almost any character, including ".", "_", "/", "-", ....

Will continued application of the above rules cause problems in the future? (Laurent, your comment above seems to find a problem with at least one of these rules.)

Regarding the filenames of CVs and the directories hosting them (which are unrelated to the model output archive), I think Dan is free to replace the "_" with "-" without consequence.

[ltroussellier]

Thanks for the precise definition of semantic terms.

Regarding the form:

For variable names, I prefer facets, DataDescriptor (for the universe), and Collection (for the project). To me, all of these are somewhat synonymous.
For values (such as "HadGEM3-GC31-LL"), I use the term terms, but "values" is fine. However, "values" might be misleading given the key-value structure in JSON-LD.
Regarding the content:

We placed few restrictions on the variable names, which might routinely include an underscore "_".

The purpose of this issue is precisely to change this rule from "_" to "-", isn't it? Does replacing " _ " with "-" apply only to repository names, or does it also affect names/DataDescriptors/Collections and values/terms?

We restricted values used to construct the DRS unique identifiers (e.g., in the DRS) that would need parsing to only certain characters. Forbidden characters included underscore "_", period ".", and "/", but hyphens were permitted. I think our rule has been to limit these strings to only alphanumeric characters plus "-", and nothing else.

This point is important for DRS parsing!

For CV values that are not part of the DRS, we allowed almost any character, including ".", "_", "/", "-", ….

Although this is technically allowed, I find it confusing to permit some values/terms to have these characters while restricting others. I would prefer a single, consistent rule regardless of whether the term appears in the DRS. I also suspect that the JSON-LD algorithm would handle this more effectively with a unified rule.

In fact, based on an initial review, there currently don’t seem to be values/terms containing ".", "_", "/", etc.,only "-" except for these three cases:

• Mixed use of "_" and "-" in license names
• A dot "." in "0.5km" resolution
• A model component containing a ","

There may be other cases, but resolving these inconsistencies seems easier than maintaining separate rules.

[taylor13]

O.K., let's try to be consistent and use "facets" and "terms".

@wolfiex: Dan, am I correct that you were proposing only to change the names of repositories or files hosting vocabularies, not modify their contents (i.e., not modify the names of facets or terms in the vocabulary)?

@ltroussellier : Laurent, I really would like to retain the freedom to use punctuation and allow phrases (with blank spaces) as vocabulary terms. I think we should allow licenses, for example, to be expressed in the usual way (e.g., "CC BY-SA 4.0"). We could include in the json file for each vocabulary a flag, indicating level of restriction on characters. For example: "character_restriction" : "TRUE" (or False). When true, we would forbid in vocabulary terms all characters except alphanumeric characters and the hyphen.

Would that be acceptable?

[wolfiex]

Correct. Eventually the contents should also reflect the names, but that is very low priority and something to consider at a later time.

[ltroussellier]

Correct. Eventually the contents should also reflect the names, but that is very low priority and something to consider at a later time.

OK, so the idea is just to define rules for the naming/syntax of github repository name and file names inside directories.

@ltroussellier : Laurent, I really would like to retain the freedom to use punctuation and allow phrases (with blank spaces) as vocabulary terms. I think we should allow licenses, for example, to be expressed in the usual way (e.g., "CC BY-SA 4.0"). We could include in the json file for each vocabulary a flag, indicating level of restriction on characters. For example: "character_restriction" : "TRUE" (or False). When true, we would forbid in vocabulary terms all characters except alphanumeric characters and the hyphen.

It seems like there is still some confusion between "value" and "term.". I'll try my best to be as clear as possible.

Let’s take an example that appears in the DRS from the current repository (cmip6plus), which is a project pointing to the universe:

here :

• "experiment_id" is a facet/collection
• "1pctco2.json" is a term

considering this issue => "experiment_id" will change to "experiment-id"

The term (1pctco2.json) points to the universe term (1pctco2.json in WCRP-universe):

here :
facet/datadescriptor is "experiment"

• It can be different between the project and the universe to facilitate harmonization across all CMIP iterations.

inside the term, there are keys/values :
for instance the value of the key description for this term is : "DECK: 1pctCO2"

• there is absolutely no problem at all to use punctuation for values inside the term

lets take a term for the license facet/datadescriptor that doesnt appears in the DRS : https://github.com/WCRP-CMIP/WCRP-universe/blob/esgvoc/license/cc_by-nc-sa_4.0.json

the filename of the this term contains mix "_" and "-" => this need to be updated according to this issue
inside the file, the key "kind" is exactly what you want : "CC BY-NC-SA 4.0"

Notes and Remarks :

• I use data of a term for the keys and values
• The JSON data structure for each term (i.e., the keys used inside JSON files) needs to be defined soon. The examples shown here were used during the development of the esgvoc library.
• The keys must be valid class attributes in Python, which imposes punctuation constraints.
• The values for each keys for a particular term can be anything
• The only key currently subject to constraints is "drs_name", which must follow the rules you described.

[taylor13]

This seems very well thought out, but I will try to study this further tomorrow and I'm sure I'll have a few questions.

My main concern is that the "synonyms" or "aliases" for basically the same thing seem to be multiplying, making it harder to keep them straight. It seems, for example, that even within a single phase of CMIP, "experimentDD" might be referred to as "experiment_id" in the file global attributes and "experiment-id" in the CVs, and possibly also as "experiment" (as in "type"). Can we collapse these 4 representations of the same thing to perhaps 2 options? I think I would be happy with "experimentDD in the CVs and experiment_id as a global attribute, but I'm open to other options.

As an example of where I'm coming from, my original idea was to define the generic "sourceDD" as something that might be represented differently in different CMIP phases. In fact, in CMIP5 the sourceDD was referred to as "model_id", while in CMIP6 it was referred to as "source_id". (There was never a reference to "source-id".)

[ltroussellier]

My main concern is that the "synonyms" or "aliases" for basically the same thing seem to be multiplying, making it harder to keep them straight. It seems, for example, that even within a single phase of CMIP, "experimentDD" might be referred to as "experiment_id" in the file global attributes and "experiment-id" in the CVs, and possibly also as "experiment" (as in "type"). Can we collapse these 4 representations of the same thing to perhaps 2 options? I think I would be happy with "experimentDD in the CVs and experiment_id as a global attribute, but I'm open to other options.

its fine i think:

"global attributes" are defined in project (specific cmip phase). it can use everything from the project facet/collection
as you mention in the example for source :

cmip5 : model_id
cmip6 : source_id

and now with the new framework we got

cmip5: model_id
cmip6: source_id
cmip7 : source-id
universe : sourceDD (now is it "source" but im fine changing it to "sourceDD")

In a specific project, a source term maps to sourceDD through JSON-LD. Each phase’s global attribute will refer to its respective collection name (model_id for CMIP5, source_id for CMIP6, source-id for CMIP7), ensuring no overlap. This way, when software validates or displays a model_id for CMIP5, it will retrieve the corresponding term in CMIP5, which then maps to sourceDD, ensuring consistency with the CV syntax.