Checksum attributes in tables

[mauzey1]

In the current version of the mip-cmor-tables, there is a checksum attribute in the headers of tables. They all currently have empty strings for values.

mip-cmor-tables/Tables/MIP_ACmon.json

Line 5 in 49d4cfb

"checksum": "",

I've also seen checksum attributes in the auxillary files. These do have a value for the attribute.

mip-cmor-tables/Auxillary_files/MIP_formula_terms.json

Line 206 in 49d4cfb

"checksum": "md5: 8f652d9d1e8741f5f81c1ee2cb0d3e2f",

I don't understand the purpose of this attribute. It shouldn't be the checksum of the file itself since creating a checksum of a file and then adding it inside the file would change its checksum. Where did the checksum for MIP_formula_terms.json come from?

If the point was to store the checksum of a table used by CMOR to create a dataset file, then CMOR must calculate the checksum of the table file. It can't be a value stored in the table.

[durack1]

@mauzey1 good catch. I believe this is a checksum of the file (non-header) contents, which is calculated in memory rather than from the output file, @wolfiex can confirm. The intention would then be to use this to evaluate if any contents have changed, which means we can catch cases where folks have hand edited files causing CMOR to fail, but didn't own up to this unsupported edit

[mauzey1]

Shouldn't the header info also be monitored for changes? It contains approx_interval, generic_levels, int_missing_value, and missing_value.

[durack1]

It very well may be. I am not certain of how or what the checksum is calculated from

[taylor13]

I can't remember why it is in the header of the input file. As I recall, that global attribute was supposed to be generated by the user (or by CMOR) by performing checksum on the entire CMOR input file and then writing it to the output files. This would tell us if an input table had been modified. I don't think we've every looked at it though, so maybe we should drop that.

[matthew-mizielinski]

I probably introduced this in a very early attempt at building the MIP tables (pre-@wolfiex). My intention was to avoid (or at least hightlight) manual changes to the tables once they are put together -- if the checksums are appropriately set then it is easy to confirm whether a copy of them has been changed relative to the repository. I've used this strategy a few times within MO systems to prevent users from unilaterally changing certain control files partway through processing.

Validating checksums in python is pretty straightforward if you remove the checksum entry first, e.g. see here for example code -- I have this in parts of our processing systems to ensure that users don't break certain files by editing them by hand :-).

There is at least one case in the CMIP6 archive where someone must have hacked the tables to get their variable onto ESGF (go looking for areacelli for an example). CMOR logs a checksum in the metadata of the file (table_info) and it would be useful if this could be checked against the repository value to quickly check conformance.

I'm not completely averse to removing this, but I think it would be better to populate it -- shouldn't be too tricky for @wolfiex or myself to implement.

Note that I don't know how easy it will be to get the current CMOR code to validate these checksums in the same way as python would -- might be something that we ignore with the current C code and pick up in CMOR v4

[taylor13]

Seems like a sensible way to proceed.