Separate internal netCDF-4 data model from HDF5 code

[edhartnett]

As discussed in PR #849, it will soon be possible to separate the internal netCDF-4 data model code (i.e. lists of atts, vars, and dims, and the functions that manipulate them) from the HDF5 code. I add this issue so that there is a place to discuss the worthiness and possible implementation of this goal.

This would enable user-defined formats (#177) and the HDF4 layer to work without HDF5. It would also be a useful simplification of the code which would make the libsrc4 code smaller and easier to maintain.

After this ticket has been accomplished several other opportunities present themselves:

• DAP has it's own internal data model (I believe), and at some point it could be simplified by using the common data model.

• The classic code in libsrc uses a separate but related implementation of the data model. This too could eventually be removed, so that all parts of the library were using the same internal data structs.

[DennisHeimbigner]

The approach I was planning to take was to take each netcdf-4 structure
(e.g. NC_VAR_INFO_T etc) and separate the fields into two parts:

1. implementation independent
2. implementation dependent
   The Use GNUInstallDirs to install into /usr/lib64 as needed #2 fields I would then separate off into their own, implementation dependent
   structure. I would then leave an opaque (void*) ptr in the implementation
   independent part. This mirrors how the NC structure has an opaque pointer
   called ncdispatchdata that points to the dispatch specific info.
   Alternatively, we could keep a hash table that maps each implementation specific
   object to the corresponding implementation independent meta data object.

[edhartnett]

The NC_VAR_INFO_T can already represent all data in the classic and enhanced models. So I would suggest that be the universal understanding (internally) of what a var is, and functions like nc_inq_var*() are already written to work off it. So I don't think anything should be taken out.

Then each user-defined format must jam its data into that model. To the extent they can do so, they take advantage of all the existing inq code that already works well.

Some parts of the model will not be used (for example, NC_CLASSIC_MODEL files cannot have user-defined types), but the code is already written to handle that. If no user-defined types are present, the list can be NULL and that's fine. The current internal data model can handle all variants of the classic and enhanced netCDF models.

I would like user-defined formats to have access to this existing model. I don't think there are going to be many user-defined formats that have a more complex model than HDF5, so I'm pretty sure they will all be able to easily fit within the existing netCDF-4 model.

In many cases, read-only access is fine, so they can then implement a dispatch layer with just a few functions, and take advantage of all the functions that already work off the existing data model.

I don't object to giving them an additional void pointer that they can use within their dispatch layer code for implementation dependent data, but I would suggest that the current netCDF enhanced data model be fully supported in the internal data structures, so that functions written to it can continue to work for many different formats, just like the inq functions currently work for both HDF4 and HDF5.

Does that fit what you thinking?

[edhartnett]

OK, I think I have a better idea of what you mean. Just so we have an example:

/* This is a struct to handle the var metadata. */
typedef struct NC_VAR_INFO
{
   char *name;
   char *hdf5_name; /* used if different from name */
   int ndims;
   int *dimids;
   NC_DIM_INFO_T **dim;
   int varid;
   int natts;
   uint32_t hash;
   nc_bool_t is_new_var;        /* True if variable is newly created */
   nc_bool_t was_coord_var;     /* True if variable was a coordinate var, but either the dim or var has been renamed */
   nc_bool_t became_coord_var;  /* True if variable _became_ a coordinate var, because either the dim or var has been renamed */
   nc_bool_t fill_val_changed;  /* True if variable's fill value changes after it has been created */
   nc_bool_t attr_dirty;        /* True if variable's attributes are dirty and should be rewritten */
   nc_bool_t created;           /* Variable has already been created (_not_ that it was just created) */
   nc_bool_t written_to;        /* True if variable has data written to it */
   struct NC_TYPE_INFO *type_info;
   hid_t hdf_datasetid;
   NC_ATT_INFO_T *att;
   nc_bool_t no_fill;           /* True if no fill value is defined for var */
   void *fill_value;
   size_t *chunksizes;
   nc_bool_t contiguous;        /* True if variable is stored contiguously in HDF5 file */
   int parallel_access;         /* Type of parallel access for I/O on variable (collective or independent) */
   nc_bool_t dimscale;          /* True if var is a dimscale */
   nc_bool_t *dimscale_attached;        /* Array of flags that are true if dimscale is attached for that dim index */
   HDF5_OBJID_T *dimscale_hdf5_objids;
   nc_bool_t deflate;           /* True if var has deflate filter applied */
   int deflate_level;
   nc_bool_t shuffle;           /* True if var has shuffle filter applied */
   nc_bool_t fletcher32;        /* True if var has fletcher32 filter applied */
   size_t chunk_cache_size, chunk_cache_nelems;
   float chunk_cache_preemption;
#ifdef USE_HDF4
   /* Stuff below is for hdf4 files. */
   int sdsid;
   int hdf4_data_type;
#endif /* USE_HDF4 */
   /* Stuff for arbitrary filters */
   unsigned int filterid;
   size_t nparams;
   unsigned int* params;
   /* Stuff for diskless data files. */
   void *diskless_data;
} NC_VAR_INFO_T;

You mean there would be a void pointer (format_infop or something), In the case of HDF5 it would contian such things as the datasetid, dimscale_hdf5_objids, and any other values that have to with HDF5 specifics.

Things like name, ndims, dim, etc., would remain as part of the netCDF internal data that is independent of format.

[DennisHeimbigner]

Remember that the NC_VAR_INFO structure has some hdf5
specific fields like hdf5_name, native_hdf5_typeid.

[edhartnett]

Yes, I get it, and I agree. A void pointer to the format-dependent data elements would work well. It would clean up the data structures very nicely.

[DennisHeimbigner]

A couple of other thoughts.

1. Are chunking and filters considered part ot the netcdf4 model or are they hdf5 specific?
2. Other formats will probably have special features (like hdf5 has chunking) that
   it might be desirable to make visible in some generic form thru the netcdf API.
   PIO probably has such, for example.

[cofinoa]

@DennisHeimbigner and @edhartnett here's my 2 cents worth.

IMO , chunking or filters are part of the storage layer not of the netcdf4 data model layer. The netcdf3, DAP2 or DAP4 are another examples of different dispatchers (storage layers).

Yes, a special API for storage layer (implementations) features would be desirable.

[edhartnett]

We can certainly separate the chunking/filter code so that it is in the HDF5 (libsrc4) directory, instead of the new internals directory (libint). We can put chunking/filter values into the proposed HDF5 var data info (let's call it NC_HDF5_VAR_INFO_T).

Recall that chunking and filters are also part of the API. So they will always be part of the libsrc4 directory for that reason. However, we can store the results in NC_HDF5_VAR_INFO_T instead.

But chunking is also used by HDF4, an addition since my departure from the project, I believe. So chunking at least is general to more than one binary format. So maybe chunksizes should stay in NC_VAR_INFO_T, and filter info, and other HDF5-only values, would move into NC_HDF5_VAR_INFO_T, and NC_VAR_INFO_T will contain a void pointer to the format specific struct.

[DennisHeimbigner]

NC_HDF5_VAR_INFO_T
Surely we can find a shorter set of names :-)

[DennisHeimbigner]

Before this goes much further, we need to have
a software architecture and design document to
codfy the API and the critical implementation
details. Something in markdown like a cross between
this https://github.com/Unidata/netcdf-c/blob/master/docs/filters.md
and this https://support.hdfgroup.org/HDF5/doc/Advanced/DynamicallyLoadedFilters/HDF5DynamicallyLoadedFilters.pdf

[edhartnett]

What, you just reuse the same comment on three threads? ;-)

First I will get a working implementation, then I will document it for you. It's easier that way. ;-)