Expose dispatch layer so that users can add their own local binary format

[edhartnett]

Introduction

In order to enable the HYCOM model to be updated to the netCDF API, and to provide a general capability to users, I propose to expose the netCDF-C dispatch layer so that users can add their own local format. This feature, already available in the netCDF-C library internals, is not currently exposed to users.

Background

HYCOM Model and Data Format

The HYCOM modeling team would like to upgrade their model to netCDF, in part to take advantage of the ParallelIO libary, which provides a standard way to ensure good performance in the I/O layer, specifically in the case of asynchronous I/O to a subset of processors using parallel I/O to write to disk.

Many legacy tools have been written based on the existing HYCOM native binary format ("AB Format") Converting the model to netCDF would break all existing tools, leading to loss of significant investment of time and effort, as well as disrupting work with a need to convert all necessary tools at the same time.

Providing and Upgrade Path for HYCOM

With the NetCDF-C dispatch layer exposed, an external plug-in can be provided for the netCDF-C library. Once this plug-in is in place, HYCOM programmers can convert model code to the netCDF API, while still reading and writing their local binary format.

All tools based on the netCDF-C library will transparently read the AB Format (including all tools in Fortran 77, Fortran 90, C++, Python, Perl, Mathmatica, etc.) These tools will also be able to write the AB Format with the netCDF API, allowing them to be converted one at a time.

This will give HYCOM modelers an upgrade path for their existing large code base of code which works with the local binary format. One by one, as time permits, these tools can also be upgraded to the netCDF API.

Eventually, when all tools are converted, the arguments of the nc_create commands in the HYCOM model can be changed to use one of the supported netCDF formats. All tools will transparently read netCDF files, and files in their local format. Historical data (in AB Format) will still be transparently available to netCDF programs. When a file is opened, the netCDF-C library will recognize it as an AB file, and use the plug-in code to read the file. User applications will be able to access the file through the netCDF API.

NettCDF-Java Plug-Ins

The netCDF Java library allows users to write a plug-in which allows them to support read/writes to their own binary format, through the netCDF Java API. The plug-in enables the netCDF library to read/write an arbitrary local format. This feature is widely used in the netCDF-Java community.

The proposed changes will bring this capability to the netCDF-C user community.

General Applicability for the NetCDF Community

Although this work will directly benefit the HYCOM mode, the need for an upgrade path from a local format to netCDF is general. I propose that this become part of the netCDF release and become available to all netCDF-C library users.

Technical Approach

The solution consists of two parts: a plug-in to read and write AB Format, and the changes to the C library necessary to support use of the plug-in, and others like it.

Changes to NetCDF-C to Expose the Dispatch Layer

The C library contains a "dispatch" layer which allows for other formats to be added to the C library. This is how netCDF-4/HDF5, OPENDAP, HDF4, and the parallel-netcdf library are currently supported.

But the C library does not expose this plug-in capability. Changing the C library source code is required in order to add a new version.

For the purposes of this discussion, AB Format, and other such formats, will be called "user-defined formats".

Dispatch Layer Library

The user is responsible for building a dispatch layer library.

This library, when linked with netCDF and a user application, allows the netCDF library to understand the user-defined format. The user-defined format dispatch library implements all the necessary functions from the netCDF dispatch table. (This is a subset of the entire netCDF API).

The user-defined format is registered with a new function, nc_def_user_format().

The AB Format

The local format for the HYCOM model is called "AB Format." It consists of two files, one with metadata, and one with the data stored in binary form.

An AB "file" consists of two files on disk, a binary file with extension "a", and an ASCII text file with metadata with a "b" extension. (The names of the file are part of the format definition.)

The A file is IEEE big-endian, direct access, with a fixed record length.

Converting the AB Format to the NetCDF Internal Model

For the netCDF library to read and write files in AB format, the plug-in must read the metadata file, and provide functions in accordance with the internal needs of the netCDF library. Reading and writing are handled separately.

Reading the AB Format

When reading the AB format the netCDF library needs functions which will:

• identify the file as an AB format file
• read metadata and provide it to netCDF internals
• read the data in subsetted arrays.

Identifying AB Files

AB Files can be identified by name. When an nc_open call is made, and the file name is not found, but two files "FILENAME.a" and FILENAME.b" are found, then the library knows it will be opening an AB file.

Reading AB File Metadata

Once a file has been identified as AB Format, the netCDF library will need to read and understand all the metadata pertaining to that file. This includes the names, types, and sizes of all variables, attributes, and dimensions in the file.

With AB format, code will be written to read the .a file and provide metadata information to the netCDF library.

Reading Subsetted Arrays of AB File Data

The core data read operation of netCDF is nc_get_vara_TYPE (where TYPE varies). This call allows the user to read a subset of a data variable into an n-dimensional array. With this operation, the netCDF-C library implements most of the other read operations.

In the case of AB Format the subsetted array read will be straightforward. Based on the information in the metadata file, the exact offsets to any element of the data can be computed.

Writing the AB Format

In writing the AB Foramt, the netCDF library must:

• Create the AB files.
• Fill the metadata file.
• Implement the subset array write operation.

Creating an AB File

The config file which is read at library load will contain the mode flag which indicates the AB format.

Writing the AB Metadata File

Once the nc_create call returns, no further disk access takes place until the nc_enddef() call. During the time between nc_create() and nc_enddef() the user can define variables, attributes, and dimensions for the file.

When nc_enddef() is called, the AB Plug-in code muse write the B file, which contains the metadata for the file.

Only the netCDF-classic data model will be supported by the AB Format Plug-in. Other restrictions on types and objects may be necessary.

Subset Write Operation

The netCDF subset write operation nc_put_vara() is the key write operation from which other write operations are built. Once implement for the AB format, all netCDF write operations will work for the files.

Conclusion

Exposing the netCDF-C library dispatch layer will benefit the HYCOM modeling group, and also other netCDF-C users (including all netCDF fortran, python, perl, etc.)

Comments and feedback welcome. Please add to this issue so that all can participate in discussion.

[edhartnett]

OK, I now have this working on HPC NetCDF. It's really neat. (@wkliao this is one of the interesting new features I mentioned).

To see it in use, take a look at this github project:
https://github.com/HPC-NetCDF/ab-dispatch

In this project, HPC NetCDF is used, and a new netcdf function is called: nc_def_user_format().

After that function is called, the AB Dispatch code is used to read the file. At the moment I've just stuck the HDF4 code in there, so instead of reading AB format, the AB Dispatch code is reading HDF4. ;-) However, that code just has to be swapped out for AB format code that does the same thing (i.e. read the file and present it in the netCDF data model).

Here's how it looks in code:

   printf("\nTesting AB format dispatch layer...");
   if ((ret = nc_def_user_format(NC_UF0, &AB_dispatcher, NULL)))
      return ret;

   if ((ret = nc_open(TEST_FILE, NC_UF0, &ncid)))
      return ret;
   if ((ret = nc_inq(ncid, &ndims, &nvars, &natts, &unlimdimid)))
      return ret;
   /* printf("ndims %d nvars %d natts %d unlimdimid %d\n", ndims, nvars, natts, unlimdimid); */
   if (ndims != 4 || nvars != 6 ||natts !=10 || unlimdimid != -1)
       return 111;
   if ((ret = nc_close(ncid)))
      return ret;

I have added 2 user formats, NC_UF0 and NC_UF1, so the user can define two different formats at the same time, and use the netCDF API to interchangeably read/write them.

It's a neat feature. I will announce it on the netCDF mailing list sometime this week.

It's going to take time to get this into Unidata netCDF, due to the backlog of my PRs. To prevent out of order merging I'll hold off on putting this up until all or most of my currently-pending PRs are merged.

[edhartnett]

Here's a slide showing how user formats will work.

[DennisHeimbigner]

Duplicate of comment at: #814 (comment)

In restrospect, I made a mistake in making the # of entries in the dispatch
table (include/ncdispatch.h) differ depending on whether or not netcdf4
was enabled. The IMO proper way to do this is to make all dispatch table
entries be included and implement them to return NC_ENOTNC4 in e.g.
libsrc and libdap2 and pnetcdf.
P.S Making the dispatch table fixed size will also help Ed attempt at dynamic loading.

[DennisHeimbigner]

One critical issue is telling the netcdf library where and how to load the
dynamic format .so (or .dll) file. It is instructive to see how HDF5 does this for
its filter plugins. I have a short description in docs/filters.md#Process.

[edhartnett]

But this turns out not to be necessary with this solution.

The user-format dispatch layer is a library, which uses the netCDF library. (And the netCDF library does not have to be re-compiled.)

Then the user writes a program, which uses the dispatch layer library he/she has just written. And that library uses netCDF.

So our code code does not have to deal with .so or .dll files. All functions are available at program compile time, either in the user-dispatch library or the netCDF library.

[DennisHeimbigner]

From my point of view, this is a bad and complicated solution.
This no different from our current situation of compiling library usage
into the netcdf library. To call it dynamic seems incorrect.

[DennisHeimbigner]

Let me paraphrase your proposal and tell me where I have it wrong.

1. You have wrapper library on top of netcdf-c
2. In order to add a new format, you implement the netcdf API
   for that format
3. You then add the format implementation code to your wrapper
   source code repo, establish some connections (e.g. pointer to the api/dispatch table
   plus a mode flag + whatever).
4. You then recompile the wrapper (with the format code).
5. The user then uses this newly compiled wrapper in place of the netcdf
   library.

[edhartnett]

First there are some changes in netCDF:

1 - I added function nc_def_user_format() which takes a pointer to a dispatch table, a mode flag (NC_UF0 or NC_UF1), and an optional magic number (not yet used).

2 - I modified the netCDF dispatch code in dfile.c so that when it gets NC_UF0 or NC_UF1 it uses the provided user-format dispatch table. (Magic number modifications still to come.)

Now for a user to implement their own binary format:

1 - User writes a library that implements the required netCDF dispatch functions. It is not a wrapper for netCDF. (Example here: https://github.com/HPC-NetCDF/ab-dispatch). It's the same code as lives in the existing dispatch directories (like HDF4), but it lives in its own repo and library. It is not mixed with netCDF code (but does use netCDF headers). The library contains a dispatch table, with pointers to all the functions needed.

2 - Only the functions needed in a dispatch layer are required. It can easily reuse standard functions that will work (as the HDF4 layer reuses all the existing nc_inq_* functions). So not every function is implemented.

3 - The user writes a program that links to both netCDF and their own, separately compiled and installed, dispatch library. (It actually does not have to be separate from the user application, but it is separate from netCDF.)

4 - The user code calls nc_def_user_format(), with a pointer to the dispatch table in the user-format dispatch library. In this call, user also associates one of two pre-defined mode flags (NC_UF0 or NC_UF1) with the format.

5 - From then on, NC_UF0 in the mode flag means use this custom dispatch table. The netCDF library is not recompiled or wrapped.

6 - Other modeling groups, working on the same HPC with the same netCDF installation, can define their own NC_UF0, by writing and linking to their own dispatch layer. Each program can have up to two user-defined formats in use at the same time (in addition to all the usual netCDF formats).

This way the code for the user-defined dispatch layer is completely and always outside the netCDF code. Users can develop dispatch layers, exchange them, or whatever. They can write a custom dispatch layer for a proprietary format, and not release it. It's a separate library, which plugs in to netCDF. Dynamic loading is taken care of when a pointer to the dispatch table is passed to nc_def_user_format().

So this is quite different from the current situation, where each dispatch layer has to be built into the library, and then the library must be rebuilt.

[edhartnett]

Here's a little sneak preview of some code. This is all working code, BTW.

/* User-defined formats. */
NC_Dispatch* UF0_dispatch_table = NULL;
NC_Dispatch* UF1_dispatch_table = NULL;

/**
 * Add handling of user-defined format.
 *
 * @param mode_flag NC_UF0 or NC_UF1
 * @param dispatch_table Pointer to dispatch table to use for this user format.
 * @param magic_numer Magic number used to identify file. Ignored if NULL.
 *
 * @return ::NC_NOERR No error.
 * @return ::NC_EINVAL Invalid input.
 * @author Ed Hartnett
 */
int
nc_def_user_format(int mode_flag, NC_Dispatch *dispatch_table, char *magic_number)
{
   if (mode_flag != NC_UF0 && mode_flag != NC_UF1)
      return NC_EINVAL;
   if (!dispatch_table)
      return NC_EINVAL;

   switch(mode_flag)
   {
   case NC_UF0:
      UF0_dispatch_table = dispatch_table;
      break;
   case NC_UF1:
      UF1_dispatch_table = dispatch_table;
      break;
   }
   
   return NC_NOERR;
}

Later, in NC_open():

   /* Check for use of user-defined format 0. */
   if (cmode & NC_UF0)
   {
      if (!UF0_dispatch_table)
         return NC_EINVAL;
      model = NC_FORMATX_UF0;
      dispatcher = UF0_dispatch_table;
   }

   /* Check for use of user-defined format 1. */
   if (cmode & NC_UF1)
   {
      if (!UF1_dispatch_table)
         return NC_EINVAL;
      model = NC_FORMATX_UF1;
      dispatcher = UF1_dispatch_table;
   }

Then, in my sample user-defined format library, I have:

NC_Dispatch AB_dispatcher = {

NC_FORMATX_UF0,

AB_create,
AB_open,

AB_redef,
AB__enddef,
AB_sync,
AB_abort,
AB_close,
AB_set_fill,
AB_inq_base_pe,
AB_set_base_pe,
AB_inq_format,
AB_inq_format_extended,

NC4_inq,
NC4_inq_type,

AB_def_dim,
NC4_inq_dimid,
NC4_inq_dim,
NC4_inq_unlimdim,
AB_rename_dim,

NC4_inq_att,
NC4_inq_attid,
NC4_inq_attname,
AB_rename_att,
AB_del_att,
NC4_get_att,
AB_put_att,

AB_def_var,
NC4_inq_varid,
AB_rename_var,
AB_get_vara,
AB_put_vara,
NCDEFAULT_get_vars,
NCDEFAULT_put_vars,
NCDEFAULT_get_varm,
NCDEFAULT_put_varm,

NC4_inq_var_all,

AB_var_par_access,
AB_def_var_fill,

NC4_show_metadata,
NC4_inq_unlimdims,

NC4_inq_ncid,
NC4_inq_grps,
NC4_inq_grpname,
NC4_inq_grpname_full,
NC4_inq_grp_parent,
NC4_inq_grp_full_ncid,
NC4_inq_varids,
NC4_inq_dimids,
NC4_inq_typeids,
NC4_inq_type_equal,
AB_def_grp,
AB_rename_grp,
NC4_inq_user_type,
NC4_inq_typeid,

AB_def_compound,
AB_insert_compound,
AB_insert_array_compound,
AB_inq_compound_field,
AB_inq_compound_fieldindex,
AB_def_vlen,
AB_put_vlen_element,
AB_get_vlen_element,
AB_def_enum,
AB_insert_enum,
AB_inq_enum_member,
AB_inq_enum_ident,
AB_def_opaque,
AB_def_var_deflate,
AB_def_var_fletcher32,
AB_def_var_chunking,
AB_def_var_endian,
AB_def_var_filter,
AB_set_var_chunk_cache,
AB_get_var_chunk_cache,

};

Note the used of NC4_ and NCDEFAULT functions to lighten the load. I only implemented the functions that start AB_, and many of those just return NC_EPERM since this is read only.

To use the AB dispatch layer, I do this:

   printf("\nTesting AB format dispatch layer...");
   if ((ret = nc_def_user_format(NC_UF0, &AB_dispatcher, NULL)))
      return ret;

   if ((ret = nc_open(TEST_FILE, NC_UF0, &ncid)))
      return ret;
   if ((ret = nc_inq(ncid, &ndims, &nvars, &natts, &unlimdimid)))
      return ret;
   /* printf("ndims %d nvars %d natts %d unlimdimid %d\n", ndims, nvars, natts, unlimdimid); */
   if (ndims != 4 || nvars != 6 ||natts !=10 || unlimdimid != -1)
       return 111;
   if ((ret = nc_close(ncid)))
      return ret;

   printf("SUCCESS!\n");
   return 0;

[DennisHeimbigner]

Ok, I see. It is actually a pretty good solution. We ought to propose
a variant to the HDF5 group as a means for defining a filter plugin.

I might suggest we provide an extra function for the dispatch table
that return true/false if an existing file path or URL can be read
by that format. Netcdf-java has such, and it would simplify
the current NC_open code in libdispatch/dfile.c.

However, this solution would not work for existing systems that
use netcdf-c: systems like R and IDL. It would also be
difficult for python and Java users.

One problem I see is mode flag conflicts. We have a limited set of
available mode flags (about 16 bits). We may need to consider extending
the arguments to nc_open and nc_create. We might also consider changing
the mode to be of type long so that is can be 64 bits on 64bit machines.

[edhartnett]

This system will work for R, IDL, Fortran, and any other layer that sits on top of the C library. BUT the user-format dispatch layer must be written in C. And there must be one C call to set the user format (although we can probably do this in most languages with a void *.)

Once that is done, it works completely within the C layer. If the Fortran program passes NC_UF0 in to nf_open() it will all work. Same with python, as it uses the C library.

It does nothing for Java but they already have this feature. We are just trying to catch up. ;-)

For most HPC users and modeling groups, this will be useful. There will be one or two non-netCDF formats, and some C programmer will write the dispatch layer, and all the fortran programmers will just link to the user-format dispatch library, just as they do to netCDF. Useful formats can even be distributed by helpful folk like @opoplawski if they become popular, and then users can just install them with some package management tool.

The user does not add mode flags. I provide two: NC_UF0 and NC_UF1. This allows two different user-defined formats to be in use in the same program. There are still some mode flags available after that. I agree about the eventual mode flag scarcity problem, but we have not run out yet.

So I would propose to separate any discussion of mode flag scarcity from the issue of user-defined formats.