Lazy arrays are already available using the index file approach, e.g.
fstd2nc.Buffer.make_index(infiles, indexfile, **kwargs)
ds = fstd2nc.Buffer.open_index(indexfile)
Where it writes the metadata into a temporary indexfile. The result is more memory-friendly for large numbers of files (metadata compressed and stored efficiently in netCDF store instead of dask graphs, also reduces communication overhead between processes). Would be nice to be able to trigger this from xr.open_dataset, e.g. something like
ds = xr.open_dataset(infiles, engine="fstd", lazy=True, **kwargs)
Lazy arrays are already available using the index file approach, e.g.
Where it writes the metadata into a temporary indexfile. The result is more memory-friendly for large numbers of files (metadata compressed and stored efficiently in netCDF store instead of dask graphs, also reduces communication overhead between processes). Would be nice to be able to trigger this from
xr.open_dataset, e.g. something like