`read` operations cause too many disk reads with unlimited dimensions

[haileyajohnson]

@lesserwhirls - the problem as far as I can tell exists in netcdf-java as well.

This is a repeat of this issue which can probably be closed in favor of this one.

The problem: we're hosting netcdf-classic files with in EFS and reading them from EC2, so we're charged per bytes read. The charges have been unexpectedly high (order of magnitude) for reads from our datasets with unlimited dimensions.

I've attached the results of some testing I've done locally on my mac using fs-usage, input files, the output of those tests, and the python script I used to count bytes read. The short summary is output from reading a fixed dimension dataset are about what we expect with both ncdump and toolsui, but way too high when reading from a record dataset.

test_disk_reads.zip

[haileyajohnson]

Updating my analysis to include some info on reading from a netcdf4 dataset with an unlimited dimension:

It works as expected with ncdump! This is great because we have full control over the datasets in question and can switch to nc4 pretty easily to save a lot of money. There are still two problems with that though:

1. ncdump makes a read of ~4.2Mb early on in the process, no matter the file (nc3, nc4, fixed dimension, unlimited), which is not inconsequential.
2. ToolsUI still does a bad job with over-reading from disk with netcdf4s, so any datasets we have exposed via THREDDS would still be a liability.

[lesserwhirls]

I am going to guess the over-reading by netCDF-Java is due to this?

[haileyajohnson]

Maybe, but we're talking about ~150 Mb of disk reads to ncdump a 7.9Mb file, which is more than just touching every memory block.
^That's with both java and C

[WardF]

Interesting; I'm taking a look at this now. I'm in rural Oregon (again), forgive the late response. I'm curious if the same behavior is observed with h5dump; it will be helpful to figure out if this is something in libnetcdf or libhdf5.

[WardF]

Also, thanks for the tip about fs-usage, I was unfamiliar.

[haileyajohnson]

Also, thanks for the tip about fs-usage, I was unfamiliar.

Honestly ChatGPT and I had some fights before finally nailing down a workflow I was reasonably confident in 😆

[WardF]

If the workflow is portable and easy to share, I wouldn't mind taking a peek and using it to recreate the issue and then see what we're able to do about it/testing any potential fixes. :)

[haileyajohnson]

It should be portable to a mac! The quick synopsis is:

• run fs_usage -w -f filesys and >> the results into a text file
• run the ncdump command I'm testing in a another window
• end fs_usage
• change the output file for fs~usage and run again on next test case

Then once I had all the logs I parsed them with python regex to find the number of bytes recorded as read by ncdump. I tried with grep and awk, but it was easier for me with python, I'm sure you'd have better luck than I there.

A couple of things I've learned while staring at these logs for hours:

• recording reads only is easier to parse, but if you record everything you can also see when ncdump opens and closes the actual dataset file, which is also useful.
• With netcdf4, all the data reads are pread instead of read (which totally makes sense but that caught me up for a while when I was only searching for reads)

[WardF]

I'll take a look when I get back to Colorado next week, when I have access to a MacOS machine with sudo. Thanks!

[WardF]

@haileyajohnson out of curiosity, is the same behavior observed if you try using the NCO tools?

[WardF]

My initial un-tested thought is that we may be re-reading the metadata any time we need to; this was probably less of an issue when file storage was all local. I've arrived back in Colorado and will sit down and see if I can set up the same workflow and then start to sort this out. If we are able to determine whether or not this is happening in NCO, it will help narrow down if this is happening in libnetcdf or in ncdump, and also potentially offer a stopgap/work around.

[haileyajohnson]

I can checkout NCO and get back to you.
Your initial untested thought seems on the right track to me, looking at metrics on AWS I can see the reads split up into data reads, which more or less make sense, and metadata reads, which seem large.

[haileyajohnson]

My theory for the netcdf3 overreading (which isn't blocking us btw) is that reading a single unlimited variable works as intended in that it reads the whole record, but reading multiple unlimited variables reads the whole record again for each variable. I'll probably self-assign looking into that in netcdf-java, but I doubt I'll get to it this week :)

[DennisHeimbigner]

That sounds like a plausible explanation for netcdf-3.

[DennisHeimbigner]

I can checkout NCO and get back to you. Your initial untested thought seems on the right track to me, looking at metrics on AWS I can see the reads split up into data reads, which more or less make sense, and metadata reads, which seem large.

I would be surprised if this was the case. Ed Hartnett did some significant changes to add lazy meta-data evaluation.