ShardListDataset cache miss rate with wids

[omri-cavnue]

Hello,

I am currently implementing a pipeline with DDP and wids. My dataloaders look like the following:

chunk_size = math.ceil(dataset_length / int(os.environ["WORLD_SIZE"]))
dataset = (
            wids.ShardListDataset(
                wids_map["shardlist"],
                cache_dir=cache_dir,
                keep=True
            )
        ).add_transform(preprocess)
loader = torch.utils.data.DataLoader(
            dataset,
            num_workers=num_workers,
            batch_size=batch_size,
            collate_fn=identify_fn,
            pin_memory=True,
            sampler=wids.DistributedChunkedSampler(dset, chunksize=chunk_size, shuffle=True) if "train" else None,
        )

While everything seems to be working correctly, I am seeing messages around cache miss rate similar to Warning: ShardListDataset has a cache miss rate of 9901.0%%. I haven't found any information on this and was wondering what these signify as it relates to ShardListDataset given the data is already cached locally on disk and the cache_dir simply points there? So I'm not sure how it would miss but still train through the iter and epoch without any performance impact (or so it seems)?

[tmbdev]

For local files, the cache misses don't matter. For remote files, it tells you that somehow your indexes don't match your shard structure very well. However, 9991% is a bit too high. Obviously, there is a bug somewhere. I'll try to see whether I can figure it out from the code, otherwise, a simple testcase that reproduces the bug would be good. You can just create the .tar file at the start of the PyTest test case.