Missing files in Hugging Face

[ADHuan]

Hi,

I recently downloaded the full Satlas pretrain dataset from Hugging Face. However, upon reviewing the file lists, I noticed that several tar files for the NAIP dataset are missing for the years 2013, 2016, 2017, and 2018. The missing files are illustrated in the attached screenshots below.

Additionally, I have roughly calculated the total data size of the available tar files for 2013, 2016, 2017, and 2018. There seems to be a mismatch between this total and the data size listed in your AWS S3 bucket.

Could you please verify the completeness of the dataset and ensure that all files are available for download on Hugging Face?

[favyen2]

Yes it is complete, and I checked the total data size matches for 2018 (1.7 TB). Your screenshots show that files from every year are available. NAIP images are captured once every 2-3 years at a given location.

[favyen2]

OK I see now what you mean about missing files. I will check it.

[favyen2]

In the meantime please download from S3: https://github.com/allenai/satlas/blob/main/satlaspretrain_urls.txt