Create cache files with CREATE_NEW & SPARSE options

[tlrx]

Cold and Frozen tiers use a local cache for some Lucene files. This cache is based on files stored on disk, and to limit the required disk space it relies on the operating system's support for sparse files.

When Elasticsearch creates the cache files it uses the following standard options:

StandardOpenOption.READ,
StandardOpenOption.WRITE,
StandardOpenOption.CREATE,
StandardOpenOption.SPARSE

Sadly those options are not enough to enable sparse file for the cache files. Instead it should use the CREATE_NEW option, as it is documented in the Java NIO API:

/**
     * Sparse file. When used with the {@link #CREATE_NEW} option then this
     * option provides a <em>hint</em> that the new file will be sparse. The
     * option is ignored when the file system does not support the creation of
     * sparse files.
     */
    SPARSE,

The consequence today is that sparse file support if not enable for operating systems that do not enable it by default, like NTFS.

Since partially mounted indices ignores disk watermarks and report a size on disk of 0 bytes it means that the cache files can grow and use much more disk space than needed, potentially filling up the disk.

Operating systems that enables sparse file by default, like most Linux distributions, should not be affected.

This pull request adds a created flag to the CacheFile class that indicates if the file is expected to already exist on disk or not. This flag is used to create or open the file with the correct options.

This bug will be documented in a follow up. I also plan to spend some time looking if we can use JNA to add assertions that the shared cache file is effectively sparse on Windows/Linux etc.

[elasticmachine]

Pinging @elastic/es-distributed (Team:Distributed)

[original-brownbear]

LGTM, though maybe adjust the naming a little :)

[original-brownbear]

Maybe name this fileExists and just describe it as "true if the physical cache file exists on disk" or so? I find the description and name very confusing tbh.

[ywelsch]

+1

[tlrx]

Sure, I pushed 3bbef1e

[ywelsch]

LGTM

[ywelsch]

+1

[tlrx]

Thanks Yannick and Armin!

[elasticsearchmachine]

💔 Backport failed

Status	Branch	Result
✅	7.x
❌	7.15	Commit could not be cherrypicked due to conflicts

You can use sqren/backport to manually backport by running backport --upstream elastic/elasticsearch --pr 79371