[META] Writable Warm Index

[rayshrey]

Please describe the end goal of this project

This meta task will keep track of all the ongoing and completed development work related to Writable Warm Index.
This includes the composite directory and file cache integration, hot/warm migration support and performance optimizations.

Supporting References

Writable Warm RFC - #12809

Issues

Composite Directory:

• [Writable Warm] Design and implement Composite Directory and integrate with FileCache #12781
• [Writable Warm] Recovery/Replication flow for Composite Directory #13647
• Add support for dynamic switching of IndexInput from Full file to Block based (TODO: Create issue)

FileCache Improvements:

• [Writable Warm] File Pinning support in FileCache #13648
• [Writable Warm] More refined stats in FileCache #17479
• FileCache bootup logic for Composite Directory (TODO: Create issue)

Hot/Warm Migration

• Support for a new node role(WARM) for Writable Warm indices (TODO: Create issue)
• Add hook in Composite Directory for hot to warm migration (TODO: Create issue)
• Add hook in Composite Directory for warm to hot migration (TODO: Create issue)
• Migration APIs for hot/warm migration

---

Follow-up AIs from the above issues:

• Optimize readInput for Block files of RemoteDirectory to avoid loading whole 8MB chunk in memory
• Explore and verify impact of temp files in directory as they are not tracked in FileCache
• Add new role for Writable Warm for FileCache initialization logic (instead of being dependent on Feature Flag)
• Refactor OnDemandSnapshotIndexInput and FileInfo to more generic names as they are now being used by Composite directory as well which is not Snapshot related.
• Optimize and make listAll() of RemoteSegmentStoredircetory more robust (fix NPE on empty metadata and improve complexity)

Backlogs:

• Use String as key of FileCache instead of Path to avoid having hard dependency of local directory being FSDirectory

Related component

Storage:Remote