-
Notifications
You must be signed in to change notification settings - Fork 2.5k
Description
Is your feature request related to a problem? Please describe
In continuation to discussion (#18833) , we are proposing to build a io uring based directory in opensearch. Mmap and NIOFS ( nio file channel) both tends to capture the carrier threads (with virtual threads). This limits the io concurrency that we can achieve at application level. However, we found from our experiments that we can achieve significantly higher io concurrency with scrappy iouring based directory implementation.
Describe the solution you'd like
We are proposing to build iouring (https://kernel.dk/io_uring.pdf) based IO capability in opensearch.
There is a helper library that makes the use of implementing iouring relatively simpler (https://github.com/axboe/liburing/tree/master/src).
I also found couple of implementation libraries in java (https://github.com/davidtos/JUring/tree/master), related blog post (https://nljug.org/foojay/async-file-io-with-java-and-io_uring/). Taking inspirations from these samples, I was able to implement read functionalities in lucene based io uring directory as a poc.
Related component
No response
Describe alternatives you've considered
No response
Additional context
No response
Metadata
Metadata
Assignees
Labels
Type
Projects
Status