Skip to content

[Feature Request] io_uring based directory #18839

@abiesps

Description

@abiesps

Is your feature request related to a problem? Please describe

In continuation to discussion (#18833) , we are proposing to build a io uring based directory in opensearch. Mmap and NIOFS ( nio file channel) both tends to capture the carrier threads (with virtual threads). This limits the io concurrency that we can achieve at application level. However, we found from our experiments that we can achieve significantly higher io concurrency with scrappy iouring based directory implementation.

Describe the solution you'd like

We are proposing to build iouring (https://kernel.dk/io_uring.pdf) based IO capability in opensearch.
There is a helper library that makes the use of implementing iouring relatively simpler (https://github.com/axboe/liburing/tree/master/src).
I also found couple of implementation libraries in java (https://github.com/davidtos/JUring/tree/master), related blog post (https://nljug.org/foojay/async-file-io-with-java-and-io_uring/). Taking inspirations from these samples, I was able to implement read functionalities in lucene based io uring directory as a poc.

Related component

No response

Describe alternatives you've considered

No response

Additional context

No response

Metadata

Metadata

Assignees

Labels

StorageIssues and PRs relating to data and metadata storageenhancementEnhancement or improvement to existing feature or request

Type

No type

Projects

Status

🆕 New

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions