[Feature Request] Virtual thread support in OpenSearch

[abiesps]

Is your feature request related to a problem? Please describe

In continuation to discussion here (#18833). We have found from our experiments that virtual threads are imperative to achieve maximum io concurrency in opensearch. Virtual threads can also make the problem of statically combining segments into partition a bit simpler, we can also assign a single segment (or part of a segment) to an independent virtual thread.

Describe the solution you'd like

We are proposing to use virtual thread as an alternative to searcher threadpool (and possibly search threadpool) to execute the queries. Virtual threads can also simplify the work assignment algorithms in intra and inter-segment concurrency.

Related component

No response

Describe alternatives you've considered

No response

Additional context

No response

[sohami]

One thing to consider with virtual threads is all the virtual threads in JVM are mapped to same instance of carrier threads which is controlled by JVM. So if there are multiple different operations using virtual thread-pools like merge/search which share the same JVM, then they will loose workload isolation which is provided by separate thread-pool today. On setup where there are different hosts (separate indexer/search) for competing operations it may not be a problem. Or for use-cases (like uploading/downloading files from remote store) where execution on the carrier thread is very lightweight it can work well. What are your thoughts on that ?

[abiesps]

I envision this change in two phases,

In the short-term,
It is much cleaner and simpler to start with virtual threads on deployments with reader-writer separation for cases where working set is almost always larger than the available memory to the hosts for queries. For a dedicated search host, the background operations (like refreshes etc) can still use the regular platform threads with control on thread pool sizes and search and searcher threads can be changed to virtual threads per request.
For writers, since writes are added to in-memory buffer and translog fsync (even when durability is per request) is buffered and only by one write thread at a time, we may still continue to use regular threads as writes are mostly CPU bound.

In the long term,
It would be ideal if JVM provides a support to control the number of platform threads onto which a particular set of virtual thread tasks can be scheduled. There are some experimental discussion about this https://builds.shipilev.net/patch-openjdk-loom-fibers/loom-docs/CustomSchedulers.md.html but it is not yet supported as per my knowledge through a stable api.

or we have to build a workload isolation framework using semaphores, workload priorities and resource utilization of the host to fully utilize virtual threads.

Till then we can continue using them in mix mode (primary IO intensive workloads can be on virtual threads and rest of the workloads can stay as is).

Let me know your thoughts.

[sohami]

It is much cleaner and simpler to start with virtual threads on deployments with reader-writer separation for cases where working set is almost always larger than the available memory to the hosts for queries. For a dedicated search host, the background operations (like refreshes etc) can still use the regular platform threads with control on thread pool sizes and search and searcher threads can be changed to virtual threads per request.

Agreed and hence I mentioned those 2 cases where it can work well. But I was wondering how can we enforce that since OpenSearch can be deployed using any model by different users. Probably using virtual thread-pool by default for search threadpools may not be a good idea and instead providing mechanism for user to opt-in explicitly will make more sense such that they can make informed decisions.

For writers, since writes are added to in-memory buffer and translog fsync (even when durability is per request) is buffered and only by one write thread at a time, we may still continue to use regular threads as writes are mostly CPU bound.

Yes, but merge operations can be offloaded to virtual threads in the indexing side.

Till then we can continue using them in mix mode (primary IO intensive workloads can be on virtual threads and rest of the workloads can stay as is).

This completely makes sense, as mentioned such cases are remote store based interactions. But search is mix of IO and compute heavy where I was raising the concern of isolation. But as mentioned above with some explicit opt-in mechanism it should be fine.

[abiesps]

Agreed @sohami , this could be an explicit opt-in.