Async versions for Reader and Writer

[splix]

I'm wondering if there are any plans (or opposition to those) for async implementation for Reader and Writer? I.e., a reader from tokio::io::AsyncRead, etc

[splix]

I've tried to add it, but quickly realized that it requires [mostly] duplicate versions for too many of internal functions.

Like for

pub fn zag_i64<R: Read>(reader: &mut R) -> AvroResult<i64> {
    let z = decode_variable(reader)?;
    Ok(if z & 0x1 == 0 {
        (z >> 1) as i64
    } else {
        !(z >> 1) as i64
    })
}

it needs an accompanying:

#[cfg(feature = "tokio")]
pub async fn zag_i64_async<R: AsyncRead>(reader: &mut R) -> AvroResult<i64> {
    let z = decode_variable_async(reader).await?;
    Ok(if z & 0x1 == 0 {
        (z >> 1) as i64
    } else {
        !(z >> 1) as i64
    })
}

and so on.

It's definitely possible to implement, but the main question if that aligns well with avro-rs plans?

[martin-g]

Duplicating the code is no go.

IMO a more feasible way is to make everything async and use a same-thread thread pool for sync usage. But I haven't seen any other library doing it so probably I miss something.

[splix]

For this particular example it's possible to move out the duplicate part into an inline function, and keep only decode_variable_async(reader).await?; and decode_variable_async(reader)?; depending on the variant. Another option is wrapping those with some macros which depending on the compilation target adds .await or not. Anyway, I agree that both options are bit ugly.

Or actually I think that many functions like this one could be transformed to something more universal which could work in both ways. I'm worried mostly about the fact that too many things must be touched to support async.

[splix]

Regarding the sync usage, there is https://docs.rs/tokio-util/0.7.4/tokio_util/io/struct.SyncIoBridge.html that allows it

[martin-g]

Closing the issue since there is no activity for almost 2 months now.
We could re-open it if there are ideas how to do it!

[Kriskras99]

Reqwest is async by default, but has a blocking feature flag. It wraps the async types and spawns a Tokio thread to execute the async functions (https://github.com/seanmonstar/reqwest/blob/master/src/blocking/client.rs#L1234).

Although reqwest uses Tokio, it might make sense to use smol to keep the impact on sync users small.

[martin-g]

What would be the benefit of using async in Avro?
File IO is not async anyway.
Computation should be sync too.
What would we gain ?

[Kriskras99]

I'm reading/writing files over the network, which is async. So currently I'm retrieving the entire file and then decoding it, instead of decoding it while it's downloading.

[splix]

I have Avro on S3 too. So the whole code is async except the Avro writing/reading

[bmitc]

This is a fairly large hiccup in work I am doing. Everything in the application, which makes file reads and HTTP requests, runs within Tokio, but to read data with the Avro reader, I need to use SyncIoBridge to be able to pass tokio_stream::Stream (via a StreamReader) data.

What would be the benefit of using async in Avro?

The benefits of async are not specific to Avro but apply to Avro. By this library being synchronous only, to read multiple files all concurrently, one needs to use threads rather than an asynchronous, task-based runtime like Tokio.

File IO is not async anyway.
Computation should be sync too.

This is not accurate. Tokio provides asynchronous file utilities via tokio::fs. For example, tokio::fs::File is AsyncRead.

[Kriskras99]

File IO is not async anyway.
Computation should be sync too.

This is not accurate. Tokio provides asynchronous file utilities via tokio::fs. For example, tokio::fs::File is AsyncRead.

It is accurate, under the hood Tokio spawns a "blocking" thread which reads the file using std::io and then pushes it to the async runtime

[Kriskras99]

@martin-g if you're interested I can write a mini-RFC discussing the options for an async variant/implementation, and after we've agreed (or decided it's not worth it) I can write the implementation.

[bmitc]

File IO is not async anyway.
Computation should be sync too.

This is not accurate. Tokio provides asynchronous file utilities via tokio::fs. For example, tokio::fs::File is AsyncRead.

It is accurate, under the hood Tokio spawns a "blocking" thread which reads the file using std::io and then pushes it to the async runtime

It isn't accurate at a practical level. tokio::fs exists because there is a need to make file operations behave asynchronously. Whether or not the file operations are actually ascynrhonous is an implementation detail that doesn't support why there shouldn't be an asynchronous version of this Avro reader.

The benefits of having this Avro library support AsyncWrite and AsyncRead are not tied to file operations but are tied to the reasons why async exists in the first place and allowing this Avro library to be more easily integrated into async and concurrent applications. If one wants to stream a lot of Avro data from many sources at the same time, then Tokio is the way to do that, not having to manually manage threads. SyncIoBridge works with this library, but it's awkward.

Lastly, things that implement Read and AsyncRead do not always come from files. As many pointed out here, it's common to have Avro data in S3 or other such cloud-based systems, where the data is streamed over HTTP, which is asynchronous.

[martin-g]

@martin-g if you're interested I can write a mini-RFC discussing the options for an async variant/implementation, and after we've agreed (or decided it's not worth it) I can write the implementation.

I am very much interested !
Any help is very welcome!