Grpc output plugin

[talfridmen]

Use Case

When sending to a custom endpoint, allow for grpc instead of http/raw socket.
Similar to http endpoint but more lightweight.

Expected behavior

Send the metrics in any supported format (influxdb,json,csv,etc..) to a grpc listener.
The protobuf format should of course be published too for others to be able to use in their own services.
Could be as basic as a list of strings - and let the server handle reading it back, or creating a specific protobuf format for each metric type.

Actual behavior

This does not currently exist, except for otel, which has many more restrictions, such as single field per metric, no string fields, etc..

Additional info

No response

[srebhan]

@talfridmen what exactly do you have in mind? What is your use case?

I'm asking because from your generic description we would need to define a GRPC protocol, i.e. a protobuf for the metrics as well as the RPC function signature and you have to implement that GRPC server... So I want to understand if that server already exists and/or how you plan to use that plugin...

[talfridmen]

@srebhan
I agree - we need to define some protobufs.
The way I see it, a list of strings is technically enough. Of course that creating specialized protobuf structures for different formats (i.e. influx line protocol, and such) could be more robust, but in actuality it will probably be more resource consuming - both to serialize and specifically the payload size.

The purpose of this, is to have an ultra lite method of sending data to custom endpoints. As I stated, similar to sending over http, which has a massive overhead.

I see it that if I want to use it, I will import the protobufs from telegraf into my custom service, and will handle it as I see fit.

My current specific usecase is to insert the metrics into a custom db we have. While I can implement an even more specific plugin for my database, I thought this wil be more generic, and could allow others to use. Plus as a bonus, it saves me from maintaining many connections to the db, and opening a small pool conconnections my custom grpc server will use to aggregate and push the data through.

I have an initial implementation I worked on (not yet production ready, but getting there). Instead of adding it locally and compiling telegraf myself, I thought that would be a nice addition to the upstream.

[srebhan]

Sorry but it's still not clear yet what you are after....

The purpose of this, is to have an ultra lite method of sending data to custom endpoints. As I stated, similar to sending over http, which has a massive overhead.

Well GRPC uses HTTP/2 under the hood, so how can it be less overhead? Or are you referring to serialization/deserialization cost?

My current specific usecase is to insert the metrics into a custom db we have. While I can implement an even more specific plugin for my database, [...]

Which would be the better alternative as it allows the output plugin to be optimized to the use-case avoiding additional copy, serialization and deserialization steps.

[...] Plus as a bonus, it saves me from maintaining many connections to the db, and opening a small pool conconnections my custom grpc server will use to aggregate and push the data through.

You could do the same with Telegraf. Many people use an "aggregator" Telegraf instance receiving data from other Telegrafs, batch them and send the data to your data source(s). Buffering and batching is one of the core-competencies of Telegraf. Furthermore, e.g. the Golang SQL framework handles the connection pooling for you and probably other database-drivers do as well.

I have an initial implementation I worked on (not yet production ready, but getting there). Instead of adding it locally and compiling telegraf myself, I thought that would be a nice addition to the upstream.

So why don't you push it to a draft PR here and we can discuss on the code?