(bug): messages bigger than 64KB are not being logged as standalone connector

[raulb]

Bug description

Testing ConduitIO/conduit#547, we noticed this connector-log doesn't show big records (as standalone).

Steps to reproduce

• Use log connector as standalone.
• Send big records to the connector
• Notice the record is not being logged out.

We noticed this when the payload size hits 100kB (might be less, as we didn't investigate the actual size when this starts happening).

Version

v0.7.2

[raulb]

Looks like it does work when using --log.level trace

[lovromazgon]

Possibly related to rs/zerolog#660. Then again, it works for built-in plugins, which also use zerolog to print the record.

[raulb]

The issue is a combination of a limited buffer size of 64KB in the go-plugin and an issue with chunking the message

Context

Not big enough buffer size in the hashicorp/go-plugin

We're not setting the PluginLogBufferSize in the PluginConfig which means that it utilizes the default value to 64KB (set in the plugin config here):

 // defaultPluginLogBufferSize is the default size of the buffer 
// used to read from stderr for plugin log lines. 
const defaultPluginLogBufferSize = 64 * 1024

When the log line increases, the line should be split in chunks.

This , however, is not working accordingly, and my guess is that it has to do with the fact that the message is a JSON and it's not being chunked accordingly.

Why built-in connectors behave differently

Standalone connectors use a different Logger: i.e.: a default context logger, being necessary reading from stderr (via go-plugin).

Possible fixes

1. Update PluginConfig to a bigger value (maybe reutilize the recently added connectors.max-receive-record-size (though message size is not necessarily the same as record size).
2. Possibly modify the source implementation of logStderr and possibly using https://pkg.go.dev/go/scanner removing the continue logic so the message is printed out at full.
3. Document that when this happens in the log connector recommend using log.level debug since the record is shown but not formatted as usual:

2025-05-23T11:29:48+00:00 DBG {"@level":"info","record":{"position":"Mg==","operation":"create","metadata":{"conduit.source.connector.id":"noop:gen","opencdc.createdAt":"1747992587488214000","opencdc.readAt":"1747992588739469000"},"key":"YmFzaXNjb3BpYw==","payload":{"before":null,"after":"...component=plugin.standalone connector_id=noop:log plugin_name=conduit-connector-log

Why using log.level debug works

Messages are shown here instead.

[hariso]

For the record: the team decided to just document this issue, given that the log connector is a built-in one, and the majority of use cases will be as such.