Ability to ignore extra fileds in TOML config (or missing fields in plugins)

[mialinx]

Use Case

To monitor our data plane, we use Telegraf with several dozen self-written plug-ins.
Plugins are supported by several teams; some of the plugins are used by several teams at once.

In such a situation, we often have problems with the version mismatch between plugins and their configs
(the configuration turns out to be newer than the plug-ins themselves),
which makes it impossible to launch telegraf.

The ideal solution, of course, would be to avoid such situations. But from the point of view of SRE,
it would be very convenient to make Telegraf more tolerant of unnecessary fields in the config,
to ensure forward compatibility with newer versions of configs.

Expected behavior

We have an option in plugin config that is not yet supported by the plugin code.
Telegraf starts, and the plugin just ignores extra settings.

Actual behavior

Telegraf fails to start, reporting the configuration specified the fields %q, but they were not used error

Additional info

No response

[srebhan]

@mialinx let me please understand your use-case a bit better. You want a kind of "forward compatibility" for your configs i.e. you want to use a config targeted for a newer version of Telegraf (or a plugin in your case) with an older version of Telegraf (or plugin) that does not know about a config option introduced by the newer version. Is this correct?

To be honest I'm a bit hesitant to add the requested option as this calls for trouble especially since the issue likely arises from your custom plugins not being part of Telegraf. One issue I'm seeing is that once you use the option you are intending Telegraf completely stops checking the configuration "silently" ignoring all options with a typo etc.
Another issue is that if a user adds a setting (e.g. myfeature = true) he expects a certain behavior but with that option, Telegraf behaves "random", depending on the Telegraf version, as it sometimes will exhibit the behavior (if the Telegraf version matches) and sometimes it will not (if the version is too old). This leaves users with hard-to-debug setups and will likely produce situation where users report issues to us just because using a too old version of Telegraf with a newer config.

The cause of your issue is in your custom plugins AFAICS and you should enforce a more strict setting design in your custom code. I understand that the plugin code may be outside of your control but this raises another problem e.g. if a plugin changes the type of an option from say bool to a list of strings. How is Telegraf supposed to handle those situations?

All this being said, I think we could solve this by extending the labels/selection framework by adding a built-in Telegraf version selector with a fixed key (e.g. _version) containing the Telegraf version as value (e.g. _version=v1.38). You could then add a config for each version with a label specifying the specific Telegraf version targeted by the config section or not providing the version label to target all versions. What do you think?

P.S.: Is there any reason you are not contributing the custom plugins upstream i.e. here?