Background
Logstash output plugins declare their concurrency model via concurrency :shared, concurrency :single, or the default :legacy. The :legacy strategy was originally designed to support a per-output workers setting that created N independent plugin instances distributed via a BlockingQueue. Currently, the workers can only be 1. This model predates the current pipeline.workers architecture.
Current State
An audit of all Elastic maintained output plugins across the logstash-plugins organisation:
:single: google_cloud_storage, icinga, stdout, tcp, zeromq:shared (via concurrency :shared or the deprecated declare_threadsafe!): elasticsearch, file, google_bigquery, google_pubsub, http, kafka, rabbitmq, s3, sns, sqs, timber, integration-aws, integration-kafka, integration-logstash, integration-rabbitmq:legacy (default, no explicit declaration): appsearch, boundary, circonus, cloudwatch, csv, datadog, datadog_metrics, elastic_app_search, email, exec, firehose, ganglia, gelf, graphite, hipchat, influxdb, irc, jira, jms, logentries, loggly, lumberjack, mongodb, nagios, nagios_nsca, newrelic, null, opentsdb, pagerduty, pipe, redis, riemann, slack, statsd, stomp, syslog, udp, webhdfs, websocket, xmpp, zabbix, zookeeper, integration-elastic_enterprise_search, integration-zeromq
None of these plugins configure a workers value. LegacyOutputStrategyExt therefore always initialises with workers = 1, creating a single instance behind a BlockingQueue of size 1. This is functionally identical to SingleOutputStrategy.
declare_workers_not_supported! is unused across the entire logstash-plugins organisation.
Problem
The :legacy strategy is still supported but no plugin actively uses its multi-worker capability. Keeping it as the default misleads plugin authors into thinking per-output worker pools are still actively maintained and in use.
Proposal
- Change the default in
outputs/base.rb from :legacy to :single
- Remove
LegacyOutputStrategyExt and its registration in RubyUtil.java
- Remove the deprecated
declare_workers_not_supported! helper
- Add a deprecation warning if any plugin explicitly declares
concurrency :legacy
Risk
No behavioural change for any current plugin. All plugins currently on :legacy with workers = 1 will have identical serialised-access semantics under :single. The only breaking case would be a custom plugin that explicitly declares concurrency :legacy and sets workers > 1 in user config, which is not supported by any known plugin.
Background
Logstash output plugins declare their concurrency model via
concurrency :shared,concurrency :single, or the default:legacy. The:legacystrategy was originally designed to support a per-outputworkerssetting that created N independent plugin instances distributed via aBlockingQueue. Currently, theworkerscan only be 1. This model predates the currentpipeline.workersarchitecture.Current State
An audit of all Elastic maintained output plugins across the
logstash-pluginsorganisation::single:google_cloud_storage,icinga,stdout,tcp,zeromq:shared(viaconcurrency :sharedor the deprecateddeclare_threadsafe!):elasticsearch,file,google_bigquery,google_pubsub,http,kafka,rabbitmq,s3,sns,sqs,timber,integration-aws,integration-kafka,integration-logstash,integration-rabbitmq:legacy(default, no explicit declaration):appsearch,boundary,circonus,cloudwatch,csv,datadog,datadog_metrics,elastic_app_search,email,exec,firehose,ganglia,gelf,graphite,hipchat,influxdb,irc,jira,jms,logentries,loggly,lumberjack,mongodb,nagios,nagios_nsca,newrelic,null,opentsdb,pagerduty,pipe,redis,riemann,slack,statsd,stomp,syslog,udp,webhdfs,websocket,xmpp,zabbix,zookeeper,integration-elastic_enterprise_search,integration-zeromqNone of these plugins configure a
workersvalue.LegacyOutputStrategyExttherefore always initialises withworkers = 1, creating a single instance behind aBlockingQueueof size 1. This is functionally identical toSingleOutputStrategy.declare_workers_not_supported!is unused across the entirelogstash-pluginsorganisation.Problem
The
:legacystrategy is still supported but no plugin actively uses its multi-worker capability. Keeping it as the default misleads plugin authors into thinking per-output worker pools are still actively maintained and in use.Proposal
outputs/base.rbfrom:legacyto:singleLegacyOutputStrategyExtand its registration inRubyUtil.javadeclare_workers_not_supported!helperconcurrency :legacyRisk
No behavioural change for any current plugin. All plugins currently on
:legacywithworkers = 1will have identical serialised-access semantics under:single. The only breaking case would be a custom plugin that explicitly declaresconcurrency :legacyand setsworkers > 1in user config, which is not supported by any known plugin.