Feature Request: Copy Processor

[anandsinghkunwar]

I found the rename processor for filebeat, couldn't find anything related to a copy field processor. My usecase is I want to copy some fields from the kubernetes processor to a root field with the original fields remaining intact. I have a similar usecase in the case of journalbeat as well.
I tried using the fields, fields_under_root which instead of deriving the name of the variable use the variable name itself. For eg.

fields:
  test: "%{[kubernetes][namespace]}"

makes a field test with value %{[kubernetes][namespace]} instead of derived value of that actual namespace. Requesting this feature. Also are you willing to accept a PR for the same?

Discussion link here

[ruflin]

I wonder if the upcoming field alias feature could solve your problem: elastic/elasticsearch#32172 to not have to duplicate the data? An other approach would be using copy_to in the template. https://www.elastic.co/guide/en/elasticsearch/reference/current/copy-to.html

Not against the copy processor but would like to understand where the other two proposals would fall short for your use case.

[anandsinghkunwar]

The usecase I see which doesn't fulfill this is when I have a kafka bus that sits between filebeat and ES. Logstash consumes from this kafka bus and pushes data to ES. I needed some sort of common field for all my logs which may be derived from other fields. Logstash reads this common field and understands what to do with the log, essentially what index to push, ignore it etc. I wanted this field to be made in the producer, i.e., filebeat and journalbeat in my case instead of on the logstash side. I wanted even my kafka topic name to be derived from this common field name, hence the need of a copy/derived field processor. My feature req. has a copy processor instead of a derived processor(baby steps) as I felt it would be easier to implement as there already is a rename processor.

[webmat]

One of my initial attempts at doing the Suricata module with the ECS schema assumed there was a copy processor in Beats. I do agree that it could be useful in some scenarios. Sending to a different transport is another good example.

[urso]

@anandsinghkunwar +1 for copy processor. A PR would be very welcome. Make sure you have signed the CLAs when contributing.

[kvch]

Added in: #11303