Compatibility with Elastic ECS?

[nick-george]

Hi,

I'm using v3.2.3 of this filter and I can see that it's producing fields that are not in the ECS. An example of its output is:

   "user_agent": {
         "build": "",
         "device": "Other",
         "major": "1",
         "minor": "11",
         "name": "aws-sdk-java",
         "original": "[S3Console/0.4, aws-internal/3 aws-sdk-java/1.11.488 Linux/4.9.137-0.1.ac.218.74.329.metal1.x86_64 OpenJDK_64-Bit_Server_VM/25.192-b12 java/1.8.0_192]",
         "os": "Linux",
         "os_major": "4",
         "os_minor": "9",
         "os_name": "Linux",
         "patch": "488"
     }

Yet the fields specified for user_agent in the ECS are here (https://github.com/elastic/ecs#user_agent)

Will this filter be updated to comply with the ECS? Or will someone be updating the ECS to be compatible with the output of this filter?

Many thanks,
Nick

[ba0708]

Looking for an answer to this as well. For now, I suppose I will have to manually reformat the data within Logstash, which is not so convenient. The release of Logstash 7.0.0 seemed like a good time to break the backwards compatibility, as there were already numerous BC breaking changes in relation to ECS. FWIW, the user agent ingest processor for Elasticsearch has been migrated to using ECS.

[ba0708]

It looks like this will not be resolved soon (and understandably so). I went ahead and implemented a workaround in my pipeline until then. It is arguably not the prettiest and can probably be done simpler, but I needed to get something working quickly (inspired by the Elasticsearch ingest pipeline). I hope it helps someone.

useragent {
  source => "[user_agent][original]"
  target => "ua_tmp"

  add_field => {
    "[user_agent][device][name]" => "%{[ua_tmp][device]}"
    "[user_agent][os][name]" => "%{[ua_tmp][os_name]}"
    "[user_agent][name]" => "%{[ua_tmp][name]}"
  }
}

# OS version ECS compatibility
if [ua_tmp][os_major] {
  mutate {
    add_field => {
      "[user_agent][os][version]" => "%{[ua_tmp][os_major]}"
    }
  }

  if [ua_tmp][os_minor] {
    mutate {
      replace => {
        "[user_agent][os][version]" => "%{[user_agent][os][version]}.%{[ua_tmp][os_minor]}"
      }
    }

    if [ua_tmp][os_patch] {
      mutate {
        replace => {
          "[user_agent][os][version]" => "%{[user_agent][os][version]}.%{[ua_tmp][os_patch]}"
        }
      }

      if [ua_tmp][os_build] {
        mutate {
          replace => {
            "[user_agent][os][version]" => "%{[user_agent][os][version]}.%{[ua_tmp][os_build]}"
          }
        }
      }
    }
  }

  mutate {
    add_field => {
      "[user_agent][os][full]" => "%{[user_agent][os][name]} %{[user_agent][os][version]}"
    }
  }
}

# User agent version ECS compatibility
if [ua_tmp][major] {
  mutate {
    add_field => {
      "[user_agent][version]" => "%{[ua_tmp][major]}"
    }
  }

  if [ua_tmp][minor] {
    mutate {
      replace => {
        "[user_agent][version]" => "%{[user_agent][version]}.%{[ua_tmp][minor]}"
      }
    }

    if [ua_tmp][patch] {
      mutate {
        replace => {
          "[user_agent][version]" => "%{[user_agent][version]}.%{[ua_tmp][patch]}"
        }
      }

      if [ua_tmp][build] {
        mutate {
          replace => {
            "[user_agent][version]" => "%{[user_agent][version]}.%{[ua_tmp][build]}"
          }
        }
      }
    }
  }
}

mutate {
  remove_field => ["ua_tmp"]
}

EDIT: Updated the workaround to include user_agent.name (thanks @Reamer).

[Reamer]

@ba0708 Thanks for your workaround. Added this additional line to your workaround. The name field includes the browser name (e.g. Chrome, Vivaldi, Java)

...
    useragent {
      source => "[user_agent][original]"
      target => "ua_tmp"
      add_field => {
        "[user_agent][device][name]" => "%{[ua_tmp][device]}"
        "[user_agent][os][name]" => "%{[ua_tmp][os_name]}"
->      "[user_agent][name]" => "%{[ua_tmp][name]}"
      }
    }
...

[kares]

ECS support shipped in #68 with plugin version 3.3.0 (see also #66)