Skip to content

Add ECS schema for user-agent ingest processor (#37727)#37984

Merged
dakrone merged 4 commits intoelastic:masterfrom
dakrone:ingest-user-agent-ecs
Jan 30, 2019
Merged

Add ECS schema for user-agent ingest processor (#37727)#37984
dakrone merged 4 commits intoelastic:masterfrom
dakrone:ingest-user-agent-ecs

Conversation

@dakrone
Copy link
Copy Markdown
Member

@dakrone dakrone commented Jan 29, 2019

(This is a forward-port with changes of #37727)

This switches the format of the user agent processor to use the schema from ECS.
So rather than something like this:

{
  "patch" : "3538",
  "major" : "70",
  "minor" : "0",
  "os" : "Mac OS X 10.14.1",
  "os_minor" : "14",
  "os_major" : "10",
  "name" : "Chrome",
  "os_name" : "Mac OS X",
  "device" : "Other"
}

The structure is now like this:

{
  "name" : "Chrome",
  "original" : "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36",
  "os" : {
    "name" : "Mac OS X",
    "version" : "10.14.1",
    "full" : "Mac OS X 10.14.1"
  },
  "device" : "Other",
  "version" : "70.0.3538.102"
}

This is now the default for 7.0. The deprecated ecs setting in 6.x is not
supported.

Resolves #37329

@dakrone
Add ECS schema for user-agent ingest processor (elastic#37727)
55adc2a 
This switches the format of the user agent processor to use the schema from [ECS](https://github.com/elastic/ecs).
So rather than something like this:

```
{
  "patch" : "3538",
  "major" : "70",
  "minor" : "0",
  "os" : "Mac OS X 10.14.1",
  "os_minor" : "14",
  "os_major" : "10",
  "name" : "Chrome",
  "os_name" : "Mac OS X",
  "device" : "Other"
}
```

The structure is now like this:

```
{
  "name" : "Chrome",
  "original" : "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36",
  "os" : {
    "name" : "Mac OS X",
    "version" : "10.14.1",
    "full" : "Mac OS X 10.14.1"
  },
  "device" : "Other",
  "version" : "70.0.3538.102"
}
```

This is now the default for 7.0. The deprecated `ecs` setting in 6.x is not
supported.

Resolves elastic#37329
@dakrone dakrone added >breaking :Distributed/Ingest Node Execution or management of Ingest Pipelines v7.0.0 labels Jan 29, 2019
@elasticmachine
Copy link
Copy Markdown
Collaborator

elasticmachine commented Jan 29, 2019

Pinging @elastic/es-core-features

@dakrone dakrone mentioned this pull request Jan 29, 2019
Closed
@JoshMock
Copy link
Copy Markdown
Member

JoshMock commented Jan 29, 2019

jenkins retest this please

@jakelandis jakelandis self-requested a review January 30, 2019 16:37
jakelandis
jakelandis approved these changes Jan 30, 2019
View reviewed changes
Copy link
Copy Markdown
Contributor

@jakelandis jakelandis left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dakrone dakrone merged commit cac6b8e into elastic:master Jan 30, 2019
jasontedor added a commit to jasontedor/elasticsearch that referenced this pull request Jan 30, 2019
@jasontedor
Merge branch 'master' into replication-tracker-primary-term
0895a5c 
* master:
  Remove types from watcher docs (elastic#38002)
  Add test coverage for Painless general casting of boolean and Boolean (elastic#37780)
  Fixed test bug, lastFollowTime is null if there are no follower indices.
  Add ECS schema for user-agent ingest processor (elastic#37727) (elastic#37984)
  Extract TransportRequestDeduplication from ShardStateAction (elastic#37870)
  Expose retention leases in shard stats (elastic#37991)
@webmat webmat mentioned this pull request Jan 30, 2019
Merged
5 tasks
jasontedor added a commit to dnhatn/elasticsearch that referenced this pull request Jan 31, 2019
@jasontedor
Merge remote-tracking branch 'elastic/master' into pr/37940
aa02208 
* elastic/master:
  ILM setPriority corrections for a 0 value (elastic#38001)
  Temporarily disable BWC for retention lease stats (elastic#38049)
  Skip Shrink when numberOfShards not changed (elastic#37953)
  Add dispatching to `HandledTransportAction` (elastic#38050)
  Update httpclient for JDK 11 TLS engine (elastic#37994)
  Reduce flaxiness of ccr recovery timeouts test (elastic#38035)
  Fix ILM status to allow unknown fields (elastic#38043)
  Fix ILM Lifecycle Policy to allow unknown fields (elastic#38041)
  Update verify repository to allow unknown fields (elastic#37619)
  [ML] Datafeed deprecation checks (elastic#38026)
  Deprecate minimum_master_nodes (elastic#37868)
  Remove types from watcher docs (elastic#38002)
  Add test coverage for Painless general casting of boolean and Boolean (elastic#37780)
  Fixed test bug, lastFollowTime is null if there are no follower indices.
  Add ECS schema for user-agent ingest processor (elastic#37727) (elastic#37984)
  Extract TransportRequestDeduplication from ShardStateAction (elastic#37870)
@simitt
Copy link
Copy Markdown
Contributor

simitt commented Jan 31, 2019

I just tried to use the changed user-agent ingest processor and realized that the device implementation differs from the ECS definition:
device is implemented as keyword but in ECS defined as object with a field name that is a keyword, see https://github.com/elastic/ecs#-user-agent-fields.

@webmat
Copy link
Copy Markdown

webmat commented Jan 31, 2019 •
edited
Loading

Oh jeez, I made the mistake in field naming, in my initial issue 🤦‍♂️Sorry about that!

The nesting of device.name is there to allow for other fields about the device eventually. Like device.category (tablet, mobile, etc).

So the nesting is kind of important, even if there's nothing else under device. yet.

@webmat
Copy link
Copy Markdown

webmat commented Jan 31, 2019

I created issue #38094 for this

@dakrone dakrone deleted the ingest-user-agent-ecs branch February 4, 2019 14:42
jakelandis added a commit to jakelandis/elasticsearch that referenced this pull request Feb 12, 2019
@jakelandis
Revert "Add ECS schema for user-agent ingest processor (elastic#37727) (
3bd64f4 
elastic#37984)"

This reverts commit cac6b8e.

Related: elastic/beats#10650

Will replace this commit with the 6.7 version
jakelandis added a commit that referenced this pull request Feb 13, 2019
@jakelandis
Make 7.0 like 6.7 user agent ecs, but default to true (#38757)
5d8a980 
This change reverts the initial 7.0 commits and replaces them
with the 6.7 variant that still allows for the ecs flag. 
This commit differs from the 6.7 variants in that ecs flag will 
now default to true. 

6.7: `ecs` : default `false`
7.x: `ecs` : default `true`
8.0: no option, but behaves as `true`

* Revert "Ingest node - user agent, move device to an object (#38115)"
This reverts commit 5b008a3.

* Revert "Add ECS schema for user-agent ingest processor (#37727) (#37984)"
This reverts commit cac6b8e.

* cherry-pick 5dfe193 
Add ECS schema for user-agent ingest processor (#37727)

* cherry-pick ec8ddc8 
Ingest node - user agent, move device to an object (#38115) (#38121)
  
* cherry-pick f63cbdb (with manual merge fixes)
Dep. check for ECS changes to User Agent processor (#38362)

* make true the default for the ecs option, and update 7.0 references and tests
jakelandis added a commit to jakelandis/elasticsearch that referenced this pull request Feb 13, 2019
@jakelandis
Make 7.0 like 6.7 user agent ecs, but default to true (elastic#38757)
08499fa 
This change reverts the initial 7.0 commits and replaces them
with the 6.7 variant that still allows for the ecs flag. 
This commit differs from the 6.7 variants in that ecs flag will 
now default to true. 

6.7: `ecs` : default `false`
7.x: `ecs` : default `true`
8.0: no option, but behaves as `true`

* Revert "Ingest node - user agent, move device to an object (elastic#38115)"
This reverts commit 5b008a3.

* Revert "Add ECS schema for user-agent ingest processor (elastic#37727) (elastic#37984)"
This reverts commit cac6b8e.

* cherry-pick 5dfe193 
Add ECS schema for user-agent ingest processor (elastic#37727)

* cherry-pick ec8ddc8 
Ingest node - user agent, move device to an object (elastic#38115) (elastic#38121)
  
* cherry-pick f63cbdb (with manual merge fixes)
Dep. check for ECS changes to User Agent processor (elastic#38362)

* make true the default for the ecs option, and update 7.0 references and tests
@jakelandis jakelandis mentioned this pull request Feb 13, 2019
Merged
jakelandis added a commit that referenced this pull request Feb 13, 2019
@jakelandis
Make 7.x like 6.7 user agent ecs, but default to true (#38828)
46bb663 
Forward port of #38757

This change reverts the initial 7.0 commits and replaces them
with the 6.7 variant that still allows for the ecs flag. 
This commit differs from the 6.7 variants in that ecs flag will 
now default to true. 

6.7: `ecs` : default `false`
7.x: `ecs` : default `true`
8.0: no option, but behaves as `true`

* Revert "Ingest node - user agent, move device to an object (#38115)"
This reverts commit 5b008a3.

* Revert "Add ECS schema for user-agent ingest processor (#37727) (#37984)"
This reverts commit cac6b8e.

* cherry-pick 5dfe193 
Add ECS schema for user-agent ingest processor (#37727)

* cherry-pick ec8ddc8 
Ingest node - user agent, move device to an object (#38115) (#38121)
  
* cherry-pick f63cbdb (with manual merge fixes)
Dep. check for ECS changes to User Agent processor (#38362)

* make true the default for the ecs option, and update 7.0 references and tests
@ba0708 ba0708 mentioned this pull request May 3, 2019
Closed
@joegallo joegallo mentioned this pull request Oct 15, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@jakelandis jakelandis jakelandis approved these changes

Assignees

No one assigned

Labels

blocker >breaking :Distributed/Ingest Node Execution or management of Ingest Pipelines v7.0.0-beta1

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

7 participants

@dakrone @elasticmachine @JoshMock @simitt @webmat @jakelandis @colings86