Skip to content

[APM] Service maps - Fix missing ML status for services with jobs but no anomalies#68486

Merged
ogupte merged 7 commits intoelastic:masterfrom
ogupte:apm-68485-fix-missing-service-ml-health-indicator
Jun 10, 2020
Merged

[APM] Service maps - Fix missing ML status for services with jobs but no anomalies#68486
ogupte merged 7 commits intoelastic:masterfrom
ogupte:apm-68485-fix-missing-service-ml-health-indicator

Conversation

@ogupte
Copy link
Copy Markdown
Contributor

@ogupte ogupte commented Jun 8, 2020 •
edited
Loading

Closes #68485 by:

  • use the ML plugin to query for all APM jobs
  • inspect the ml job groups to find all jobs related to a particular service
  • use the mlAnomalySearch client to get ml job buckets with the max anomaly score & actual/median values for the ML description
  • return the relevant ML job with the max anomaly score for a service
  • indicate to the user that no anomalies were found for a service with an ml job
    Screen Shot 2020-06-09 at 9 12 49 AM

@ogupte
Closes elastic#68485 by:
acc3db6 
  - use the ML plugin to query for all APM jobs
  - inspect the ml job groups to find all jobs related to a particular service
  - use the mlAnomalySearch client to get ml job buckets with the max anomaly score
  - query for the model_plot buckets to obtain actual/median values for the ML description
  - return the relevant ML job with the max anomaly score for a service
  - indicate to the user that no anomalies were found for a service with an ml job
@ogupte ogupte added release_note:skip Skip the PR/issue when compiling release notes v7.9.0 v7.8 labels Jun 8, 2020
@ogupte ogupte requested a review from a team June 8, 2020 03:56
@ogupte ogupte requested a review from a team as a code owner June 8, 2020 03:56
@botelastic botelastic bot added the Team:APM - DEPRECATED Use Team:obs-ux-infra_services. label Jun 8, 2020
@elasticmachine
Copy link
Copy Markdown
Contributor

elasticmachine commented Jun 8, 2020

Pinging @elastic/apm-ui (Team:apm)

jgowdyelastic
jgowdyelastic reviewed Jun 8, 2020
View reviewed changes
formgeist
formgeist reviewed Jun 8, 2020
View reviewed changes
Copy link
Copy Markdown
Contributor

@formgeist formgeist left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a suggestion on the empty state copy

@ogupte
- Use the anomalyDetectorsProvider jobs API rather than the search en…
08b94cf 
…dpoint directly

- Defines a specific return type for the ml jobs api
- Update the empty anomaly data message
@ogupte ogupte requested a review from jgowdyelastic June 8, 2020 16:10
sorenlouv
sorenlouv reviewed Jun 8, 2020
View reviewed changes
jgowdyelastic
jgowdyelastic approved these changes Jun 8, 2020
View reviewed changes
Copy link
Copy Markdown
Member

@jgowdyelastic jgowdyelastic left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ML changes LGTM

sorenlouv
sorenlouv reviewed Jun 8, 2020
View reviewed changes
sorenlouv
sorenlouv reviewed Jun 8, 2020
View reviewed changes
sorenlouv
sorenlouv reviewed Jun 8, 2020
View reviewed changes
ogupte added 3 commits June 8, 2020 12:39
@ogupte
Code and types cleanup
bd1ca55
@ogupte
Return to using record result type on anomaly queries. These are the
106d0f2 
same values used in the anomaly explorer and it includes actual &
typical values which greatly improve performance of the previous query.
@ogupte
- If anomaly data is missing show a gray border around node
143c0a8
smith
smith approved these changes Jun 9, 2020
View reviewed changes
Copy link
Copy Markdown
Contributor

@smith smith left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A few nits in the comments, but works well for me. Thanks!

@ogupte
Copy link
Copy Markdown
Contributor Author

ogupte commented Jun 9, 2020

@elasticmachine merge upstream

@rayafratkina rayafratkina added v7.8.0 and removed v7.8 labels Jun 9, 2020
@ogupte
Copy link
Copy Markdown
Contributor Author

ogupte commented Jun 9, 2020

retest

@kibanamachine
Copy link
Copy Markdown
Contributor

kibanamachine commented Jun 10, 2020

💚 Build Succeeded

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

@ogupte ogupte merged commit cac9018 into elastic:master Jun 10, 2020
@ogupte ogupte mentioned this pull request Jun 10, 2020
Merged
ogupte added a commit to ogupte/kibana that referenced this pull request Jun 10, 2020
@ogupte @elasticmachine
[APM] Service maps - Fix missing ML status for services with jobs but…
56a37be 
… no anomalies (elastic#68486)

* Closes elastic#68485 by:
  - use the ML plugin to query for all APM jobs
  - inspect the ml job groups to find all jobs related to a particular service
  - use the mlAnomalySearch client to get ml job buckets with the max anomaly score
  - query for the model_plot buckets to obtain actual/median values for the ML description
  - return the relevant ML job with the max anomaly score for a service
  - indicate to the user that no anomalies were found for a service with an ml job

* - Use the anomalyDetectorsProvider jobs API rather than the search endpoint directly
- Defines a specific return type for the ml jobs api
- Update the empty anomaly data message

* Code and types cleanup

* Return to using record result type on anomaly queries. These are the
same values used in the anomaly explorer and it includes actual &
typical values which greatly improve performance of the previous query.

* - If anomaly data is missing show a gray border around node

* - moved AnomalyDetection out of service map Contents into own component

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
ogupte added a commit to ogupte/kibana that referenced this pull request Jun 10, 2020
@ogupte @elasticmachine
[APM] Service maps - Fix missing ML status for services with jobs but…
23f75d0 
… no anomalies (elastic#68486)

* Closes elastic#68485 by:
  - use the ML plugin to query for all APM jobs
  - inspect the ml job groups to find all jobs related to a particular service
  - use the mlAnomalySearch client to get ml job buckets with the max anomaly score
  - query for the model_plot buckets to obtain actual/median values for the ML description
  - return the relevant ML job with the max anomaly score for a service
  - indicate to the user that no anomalies were found for a service with an ml job

* - Use the anomalyDetectorsProvider jobs API rather than the search endpoint directly
- Defines a specific return type for the ml jobs api
- Update the empty anomaly data message

* Code and types cleanup

* Return to using record result type on anomaly queries. These are the
same values used in the anomaly explorer and it includes actual &
typical values which greatly improve performance of the previous query.

* - If anomaly data is missing show a gray border around node

* - moved AnomalyDetection out of service map Contents into own component

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
# Conflicts:
#	x-pack/plugins/ml/server/shared.ts
@ogupte ogupte mentioned this pull request Jun 10, 2020
Merged
ogupte added a commit that referenced this pull request Jun 10, 2020
@ogupte @elasticmachine
[APM] Service maps - Fix missing ML status for services with jobs but…
8d212ea 
… no anomalies (#68486) (#68734)

* Closes #68485 by:
  - use the ML plugin to query for all APM jobs
  - inspect the ml job groups to find all jobs related to a particular service
  - use the mlAnomalySearch client to get ml job buckets with the max anomaly score
  - query for the model_plot buckets to obtain actual/median values for the ML description
  - return the relevant ML job with the max anomaly score for a service
  - indicate to the user that no anomalies were found for a service with an ml job

* - Use the anomalyDetectorsProvider jobs API rather than the search endpoint directly
- Defines a specific return type for the ml jobs api
- Update the empty anomaly data message

* Code and types cleanup

* Return to using record result type on anomaly queries. These are the
same values used in the anomaly explorer and it includes actual &
typical values which greatly improve performance of the previous query.

* - If anomaly data is missing show a gray border around node

* - moved AnomalyDetection out of service map Contents into own component

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
gmmorris added a commit to gmmorris/kibana that referenced this pull request Jun 10, 2020
@gmmorris
Merge branch 'master' into alerting/consumer-based-rbac
86f7d73 
* master: (22 commits)
  Partial revert of "Sync Kerberos + Anonymous access tests with the latest `security/_authenticate` API (user roles now include roles of anonymous user)." (elastic#68624)
  adapt some snapshot test (elastic#68489)
  [APM] Service maps - Fix missing ML status for services with jobs but no anomalies (elastic#68486)
  [skip test] apis Kerberos security Kerberos authentication finishing SPNEGO should properly set cookie and authenticate user
  [SIEM][Exceptions] - ExceptionsViewer UI component part 2 (elastic#68294)
  Surface data streams in Index Management. (elastic#67806)
  Fix edit datasource not working following changes in elastic#67234 (elastic#68583)
  [Logs + Metrics UI] Clean up async plugin initialization (elastic#67654)
  APM Storybook fixes (elastic#68671)
  Upgrade EUI to v24.1.0 (elastic#68141)
  [ML] DF Analytics: Creation wizard part 2 (elastic#68462)
  [Uptime] Fix race on overview page query (elastic#67843)
  Prefer using npm_execpath when spawning Yarn (elastic#68673)
  [Security] [Cases] Attach timeline to existing case (elastic#68580)
  Use Search API in Vega (elastic#68257)
  [Component templates] Table view (elastic#68031)
  [Uptime] Added relative date info in cert status column (elastic#67612)
  [Endpoint] Re-enable Functional test case for Endpoint related pages (elastic#68445)
  run page_load_metrics tests in visual regresssion jobs (elastic#68570)
  Enable exhaustive-deps; correct any lint warnings (elastic#68453)
  ...
ogupte added a commit that referenced this pull request Jun 10, 2020
@ogupte @elasticmachine
[7.8] [APM] Service maps - Fix missing ML status for services with jo…
2dc4c8b 
…bs but no anomalies (#68486) (#68735)

* [APM] Service maps - Fix missing ML status for services with jobs but no anomalies (#68486)

* Closes #68485 by:
  - use the ML plugin to query for all APM jobs
  - inspect the ml job groups to find all jobs related to a particular service
  - use the mlAnomalySearch client to get ml job buckets with the max anomaly score
  - query for the model_plot buckets to obtain actual/median values for the ML description
  - return the relevant ML job with the max anomaly score for a service
  - indicate to the user that no anomalies were found for a service with an ml job

* - Use the anomalyDetectorsProvider jobs API rather than the search endpoint directly
- Defines a specific return type for the ml jobs api
- Update the empty anomaly data message

* Code and types cleanup

* Return to using record result type on anomaly queries. These are the
same values used in the anomaly explorer and it includes actual &
typical values which greatly improve performance of the previous query.

* - If anomaly data is missing show a gray border around node

* - moved AnomalyDetection out of service map Contents into own component

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
# Conflicts:
#	x-pack/plugins/ml/server/shared.ts

* - exposes ML types to be referenced in APM
- uses existing mlSearch API rather than the mlAnomalySearch API

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@formgeist formgeist formgeist left review comments

@bmorelli25 bmorelli25 bmorelli25 left review comments

@smith smith smith approved these changes

@jgowdyelastic jgowdyelastic jgowdyelastic approved these changes

+1 more reviewer

@sorenlouv sorenlouv sorenlouv left review comments

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

release_note:skip Skip the PR/issue when compiling release notes Team:APM - DEPRECATED Use Team:obs-ux-infra_services. v7.8.0 v7.9.0

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[APM] Service map bug doesn't always display ML health indicators

9 participants

@ogupte @elasticmachine @kibanamachine @smith @sorenlouv @formgeist @bmorelli25 @jgowdyelastic @rayafratkina