[ML-DataFrame] add a stats endpoint

[hendrikmuhs]

Feature Branch PR

add a stats endpoint for data frame jobs:

GET _xpack/feature_index_builder/jobs/_all/_stats

(instead of _all you can specify an exact/explicit job id)
example output:

{
  "count" : 2,
  "jobs" : [
    {
      "id" : "pivot_reviews_4",
      "state" : {
        "job_state" : "stopped"
      },
      "stats" : {
        "pages_processed" : 0,
        "documents_processed" : 0,
        "documents_indexed" : 0,
        "trigger_count" : 0
      }
    },
    {
      "id" : "pivot_reviews_5",
      "state" : {
        "job_state" : "indexing",
        "current_position" : {
          "business" : "a_xdfGzrJ1mx3mBx6m14Ebhw",
          "reviewer" : "a_U2dPmHaUtlcjDhJjX7oN3w"
        }
      },
      "stats" : {
        "pages_processed" : 330,
        "documents_processed" : 330001,
        "documents_indexed" : 329000,
        "trigger_count" : 2
      }
    }
  ]
}

Notes:

• the path (_xpack/feature...) will be renamed in a separate PR.
• the output for state will be changed to make it more user-friendly in an upcoming PR

[elasticmachine]

Pinging @elastic/ml-core

[dimitris-athanasiou]

It seems all 3 fields should not be null. Add Objects.requireNotNull checks? (Btw, just realised the main reason I like those is the way they serve as documentation)

[dimitris-athanasiou]

Also init jobsStateAndStats to empty list?

[dimitris-athanasiou]

You should be able to simply do builder.field(JOBS.getPreferredName(), jobsStateAndStats) here. Unless we need to use Params it should be fine.

[dimitris-athanasiou]

This might be simpler if instead of declaring the return value we call the listener once for each branch directly like:

listener.onResponse(CollectionsSingletonList(jobStateAndStats)

etc. Just a suggestion, I know this is quite subjective :-)

[dimitris-athanasiou]

I think you should be able to squash this comment in 2 lines

[hendrikmuhs]

Eclipse formatting drives me crazy! -> #35984

[dimitris-athanasiou]

For anomaly detection, the equivalent action is called GetJobStatsAction where job is singular. We might want to be consistent but I don't feel strongly about it.

[hendrikmuhs]

Discussed this with @droberts195 on a previous PR and we went all for plural, all the endpoint have changed to .../jobs/{id}/... GetDataFrameJob changed to GetDataFrameJobs, etc.

Anyway, it's a good point and I suggest to keep it as is for now but have a session about naming where we can go over all endpoints and then do a renaming PR.

[droberts195]

The ML actions are inconsistent in this respect: we have GetJobsStatsAction and GetDatafeedsStatsAction but GetJobStatsActionRequest and RestGetJobStatsAction. I think plural is correct.

[hendrikmuhs]

addressed the review comments.

As #35471 has been merged yesterday, the stats API is going to break with the next merge, I will therefore do a master merge now and add fixes as part of this PR.

[hendrikmuhs]

@droberts195 @dimitris-athanasiou Thanks for your comments, this PR should be ready now.

[droberts195]

LGTM

[dimitris-athanasiou]

LGTM

[dimitris-athanasiou]

Oh, just realised this is missing the REST endpoint specification which also enables writing YAML tests.

[hendrikmuhs]

@dimitris-athanasiou Yes, you are right. This is true for all endpoints, tracked in the team repro issue number 7.