[BUG] Discrepancy between Profile API and realtime tasks API

[amitgalitz]

What is the bug?
There is discrepancy between the results returned from the profile API and search tasks API. This occurs when restarting an Opensearch cluster that already has a HCAD running.
How can one reproduce the bug?
Steps to reproduce the behavior:

1. Start Opensearch cluster and dashboards with AD plugin
2. create and wait for HCAD to be running and returning anomaly results
3. stop and start OpenSearch cluster
4. HCAD will go back to showing as initializing again on frontend
5. Profile API for detector will return response of running
6. Search request to ..._plugins/_anomaly_detection/detectors/tasks/_search with body of:

{
  "query": {
    "bool": {
      "filter": [
        {
          "term": {
            "detector_id": "N64oKYABE8ZUzJlQp33e"
          }
        },
        {
          "term": {
            "task_type": "REALTIME_HC_DETECTOR"
          }
        },
        {
          "term": {
            "is_latest": "true"
          }
        }
      ]
    }
  }
}

returns response with initializing

What is the expected behavior?
HCAD to show as running on frontend and all relevant API responses

What is your host/environment?

• Opensearch 2.0.0-alpha1-snapshot (4/14/22)

[ylwu-amzn]

For HC detector:

• Realtime task calculate state&init_progress with RCF total updates, check https://github.com/opensearch-project/anomaly-detection/blob/main/src/main/java/org/opensearch/ad/AnomalyDetectorJobRunner.java#L550-L556, If detector restarted, the RCF model cache will be cleaned, and detector will not execute until next job run. So the realtime task get 0 total updates and will show as INIT.

• Profile API will query AD result index to check if there are any result generated for detector to tell whether it passed initialization of not , check https://github.com/opensearch-project/anomaly-detection/blob/main/src/main/java/org/opensearch/ad/AnomalyDetectorProfileRunner.java#L451-L461, if detector passes initialization, it will generate some AD result and store in result index. When detector restarted, the AD result still in result index, so profile API can find some AD result and regard the state as RUNNING. But searching AD result index is quite heavy, that's why we made trade off to use realtime task state rather than use the profile.

[kaituo]

I experienced it myself without restarting a node. My case is when the data runs out the HCAD detector status change to init. Will debug about this.

[kaituo]

Another case is when I have long interval detector (e.g., 2hr). Hourly Cron cleared the model cache and we never update the state to running since the model cache is always empty. Real time task api needs to send a broadcast to get total updates from model cache. If model cache is always empty, real time task will deem the detector to be in initialization status. The detector will stay init forever on the frontend. But profile api returns running.

GET _plugins/_anomaly_detection/detectors/gQf0SX0Bof7RpY_a90aO/_profile/init_progress
{
"init_progress" : {
"percentage" : "100%"
}
}

Front end call shows init progress to 0%:

GET
http://ad-12-netwo-8c0fibvpdi7x-a8dcec46e0f0d9a6.elb.us-west-2.amazonaws.com:8443/api/anomaly_detectors/detectors/gQf0SX0Bof7RpY_a90aO
[9:47](https://amzn-aws.slack.com/archives/D01DJ386XN0/p1637689640002600)
response:
{“ok”:true,“response”:{“id”:“gQf0SX0Bof7RpY_a90aO”,“primaryTerm”:1,“seqNo”:3,“name”:“kaituo-hc-2hr”,“description”:“”,“timeField”:“@timestamp”,“indices”:[“host-cloudwatch”],“detectionInterval”:{“period”:{“interval”:120,“unit”:“Minutes”}},“windowDelay”:{“period”:{“interval”:1,“unit”:“Minutes”}},“shingleSize”:8,“schemaVersion”:0,“lastUpdateTime”:1637623330702,“categoryField”:[“host”],“user”:{“name”:“admin”,“backendRoles”:[“admin”],“roles”:[“own_index”,“all_access”],“customAttributeNames”:[],“userRequestedTenant”:null},“detectorType”:“MULTI_ENTITY”,“filterQuery”:{“match_all”:{“boost”:1}},“featureAttributes”:[{“featureId”:“gAf0SX0Bof7RpY_a90Z3”,“featureName”:“avg_cpu”,“featureEnabled”:true,“aggregationQuery”:{“avg_cpu”:{“avg”:{“field”:“cpu”}}}}],“uiMetadata”:{“features”:{“avg_cpu”:{“aggregationBy”:“avg”,“aggregationOf”:“cpu”,“featureType”:“simple_aggs”}},“filters”:[]},“enabled”:true,“enabledTime”:1637623330802,“curState”:“Initializing”,“stateError”:“”,“initProgress”:{“percentageStr”:“0%“,”estimatedMinutesLeft”:3840},“taskState”:“Stopped”,“taskError”:“”}}

[kaituo]

Proposed fix: to query the result index when the total updates is 0 like profile API. The con is extra query. Considering the query is to get 1 result instead of multiple results from result index. It is relatively cheap. It won't cause the frontend to wait longer as the query is initialized by the job scheduler run, not frontend. The frontend still reads states from task api

GET _plugins/_anomaly_detection/detectors/<detectorId>?task=true

Alternative considered:

1. ignore the discrepancy. The cons is that the longer detector interval, the longer cx needs to wait to see the detector become running again. Also, it won't solve the issue I mentioned.
2. remove hourly cron model cache reset. The cache reset clears unused memory and has been there since the first version of AD. We do not want to solve the state issue while ditching the resource management.
3. Once the state becomes running, write logic that prevents the state from going back to init again. The solution does not solve the case I mentioned.

[kaituo]

One scenario that is not covered in #753 is: what if there is no data left anymore? Without data, we won't trigger cold start and the detector will stay initialized until data appears again.