[BUG] Admin user can search system index registered via SystemIndexPlugin.getSystemIndexDescriptors

[kaituo]

What is the bug?
We added the .opensearch-forecast-state index to the system indices via this pull request by implementing the SystemIndexPlugin.getSystemIndexDescriptors method. I am unable to write to this system index using the admin user, but I can perform search queries. This behavior is inconsistent with other system indices added via an OpenSearch setting, where the admin user cannot write or search.

How can one reproduce the bug?
Steps to reproduce the behavior:

1. Download and start OpenSearch 3.0.0 container.

docker run -p 9200:9200 -d -p 9600:9600 -e "OPENSEARCH_INITIAL_ADMIN_PASSWORD=myStrongPassword123\!" -e "discovery.type=single-node" opensearch-ad:test

2. As the super admin, create the system indices .opensearch-forecast-state and .opendistro-anomaly-detectors, and add some content to them.

• .opensearch-forecast-state is registered as a system index via SystemIndexPlugin.getSystemIndexDescriptors.
• .opendistro-anomaly-detectors is registered as a system index via an OpenSearch setting.

3. The admin user cannot write to either index, which is the expected behavior.

(base) kaituo@88665a53bc93 github % curl -XPUT 'https://localhost:9200/.opensearch-forecast-state/_doc/1' -u admin:myStrongPassword123! --insecure -H 'Content-Type: application/json' -d'
{
  "title": "Document with ID 1",
  "content": "This document has a specific ID of 1.",
  "date": "2023-10-06"
}
'
{"error":{"root_cause":[{"type":"security_exception","reason":"no permissions for [] and User [name=admin, backend_roles=[admin], requestedTenant=null]"}],"type":"security_exception","reason":"no permissions for [] and User [name=admin, backend_roles=[admin], requestedTenant=null]"},"status":403}%
(base) kaituo@88665a53bc93 github % curl -XPUT 'https://localhost:9200/.opendistro-anomaly-detectors/_doc/1' -u admin:myStrongPassword123! --insecure -H 'Content-Type: application/json' -d'
{
  "title": "Document with ID 1",
  "content": "This document has a specific ID of 1.",
  "date": "2023-10-06"
}
'
{"error":{"root_cause":[{"type":"security_exception","reason":"no permissions for [] and User [name=admin, backend_roles=[admin], requestedTenant=null]"}],"type":"security_exception","reason":"no permissions for [] and User [name=admin, backend_roles=[admin], requestedTenant=null]"},"status":403}%

4. The admin user can query .opensearch-forecast-state but cannot query .opendistro-anomaly-detectors. This is inconsistent behavior.

(base) kaituo@88665a53bc93 github % curl -XGET 'https://localhost:9200/.opensearch-forecast-state/_search' -u admin:myStrongPassword123! --insecure -H 'Content-Type: application/json' -d'
{
"query": {
  "match_all": {}
},
"size": 10
}'
{"took":2,"timed_out":false,"_shards":{"total":1,"successful":1,"skipped":0,"failed":0},"hits":{"total":{"value":1,"relation":"eq"},"max_score":1.0,"hits":[{"_index":".opensearch-forecast-state","_id":"8FbKKpIBSvuCJXdtvH-J","_score":1.0,"_source":{"last_update_time":1727294782601,"started_by":"forecaster_full_user","state":"CREATED","task_progress":0.0,"init_progress":0.0,"execution_start_time":1727294782601,"is_latest":true,"task_type":"REALTIME_FORECAST_HC_FORECASTER","user":{"name":"forecaster_full_user","backend_roles":[],"roles":["own_index","forecast_full_access"],"custom_attribute_names":[],"user_requested_tenant":null},"forecaster_id":"7VbKKpIBSvuCJXdtu3-L","forecaster":{"name":"Second-Test-Forecaster-4","description":"ok rate","time_field":"timestamp","indices":["rule"],"filter_query":{"match_all":{"boost":1.0}},"window_delay":{"period":{"interval":211897,"unit":"Minutes"}},"shingle_size":8,"schema_version":2,"feature_attributes":[{"feature_id":"max1","feature_name":"max1","feature_enabled":true,"aggregation_query":{"max1":{"max":{"field":"visitCount"}}}}],"recency_emphasis":2560,"history":40,"ui_metadata":{"aabb":{"ab":"bb"}},"last_update_time":1727294782347,"category_field":["cityName"],"user":{"name":"forecaster_full_user","backend_roles":[],"roles":["own_index","forecast_full_access"],"custom_attribute_names":[],"user_requested_tenant":null},"forecast_interval":{"period":{"interval":10,"unit":"Minutes"}},"horizon":24}}}]}}%
(base) kaituo@88665a53bc93 github % curl -XGET 'https://localhost:9200/.opendistro-anomaly-detectors/_search' -u admin:myStrongPassword123! --insecure -H 'Content-Type: application/json' -d'
{
"query": {
  "match_all": {}
},
"size": 10
}'
{"took":1,"timed_out":false,"_shards":{"total":1,"successful":1,"skipped":0,"failed":0},"hits":{"total":{"value":0,"relation":"eq"},"max_score":null,"hits":[]}}%

5. The super admin can query .opendistro-anomaly-detectors

[opensearch@d267fcfcef53 ~]$ curl -XGET 'https://localhost:9200/.opendistro-anomaly-detectors/_search' -u admin:myStrongPassword123! --insecure --cacert /usr/share/opensearch/config/root-ca.pem --cert /usr/share/opensearch/config/kirk.pem --key /usr/share/opensearch/config/kirk-key.pem -H 'Content-Type: application/json' -d '
{
"query": {
  "match_all": {}
},
"size": 10
}'
{"took":4,"timed_out":false,"_shards":{"total":1,"successful":1,"skipped":0,"failed":0},"hits":{"total":{"value":1,"relation":"eq"},"max_score":1.0,"hits":[{"_index":".opendistro-anomaly-detectors","_id":"XFZjL5IBSvuCJXdtSYE0","_score":1.0,"_source":{"name":"Second-Test-Forecaster-4","description":"ok rate","time_field":"timestamp","indices":["rule"],"filter_query":{"match_all":{"boost":1.0}},"window_delay":{"period":{"interval":213182,"unit":"Minutes"}},"shingle_size":8,"schema_version":2,"feature_attributes":[{"feature_id":"max1","feature_name":"max1","feature_enabled":true,"aggregation_query":{"max1":{"max":{"field":"visitCount"}}}}],"recency_emphasis":2560,"history":40,"ui_metadata":{"aabb":{"ab":"bb"}},"last_update_time":1727371888948,"category_field":["cityName"],"user":{"name":"forecaster_full_user","backend_roles":[],"roles":["own_index","forecast_full_access"],"custom_attribute_names":[],"user_requested_tenant":null},"forecast_interval":{"period":{"interval":10,"un~]

What is the expected behavior?
The admin user should not be able to query the .opensearch-forecast-state system index, consistent with the behavior of other system indices added via OpenSearch settings.

Do you have any screenshots?
If applicable, add screenshots to help explain your problem.

Do you have any additional context?
opensearch-project/OpenSearch#14415
#4471
opensearch-project/anomaly-detection#1251