[BUG] <title>Cross-cluster search that includes self no longer works 2.16-2.19

[kmfreder1]

Describe the bug

We have an Opensearch 2.16.0 installation from where we set up cross cluster search to a handful of other Opensearch 2.16.0 systems. Pre-opensearch 2.16.0, we could create a *:index-name* pattern and view the events of our other Opensearch clusters along with the events from the Opensearch cluster we were searching from. We recently upgraded from Opensearch 2.8.0 where was being done.

As an a trouble-shooting step, I installed a separate Opensearch 2.19 and set up the installation as a cross-cluster search to itself. When I attempted to make an index pattern like remote_host:index-name*, it failed the same as with Opensearch 2.16.0 even though we were doing this successfully with Opensearch <= 2.8.0.

Related component

Search:Remote Search

To Reproduce

1. Go to 'Dev Tools' and set up the local host as a remote cluster by entering the following:

PUT _cluster/settings
{
    "persistent": {
        "cluster.remote": { 
            "my_remote": {
                "seeds": ["localhost:9300"],
                "skip_unavailable": true
            }
        }
    }
}

2. Check that the remote cluster connected by entering the following:
   GET /_remote/info
   The output should show "connected": true as below

{
  "my_remote": {
    "connected": true,
    "mode": "sniff",
    "seeds": [
      "localhost:9300"
    ],
    "num_nodes_connected": 1,
    "max_connections_per_cluster": 3,
    "initial_connect_timeout": "30s",
    "skip_unavailable": true
  }
}

For this test, there should be no other remote cluster set up but the local cluster.
3. Go to Management > Dashboard Management > Index Patterns and click on the Create index pattern button.

4. In the index pattern name field, enter the name of the remote followed by a known index name pattern separated by a colon:my_remote:security-auditlog*

5. See error
   

Expected behavior

The expected behavior is to be able to review events from the local SIEM that has been set up as a remote cluster along with the events from other remote clusters as we were doing in <= Opensearch 2.8.

Additional Details

Plugins
Please list all plugins currently enabled.

/usr/share/opensearch/bin/opensearch-plugin list

opensearch-alerting
opensearch-anomaly-detection
opensearch-asynchronous-search
opensearch-cross-cluster-replication
opensearch-custom-codecs
opensearch-flow-framework
opensearch-geospatial
opensearch-index-management
opensearch-job-scheduler
opensearch-knn
opensearch-ltr
opensearch-ml
opensearch-neural-search
opensearch-notifications
opensearch-notifications-core
opensearch-observability
opensearch-performance-analyzer
opensearch-reports-scheduler
opensearch-security
opensearch-security-analytics
opensearch-skills
opensearch-sql
opensearch-system-templates
query-insights

Screenshots
If applicable, add screenshots to help explain your problem.

This is what happens on 2.8 when I attempt to set up an index pattern to the local "remote-cluster":

This is what happens in 2.19 (and 2.16) wen I do the same:

Host/Environment (please complete the following information):

• OS: Ubuntu
• Version 22.04

[kkewwei]

@kmfreder1 Can you help check is the search failed/timeout or the bug? When you clieck Create index pattern, it will send a very time consuming query(opensearch-project/OpenSearch-Dashboards#8596), maybe it will time out?

I I encountered a similar issue (opensearch-project/OpenSearch-Dashboards#8596, opensearch-project/index-management-dashboards-plugin#1178), this is currently a bug and no reply.

[kmfreder1]

@kkewwei No, it does not timeout. It immediately shows the error above when I type in the above remote cluster name and index pattern.

[HenryTheSir]

Hi,

this should solve your issue
https://forum.opensearch.org/t/ccs-query-to-same-cluster-is-not-working/17995/2

best regards

[kmfreder1]

@HenryTheSir That is a great help! Thanks for sharing.

[sandeshkr419]

[Search Triage]

@HenryTheSir Thanks for sharing the resource.
If this is not an issue anymore, should we close this out?

[kmfreder1]

I think what @HenryTheSir has described is a work-around for the apparent bug and change in default behavior.

[kmfreder1]

Unfortunately, the work-around does not appear to be working for us. This does not appear to be resolved, or we are doing something wrong when we try to apply the work-around mentioned by @HenryTheSir. We added this line to our opensearch.yml file:

plugins.security.unsupported.passive_intertransport_auth_initially: true

Step 2 was a little unclear, but we did what we believe it was suggesting.

1. We added a user called opendistro_security_default_transport_user. (We chose that user.name because we found it referenced in the logs)
2. We gave that user the backend role of opendistro_security_default_transport_backendrole.
3. We created a role called ccs_to_self_role with the indices:data/read/search index permissions and mapped it to the above user.
   But, then, while it looks like it is going to allow us to create an index pattern, it never brings up the date field to chose from and when I attempt to use the already created *:index index pattern, it just spins forever and never brings up the data, whether I am in a Discover window or in a custom dashboard.

This used to work fine in 2.8 prior to our upgrade. Here are some screenshots of what we have now:

When I attempt to make an index pattern, it finds the indices:

However, on the next page, it never finds the date field and just hangs:

In the Discover windows looking at the *:index index pattern after quite a while of spinning, it ends here:

I am not seeing any indicators in the log as to what the problem might be or why it hangs and ends with an empty set, presumably due to timing out.

Any further help would be much appreciated. We rely on this pretty heavily.

[kmfreder1]

This continues to be a significant issue for us. Are there other things we could try? Is this on a roadmap to be addressed?

The documented solution here:
https://forum.opensearch.org/t/ccs-query-to-same-cluster-is-not-working/17995/2

did not work. Please advise.