Should search preference _primary take primary relocation target into account?

[ywelsch]

When using the search preference _primary, and executing a search on a cluster where a concerned primary is about to complete relocation, it's possible that the search for that primary will fail, as it will hit the primary relocation source just when the relocation is completed and the source shard is shut down. My question is:

Should the search preference _primary take the primary relocation target into account?
Put differently, should OperationRouting.preferenceActiveShardIterator(...) return the primary relocation target as backup when specifying _primary preference?

Note that the default search preference always takes relocation targets into account (and puts them last together with the "regular" initializing shards).

Related question:

Should the search preference _replica also take replica relocation targets into account?

More background to this:

There was a test failure of SearchWhileCreatingIndexIT which illustrates the issue:

https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+g1gc/4012/consoleFull

[bleskes]

I'm struggling to find a use case for _primary and _replica. Should we deprecate and remove them?

[dakrone]

If I recall correctly, _replica was originally added back in the day for shadow replicas (see: #12244). Which have been removed, so +1 on removing _replica and _replica_first.

_primary was useful back in the days of asynchronous replication, since your document may not have reached the replicas yet. We don't allow it any more though, so it's probably not strictly necessary.

However, having _primary and _replica can be nice for debugging things, for instance, if you suspect that a copy of the data is somehow misbehaving, or you want to compare an analyzer on one node with a (perhaps misconfigured) analyzer on a second node, it can be very helpful to be able to use _primary and _replica to route the request to the desired node, so overall I'm probably +0.5 on keeping them.

[bleskes]

@dakrone thanks for filling in the context and history. I should have done it in the first place.

However, having _primary and _replica can be nice for debugging things

I think this use case is better served by the _prefer_nodes and _only_nodes options. It works for primaries (with some extra effort of figuring out where the primary is) and is actually needed for replicas when people run with >1 replicas.

[dakrone]

I think this use case is better served by the _prefer_nodes and _only_nodes options.

Those work also, so +1 to removing the others if no one else objects. Perhaps the title of this issues should be changed to reflect the discussion?

[liketic]

From what I understand, we only need to keep the two preference _prefer_nodes and _only_nodes and remove all others? Thanks!

[dakrone]

@liketic there are others that should still be kept, I believe the consensus was just to remove _primary and _replica

[bleskes]

@dakrone Do we have a valid use case for (_primary|_replica)_first?

[dakrone]

@bleskes other than them being nicer than having to determine the nodes themselves? I'm not sure. The only thing I could think would be a very script-heavy update where you wanted to use _replica so that you avoided load on the primaries that were doing the fetch-run-script-and-update.

[bleskes]

@bleskes other than them being nicer than having to determine the nodes themselves?

The question is why would you like to do that? I would op to remove those. It just confuses people to think that primaries are special (and we should remove cases where they are..)

[dakrone]

The question is why would you like to do that?

That (the update scenario) was the only scenario I could think of

I would op to remove those.

Sure, if you think the scenario I mentioned isn't valid

It just confuses people to think that primaries are special (and we should remove cases where they are..)

I'm not sure if we're going to be able to move away from update scripts running on the primary shard? At least definitely not on the short term