[DISCUSS] Add back preference for searching _primaries or _replicas

[mattweber]

With segment replication and the potential lag of replicas, it would be nice to support search routing for _primaries and _replicas. With this preference we can keep traffic off our indexers and/or only on primaries to ensure the most up-to-date data. This was previously supported in Elaslticsearch <= 6x but removed for 7x+. I imagine it wouldn't be too hard to revert that PR (it is pre-license change) to re-enable this support. What do you think?

[mattweber]

cc @reta @dblock I can work on this if you would consider merging it

[andrross]

@mch2 What do you think about this idea?

[andrross]

We have some longer term plans regarding separating readers and writers, but this seems like low hanging fruit for a useful capability today.

[mch2]

Love this idea, @mattweber Thanks for raising this! We had been considering the possibility of a priority/urgent request param to fwd requests only to primaries given the added replication lag, but having an option to control routing to both primaries & replicas is even better. This would be a great addition.

[anasalkouz]

@mattweber Thanks for raising this idea. Feel free to pick up this and submit PR. I will assigned it to you.

[nknize]

@mattweber I remember the conversation to remove that feature in 7x+ to avoid trappy hot spots. It was a controversial change for sure.

I'm all for reverting as a simple mechanism, but maybe just enable it for indexes using segrep?

Another idea we had been toying w/ as @mch2 mentioned was to add something like priority={CRITICAL | HIGH | MEDIUM | LOW} as an API mechanism to help triage query routing and guide ARS. CRITICAL queries will only be executed on the primary and pre-empt other lower priority queries, lower priority queries can run on primaries or replicas but they're ordered and pre-empted by priority. Perhaps that could be a follow on enhancement.

[andrross]

With this preference we can keep traffic off our indexers

@mattweber Just curious how far this feature will get you towards this goal, given that the default behavior is to allocate primary and replica shards evenly across all nodes. Do you have any mechanism to change the shard allocation?

[mattweber]

@andrross you are right by default and where nodes have more than one shard this still won't 100% solve things. I main use-case is currently querying the primary for most up-to-date data to validate document counts after a load.

[mattweber]

@nknize sure I can do that if you prefer. IMO, I guess a trappy hot spot is still possible even with segrep so it might be better to just add it back and document the potential host spot?

[nknize]

main use-case is currently querying the primary for most up-to-date data to validate document counts after a load.

This is the purpose of the priority query idea mentioned by @mch2. We haven't concretely pursued this yet because we haven't quantified (average) consistency metrics yet to determine if it's a mechanism worth introducing or if it will just exacerbate hot spots. (@mch2 maybe you have more info here). In essence, mechanisms like preference are really just "hacks" to push the system closer to "strongly consistent" than it's "eventually consistent" nature.

[nknize]

... add it back and document the potential host spot?

I lean towards this as well. Some folks are less accommodating, though, and strongly oppose trappy leniency so maybe we open a PR to do this (revert the change) and give sufficient feedback time for strong viewpoints to veto and offer an alternative?