Message prioritization to certain consumers

[plaisted]

I have a data processing application that runs in a kubernetes cluster that uses NATS Jetstream work queues and pull subscriptions. As it stands now the consumer that gets a message is relatively random (combination of round robin from NATS and timing of an open worker). I'm looking for a way to promote data locality, in this case when a message is published it would prefer to be read by a worker on the same node (so that a local cache would be used to read the results of the last step) but if there are no local workers available it could be pulled by a worker on another node.

I could publish to node specific topics (eg. work.{node name}) and have the stream read all the topics for the work queue but don't see a way to get it routed locally if possible without writing some sort of custom message broker to route the work around. Is there some existing functionality that could support a use case like this that I've missed?

[derekcollison]

Can you describe the state that is being read, the one you want locality for? Is it a file read? A Key Value pair?

[plaisted]

They are files persisted to s3 but also cached locally on the node SSD. The messages include the s3 key but grab the local cached file if it exists.

[derekcollison]

Maybe consider using JetStream KV or Object store and have realtime mirrors on all the nodes.

These can be configured to expire after a certain amount of time or when new entries are added.

[plaisted]

Interesting suggestion, my original intent for the prioritization to certain consumers was to avoid copying the data around (relatively large files) and paying for AWS network traffic / S3 traffic. However, if NATS is moving the data locally to an availability zone that's free and fast. Don't think it's a great fit for our use case (files need to be mmap'd so would need to be copied locally again anyway) but good to think about.

[phamann]

@plaisted, I have a very similar use case to yours (reading large files and want to try routing within an AZ if possible to reduce costs). Did you settle on a NATS pattern that facilitated this / that you were happy with?

[jnmoyne]

One possible option that may work for you, assuming the data is really just caching data and therefore initially created by the node that receives the first request, and assuming you have something in the subject that you can use to identify related requests: partitioned consumers groups https://github.com/synadia-io/orbit.go/tree/main/pcgroups will deterministically distribute the messages in a stream between a set of 'members' (consumers).