vert.x cluster using NATS

[magicprinc]

Proposed change

? Should this proposal go into main NATS issue tracker?🤷‍♀️

As you know, Vert.x can work in a cluster.
But it doesn't handle clustering itself, instead it delegates this to other projects, like
https://vertx.io/docs/vertx-ignite/java/
https://vertx.io/docs/vertx-hazelcast/java/
https://vertx.io/docs/vertx-zookeeper/java/

I think it's a good idea to add NATS support to vert.x!

Use case

Vert.x can work in cluster using NATS.

This will serve to popularize NATS among Java developers.

Contribution

No response

[scottf]

It seems like these cluster managers are different ways to cluster vertx itself. Does this mean client code needs to be cluster aware? How is "aware" accomplished? For this client, which is a thin wrapper around the nats client, it's up to the user to provide the vertx implementation which I assume the user can get a cluster aware vertx implementation, so I think this is a non issue.

[magicprinc]

Here is information from vert.x documentation about this functionality
https://vertx.io/docs/vertx-core/java/#_cluster_manager_spi

There are two prefect examples:
https://github.com/extenda/vertx-redis-clustermanager

https://github.com/vert-x3/vertx-zookeeper

Zookeeper Cluster Manager is an official, "out of the box" project, so it has good documentation:
https://vertx.io/docs/vertx-zookeeper/java/

PS:
I dislike this part from _cluster_manager_spi documentation:

"Cluster managers do not handle the event bus inter-node transport, this is done directly by Vert.x with TCP connections."

It makes sense from vert.x point of view (vert.x can TCP and makes it very efficient),
but you have to configure extra how vert.x nodes connect to each other, and it is "best effort" (no persistence/durability). If there is no consumer, then you simply lose your messages sent through vert.x EventBus 🤷‍♀️

NATS is like a vert.x EventBus in Clustered mode, but much much better.

PPS
Other examples are vert.x Ignite ClusterManager and vert.x Hazelcast ClusterManager, but they are quite different. They don't have a separate server like NATS or Redis, but everything is implemented in Java.

[magicprinc]

Does this mean client code needs to be cluster aware?

I assume you speak about your NATS vert.x client?

Yes, it is beyond simply NATS client. 🙏
Some Vert.x internal interfaces must be implemented.
It can be in this project, or it can be a new sibling project (like spring nats).

I could sketch some drafts, but I am not NATS expert or a nats.java expert, so the code will need review and some rework/optimization.

[magicprinc]

I came to NATS from Apache Ignite.
So I have used vert.x Apache Ignite Cluster.

Apache Ignite is a great replacement for Redis.
Amazing IMDB (with key-value and SQL). Computation Grid, etc, etc.
BUT pub/sub and queues are primitive, no super-cluster: connectivity between data centers and some disadvantages of Kafka from your white paper are applicable 😭

vert.x Apache Ignite Cluster has disadvantages:
hard to configure in some complex scenarios (separate configuration for Ignite and extra configuration for vert.x)
and after all efforts it is like NATS without JetStream.

[scottf]

If someone were to do the development themself, in my role as maintainer, I would administer the code being put into the project. This is an open source project and we are happy to have contributions. If it would be more appropriate to have a sibling project, I would maintain that as well.

If someone wants to pay for the development, a consulting agreement could be made to do the work.