Distribute MQTT topic space across different Kafka topics

[jfallows]

The MQTT topic space is more like a single space, treating topic names as a filter. However the choices in how these are mapped to the Kafka topic space has impact on both scalability and performance as no server-side filtering exists in Kafka. If we use only a single underlying Kafka topic, then Zilla will receive all messages in the entire MQTT topic space at all Zilla instances.

Mapping an exact MQTT topic to the corresponding Kafka topic is straightforward.

Complexity increases as we attempt to split the MQTT topic space over different Kafka topics and still attempt to treat it as a unified MQTT topic space for wildcard topic filters.

MQTT to Kafka mappings can make use of patterns, to route to the appropriate underlying Kafka topic, perhaps even parameterized by a named segment of the MQTT topic name.

Therefore, when an MQTT wildcard topic filter is specified, we may not have explicit values for all of the parameterized segments of the Kafka topic name, but we will have the pattern. So we can interrogate Kafka for the full list of topic names and discern the subset that matches the pattern. Then we can subscribe to all Kafka topics in that subset, dynamically adjusting as new topics are created or existing topics are destroyed. They would all follow consistent filtering rules.