Basic Docker Compose Setup Clogs CPU With Error Messages

[halkony]

Describe the bug
Running a basic docker compose file with kafka and zilla clogs up CPU with "invalid number of brokers" message. Consumes every available cycle.

To Reproduce
Copy mqtt.kafka.broker from zilla-examples.
Replace ./docker/compose/docker-compose.yaml with the following file and run docker compose up in the directory.

version: '3'
services:
  zookeeper:
    image: wurstmeister/zookeeper
    container_name: zookeeper
    ports:
      - "2181:2181"
    networks:
      - default

  kafka:
    image: confluentinc/cp-kafka:7.1.3
    container_name: kafka
    depends_on:
      - zookeeper
    ports:
      - "9093:9093"
    environment:
      KAFKA_ADVERTISED_LISTENERS: BROKER://kafka:9092,OUTSIDE://localhost:9093
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: BROKER:PLAINTEXT,OUTSIDE:PLAINTEXT
      KAFKA_LISTENERS: BROKER://0.0.0.0:9092,OUTSIDE://0.0.0.0:9093
      KAFKA_INTER_BROKER_LISTENER_NAME: BROKER
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_CREATE_TOPICS: "poopootopic:1:1"
      KAFKA_BROKER_ID: '1'
    networks:
      - default

  zilla:
    image: ghcr.io/aklivity/zilla
    container_name: zilla
    pull_policy: always
    restart: unless-stopped
    depends_on:
      kafka-init:
        condition: service_completed_successfully
    ports:
      - 7114:7114
      - 7143:7143
      - 7183:7183
      - 7883:7883
    environment:
      KEYSTORE_PASSWORD: generated
      KAFKA_HOST: kafka
      KAFKA_PORT: 9092
    volumes:
      - ../../zilla.yaml:/etc/zilla/zilla.yaml
      - ../../tls:/etc/zilla/tls
    command: start -v -e

  kafka-init:
    image: bitnami/kafka:3.2
    command:
      - "/bin/bash"
      - "-c"
      -  |
        /opt/bitnami/kafka/bin/kafka-topics.sh --bootstrap-server kafka:9092 --create --if-not-exists --topic mqtt-messages
        /opt/bitnami/kafka/bin/kafka-topics.sh --bootstrap-server kafka:9092 --create --if-not-exists --topic mqtt-sessions --config cleanup.policy=compact
        /opt/bitnami/kafka/bin/kafka-topics.sh --bootstrap-server kafka:9092 --create --if-not-exists --topic mqtt-retained --config cleanup.policy=compact
    depends_on:
      - kafka
    init: true

networks:
  default:
    driver: bridge

Running on Windows 10,

Expected behavior
I expect to run this docker compose stack and have Kafka available on port 9093 and zilla's mqtt functionality available on port 7183.

Screenshots

Desktop (please complete the following information):

• OS: Windows 10
• Docker Compose V2, Docker Desktop v4.26.1

Additional context
Temporary solution seems to be running a stack with at least 3 brokers. Will try this with a kraft stack later.

If this problem is unique to zilla, it'd be nice to have a timeout on these requests so my CPU doesn't get bogged down.

[halkony]

zilla.yml for reference, copied from the mqtt broker example

---
name: zilla-mqtt-kafka-broker
bindings:
  # Proxy service entrypoint
  north_tcp_server:
    type: tcp
    kind: server
    options:
      host: 0.0.0.0
      port:
        - 7114
        - 7143
        - 7183
        - 7883
    routes:
      - when:
          - port: 7114
        exit: north_http_server
      - when:
          - port: 7143
        exit: north_wss_tls_server
      - when:
          - port: 7183
        exit: north_mqtt_server
      - when:
          - port: 7883
        exit: north_mqtts_tls_server

  # TLS decrypting
  north_mqtts_tls_server:
    type: tls
    kind: server
    vault: my_servers
    options:
      keys:
        - localhost
      sni:
        - localhost
    exit: north_mqtt_server
  north_wss_tls_server:
    type: tls
    kind: server
    vault: my_servers
    options:
      keys:
        - localhost
      sni:
        - localhost
    exit: north_http_server

  # WebSocket server
  north_http_server:
    type: http
    kind: server
    routes:
      - when:
          - headers:
              :scheme: http
              :authority: localhost:7114
              upgrade: websocket
          - headers:
              :scheme: https
              :authority: localhost:7143
              upgrade: websocket
        exit: north_ws_server
  north_ws_server:
    type: ws
    kind: server
    routes:
      - when:
          - protocol: mqtt
        exit: north_mqtt_server

  # Shared MQTT server
  north_mqtt_server:
    type: mqtt
    kind: server
    exit: north_mqtt_kafka_mapping

  # MQTT messages to Kafka topics
  north_mqtt_kafka_mapping:
    type: mqtt-kafka
    kind: proxy
    options:
      server: localhost:7183
      topics:
        sessions: mqtt-sessions
        messages: mqtt-messages
        retained: mqtt-retained
    clients:
      - place/{identity}/#
    routes:
      - when:
          - publish:
              - topic: place/+/device/#
              - topic: device/#
          - subscribe:
              - topic: place/+/device/#
              - topic: device/#
        with:
          messages: mqtt-devices
        exit: north_kafka_cache_client
    exit: north_kafka_cache_client

  # Kafka sync layer
  north_kafka_cache_client:
    type: kafka
    kind: cache_client
    exit: south_kafka_cache_server
  south_kafka_cache_server:
    type: kafka
    kind: cache_server
    options:
      bootstrap:
        - mqtt-messages
        - mqtt-retained
        - mqtt-devices
    exit: south_kafka_client

  # Connect to Kafka
  south_kafka_client:
    type: kafka
    kind: client
    exit: south_tcp_client
  south_tcp_client:
    type: tcp
    kind: client
    options:
      host: ${{env.KAFKA_HOST}}
      port: ${{env.KAFKA_PORT}}
    routes:
      - when:
          - cidr: 0.0.0.0/0

vaults:
  my_servers:
    type: filesystem
    options:
      keys:
        store: tls/localhost.p12
        type: pkcs12
        password: ${{env.KEYSTORE_PASSWORD}}

[halkony]

Tested this with 3 brokers and it solved the issue of the CPU bloat.

Unfortunately, I still have the same problem I describe in #724

[vordimous]

@halkony Looking at the docker logs output you posted am I correct that the issue was with zookeeper and confluent kafka?

[halkony]

Is zilla responsible for the __consumer-offsets topic? I didn't have this problem when working with the 1 broker example using bitnami kafka + kraft, so the combination of these images must be causing the issue.

[vordimous]

No, Zilla doesn't create any topics on Kafka. It will start up normally if a topic it needs doesn't exist and wait for it to become available. Zilla manages any relevant Kafka information in it's Kafka cache and loads that from the underlying Kafka cluster, making each Zilla instance stateless.

[vordimous]

@halkony, were you able to determine the cause of the CPU error messages? If the error was a configuration issue I will go ahead and close this ticket.

[vordimous]

This issue has grown stale; please reopen it if it comes up again.