[4.1.x] Management UI stuck in "Management only mode" after toggling management_metrics_collection back to true

[olakunle-oladimeji-viadialog]

Describe the bug

In RabbitMQ 4.1.0, the legacy metrics bridge (management_metrics_collection) appears to exhibit "sticky" behavior. If the feature is disabled and then re-enabled via rabbitmq.conf, the cluster refuses to transition back to the permitted state.

This results in the Management UI staying in "Management only mode" indefinitely, even after rolling restarts and verifying that the configuration file is being read correctly by the nodes.

Environment :

RabbitMQ Version: 4.1.0
Erlang Version: 27.3.4
Deployment: Kubernetes (StatefulSet) using the Cluster Operator
Cluster Size: 3 nodes

Reproduction steps

1. Initialize Cluster: Deploy a cluster with deprecated_features.permit.management_metrics_collection = true. Confirm charts are visible in the Management UI.
2. Disable Metrics: Update the config to false and perform a rolling restart.
3. Verify UI Change: Open the Management UI. The header now displays "Overview: Management only mode" and all charts (Message rates, Connection counts, etc.) have disappeared.
4. Re-enable Metrics: Revert the configuration to true.
5. Rolling Restart: Perform a rolling restart of all nodes to apply the change.
6. Final UI Check: Refresh the Management UI.

Expected behavior

The Management UI should transition back to showing live metrics and charts once the management_metrics_collection flag is set to true and the nodes are restarted.

Additional context

Current Config:

    rabbitmq:
      additionalConfig: |
        # --- defaults ---
        cluster_formation.discovery_retry_limit = 60
        cluster_formation.discovery_retry_interval = 2000
        channel_max = 5000
        consumer_timeout = 600000
        ssl_options.verify = verify_none
        # definitions.tls.verify = verify_none
        cluster_formation.node_cleanup.interval = 10
        cluster_formation.node_cleanup.only_log_warning = true
        cluster_partition_handling = pause_minority
        # definitions.import_backend = local_filesystem
        # definitions.local.path = /etc/rabbitmq/definitions
        # definitions.skip_if_unchanged = true
        heartbeat = 30
        tcp_listen_options.nodelay = true
        tcp_listen_options.backlog = 128
        tcp_listen_options.keepalive = true
        vm_memory_high_watermark.relative = 0.75
        disk_free_limit.relative = 1.0
        log.file.formatter = json
        log.console.formatter = json
        log.syslog.formatter = json
        management.enable_queue_totals = true
        management_agent.disable_metrics_collector = false
        # --- appended from values.yaml ---
        deprecated_features.permit.management_metrics_collection = true
        num_acceptors.tcp = 32
        queue_leader_locator = balanced
        tcp_listen_options.backlog = 1024

[michaelklishin]

@olakunle-oladimeji-viadialog we do not consider such back-and-forth switching to be an important scenario. In particular given that the management metrics will likely get undeprecated.

Clearing the management UI state (cache, local state) and resetting the statistics DB should help.

As will adopting Prometheus and Grafana which is the recommended option for monitoring in any case.

[olakunle-oladimeji-viadialog]

Thanks for the help! A full cluster restart (scale to 0 → scale up) resolved the UI graph issue.

We’re now moving toward a Prometheus/Grafana setup, but we've hit a roadblock: we need to monitor the list of active consumers of a queue (including IPs and ports).

Since Prometheus typically focuses on aggregated metrics, is there a recommended way to expose this granular connection data? We'd appreciate any advice on workarounds if this isn't supported out of the box.

[mkuratczyk]

There are /metrics/per-object and /metrics/detailed so it's not just aggregated data, but high-cardinality values such as IPs and ports are not present (see https://prometheus.io/docs/practices/naming/#labels). We have Erlang PIDs in some places (also high cardinality value) but I think we should remove that.

IPs and ports are logged when a connection is initiated/terminated - that's one more option.

But the API is still there if you need it. Out of curiosity - why do you monitor IPs/ports of consumers?