Web STOMP connections can log a crash with {noproc,{sys,terminate}} in terminate_heartbeaters/1

[moritzwiechers]

Describe the bug

We are observing a recurring crash loop of the Web-STOMP listener in Kubernetes.
The rabbit_web_stomp_connection_sup supervisor reaches reached_max_restart_intensity
due to repeated abnormal exits of cowboy_clear workers.

The crash consistently happens during WebSocket connection termination and appears to be
caused by an unhandled {noproc} exit when terminating heartbeat processes.

The log gets filled up with error stack traces.

Communication works as expected.

Reproduction steps

• Nothing special here.
• Startet RabbitMQ and client connect (npm @stomp/stompjs version 7.0.0)
• Communication works fine
• Server Logs get filled up

Expected behavior

Web-STOMP WebSocket connections should terminate gracefully.
Heartbeat termination should not crash the handler if the heartbeat process
has already exited.

Additional context

RabbitMQ version: 4.2.4
Erlang/OTP version: OTP 27
Deployment: Kubernetes (StatefulSet),
via. docker.io/rabbitmqoperator/cluster-operator:2.19.1 (Applying default config: https://github.com/rabbitmq/cluster-operator/releases/download/v2.19.1/cluster-operator.yml)
Plugin: rabbitmq_web_stomp 4.2.4

RabbitmqCluster yaml part: rabbitmq. conf:

spec:
  delayStartSeconds: 30
  image: rabbitmq:4.2.4-management
  override:
    statefulSet:
      spec:
        template:
          spec:
            containers:
              - name: rabbitmq
                resources: {}
                volumeMounts:
                  - mountPath: /app/
                    name: definitions
            volumes:
              - configMap:
                  name: rabbitmq-definitions
                name: definitions
  persistence:
    storage: 10Gi
  rabbitmq:
    additionalConfig: >
      vm_memory_high_watermark.relative = 0.8  
      management.path_prefix = /rabbit
      load_definitions = /app/definitions.json # Path to the mounted definitions file    
      loopback_users = none
    additionalPlugins:
      - rabbitmq_web_stomp
      - rabbitmq_shovel
      - rabbitmq_shovel_management
  replicas: 3
  resources:
    limits:
      cpu: "1"
      memory: 2Gi
    requests:
      cpu: 500m
      memory: 512Mi
  secretBackend:
    externalSecret:
      name: ""
  service:
    type: ClusterIP
  terminationGracePeriodSeconds: 604800

Stack-Trace

crasher:
     initial call: cowboy_clear:connection_process/4
     pid: <0.13671828.0>
     registered_name: []
     exception exit: {noproc,{sys,terminate,[none,shutdown]}}
       in function  sys:send_system_msg/2 (sys.erl, line 754)
       in call from rabbit_web_stomp_handler:terminate_heartbeaters/1 (rabbit_web_stomp_handler.erl, line 306)
       in call from cowboy_websocket:terminate/3 (src/cowboy_websocket.erl, line 773)
       in call from cowboy_http:loop/1 (src/cowboy_http.erl, line 279)
     ancestors: [<0.13671760.0>,<0.842.0>,<0.841.0>,<0.840.0>,ranch_sup,
                   <0.118.0>]
     message_queue_len: 3
     messages: [{tcp_passive,#Port<0.157676>},
                   {'EXIT',<0.13671789.0>,normal},
                   {'EXIT',<0.13671806.0>,{shutdown,normal}}]
     links: [#Port<0.157676>,<0.13671760.0>]
     dictionary: [{gen_server_call_timeout,130000},
                   {guid_secure,{{1,
                                  'rabbit@rabbitmq-server-2.rabbitmq-nodes.prod',
                                  #Ref<0.1208331358.2416967681.147005>},
                                 2}}]
     trap_exit: true
     status: running
     heap_size: 1598
     stack_size: 29
     reductions: 135809
   neighbours:
 
     supervisor: {<0.13671760.0>,rabbit_web_stomp_connection_sup}
     errorContext: child_terminated
     reason: {noproc,{sys,terminate,[none,shutdown]}}
     offender: [{pid,<0.13671828.0>},
                {id,cowboy_clear},
                {mfargs,
                 {cowboy_clear,start_link,
                  [{acceptor,{0,0,0,0,0,0,0,0},15674},
                   ranch_tcp,
                   #{env =>
                      #{dispatch =>
                         [{'_',[],
                           [{[<<"ws">>],
                             [],rabbit_web_stomp_handler,
                             [{type,text},{ws_opts,#{}}]}]}],
                        keepalive_sup => <0.13671785.0>},
                     middlewares =>
                      [cowboy_router,rabbit_web_stomp_middleware,
                       cowboy_handler],
                     stream_handlers =>
                      [rabbit_web_stomp_stream_handler,cowboy_stream_h],
                     proxy_header => false}]}},
                {restart_type,transient},
                {significant,true},
                {shutdown,300000},
                {child_type,worker}]
 
     supervisor: {<0.13671760.0>,rabbit_web_stomp_connection_sup}
     errorContext: shutdown
     reason: reached_max_restart_intensity
     offender: [{pid,<0.13671828.0>},
                {id,cowboy_clear},
                {mfargs,
                 {cowboy_clear,start_link,
                  [{acceptor,{0,0,0,0,0,0,0,0},15674},
                   ranch_tcp,
                   #{env =>
                      #{dispatch =>
                         [{'_',[],
                           [{[<<"ws">>],
                             [],rabbit_web_stomp_handler,
                             [{type,text},{ws_opts,#{}}]}]}],
                        keepalive_sup => <0.13671785.0>},
                     middlewares =>
                      [cowboy_router,rabbit_web_stomp_middleware,
                       cowboy_handler],
                     stream_handlers =>
                      [rabbit_web_stomp_stream_handler,cowboy_stream_h],
                     proxy_header => false}]}},
                {restart_type,transient},
                {significant,true},
                {shutdown,300000},
                {child_type,worker}]

[michaelklishin]

The crash consistently happens during WebSocket connection termination and appears to be
caused by an unhandled {noproc} exit when terminating heartbeat processes.

Those crash reports are just log noise unless you have enough connections abruptly closing connections at once to the point that the supervisor has to stop due to the maximum allowed restart intensity being reached.

[michaelklishin]

#15607 should avoid this exception.

Thank you for providing a fairly detailed report, @moritzwiechers!