Quadlet container stucks in stopping state

[NotSoFancyName]

Issue Description

I am trying to achieve the same behavior as with podman stop -t 0 <nameOrId> for a Quadlet container (I want to stop it immediately, regardless of its configuration). To do this, I use systemctl --user kill --signal=SIGKILL <service>. However, I am experiencing an issue while running this command when the container is already in stopping state, triggered by systemctl --user stop <service> . The container stays in the stopping state forever.

Since @mheon mentioned that this is expected behaviour I have also have created a question in the Discussion section.

Steps to reproduce the issue

Steps to reproduce the issue

1. Create a quadlet container unit file ~/.config/containers/systemd/test.container with following content:

[Container]
Image=alpine
ContainerName=test
Exec=sleep 1000

[Service]
TimeoutStartSec=600

2. Do systemctl --user daemon-reload
3. Do systemctl --user start test
4. Do systemctl --user stop test
5. In the other terminal tab do systemctl --user kill --signal=SIGKILL test
6. Do podman ps -a

Describe the results you received

After doing podman ps -a container stays in the stopping state:

322847aaa139  docker.io/library/alpine:latest             sleep 1000              33 seconds ago       Stopping                           test

and the systemd service status:

× test.service
     Loaded: loaded (/home/vova/.config/containers/systemd/test.container; generated)
    Drop-In: /etc/systemd/user/service.d
             └─10-timeout-abort.conf
     Active: failed (Result: signal) since Tue 2025-07-15 11:22:21 CEST; 53min ago
   Duration: 16.652s
 Invocation: fdec367daf91437fabe74e5f0504efe5
    Process: 2510870 ExecStart=/usr/bin/podman run --name test --cidfile=/run/user/1000/test.cid --replace --rm --cgroups=split --sdnotify=conmon -d alpine sleep 1000 (code=killed, signal=KILL)
    Process: 2511073 ExecStop=/usr/bin/podman rm -v -f -i --cidfile=/run/user/1000/test.cid (code=killed, signal=KILL)
    Process: 2511160 ExecStopPost=/usr/bin/podman rm -v -f -i --cidfile=/run/user/1000/test.cid (code=killed, signal=KILL)
   Main PID: 2510870 (code=killed, signal=KILL)
   Mem peak: 45.8M
        CPU: 306ms

Describe the results you expected

There is no container in the list after doing podman ps -a

podman info output

host:
  arch: amd64
  buildahVersion: 1.40.1
  cgroupControllers:
  - cpu
  - io
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.13-1.fc42.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.13, commit: '
  cpuUtilization:
    idlePercent: 94.52
    systemPercent: 1.94
    userPercent: 3.55
  cpus: 16
  databaseBackend: sqlite
  distribution:
    distribution: fedora
    variant: workstation
    version: "42"
  eventLogger: journald
  freeLocks: 2026
  hostname: dhcp-10-29-246-140
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 524288
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 524288
      size: 65536
  kernel: 6.15.3-200.fc42.x86_64
  linkmode: dynamic
  logDriver: journald
  memFree: 668872704
  memTotal: 32914616320
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.15.0-1.fc42.x86_64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.15.0
    package: netavark-1.15.2-1.fc42.x86_64
    path: /usr/libexec/podman/netavark
    version: netavark 1.15.2
  ociRuntime:
    name: crun
    package: crun-1.21-1.fc42.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.21
      commit: 10269840aa07fb7e6b7e1acff6198692d8ff5c88
      rundir: /run/user/1000/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt-0^20250611.g0293c6f-1.fc42.x86_64
    version: |
      pasta 0^20250611.g0293c6f-1.fc42.x86_64
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: true
    path: /run/user/1000/podman/podman.sock
  rootlessNetworkCmd: pasta
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: ""
    package: ""
    version: ""
  swapFree: 7954321408
  swapTotal: 8589930496
  uptime: 167h 25m 13.00s (Approximately 6.96 days)
  variant: ""
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
store:
  configFile: /home/vova/.config/containers/storage.conf
  containerStore:
    number: 15
    paused: 0
    running: 13
    stopped: 2
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/vova/.local/share/containers/storage
  graphRootAllocated: 1022488477696
  graphRootUsed: 87937064960
  graphStatus:
    Backing Filesystem: btrfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Supports shifting: "false"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 1265
  runRoot: /run/user/1000/containers
  transientStore: false
  volumePath: /home/vova/.local/share/containers/storage/volumes
version:
  APIVersion: 5.5.2
  BuildOrigin: Fedora Project
  Built: 1750723200
  BuiltTime: Tue Jun 24 02:00:00 2025
  GitCommit: e7d8226745ba07a64b7176a7f128e4ef53225a0e
  GoVersion: go1.24.4
  Os: linux
  OsArch: linux/amd64
  Version: 5.5.2

Podman in a container

No

Privileged Or Rootless

Rootless

Upstream Latest Release

Yes

Additional environment details

No response

Additional information

No response

[Luap99]

My recommendation would be to set KillSignal/StopSignal #26639 (comment)

If systemctl kill also kills the ExecStopPost then yeah we cannot perform cleanup and leak resources.
Reading the systemctl man page it seems there is the option to use --kill-whom=main on systemctl kill which should only kill the main process and still allow our cleanup process to run correctly then I guess. So try that maybe.

[NotSoFancyName]

I have already tried systemctl --user kill --kill-whom=main --signal=SIGKILL test and the container still stays, but at least it is not in the stopping state:

75f8cbbdbf10  docker.io/library/alpine:latest      sleep 1000       About a minute ago    Exited (-1) 8 seconds ago     test            

The reason why we want to avoid using StopSignal with SIGKILL is because we want to have two options: stop the container normally (if the proper handling of the SIGTERM was configured) or do the force shutdown without waiting till the --stop-timeout exceeds. It seems that in case of quadlet units it is impossible to overwrite that behaviour after the systemd container service is up.

[Luap99]

I have already tried systemctl --user kill --kill-whom=main --signal=SIGKILL test and the container still stays, but at least it is not in the stopping state:

Mhh, weird. If we can convert it to exited I am not sure why it was not removed. Does the ExecStopPost= command ran in this case?

The reason why we want to avoid using StopSignal with SIGKILL is because we want to have two options: stop the container normally (if the proper handling of the SIGTERM was configured) or do the force shutdown without waiting till the --stop-timeout exceeds. It seems that in case of quadlet units it is impossible to overwrite that behaviour after the systemd container service is up.

I mean you can always directly send kill -9 to the main container pid or use podman kill on the container which should allow you to force the shutdown right away.

[NotSoFancyName]

Mhh, weird. If we can convert it to exited I am not sure why it was not removed. Does the ExecStopPost= command ran in this case?

Yes it did but it exited with exit code 125

× test.service
     Loaded: loaded (/home/vova/.config/containers/systemd/test.container; generated)
    Drop-In: /etc/systemd/user/service.d
             └─10-timeout-abort.conf
     Active: failed (Result: signal) since Tue 2025-07-15 14:48:27 CEST; 3s ago
   Duration: 5.789s
 Invocation: 9cfbad6063bb4dc38b929a4d6758f564
    Process: 2699991 ExecStart=/usr/bin/podman run --name test --cidfile=/run/user/1000/test.cid --replace --rm --cgroups=split --sdnotify=conmon -d alpine sleep 1000 (code=killed, signal=KILL)
    Process: 2700025 ExecStopPost=/usr/bin/podman rm -v -f -i --cidfile=/run/user/1000/test.cid (code=exited, status=125)
   Main PID: 2699991 (code=killed, signal=KILL)
   Mem peak: 45.5M
        CPU: 243ms

Jul 15 14:48:21 dhcp-10-29-246-140 systemd[2487]: Started test.service.
Jul 15 14:48:21 dhcp-10-29-246-140 test[2699960]: f63591faf9252cbb970e2404ceb77b72bd895f2ce17ca6d1abbc11eb0143a060
Jul 15 14:48:27 dhcp-10-29-246-140 systemd[2487]: test.service: Sent signal SIGKILL to main process 2699991 (conmon) on client request.
Jul 15 14:48:27 dhcp-10-29-246-140 systemd[2487]: test.service: Main process exited, code=killed, status=9/KILL
Jul 15 14:48:27 dhcp-10-29-246-140 systemd[2487]: test.service: Killing process 2699993 (sleep) with signal SIGKILL.
Jul 15 14:48:27 dhcp-10-29-246-140 systemd[2487]: test.service: Killing process 2699989 (pasta.avx2) with signal SIGKILL.
Jul 15 14:48:27 dhcp-10-29-246-140 podman[2700025]: 2025-07-15 14:48:27.408094156 +0200 CEST m=+0.071154365 container stop f63591faf9252cbb970e2404ceb77b72bd895f2ce17ca6d1abbc11eb0143a060 (image=docker.io/librar>
Jul 15 14:48:27 dhcp-10-29-246-140 podman[2700025]: 2025-07-15 14:48:27.408504613 +0200 CEST m=+0.071564822 container died f63591faf9252cbb970e2404ceb77b72bd895f2ce17ca6d1abbc11eb0143a060 (image=docker.io/librar>
Jul 15 14:48:27 dhcp-10-29-246-140 test[2700025]: Error: cannot remove container f63591faf9252cbb970e2404ceb77b72bd895f2ce17ca6d1abbc11eb0143a060 as it could not be stopped: container f63591faf9252cbb970e2404ceb>
Jul 15 14:48:27 dhcp-10-29-246-140 systemd[2487]: test.service: Failed with result 'signal'.

I mean you can always directly send kill -9 to the main container pid or use podman kill on the container which should allow you to force the shutdown right away.

In our case, we cannot use kill -9 because the process to be killed is not in the same process namespace in which we do the container orchestration.

[Luap99]

Error: cannot remove container f63591faf9252cbb970e2404ceb77b72bd895f2ce17ca6d1abbc11eb0143a060 as it could not be stopped: container f63591faf9252cbb970e2404ceb>

Can you provide the full error text. This seem like a legit podman bug if we are unable to remove the container after we conmon got killed that we should be able to fix. That should be enough to make --kill-whom=main work properly then.

[NotSoFancyName]

Sure, here it is

Jul 15 14:48:27 dhcp-10-29-246-140 test[2700025]: Error: cannot remove container f63591faf9252cbb970e2404ceb77b72bd895f2ce17ca6d1abbc11eb0143a060 as it could not be stopped: container f63591faf9252cbb970e2404ceb77b72bd895f2ce17ca6d1abbc11eb0143a060 conmon exited prematurely, exit code could not be retrieved: internal libpod error

[Luap99]

Feel free to give #26643 a try, I it should fix the error and correctly remove the container when conmon is killed