Podman 5.6.x fails for cache mounts where 5.4.x did not

[jonner]

Issue Description

After updating podman in fedora, I had several builds that were working before that started failing. It seems to happen for image builds that are using a non-root user and cache mounts. In 5.4.x, the cache mount worked and the build succeeded. In 5.6.x, the cache mount fails with a permission error

Steps to reproduce the issue

Steps to reproduce the issue

1. Use this Containerfile:

FROM quay.io/centos/centos:stream9 AS builder
USER 1001
ENV CACHE=/foo/cache
RUN --mount=type=cache,target=${CACHE},uid=1001 touch ${CACHE}/test

2. build the image: podman build -t test-podman-mount-cache -f Containerfile .
3. observe results

Describe the results you received

In podman 5.4.1, I get the following:

podman build -t test-podman-mount-cache -f Containerfile .
STEP 1/4: FROM quay.io/centos/centos:stream9 AS builder
STEP 2/4: USER 1001
--> 8bf6be39e826
STEP 3/4: ENV CACHE=/foo/cache
--> 597f9f46d120
STEP 4/4: RUN --mount=type=cache,target=${CACHE},uid=1001 touch ${CACHE}/test
COMMIT test-podman-mount-cache
--> 8c0696c1bb7d
Successfully tagged localhost/test-podman-mount-cache:latest
8c0696c1bb7d8195cb39e9c9f6d7fc3862c143cc710415fc538836726a679c94

In podman 5.6.1, I get the following:

podman build -t test-podman-mount-cache -f Containerfile .
STEP 1/4: FROM quay.io/centos/centos:stream9 AS builder
STEP 2/4: USER 1001
--> 180fa21c6dc4
STEP 3/4: ENV CACHE=/foo/cache
--> e712763e35b0
STEP 4/4: RUN --mount=type=cache,target=${CACHE},uid=1001 touch ${CACHE}/test
touch: cannot touch '/foo/cache/test': Permission denied
Error: building at STEP "RUN --mount=type=cache,target=${CACHE},uid=1001 touch ${CACHE}/test": while running runtime: exit status 1
make: *** [Makefile:2: image] Error 1

Describe the results you expected

I expect both versions to behave the same.

podman info output

host:
  arch: amd64
  buildahVersion: 1.41.4
  cgroupControllers:
  - cpu
  - io
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.13-1.fc42.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.13, commit: '
  cpuUtilization:
    idlePercent: 93.92
    systemPercent: 1.09
    userPercent: 4.99
  cpus: 8
  databaseBackend: boltdb
  distribution:
    distribution: fedora
    variant: workstation
    version: "42"
  emulatedArchitectures:
  - linux/arm
  - linux/arm64
  - linux/arm64be
  - linux/loong64
  - linux/mips
  - linux/mips64
  - linux/ppc
  - linux/ppc64
  - linux/ppc64le
  - linux/riscv32
  - linux/riscv64
  - linux/s390x
  eventLogger: journald
  freeLocks: 1968
  hostname: himantopus
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 21811
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 21811
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 6.15.6-200.fc42.x86_64
  linkmode: dynamic
  logDriver: journald
  memFree: 926199808
  memTotal: 33334685696
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.16.0-1.fc42.x86_64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.16.0
    package: netavark-1.16.1-1.fc42.x86_64
    path: /usr/libexec/podman/netavark
    version: netavark 1.16.1
  ociRuntime:
    name: crun
    package: crun-1.23.1-1.fc42.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.23.1
      commit: d20b23dba05e822b93b82f2f34fd5dada433e0c2
      rundir: /run/user/21811/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt-0^20250805.g309eefd-2.fc42.x86_64
    version: |
      pasta 0^20250805.g309eefd-2.fc42.x86_64
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: true
    path: /run/user/21811/podman/podman.sock
  rootlessNetworkCmd: pasta
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: ""
    package: ""
    version: ""
  swapFree: 9351634944
  swapTotal: 17003700224
  uptime: 1227h 56m 40.00s (Approximately 51.12 days)
  variant: ""
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
store:
  configFile: /home/jjongsma/.config/containers/storage.conf
  containerStore:
    number: 19
    paused: 0
    running: 7
    stopped: 12
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/jjongsma/.local/share/containers/storage
  graphRootAllocated: 730684710912
  graphRootUsed: 580832305152
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Supports shifting: "true"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 2145
  runRoot: /run/user/21811/containers
  transientStore: false
  volumePath: /home/jjongsma/.local/share/containers/storage/volumes
version:
  APIVersion: 5.6.1
  BuildOrigin: Fedora Project
  Built: 1756944000
  BuiltTime: Wed Sep  3 19:00:00 2025
  GitCommit: 1e2b2315150b2ffa0971596fb5da8cd83f3ce0e1
  GoVersion: go1.24.6
  Os: linux
  OsArch: linux/amd64
  Version: 5.6.1

Podman in a container

No

Privileged Or Rootless

Rootless

Upstream Latest Release

Yes

Additional environment details

No response

Additional information

No response

[Luap99]

This sounds like a buildah issue and likely a duplicate of containers/buildah#6361

[Luap99]

cc @nalind @flouthoc

[jonner]

I initially thought it might be buildah, but since simply upgrading / downgrading podman (without touching buildah) was enough to trigger the behavior change, I decided to file it against podman.

Also, FWIW, I just checked and the behavior is the same even without the uid parameter provided to the cache mount: 5.4.x works, 5.6.x fails.

FROM quay.io/centos/centos:stream9 AS builder
USER 1001
ENV CACHE=/foo/cache
RUN --mount=type=cache,target=${CACHE} touch ${CACHE}/test

[baude]

podman uses the same code that buildah produces (statically linked). so changing the buildah package will not change podman's build, but they are the same code base.