libcdb CI cache doesn't start sometimes

[peace-maker]

The nginx container in CI fails to parse the config sometimes resulting in all libcdb tests to fail. Here are the logs of two runs from the same overall action run.

Failed run

Print service container logs: 2b255856d2444f259f43b2fe22030948_nginx_9b088c
/usr/bin/docker logs --details e027be1826ab33dc6ab2f2e22ede272edba2346d449a0cb586fa5fe6aa3b66af
 /docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
Notice:  2026/04/13 12:30:22 [notice] 1#1: using the "epoll" event method
Notice:  2026/04/13 12:30:22 [notice] 1#1: nginx/1.29.8
Notice:  2026/04/13 12:30:22 [notice] 1#1: built by gcc 14.2.0 (Debian 14.2.0-19) 
Notice:  2026/04/13 12:30:22 [notice] 1#1: OS: Linux 6.17.0-1010-azure
Notice:  2026/04/13 12:30:22 [notice] 1#1: getrlimit(RLIMIT_NOFILE): 65536:65536
Notice:  2026/04/13 12:30:22 [notice] 1#1: start worker processes
Notice:  2026/04/13 12:30:22 [notice] 1#1: start worker process 29
Notice:  2026/04/13 12:30:22 [notice] 1#1: start worker process 30
Notice:  2026/04/13 12:30:22 [notice] 1#1: start worker process 31
Notice:  2026/04/13 12:30:22 [notice] 1#1: start worker process 32
Notice:  2026/04/13 12:30:26 [notice] 1#1: signal 3 (SIGQUIT) received, shutting down
Notice:  2026/04/13 12:30:26 [notice] 29#29: gracefully shutting down
Notice:  2026/04/13 12:30:26 [notice] 30#30: gracefully shutting down
Notice:  2026/04/13 12:30:26 [notice] 29#29: exiting
Notice:  2026/04/13 12:30:26 [notice] 30#30: exiting
Notice:  2026/04/13 12:30:26 [notice] 29#29: exit
Notice:  2026/04/13 12:30:26 [notice] 30#30: exit
Notice:  2026/04/13 12:30:26 [notice] 31#31: gracefully shutting down
Notice:  2026/04/13 12:30:26 [notice] 31#31: exiting
Notice:  2026/04/13 12:30:26 [notice] 32#32: gracefully shutting down
Notice:  2026/04/13 12:30:26 [notice] 32#32: exiting
Notice:  2026/04/13 12:30:26 [notice] 31#31: exit
Notice:  2026/04/13 12:30:26 [notice] 32#32: exit
Notice:  2026/04/13 12:30:26 [notice] 1#1: signal 17 (SIGCHLD) received from 30
Notice:  2026/04/13 12:30:26 [notice] 1#1: worker process 30 exited with code 0
Notice:  2026/04/13 12:30:26 [notice] 1#1: signal 29 (SIGIO) received
Notice:  2026/04/13 12:30:26 [notice] 1#1: signal 17 (SIGCHLD) received from 32
Notice:  2026/04/13 12:30:26 [notice] 1#1: worker process 32 exited with code 0
Notice:  2026/04/13 12:30:26 [notice] 1#1: signal 17 (SIGCHLD) received from 31
Notice:  2026/04/13 12:30:26 [notice] 1#1: worker process 29 exited with code 0
Notice:  2026/04/13 12:30:26 [notice] 1#1: worker process 31 exited with code 0
Notice:  2026/04/13 12:30:26 [notice] 1#1: exit
 2026/04/13 12:30:26 [emerg] 1#1: host not found in upstream "debuginfod.elfutils.org" in /etc/nginx/nginx.conf:26
 nginx: [emerg] host not found in upstream "debuginfod.elfutils.org" in /etc/nginx/nginx.conf:26
 /docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
 /docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
 10-listen-on-ipv6-by-default.sh: info: Getting the checksum of /etc/nginx/conf.d/default.conf
 10-listen-on-ipv6-by-default.sh: info: Enabled listen on IPv6 in /etc/nginx/conf.d/default.conf
 /docker-entrypoint.sh: Sourcing /docker-entrypoint.d/15-local-resolvers.envsh
 /docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
 /docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
 /docker-entrypoint.sh: Configuration complete; ready for start up
 /docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
 /docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
 /docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
 10-listen-on-ipv6-by-default.sh: info: IPv6 listen already enabled
 /docker-entrypoint.sh: Sourcing /docker-entrypoint.d/15-local-resolvers.envsh
 /docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
 /docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
 /docker-entrypoint.sh: Configuration complete; ready for start up
Stop and remove container: 2b255856d2444f259f43b2fe22030948_nginx_9b088c
/usr/bin/docker rm --force e027be1826ab33dc6ab2f2e22ede272edba2346d449a0cb586fa5fe6aa3b66af
e027be1826ab33dc6ab2f2e22ede272edba2346d449a0cb586fa5fe6aa3b66af
Remove container network: github_network_e1beb0b502584c518ed16aa69f3b613e
/usr/bin/docker network rm github_network_e1beb0b502584c518ed16aa69f3b613e
github_network_e1beb0b502584c518ed16aa69f3b613e

Working run

Print service container logs: 0c9c867e08d74470866d720f861193e0_nginx_be571a
/usr/bin/docker logs --details 25036fd6c42eef6b9007f8a021737f3f02df69762c8c91e22be721c13fce1a17
Notice:  2026/04/13 12:30:24 [notice] 1#1: using the "epoll" event method
 /docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
Notice:  2026/04/13 12:30:24 [notice] 1#1: nginx/1.29.8
Notice:  2026/04/13 12:30:24 [notice] 1#1: built by gcc 14.2.0 (Debian 14.2.0-19) 
Notice:  2026/04/13 12:30:24 [notice] 1#1: OS: Linux 6.17.0-1010-azure
Notice:  2026/04/13 12:30:24 [notice] 1#1: getrlimit(RLIMIT_NOFILE): 65536:65536
Notice:  2026/04/13 12:30:24 [notice] 1#1: start worker processes
Notice:  2026/04/13 12:30:24 [notice] 1#1: start worker process 29
 /docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
Notice:  2026/04/13 12:30:24 [notice] 1#1: start worker process 30
Notice:  2026/04/13 12:30:24 [notice] 1#1: start worker process 31
Notice:  2026/04/13 12:30:24 [notice] 1#1: start worker process 32
Notice:  2026/04/13 12:30:29 [notice] 1#1: signal 3 (SIGQUIT) received, shutting down
Notice:  2026/04/13 12:30:29 [notice] 29#29: gracefully shutting down
Notice:  2026/04/13 12:30:29 [notice] 31#31: gracefully shutting down
Notice:  2026/04/13 12:30:29 [notice] 29#29: exiting
Notice:  2026/04/13 12:30:29 [notice] 31#31: exiting
Notice:  2026/04/13 12:30:29 [notice] 29#29: exit
Notice:  2026/04/13 12:30:29 [notice] 31#31: exit
Notice:  2026/04/13 12:30:29 [notice] 30#30: gracefully shutting down
Notice:  2026/04/13 12:30:29 [notice] 30#30: exiting
Notice:  2026/04/13 12:30:29 [notice] 32#32: gracefully shutting down
Notice:  2026/04/13 12:30:29 [notice] 30#30: exit
Notice:  2026/04/13 12:30:29 [notice] 32#32: exiting
Notice:  2026/04/13 12:30:29 [notice] 32#32: exit
Notice:  2026/04/13 12:30:29 [notice] 1#1: signal 17 (SIGCHLD) received from 31
Notice:  2026/04/13 12:30:29 [notice] 1#1: worker process 29 exited with code 0
Notice:  2026/04/13 12:30:29 [notice] 1#1: worker process 31 exited with code 0
Notice:  2026/04/13 12:30:29 [notice] 1#1: worker process 32 exited with code 0
Notice:  2026/04/13 12:30:29 [notice] 1#1: signal 29 (SIGIO) received
Notice:  2026/04/13 12:30:29 [notice] 1#1: signal 17 (SIGCHLD) received from 29
Notice:  2026/04/13 12:30:29 [notice] 1#1: signal 17 (SIGCHLD) received from 30
Notice:  2026/04/13 12:30:29 [notice] 1#1: worker process 30 exited with code 0
Notice:  2026/04/13 12:30:29 [notice] 1#1: exit
 /docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
 10-listen-on-ipv6-by-default.sh: info: Getting the checksum of /etc/nginx/conf.d/default.conf
 10-listen-on-ipv6-by-default.sh: info: Enabled listen on IPv6 in /etc/nginx/conf.d/default.conf
 /docker-entrypoint.sh: Sourcing /docker-entrypoint.d/15-local-resolvers.envsh
 /docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
 /docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
 /docker-entrypoint.sh: Configuration complete; ready for start up
 /docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
 /docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
 /docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
 10-listen-on-ipv6-by-default.sh: info: IPv6 listen already enabled
 /docker-entrypoint.sh: Sourcing /docker-entrypoint.d/15-local-resolvers.envsh
 /docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
 /docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
 /docker-entrypoint.sh: Configuration complete; ready for start up
 172.18.0.1 - - - HIT [13/Apr/2026:12:34:00 +0000]  "POST /api/find HTTP/1.1" 200 818 "-" "python-requests/2.33.1"

The nginx: [emerg] host not found in upstream "debuginfod.elfutils.org" in /etc/nginx/nginx.conf:26 line should have been fixed by specifying the resolver in the config to delay resolving the domain when the first request arrives instead of on config parse time.

Maybe it's the different matrix runs influencing each other's nginx containers? I'm not sure how services interact with matrix runs in Github Actions. If every matrix job gets its own service container or not. Needs investigating.

One recent instance was here. The nginx logs are in the Stop containers step at the bottom.