podman compose unable to run docker lifecycle hooks

[MoralCode]

When trying to run the compose file, even with patches from #3211, i run into this error:

[augur]        | Python 3.11.13
[augur]        | 2025-07-03 13:39:57 a91d132b829c augur[1] INFO Flushing all redis databases this instance was using
[augur]        | Traceback (most recent call last):
[augur]        |   File "/augur/.venv/bin/augur", line 10, in <module>
[augur]        |     sys.exit(run())
[augur]        | (augur) Reseting log files
[augur]        | Failed to create file handler: [Errno 13] Permission denied: '/logs/augur.info'
[augur]        | Failed to create file handler: [Errno 13] Permission denied: '/logs/augur.error'
[augur]        |              ^^^^^
[augur]        |   File "/augur/.venv/lib/python3.11/site-packages/click/core.py", line 1128, in __call__
[augur]        |     return self.main(*args, **kwargs)
[augur]        |            ^^^^^^^^^^^^^^^^^^^^^^^^^^
[augur]        |   File "/augur/.venv/lib/python3.11/site-packages/click/core.py", line 1053, in main
[augur]        |     rv = self.invoke(ctx)
[augur]        |          ^^^^^^^^^^^^^^^^
[augur]        |   File "/augur/.venv/lib/python3.11/site-packages/click/core.py", line 1659, in invoke
[augur]        |     return _process_result(sub_ctx.command.invoke(sub_ctx))
[augur]        |                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[augur]        |   File "/augur/.venv/lib/python3.11/site-packages/click/core.py", line 1659, in invoke
[augur]        |     return _process_result(sub_ctx.command.invoke(sub_ctx))
[augur]        |                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[augur]        |   File "/augur/.venv/lib/python3.11/site-packages/click/core.py", line 1395, in invoke
[augur]        |     return ctx.invoke(self.callback, **ctx.params)
[augur]        |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[augur]        |   File "/augur/.venv/lib/python3.11/site-packages/click/core.py", line 754, in invoke
[augur]        |     return __callback(*args, **kwargs)
[augur]        |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
[augur]        |   File "/augur/.venv/lib/python3.11/site-packages/click/decorators.py", line 26, in new_func
[augur]        |     return f(get_current_context(), *args, **kwargs)
[augur]        |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[augur]        |   File "/augur/augur/application/cli/__init__.py", line 45, in new_func
[augur]        |     return ctx.invoke(function_internet_connection, *args, **kwargs)
[augur]        |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[augur]        |   File "/augur/.venv/lib/python3.11/site-packages/click/core.py", line 754, in invoke
[augur]        |     return __callback(*args, **kwargs)
[augur]        |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
[augur]        |   File "/augur/.venv/lib/python3.11/site-packages/click/decorators.py", line 26, in new_func
[augur]        |     return f(get_current_context(), *args, **kwargs)
[augur]        |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[augur]        |   File "/augur/augur/application/cli/__init__.py", line 57, in new_func
[augur]        |     return ctx.invoke(function_db_connection, *args, **kwargs)
[augur]        |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[augur]        |   File "/augur/.venv/lib/python3.11/site-packages/click/core.py", line 754, in invoke
[augur]        |     return __callback(*args, **kwargs)
[augur]        |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
[augur]        |   File "/augur/.venv/lib/python3.11/site-packages/click/decorators.py", line 26, in new_func
[augur]        |     return f(get_current_context(), *args, **kwargs)
[augur]        |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[augur]        |   File "/augur/augur/application/cli/__init__.py", line 105, in new_func
[augur]        |     return ctx.invoke(f, *args, **kwargs)
[augur]        |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[augur]        |   File "/augur/.venv/lib/python3.11/site-packages/click/core.py", line 754, in invoke
[augur]        |     return __callback(*args, **kwargs)
[augur]        |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
[augur]        |   File "/augur/.venv/lib/python3.11/site-packages/click/decorators.py", line 26, in new_func
[augur]        |     return f(get_current_context(), *args, **kwargs)
[augur]        |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[augur]        |   File "/augur/augur/application/cli/backend.py", line 91, in start
[augur]        |     cleanup_collection_status_and_rabbit(logger, ctx.obj.engine)
[augur]        |   File "/augur/augur/application/cli/backend.py", line 350, in cleanup_collection_status_and_rabbit
[augur]        |     clear_redis_caches()
[augur]        |   File "/augur/augur/application/cli/backend.py", line 363, in clear_redis_caches
[augur]        |     logger.info("Flushing all redis databases this instance was using")
[augur]        |   File "/usr/local/lib/python3.11/logging/__init__.py", line 1489, in info
[augur]        |     self._log(INFO, msg, args, **kwargs)
[augur]        |   File "/usr/local/lib/python3.11/logging/__init__.py", line 1634, in _log
[augur]        |     self.handle(record)
[augur]        |   File "/usr/local/lib/python3.11/logging/__init__.py", line 1644, in handle
[augur]        |     self.callHandlers(record)
[augur]        |   File "/usr/local/lib/python3.11/logging/__init__.py", line 1705, in callHandlers
[augur]        |     if record.levelno >= hdlr.level:
[augur]        |                          ^^^^^^^^^^
[augur]        | AttributeError: 'NoneType' object has no attribute 'level'

It appears that there is supposed to be a post_start script to open up the permissions on these files at https://github.com/chaoss/augur/blob/cf69b9ea9ebcbe235404d856c914947a8daa2a39/docker-compose.yml#L87C1-L90C19 but it didnt seem to work when using podman compose

I was able to find containers/podman#25370 and containers/podman-compose#794 in upstream podman related repos indicating these lifecycle variables may not be supported yet.

Is podman compose a supported way to run augur?

[sgoggins]

@MoralCode : We do support Docker. I think we would need somebody to sort a compose for Podman as distinct from Docker. Alternately, I think its likely Podman will support these new parameters shortly, since they are working as a Docker alternative. Though its possible the particular permission switches Docker enables will not be supported for security reasons.

[MoralCode]

Is podman also supported in addition to docker? I know in theory they're meant to essentially be replacements for each other, but in practice it seems like podman lags behind a little in the feature support and seems to be a lot more common in the enterprise linux world, especially since fedora ships it by default

[MoralCode]

@JohnStrunk The way I would solve this is probably to add a section at the start of the dockerfile that goes something like

USER root
RUN mkdir  /facade /logs /config /cache
RUN chmod 777  /facade /logs /config /cache

USER augur

Do you think that would break the openshift usecase? My understanding is that openshift is just concerned about avoiding running as root at runtime, so this seems like it should be fine.

[JohnStrunk]

It's not going to break openshift, but I don't think it's going to fix the problem, either.
The lifecycle hook is making sure the volumes are properly writable, so I think they need to be mounted first, before the chmod happens. OCP (and kube) have their own way of handling the volume permissions and aren't affected by this.

Maybe adding a set of VOLUME statements to the dockerfile, plus a chmod 777 would have an effect and allow you to get rid of the lifecycle hook. I'm saying maybe just because of how docker treats new volumes when containers are newly brought up (copying contents... so maybe it'll copy the permissions).

Also of note: The build is already running w/ user root, and there is no augur user defined in the container.

[MoralCode]

Ok, so based on my testing the

touch: cannot touch '/facade/.git-credentials': Permission denied
[augur]        | ./scripts/install/config.sh: line 231: /facade/.git-credentials: Permission denied
[augur]        | ./scripts/install/config.sh: line 232: /facade/.git-credentials: Permission denied
[augur]        | error: could not lock config file /augur/.gitconfig: Permission denied

errors (and i guess also the ones from the original PR description) were caused because the volumes I was using are ones ive been using for a while, and thus presumably had permissions set from sometime before things broke (which i assume was when permissions/user IDs got changed in #3131).

Bringing down the stack and changing the name of the 4 volumes used by this container caused fresh volumes to be created, which do not experience this issue because the user id they were created with and run with are the same.

using chmod 777 feels like a bit of a hack and, based on the above info, it seems like maybe some form of docs - such as a comment near this hook, or a note in the docker setup docs - would be a good way to resolve this.

This seems to only be an issue for podman in very narrow circumstances too, so that's definitely helpful

[MoralCode]

While the core issue here was fundamentally a PEBCAK on my part, I think its still worth keeping this open to decide what to do with the lifecycle hooks.

It sounds like they don't make a difference for openshift (since openshift handles volume permissions differently), are only supported in current versions of docker, and may change permissions to something (777) that is often undesirable from a security standpoint. I feel like this may be better to resolve as a documentation fix that explains how to fix permissions issues in these directories in docker

[MoralCode]

I think i found another piece of this puzzle

• during docker build, init.sh is run. oops this is wrong. ugh
• init.sh calls ./scripts/install/config.sh, which writes a .git-credentials file in /facade (as root)
• the container, running as an arbitrary nonroot user, cant access this file, and crashes

This is shown in the logs a little bit before the exception above

[augur]        | Using it in the config
[augur]        | Please unset RABBITMQ_CONN_STRING if you would like to be prompted for the rabbit MQ connection string
[augur]        | 
[augur]        | init with redis redis://redis:6379
[augur]        | touch: cannot touch '/facade/.git-credentials': Permission denied
[augur]        | ./scripts/install/config.sh: line 231: /facade/.git-credentials: Permission denied
[augur]        | ./scripts/install/config.sh: line 232: /facade/.git-credentials: Permission denied
[augur]        | error: could not lock config file /augur/.gitconfig: Permission denied

[MoralCode]

JK, turns out my local setup reverted back to the old volumes, not the recreated ones that made me realize this was largely a PEBCAK.

podman compose down --volumes seems to have fixed the issue

[MoralCode]

when running docker compose logs using Docker version 20.10.24+dfsg1, build 297e128 in a debian 12 environment, it says

validating /opt/git/augur/docker-compose.yml: services.augur Additional property post_start is not allowed

So removing this may also help augur be compatible with more different host OSses that people may choose to run on their servers