S6 overlay v3 scripts order having issue

[tulsis]

Hi Team,

We have an Ubuntu-based image that includes several scripts running as oneshot and longrun services under /etc/s6-overlay/s6-rc.d.
On top of this base image, we’re building another image that adds more scripts.
However, the issue is that the services from the new image start before the base Ubuntu services, whereas the base services are supposed to start first.
Even after adding dependencies, the startup order is still incorrect.

Thanks

[skarnet]

You're not giving me much to work with here.

How are you adding the new scripts? how are you adding the dependencies? where are your service definitions? etc.

[tulsis]

Hi Skarnet,

Apologies for not including the screenshots and other information earlier.
I’ve now added the screenshots for the base image services. Likewise, there are more than 10 services running in the application image, which is built on top of the base image.

base image services

Application services

Please let me know if you need any additional information — I’ll be happy to provide it.

[skarnet]

Your dependency tree looks incomplete. ca-certs does not depend on base, so it will start early, and roche-certs can follow as soon as ca-certs completes. sshd depends on sssd, but I don't know what sssd depends on. If it doesn't depend on base, everything will start early.

All the services that you want to start late need to depend on base, or on something that depends on base.

[tulsis]

After adding it to the base as a dependency, the application is now working as expected. However, there’s an issue with the application services — they are starting before the base image services.

I’ve attached the base image logs and screenshots for your reference.

base_image_logs.txt

[tulsis]

I am attaching the application log for your reference.

app_logs.txt

[skarnet]

Same symptoms, same causes. The services that you see start early, such as static-medata and sslcert, do not depend on base. Everything you add should depend on base.

[tulsis]

I’ve added the required dependencies in my application image, and the image builds successfully. However, I’ve noticed that the base image services are still starting after the application services.

I’ve attached the logs for your reference.

app_logs1.txt

[skarnet]

I don't know what you call "base image services" and what you call "application services".

It's not difficult. If you want service B to start after service A, then you must declare that B depends on A. This is true for all your services.

[tulsis]

We’re building our images using Ubuntu, which includes the necessary libraries such as sssd, Python, and others. The base image contains several essential services like sssd, sshd, and certificate-related services.

Once the base image is built, we create application images (e.g., Jupyter, API, etc.) on top of it.

In s6-overlay v2, this setup worked without any issues — base image services (like sssd) started first to handle identity management, followed by the application services under /etc/cont-init.d.

As part of our upgrade to s6-overlay v3, we’re trying to achieve the same behavior. However, in v3, when we run the application image, the application services start before the base image services, which causes problems.

We’d like to ensure that all base image services start first, and only then the application services should begin.

Could you please guide us on how to achieve this behavior in s6-overlay v3?

[skarnet]

• Make a bundle containing all the services in the base image. mkdir -p /etc/s6-overlay/s6-rc.d/base-services/contents.d ; for i in sssd sshd ca-certs ... ; do touch /etc/s6-overlay/s6-rc.d/base-services/contents.d/$i ; done
• For all your application services that need to start after the base image services, add a dependency from that service to the base image bundle. for i in phc-folder install-flywheel ... ; do mkdir -p /etc/s6-overlay/s6-rc.d/$i/dependencies.d ; touch /etc/s6-overlay/s6-rc.d/$i/dependencies.d/base-services ; done
• That tells s6-rc to only attempt to start the application services after all the base image services have been successfully started.

[tulsis]

I followed your instructions and updated both the base and app images. However, when I build the app image, the logs only show the base image services — the app image services aren’t appearing.

.
├── awscli-setup
│ ├── dependencies.d
│ │ └── sssd
│ ├── type
│ └── up
├── base-services
│ ├── contents.d
│ │ ├── awscli-setup
│ │ ├── ca-certs
│ │ ├── roche-certs
│ │ ├── sshd
│ │ ├── sssd
│ │ └── sssd-config
│ └── type
├── ca-certs
│ ├── dependencies.d
│ │ └── base
│ ├── type
│ └── up
├── roche-certs
│ ├── dependencies.d
│ │ └── ca-certs
│ ├── type
│ └── up
├── sshd
│ ├── dependencies.d
│ │ └── sssd
│ ├── run
│ └── type
├── sssd
│ ├── dependencies.d
│ │ └── sssd-config
│ ├── run
│ └── type
├── sssd-config
│ ├── dependencies.d
│ │ └── roche-certs
│ ├── type
│ └── up
├── user
│ ├── contents.d
│ │ └── base-services
│ └── type
└── user2
├── contents.d
└── type

This is the application structure.

.
├── ace-job
│ ├── dependencies.d
│ │ └── base-services
│ ├── type
│ └── up
├── alblogs
│ ├── dependencies.d
│ │ └── base-services
│ ├── type
│ └── up
├── awscli-setup
│ ├── dependencies.d
│ │ └── sssd
│ ├── type
│ └── up
├── base-services
│ ├── contents.d
│ │ ├── awscli-setup
│ │ ├── ca-certs
│ │ ├── roche-certs
│ │ ├── sshd
│ │ ├── sssd
│ │ └── sssd-config
│ └── type
├── ca-certs
│ ├── dependencies.d
│ │ └── base
│ ├── type
│ └── up
├── homedir
│ ├── dependencies.d
│ │ └── base-services
│ ├── type
│ └── up
├── install-flywheel
│ ├── dependencies.d
│ │ └── base-services
│ ├── type
│ └── up
├── jupyterhub
│ ├── dependencies.d
│ │ └── base-services
│ ├── finish
│ ├── run
│ └── type
├── modify-access
│ ├── dependencies.d
│ │ └── base-services
│ ├── type
│ └── up
├── phc-folder
│ ├── dependencies.d
│ │ └── base-services
│ ├── type
│ └── up
├── roche-certs
│ ├── dependencies.d
│ │ └── ca-certs
│ ├── type
│ └── up
├── set-env-variables
│ ├── dependencies.d
│ │ └── base-services
│ ├── type
│ └── up
├── sshd
│ ├── dependencies.d
│ │ └── sssd
│ ├── run
│ └── type
├── sslcert
│ ├── dependencies.d
│ │ └── base-services
│ ├── type
│ └── up
├── sssd
│ ├── dependencies.d
│ │ └── sssd-config
│ ├── run
│ └── type
├── sssd-config
│ ├── dependencies.d
│ │ └── roche-certs
│ ├── type
│ └── up
├── static-metadata
│ ├── dependencies.d
│ │ └── base-services
│ ├── type
│ └── up
├── user
│ ├── contents.d
│ │ ├── ace-job
│ │ ├── alblogs
│ │ ├── base-services
│ │ ├── homedir
│ │ ├── install-flywheel
│ │ ├── jupyterhub
│ │ ├── modify-access
│ │ ├── phc-folder
│ │ ├── set-env-variables
│ │ ├── sslcert
│ │ └── static-metadata
│ ├── dependencies.d
│ │ └── base
│ └── type
└── user2
├── contents.d
└── type

[skarnet]

What does the container logs say with S6_VERBOSITY=2? Does s6-rc attempt to start the application services, or not at all?

What it looks like is that the files in the user/contents.d part of your application structure are not properly copied to the container and do not appear, are not part of the user bundle, and so s6-rc does not even attempt to start them. But I could be wrong: only logs will tell that.

[tulsis]

I have attached the application image logs for your reference.

jupiter_logs.txt

[skarnet]

Okay. What is happening is that your awscli-setup service is failing. Services that depend on it will only start if it succeeds. If service B depends on service A, then service B will only be started after A succeeds; if A fails, B will not be started at all.

If it does not matter whether it succeeds or fails, and you want to start the application services anyway, then remove awscli-setup from the base-services bundle. The application services will depend on the other base services, but awscli-setup can fail without impacting anything else if nothing depends on it.

[skarnet]

Have you solved your issue? Can I close this?