ERGURNT: s6-overlay failed to start service on docker-1.10.3

[pubyun]

after we upgrade docker 1.9.1 to docker-1.10.3 on centos 7, all docker container failed to start service:

s6-supervise (child): fatal: unable to exec run: Permission denied
s6-supervise (child): fatal: unable to exec run: Permission denied
s6-supervise supervisord: warning: unable to spawn ./run - waiting 10 seconds
s6-supervise sshd: warning: unable to spawn ./run - waiting 10 seconds
s6-supervise (child): fatal: unable to exec run: Permission denied
s6-supervise cron: warning: unable to spawn ./run - waiting 10 seconds

[pubyun]

downgrade to docker 1.9.1, then it's OK

[jprjr]

@pubyun can you provide an example Dockerfile and run script?

[pubyun]

FROM docker.pubyun.com/pubyun/centos

ENV S6_OVERLAY_VERSION v1.18.1.0

ADD file/ /

ADD https://github.com/just-containers/s6-overlay/releases/download/v1.11.0.1/s6-overlay-amd64.tar.gz /tmp/
RUN tar xzf /tmp/s6-overlay-amd64.tar.gz -C /
&& mv /bin /bin-s6
&& rm /usr/bin/execlineb
&& /bin-s6/s6-update-symlinks /bin /bin-s6 /usr/bin
&& ln -s /bin/execlineb /usr/bin/execlineb
&& mv /sbin /sbin-s6
&& /bin-s6/s6-update-symlinks /sbin /sbin-s6 /usr/sbin
&& rm /tmp/s6-overlay-amd64.tar.gz

EXPOSE 22

ENTRYPOINT ["/init"]

[pubyun]

$ cat file/etc/services.d/sshd/run
#!/usr/bin/execlineb -P

/usr/sbin/sshd -D

[pubyun]

type error in Dockfile, i have update to latest s6-overlay, same error:
ADD https://github.com/just-containers/s6-overlay/releases/download/v1.18.1.1/s6-overlay-amd64.tar.gz

[jprjr]

I can't pull from docker.pubyun.com/pubyun/centos, but here's a something to try out.

I think you're doing a lot to work around a known issue: https://github.com/just-containers/s6-overlay#bin-and-sbin-are-symlinks - try just splitting up your tar extraction into two steps

I made this Dockerfile:

FROM centos:7

ENV S6_OVERLAY_VERSION v1.18.1.1

ADD file/ /

ADD https://github.com/just-containers/s6-overlay/releases/download/v1.18.1.1/s6-overlay-amd64.tar.gz /tmp/
RUN tar xzf /tmp/s6-overlay-amd64.tar.gz -C / --exclude="./bin" --exclude="./sbin" \
&& tar xzf /tmp/s6-overlay-amd64.tar.gz -C /usr ./bin ./sbin \
&& yum install -y openssh-server

EXPOSE 22

ENTRYPOINT ["/init"]

I was able to build and run it just fine on docker 1.11.2

[pubyun]

i am runing docker on CentOS 7.2, which have many patches from RedHat.

maybe this patch trigger the problem. it will merge into docker 1.12, but have patched for docker 1.10.3 of redhat:
moby/moby#23301

https://github.com/projectatomic/docker.git

i add one line to Dockerfile, it works now.

VOLUME ["/run"]

can you test it on Docker 1.12?

[jprjr]

It sounds like your problem might be with the CentOS version of Docker? I'm unable to replicate your issue so I'm not sure if there's anything I can do.

[pubyun]

i test package from docker.com, version 1.11 and 1.12, it's OK.
https://yum.dockerproject.org/repo/main/centos/7/
https://yum.dockerproject.org/repo/testing/centos/7/

it fails when i use official package from centos:
http://mirror.centos.org/centos/7/extras

the "/run" directory was locked, and any files can't be executed under it.

i add one line to Dockerfile, then it works:

VOLUME ["/run"]

[pubyun]

i find the reason, there is noexec flag on /run:

sudo docker run --rm=true -ti centos:s6 /bin/bash -c 'mount | grep run'

/dev/mapper/vg0-var on /run/secrets type xfs (rw,relatime,attr2,inode64,noquota)
tmpfs on /run type tmpfs (rw,nosuid,nodev,noexec,relatime,size=65536k,mode=755)

[glerchundi]

@pubyun is this something centos+s6 related? Or just your custom docker image which overrides the /run folder? Thanks

[nomadalex]

@glerchundi when use without /init as ENTRYPOINT, like /bin/bash, the /run is not mount use tmpfs.
but when use /init as ENTRYPOINT, the /run is mount use tmpfs with noexec flag. I don't know why, but it make s6-overlay useless on CentOS 7, my host is CentOS 7 with official docker, docker base container is centos:7. Can you change the service dir for better compatible?

[pubyun]

@glerchundi this is centos7 + s6 related. the script can't run for the noexec flag of /run

[danielunderwood]

I'm having this issue as well on CentOS 7.2 with Docker 1.10.3. It seems to be resolved after updating to Docker 1.12.1 with the directions in the Docker docs. I also forced recreation of the images after updating docker, which may have had some effect on SELinux flags (z on volumes).