Excessive amount of image pulls

[awptechnologies]

i have 50 containers running. 28 stacks but of course being docker swarm i have some containers replicated.
Shepherd is going through all 200 of my pulls before its able to update them all.
Is there any way to tone this down?? i have multiple stacks that rely on the same image. For example mariadb. but it still tries to pull a new image for every service even if they are running same image.

[moschlar]

Hm.

I suppose we could optimize that by aggregating the used images to eliminate duplicates - but the actual docker swarm service update will be individually per service, I think.

[shizunge]

What is your top concern?

1. hit the docker hub rate limit https://docs.docker.com/docker-hub/download-rate-limit/ ?
2. or image download uses too much internet bandwidth / server resources ?

The former one is due to

1. Both docker manifest inspect and docker service update contributes to the dockerhub rate usage.
2. Shepherd runs docker manifest inspect on all services, even they have the same image before updating. We do need to check whether there is an update on all services, but we can cache the results.
   As a result, today, Shepherd would consume at least dockerhub rate 56 for your 28 stacks.
   But it seems shepherd does more than I understand, and consumes more than 200 dockerhub rate. Maybe the commands use more than 1 rate.

For later one, I guess (no proof) docker service update https://docs.docker.com/engine/reference/commandline/service_update/ would use the local image if available. But I don't think it accepts the digest of the image to avoid querying the register.

[shizunge]

Run a quick experiment: 2 docker manifest inspect commands consume 1 dockerhub rate.

[moschlar]

FWIW: See the workaround to use a registry mirror at #73 (comment)

[shizunge]

FWIW: See the workaround to use a registry mirror at #73 (comment)

Registry mirroring is also one thing I did. You may use this as a template to setup your own repo https://github.com/shizunge/dockerhub-mirror

[awptechnologies]

Im not 100% up to date on this stuff. What exactly would the registry mirroring do and how would it help in my situation?

[martadinata666]

essentially pull images from docker registry to your github registry. Then deploy the image from github registry, as github registry doesn't have a stupid rate limit.