Fix Complement CI not running against the code from our PRs (remote images being chosen over local)

[MadLittleMods]

Fix remote images being chosen over the local ones we just built with Complement in CI (any Docker environment using the containerd image store). This problem means that Complement jobs in CI don't actually test against the code from the PR (since 2026-02-10).

This PR approaches the problem the same way that @AndrewFerr proposed in #18210. This is better than the alternative listed below as we can just make our code compatible with whatever image store is being used.

Problem

Spawning from #19460 (comment) where we found that our Complement jobs in CI don't actually test against the code from the PR at the moment.

This is caused by a change in Docker Engine 29.0.0:

containerd image store is now the default for fresh installs. This doesn't apply to daemons configured with userns-remap (see moby#47377).

-- 29.0.0 (2025-11-10), https://docs.docker.com/engine/release-notes/29/#2900

And our ubuntu-latest GitHub runner (Current runner version: '2.331.0') points to using Docker client/server 29.1.5 🎯

This Docker version bump happened on actions/runner-images@416418d (2026-02-10) (28.0.4 -> 29.1.5). Specific PR: actions/runner-images#13633

---

I found this because I reviewed and remembered #18210 was a thing that @AndrewFerr ran into. And then running dockers system prune also revealed the problematic containerd in CI. Checking the Docker changelogs, I found the new default culprit and then could trace down where the GitHub runners made the dependency update.

Alternatives

Use non-published tags

We could use tags that aren't published like matrixdotorg/synapse:testing.

Disable the containerd image store

Looking at the PR's cross-linked from actions/runner-images#13633, I saw NixOS/nix#15252 which disables the containerd image store. Seems like it could work but we might as well update our code to be compatible with whatever image store is being used (especially when someone has already put in the work to figure it out).

      - name: Disable containerd image store
        run: |
          echo '{"features":{"containerd-snapshotter":false}}' | sudo tee /etc/docker/daemon.json > /dev/null
          sudo systemctl restart docker

Dev notes

I've given a heads-up to other developers about the breaking change, actions/runner-images#13633 (comment)

Todo

• Update docs like complement.sh: don't use remote base synapse image #18210
• Add in-repo Complement test that checks that the current commit sha matches the one that Synapse is using (/version endpoint), discussion below
  • -> handled as a separate PR: Add in-repo Complement test to sanity check Synapse version matches git checkout #19476

Pull Request Checklist

• Pull request is based on the develop branch
• Pull request includes a changelog file. The entry should:
  • Be a short description of your change which makes sense to users. "Fixed a bug that prevented receiving messages from other servers." instead of "Moved X method from EventStore to EventWorkerStore.".
  • Use markdown where necessary, mostly for code blocks.
  • End with either a period (.) or an exclamation mark (!).
  • Start with a capital letter.
  • Feel free to credit yourself, by adding a sentence "Contributed by @github_username." or "Contributed by [Your Name]." to the end of the entry.
• Code style is correct (run the linters)

[MadLittleMods]

One difference from #18210 is that I've updated everything we build to use the local dummy registry namespace.

This way we don't have to worry about the other images being published or not at some point in the future. It's all just namespaced and pulls from the same place across the board.

[MadLittleMods]

Ideally, to detect this kind of problem, we'd have an in-repo Complement test that checks if the current git commit sha matches the one that Synapse is using for Complement (via /version endpoint). I'm tackling this separately as #19476

[MadLittleMods]

Overall, this docs section doesn't seem that useful. To make this PR whole, I'm updating the docs but I think it's worth ripping this all out (just point to complement.sh and those docs) in a follow-up PR.

[MadLittleMods]

Related discussion below with some favor on removing this section, #19475 (comment)

[reivilibre]

Is it worth just dropping this 'global' tag here and keeping only the local tag?

Since it's a build that was never published, it feels almost preferable to not give it the global tag

[MadLittleMods]

Updated everything to just use localhost/... straight-up.

We do publish the complement-synapse image built from complement.sh but if I squint at the CI code there, I think it still works:

synapse/.github/workflows/push_complement_image.yml

Lines 56 to 74 in 3833eb4

	- name: Work out labels for complement image
	id: meta
	uses: docker/metadata-action@c299e40c65443455700f0fdfc63efafe5b349051 # v5.10.0
	with:
	images: ghcr.io/${{ github.repository }}/complement-synapse
	tags: |
	type=schedule,pattern=nightly,enable=${{ github.event_name == 'schedule'}}
	type=raw,value=develop,enable=${{ github.event_name == 'schedule' || inputs.branch == 'develop' }}
	type=raw,value=latest,enable=${{ github.event_name == 'push' || inputs.branch == 'master' }}
	type=sha,format=long
	- name: Run scripts-dev/complement.sh to generate complement-synapse:latest image.
	run: scripts-dev/complement.sh --build-only
	- name: Tag and push generated image
	run: |
	for TAG in ${{ join(fromJson(steps.meta.outputs.json).tags, ' ') }}; do
	echo "tag and push $TAG"
	docker tag complement-synapse $TAG
	docker push $TAG
	done

[reivilibre]

hmm, would this need to say

docker tag localhost/complement-synapse $TAG 

for that to work?

[MadLittleMods]

Just seeing this now (not sure how I missed).

Good spot! I think the answer is yes based on these CI failures:

❌ https://github.com/element-hq/synapse/actions/runs/22609655282/job/65509315002#step:8:39

Error response from daemon: No such image: complement-synapse:latest

[MadLittleMods]

Fixed in #19523

[reivilibre]

Thanks! Some minor points but I'm not fussy about them

[MadLittleMods]

Thanks for the review @reivilibre 🦜