Upconvert remote schema 1 images

[jonjohnsonjr]

It's not too hard to convert a schema 1 image to a schema 2 image. We can do this whenever we encounter a schema 1 image in remote.Image in order to support pulling older images or pulling from registries that don't support schema 2 yet.

non-goal: Pushing schema 1 images. These are legacy and we shouldn't help keep them alive.

See: GoogleContainerTools/kaniko#509

It's expensive to do this conversion because we have to gunzip and sha256 each layer to compute the DiffIDs. In some cases we can do this lazily. In other cases, doing this expensive conversion is better than nothing.

[xingao267]

Hi @jonjohnsonjr, I'm looking at migrating rules_docker to base on this repo instead of the python version of it. Just to confirm the schema 1 you would like to upconvert is https://docs.docker.com/registry/spec/manifest-v2-1/ right, not https://github.com/moby/moby/blob/master/image/spec/v1.md? So basically, this go-containerregistry repo only supports https://docs.docker.com/registry/spec/manifest-v2-2? I got a little confused because of the naming of v1 folder in this repo.

[jonjohnsonjr]

I see, yeah that is confusing. The v1 folder here refers to the OCI version, e.g. https://github.com/opencontainers/image-spec/blob/master/media-types.md

The images we'd upconvert are v2 schema 1; we currently only support v2 schema 2 (AKA OCI images, kind of).

Why do you ask?

[xingao267]

Thanks for the confirmation. Our team maintains rules_docker, and we are thinking of migrating rules_docker from the python containerregistry to this one to get better support and solve the py2/3 compatibility issue altogether. I'm currently evaluating the differences between the two containerregistry repos and see if there are feature parity and estimate the gaps we need to fill, which means we might need to request features and sending PRs in this repo if needed. I'm also putting up an internal document. I'll share that with you once finished. Thanks!

[glyn]

I hit this with some Istio-related images (prom/statsd-exporter and quay.io/coreos/hyperkube).

@jonjohnsonjr Apart from the performance overhead, which is "pay as you go", you mentioned on slack that there are some disadvantages of implementing this issue. Please could you elaborate?

[jonjohnsonjr]

Schema 1 images are generally super ancient and will have tons of unaddressed vulnerabilities. The exception to this is quay.io, which for reasons I cannot understand didn't implement schema 2 support until very recently. Poking around, most images are still schema1 so it seems not to be completely rolled out yet?

It's mostly a philosophical point, but by not supporting schema 1 in this library, we're shining a light on unexpected dependencies. The prom/statsd-exporter:v0.6.0 gets dropped in the latest istio release, rightly.

Istios' hyperkube dependency is weirder, it seems like they just use it as a bash shell that happens to have kubectl installed. That doesn't seem to be the purpose of hyperkube, which AFAICT is basically busybox but for kubernetes API components.

I guess I'm reluctant to enable people to continue to depend on schema 1 images, because people should basically never be depending on schema 1 images, except for quay, which I think is being fixed.

FWIW I started playing with this a while ago to see how annoying it would be:
master...jonjohnsonjr:schema1

I don't want to merge that in because it's not well tested, but it seems to work as a workaround if my protests aren't convincing enough :)

[glyn]

Schema 1 images are generally super ancient and will have tons of unaddressed vulnerabilities. The exception to this is quay.io, which for reasons I cannot understand didn't implement schema 2 support until very recently. Poking around, most images are still schema1 so it seems not to be completely rolled out yet?

It's mostly a philosophical point, but by not supporting schema 1 in this library, we're shining a light on unexpected dependencies. The prom/statsd-exporter:v0.6.0 gets dropped in the latest istio release, rightly.

Istios' hyperkube dependency is weirder, it seems like they just use it as a bash shell that happens to have kubectl installed. That doesn't seem to be the purpose of hyperkube, which AFAICT is basically busybox but for kubernetes API components.

I guess I'm reluctant to enable people to continue to depend on schema 1 images, because people should basically never be depending on schema 1 images, except for quay, which I think is being fixed.

Of the published images for quay.io/coreos/hyperkube, the highest tag (v1.10.5_coreos.0, 2 Jul 2018) and the most recent tag (v1.9.6_coreos.2, 2 Feb 2019) still use schema v1. So it may be a while until that image is available with schema v2.

FWIW I started playing with this a while ago to see how annoying it would be:
master...jonjohnsonjr:schema1

I don't want to merge that in because it's not well tested, but it seems to work as a workaround if my protests aren't convincing enough :)

The conversion seems to work (slowly). However, not surprisingly, the image digest isn't preserved, which is another reason for not merging IMO.

[josephschorr]

Schema 1 images are generally super ancient and will have tons of unaddressed vulnerabilities. The exception to this is quay.io, which for reasons I cannot understand didn't implement schema 2 support until very recently. Poking around, most images are still schema1 so it seems not to be completely rolled out yet?

(Tech lead and cofounder of Quay here)

Long story aside as to why it has taken us significant time to rollout Schema 2 support (best told in a talk I intend to give at some point soon), even once Schema 2 support is fully rolled out, we are not going to be up-converting Schema 1 manifests into Schema 2 manifests, as it would change the content digests for existing manifests which are attached to tags.

The expectation, as per our read of the Schema 1 -> Schema 2 migration, has always been that clients will remain supporting both versions for a significant period of time after push support for Schema 1 has been deprecated and removed from Docker and other clients (which seems to be sometime this month for Docker).

If we find this to not be the case, we might reassess and expand our conversion code to return a Schema 2 form of Schema 1 manifests when supported, but as this would, as mentioned, change the manifest content digest, we'd highly prefer not to do so, as it will break image signing/verification by users.