Codefresh image registry manifest-tool push errors

codefresh-manifest-tool-errors.md

Codefresh has a Docker v2-API supporting image registry that is "Powered by Google Container Registry (gcr.io)" (per blog post link).

While gcr.io has its own issues with support of manifest list objects defined in the v2.2 image spec, Codefresh registry seems to have slightly different support than the current implementation of GCR, in that cross-repo blob mount/manifest references don't work at all (they do in GCR); but both registries refuse to accept the media type for a manifest list PUT API call.

Attempting cross-repository push to codefresh registry

This YAML file tries to push a manifest list object which references four images from other repositories within the same namespace; the mount of the cross-repo blobs (which should be a V2-supported API call) gets a 500 HTTP response:

$ ./manifest-tool --debug push from-spec alpine-codefresh.yaml 
DEBU[0000] endpoints: [{false https://r.cfcr.io v2 false true 0xc42028a180} {false http://r.cfcr.io v2 false true 0xc42028a180} {false https://r.cfcr.io v1 false true 0xc42028a300} {false http://r.cfcr.io v1 false true 0xc42028a300}] 
DEBU[0000] repoName: estesp/alpine                      
INFO[0000] Retrieving digests of images...              
DEBU[0000] authConfig for r.cfcr.io: estesp             
DEBU[0000] endpoints: [{false https://r.cfcr.io v2 false true 0xc420316300} {false http://r.cfcr.io v2 false true 0xc420316300}] 
DEBU[0000] Trying to fetch image manifest of r.cfcr.io/estesp/alpine_ppc64le repository from https://r.cfcr.io v2 
INFO[0002] Image "r.cfcr.io/estesp/alpine_ppc64le:latest" is digest sha256:15780ed5342bafabe42cb609d2ca62534c12c1ae7681b10b978d2a8e781e0957; size: 528 
DEBU[0002] Adding manifest references of "r.cfcr.io/estesp/alpine_ppc64le:latest" to blob mount requests 
DEBU[0002] Adding manifest "estesp/alpine_ppc64le" -> to be pushed to "estesp/alpine" as a manifest reference 
DEBU[0002] authConfig for r.cfcr.io: estesp             
DEBU[0002] endpoints: [{false https://r.cfcr.io v2 false true 0xc420330000} {false http://r.cfcr.io v2 false true 0xc420330000}] 
DEBU[0002] Trying to fetch image manifest of r.cfcr.io/estesp/alpine_amd64 repository from https://r.cfcr.io v2 
INFO[0004] Image "r.cfcr.io/estesp/alpine_amd64:latest" is digest sha256:1072e499f3f655a032e88542330cf75b02e7bdf673278f701d7ba61629ee3ebe; size: 528 
DEBU[0004] Adding manifest references of "r.cfcr.io/estesp/alpine_amd64:latest" to blob mount requests 
DEBU[0004] Adding manifest "estesp/alpine_amd64" -> to be pushed to "estesp/alpine" as a manifest reference 
DEBU[0004] authConfig for r.cfcr.io: estesp             
DEBU[0004] endpoints: [{false https://r.cfcr.io v2 false true 0xc42028ac00} {false http://r.cfcr.io v2 false true 0xc42028ac00}] 
DEBU[0004] Trying to fetch image manifest of r.cfcr.io/estesp/alpine_s390x repository from https://r.cfcr.io v2 
INFO[0006] Image "r.cfcr.io/estesp/alpine_s390x:latest" is digest sha256:4313f086b6ec7a18c21effbb86b01a643d08529c79684c823655c3136a2e3887; size: 528 
DEBU[0006] Adding manifest references of "r.cfcr.io/estesp/alpine_s390x:latest" to blob mount requests 
DEBU[0006] Adding manifest "estesp/alpine_s390x" -> to be pushed to "estesp/alpine" as a manifest reference 
DEBU[0006] authConfig for r.cfcr.io: estesp             
DEBU[0006] endpoints: [{false https://r.cfcr.io v2 false true 0xc420302300} {false http://r.cfcr.io v2 false true 0xc420302300}] 
DEBU[0006] Trying to fetch image manifest of r.cfcr.io/estesp/alpine_aarch64 repository from https://r.cfcr.io v2 
INFO[0008] Image "r.cfcr.io/estesp/alpine_aarch64:latest" is digest sha256:b06041f25eb362e78cac279242fa2ffb548a6788dd1663a45bf4b5b01994a8ce; size: 528 
DEBU[0008] Adding manifest references of "r.cfcr.io/estesp/alpine_aarch64:latest" to blob mount requests 
DEBU[0008] Adding manifest "estesp/alpine_aarch64" -> to be pushed to "estesp/alpine" as a manifest reference 
DEBU[0008] Manifest list push url: https://r.cfcr.io/v2/estesp/alpine/manifests/latest 
DEBU[0008] mediaType of manifestList: application/vnd.docker.distribution.manifest.list.v2+json 
DEBU[0008] authConfig for r.cfcr.io: estesp             
FATA[0008] Couldn't mount blobs for cross-repository push: V2 registry POST of blob mount failed: Post https://r.cfcr.io/v2/estesp/alpine/blobs/uploads/?from=estesp%2Falpine_ppc64le&mount=sha256%3A94de54a1384eecad706e385f2e4c4a4d050a5e87643b35625b86cabec49e636b: received unexpected HTTP status: 500 Internal Server Error 

Attempt direct push of a manifest list with no cross-repo references to codefresh registry

This YAML file tries a simpler manifest list object which references four images in the same target repo, allowing it to work without any cross-repo references. The PUT of the manifest list media type/object fails with an HTTP 500 reponse:

$ ./manifest-tool --debug push from-spec alpine-codefresh.yaml 
DEBU[0000] endpoints: [{false https://r.cfcr.io v2 false true 0xc420354000} {false http://r.cfcr.io v2 false true 0xc420354000} {false https://r.cfcr.io v1 false true 0xc420326180} {false http://r.cfcr.io v1 false true 0xc420326180}] 
DEBU[0000] repoName: estesp/alpine                      
INFO[0000] Retrieving digests of images...              
DEBU[0000] authConfig for r.cfcr.io: estesp             
DEBU[0000] endpoints: [{false https://r.cfcr.io v2 false true 0xc420326300} {false http://r.cfcr.io v2 false true 0xc420326300}] 
DEBU[0000] Trying to fetch image manifest of r.cfcr.io/estesp/alpine repository from https://r.cfcr.io v2 
INFO[0001] Image "r.cfcr.io/estesp/alpine:ppc64le" is digest sha256:15780ed5342bafabe42cb609d2ca62534c12c1ae7681b10b978d2a8e781e0957; size: 528 
DEBU[0001] authConfig for r.cfcr.io: estesp             
DEBU[0001] endpoints: [{false https://r.cfcr.io v2 false true 0xc420326780} {false http://r.cfcr.io v2 false true 0xc420326780}] 
DEBU[0001] Trying to fetch image manifest of r.cfcr.io/estesp/alpine repository from https://r.cfcr.io v2 
INFO[0004] Image "r.cfcr.io/estesp/alpine:amd64" is digest sha256:1072e499f3f655a032e88542330cf75b02e7bdf673278f701d7ba61629ee3ebe; size: 528 
DEBU[0004] authConfig for r.cfcr.io: estesp             
DEBU[0004] endpoints: [{false https://r.cfcr.io v2 false true 0xc420326c00} {false http://r.cfcr.io v2 false true 0xc420326c00}] 
DEBU[0004] Trying to fetch image manifest of r.cfcr.io/estesp/alpine repository from https://r.cfcr.io v2 
INFO[0005] Image "r.cfcr.io/estesp/alpine:s390x" is digest sha256:4313f086b6ec7a18c21effbb86b01a643d08529c79684c823655c3136a2e3887; size: 528 
DEBU[0005] authConfig for r.cfcr.io: estesp             
DEBU[0005] endpoints: [{false https://r.cfcr.io v2 false true 0xc42028c600} {false http://r.cfcr.io v2 false true 0xc42028c600}] 
DEBU[0005] Trying to fetch image manifest of r.cfcr.io/estesp/alpine repository from https://r.cfcr.io v2 
INFO[0007] Image "r.cfcr.io/estesp/alpine:aarch64" is digest sha256:b06041f25eb362e78cac279242fa2ffb548a6788dd1663a45bf4b5b01994a8ce; size: 528 
DEBU[0007] Manifest list push url: https://r.cfcr.io/v2/estesp/alpine/manifests/latest 
DEBU[0007] mediaType of manifestList: application/vnd.docker.distribution.manifest.list.v2+json 
DEBU[0007] authConfig for r.cfcr.io: estesp             
FATA[0007] V2 registry PUT of manifest list failed: Put https://r.cfcr.io/v2/estesp/alpine/manifests/latest: received unexpected HTTP status: 500 Internal Server Error