Debugging MANIFEST_UNKNOWN

Notes

I push an image:

    > docker push registry.gsc.wustl.edu/mcallawa/hello-world
    The push refers to a repository [registry.gsc.wustl.edu/mcallawa/hello-world]
    5f70bf18a086: Pushed
    b652ec3a27e7: Pushed
    latest: digest: sha256:fea8895f450959fa676bcc1df0611ea93823a735a01205fd8622846041d0c7cf size: 708

I try to pull that image from a different location:

    root@blade17-1-1:~# docker pull registry.gsc.wustl.edu/mcallawa/hello-world
    Using default tag: latest
    Pulling repository registry.gsc.wustl.edu/mcallawa/hello-world
    Error: image mcallawa/hello-world not found

Similarly, from the same place I pushed from:

    -> curl -k -L -H "Authorization: Bearer $TOKEN" 'https://10.100.0.20/v2/mcallawa/hello-world/manifests/latest'
    {"errors":[{"code":"MANIFEST_UNKNOWN","message":"manifest unknown","detail":{"Path":"/docker/registry/v2/repositories/mcallawa/hello-world/_manifests/revisions/sha256/fea8895f450959fa676bcc1df0611ea93823a735a01205fd8622846041d0c7cf/signatures/sha256","DriverName":"filesystem"}}]}

So take note of the Path in that error JSON:

  "Path":"/docker/registry/v2/repositories/mcallawa/hello-world/_manifests/revisions/sha256/fea8895f450959fa676bcc1df0611ea93823a735a01205fd8622846041d0c7cf/signatures/sha256"

Now, docker-registry is running in a container where the registry is at /var/lib/registry/

Under that I have:

    /docker/registry/v2/repositories/mcallawa/hello-world/_manifests/revisions/sha256/fea8895f450959fa676bcc1df0611ea93823a735a01205fd8622846041d0c7cf/link

    > cat ./repositories/mcallawa/hello-world/_manifests/revisions/sha256/fea8895f450959fa676bcc1df0611ea93823a735a01205fd8622846041d0c7cf/link
    sha256:fea8895f450959fa676bcc1df0611ea93823a735a01205fd8622846041d0c7cf

If I go looking for the contents of that "link" file in the ./blobs store, I find it:

    > cat ./blobs/sha256/fe/fea8895f450959fa676bcc1df0611ea93823a735a01205fd8622846041d0c7cf/data
    {
       "schemaVersion": 2,
       "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
       "config": {
          "mediaType": "application/octet-stream",
          "size": 1387,
          "digest": "sha256:690ed74de00f99a7d00a98a5ad855ac4febd66412be132438f9b8dbd300a937d"
       },
       "layers": [
          {
             "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
             "size": 601,
             "digest": "sha256:03f4658f8b782e12230c1783426bd3bacce651ce582a4ffb6fbbfa2079428ecb"
          },
          {
             "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
             "size": 32,
             "digest": "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4"
          }
       ]
    }

Once I got the cert thing fixed (/var/lib/boot2docker/certs), I was able to run "docker login" and push from my macbook. I then logged in to blade17-1-10 and ran the image I had just pushed, with no issues.

The image is registry.gsc.wustl.edu/rfugina/hello-world, and I've now run it on blade17-1-9 through 16 without issue.