Re-create Windows Docker Image by Hand

Setup

Install docker
Install registry.exe from distribution/distribution: The toolkit to pack, ship, store, and deliver container content (github.com)
- Since they don’t have a Windows distribution, install Go.
- Run:
```
go install -v github.com/distribution/distribution/v3/cmd/registry@latest
```

Setup the registry locally

$regPath = "C:\reg"
# choose a directory where the registry will reside, e.g. C:\reg
mkdir $regPath
cd $regPath
# create the two sub-directories [data, config]
mkdir data
mkdir config
# create config file, update the `rootdirectory` path appropriately
Set-Content config\config.yml @"
version: 0.1
log:
  level: debug
  fields:
    service: registry
    environment: development
storage:
  filesystem:
    rootdirectory: "$($regPath -replace '\\', '/')/data"
http:
  addr: :5000
  secret: sirid3v
"@
# start the server
registry.exe serve $regPath\config\config.yml
# you should see something similar to:
# time="2024-06-19T17:39:51.3137525+03:00" level=debug msg="using \"text\" logging formatter"
# time="2024-06-19T17:39:51.315399+03:00" level=info msg="Starting upload purge in 52m0s" environment=development go.version=go1.21.0 # instance.id=4b3e5cae-0d44-4251-969b-9f7e7079e571 service=registry version=v3.0.0-alpha.1
# time="2024-06-19T17:39:51.315399+03:00" level=info msg="redis not configured" environment=development go.version=go1.21.0 # instance.id=4b3e5cae-0d44-4251-969b-9f7e7079e571 service=registry version=v3.0.0-alpha.1
# time="2024-06-19T17:39:51.3183262+03:00" level=info msg="listening on [::]:5000" environment=development go.version=go1.21.0 instance.id=4b3e5cae-0d44-4251-969b-9f7e7079e571 service=registry version=v3.0.0-alpha.1

Setup baseline image

We will start off with the current servercore:ltsc2019 to be our baseline image which we will then patch.

# pull it from the official registry
docker pull mcr.microsoft.com/windows/servercore:ltsc2019
# push it to our local registry
docker tag mcr.microsoft.com/windows/servercore:ltsc2019 localhost:5000/servercore
docker push localhost:5000/servercore
# you should see something similar to:
# Using default tag: latest
# The push refers to repository [localhost:5000/servercore]
# 1b5c07654576: Pushed
# da2d874340bd: Pushed
# latest: digest: sha256:d04f8115dad2fde6b2d8bf327a1422147b4c75c9cdb3e3cdc69cea5d4cfe1146 size: 743

Let’s explore our data directory, there a new subdirectory docker\registry\v2 created there that has all the manifests and tarball files (layers) for the image we just pushed:

cd docker\registry\v2\
PS C:\reg\data\docker\registry\v2> tree .
C:\REG\DATA\DOCKER\REGISTRY\V2
├───blobs
│   └───sha256
│       ├───0d
│       │   └───0d0007bbd07e0d969570201e995b24f19a3dce1efd0f23bb979ba976cb745048
│       ├───56
│       │   └───56a5fd77f8cb6921d3e283f98213bf8c163d3502a75b4a8e4a809a15654f7d1a
│       ├───c9
│       │   └───c9226d61d3bdbf9f09821b32f5878623b8daaa5fb4f875cb63c199f87a26d57e
│       └───d0
│           └───d04f8115dad2fde6b2d8bf327a1422147b4c75c9cdb3e3cdc69cea5d4cfe1146
└───repositories
    └───servercore
        ├───_layers
        │   └───sha256
        │       ├───0d0007bbd07e0d969570201e995b24f19a3dce1efd0f23bb979ba976cb745048
        │       ├───56a5fd77f8cb6921d3e283f98213bf8c163d3502a75b4a8e4a809a15654f7d1a
        │       └───c9226d61d3bdbf9f09821b32f5878623b8daaa5fb4f875cb63c199f87a26d57e
        ├───_manifests
        │   ├───revisions
        │   │   └───sha256
        │   │       └───d04f8115dad2fde6b2d8bf327a1422147b4c75c9cdb3e3cdc69cea5d4cfe1146
        │   └───tags
        │       └───latest
        │           ├───current
        │           └───index
        │               └───sha256
        │                   └───d04f8115dad2fde6b2d8bf327a1422147b4c75c9cdb3e3cdc69cea5d4cfe1146
        └───_uploads

We will be patching if on those leaf nodes in the directory tree.

Patching

1. Patch the base and delta layers

Make sure you have the following files (renamed for clarity), staged somewhere on a local directory, for my case C:\__shuttle\7B:

base.tar.gz    # for the base layer
delta.tar.gz   # for the delta layer
# extra metadata files
delta.tar.gz.manifest.json
delta.tar.gz.config.json
delta.tar.gz.digest # though if you can calculate the SHA's by your own
base.tar.gz.digest

From our C:\reg\data\docker\registry\v2 tree above, we start off with the index which tells us how everything relates: sha256:d04f8115dad2fde6b2d8bf327a1422147b4c75c9cdb3e3cdc69cea5d4cfe1146.

# take note of the SHA256 for the index file:
$oldIndexSHA = "d04f8115dad2fde6b2d8bf327a1422147b4c75c9cdb3e3cdc69cea5d4cfe1146"

{
   "schemaVersion": 2,
   "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
   "config": {
      "mediaType": "application/vnd.docker.container.image.v1+json",
      "size": 786,
      "digest": "sha256:0d0007bbd07e0d969570201e995b24f19a3dce1efd0f23bb979ba976cb745048"
   }
   "layers": [
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 1650620357,
         "digest": "sha256:c9226d61d3bdbf9f09821b32f5878623b8daaa5fb4f875cb63c199f87a26d57e"
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 570060810,
         "digest": "sha256:56a5fd77f8cb6921d3e283f98213bf8c163d3502a75b4a8e4a809a15654f7d1a"
      }
   ]
}

In the staged delta.tar.gz.manifest.json file, we already have our final index file (formatted for readability):

{
    "schemaVersion": 2,
    "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
    "config": {
        "mediaType": "application/vnd.docker.container.image.v1+json",
        "size": 788,
        "digest": "sha256:bf8e16a59e4337c84640f8c1815ddb36dc01d7cdb2cf580f12556c718e9d80cc"
    },
    "layers": [
        {
            "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
            "size": 1604019297,
            "digest": "sha256:cb524f6f22159378ea820d234d80ca09b79c2f0cc91315eeef11904e3ff36a21"
        },
        {
            "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
            "size": 574224024,
            "digest": "sha256:0dbe305fcd4fddcf1d31170ba2a50504e9d1556971a4f83f35dd83d97606f3e6"
        }
    ]
}

This content-addressing, so everything is referenced by it’s SHA256, for the above index file above for instance, you can see that it’s SHA256 is the same as the file directory name:

$filepath = ".\blobs\sha256\d0\d04f8115dad2fde6b2d8bf327a1422147b4c75c9cdb3e3cdc69cea5d4cfe1146\data"
(Get-FileHash -Path $filepath -Algorithm SHA256).Hash.ToLower()
# d04f8115dad2fde6b2d8bf327a1422147b4c75c9cdb3e3cdc69cea5d4cfe1146

Let’s start off by patching the layers first.

# copy the tarballs
$basePath = "C:\__shuttle\7B\base.tar.gz"
$deltaPath = "C:\__shuttle\7B\delta.tar.gz"
$manifestFile = "C:\__shuttle\7B\delta.tar.gz.manifest.json" # aka, the new index file
$configFile = "C:\__shuttle\7B\delta.tar.gz.config.json"

$baseSHA = (Get-Content C:\__shuttle\7B\base.tar.gz.digest).Split(':')[1]
$deltaSHA = (Get-Content C:\__shuttle\7B\delta.tar.gz.digest).Split(':')[1]
echo $baseSHA $deltaSHA
# cb524f6f22159378ea820d234d80ca09b79c2f0cc91315eeef11904e3ff36a21
# 0dbe305fcd4fddcf1d31170ba2a50504e9d1556971a4f83f35dd83d97606f3e6

Prepare the blob directories and copy over the tarball files:

cd .\data\docker\registry\v2\blobs\sha256\ # for short paths

$baseBlobPath = $baseSHA.Substring(0,2) + "/" + $baseSHA
$deltaBlobPath = $deltaSHA.Substring(0,2) + "/" + $deltaSHA

mkdir $baseBlobPath
mkdir $deltaBlobPath

# copy
cp $basePath $baseBlobPath\data
cp $deltaPath $deltaBlobPath\data

2. Patch the Config File

# get the new hashes
$manifestSHA = (Get-FileHash -Path $manifestFile -Algorithm SHA256).Hash.ToLower()
$configSHA = (Get-FileHash -Path $configFile -Algorithm SHA256).Hash.ToLower()

# create the blob directories
$manifestBlobPath = $manifestSHA.Substring(0,2) + "/" + $manifestSHA
$configBlobPath = $configSHA.Substring(0,2) + "/" + $configSHA
mkdir $configBlobPath
mkdir $manifestBlobPath

# copy over the files
cp $configFile $configBlobPath/data
cp $manifestFile $manifestBlobPath/data

3. Update the metadata files in `repositories\servercore\*`

Remember that the manifestFile is the index file; and it's SHA256 is in $manifestSHA.

cd ..\..\repositories\servercore\
# patch manifests
# index references
Rename-Item .\_manifests\tags\latest\index\sha256\$oldIndexSHA $manifestSHA
# update the link files
"sha256:$manifestSHA" | Out-File .\_manifests\tags\latest\index\sha256\$manifestSHA\link -NoNewline
"sha256:$manifestSHA" | Out-File .\_manifests\tags\latest\current\link -NoNewline

# revision references
Rename-Item .\_manifests\revisions\sha256\$oldIndexSHA $manifestSHA
"sha256:$manifestSHA" | Out-File .\_manifests\revisions\sha256\$manifestSHA\link -NoNewline

# patch _layers references
rm -Recurse _layers

# config file ref
mkdir _layers/sha256/$configSHA
"sha256:$configSHA" | Out-File .\_layers\sha256\$configSHA\link -NoNewline

# base layer
mkdir _layers/sha256/$baseSHA
"sha256:$baseSHA" | Out-File .\_layers\sha256\$baseSHA\link -NoNewline

# delta layer
mkdir _layers/sha256/$deltaSHA
"sha256:$deltaSHA" | Out-File .\_layers\sha256\$deltaSHA\link -NoNewline

Pull the patched Image

We can now pull back the patched image from the local registry and test it.

docker pull localhost:5000/servercore

# Using default tag: latest
# latest: Pulling from servercore
# cb524f6f2215: Already exists
# 0dbe305fcd4f: Extracting [============================>                      ]  325.9MB/574.2MB

# 0dbe305fcd4f: Pull complete
# Digest: sha256:dc66f1b8e8174103a1692d22b8303154341741074bac2a9eebad3466e66d6095
# Status: Downloaded newer image for localhost:5000/servercore:latest
# localhost:5000/servercore:latest

# run the image
docker run -it localhost:5000/servercore

profnandaa/recreate_windows_docker_image.md