APFS Container cloning/replicating under macOS 11.0 (Big Sur) - with a bootable system

readme.md

It is the year 2020 and replicating APFS containers still sucks. One would expect it would be a simple copy and paste in the Disk Utility app but this is still far from reality.

Last year I wrote how I managed to clone my macOS system under Catalina. The main trick was to create a DMG file with multiple volumes, mount it on target machine and drop to command-line to do asr restore from synthetised disk while avoiding possible pitfalls.

The good news is that Apple devs definitely worked on improving this under Big Sur and added some documentation (see man asr). But I didn't understand it fully on first read. Maybe someone could explain how is this supposed to work?

Unfortunately they broke the workflow I described in the previous article. The problem is that asr newly tests source of restore operation and if it contains multiple system/data volumes then it simply gives up. For example:

> diskutil list
/dev/disk0 (internal, physical):
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      GUID_partition_scheme                        *500.3 GB   disk0
   1:                        EFI ⁨EFI⁩                     209.7 MB   disk0s1
   2:                 Apple_APFS ⁨Container disk1⁩         499.4 GB   disk0s2
   3:           Windows Recovery ⁨⁩                        552.6 MB   disk0s3
   4:                 Apple_Boot ⁨⁩                        134.2 MB   disk0s4

/dev/disk1 (synthesized):
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      APFS Container Scheme -                      +499.4 GB   disk1
                                 Physical Store disk0s2
   1:                APFS Volume ⁨BismarckOS - Data⁩       439.6 GB   disk1s1
   2:                APFS Volume ⁨Preboot⁩                 82.1 MB    disk1s2
   3:                APFS Volume ⁨Recovery⁩                529.0 MB   disk1s3
   4:                APFS Volume ⁨VM⁩                      2.1 GB     disk1s4
   5:                APFS Volume ⁨BismarckOS⁩              11.5 GB    disk1s5
   6:                APFS Volume ⁨Nix Store⁩               616.0 MB   disk1s6

...
/dev/disk6 (external):
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      GUID_partition_scheme                         1.0 TB     disk6
   1:                        EFI ⁨EFI⁩                     209.7 MB   disk6s1
   2:                 Apple_APFS ⁨Container disk7⁩         500.8 GB   disk6s2
   3:                 Apple_APFS ⁨Container disk8⁩         499.5 GB   disk6s3
   
/dev/disk7 (synthesized):
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      APFS Container Scheme -                      +500.8 GB   disk7
                                 Physical Store disk6s2
...

> asr restore --source /dev/disk1 --target /dev/disk7 --erase
	Validating target...done
	Validating source...
Source "/dev/disk1" contains multiple system/data volumes.
You must pass --sourcevolumename or --sourcevolumeUUID to specify which volume to restore.
Could not validate source - Invalid argument

Not sure if non-standard ⁨Nix Store volume has anything to with it. Feel free to comment with your experiences.

And if you try to pass in --sourcevolumename, it answers in gibberish:

> asr restore --source /dev/disk1 --target /dev/disk7 --erase --sourcevolumename BismarckOS
	Validating target...done
	Validating source...done
	Erase contents of /dev/disk7 ()? [ny]: y

Couldn't set up partitions on target device - operation AddAPFSVolumeToContainer, line #5334 - error 49165

Well, we could try to specify container partition as the source of restore operation. I'm 99% sure this didn't work last year...

> sudo asr restore --source /dev/disk0s2 --target /dev/disk7 --erase
Password:
	Validating target...done
	Validating source...
Source "/dev/disk1" contains multiple system/data volumes.
You must pass --sourcevolumename or --sourcevolumeUUID to specify which volume to restore.
Could not validate source - Invalid argument

Hmm... what about this?

> sudo asr restore --source /dev/disk0s2 --target /dev/disk7 --erase --sourcevolumename BismarckOS
	Validating target...done
	Validating source...done
	Erase contents of /dev/disk7 ()? [ny]: y
	Replicating ....10....20....30....40....50....60....70....80....90....100
	Replicating ....10....20....30....40....50....60....70....80....90....100
	Restored target device is /dev/disk7s2.
Restore completed successfully.

Wut?

> diskutil list
...
/dev/disk7 (synthesized):
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      APFS Container Scheme -                      +500.8 GB   disk7
                                 Physical Store disk6s2
   1:                APFS Volume ⁨BismarckOS - Data⁩       439.5 GB   disk7s1
   2:                APFS Volume ⁨BismarckOS⁩              11.5 GB    disk7s2
   3:                APFS Volume ⁨Preboot⁩                 82.1 MB    disk7s3
   4:                APFS Volume ⁨Recovery⁩                533.8 MB   disk7s4
...

This /dev/disk7 system turned out to be bootable and was a good copy of the previous system (as expected).

Observations

1. It is possible to clone/copy APFS-resident bootable macOS system from command-line.
2. I don't know how to do it with DMGs anymore. Instead, by accident, I found a way how to do it directly with asr restore disk-to-disk (I used target disk mode to connect old machine to new machine).
3. asr restore has a custom logic how to copy system/data volumes, it does not do what would be normally expected, I mean: "to copy all volumes one-by-one". See that Nix Store⁩ and VM volumes were not copied/restored.

Please note that all this info is relevant to public release of macOS 11.0.1 (Big Sur). I expect this behaviour to be subject of future changes.

Pro tip: If you need to boot into some other macOS system to do some asr work please make sure you are using latest versions of these tools because they differ between OS releases.

> sw_vers
ProductName:	macOS
ProductVersion:	11.0.1
BuildVersion:	20B29

I had a disaster during Catalina security update and ended up with unbootable system. When I inspected the system volume it was empty, but the installation message said that it could not remove previous system. So I ended up creating a R/W container image and reinstalling the system from scratch. Is it now possible to restore only the DATA volume? And will all user/system settings be there? Another interesting thing is that the DMG system size shows 10.5 GB, but the volume is actually empty:

/dev/disk18 (synthesized):
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      APFS Container Scheme -                      +249.7 GB   disk18
                                 Physical Store disk17
   1:                APFS Volume CAT1 - Data             193.3 GB   disk18s1
   2:                APFS Volume CAT1                    10.5 GB    disk18s2
   3:                APFS Volume Preboot                 23.8 MB    disk18s3
   4:                APFS Volume Recovery                20.5 KB    disk18s4
   5:                APFS Volume VM                      4.3 GB     disk18s5
   6:                APFS Volume Update                  1.3 MB     disk18s6

fascinating…

I'm trying to figure out how to make a system/data pair that are unencrypted and then set them to be encrypted like enabling File Vault does in the UI, but from the command line.

Reading Apple File System Reference it appears that each of the two volumes is encrypted and then their keys (which might include disk and multiple user keys) are wrapped and encrypted with the container key (I think) and stored in the container key bag. Unclear how to do this using command line tools (only documented at a much lower level).

Following this https://gist.github.com/darwin/3c92ac089cf99beb54f1108b2e8b4b9f#gistcomment-3306783 instead did the trick for me.

I found this process that might be helpful for others:

Booted to another external USB disk with Big Sur OS on it.. Assuming the drive you want to copy is disk1s3
-----Create DMG of Big Sur Machine----

#!/bin/sh
#Create SparseBundle to use to copy Image To
hdiutil create /tmp/DSNetworkRepository/Masters/APFS/TempImage.sparsebundle -volname "MacPrep" -fs apfs -size 50GB
hdiutil mount /tmp/DSNetworkRepository/Masters/APFS/TempImage.sparsebundle
#Figure out the disk number of the mounted temp image
disknumber=$(diskutil list MacPrep | grep Scheme | awk {'print $8'})
diskutil apfs deletevolume MacPrep
echo $disknumber
#Get snapshotID of BootDrive
snapshotID=$(diskutil apfs listSnapshots /dev/disk1s3 | grep "Name" | awk {'print $2'})
echo "using drive snapshot: "$snapshotID
asr --source /dev/disk1s3 --target /dev/$disknumber -toSnapshot $snapshotID
#unmount restored disk in order to convert it
diskutil unmountdisk $disknumber
#Pause 5 seconds while waiting for Unmount
sleep 5
#Eject the Image File
hdiutil detach $disknumber
sleep 5
#convert from sparseBundle to DMG
hdiutil convert /tmp/DSNetworkRepository/Masters/APFS/TempImage.sparsebundle -format UDZO -o /tmp/DSNetworkRepository/Masters/APFS/TempImage-Converted.DMG

exit 0

-------------RESTORE DMG Back to Disk---
#!/bin/sh

imagefile="BigSur_11.1_OS.dmg"
imagepath="/Location/To/DMG/"
drivelabel="Macintosh HD"

#Delete and Re-partition creating an APFS Container and temp volume, then delete Temp Volume
diskutil partitionDisk /dev/disk0 1 gpt apfs "MacOSPrep" 100%
#Get Disk Number of new APFS Volume
targetdisk=$(diskutil list MacOSPrep | grep Scheme | awk {'print $8'})
echo "Targetdisk: "$targetdisk
diskutil apfs deletevolume MacOSPrep
sleep 2
#Restore DMG - Must be running macOS 11.1 for next step to work and restore both System and Data Drives
#Mount DMG as Read/Write using shadow file and Get the name of the disk mounted
echo "using "$imagepath/$imagefile" with drive label: "$drivelabel
sourcedisk=$(hdiutil mount $imagepath/$imagefile -shadow /tmp/img.shadow -noverify | grep -Ei "$drivelabel" | grep -Eiv " Data" | awk {'print $1'})
sleep 2
echo "DMG mounted source drive as:" $sourcedisk
#Get snapshotID of BootDrive
snapshotID=$(diskutil apfs listSnapshots $sourcedisk | grep "Name" | awk {'print $2'})
echo "using drive snapshot: "$snapshotID
asr --source $sourcedisk --target /dev/$targetdisk -toSnapshot $snapshotID
#Unmount the DMG so its volume name doesn't conflict
hdiutil detach $sourcedisk
#pause 2 seconds to allow detach
sleep 2
#make sure Disk is Mounted to configure the rest
diskutil mountDisk disk1
#Pause 2 seconds to allow disk to mount
sleep 2
exit 0

THANK YOU! I’ve been looking for a solution for weeks! I can tell this works in 11.2.3 at a Mac with T2 chip.

Excellent explanations. I'm still trying to make an asr restore from a read-only system volume in a Big Sur (for forensic reasons). It seams to work for the first system volume, then it fails when it comes to the data volume (Volume replication failed - read-only file system). What makes no sense, since reading is all It's needed. An "asr restore" bug, perhaps? All seams to work fine when the volume is mounted read-write. Any suggestions that can permit a less intrusive method (forensically sound)?

Following this https://gist.github.com/darwin/3c92ac089cf99beb54f1108b2e8b4b9f#gistcomment-3306783 instead did the trick for me.

thank you for confirming this!!! @yangx2000’s instructions were very clear and very strightforward and intuitive as you would expect it to work. I didn’t even need to bless the drive - it was a clone and boot!

the only snafu I ran into was getting a new partition to format to APFS on mixed-partition (ext4, hfs, APFS) drive - it would only format to FAT32 or exFAT. Disk Utility would throw a “Couldn’t modify partition map.: -69874” every time I chose “APFS” (or HFS+). First Aid on the partition would report “Invalid BS_jmpBoot in boot block: 000000”

The “trick” was to run diskutil repairDisk which shrunk the previous partition about 127MB to create room for “loader space”

Then I was able to format the partition to APFS in Recovery Mode (it wouldn’t work in macOS, it would complain about the partition in use by fseventsd despite it being umounted)

Once you get a target APFS partition (if you can nuke an entire disk, this is by far the path with least resistance), you’re home free :)

So glad Recovery Mode makes to simple to clone my main drive to a small partition on a 6TB HDD external and boot off it when I get into a bind - hallelujah!

EDIT: apparently I celebrated a bit too soon: ironically, I can boot my Catalina clone on an early 2015 MBP and if I try to boot off the clone on the source 2010 mac mini server it hangs (go figure). For completeness, I verified I can use these steps to clone macOS Monterey on the 2015 MBP to the same external HDD and boot off it on the same machine. So at the end of the day, these steps are valid if cloning a bootable APFS is what you’re up to. I’ve thus decided my time with Catalina is coming to an end and going to rebuild my mac mini server to dual boot Ubuntu 22.04 and OpenCore Monterey (or Ventura/Sonoma if it can take it). Thanks everyone for keeping this project moving - y’all have opened up so many new doors! 🚪