APFS Container cloning/replicating under macOS 11.0 (Big Sur) - with a bootable system

readme.md

It is the year 2020 and replicating APFS containers still sucks. One would expect it would be a simple copy and paste in the Disk Utility app but this is still far from reality.

Last year I wrote how I managed to clone my macOS system under Catalina. The main trick was to create a DMG file with multiple volumes, mount it on target machine and drop to command-line to do asr restore from synthetised disk while avoiding possible pitfalls.

The good news is that Apple devs definitely worked on improving this under Big Sur and added some documentation (see man asr). But I didn't understand it fully on first read. Maybe someone could explain how is this supposed to work?

Unfortunately they broke the workflow I described in the previous article. The problem is that asr newly tests source of restore operation and if it contains multiple system/data volumes then it simply gives up. For example:

> diskutil list
/dev/disk0 (internal, physical):
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      GUID_partition_scheme                        *500.3 GB   disk0
   1:                        EFI ⁨EFI⁩                     209.7 MB   disk0s1
   2:                 Apple_APFS ⁨Container disk1⁩         499.4 GB   disk0s2
   3:           Windows Recovery ⁨⁩                        552.6 MB   disk0s3
   4:                 Apple_Boot ⁨⁩                        134.2 MB   disk0s4

/dev/disk1 (synthesized):
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      APFS Container Scheme -                      +499.4 GB   disk1
                                 Physical Store disk0s2
   1:                APFS Volume ⁨BismarckOS - Data⁩       439.6 GB   disk1s1
   2:                APFS Volume ⁨Preboot⁩                 82.1 MB    disk1s2
   3:                APFS Volume ⁨Recovery⁩                529.0 MB   disk1s3
   4:                APFS Volume ⁨VM⁩                      2.1 GB     disk1s4
   5:                APFS Volume ⁨BismarckOS⁩              11.5 GB    disk1s5
   6:                APFS Volume ⁨Nix Store⁩               616.0 MB   disk1s6

...
/dev/disk6 (external):
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      GUID_partition_scheme                         1.0 TB     disk6
   1:                        EFI ⁨EFI⁩                     209.7 MB   disk6s1
   2:                 Apple_APFS ⁨Container disk7⁩         500.8 GB   disk6s2
   3:                 Apple_APFS ⁨Container disk8⁩         499.5 GB   disk6s3
   
/dev/disk7 (synthesized):
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      APFS Container Scheme -                      +500.8 GB   disk7
                                 Physical Store disk6s2
...

> asr restore --source /dev/disk1 --target /dev/disk7 --erase
	Validating target...done
	Validating source...
Source "/dev/disk1" contains multiple system/data volumes.
You must pass --sourcevolumename or --sourcevolumeUUID to specify which volume to restore.
Could not validate source - Invalid argument

Not sure if non-standard ⁨Nix Store volume has anything to with it. Feel free to comment with your experiences.

And if you try to pass in --sourcevolumename, it answers in gibberish:

> asr restore --source /dev/disk1 --target /dev/disk7 --erase --sourcevolumename BismarckOS
	Validating target...done
	Validating source...done
	Erase contents of /dev/disk7 ()? [ny]: y

Couldn't set up partitions on target device - operation AddAPFSVolumeToContainer, line #5334 - error 49165

Well, we could try to specify container partition as the source of restore operation. I'm 99% sure this didn't work last year...

> sudo asr restore --source /dev/disk0s2 --target /dev/disk7 --erase
Password:
	Validating target...done
	Validating source...
Source "/dev/disk1" contains multiple system/data volumes.
You must pass --sourcevolumename or --sourcevolumeUUID to specify which volume to restore.
Could not validate source - Invalid argument

Hmm... what about this?

> sudo asr restore --source /dev/disk0s2 --target /dev/disk7 --erase --sourcevolumename BismarckOS
	Validating target...done
	Validating source...done
	Erase contents of /dev/disk7 ()? [ny]: y
	Replicating ....10....20....30....40....50....60....70....80....90....100
	Replicating ....10....20....30....40....50....60....70....80....90....100
	Restored target device is /dev/disk7s2.
Restore completed successfully.

Wut?

> diskutil list
...
/dev/disk7 (synthesized):
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      APFS Container Scheme -                      +500.8 GB   disk7
                                 Physical Store disk6s2
   1:                APFS Volume ⁨BismarckOS - Data⁩       439.5 GB   disk7s1
   2:                APFS Volume ⁨BismarckOS⁩              11.5 GB    disk7s2
   3:                APFS Volume ⁨Preboot⁩                 82.1 MB    disk7s3
   4:                APFS Volume ⁨Recovery⁩                533.8 MB   disk7s4
...

This /dev/disk7 system turned out to be bootable and was a good copy of the previous system (as expected).

Observations

1. It is possible to clone/copy APFS-resident bootable macOS system from command-line.
2. I don't know how to do it with DMGs anymore. Instead, by accident, I found a way how to do it directly with asr restore disk-to-disk (I used target disk mode to connect old machine to new machine).
3. asr restore has a custom logic how to copy system/data volumes, it does not do what would be normally expected, I mean: "to copy all volumes one-by-one". See that Nix Store⁩ and VM volumes were not copied/restored.

Please note that all this info is relevant to public release of macOS 11.0.1 (Big Sur). I expect this behaviour to be subject of future changes.

Pro tip: If you need to boot into some other macOS system to do some asr work please make sure you are using latest versions of these tools because they differ between OS releases.

> sw_vers
ProductName:	macOS
ProductVersion:	11.0.1
BuildVersion:	20B29

I can not restore a dmg container backup due to existing Snapshot that can NOT be removed: (no matter if doing it in another machine Catalina, or Recovery Catalina or Recovery Big Sur on actual machine)

me@p310 ~ % diskutil apfs listsnapshots /dev/disk5s5
Snapshot for disk5s5 (1 found)
|
+-- 4374EE37-CF4E-4FC9-8A13-7657D818842A
    Name:        com.apple.bless.49B69FB7-5A74-4405-9B2A-BE4CE2725269
    XID:         9223372036854776736
    Purgeable:   Yes
    NOTE:        This snapshot limits the minimum size of APFS Container disk5
me@p310  ~ % sudo diskutil apfs listsnapshots /dev/disk5s5 | grep "+--" | cut -d" " -f2 | xargs -I{} sudo diskutil apfs deleteSnapshot disk5s5 -uuid {}
Password:
Deleting APFS Snapshot 4374EE37-CF4E-4FC9-8A13-7657D818842A "com.apple.bless.49B69FB7-5A74-4405-9B2A-BE4CE2725269" from 
APFS Volume disk5s5
Started APFS operation
Error: -69863: Insufficient privileges 

Actual restore also fails:

Deleting APFS Snapshot 4374EE37-CF4E-4FC9-8A13-7657D818842A "com.apple.bless.49B69FB7-5A74-4405-9B2A-BE4CE2725269" from APFS Volume disk27s5
Started APFS operation
Error: -69863: Insufficient privileges

-bash-3.2# asr restore --source /dev/disk27 --target /dev/disk0s2 --erase --noprompt --useInverter
	Validating target...done
	Validating source...done
	Validating sizes...nx_kernel_mount:1387: : reloading after unclean unmount, checkpoint xid 1457, superblock xid 1454
done
	Restoring  ....10....20....30....40....50....60....70....80....90....100
	Verifying  ....10....20....30....40....50....60....70....80....90....100
	Inverting target volume...lookup_topology:1234: synth container: disk1
lookup_topology:1235: synth vol: disk1s1
lookup_topology:1240: main tier: /dev/disk0s2
lookup_topology:1248: encrypted: no
*** Mounting outer volume (/dev/disk1 s1)...
nx_kernel_mount:1385: /dev/disk1: checkpoint search: largest xid 23, best xid 23 @ 45
spaceman_metazone_init:238: metazone for device 0 of size 914187 blocks (encrypted: 0-457093 unencrypted: 457093-914187)
spaceman_datazone_init:491: allocation zone on dev 0 for allocations of 1 blocks starting at paddr 4096000
spaceman_datazone_init:491: allocation zone on dev 0 for allocations of 2 blocks starting at paddr 917504
spaceman_datazone_init:491: allocation zone on dev 0 for allocations of 3 blocks starting at paddr 950272
spaceman_datazone_init:491: allocation zone on dev 0 for allocations of 4 blocks starting at paddr 983040
spaceman_trim_free_blocks:3326: scan took 0.042423 s, trims took 0.000000 s

*** Getting image dstream info...
    ContainerToInvert: dstream_id=1152921500311879696, size=42000003072

*** Mounting inner volume (ContainerToInvert)...
nx_kernel_mount:1385: : checkpoint search: largest xid 1462, best xid 1462 @ 187
sanity_check_alloced_blocks:270: fs_alloc_count mismatch: fs root nodes 53003 extent 1 omap 591 snap_meta 1 er: 0 udata: 3887202 fs_alloc_count 3944194
mount_inner_volume:884: Inner volume has snapshots

APFS inverter failed to invert the volume - Invalid argument

Restore (APFS Container) from Disk Util GUI did work using the same (with above Snapshot) dmg image.

Only volume that did NOT get restored and I could not create it by hand was the Update Volume (-role E) 👍

APFS Volumes can be tagged with certain role metadata flags; you can supply the roles parameter with any
combination of one or more of the characters BRVITSDUNEXHLCYG, or 0 as a no-op for scripting convenience;
the meaning of these characters is, respectively: 
B=Preboot (boot loader), 
R=Recovery, 
V=VM (swap space),
I=Installer (temporary usage), 
T=Backup (Time Machine), 
S=System, D=Data, 
U=User, 
N=Baseband, 
E=Update,
X=XART (hardware security), 
H=Hardware, L=Internal, 
C=Sidecar (Time Machine), 
Y=Enterprise (data), and
G=iDiagnostics (EFI).  
Note that you may be limited to only one role at a time and various other rules.

I have manager to get it to restore from .dmg with asr only from Refind bootable recovery USB (copy UUID folder from Recovery to USB stick with Refind installed to EFI partition having configured in refind.conf:

showtools shell, bootorder, gdisk, memtest, mok_tool, apple_recovery, windows_recovery, about, hidden_tags, reboot, exit, firmware, fwupdate, csr_rotate

To me it is way more convenient & quicker than booting full installer.

-rw-r--r--@  1 _unknown  _unknown  22125765821  5 Dec 11:25 BigSur-MM2012.dmg
-rw-r--r--   1 _unknown  _unknown    330339840  5 Dec 12:45 BigSur-MM2012.dmg.shadow

Then:

[-bash-3.2# hdiutil attach /Volumes/Image\ Volume/BigSur-MM2012.dmg -noverify -nomount
/dev/disk5          	                               	
/dev/disk6          	EF57347C-0000-11AA-AA11-0030654	
/dev/disk6s1        	41504653-0000-11AA-AA11-0030654	
/dev/disk6s2        	41504653-0000-11AA-AA11-0030654	
/dev/disk6s3        	41504653-0000-11AA-AA11-0030654	
/dev/disk6s4        	41504653-0000-11AA-AA11-0030654	
/dev/disk6s5        	41504653-0000-11AA-AA11-0030654	
/dev/disk6s6        	41504653-0000-11AA-AA11-0030654	
-bash-3.2# asr restore --source /dev/disk6 --target /dev/disk0s2 --erase
	Validating target...done
	Validating source...done
	Replicating ....10....20....30....40....50....60....70....80....90....100
	Replicating ....10....20....30....40....50....60....70....80....90....100
	Restored target device is /dev/disk1s2.
Restore completed successfully.

As we can see it is Replication (instead of Restore)
APFS looks like that:

[-bash-3.2# diskutil apfs list
APFS Containers (2 found)
|
+-- Container disk1 1A7FE82D-506E-45F9-BA77-96F266975E9B
|   ====================================================
|   APFS Container Reference:     disk1
|   Size (Capacity Ceiling):      119824367616 B (119.8 GB)
|   Capacity In Use By Volumes:   22052995072 B (22.1 GB) (18.4% used)
|   Capacity Not Allocated:       97771372544 B (97.8 GB) (81.6% free)
|   |
|   +-< Physical Store disk0s2 A68CB599-1BAE-4C16-A978-AB41BF1F4731
|   |   -----------------------------------------------------------
|   |   APFS Physical Store Disk:   disk0s2
|   |   Size:                       119824367616 B (119.8 GB)
|   |
|   +-> Volume disk1s3 A68F2D28-382C-4D04-B368-5B60083BB4BC
|   |   ---------------------------------------------------
|   |   APFS Volume Disk (Role):   disk1s3 (Preboot)
|   |   Name:                      Preboot (Case-insensitive)
|   |   Mount Point:               Not Mounted
|   |   Capacity Consumed:         289579008 B (289.6 MB)
|   |   Sealed:                    No
|   |   FileVault:                 No
|   |
|   +-> Volume disk1s4 F230DC7B-BAF4-4323-A70F-56087263DE07
|   |   ---------------------------------------------------
|   |   APFS Volume Disk (Role):   disk1s4 (Recovery)
|   |   Name:                      Recovery (Case-insensitive)
|   |   Mount Point:               Not Mounted
|   |   Capacity Consumed:         655564800 B (655.6 MB)
|   |   Sealed:                    No
|   |   FileVault:                 No
|   |
|   +-> Volume disk1s1 7FEB70D1-C678-47D6-A449-010D1443DD7E
|   |   ---------------------------------------------------
|   |   APFS Volume Disk (Role):   disk1s1 (Data)
|   |   Name:                      BiGSur - Data (Case-insensitive)
|   |   Mount Point:               Not Mounted
|   |   Capacity Consumed:         4853907456 B (4.9 GB)
|   |   Sealed:                    No
|   |   FileVault:                 No
|   |
|   +-> Volume disk1s2 257E10CB-2354-4D2F-95E7-82BCF7CAC307
|       ---------------------------------------------------
|       APFS Volume Disk (Role):   disk1s2 (System)
|       Name:                      BiGSur (Case-insensitive)
|       Mount Point:               Not Mounted
|       Capacity Consumed:         16127627264 B (16.1 GB)
|       Sealed:                    No
|       FileVault:                 No
|

Only failing to re-create Update volume:

[-bash-3.2# diskutil apfs addVolume disk1 APFS Update -role E
Will export new APFS Volume "Update" from APFS Container Reference disk1
Started APFS operation on disk1
Preparing to add APFS Volume to APFS Container disk1
Creating APFS Volume
Error: -69624: Unable to add a new APFS Volume to an APFS Container

edit:
Update Volume gets auto created or copied (do not know), it simply does not show in Big Sur (so boot to Catalina Recovery with another Refind USB stick, and it shows there just fine!)

Mission accomplished, can still image non-T2 machines with Big Sur!

I had a disaster during Catalina security update and ended up with unbootable system. When I inspected the system volume it was empty, but the installation message said that it could not remove previous system. So I ended up creating a R/W container image and reinstalling the system from scratch. Is it now possible to restore only the DATA volume? And will all user/system settings be there? Another interesting thing is that the DMG system size shows 10.5 GB, but the volume is actually empty:

/dev/disk18 (synthesized):
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      APFS Container Scheme -                      +249.7 GB   disk18
                                 Physical Store disk17
   1:                APFS Volume CAT1 - Data             193.3 GB   disk18s1
   2:                APFS Volume CAT1                    10.5 GB    disk18s2
   3:                APFS Volume Preboot                 23.8 MB    disk18s3
   4:                APFS Volume Recovery                20.5 KB    disk18s4
   5:                APFS Volume VM                      4.3 GB     disk18s5
   6:                APFS Volume Update                  1.3 MB     disk18s6

fascinating…

I'm trying to figure out how to make a system/data pair that are unencrypted and then set them to be encrypted like enabling File Vault does in the UI, but from the command line.

Reading Apple File System Reference it appears that each of the two volumes is encrypted and then their keys (which might include disk and multiple user keys) are wrapped and encrypted with the container key (I think) and stored in the container key bag. Unclear how to do this using command line tools (only documented at a much lower level).

Following this https://gist.github.com/darwin/3c92ac089cf99beb54f1108b2e8b4b9f#gistcomment-3306783 instead did the trick for me.

I found this process that might be helpful for others:

Booted to another external USB disk with Big Sur OS on it.. Assuming the drive you want to copy is disk1s3
-----Create DMG of Big Sur Machine----

#!/bin/sh
#Create SparseBundle to use to copy Image To
hdiutil create /tmp/DSNetworkRepository/Masters/APFS/TempImage.sparsebundle -volname "MacPrep" -fs apfs -size 50GB
hdiutil mount /tmp/DSNetworkRepository/Masters/APFS/TempImage.sparsebundle
#Figure out the disk number of the mounted temp image
disknumber=$(diskutil list MacPrep | grep Scheme | awk {'print $8'})
diskutil apfs deletevolume MacPrep
echo $disknumber
#Get snapshotID of BootDrive
snapshotID=$(diskutil apfs listSnapshots /dev/disk1s3 | grep "Name" | awk {'print $2'})
echo "using drive snapshot: "$snapshotID
asr --source /dev/disk1s3 --target /dev/$disknumber -toSnapshot $snapshotID
#unmount restored disk in order to convert it
diskutil unmountdisk $disknumber
#Pause 5 seconds while waiting for Unmount
sleep 5
#Eject the Image File
hdiutil detach $disknumber
sleep 5
#convert from sparseBundle to DMG
hdiutil convert /tmp/DSNetworkRepository/Masters/APFS/TempImage.sparsebundle -format UDZO -o /tmp/DSNetworkRepository/Masters/APFS/TempImage-Converted.DMG

exit 0

-------------RESTORE DMG Back to Disk---
#!/bin/sh

imagefile="BigSur_11.1_OS.dmg"
imagepath="/Location/To/DMG/"
drivelabel="Macintosh HD"

#Delete and Re-partition creating an APFS Container and temp volume, then delete Temp Volume
diskutil partitionDisk /dev/disk0 1 gpt apfs "MacOSPrep" 100%
#Get Disk Number of new APFS Volume
targetdisk=$(diskutil list MacOSPrep | grep Scheme | awk {'print $8'})
echo "Targetdisk: "$targetdisk
diskutil apfs deletevolume MacOSPrep
sleep 2
#Restore DMG - Must be running macOS 11.1 for next step to work and restore both System and Data Drives
#Mount DMG as Read/Write using shadow file and Get the name of the disk mounted
echo "using "$imagepath/$imagefile" with drive label: "$drivelabel
sourcedisk=$(hdiutil mount $imagepath/$imagefile -shadow /tmp/img.shadow -noverify | grep -Ei "$drivelabel" | grep -Eiv " Data" | awk {'print $1'})
sleep 2
echo "DMG mounted source drive as:" $sourcedisk
#Get snapshotID of BootDrive
snapshotID=$(diskutil apfs listSnapshots $sourcedisk | grep "Name" | awk {'print $2'})
echo "using drive snapshot: "$snapshotID
asr --source $sourcedisk --target /dev/$targetdisk -toSnapshot $snapshotID
#Unmount the DMG so its volume name doesn't conflict
hdiutil detach $sourcedisk
#pause 2 seconds to allow detach
sleep 2
#make sure Disk is Mounted to configure the rest
diskutil mountDisk disk1
#Pause 2 seconds to allow disk to mount
sleep 2
exit 0

THANK YOU! I’ve been looking for a solution for weeks! I can tell this works in 11.2.3 at a Mac with T2 chip.

Excellent explanations. I'm still trying to make an asr restore from a read-only system volume in a Big Sur (for forensic reasons). It seams to work for the first system volume, then it fails when it comes to the data volume (Volume replication failed - read-only file system). What makes no sense, since reading is all It's needed. An "asr restore" bug, perhaps? All seams to work fine when the volume is mounted read-write. Any suggestions that can permit a less intrusive method (forensically sound)?

Following this https://gist.github.com/darwin/3c92ac089cf99beb54f1108b2e8b4b9f#gistcomment-3306783 instead did the trick for me.

thank you for confirming this!!! @yangx2000’s instructions were very clear and very strightforward and intuitive as you would expect it to work. I didn’t even need to bless the drive - it was a clone and boot!

the only snafu I ran into was getting a new partition to format to APFS on mixed-partition (ext4, hfs, APFS) drive - it would only format to FAT32 or exFAT. Disk Utility would throw a “Couldn’t modify partition map.: -69874” every time I chose “APFS” (or HFS+). First Aid on the partition would report “Invalid BS_jmpBoot in boot block: 000000”

The “trick” was to run diskutil repairDisk which shrunk the previous partition about 127MB to create room for “loader space”

Then I was able to format the partition to APFS in Recovery Mode (it wouldn’t work in macOS, it would complain about the partition in use by fseventsd despite it being umounted)

Once you get a target APFS partition (if you can nuke an entire disk, this is by far the path with least resistance), you’re home free :)

So glad Recovery Mode makes to simple to clone my main drive to a small partition on a 6TB HDD external and boot off it when I get into a bind - hallelujah!

EDIT: apparently I celebrated a bit too soon: ironically, I can boot my Catalina clone on an early 2015 MBP and if I try to boot off the clone on the source 2010 mac mini server it hangs (go figure). For completeness, I verified I can use these steps to clone macOS Monterey on the 2015 MBP to the same external HDD and boot off it on the same machine. So at the end of the day, these steps are valid if cloning a bootable APFS is what you’re up to. I’ve thus decided my time with Catalina is coming to an end and going to rebuild my mac mini server to dual boot Ubuntu 22.04 and OpenCore Monterey (or Ventura/Sonoma if it can take it). Thanks everyone for keeping this project moving - y’all have opened up so many new doors! 🚪