Today I wanted to move existing APFS-resident macOS Catalina installation to a new disk. I upgraded my late 2014 Mac Mini with a shiny new 1TB SSD. This took way too many hours of my life I will never get back. Hope this saves some time to you.
Good news:
- it is possible to create a DMG image from existing APFS container with macOS Catalina installation including metadata needed for complete restore (the DMG contains OS, OS Data, Preboot, Recovery and VM volumes)
- it is possible to restore this DMG image into empty APFS container and get a bootable copy of the original system
This information is relevant for Catalina (I'm currently running macOS 10.15.1).
There are major tricks how to make it work, the second one quite recent and third one is its consequence:
Source of restore operation must be a synthetised disk mounted from the DMG. Not the DMG file itself!
"APFS Volume Groups", a new feature introduced in Catalina does not
play well with current Disk Utility.app, so you have to drop to command-line and force using legacy method of restoring the
container.
Source DMG must not contain any APFS Volume with APFS snapshots.
This can be done comfortably from Disk Utility.app. You must be booted into some other system than your main system. In the
left sidebar of Disk Utility.app switch to "Show All Devices". And then right-click the whole APFS container containing your
system. It should have a name like "container diskN" and none of its volumes should be currently mounted. In the context menu
you should see an available option to "Image from container diskN".
Wait, hold your horses. Before making image you might consider deleting all existing snapshots in all APFS Volumes in the container. It will save you headaches later. Better finish reading the article first.
Can I boot into RecoveryOS to do this DMG image?
No. RecoveryOS resides in one of APFS Volumes in the container you want to image so this is not possible, I think (not tested).
Oh, what to do if I don't have a secondary system?
I would recommend using a recent macOS system installed on an external drive and boot from there.
Alternatively, you can downsize the existing APFS container, create a new APFS container on your main disk and install a
secondary macOS there and boot into it. This is quite easily doable just by clicking in Disk Utility.app if you have enough
space on the main drive and a macOS installer at hand.
Disk Utility.app seems to be using asr command under the hood. You can man asr to read the docs, especially the sections
about APFS restore.
When restoring you specify source and target. asr supports different ways how to specify source and target to cover
scenarios related to previous filesystems. Source could be a DMG image file path or some existing disk in the form of
/dev/diskN.
One might expect that specifying a DMG image file as source and some existing empty APFS container as target should do the right thing. No! You get cryptic errors. The trick is to:
- first, mount your DMG file => it creates a synthetised disk
- second, figure out which
/dev/diskNcorresponds to thesynthetiseddisk created in step 1 - finally, use this synthetised disk as source for your
asrrestore operation
Make sense?
Let's be more concrete with my case. My DMG file mounted as disk9 which created synthetised disk10. As a target I created
an empty APFS container in disk0 with single empty APFS volume via Disk Utility.app. The synthetised target container had
identifier /dev/disk1:
> diskutil list
/dev/disk0 (internal, physical):
#: TYPE NAME SIZE IDENTIFIER
0: GUID_partition_scheme *1.0 TB disk0
1: EFI EFI 209.7 MB disk0s1
2: Apple_APFS Container disk1 1000.0 GB disk0s2
/dev/disk1 (synthesized):
#: TYPE NAME SIZE IDENTIFIER
0: APFS Container Scheme - +1000.0 GB disk1
Physical Store disk0s2
1: APFS Volume NewEmptyOS
...
/dev/disk9 (disk image):
#: TYPE NAME SIZE IDENTIFIER
0: +499.2 GB disk9
/dev/disk10 (synthesized):
#: TYPE NAME SIZE IDENTIFIER
0: APFS Container Scheme - +499.2 GB disk10
Physical Store disk9
1: APFS Volume MinimeOS - Data 311.1 GB disk10s1
2: APFS Volume Preboot 82.2 MB disk10s2
3: APFS Volume Recovery 528.5 MB disk10s3
4: APFS Volume VM 4.3 GB disk10s4
5: APFS Volume MinimeOS 10.9 GB disk10s5
To execute restore I had to use asr command from command-line:
> sudo asr restore --source /dev/disk10 --target /dev/disk1
...
Volume replication failed - Read-only file systemThe --useInverter is the tricky option here. Replication functionality which is enabled by default is quite recent and it
does not play well with APFS Volume Groups. It fails when trying to do something with read-only system volume:
I consider this to be a bug and to be resolved in some future macOS version. Unfortunately Disk Utilty.app seem to be using
asr without the --useInverter flag and gives me the same error in the GUI when trying to restore mounted DMG into existing
container.
So I ran it and got:
> sudo asr restore --source /dev/disk10 --target /dev/disk1 --useInverter
...
APFS inverter failed to invert the volume - Invalid argument
Uh, oh!? Trying to run the restore again with --verbose and --debug flag gave me a hint:
> sudo asr restore --source /dev/disk10 --target /dev/disk1 --useInverter --verbose --debug
...
*** Mounting inner volume (ContainerToInvert)...
...
mount_inner_volume:884: Inner volume has snapshots
APFS inverter failed to invert the volume - Invalid argument
Ah, now I remember that the man page stated that --useInverter does not work with snapshots.
The error does not tell me which volume has this issue but sure I can figure this out:
> diskutil apfs listSnapshots disk10s1
Snapshots for disk10s1 (20 found)
|
+-- 9801FBE4-E581-4B95-A055-8A94A9C2ABE9
| Name: com.apple.TimeMachine.2019-11-13-000654.local
| XID: 2200904
| Purgeable: Yes
| NOTE: This snapshot limits the minimum size of APFS Container disk10
|
...
|
+-- 46C315AB-6014-4275-AF94-B284E476718A
| Name: com.apple.TimeMachine.2019-11-13-220854.local
| XID: 2211035
| Purgeable: Yes
|
+-- 72FE9F27-A358-42F0-91CE-CBC0E8140B46
Name: com.apple.TimeMachine.2019-11-13-231202.local
XID: 2212680
Purgeable: Yes
> diskutil apfs listSnapshots disk10s5
Snapshots for disk10s5 (3 found)
|
+-- B07AE72B-1EE9-4145-827F-CB31E06635BB
| Name: com.apple.TimeMachine.2019-11-13-211042.local
| XID: 2210200
| Purgeable: Yes
| NOTE: This snapshot limits the minimum size of APFS Container disk10
|
+-- D124E17B-B061-4143-B69D-6626E5E8B41A
| Name: com.apple.TimeMachine.2019-11-13-220854.local
| XID: 2211035
| Purgeable: Yes
|
+-- 1430DB4E-6527-4FA3-B226-77006055CA99
Name: com.apple.TimeMachine.2019-11-13-231202.local
XID: 2212676
Purgeable: Yes
> diskutil apfs listSnapshots disk10s4
No snapshots for disk10s4
> diskutil apfs listSnapshots disk10s3
No snapshots for disk10s3
> diskutil apfs listSnapshots disk10s2
No snapshots for disk10s2
So the Data volume disk10s1 and OS volume disk10s5 seem to have some snapshots created by TimeMachine.
My task now is to delete them and try to run restore again. The problem is that my disk10 disks are read-only because they
were mounted from read-only DMG.
... few hours of banging my head into a wall
Ok, I will spare you and won't describe all my trial and errors here.
Finally I found a way how to strip all snapshots from disk10s1:
> hdiutil attach -owners on /Volumes/W1/backups/mos.dmg -shadow
expected CRC32 $0A545D85
/dev/disk11
/dev/disk12 EF57347C-0000-11AA-AA11-0030654
/dev/disk12s1 41504653-0000-11AA-AA11-0030654 /Volumes/MinimeOS - Data 1
/dev/disk12s2 41504653-0000-11AA-AA11-0030654 /Volumes/Preboot 1
/dev/disk12s3 41504653-0000-11AA-AA11-0030654 /Volumes/Recovery 1
/dev/disk12s4 41504653-0000-11AA-AA11-0030654 /Volumes/VM 1
/dev/disk12s5 41504653-0000-11AA-AA11-0030654 /Volumes/MinimeOS 1
The magical -shadow option mounts the image as read-write and keeps diff of changes in /Volumes/W1/backups/mos.dmg.shadow
file. So if you try this at home make sure you have enough space available for the shadow file. Not sure if -owners on is
important but this worked for me after some previous problems. Also note that an alternative solution would be to use
hdiutil convert -format UDRW ... and continue with converted read-write image. See the Q/A section below.
Anyways, having mounted writable disk allowed me to enumerate all snapshots and delete them:
# this is just a dry-run to see all snapshot GUIDs to be deleted
> diskutil apfs listsnapshots /dev/disk12s1 | grep "+--" | cut -d" " -f2
9801FBE4-E581-4B95-A055-8A94A9C2ABE9
DA44B969-EBA6-4EFE-ADCA-B4BB43FF6B37
8AEFF1F4-BC35-4C05-A21C-3122A97569B4
0149B1B9-9095-4896-B883-CD976802E419
A6E252E6-2007-4DD5-9313-EE2156C0C813
F3C38242-8AF7-42A8-9EC6-CF6381F647C0
D6391D9B-4ADB-44D4-9F05-C58FABC730AD
9281E5D8-867D-423D-A95F-7C4E3B4F34E2
82EA6CA1-FB41-40DC-8F80-B8C34311E491
EEE95161-ABFA-4E99-B893-2887DF4EC317
779FAA27-DACD-41C2-A408-E731DC6E96A1
36E7F342-9E5E-43F4-BA45-0441F9067244
8359CA22-79A6-43DF-A265-35BEB8D4FC7E
70105B62-19B0-4F14-9FFE-E73C6E32C74A
3C581209-FC4B-4D14-9491-95AE656B3335
068E2DD8-B6CF-454E-A5CC-8E68261EC9A6
422788AF-2A6E-40D6-976D-CC3F93A925CA
B647F1C2-E4CF-4F90-8A77-3C1CB2B992C9# delete them all
> diskutil apfs listsnapshots /dev/disk12s1 | grep "+--" | cut -d" " -f2 | xargs -I{} sudo diskutil apfs deleteSnapshot disk12s1 -uuid {}
Deleting APFS Snapshot DA44B969-EBA6-4EFE-ADCA-B4BB43FF6B37 "com.apple.TimeMachine.2019-11-13-011547.local" from APFS Volume disk10s1
Started APFS operation
Finished APFS operation
Deleting APFS Snapshot 8AEFF1F4-BC35-4C05-A21C-3122A97569B4 "com.apple.TimeMachine.2019-11-13-030651.local" from APFS Volume disk10s1
Started APFS operation
Finished APFS operation
...
Deleting APFS Snapshot B647F1C2-E4CF-4F90-8A77-3C1CB2B992C9 "com.apple.TimeMachine.2019-11-13-211042.local" from APFS Volume disk10s1
Started APFS operation
Finished APFS operationTo double-check the result, I ran:
> diskutil apfs listsnapshots /dev/disk12s1
No snapshots for disk12s1I repeated the same snapshot purging with disk12s5 and confirmed:
> diskutil apfs listsnapshots /dev/disk12s5
No snapshots for disk12s5Ok, hopefully there are no snapshots in any volume in whole container and I can run the restore again using the massaged disk12:
> sudo asr restore --source /dev/disk10 --target /dev/disk1 --useInverter
Validating target...done
Validating source...done
Validating sizes...done
Restoring ....10....20....30....40....50....60....70....80....90....100
Verifying ....10....20....30....40....50....60....70....80....90....100
Inverting target volume...done
Restoring ....10....20....30....40....50....60....70....80....90....100
Verifying ....10....20....30....40....50....60....70....80....90....100
Inverting target volume...done
Restored target device is /dev/disk1s2.
The disk is bootable and seems to be 1:1 copy of the original. Sure not really a device-level block-by-block copy, but good enough.
How do I mount the DMG?
Simply double-click the DMG file and it should mount. Then use diskutil list or Disk Utility.app to figure out which
synthetised disk it was mounted as. It should have "APFS Container Scheme".
/dev/disk9 (disk image):
#: TYPE NAME SIZE IDENTIFIER
0: +499.2 GB disk9
/dev/disk10 (synthesized):
#: TYPE NAME SIZE IDENTIFIER
0: APFS Container Scheme - +499.2 GB disk10
Physical Store disk9
1: APFS Volume MinimeOS - Data 311.1 GB disk10s1
2: APFS Volume Preboot 82.2 MB disk10s2
3: APFS Volume Recovery 528.5 MB disk10s3
4: APFS Volume VM 4.3 GB disk10s4
5: APFS Volume MinimeOS 10.9 GB disk10s5
> hdiutil attach /Volumes/W1/backups/mos.dmg
Checksumming whole disk (Apple_APFS : 0)…
....................................................................................................................
whole disk (Apple_APFS : 0): verified CRC32 $09A8F547
verified CRC32 $0A545D85
/dev/disk10 EF57347C-0000-11AA-AA11-0030654
/dev/disk10s1 41504653-0000-11AA-AA11-0030654 /Volumes/MinimeOS - Data
/dev/disk10s2 41504653-0000-11AA-AA11-0030654
/dev/disk10s3 41504653-0000-11AA-AA11-0030654
/dev/disk10s4 41504653-0000-11AA-AA11-0030654
/dev/disk10s5 41504653-0000-11AA-AA11-0030654 /Volumes/MinimeOS
/dev/disk9
> hdiutil attach -owners on /Volumes/W1/backups/mos.dmg -shadow
expected CRC32 $0A545D85
/dev/disk11
/dev/disk12 EF57347C-0000-11AA-AA11-0030654
/dev/disk12s1 41504653-0000-11AA-AA11-0030654 /Volumes/MinimeOS - Data
/dev/disk12s2 41504653-0000-11AA-AA11-0030654 /Volumes/Preboot
/dev/disk12s3 41504653-0000-11AA-AA11-0030654 /Volumes/Recovery
/dev/disk12s4 41504653-0000-11AA-AA11-0030654 /Volumes/VM
/dev/disk12s5 41504653-0000-11AA-AA11-0030654 /Volumes/MinimeOS
How to convert read-only DMG to writable DMG?
This is what worked for me:
> hdiutil convert /Volumes/W1/backups/mos.dmg -format UDRW -o /Volumes/W1/backups/mos-rw.dmg
nx_kernel_mount:1387: : reloading after unclean unmount, checkpoint xid 2213158, superblock xid 2213151
Reading whole disk (Apple_APFS : 0)…
....................................................................................................
Elapsed Time: 3h 14m 56.431s
Speed: 40.7Mbytes/sec
Savings: 0.0%
created: /Volumes/W1/backups/mos-rw.dmg
You can then follow case 1 to mount /Volumes/W1/backups/mos-rw.dmg instead of /Volumes/W1/backups/mos.dmg.
As you can see from the output this conversion can take quite some time if the file is on external/slower disk, so it is
preferable to use -shadow mounting as described in the article.
I've launched hdiutil/asr and it does seem to be stuck, what now?
With larger images/disks it might take up to several hours for operation to complete. The progress indicator in terminal
output is not granular enough to display any action for several minutes. I would recommend to open Activity Monito.App
switch to Disk view and observe most busy reader/writer processes. This should give you sense if your command is doing any
work or is slacking. It could give you also rough lower bound estimate how far it is in progress because it usually needs to
read or write its source or target data at least once. So if you have 300GB DMG and see reading speed 1GB/s you can get an
idea if you can take a nap or not. Also better disable power-saving because your Mac could hibernate unexpectedly while
working.
This was super, super, super helpful in terms of understanding APFS' operations. Thank you. For this particular use case, I would just use Carbon Copy Cloner, but I wanted a lower level understanding. After reading your piece, and closely persuing "man asr", it appears to me that restoring all the volumes in a multi-volume container requires a one-volume-at-a-time process, which just seems crazy to me. Even if Apple's not going to provide a block-copy of the whole container, you'd think they could automate serial multiple restore from a single container.
Though I also don't know why they couldn't provide a block copy if a container is your target, and then fix things up at the destination; it practically does that anyway during the inversion, but that process only lets you operate on one source volume, even after it's already block-copied over the whole container. (A true block-copy manual process for keeping all volumes is laid out here: http://blog.tempel.org/2019/05/cloning-apfs-volumes-containers-apfs.html, though it seems best suited to unencrypted disks, and perhaps only those on non-T2 Macs.)
@tschloeter I think while you got a mostly-block-copy, you also got only a single volume out of your source. That's fine if you only had a single volume anyway, but I think you would have had problems with more. If I understand the asr man page correctly, it no longer copies at the file level anyway, and if you specify --erase with the whole APFS container as a target, all it does is delete the container before making a new blank empty volume to receive the container copy, then inverts the one volume it finds inside that copy. If you'd had more than one, you would have had to manually specify which one you wanted, and then subsequent volumes would have been copied (inefficiently) via the same process, but without --erase, in which case it adds a new volume without deleting the container or the previous volume.
@maxscience I wonder if this has anything to do with T2 Macs storing all data encrypted, whether or not you elect to use FileVault.