Hello! It's been a while since I've done any write-ups, but I thought I'd do one to (hopefully) bring a little clarity to a situation that's been causing a lot of confusion and misinformation.
I'm thinking I'll go through what the problem is and why it happens. So, I guess, let's dive in:
Some people are finding that upon updating to 17.0.0, their consoles are getting a blackscreen and refusing to boot further.
Worse, some people are reporting that this is happening even on consoles that aren't using CFW in any way.
The root cause of this turns out to be an under-the-hood change Nintendo made to save file management, in 17.0.0. This change interacts badly with the way some users have modified the SYSTEM partition in their eMMC in the past; in particular, it turns out to cause systems which have (at any point in the past) used certain unofficial methods to "reset"/"wipe" the internal eMMC to completely fail to boot.
So, what change did they make? To answer that, we need to talk about parallel universes how save files are managed. For those not interested in technical details, you may wish to proceed to the end.
Before that, though, let me get this out of the way:
- This issue will never occur on consoles which have never been wiped/reset using unofficial methods in the past.
- Consoles which have only ever been wiped/reset using official methods are unaffected.
- This issue is probably not intentional on Nintendo's part, it's a side-effect of a legitimate change.
The Switch uses a custom and proprietary archive format for save files, which supports lots of niceties like journaling and rollback. Because the format is so nice, Nintendo uses it for both user game saves and to manage internal system settings/data. Under the hood, just about everything that you can edit on your console (from your Wi-Fi settings, to your usernames, to what's installed on the console) are stored in some specific save file.
Every save file has a 64-bit identifier (its "SaveDataId"). When some process on the Switch asks to open a save file, it provides the following information:
- What partition the save is located (in the SYSTEM partition, in the USER partition, on the SD card, etc).
- The application id for the save, if the save belongs to a specific game (this is zero for system savedata).
- The user id for the save, if the save belongs to a specific user (this is also zero for system savedata).
- The save data id.
- Some other miscellaneous information, none of which is important to this post.
Save files are stored in a directory in whatever partition they're on, with their filename equal to their save data id encoded as hexadecimal.
So, for example, save file 8000000000000120 in the SYSTEM partition is stored inside the file SYSTEM:/save/8000000000000120.
One thing that might not be obvious at first glance is that it might not always be convenient to specify a save data's id. This will be used by games, and it would be kind of a hassle to assign every game a unique save ID, and make sure games don't conflict with eachother.
So, to make this all work more nicely, Nintendo does the following:
- Nintendo creates an index of save files. This index is essentially a list of what save data ids have what application ids/user ids, alongside other metadata/attributes.
- When a save file is created, its id gets added to the index. Games are allowed to specify a save data id of zero, which will automatically allocate a new save id and put it in the index.
- When opening a save file, Nintendo can look up the provided information in the index (if a game specifies an id of zero, its application id/user id can be used for lookup).
This works out pretty nicely! However, Nintendo needs to save data about the index somewhere, and, it turns out, they made the choice to have the index be another save file.
Having the index itself be a save file is nice, because the save file format provides a lot of great features. However, it introduces a chicken-and-egg problem. When opening save files, their data is located in the index. But the index is a save file, which needs to be opened.
To work around this, Nintendo adds the concept of a static save file. If a save file is static, then it doesn't need to be looked up in the index. If the index save file is static, then it doesn't need to be looked up in itself.
Nintendo decided on the static save id value 8000000000000000 for the system index save file, and so the chicken-and-egg problem is solved: the index is always located at SYSTEM:/save/8000000000000000.
The actual change in 17.0.0 has to do with how the OS decides if a save file is static. The specific change is in nn::fssrv::SaveDataFileSystemService::OpenSaveDataFileSystemCore; pseudo-code for before-and-after would look something like this:
- Prior to 17.0.0:
if (SaveDataId != 0 && UserId == InvalidUserId) {
IsStatic = true
}
- In 17.0.0:
if (SaveDataId == 0x8000000000000000) {
IsStatic = true
}
If you're not good at understanding code, the gist is that before, a save file was considered static if it didn't belong to a specific user.
In practice, this meant that prior to 17.0.0 every system save file was considered static.
In 17.0.0, only the index save file is considered static.
On an unmodified console, this change is guaranteed completely safe and fine! The only way for a save file to be created officially is via CreateSaveFile(), which always adds it to the index. This means every system save file will be in the index, so even though they'll now be looked up in the index (where they weren't looked up before), this will always succeed, and everything will be fine.
However, if a console has been modified, this nice guarantee is kind of out the window. People can and will do wild things to their consoles.
Consider what would happen if a system save file was (somehow) present on the filesystem, but not in the index.
Prior to 17.0.0, the system save file would be static. So, it would be opened directly without needing to go through the index. Because system save files have fixed system save ids (unlike game save files), this would directly open the system save file, and would succeed.
On 17.0.0, the system save file would not be static. Because it's not in the index, looking it up will fail, and the OS will conclude that the save does not exist. This is equivalent to deleting that system save file.
As you might have guessed, the problem here arises because some people have ended up in a state where they have an important system save file that isn't in the index. When updating to 17.0.0, this save file gets deleted, and everything breaks.
So...why does this happen?
Most of the time, it's actually not a big deal if a system save file is deleted. Most parts of the OS will simply create a new (empty) save, and it's like resetting that part of the OS back to how it was at the factory.
However, there are two save files that are pretty important:
- The first one is the system "Content Meta Database" save. This save file has id
8000000000000120, and it tracks what OS modules are installed. If it is deleted, the OS believes nothing is installed, and cannot launch or access any of its components. - The second one is (naturally) the index save. If it's deleted, it will be recreated, and will start fresh as empty.
The OS can actually recover from deleting literally any save other than the content meta database save.
However, consider what happens if the index save is deleted, but the content meta database save isn't.
So, prior to 17.0.0, all system save files were considered static. So, the content meta database save wouldn't be in the index, but it could still be opened and used normally.
However, after updating to 17.0.0, the content meta database save would no longer be static. It would be treated as deleted...and the OS would become unable to find any of its programs, and boot would fail.
This is exactly what's happening.
It turns out that a lot of homebrew tools for "cleaning" or "resetting" a console have been relying on this behavior!
Tools like Haku33 or TegraExplorer or EmmcHaccGen or ChoiDujour used to clean or reset a console do this by deleting every save other than the content meta database save, including deleting the index.
Prior to 17.0.0, this seemed to conveniently reset the Switch completely, seeming to put it back to how it was at the factory.
A complete wipe can be useful, and I am even guilty of doing this myself; I have even told users to do this, as a last resort way to get a console back to a known-good state.
Except, it turns out, that state isn't exactly good, because there's an important difference from an actually cleanly-reset console: the content meta database save isn't in the index.
This becomes a time-bomb, and when the console is updated to 17.0.0, the system can no longer open the content meta database save, re-creates it as empty, and bricks.
In the worst cases, a user might do this to "reset" their console before reselling it to someone who doesn't even know what homebrew or CFW is, and when that unsuspecting user updates, they'll find their console no longer boots.
I learned about this problem prior to releasing atmosphère 1.6.0, and tried to add a workaround. If the save file describing where the OS files are installed was detected as deleted on 17.0.0, atmosphère would automatically re-build it, under the assumption that the issue with the index had occurred.
So long as the user was using atmosphère for their first boot into 17.0.0, this would automatically solve the problem. However, if the issue had already happened, the save file wouldn't be deleted -- it would exist, it would just be empty. This would still fail.
I (perhaps naively) hoped that this issue wouldn't be so common, and devised a few short steps to follow for users who ended up in that state. I started telling people to DM me on discord for help, thinking I'd guide them through the steps and that would be that.
This was...a mistake. I severely underestimated how common this issue would be! I received over 100 direct requests for help on discord, and more help requests in support channels, and I can only assume that only a small fraction of people encountering the problem will actually know to reach out to me.
So, I've streamlined this, I hope. Starting in 1.6.1, atmosphère will attempt to additionally detect if the issue has happened in the past (if the list of installed OS programs is empty), and automatically re-build in that case, too. This means that booting into atmosphère will automatically fix the problem, without the need for users to follow additional steps that they might not understand.
Repeating that in bold:
- If this issue has happened to you, all you need to do to fix it is boot atmosphère 1.6.1 (or higher).
- If you are using atmosphère, you do not need to worry about this issue.
I predict some follow-up questions. Hopefully, they're covered here:
- The issue happens when I boot into the stock OS/OFW, without atmosphère. I don't even have CFW installed! What should I do?
- You should boot atmosphère 1.6.1 or higher on sysmmc.
- To be sure that you are using sysmmc, you can inject hekate, and select "emuMMC" -> "Change emuMMC" -> "Disable" (in the upper right corner).
- After doing this, you can boot sysmmc with fusee.bin (or using hekate), and the problem will be automatically fixed.
- I don't want to boot into atmosphère. Do I have any other options?
- I'm sorry, but no. Any alternative would be much less safe! Someone could make a tool to fix it (which you could run separately), I suppose, but no such tool exists at present. If you're interested in writing one, you can reach out to me, but I genuinely believe this would be a bad use of your time and don't recommend it.
- I am using emummc, and emummc works fine. But the problem occurs when I try to boot into OFW. What should I do?
- You should boot atmosphère 1.6.1 or higher on sysmmc.
- Booting into emummc will not fix the problem on sysmmc. In order to fix the problem on OFW/stock, you need to boot atmosphère into sysmmc.
- Are there any cases where this cannot be fixed?
- No, I don't believe so. Every case should be fixable by booting into atmosphère 1.6.1 or higher.
- Can Nintendo fix this via an official system update?
- No, they cannot. Because the console does not know where anything is installed, it cannot boot far enough to install a new system update, and cannot boot far enough to launch a game-card to fix the problem. They could fix it by using an officially-signed tool with RCM mode (to repair consoles in their repair centers), however it seems unlikely to me that they will do this, especially as it only affects consoles which have been hacked in the past.
- Should I stop using tools like this to reset my console?
- Tools like this should probably be updated to not cause a situation where the content meta database isn't in the index. However, research will need to be done to determine the best way to make this happen, and then tools will need to be updated after that.
- A safer alternative would be to use a homebrew program to call the official code for resetting to factory settings ("ResetToFactorySettingsForRefurbishment"). If such a program is made in the future, I will add a link to it here.
- However, if you really want to use the old tools, so long as you make sure that you're on 17.0.0 when resetting, and boot into atmosphère 1.6.1 successfully at least once after resetting, it should be fine.
have you had success fixing this altar?