Skip to content

Instantly share code, notes, and snippets.

@XenoPanther
Forked from 0xbadfca11/00_README.md
Last active April 2, 2025 22:00
Show Gist options
  • Save XenoPanther/15d8fad49fbd51c6bd946f2974084ef8 to your computer and use it in GitHub Desktop.
Save XenoPanther/15d8fad49fbd51c6bd946f2974084ef8 to your computer and use it in GitHub Desktop.
Windows ReFS versions

Version number is reported by fsutil fsinfo refsinfo, available on Windows 10 and Windows Server 2016.

ReFS 1.1

  • Version of formatted by Windows Server 2012.
  • Version 1.1 is used already in Windows Server 8 Beta. I have never seen version 1.0.
  • Can use and store alternate data streams, when mount on 8.1/2012 R2 or later.

ReFS 1.2

  • Version of formatted by Windows 8.1, Windows 10 v1507 to v1607, Windows Server 2012 R2, and when specified ReFSv1 on Windows Server 2016 or later.
  • Cannot use alternate data streams, when mount on 2012.

ReFS 7.2

  • Version that can be formatted with Windows 10 Build 9780

ReFS 9.2

  • Version that can be formatted with Windows 10 Technical Preview build 9841 to 9860 and Windows Server 2016 TP1 (It's not default). Could not mount in 10 9879 or 2016 TP2 and later.

ReFS 11.2

  • Version that can be formatted with Windows 10 Technical Preview build 9879 (It's not default). Could not mount in 9926 and later.

ReFS 12.2

  • Version that can be formatted with Windows 10 Technical Preview build 9926 (It's not default). Could not mount in 10041 and later.

ReFS 22.2

  • Version that can be formatted with Windows 10 Technical Preview build 10041 to 10049 (It's not default). Could not mount in 10061 and later.

ReFS 2.0

  • Version of formatted by Windows Server 2016 TP2/TP3.
  • Version that can be formatted with between Windows 10 Technical Preview build 10061 or later and earlier than 10130 (It's not default). Could not mount in Windows 10 Insider Preview build 10130 and later, or Windows Server 2016 TP4 and later.

ReFS v2 Overview
http://www.snia.org/sites/default/files/SDC15_presentations/file_sys/JRTipton_ReFS_v2.pdf
http://www.snia.org/events/storage-developer/presentations15#file_sys

ReFS 3.0

  • Version of formatted by Windows Server 2016 TP4/TP5.
  • Upgrade to 3.1 when writable mount from Windows Server 2016 RTM.

ReFS 3.1

  • Version of formatted by Windows Server 2016.

ReFS 3.2

  • Version of formatted by Windows 10 v1703.
  • Version that can be formatted with Windows 10 Insider Preview 15002 or later (It's not default at 15002) and Windows Server Insider Preview build 16237.
    • It became the default between after than 15002 and 15019 or earlier.

ReFS 3.3

ReFS 3.4

  • Version of formatted by Windows 10 Enterprise v1803, Windows Server 2019 and Windows Server version 1803.
  • Version of formatted by Windows 10 Enterprise Insider Preview build 17083 and Windows Server Insider Preview build 17079.

ReFS 3.5

  • Version of formatted by Windows 10 Enterprise Insider Preview build 19536 and Windows Server Insider Preview build 19551.
  • Added hardlink support if fresh formatted volume.
    • Can't use hardlink if upgraded from previous version.
  • Upgrade to 3.6 when writable mount from Windows 10 Insider Preview build 21292.

ReFS 3.6

  • Version of formatted by Windows 10 Enterprise Insider Preview build 21292 and Windows Server Insider Preview build 20282.
  • Upgrade to 3.7 when writable mount from Windows 10 Insider Preview build 21313.

ReFS 3.7

  • Version of formatted by Windows 11 Enterprise v21H2 and Windows Server 2022.
  • Version of formatted by Windows 10 Enterprise Insider Preview build 21313 and Windows Server Insider Preview build 20303.

ReFS 3.9

  • Version of formatted by Windows 11 Enterprise v22H2.
  • Version of formatted by Windows 11 Enterprise Insider Preview build 22598 and Windows Server Insider Preview build 25099.
  • Added post process compression with LZ4 & ZSTD, transparent decompression.

ReFS 3.10

  • Version formatted by Windows 11 Enterprise Insider build 25324 through 25997 and builds 26010

ReFS 3.12

  • Version formatted by Windows 11 Enterprise Insider build 26002. Build 26010 reverts back to 3.10

Mountability

ReFS\Windows 2012 8.1/2012 R2 10 v1507 2016 10 v1703 10 v1709 10 v1803/2019 11 v21H2/2022 11 v22H2 Canary Channel
1.1 Yes Yes1 Yes1 Yes1 Yes1 Yes1 Yes1 Yes1 Yes1
1.2 Yes Yes Yes Yes Yes Yes Yes Yes Yes
3.1 No No No Yes Yes2 Yes3 Yes4 Yes56 Yes76
3.2 No No Yes Yes3 Yes4 Yes56 Yes76
3.3 No No Yes Yes4 Yes56 Yes76
3.4 No Yes Yes56 Yes76
3.7 No Yes Yes8
3.9 No Yes
3.10 No No Yes9
3.12 No No Yes

License: CC BY

Footnotes

  1. "Volume "?:" was mounted in an older version of Windows. Some features may be lost." was recorded to event log when writable mount. I don't know what's been lost. 2 3 4 5 6 7 8

  2. Upgrade to 3.2 when writable mount.

  3. Upgrade to 3.3 when writable mount. 2

  4. Upgrade to 3.4 when writable mount. 2 3

  5. Upgrade to 3.7 when writable mount. 2 3 4

  6. Can't use hardlink. 2 3 4 5 6 7 8

  7. Upgrade to 3.9 when writable mount. Can't mount read-only. 2 3 4

  8. Upgrade to 3.9 when writable mount.

  9. Builds 25324 to 25997

@mdealer
Copy link

mdealer commented Dec 16, 2024

Coming over from the source gist, so 23H2 corrupted my ReFS volumes, but kept the file structure and file contents on disk. The files got "disconnected" from the actual data on disk. Any ideas how to connect them back? I am already exporting the most important files using refsutil and so far 99.9999% are intact, just not accessible by path ("file not found" error on every existing file but not on new ones). Version reported is 3.4. I run W11 24H2 now.

Tools that I have run so far do not report any errors, but I am no ReFS expert to know all of them.

@Karl-WE
Copy link

Karl-WE commented Dec 18, 2024

Hi @mdealer with no promises.

Could you describe what's the former setting and today's I would like to repro and then escalate it.

@mdealer
Copy link

mdealer commented Dec 19, 2024

Hi @mdealer with no promises.

Could you describe what's the former setting and today's I would like to repro and then escalate it.

I want to understand what went wrong and prevent this from happening for anyone else and find potential easy solutions, if any. I don't need to actually recover the partitions per se. refsutil will do fine for me.

I was running W10 22H2 Pro for Workstations. I don't know with which ReFS version I started with, but at the moment refsutil reports v9.4.

System disk: 1TB NVMe NTFS
CPU: Threadripper 2950x
RAM: 64GB

The problematic ReFS partitions were using AMD RAID:

  • Spanned partition with 4x NVMe disks 512GB each with around 20% free
  • RAID 1 with 2x HDDs 14TB each with around 65% free

What I roughly did:

  1. I waited years for W11 to get stable as I was already severely burned by XP->Vista upgrade.
  2. On day X, I made a backup of the system drive as I thought "ReFS will surely survive, it is named resilient after all"
  3. Then I reenabled the TPM module (disabled it so that I don't get nagged about upgrades) and was able to download and install W11 23H2.
  4. The installation took kinda long, but I saw no errors, the system booted fine.
  5. I started hearing disk access on the HDDs instantly after boot. Checking Task Manager showed 30MB/s read 1MB/s write non-stop.
  6. The system barely responded from Desktop, it appeared as if I am unable to open anything, but it just took more than 1min to do anything, like open an explorer window or Task Manager.
  7. AMD RAID displayed no errors, everything was normal in its control panel, but it showed 100K IOP/s.
  8. I started checking, disks were shown as RAW in Disk Management and the System process was doing the IO, there were the typical errors in event logs as reported from other users.
  9. I kind of panicked because I didn't understand how the System is not recognizing ReFS but writing to it, I am a noob in this particular area of Windows related to device mounting and file system upgrades.
  10. I checked for more updates and there were some, so I tried to install them. They took forever to install but succeeded. I think I let the computer run for an hour or so for this, then restarted. The restart went fine, but maybe I was not watching.
  11. The partitions were still shown as RAW and System was writing to them. Eventually the system stopped writing to the SSD partition but it kept restarting the process after OS restart I think. Both partitions got corrupted the same way.
  12. Installing more driver & co updates took forever, the machine was barely usable, the disks were already written to, so I decided to force abort this and put the disks Offline in disk management as I thought no way it can take this long to upgrade metadata on spanned SSDs and wanted to avoid actual data loss on the HDDs. The machine became usable.
  13. Then after Googling I found out that 24H2 added support for ReFS. I couldn't believe my eyes, how did Microsoft nag me about W11 all this time, and basically ignored my ReFS partitions?!?
  14. The update was not offered to me. I had to force it in the registry. It installed fine without errors.
  15. Partitions became "readable", no disk activity. I thought all is fine. Two days later I tried to open a file and was greeted by "File not found".
  16. I am now slowly recovering the files using refsutil (need to procure disks) and so far I see no obvious problems after recovering around 1TB of data with around 40K files.

Let me know if you need more. I understand that somewhere it might be this platform specific, the Threadripper has problems with too slow internal CPU bus that slows down literally everything when accessing PCIe lanes like the spanned SSDs.

EDIT: I completely forgot that I changed from RAID0 to spanned volume on the NVMe SSDs a year ago due to microstutters on high throughput (known platform quirk).

@swph
Copy link

swph commented Dec 19, 2024 via email

@Karl-WE
Copy link

Karl-WE commented Dec 23, 2024

Hi @mdealer that's a very helpful description.
as said the metadata upgrade will cause read and writes on the "raw" ReFS volume to upgrade it to the latest.

Question:
What happened about the NVMe drives did these upgrade correctly?

Personal opinion:
Generally, there is no kind of warning in the docs, outside clustered volumes (CSV).
I would be hesitant using ReFS on top of RAID or software RAID controllers.

Microsoft expects ReFS:

  • on standalone drives / partitions on single drives, using Microsoft default storage drives (AHCI or stornvme)
  • on multiple drives using Storage Spaces (Consumer OS or Windows Server, standalone computer)
  • on multiple drives using Storage Spaces Direct (Consumer OS or Windows Server , on clustered servers, that fulfill certain system requirements).

Why:
ReFS is resilient and has benefits a number of benefits over NTFS for any of the configurations above, but even more when in use with Storage Spaces or Storage Spaces direct.

It is unclear if Microsoft does any testing of ReFS on HW or SW RAID. What is clear though that Storage Spaces or Storage Spaces Direct would not be supported using RAID / SW RAID. That's not the case here.

personal wish:
I would really appreciate if the ReFS metadata upgrade procedure and progress would be transparent to the user, in File Explorer. It is not.

@swph
Copy link

swph commented Dec 23, 2024 via email

@Karl-WE
Copy link

Karl-WE commented Dec 23, 2024

Thank you swph. That's an important note.
Anyone opened a support ticket eventually?
A SR number would be helpful to work on this in 2025. The amount of reports is concerning.
Again, didn't meant to say that ReFS on RAID, with the described scenario, is generally a problem.

@Karl-WE
Copy link

Karl-WE commented Dec 24, 2024

Wrote a guidance on this techcommunity.microsoft.com thread. Hope it helps.

@SanicBTW
Copy link

I'm currently facing issues with ReFS, I'm working on a project and I thought on using a dev drive to be able to move the files faster, I have the drive on a SD card and when I first created the drive and stored it in the SD card it was fine, then I moved to another PC and it was working correctly but going back to my laptop (the origin of the drive creation) it couldn't get mounted and reported a RAW filesystem.

The laptop is running Windows 11 23H2, my PC is running Windows 11 24H2, I believe it's something related to the ReFS version from each host and not being able to downgrade the drive version, the thing is that the drive gets only mounted on my PC, not my laptop which is uhh something that I don't really like.

@Karl-WE
Copy link

Karl-WE commented Dec 24, 2024

@SanicBTW see the linked thread. It will answer your situation.

@SanicBTW
Copy link

I got it working again, thanks!

@Karl-WE
Copy link

Karl-WE commented Dec 24, 2024

Mind to share how?

@SanicBTW
Copy link

Oh I only added the registry key, then on the PC that could open the dev drive i copied over the contents of it to the other dev drive which was created over on the laptop.

PC was running ReFS 3.14 and Laptop was running ReFS 3.10, when I meant that I got it working again I wasn't mentioning some kind of ReFS rollback, I only followed your steps on the guide.

@mdealer
Copy link

mdealer commented Dec 25, 2024

Question: What happened about the NVMe drives did these upgrade correctly?

Hi, the symptoms are in the end the same for NVMe (spanned) as HDDs (RAID1), both seem broken the same way.

It is unclear if Microsoft does any testing of ReFS on HW or SW RAID. What is clear though that Storage Spaces or Storage Spaces Direct would not be supported using RAID / SW RAID. That's not the case here.

personal wish: I would really appreciate if the ReFS metadata upgrade procedure and progress would be transparent to the user, in File Explorer. It is not.

Yes, I also think it should show progress but I absolutely don't get how it starts upgrading without having the capability to read it first? The logic is like, "I don't know what this car is but lets upgrade the ECU."

@mdealer
Copy link

mdealer commented Jan 3, 2025

Something strange happened and the problem fixed itself. I am only 99.9% sure which step did it, so take it with a grain of salt:

Start refsutil salvage -QS ... and let it finish. Then start refsutil salvage -FS ..., let it go over a few MB and then cancel it (CTRL+C). It fixed the file system, BUT, it also upgraded it 3.14, so refsutil no longer works as it complains about unsupported version...

Maybe you don't need -QS, but I can't check as the problem went away on all affected partitions instantly after running -FS on one of them.

Maybe someone can confirm or deny if this changes anything. Other than that I can only guess that my PC is partly to blame (I consider the AMD X399 platform as a failed experiment, things are just NOT stable because of laughably slow CPU internal bus). Microsoft is also partly to blame for unnecessarily forcing ReFS updates on non-system disks. This stuff should be optional, period.

@Karl-WE
Copy link

Karl-WE commented Jan 3, 2025

ReFS is only partly available on system disks (Trusted VMs), not for usual business. I would like to disagree with upgrades being not necessary. Over the course some Versions had limitations and instabilities, so upgrades helped here.

From some scenarios that I am understanding from the thread it's questionable why people used ReFS in the first place over NTFS.

The complexity we are reading and experiencing, must be a reason why Microsoft removed the ability to format drives with ReFS in Windows Consumer Editions.

Imho, most of the issues could be solved with an active upgrade button in File Explorer and progress bar during metadata upgrade (plus checking for free space available to do so).

The auto-mount upgrade with no way to downgrade (understandably) is a complicated design.

While we would need a GitHub discussion thread I would be very interested which OS SKU people are using here and why they choose ReFS back then, just for educative reasons.
You wrote 9.14, it should be 3.14, which is yet missing in this gist XenoPanther.

@mdealer
Copy link

mdealer commented Jan 3, 2025

ReFS is only partly available on system disks (Trusted VMs), not for usual business. I would like to disagree with upgrades being not necessary. Over the course some Versions had limitations and instabilities, so upgrades helped here.

From some scenarios that I am understanding from the thread it's questionable why people used ReFS in the first place over NTFS.

The complexity we are reading and experiencing, must be a reason why Microsoft removed the ability to format drives with ReFS in Windows Consumer Editions.

From the standpoint of avoiding total data loss, ReFS seems alright. I really do not want to have a file system that can corrupt itself and needs proactive scans on 20TB disks... that should be a thing of the past. There are not really other options on Windows. In this regard, ReFS is right on the spot. I have lost enough data because things get corrupted on "accidental" power loss on NTFS in my time, the 'repair' actually killing the rest of the files in the process.

Another aspect is performance. In my tests ReFS deals with small files better. Also, less risk of file system corruption means the cache can be used more aggressively, albeit I would only recommend non-volatile caches.

Imho, most of the issues could be solved with an active upgrade button in File Explorer and progress bar during metadata upgrade (plus checking for free space available to do so).

The auto-mount upgrade with no way to downgrade (understandably) is a complicated design.

Yes, agreed.

While we would need a GitHub discussion thread I would be very interested which OS SKU people are using here and why they choose ReFS back then, just for educative reasons. You wrote 9.14, it should be 3.14, which is yet missing in this gist XenoPanther.

I fixed the number, thx.

@Karl-WE
Copy link

Karl-WE commented Jan 3, 2025

Well NTFS is a bit less stable for sure when it comes to shutdowns due power outage.
But since Windows Server 2012 R2, Windows 8.1 the chkdsk improved.
It will no longer check the entire disk / volume which would take years on 20 TB.
But it has the logic to recognize defective areas.

It can be still slow though.

When it comes to power outage there's a risk of data corruption nevertheless for both ReFS and NTFS.

When writing files it will now try to commit into Cache RAM and reporting the application the write is completed, while then slowly writing to disk (lazy write back). This is the default behaviour since Windows Server 2019 / W10 1809 for buffered writes, which are the default.

This means that when power outage occurs writes might not be finished and lost in RAM.

@Borgquite
Copy link

Just tested ReFS 3.14 and it is still capable of corrupting good data with bad data with the Integrity Streams feature enabled. Just a public service announcement not to use it for that purpose.

https://www.reddit.com/r/DataHoarder/comments/scdclm/testing_refs_data_integrity_streams_corrupt_data/

@Karl-WE
Copy link

Karl-WE commented Jan 3, 2025

@Borgquite thanks Chris, I personally hear few people / customers asking for integrity streams due to performance impact on CSV.
That you can still repro the problem is quite concerning. Quick read the thread. Do you like to necro your reddit with this Information and Windows Server 2025, you have used for testing?
Asking become someone from ReFS PG replied.
If you don't want to post on reddit, I would suggest opening a regular support case with your findings, I help to get this to the right folks as soon I have the SR with repro steps.

Thank you very much in advance.

@Borgquite
Copy link

Borgquite commented Jan 3, 2025

@Karl-WE I'm unable to comment on the Reddit post further as it's been auto-archived (nothing I can do about that).

Regarding a support case, if you read the thread carefully, I already tried to open a Premier (paid) support case in 2022 (#2204260040003921), but despite being quite persistent, the very unhelpful Microsoft Windows Platforms Support senior engineers were unwilling to even acknowledge that there were any bugs let alone work with me to resolve them. The only thing I got was a request to submit the issue to the Feedback Hub app in Windows (which I did). I raised a formal complaint, which was ignored.

After some more Reddit digging I eventually managed to establish contact with the very helpful Billy Smolen, in his role as the Storage & File Systems Team Product Manager, but unfortunately that trail went dead (no replies to my Teams messages) since July 2023 and based on his LinkedIn profile, he no longer works at Microsoft, so that option is no longer available.

It's now languishing and continuing to be ignored in the awful Feedback Hub app, I'm afraid, and there's nothing more I can do about it. I'm certainly not raising a support case again given the waste of time that it was last time. Microsoft don't really care about the ReFS Integrity Streams feature (if they did, they would have fixed it a long time ago as it's demonstrably broken) and it's still actively corrupting data - so the message Microsoft are sending right now is that they also don't care about protecting your data. It's been broken for more than 11 years. All I can do is warn people off the feature.

@Karl-WE
Copy link

Karl-WE commented Jan 3, 2025

Your warning taken. Let me see if can connect you. You can find me as Karl-WE on techcommunity.microsoft.com. If you could write me more details via DM, I would be glad to get this back on track. Yes it's very unfortunate if people leaving a team things get stuck in backlog.
Your concerns are important and the long duration of this being is not easy to deal with.

If the fix would implicate a new ReFS Version rather than a LCU fix, then I follow the impression of Billy that it's really hard to address it via backports. Thanks for your time. Hope you can give it another shot.

@Borgquite
Copy link

@Karl-WE Sent you a DM 👍

@Borgquite
Copy link

Borgquite commented Jan 3, 2025

@Karl-WE Billy explained about backporting etc which I understand - the disappointing part is that I raised this issue back in 2022, which gave plenty of time for it to be resolved in a later release of Windows 11 and especially Windows Server 2025, but that window of opportunity has been utterly squandered.

@Borgquite
Copy link

NB Just tested Windows 11 Enterprise 24H2 (also ReFS 3.14) - integrity streams still broken.

@Knewest
Copy link

Knewest commented Feb 18, 2025

Thank you so much for the warning, Borgquite. I've ensured integrity streams are disabled on all my ReFS drives. Shame it's been broken for +11 years.

@Karl-WE
Copy link

Karl-WE commented Mar 3, 2025

@XenoPanther I have checked my ReFS version in Windows Server 2025 / W11 24H2 which is now 3.14. Are there plans update this precious gist?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment