Fix CMOS/NVRAM issues for Native Boot UEFI OS from NVMe SSD in PCIe x1 adapter in ASRock Q1900DC-ITX Bay Trail motherboard
- Boot TrueNAS Scale 25.10 (or any other UEFI-bootable OS) from an internal NVMe SSD in my ASRock Q1900DC-ITX Bay Trail motherboard
- Does not require a USB flash drive with ReFind, Clover, or UEFI helper to boot the OS on the NVMe SSD.
I wanted to run TrueNAS Scale 25.10 on my ancient ASRock Q1900DC-ITX motherboard (2014), booting the OS from an internal NVMe SSD.
The UEFI firmware in the motherboard does not natively support NVMe SSD's, nor is there an M.2 slot for installing an SSD. Various BIOS-modding communities figured out how to extract the NVMe support modules from later motherboards, and inject them into BIOS images for earlier motherboards. I was unable to find a pre-modded BIOS for my motherboard, but there were modded BIOS's for the very similar ASRock Q1900-ITX.
I was able to collect the necessary tools (MMTool 5.02.0025 and UBU v1_80_b1) to mod the BIOS on my motherboard, and successfully flashed the modded BIOS.
I purchased a PCIe x1 NVMe SSD adapter card for $10, attached a 128 Gb NVMe SSD to the adapter, and inserted it into the PCIe x1 slot of the Q1900DC-ITX motherboard.
Unbelievably, it all worked on the first try! I was able to install TrueNAS Scale 25.10 on the NVMe SSD and my system was up and running.
A month later, I found 2 x 8GB DDR3L SODIMM's that I had forgotten about in a drawer. I decided to replace the 2 x 4 GB DRAM sticks that had been installed in the motherboard since 2014 with these sticks, doubling my memory. I swapped in the new DRAM sticks and booted.
The BIOS message said: UEFI defaults loaded. I went through the UEFI Setup process, turned off CSM (BIOS Compatability Support Module), and rebooted. The NVMe SSD would no longer boot, could no longer be detected, and did not appear in the Boot Priority list.
In the Boot Priority List, the only disk entry was PATA:, and there was not even a PATA header or controller on the motherboard.
I tried clearing the CMOS with the jumper, removing and replacing the CMOS battery, etc. Nothing worked. I removed the NVMe SSD and adapter card from the PC, and installed the NVMe SSD in a USB to NVMe adapter. The system booted TrueNAS perfectly from the SSD in the USB adapter, so there was nothing wrong with the SSD.
I re-installed the NVMe SSD on the PCIe x1 adapter card, and booted a Linux Mint Live USB on the system. It could access the EFI partition on the NVME SSD perfectly, so nothing was wrong with the PCIe x1 adapter or the motherboard slot.
For some reason, a CMOS reset or Load UEFI defaults on a modded BayTrail motherboard will mess up the detection/PCIe enumeration of the NVMe SSD and/or the NVRAM Boot Priority List.
Use UEFI Shell to fix the BOOT entries in the NVRAM
If the NVRAM BOOT list could be corrupted, maybe it could also be de-corrupted! I created a UEFI Shell USB Flash drive, allowing me to fix the BOOT entries in the NVRAM without the help of the motherboard's sketchy BIOS.
Boot the UEFI Shell USB. It will display a list of all the USB, SATA, and NVME file systems it finds. On my system, the FS0 entry was the EFI partition on the Truenas Scale 25.10 NVMe SSD.
::
:: display the NVRAM BOOT entries
::
bcfg boot dump
::
:: you might want to delete some entries by number (optional)
::
bcfg boot rm 2
bcfg boot rm 3
::
:: add a boot entry for the EFI boot program for your OS, and make it first on the priority list
::
bcfg boot add 0 FS0:\EFI\boot\bootx64.efi "TrueNAS NVme"
bcfg mv 0 0Boot the system. Press F11 to get the Boot Device list. You should see "TrueNAS NVme". Select it and boot TrueNAS.
If you go back into the BIOS setup again, you will have the NVME SSD detected, and will be in place in the Boot Priority List.
Hang onto the UEFI Shell Boot USB. You can use it fix the NVRAM boot list if it becomes corrupted in the future
This fix did not just drop out of the sky. I spent several days modding, re-modding, and re-flashing the BIOS with various AMI NVMe driver packages and NvmExpressDXE.ffs. I'm not going to explain the details of the modding process in this gist. In the end, my modded BIOS used only the AMI Nvme, NvmeDynamicSetup, NvmeInt13, and NvmeSmm modules, and no NvmExpressDXE module. Using both sets of modules messed up display of the NVME drive in the UEFI Setup, so I went with just the AMI basic set.
BIOS modding does not have to be a nightmare, especially if someone has already made available the mod you want for the motherboard you have. But be prepared to explore a maze of forum threads if you need to assemble a mod yourself. Perhaps I will host my NVMe modded ASRock Q1900DC-ITX bios on Github.
If you normally use Linux, you can use the efivar and efibootmgr programs on a Linux Live USB to manipulate the UEFI NVRAM environment. I used the bootable UEFI shell because it was minimally simple. I could imagine making a UEFI Shell script to find bootable ESP/EFI partitions on your disks, and create boot entries for them. Or just a simple script to execute the bcfg commands shown earlier.
I have only surface-level knowledge of bios modding, UEFI, NVRAM, etc. Be prepared for surprises.