Some notes about BIOS settings for a HP Z6 G4.
Came with Windows 11 Pro for Workstations 24H2 installed. Updated to Version 24H2 (OS Build 26100.2894).
Windows Control Panel -> System and Security -> BitLocker Drive Encryption reports for the Operating system driver C:BitLocker off. There are currently no other drives fitted. The reason for checking if BitLocker was enabled was that when checking for a BIOS update within the BIOS got a warning that BitLocker should be disabled prior to updating the BIOS.
The Windows System Information reports BIOS Version/Date HP P60 v02.94, 17/05/2024
Downloaded sp143621.exe from HP BIOS Configuration Utility (BCU) and installed.
From an Administrator Command Prompt saved the BIOS settings as delivered:
C:\SWSetup\SP143621>BiosConfigUtility64.exe /get:01_settings_as_delivered.txt
<BIOSCONFIG Version="" Computername="DESKTOP-BVUMP11" Date="2025/01/25" Time="19:18:03" UTC="0">
<SUCCESS msg="No errors occurred" />
<Information msg="BCU return value" real="0" translated="0" />
</BIOSCONFIG>
In the BIOS checked for updates, and updated to BIOS Version/Date HP P60 v02.95, 21/11/2024
Saved the settings after the update:
C:\SWSetup\SP143621>BiosConfigUtility64.exe /get:02_update_to_bios_v02.95.txt
<BIOSCONFIG Version="" Computername="DESKTOP-BVUMP11" Date="2025/01/25" Time="20:03:19" UTC="0">
<SUCCESS msg="No errors occurred" />
<Information msg="BCU return value" real="0" translated="0" />
</BIOSCONFIG>
Excluding version information, only change to settings is that the DNS Addresses is now 192.168.0.1, whereas previously was blank. That is the IP address of the DNS server used on the network, and presumably get set as part of updating the BIOS version from inside the BIOS.
VTd was disabled by default. Enabled with:
C:\SWSetup\SP143621>BiosConfigUtility64.exe /setvalue:"Virtualization Technology for Directed I/O (VTd)","Enable"
<BIOSCONFIG Version="" Computername="DESKTOP-BVUMP11" Date="2025/01/25" Time="20:15:01" UTC="0">
<SETTING changeStatus="pass" name="Virtualization Technology for Directed I/O (VTd)" returnCode="0">
<OLDVALUE><![CDATA[Disable]]></OLDVALUE>
<VALUE><![CDATA[Enable]]></VALUE>
</SETTING>
<SUCCESS msg="No errors occurred" />
<Information msg="BCU return value" real="0" translated="0" />
</BIOSCONFIG>
Due to the issue in Windows 11 not booting after update to 24H2 on a HP Z4 G4 on the 1st reboot after enabling VTd checked could boot the Windows 11 24H2 install media, since that also failed on the HP Z4 G4 when VTd was enabled.
On the HP Z6 G4 with VTd enabled:
- The Windows 11 24H2 install media could boot.
- The Windows 11 24H2 on the NVMe could still boot.
- Booting AlmaLinux 9 from a live image and adding
intel_iommu=onto the command line showed the IOMMU was enabled.
Before enabling these options System Information reported (Kernel DMA Protection was off but omitted to copy the output):
- App Control for Business policy Enforced
- App Control for Business user mode policy Off
- Automatic Device Encryption Support Elevation Required to View
- Hyper-V - VM Monitor Mode Extensions Yes
- Hyper-V - Second Level Address Translation Extensions Yes
- Hyper-V - Virtualisation Enabled in Firmware No
- Hyper-V - Data Execution Protection Yes
And in this state the option to install the Windows Sandbox feature is grayed out, with a tooltip:
Windows Sandbox cannot be installed: Virtualisation support is disabled in the firmware.
Enable VTx and TXT:
C:\SWSetup\SP143621>BiosConfigUtility64.exe /setvalue:"Virtualization Technology (VTx)","Enable"
<BIOSCONFIG Version="" Computername="DESKTOP-BVUMP11" Date="2025/01/25" Time="20:44:27" UTC="0">
<SETTING changeStatus="pass" name="Virtualization Technology (VTx)" returnCode="0">
<OLDVALUE><![CDATA[Disable]]></OLDVALUE>
<VALUE><![CDATA[Enable]]></VALUE>
</SETTING>
<SUCCESS msg="No errors occurred" />
<Information msg="BCU return value" real="0" translated="0" />
</BIOSCONFIG>
C:\SWSetup\SP143621>BiosConfigUtility64.exe /setvalue:"Trusted Execution Technology (TXT)","Enable"
<BIOSCONFIG Version="" Computername="DESKTOP-BVUMP11" Date="2025/01/25" Time="20:45:03" UTC="0">
<SETTING changeStatus="pass" name="Trusted Execution Technology (TXT)" returnCode="0">
<OLDVALUE><![CDATA[Disable]]></OLDVALUE>
<VALUE><![CDATA[Enable]]></VALUE>
</SETTING>
<SUCCESS msg="No errors occurred" />
<Information msg="BCU return value" real="0" translated="0" />
</BIOSCONFIG>
After a reboot System Information reported:
- Kernel DMA Protection Off
- Virtualisation-based security Not enabled
- App Control for Business policy Enforced
- App Control for Business user mode policy Off
- Automatic Device Encryption Support Elevation Required to View
- Hyper-V - VM Monitor Mode Extensions Yes
- Hyper-V - Second Level Address Translation Extensions Yes
- Hyper-V - Virtualisation Enabled in Firmware Yes
- Hyper-V - Data Execution Protection Yes
I.e. Virtualisation is now Enabled in Firmware, but Kernel DMA Protection is still Off.
Was able to install Windows Sandbox, and after installing System Information reports the following showing a hypervisor is running:
- Kernel DMA Protection Off
- Virtualisation-based security Running
- Virtualisation-based security required security properties
- Virtualisation-based security available security properties Base Virtualisation Support, Secure Boot, DMA Protection, UEFI Code Readonly, SMM Security Mitigations 1.0, Mode Based Execution Control, APIC Virtualisation
- Virtualisation-based security services configured
- Virtualisation-based security services running
- App Control for Business policy Enforced
- App Control for Business user mode policy Off
- Automatic Device Encryption Support Elevation Required to View
- A hypervisor has been detected. Features required for Hyper-V will not be displayed.
Device Security in the Windows Security App explains to enable Core Isolation. Enabled Memory Integrity which was the only disabled option. Kernel DMA Protection Off was still reported after a reboot.
- Shows a Memory Access Protection section on the Core Isolation settings. However, threre is no Memory Access Protection shown on the PC.
- Says if after enabling VTx and VTd:
If the state of Kernel DMA Protection remains Off, then the system doesn't support Kernel DMA Protection.
Not dure if Memory Access Protection is only related for Thunderbold and USB4
Disable the internal speakers to avoid unwanted sound effects:
C:\SWSetup\SP143621>BiosConfigUtility64.exe /setvalue:"Internal Speakers","Disable"
<BIOSCONFIG Version="" Computername="DESKTOP-BVUMP11" Date="2025/03/22" Time="09:43:41" UTC="0">
<SETTING changeStatus="pass" name="Internal Speakers" returnCode="0">
<OLDVALUE><![CDATA[Enable]]></OLDVALUE>
<VALUE><![CDATA[Disable]]></VALUE>
</SETTING>
<SUCCESS msg="No errors occurred" />
<Information msg="BCU return value" real="0" translated="0" />
</BIOSCONFIG>
Following this change, after booting into Windows 11 the task bar now shows the speaker icon with a X and says "No audio device is installed".
Enable booting from a SD card, since use that to boot live Linux images and the built in front panel SD card slot seems faster than when using a USB to SD card adapter:
C:\SWSetup\SP143621>BiosConfigUtility64.exe /setvalue:"SD Card Boot","Enable"
<BIOSCONFIG Version="" Computername="DESKTOP-BVUMP11" Date="2025/03/22" Time="09:45:49" UTC="0">
<SETTING changeStatus="pass" name="SD Card Boot" returnCode="0">
<OLDVALUE><![CDATA[Disable]]></OLDVALUE>
<VALUE><![CDATA[Enable]]></VALUE>
</SETTING>
<SUCCESS msg="No errors occurred" />
<Information msg="BCU return value" real="0" translated="0" />
</BIOSCONFIG>
On trying to "remove" the plastic in the SD Card slot found How Do One Open the SD Card Slot on HP Z4 Workstation? which says the SD Card is an option, for which need HP SD 4 Card Reader Zx G4 (2VK54AA)
Installed FPGA boards in slots 2 and 4.
The board in slot 2 enumerated on the PCIe at power up. After loaded a different FPGA bitstream over JTAG while the PC was booted into Windows 11 the PC ended up on a BIOS error screen about suprise link-down.
The previous saved BIOS configuration shows Hot Plut disabled on all 6 PCIe slots and the two M.2 slots.
For both slots enabled Hot Plug and 8 Hot Plug buses:
C:\SWSetup\SP143621>BiosConfigUtility.exe /setvalue:"Slot 2 Hot Plug","Enable"
<BIOSCONFIG Version="" Computername="DESKTOP-BVUMP11" Date="2025/07/26" Time="17:50:31" UTC="1">
<SETTING changeStatus="pass" name="Slot 2 Hot Plug" returnCode="0">
<OLDVALUE><![CDATA[Disable]]></OLDVALUE>
<VALUE><![CDATA[Enable]]></VALUE>
</SETTING>
<SUCCESS msg="No errors occurred" />
<Information msg="BCU return value" real="0" translated="0" />
</BIOSCONFIG>
C:\SWSetup\SP143621>BiosConfigUtility.exe /setvalue:"Slot 4 Hot Plug","Enable"
<BIOSCONFIG Version="" Computername="DESKTOP-BVUMP11" Date="2025/07/26" Time="17:51:05" UTC="1">
<SETTING changeStatus="pass" name="Slot 4 Hot Plug" returnCode="0">
<OLDVALUE><![CDATA[Disable]]></OLDVALUE>
<VALUE><![CDATA[Enable]]></VALUE>
</SETTING>
<SUCCESS msg="No errors occurred" />
<Information msg="BCU return value" real="0" translated="0" />
</BIOSCONFIG>
C:\SWSetup\SP143621>BiosConfigUtility.exe /setvalue:"Slot 2 Hot Plug Buses","8"
<BIOSCONFIG Version="" Computername="DESKTOP-BVUMP11" Date="2025/07/26" Time="17:51:40" UTC="1">
<SETTING changeStatus="pass" name="Slot 2 Hot Plug Buses" returnCode="0">
<OLDVALUE><![CDATA[0]]></OLDVALUE>
<VALUE><![CDATA[8]]></VALUE>
</SETTING>
<SUCCESS msg="No errors occurred" />
<Information msg="BCU return value" real="0" translated="0" />
</BIOSCONFIG>
C:\SWSetup\SP143621>BiosConfigUtility.exe /setvalue:"Slot 4 Hot Plug Buses","8"
<BIOSCONFIG Version="" Computername="DESKTOP-BVUMP11" Date="2025/07/26" Time="17:52:10" UTC="1">
<SETTING changeStatus="pass" name="Slot 4 Hot Plug Buses" returnCode="0">
<OLDVALUE><![CDATA[0]]></OLDVALUE>
<VALUE><![CDATA[8]]></VALUE>
</SETTING>
<SUCCESS msg="No errors occurred" />
<Information msg="BCU return value" real="0" translated="0" />
</BIOSCONFIG>
After using BiosConfigUtility.exe /get:05_enable_PCIe_slot_hot_plug.txt there was no record of the BIOS error about an unexpected PCIe link down. The BIOS Event Logging setting is Disable which probably explains the lack of the BIOS error.
Following enabling PCIe Hot Plug, when reloaded both FPGAs while Windows 11 was running, no BIOS error occurred. Windows 11 also appears to support hot-plug since the expected devices are present:
PS C:\Users\mr_halfword> Get-PnpDevice -InstanceId '*VEN_10EE*' -PresentOnly | Format-Table -AutoSize
Status Class FriendlyName InstanceId
------ ----- ------------ ----------
OK System PCI standard RAM Controller PCI\VEN_10EE&DEV_7024&SUBSYS_00180002&REV_00\4&1F221E6C&0&0000
OK System PCI standard RAM Controller PCI\VEN_10EE&DEV_9038&SUBSYS_001A0002&REV_00\4&170E5B98&0&0000
Ran the script from Is there a way to identify the PCIE Speed for a device using powershell (Win10)?:
PS C:\Users\mr_halfword> $pciStats | Format-Table -AutoSize -Wrap
Name ExpressSpecVersion MaxLinkSpeed MaxLinkWidth CurrentLinkSpeed Curren
tLinkW
idth
---- ------------------ ------------ ------------ ---------------- ------
Standard NVM Express Controller 2 3 4 3 4
Intel(R) Ethernet Connection X722 for 10GBASE-T 2 1 1 1 1
Intel(R) Ethernet Connection X722 for 10GBASE-T #2 2 1 1 1 1
Intel(R) Ethernet Connection X722 for 1GbE 2 1 1 1 1
High Definition Audio Controller 2 3 16 1 4
NVIDIA Quadro K620 2 3 16 1 4
PCI standard RAM Controller 2 2 4 2 4
PCI standard RAM Controller 2 3 8 3 4
Mellanox ConnectX-4 Adapter 2 3 16 3 16
In 7) above:
-
Slot 2 was poplated with a TOSING_160T board, which has a PCIe Gen2 x4 interface. Slot 2 is PCIe Gen3 x16 and as expected the board enumerated as PCIe Gen2 x4.
-
Slot 4 was populated with a AS02MC04 board, which has a PCIe v3 x8 interface. HP Z6 G4 Workstation QuickSpecs contains:
Slot 4: PCI Express Gen3 x8 – CPU with open-ended connector (slot converts to x4 electrical when SSD is installed in 2nd M.2 slot)
Figure 1. HP Z6 G4 Workstation Block Diagram in HP Z6 G4 WORKSTATION contains:
The HP Z6 G4 provides two PCIe3 x16 and three PCIe3 x4 dedicated electrical slots. An additional two PCIe3 x4 buses feed the two on-board M.2 slots. The sixth high-performance I/O slot serves as a PCIe3 x4 or x8 slot depending on whether the second M.2 slot is occupied. A PCIe mux switches in an additional four PCIe lanes when the second M.2 slot is not used.
Since the 2nd M.2 slot isn't populated, expected slot 4 to operate at a x8 width. 3.5. TOSING_160T_dma_stream_crc64 and AS02MC04_dma_stream_crc64 shows the slot 4 root port was reporting Link capabilities: Max speed 8 GT/s Max width x4. I.e. the root port was only a x4, rather than the expected x8 width.
In case the issue was that even though M.2 SSD1 wasn't populated but enabled, disabled in the BIOS:
C:\SWSetup\SP143621>BiosConfigUtility.exe /setvalue:"M.2 SSD1","Disable"
<BIOSCONFIG Version="" Computername="DESKTOP-BVUMP11" Date="2025/07/26" Time="21:39:02" UTC="1">
<SETTING changeStatus="pass" name="M.2 SSD1" returnCode="0">
<OLDVALUE><![CDATA[Enable]]></OLDVALUE>
<VALUE><![CDATA[Disable]]></VALUE>
</SETTING>
<SUCCESS msg="No errors occurred" />
<Information msg="BCU return value" real="0" translated="0" />
</BIOSCONFIG>
The above was to try and make Slot 4 as x8 width. However, still a x4 width:
PS C:\Users\mr_halfword> $pciStats | Format-Table -AutoSize
Name ExpressSpecVersion MaxLinkSpeed MaxLinkWidth CurrentLinkSpeed Curren
tLinkW
idth
---- ------------------ ------------ ------------ ---------------- ------
Standard NVM Express Controller 2 3 4 3 4
PCI Memory Controller 2 2 4 2 4
Intel(R) Ethernet Connection X722 for 10GBASE-T 2 1 1 1 1
Intel(R) Ethernet Connection X722 for 10GBASE-T #2 2 1 1 1 1
Intel(R) Ethernet Connection X722 for 1GbE 2 1 1 1 1
High Definition Audio Controller 2 3 16 1 4
NVIDIA Quadro K620 2 3 16 1 4
PCI standard RAM Controller 2 3 8 3 4
Mellanox ConnectX-4 Adapter 2 3 16 3 16
Moved the boards:
- Moved the TOSING_160T board from slot 2 to slot 3 (Gen3 x4).
- Moved the AS02MC04 board from slot slot 4 to slot 2 (Gen3 x16).
Following the move the TOSING_160T board still enumerated, but the AS02MC04 board didn't.
Dumping the PCIe root ports information:
linux@DESKTOP-BVUMP11:~/fpga_sio/software_tests/eclipse_project/bin/release> dump_info/dump_pci_info_pciutils 8086:2030
domain=0000 bus=98 dev=00 func=00 rev=04
vendor_id=8086 (Intel Corporation) device_id=2030 (Sky Lake-E PCI Express Root Port A)
iommu_group=134
driver=pcieport
physical_slot=0-5
control: I/O+ Mem+ BusMaster+ ParErr- SERR- DisINTx+
status: INTx- <ParErr- >TAbort- <TAbort- <MAbort- >SERR- DetParErr-
Capabilities: [40] Bridge subsystem vendor/device ID
Capabilities: [60] Message Signaled Interrupts
Capabilities: [90] PCI Express v2 Root Port, MSI 0
Link capabilities: Max speed 8 GT/s Max width x4
Negotiated link status: Current speed 2.5 GT/s Width x0
Link capabilities2: Supported link speeds 2.5 GT/s 5.0 GT/s 8.0 GT/s
DevCap: MaxPayload 256 bytes PhantFunc 0 Latency L0s Maximum of 64 ns L1 Maximum of 1 μs
ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 0.000W
DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
LnkCap: Port # 0 ASPM L1
L0s Exit Latency 256 ns to less than 512 ns
L1 Exit Latency 2 μs to less than 4 μs
ClockPM- Surprise+ LLActRep+ BwNot+ ASPMOptComp+
LnkCtl: ASPM Disabled RCB 64 bytes Disabled+ CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- ABWMgmt-
LnkSta: TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
Capabilities: [e0] Power Management
domain=0000 bus=80 dev=00 func=00 rev=04
vendor_id=8086 (Intel Corporation) device_id=2030 (Sky Lake-E PCI Express Root Port A)
iommu_group=102
driver=pcieport
physical_slot=0
control: I/O+ Mem+ BusMaster+ ParErr- SERR- DisINTx+
status: INTx- <ParErr- >TAbort- <TAbort- <MAbort- >SERR- DetParErr-
Capabilities: [40] Bridge subsystem vendor/device ID
Capabilities: [60] Message Signaled Interrupts
Capabilities: [90] PCI Express v2 Root Port, MSI 0
Link capabilities: Max speed 8 GT/s Max width x4
Negotiated link status: Current speed 2.5 GT/s Width x0
Link capabilities2: Supported link speeds 2.5 GT/s 5.0 GT/s 8.0 GT/s
DevCap: MaxPayload 256 bytes PhantFunc 0 Latency L0s Maximum of 64 ns L1 Maximum of 1 μs
ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 0.000W
DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
LnkCap: Port # 0 ASPM not supported
L0s Exit Latency 256 ns to less than 512 ns
L1 Exit Latency 8 μs to less than 16 μs
ClockPM- Surprise+ LLActRep+ BwNot+ ASPMOptComp+
LnkCtl: ASPM Disabled RCB 64 bytes Disabled- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- ABWMgmt-
LnkSta: TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
Capabilities: [e0] Power Management
domain=0000 bus=18 dev=00 func=00 rev=04
vendor_id=8086 (Intel Corporation) device_id=2030 (Sky Lake-E PCI Express Root Port A)
iommu_group=34
driver=pcieport
physical_slot=4
control: I/O+ Mem+ BusMaster+ ParErr- SERR- DisINTx+
status: INTx- <ParErr- >TAbort- <TAbort- <MAbort- >SERR- DetParErr-
Capabilities: [40] Bridge subsystem vendor/device ID
Capabilities: [60] Message Signaled Interrupts
Capabilities: [90] PCI Express v2 Root Port, MSI 0
Link capabilities: Max speed 8 GT/s Max width x8
Negotiated link status: Current speed 2.5 GT/s Width x0
Link capabilities2: Supported link speeds 2.5 GT/s 5.0 GT/s 8.0 GT/s
DevCap: MaxPayload 256 bytes PhantFunc 0 Latency L0s Maximum of 64 ns L1 Maximum of 1 μs
ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 0.000W
DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq-
RlxdOrd- ExtTag+ PhantFunc- AuxPwr- NoSnoop-
DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
LnkCap: Port # 1 ASPM not supported
L0s Exit Latency 256 ns to less than 512 ns
L1 Exit Latency 8 μs to less than 16 μs
ClockPM- Surprise+ LLActRep+ BwNot+ ASPMOptComp+
LnkCtl: ASPM Disabled RCB 64 bytes Disabled- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- ABWMgmt-
LnkSta: TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug+ Surprise-
Slot #4 PowerLimit 0.000W Interlock- NoCompl-
Capabilities: [e0] Power Management
domain=0000 bus=30 dev=00 func=00 rev=04
vendor_id=8086 (Intel Corporation) device_id=2030 (Sky Lake-E PCI Express Root Port A)
iommu_group=52
driver=pcieport
physical_slot=2
control: I/O+ Mem+ BusMaster+ ParErr- SERR- DisINTx+
status: INTx- <ParErr- >TAbort- <TAbort- <MAbort- >SERR- DetParErr-
Capabilities: [40] Bridge subsystem vendor/device ID
Capabilities: [60] Message Signaled Interrupts
Capabilities: [90] PCI Express v2 Root Port, MSI 0
Link capabilities: Max speed 8 GT/s Max width x4
Negotiated link status: Current speed 2.5 GT/s Width x0
Link capabilities2: Supported link speeds 2.5 GT/s 5.0 GT/s 8.0 GT/s
DevCap: MaxPayload 256 bytes PhantFunc 0 Latency L0s Maximum of 64 ns L1 Maximum of 1 μs
ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 0.000W
DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq-
RlxdOrd- ExtTag+ PhantFunc- AuxPwr- NoSnoop-
DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
LnkCap: Port # 5 ASPM not supported
L0s Exit Latency 256 ns to less than 512 ns
L1 Exit Latency 8 μs to less than 16 μs
ClockPM- Surprise+ LLActRep+ BwNot+ ASPMOptComp+
LnkCtl: ASPM Disabled RCB 64 bytes Disabled- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- ABWMgmt-
LnkSta: TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug+ Surprise-
Slot #2 PowerLimit 0.000W Interlock- NoCompl-
Capabilities: [e0] Power Management
domain=0000 bus=a4 dev=00 func=00 rev=04
vendor_id=8086 (Intel Corporation) device_id=2030 (Sky Lake-E PCI Express Root Port A)
iommu_group=167
driver=pcieport
physical_slot=0-9
control: I/O+ Mem+ BusMaster+ ParErr- SERR- DisINTx+
status: INTx- <ParErr- >TAbort- <TAbort- <MAbort- >SERR- DetParErr-
Capabilities: [40] Bridge subsystem vendor/device ID
Capabilities: [60] Message Signaled Interrupts
Capabilities: [90] PCI Express v2 Root Port, MSI 0
Link capabilities: Max speed 8 GT/s Max width x4
Negotiated link status: Current speed 2.5 GT/s Width x0
Link capabilities2: Supported link speeds 2.5 GT/s 5.0 GT/s 8.0 GT/s
DevCap: MaxPayload 256 bytes PhantFunc 0 Latency L0s Maximum of 64 ns L1 Maximum of 1 μs
ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 0.000W
DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
LnkCap: Port # 0 ASPM L1
L0s Exit Latency 256 ns to less than 512 ns
L1 Exit Latency 2 μs to less than 4 μs
ClockPM- Surprise+ LLActRep+ BwNot+ ASPMOptComp+
LnkCtl: ASPM Disabled RCB 64 bytes Disabled+ CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- ABWMgmt-
LnkSta: TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
Capabilities: [e0] Power Management
domain=0000 bus=8c dev=00 func=00 rev=04
vendor_id=8086 (Intel Corporation) device_id=2030 (Sky Lake-E PCI Express Root Port A)
iommu_group=117
driver=pcieport
physical_slot=0-1
control: I/O+ Mem+ BusMaster+ ParErr- SERR- DisINTx+
status: INTx- <ParErr- >TAbort- <TAbort- <MAbort- >SERR- DetParErr-
Capabilities: [40] Bridge subsystem vendor/device ID
Capabilities: [60] Message Signaled Interrupts
Capabilities: [90] PCI Express v2 Root Port, MSI 0
Link capabilities: Max speed 8 GT/s Max width x4
Negotiated link status: Current speed 2.5 GT/s Width x0
Link capabilities2: Supported link speeds 2.5 GT/s 5.0 GT/s 8.0 GT/s
DevCap: MaxPayload 256 bytes PhantFunc 0 Latency L0s Maximum of 64 ns L1 Maximum of 1 μs
ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 0.000W
DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
LnkCap: Port # 0 ASPM L1
L0s Exit Latency 256 ns to less than 512 ns
L1 Exit Latency 2 μs to less than 4 μs
ClockPM- Surprise+ LLActRep+ BwNot+ ASPMOptComp+
LnkCtl: ASPM Disabled RCB 64 bytes Disabled+ CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- ABWMgmt-
LnkSta: TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
Capabilities: [e0] Power Management
domain=0000 bus=49 dev=00 func=00 rev=04
vendor_id=8086 (Intel Corporation) device_id=2030 (Sky Lake-E PCI Express Root Port A)
iommu_group=85
driver=pcieport
physical_slot=5
control: I/O+ Mem+ BusMaster+ ParErr+ SERR+ DisINTx+
status: INTx- <ParErr- >TAbort- <TAbort- <MAbort- >SERR- DetParErr-
Capabilities: [40] Bridge subsystem vendor/device ID
Capabilities: [60] Message Signaled Interrupts
Capabilities: [90] PCI Express v2 Root Port, MSI 0
Link capabilities: Max speed 8 GT/s Max width x16
Negotiated link status: Current speed 8 GT/s Width x16
Link capabilities2: Supported link speeds 2.5 GT/s 5.0 GT/s 8.0 GT/s
DevCap: MaxPayload 256 bytes PhantFunc 0 Latency L0s Maximum of 64 ns L1 Maximum of 1 μs
ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 0.000W
DevCtl: CorrErr- NonFatalErr+ FatalErr+ UnsupReq-
RlxdOrd- ExtTag+ PhantFunc- AuxPwr- NoSnoop-
DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
LnkCap: Port # 9 ASPM not supported
L0s Exit Latency 256 ns to less than 512 ns
L1 Exit Latency 8 μs to less than 16 μs
ClockPM- Surprise+ LLActRep+ BwNot+ ASPMOptComp+
LnkCtl: ASPM Disabled RCB 64 bytes Disabled- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- ABWMgmt-
LnkSta: TrErr- Train- SlotClk+ DLActive+ BWMgmt+ ABWMgmt-
SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surprise-
Slot #5 PowerLimit 0.000W Interlock- NoCompl-
Capabilities: [e0] Power Management
Which shows:
physical_slot=2which is populated with the AS02MC04 board is reported asLink capabilities: Max speed 8 GT/s Max width x4. I.e. x4 width rather than the expected x8. 3.5. TOSING_160T_dma_stream_crc64 and AS02MC04_dma_stream_crc64 shows when the TOSING_160T board was fitted to slot 2, the width was x16 as expected.physical_slot=4which is now unpopulated is reported asLink capabilities: Max speed 8 GT/s Max width x8. I.e. the expected x8 width.
Therefore, appears the AS02MC04 board is somehow impacting the maximum width of the PCIe root port of the slot the board is fitted in.
Restored the original board positions:
- With the TOSING_160T board back in slot 2, it still enumerates as expected.
- With the AS02MC04 board back in slot 4, the original behaviour of the board enumerating as Gen3 x4 is back with the PCIe root port reporting a max width of x4.
Re-enabled M.2 SSD1, since disabling it didn't help:
C:\SWSetup\SP143621>BiosConfigUtility.exe /setvalue:"M.2 SSD1","Enable"
<BIOSCONFIG Version="" Computername="DESKTOP-BVUMP11" Date="2025/07/27" Time="14:46:16" UTC="1">
<SETTING changeStatus="pass" name="M.2 SSD1" returnCode="0">
<OLDVALUE><![CDATA[Disable]]></OLDVALUE>
<VALUE><![CDATA[Enable]]></VALUE>
</SETTING>
<SUCCESS msg="No errors occurred" />
<Information msg="BCU return value" real="0" translated="0" />
</BIOSCONFIG>
The current BIOS settings has:
Slot 4 Bifurcation
*Auto
x8
x4x4
In case the automatic bifurcation is having an effect, disable it:
C:\SWSetup\SP143621>BiosConfigUtility.exe /setvalue:"Slot 4 Bifurcation","x8"
<BIOSCONFIG Version="" Computername="DESKTOP-BVUMP11" Date="2025/07/27" Time="14:50:52" UTC="1">
<SETTING changeStatus="pass" name="Slot 4 Bifurcation" returnCode="0">
<OLDVALUE><![CDATA[Auto]]></OLDVALUE>
<VALUE><![CDATA[x8]]></VALUE>
</SETTING>
<SUCCESS msg="No errors occurred" />
<Information msg="BCU return value" real="0" translated="0" />
</BIOSCONFIG>
That still left slot 4 as only x4 width, when the AS02MC04 board was fitted.
Also, seemed to coincide with the 10 GbE X722 ports no longer enumerating.
Re-enabled Auto Bifurcation:
C:\SWSetup\SP143621>BiosConfigUtility.exe /setvalue:"Slot 4 Bifurcation","Auto"
<BIOSCONFIG Version="" Computername="DESKTOP-BVUMP11" Date="2025/07/27" Time="15:20:57" UTC="1">
<SETTING changeStatus="pass" name="Slot 4 Bifurcation" returnCode="0">
<OLDVALUE><![CDATA[x8]]></OLDVALUE>
<VALUE><![CDATA[Auto]]></VALUE>
</SETTING>
<SUCCESS msg="No errors occurred" />
<Information msg="BCU return value" real="0" translated="0" />
</BIOSCONFIG>
That didn't cause the 10 GbE X722 ports to re-appear. After wiggling the cable connecting the board with the 10 GbE X722 ports they started to enumerate again, so possibly the cable worked loose while moving the boards.
As suggested in General Debug Checklist tried changing the design to change the "Link Partner TX Preset" from the default of 4 to 5. This still caused slot 4 with the AS02MC04 board fitted to have a width of x8.
Changed the design to drop to a max link speed of 5 GT/s. Enumerated at 5 GT/s with x4 rather than x8. I.e. still enumerating at a lower than expected link width.
Programmed the configuration memory with the AS02MC04_dma_stream_crc64 design, in which the user_lnk_up signal from the DMA/Bridge Subsystem for PCI Express used to light the green LED on the edge of the board when the PCI Express core is linked up
with a host device.
The sequence occurs:
- Power on the PC, and the LED doesn't indicate the link is up.
- The PCIe endpoint hasn't enumerated.
- Connect the Vivado Hardware Manager over JTAG, which indicates the FPGA is programmed.
- Use the Hardware Manager to select Boot from Configuration Memory Device and the LED indicates the link is up.
- The PCIe endpoint has enumerated as Gen3 x4.
Enable PCIe Training Reset to see if that allows the link to come up at power on:
C:\SWSetup\SP143621>BiosConfigUtility.exe /setvalue:"PCIe Training Reset","Enable"
<BIOSCONFIG Version="" Computername="DESKTOP-BVUMP11" Date="2025/08/01" Time="22:11:48" UTC="1">
<SETTING changeStatus="pass" name="PCIe Training Reset" returnCode="0">
<OLDVALUE><![CDATA[Disable]]></OLDVALUE>
<VALUE><![CDATA[Enable]]></VALUE>
</SETTING>
<SUCCESS msg="No errors occurred" />
<Information msg="BCU return value" real="0" translated="0" />
</BIOSCONFIG>
The first power cycle after enabling PCIe Training Reset allowed the link to come up, but at Gen3 x4.
On a further two power cycles the link didn't come up.
In the state where the FPGA loads at power up but the link doesn't come up in the Vivado Hardware Manager ran source AS02MC04_dma_stream_crc64/AS02MC04_dma_stream_crc64.gen/sources_1/bd/AS02MC04_dma_stream_crc64/ip/AS02MC04_dma_stream_crc64_xdma_0_0/ip_0/AS02MC04_dma_stream_crc64_xdma_0_0_pcie4_ip/pcie_debugger/test_rd.tcl. The output from the analysis scripts:
draw_ltssm.tclshows a single transition fromPolling.ActivetoPolling.Config.draw_reset.tclshows no transitionsdraw_rxdet.tclshows no lanes detected and:- Max link width: 2
- Negotiated Link width: 2
Used Vivado Hardware Manager to boot from the configuration memory, and the link came up at Gen3 x4. Rerunning the PCIe debugger scripts:
draw_ltssm.tclshows a single transition fromL1.IdletoR.Lockdraw_reset.tclshows no transitionsdraw_rxdet.tclshows no lanes detected and:- Max link width: 24
- Negotiated Link width: 18
Fitted a ConnectX-3 Pro to slot 4, which enumerated as Gen 3 x8:
linux@DESKTOP-BVUMP11:~/fpga_sio/software_tests/eclipse_project/bin/release> dump_info/dump_pci_info_pciutils 15b3:1007
domain=0000 bus=19 dev=00 func=00 rev=00
vendor_id=15b3 (Mellanox Technologies) device_id=1007 (MT27520 Family [ConnectX-3 Pro]) subvendor_id=15b3 subdevice_id=0006
iommu_group=48
driver=mlx4_core
physical_slot=4-2
control: I/O- Mem+ BusMaster+ ParErr+ SERR+ DisINTx+
status: INTx- <ParErr- >TAbort- <TAbort- <MAbort- >SERR- DetParErr-
bar[0] base_addr=96200000 size=100000 is_IO=0 is_prefetchable=0 is_64=1
bar[2] base_addr=387f70000000 size=800000 is_IO=0 is_prefetchable=1 is_64=1
Capabilities: [40] Power Management
Capabilities: [48] Vital Product Data
Capabilities: [9c] MSI-X
Capabilities: [60] PCI Express v2 Express Endpoint, MSI 0
Link capabilities: Max speed 8 GT/s Max width x8
Negotiated link status: Current speed 8 GT/s Width x8
Link capabilities2: Supported link speeds 2.5 GT/s 5.0 GT/s 8.0 GT/s
DevCap: MaxPayload 512 bytes PhantFunc 0 Latency L0s Maximum of 64 ns L1 No limit
ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 116.000W
DevCtl: CorrErr- NonFatalErr+ FatalErr+ UnsupReq-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
DevSta: CorrErr+ NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
LnkCap: Port # 8 ASPM L0s
L0s Exit Latency More than 4 μs
L1 Exit Latency More than 64 μs
ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled RCB 64 bytes Disabled- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- ABWMgmt-
LnkSta: TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
Capabilities: [c0] Vendor specific
domain=0000 bus=18 dev=00 func=00 rev=04
vendor_id=8086 (Intel Corporation) device_id=2030 (Sky Lake-E PCI Express Root Port A)
iommu_group=32
driver=pcieport
physical_slot=4
control: I/O+ Mem+ BusMaster+ ParErr+ SERR+ DisINTx+
status: INTx- <ParErr- >TAbort- <TAbort- <MAbort- >SERR- DetParErr-
Capabilities: [40] Bridge subsystem vendor/device ID
Capabilities: [60] Message Signaled Interrupts
Capabilities: [90] PCI Express v2 Root Port, MSI 0
Link capabilities: Max speed 8 GT/s Max width x8
Negotiated link status: Current speed 8 GT/s Width x8
Link capabilities2: Supported link speeds 2.5 GT/s 5.0 GT/s 8.0 GT/s
DevCap: MaxPayload 256 bytes PhantFunc 0 Latency L0s Maximum of 64 ns L1 Maximum of 1 μs
ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 0.000W
DevCtl: CorrErr- NonFatalErr+ FatalErr+ UnsupReq-
RlxdOrd- ExtTag+ PhantFunc- AuxPwr- NoSnoop-
DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
LnkCap: Port # 1 ASPM not supported
L0s Exit Latency 256 ns to less than 512 ns
L1 Exit Latency 8 μs to less than 16 μs
ClockPM- Surprise+ LLActRep+ BwNot+ ASPMOptComp+
LnkCtl: ASPM Disabled RCB 64 bytes Disabled- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- ABWMgmt-
LnkSta: TrErr- Train- SlotClk+ DLActive+ BWMgmt- ABWMgmt-
SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug+ Surprise-
Slot #4 PowerLimit 0.000W Interlock- NoCompl-
Capabilities: [e0] Power Management