Notes about XCKU5P PCIE 3.0 QSFP X2

0. Introduction

This contains notes about using the Kintex UltraScale XCKU5P PCIE 3.0 QSFP X 2 or SFP X 2 xilinx board xilinx fpga board xilixn fpga development board pcie board with Bundle: KU5P full heightQSFP.

Read the Device DNA over JTAG and using AMD Device Lookup reports:

Device	XCKU5P
Package-Pin	FFVB676
Revision Code	AAZ
SpeedGrade	2I

No schematic or PCB layout is available for the board. Asking the seller after purchase got sent an email containing:

Spreadsheet with the XCKU5P pinout. Typo in that the pcie_p_tx6 and pcie_n_tx6 pins are set to the pins for pcie_p_rx6 and pcie_n_rx6. Noticed when checking the IO Placed Report of a design. The XDC file used only the rxp pins for each lane
Picture of the Add Configuration Memory Device dialog in Vivado with name mt25qu256-spi-x1_x2_x4.
A pcie4_uscale_plus_0_ex Vivado v2021.1 project. The part is xcku5p-ffvb676-2-i. The design has:
- A UltraScale+ Integrated Block (PCIE4) for PCI Express (1.3) IP block set for 8 lanes and 8.0 GT/s and one 128K BAR.
- Some verilog example code which implements 2 Kbytes of memory space.
- Only the PCIe signals connected.
A ibert_ultrascale_gty_0_ex Vivado v2021.1 project. The part is xcku5p-ffvb676-2-i. Can re-generate the bitstream in Vivado 2021.1 under Ubuntu. The design has:
- IBERT Ultrascale GTY (1.3) with 8 lanes with a Line Rate of 10.3125 Gbps.
- RefClk of 156.25 MHz provided on GTY Refclk ,bank227 clk1.
- System Clock of 100 MHz provided on a single ended input with IO standard LVCMOS18.
- Top level fixing QSFP_RESET_A and QSFP_RESET_B output signals as 1 with IO standard LVCMOS33.

There is a Micron device with RW167 FBGA code on the board. The Micron FBGA and component marking decoder identifies this as MT25QU256ABA1EW9-0SIT. This matches the part shown in the Add Configuration Memory Device Vivado dialog. The I/O voltage is 1.8V. MT25QL256ABA1EW9-0SIT on the Micron website has a download link for the MT25QU256ABA datasheet updated 2024-02-21, but need to register. MT25QU256ABA on the Mouser website is Rev. L – 02/2024 of the datasheet.

1. PCIe inteface

The PCIE section of the pinout spreadsheet has the RESET on pin T19 but does't describe the bank voltage.

The PCIE_5P/PCIE/pcie4_uscale_plus_0_ex/imports/xilinx_pcie4_uscale_plus_x0y0.xdc file from the example project has:

create_clock -period 10.000 -name sys_clk [get_ports sys_clk_p]
#
set_false_path -from [get_ports sys_rst_n]
set_property PULLUP true [get_ports sys_rst_n]

set_property IOSTANDARD LVCMOS18 [get_ports sys_rst_n]

#
#create_waiver -type METHODOLOGY -id {LUTAR-1} -user "pcie4_uscale_plus" -desc "user link up is synchroized in the user clk so it is safe to ignore"  -internal -scoped -tags 1024539  -objects [get_cells { pcie_app_uscale_i/PIO_i/len_i[5]_i_4 }] -objects [get_pins { pcie4_uscale_plus_0_i/inst/user_lnk_up_cdc/arststages_ff_reg[0]/CLR pcie4_uscale_plus_0_i/inst/user_lnk_up_cdc/arststages_ff_reg[1]/CLR }]

set_property PACKAGE_PIN T19 [get_ports sys_rst_n]

2. FPGA image loaded into configuration memory on delivery

This enumerated on the PCIe bus with the following, taken after running bind_xilinx_devices_to_vfio.sh:

$ sudo lspci -d 10ee: -vv
[sudo] password for mr_halfword: 
01:00.0 Memory controller: Xilinx Corporation Device 9038
	Subsystem: Xilinx Corporation Device 0007
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	IOMMU group: 0
	Region 0: Memory at fe300000 (32-bit, non-prefetchable) [disabled] [size=128K]
	Capabilities: [40] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
		Status: D3 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [48] MSI: Enable- Count=1/1 Maskable- 64bit+
		Address: 0000000000000000  Data: 0000
	Capabilities: [70] Express (v2) Endpoint, MSI 00
		DevCap:	MaxPayload 1024 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
			ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 0.000W
		DevCtl:	CorrErr+ NonFatalErr+ FatalErr+ UnsupReq+
			RlxdOrd- ExtTag+ PhantFunc- AuxPwr- NoSnoop+
			MaxPayload 128 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
		LnkCap:	Port #0, Speed 8GT/s, Width x8, ASPM not supported
			ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
		LnkCtl:	ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 5GT/s (downgraded), Width x8 (ok)
			TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
		DevCap2: Completion Timeout: Range BC, TimeoutDis+ NROPrPrP- LTR-
			 10BitTagComp- 10BitTagReq- OBFF Not Supported, ExtFmt- EETLPPrefix-
			 EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
			 FRS- TPHComp- ExtTPHComp-
			 AtomicOpsCap: 32bit- 64bit- 128bitCAS-
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR- OBFF Disabled,
			 AtomicOpsCtl: ReqEn-
		LnkCap2: Supported Link Speeds: 2.5-8GT/s, Crosslink- Retimer- 2Retimers- DRS-
		LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete- EqualizationPhase1-
			 EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest-
			 Retimer- 2Retimers- CrosslinkRes: unsupported
	Capabilities: [100 v1] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UESvrt:	DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr-
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
		AERCap:	First Error Pointer: 00, ECRCGenCap- ECRCGenEn- ECRCChkCap- ECRCChkEn-
			MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
		HeaderLog: 00000000 00000000 00000000 00000000
	Capabilities: [1c0 v1] Secondary PCI Express
		LnkCtl3: LnkEquIntrruptEn- PerformEqu-
		LaneErrStat: 0
	Kernel driver in use: vfio-pci

$ dump_info/dump_pci_info_pciutils 
domain=0000 bus=01 dev=00 func=00
  vendor_id=10ee (Xilinx Corporation) device_id=9038 (Device 9038) subvendor_id=10ee subdevice_id=0007
  iommu_group=0
  driver=vfio-pci
  control: I/O- Mem- BusMaster- ParErr- SERR- DisINTx-
  status: INTx- <ParErr- >TAbort- <TAbort- <MAbort- >SERR- DetParErr-
  bar[0] base_addr=fe300000 size=20000 is_IO=0 is_prefetchable=0 is_64=0
  Capabilities: [40] Power Management
  Capabilities: [48] Message Signaled Interrupts
  Capabilities: [70] PCI Express v2 Express Endpoint, MSI 0
    Link capabilities: Max speed 8.0 GT/s Max width x8
    Negotiated link status: Current speed 5.0 GT/s Width x8
    Link capabilities2: Supported link speeds 2.5 GT/s 5.0 GT/s 8.0 GT/s
    DevCap: MaxPayload 1024 bytes PhantFunc 0 Latency L0s Maximum of 64 ns L1 Maximum of 1 μs
            ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 0.000W
    DevCtl: CorrErr+ NonFatalErr+ FatalErr+ UnsupReq+
            RlxdOrd- ExtTag+ PhantFunc- AuxPwr- NoSnoop+
    DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
    LnkCap: Port # 0 ASPM not supported
            L0s Exit Latency More than 4 μs
            L1 Exit Latency More than 64 μs
            ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
    LnkCtl: ASPM Disabled RCB 64 bytes Disabled- CommClk+
            ExtSynch- ClockPM- AutWidDis- BWInt- ABWMgmt-
    LnkSta: TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
  domain=0000 bus=00 dev=01 func=00
    vendor_id=8086 (Intel Corporation) device_id=0101 (Xeon E3-1200/2nd Generation Core Processor Family PCI Express Root Port)
    driver=pcieport
    control: I/O+ Mem+ BusMaster+ ParErr- SERR- DisINTx+
    status: INTx- <ParErr- >TAbort- <TAbort- <MAbort- >SERR- DetParErr-
    Capabilities: [88] Bridge subsystem vendor/device ID
    Capabilities: [80] Power Management
    Capabilities: [90] Message Signaled Interrupts
    Capabilities: [a0] PCI Express v2 Root Port, MSI 0
      Link capabilities: Max speed 5.0 GT/s Max width x16
      Negotiated link status: Current speed 5.0 GT/s Width x8
      Link capabilities2: Not implemented
      DevCap: MaxPayload 128 bytes PhantFunc 0 Latency L0s Maximum of 64 ns L1 Maximum of 1 μs
              ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 0.000W
      DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq-
              RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
      DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
      LnkCap: Port # 2 ASPM L0s and L1
              L0s Exit Latency 128 ns to less than 256 ns
              L1 Exit Latency 2 μs to less than 4 μs
              ClockPM- Surprise- LLActRep- BwNot+ ASPMOptComp-
      LnkCtl: ASPM Disabled RCB 64 bytes Disabled- CommClk+
              ExtSynch- ClockPM- AutWidDis- BWInt- ABWMgmt-
      LnkSta: TrErr- Train- SlotClk+ DLActive- BWMgmt+ ABWMgmt-
      SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surprise-
              Slot #0 PowerLimit 0.000W Interlock- NoCompl+

The negotiated width is x8 as expected.

The negotiated speed is 5GT/s as expected. The FPGA design supports 2.5-8GT/s, but the root complex a max of 5GT/s.

If run the program which uses VFIO to access the device, and therefore perform a PCI reset during the open, then the link speed drops to 2.5 GT/s

$ dump_info/dump_pci_info_vfio 
Opening device 0000:01:00.0 (10ee:9038) with IOMMU group 0
domain=0000 bus=01 dev=00 func=00
  vendor_id=10ee (Xilinx Corporation) device_id=9038 (Device 9038) subvendor_id=10ee subdevice_id=0007
  iommu_group=0
  driver=vfio-pci
  control: I/O- Mem+ BusMaster- ParErr- SERR- DisINTx-
  status: INTx- <ParErr- >TAbort- <TAbort- <MAbort- >SERR- DetParErr-
  bar[0] base_addr=fe300000 size=20000 is_IO=0 is_prefetchable=0 is_64=0
  Capabilities: [40] Power Management
  Capabilities: [48] Message Signaled Interrupts
  Capabilities: [70] PCI Express v2 Express Endpoint, MSI 0
    Link capabilities: Max speed 8.0 GT/s Max width x8
    Negotiated link status: Current speed 2.5 GT/s Width x8
    Link capabilities2: Supported link speeds 2.5 GT/s 5.0 GT/s 8.0 GT/s
    DevCap: MaxPayload 1024 bytes PhantFunc 0 Latency L0s Maximum of 64 ns L1 Maximum of 1 μs
            ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 0.000W
    DevCtl: CorrErr+ NonFatalErr+ FatalErr+ UnsupReq+
            RlxdOrd- ExtTag+ PhantFunc- AuxPwr- NoSnoop+
    DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
    LnkCap: Port # 0 ASPM not supported
            L0s Exit Latency More than 4 μs
            L1 Exit Latency More than 64 μs
            ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
    LnkCtl: ASPM Disabled RCB 64 bytes Disabled- CommClk+
            ExtSynch- ClockPM- AutWidDis- BWInt- ABWMgmt-
    LnkSta: TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-

The negotiated link speed dropping from 5.0 GT/s to 2.5 GT/s after a PCI reset caused when VFIO opened the device has been seen with a different FPGA card in the same PC, so likely the link speed drop is due to something in the PC.

Also, sometimes the width enumerates as 1x, but following a reboot of the PC then enumerates as the expected x8 width. Seen after:

Powered on the PC, and the FPGA loads from the as delivered bitstream in the configuration flash.
After loading a new bitstream over JTAG.

3. Accessing the configuration flash over the PCIe bus

3.1. Success with a exp_spi_clk of 250 MHz in XCKU5P_DUAL_QSFP_dma_stream_loopback design

The actual configuration flash is a Micron MT25QU256ABA. Running the driver which was written for a Micron N25Q256A for a different board was able to identify the flash are parse a valid configuration bitstream (with the as delivered configuration contents):

$ xilinx_quad_spi/quad_spi_flasher 
Opening device 0000:01:00.0 (10ee:9038) with IOMMU group 0
Enabled bus master for 0000:01:00.0

Displaying information for SPI flash using XCKU5P_DUAL_QSFP_dma_stream_loopback design in PCI device 0000:01:00.0 IOMMU group 0
Initial device identification incorrect - ignoring due to Quad SPI core not outputting initial clock cycles
FIFO depth=256
Flash device : Micron N25Q256A
Manufacturer ID=0x20  Memory Interface Type=0xbb  Density=0x19
Flash Size Bytes=33554432  Page Size Bytes=256  Num Address Bytes=4
Successfully parsed bitstream of length 4958368 bytes with 299662 configuration packets
Read 4980736 bytes from SPI flash starting at address 0
Sync word at byte index 0x50
  Type 1 packet opcode NOP
  Type 1 packet opcode write register BSPI words 0000066C
  Type 1 packet opcode write register CMD command BSPI_READ
  Type 1 packet opcode NOP (2 consecutive)
  Type 1 packet opcode write register TIMER words 00000000
  Type 1 packet opcode write register WBSTAR words 00000000
  Type 1 packet opcode write register CMD command NULL
  Type 1 packet opcode NOP
  Type 1 packet opcode write register CMD command RCRC
  Type 1 packet opcode NOP (2 consecutive)
  Type 1 packet opcode write register FAR words 00000000
  Type 1 packet opcode write register RBCRC_SW words 00000000
  Type 1 packet opcode write register COR0 words 383C3FE5
  Type 1 packet opcode write register COR1 words 00400000
  Type 1 packet opcode write register IDCODE KU5P
  Type 1 packet opcode write register CMD command FALL_EDGE
  Type 1 packet opcode write register CMD command SWITCH
  Type 1 packet opcode NOP
  Type 1 packet opcode write register MASK words 00000001
  Type 1 packet opcode write register CTL0 words 00000101
  Type 1 packet opcode write register MASK words 00001000
  Type 1 packet opcode write register CTL1 words 00001000
  Type 1 packet opcode NOP (8 consecutive)
  Configuration data writes consisting of:
    222217 NOPs
    38213 FAR writes
    371 WCFG commands
    371 FDRI writes with a total of 97185 words
    80 MFW commands
    80 NULL commands
    37842 MFWR writes with a total of 529788 words
    25 Type 2 packets with a total of 274164 words
  Type 1 packet opcode write register CRC words B6F9F409
  Type 1 packet opcode NOP (2 consecutive)
  Type 1 packet opcode write register CMD command GRESTORE
  Type 1 packet opcode NOP (2 consecutive)
  Type 1 packet opcode write register CMD command DGHIGH_LFRM
  Type 1 packet opcode NOP (20 consecutive)
  Type 1 packet opcode write register MASK words 00001000
  Type 1 packet opcode write register CTL1 words 00000000
  Type 1 packet opcode write register CMD command START
  Type 1 packet opcode NOP
  Type 1 packet opcode write register FAR words 07FC0000
  Type 1 packet opcode write register MASK words 00000101
  Type 1 packet opcode write register CTL0 words 00000101
  Type 1 packet opcode write register CRC words 5FFE959E
  Type 1 packet opcode NOP (2 consecutive)
  Type 1 packet opcode write register CMD command DESYNC
  Type 1 packet opcode NOP (393 consecutive)

For the above XCKU5P_DUAL_QSFP_dma_stream_loopback design, the exp_spi_clk of the AXI QUAD SPI was connected to the axi_aclk output of the DMA Bridge Subsystem for PCI Express, such that exp_spi_clk is 250 MHz.

3.2. Failure with a exp_spi_clk of 250 MHz in XCKU5P_DUAL_QSFP_ibert design

The XCKU5P_DUAL_QSFP_ibert_4.166 design copied the QUAD SPI configuration from the XCKU5P_DUAL_QSFP_dma_stream_loopback design.

XCKU5P_DUAL_QSFP_ibert fails to read QSPI with:

$ xilinx_quad_spi/quad_spi_flasher -d 0000:2d:00.0
Opening device 0000:2d:00.0 (10ee:9038) with IOMMU group 25

Displaying information for SPI flash using XCKU5P_DUAL_QSFP_ibert design in PCI device 0000:2d:00.0 IOMMU group 25
Unsupported flash manufacturer_id 0xa0 (memory_interface_type 0xbb density 0x19)
Failed to initialise Quad SPI controller

The memory_interface_type and density are the expected values, but the manufacturer_id is read as 0xa0 rather than than the expected 0x20.

3.3. Reduce exp_spi_clk frequency

PG153 says that in QUAD mode, ext_spi_clk is double the frequency of the intended SPI clock.

So, with exp_spi_clk as 250 MHz the external SPI clock will be 125 MHz.

The op-code which failed was 9F Read identification ID. From the N25Q256A this op-code uses the extended SPI protocol which allows a clock frequency of up to 166 MHz. I.e. should be OK with an external SPI clock frequency of 125 MHz.

Initially attmepted to add a Clocking Wizard which takes the 250 MHz axi_clk as an input and produces a 80 MHz exp_spi_clk output, without using and reset or locked logic. On some runs of quad_spi_flasher failed with:

$ xilinx_quad_spi/quad_spi_flasher -d 0000:2d:00.0
Opening device 0000:2d:00.0 (10ee:9038) with IOMMU group 25

Displaying information for SPI flash using XCKU5P_DUAL_QSFP_ibert design in PCI device 0000:2d:00.0 IOMMU group 25
Invalid Quad SPI core fifo_depth of 0
Failed to initialise Quad SPI controller

Connected the exp_spi_clk to the 100 MHz gty_sysclkp_i, rather than 250 MHz axi_clk. Where gty_sysclkp_i is an external clock and therefore should always be running even when the PCIe block gets a hot reset. Initially still had some Invalid Quad SPI core fifo_depth of 0 errors but then couldn't repeat.

Temporarily used a Clock Wizard with the 100 MHz input from gty_sysclkp_i and a 6.25 MHz output (minimum possible) for exp_spi_clk. With this slower clock speed:

When run at full speed faild every time.
When single stepped the statement which the write which asserts a software reset of the AXI Quad SPI then could read flash OK.

The above pointed at a timing problem with how quickly attempted to use the Quad SPI after the software reset. After inserting a software added code to poll the Quad SPI status register and wait for the Tx FIFO full bit to clear. This prevented the failures with a 6.5 MHz exp_spi_clk. Coverage showed took 7 Tx FIFO full bit to clear.

The software poll was added since PG153 does't seem to describe how to determine when the software reset has completed, and didn't try a simulation.

Committed the software change to wait for the reset to complete, along with the FPGA to use 100 MHz for exp_spi_clk. Seems stable.

4. Banks in use

Looking at the usage of different banks, to determine potential for VUSER supplies to monitor:

Signal(s)	Bank	IO standard
PCIe lanes 0-3	225
PCIe lanes 4-7	224
CLK_PCIe_100MHz_clk	225
PCI_PERSTN	65	LVCMOS18
QSFP lanes 0-3	226
QSFP lanes 4-7	227
gty_refclk1	227
gty_refclk0	227
QSFP_RESET_B	86	LVCMOS33
QSFP_RESET_A	84	LVCMOS33
gty_sysclkp_i	66	LVCMOS18
QSFP A SCL	87	LVCMOS33

The above is partial, and doesn't cover all the discrete signals.

Will enable the following user supplies in the SYSMON, selected to try and get different voltages:

Channel	Bank	Monitored supply	Comment
USER0	226	AVCC	Analog supply voltage for transceiver circuits
USER1	226	AVTT	Analog supply voltage for transceiver termination circuits
USER2	65	VCCO	Expect 1.8V
USER3	87	VCCO	Expect 3.3V

The values read were:

$ xilinx_sensors/display_sensor_values 
Opening device 0000:01:00.0 (10ee:9038) with IOMMU group 0
Enabled bus master for 0000:01:00.0
Displaying SYSMON values for design XCKU5P_DUAL_QSFP_dma_stream_loopback in PCI device 0000:01:00.0 IOMMU group 0:
SYSMON samples using Continuous sequence mode
Number of samples averaged none
Current enabled channels in sequencer: Temp    Vccint  Vccaux  Vbram   Cal     Vuser0  Vuser1  Vuser2  Vuser3 
Analog Bus configuration 0x0E98
  Channel  Measurement     Min           Max
  Temp       57.4881C     55.0012C      60.4723C
  Vccint      0.8877V      0.8818V       0.8936V
  Vccaux      1.8252V      1.8223V       1.8311V
  Vbram       0.8877V      0.8818V       0.8936V
  Vuser0      0.8965V      0.8936V       0.9023V
  Vuser1      1.2041V      1.1982V       1.2100V
  Vuser2      1.8457V      1.8398V       1.8574V
  Vuser3      3.3164V      3.3105V       3.3398V

The values look valid. For Vuser3 the scaling has been set to a 6V range as expected. Whereas the value displayed by the Vivado Hardware Manager when read the SYSMON values over JTAG was half the expected value. I.e. appears the Vivado Hardware Manager doesn't detect when the supply being monitored is a high IO range.

The XCKU5P_DUAL_QSFP_dma_stream_loopback/XCKU5P_DUAL_QSFP_dma_stream_loopback.gen/sources_1/bd/XCKU5P_DUAL_QSFP_dma_stream_loopback/ip/XCKU5P_DUAL_QSFP_dma_stream_loopback_system_management_wiz_0_0/XCKU5P_DUAL_QSFP_dma_stream_loopback_system_management_wiz_0_0_xadc_core_drp.vhd generated file in the XCKU5P_DUAL_QSFP_dma_stream_loopback project which instantiates SYSMONE4 has the following, which matches the value reported for Analog Bus configuration by the program:

        INIT_45 => X"0E98", -- Analog Bus Register

From running display_sensor_values it appears that when the device is opened or closed, which performs a device resett, that the min/max sensor values are reset.

5. PCIe errors when fittd to a HP Pavilion 590-p0053na

The card was fitted to a HP Pavilion 590-p0053na, which uses a AMD IOMMU as part of checking use of VFIO with a AMD IOMMU. The FPGA card is fitted in a x16 PCIe 3.0 slot, so should use the full FPGA capability of 8.0 GT/s at x8 width.

5.1. Issues seen with openSUSE and AlmaLinux 8.10

Some issues seen are:

On one reboot the FPGA enumerated as only x2 rather than x8 width which cause the throughput to drop for test_dma_bridge_parallel_streams

A reboot into openSUSE leap 15.5 made the FPGA enumerate at x8 width. However, when monitoring dmesg noticed that when DMA tests were run numerous errors of the following form were reported:

[  +0.018245] pcieport 0000:00:01.1: AER: Corrected error message received from 0000:00:01.0
[  +0.000012] pcieport 0000:00:01.1: PCIe Bus Error: severity=Corrected, type=Data Link Layer, (Receiver ID)
[  +0.000003] pcieport 0000:00:01.1:   device [1022:15d3] error status/mask=00000040/00006000
[  +0.000003] pcieport 0000:00:01.1:    [ 6] BadTLP

Where the above errors are reported from the PCIe root port to which the FPGA is connected.

For the PCIe correctable errors:

Rebooted into AlmaLinux 8.10and didn't get the AER errors in dmesg
Rebooted back into openSUSE, and got the AER correctable errors when DMA was running.

Saved the lspci -vvv output for both OS's. Differences are:

For the PCIe root port openSUSE is enabling reporting of correctable errors in DevCtrl and AlmaLinux 8.10 is disabling reporting.
For the FPGA endpoint openSUSE is disabling reporting of correctable errors in DevCtrl and AlmaLinix 8.10 is enabling reporting.

5.2. Issues seen with AlmaLinux 9.4

Having installed AlmaLinux 9.4 on the PC noticed:

When the VFIO device is opened or closed sometimes the negotiated link width changes, have seen x8, x4 and x2 used. The change in negotiated width lead to a corresponding change in the bandwidth reported by test_dma_bridge_parallel_streams.
Eventually the link negotiated as x8 2/5 GT/s and didn't change to a higher rate over device opens / closes. Secure boot is enabled so unable to run PCIe Set Speed nor pcie_set_speed.c to trigger link retraining after the device has been opened.

Considered creating a VFIO based program to create the functionality as the pcie_set_speed.sh. However, the Retrain Link bit in the Link Control Register is not applicable for endpoints. Given that the vfio-pci module can't be bound to a PCIe root port, then it is not currently possible to use VFIO to perform a link retrain.

[Qemu-devel] QEMU PCIe link "negotiation" from 2018 has:

This email is already too long, but I also wonder whether we should consider additional vfio-pci interfaces to trigger a link retraining or allow virtualized access to the physical upstream port config space.

Not sure if has been any work on allowing vfio-pci to trigger link retraining on real hardware. Not sure how much of that message was about virtualised PCIe links for QEMU.

In the XCKU5P_DUAL_QSFP_dma_ram design for the DMA/Bridge Subsystem for PCI Express using the Advanced mode:

Set enable_auto_rxeq true, but have still seen link speed and width changes when vfio has opened the device.
Enable debug options, but haven't yet used them to investigate.

The HP Pavilion 590-p0053na PC sometimes hangs booting AlmaLinux, and have seen the NVMe with the Windows installation sometimes disappear. I.e. maybe some issues with the PC which contribute to the PCIe link speed and width changing issues.

6. No obvious PCIe issues when fitted in a HP Z4 G4

The card was moved to a HP Z4 G4. The PCIe interface in the XCKU5P_DUAL_QSFP_dma_stream_loopback design seems stable and remains as 8 GT/s Width x8

Example test program run:

[mr_halfword@skylake-alma release]$ xilinx_dma_bridge_for_pcie/test_dma_bridge_parallel_streams 
Using num_descriptors=64 bytes_per_buffer=0x1000000 data_mapping_size_words=0x10000000
Selecting test of TEF1001_dma_stream_loopback design PCI device 0000:15:00.0 IOMMU group 41 H2C channel 0 C2H channel 1
Selecting test of TEF1001_dma_stream_loopback design PCI device 0000:15:00.0 IOMMU group 41 H2C channel 1 C2H channel 0
Selecting test of XCKU5P_DUAL_QSFP_dma_stream_loopback design PCI device 0000:2d:00.0 IOMMU group 85 H2C channel 0 C2H channel 1
Selecting test of XCKU5P_DUAL_QSFP_dma_stream_loopback design PCI device 0000:2d:00.0 IOMMU group 85 H2C channel 1 C2H channel 0
Selecting test of XCKU5P_DUAL_QSFP_dma_stream_loopback design PCI device 0000:2d:00.0 IOMMU group 85 H2C channel 2 C2H channel 3
Selecting test of XCKU5P_DUAL_QSFP_dma_stream_loopback design PCI device 0000:2d:00.0 IOMMU group 85 H2C channel 3 C2H channel 2
Selecting test of NiteFury_dma_stream_loopback design PCI device 0000:36:00.0 IOMMU group 86 H2C channel 0 C2H channel 1
Selecting test of NiteFury_dma_stream_loopback design PCI device 0000:36:00.0 IOMMU group 86 H2C channel 1 C2H channel 0
Press Ctrl-C to stop test
  0000:15:00.0 0 -> 1 818.598 Mbytes/sec (8170504192 bytes in 9.981100 secs)
  0000:15:00.0 1 -> 0 818.597 Mbytes/sec (8170504192 bytes in 9.981105 secs)
  0000:2d:00.0 0 -> 1 1485.865 Mbytes/sec (14847836160 bytes in 9.992722 secs)
  0000:2d:00.0 1 -> 0 1485.864 Mbytes/sec (14847836160 bytes in 9.992728 secs)
  0000:2d:00.0 2 -> 3 1485.864 Mbytes/sec (14847836160 bytes in 9.992730 secs)
  0000:2d:00.0 3 -> 2 1485.864 Mbytes/sec (14847836160 bytes in 9.992730 secs)
  0000:36:00.0 0 -> 1 818.587 Mbytes/sec (8170504192 bytes in 9.981231 secs)
  0000:36:00.0 1 -> 0 818.586 Mbytes/sec (8170504192 bytes in 9.981236 secs)

<<snip>>
  0000:15:00.0 0 -> 1 818.597 Mbytes/sec (8187281408 bytes in 10.001600 secs)
  0000:15:00.0 1 -> 0 818.597 Mbytes/sec (8187281408 bytes in 10.001599 secs)
  0000:2d:00.0 0 -> 1 1485.917 Mbytes/sec (14864613376 bytes in 10.003662 secs)
  0000:2d:00.0 1 -> 0 1485.917 Mbytes/sec (14864613376 bytes in 10.003662 secs)
  0000:2d:00.0 2 -> 3 1485.917 Mbytes/sec (14864613376 bytes in 10.003663 secs)
  0000:2d:00.0 3 -> 2 1485.917 Mbytes/sec (14864613376 bytes in 10.003663 secs)
  0000:36:00.0 0 -> 1 818.597 Mbytes/sec (8187281408 bytes in 10.001599 secs)
  0000:36:00.0 1 -> 0 818.597 Mbytes/sec (8187281408 bytes in 10.001599 secs)

^C  0000:15:00.0 0 -> 1 818.597 Mbytes/sec (8187281408 bytes in 10.001600 secs)
  0000:15:00.0 1 -> 0 818.597 Mbytes/sec (8187281408 bytes in 10.001601 secs)
  0000:2d:00.0 0 -> 1 1485.879 Mbytes/sec (14277410816 bytes in 9.608732 secs)
  0000:2d:00.0 1 -> 0 1485.879 Mbytes/sec (14277410816 bytes in 9.608730 secs)
  0000:2d:00.0 2 -> 3 1485.879 Mbytes/sec (14277410816 bytes in 9.608729 secs)
  0000:2d:00.0 3 -> 2 1485.879 Mbytes/sec (14277410816 bytes in 9.608729 secs)
  0000:36:00.0 0 -> 1 818.592 Mbytes/sec (8187281408 bytes in 10.001664 secs)
  0000:36:00.0 1 -> 0 818.592 Mbytes/sec (8187281408 bytes in 10.001664 secs)

  0000:15:00.0 0 -> 1 127.906 Mbytes/sec (167772160 bytes in 1.311687 secs)
  0000:15:00.0 1 -> 0 127.906 Mbytes/sec (167772160 bytes in 1.311683 secs)
  0000:2d:00.0 0 -> 1 No completed transfers
  0000:2d:00.0 1 -> 0 No completed transfers
  0000:2d:00.0 2 -> 3 No completed transfers
  0000:2d:00.0 3 -> 2 No completed transfers
  0000:36:00.0 0 -> 1 127.904 Mbytes/sec (167772160 bytes in 1.311703 secs)
  0000:36:00.0 1 -> 0 127.904 Mbytes/sec (167772160 bytes in 1.311699 secs)

Overall test statistics:
  0000:15:00.0 0 -> 1 818.588 Mbytes/sec (4870761349120 bytes in 5950.196814 secs)
  0000:15:00.0 1 -> 0 818.588 Mbytes/sec (4870761349120 bytes in 5950.196815 secs)
  0000:2d:00.0 0 -> 1 1485.896 Mbytes/sec (8840502312960 bytes in 5949.608534 secs)
  0000:2d:00.0 1 -> 0 1485.896 Mbytes/sec (8840502312960 bytes in 5949.608539 secs)
  0000:2d:00.0 2 -> 3 1485.896 Mbytes/sec (8840502312960 bytes in 5949.608539 secs)
  0000:2d:00.0 3 -> 2 1485.896 Mbytes/sec (8840502312960 bytes in 5949.608539 secs)
  0000:36:00.0 0 -> 1 818.590 Mbytes/sec (4870778126336 bytes in 5950.203528 secs)
  0000:36:00.0 1 -> 0 818.590 Mbytes/sec (4870778126336 bytes in 5950.203529 secs)

0000:15:00.0 0 -> 1 Test pattern verified in 268435456 words
0000:15:00.0 1 -> 0 Test pattern verified in 268435456 words
0000:2d:00.0 0 -> 1 Test pattern verified in 268435456 words
0000:2d:00.0 1 -> 0 Test pattern verified in 268435456 words
0000:2d:00.0 2 -> 3 Test pattern verified in 268435456 words
0000:2d:00.0 3 -> 2 Test pattern verified in 268435456 words
0000:36:00.0 0 -> 1 Test pattern verified in 268435456 words
0000:36:00.0 1 -> 0 Test pattern verified in 268435456 words

Overall PASS

7. PCIe bandwidth

Looking at the PCIe bandwidth of the XCKU5P_DUAL_QSFP_dma_stream_loopback design running the test_dma_bridge_parallel_streams program.

The card is fitted in a HP Z4 G4 Intel Xeon W-2123 processor. The PCIe slot root port is vendor_id=8086 (Intel Corporation) device_id=2030 (Sky Lake-E PCI Express Root Port A) with a max width of x8 and a max speed of 8.0 GT/s.

Ran the following to measure the achived bandwidth:

xilinx_dma_bridge_for_pcie/test_dma_bridge_parallel_streams --stream_device 0000:2d:00.0,0

The IP configuration of the PCIe bridge was adjusted for different speeds and widths to see how the achived bandwidth changed, compared to the theortical PCI Express link performance:

FPGA PCIe configuration	Achieved full-duplex bandwidth	Theoretical bandwidth	Percentage achieved of theoretical bandwidth
speed 8 GT/s Width x8	5948.260 Mbytes/sec	7.877 GB/s	75.5%
speed 8 GT/s Width x4	3214.380 Mbytes/sec	3.938 GB/s	81.6%
speed 8 GT/s Width x2	1593.180 Mbytes/sec	1.969 GB/s	80.9%
speed 8 GT/s Width x1	791.394 Mbytes/sec	0.985 GB/s	80.3%
speed 5 GT/s Width x8	3279.241 Mbytes/sec	4.000 GB/s	82.0%
speed 5 GT/s Width x4	1617.207 Mbytes/sec	2.000 GB/s	80.9%
speed 5 GT/s Width x2	808.239 Mbytes/sec	1.000 GB/s	80.8%
speed 5 GT/s Width x1	396.240 Mbytes/sec	0.500 GB/s	79.2%
speed 2.5 GT/s Width x8	1613.183 Mbytes/sec	2.000 GB/s	80.7%
speed 2.5 GT/s Width x4	795.074 Mbytes/sec	1.000 GB/s	79.5%
speed 2.5 GT/s Width x2	396.209 Mbytes/sec	0.500 GB/s	79.2%
speed 2.5 GT/s Width x1	194.740 Mbytes/sec	0.250 GB/s	77.9%

8. JTAG-to-AXI core

Used the XCKU5P_DUAL_QSFP_axi_jtag design, programmed using Vivado Hardware Manager. A hw_axi_1 core is present in the hardware. Followed Using Tcl Commands to Interact with a JTAG-to-AXI Master Core to read one 32-bit word from address zero, which is the value from the usr_accesse2_read module. Output from the Vivado Tcl Console after running the command to read a value:

refresh_hw_device [lindex [get_hw_devices] 0]
INFO: [Labtools 27-2302] Device xcku5p (JTAG device index = 0) is programmed with a design that has 1 JTAG AXI core(s).
reset_hw_axi [get_hw_axis hw_axi_1]
create_hw_axi_txn read_txn [get_hw_axis hw_axi_1] -type read -address 0 -len 1
read_txn
run_hw_axi [get_hw_axi_txns read_txn]
INFO: [Labtoolstcl 44-481] READ DATA is: 9e314cb1

Where the value 9e314cb1 matches the expected value from the bitstream:

 ~/fpga_sio/software_tests/eclipse_project/bin/debug/xilinx_quad_spi/parse_bitstream_file XCKU5P_DUAL_QSFP_axi_jtag_wrapper.bit |grep AXSS
  Type 1 packet opcode write register AXSS 9E314CB1 - 19/12/2024 20:50:49

9. Bank voltages

Temporarily modified the System Management Wizard User Supply Selection for VUSER settings to attempt to identify the different bank supplies in use, due to no schematic being provided. Ran display_sensor_values to obtain the values.

USER SUPPLY0:

Bank Supply Measurement

226 AVCC 0.8877V

64 VCCO 1.8428V

87 VCCAUX 1.8281V

86 VCCINT 0.8818V

224 AVCC 0.8877V
USER SUPPLY1:

Bank Supply Measurement

226 AVTT 1.1865V

65 VCCINT 0.8818V

65 VCCAUX 1.8164V

67 VCCAUX 1.8164V

225 AVTT 1.1836V
USER SUPPLY2:

Bank Supply Measurement

65 VCCO 1.8428V

66 VCCAUX 1.8164V

66 VCCO 1.8428V

84 VCCAUX 1.8164V

227 MGTTVCAUX 1.8105V
USER SUPPLY3:

Bank Supply Measurement

87 VCCO 3.2988V

67 VCCINT 0.8818V

67 VCCO 1.8281V

86 VCCO 3.3164V

84 VCCO 3.3164V

Bank	Supply	Measurement
226	AVCC	0.8877V
64	VCCO	1.8428V
87	VCCAUX	1.8281V
86	VCCINT	0.8818V
224	AVCC	0.8877V

Bank	Supply	Measurement
226	AVTT	1.1865V
65	VCCINT	0.8818V
65	VCCAUX	1.8164V
67	VCCAUX	1.8164V
225	AVTT	1.1836V

Bank	Supply	Measurement
65	VCCO	1.8428V
66	VCCAUX	1.8164V
66	VCCO	1.8428V
84	VCCAUX	1.8164V
227	MGTTVCAUX	1.8105V

Bank	Supply	Measurement
87	VCCO	3.2988V
67	VCCINT	0.8818V
67	VCCO	1.8281V
86	VCCO	3.3164V
84	VCCO	3.3164V

The above combinations were to identify the VCCO of the non-GTY banks:

1.8V : 64, 65, 66, 67
3.3V : 84, 86, 87

10. Debugging no response to QSFP B I2C

With the initial XCKU5P_DUAL_QSFP_ibert_4.166 design there was no response attempting to read the I2C interface for QSFP B, for this example when a DAC cable fitted:

 i2c_probe/i2c_probe -a 0x50:0x50
Opening device 0000:15:00.0 (10ee:7024) with IOMMU group 41
Enabled bus master for 0000:15:00.0
Opening device 0000:2d:00.0 (10ee:9038) with IOMMU group 25
Opening device 0000:36:00.0 (10ee:7011) with IOMMU group 86
Enabled bus master for 0000:36:00.0
Enabling MOD_SEL for QSFP port A
Probing QSFP port A 
Using design XCKU5P_DUAL_QSFP_ibert in device 0000:2d:00.0
Iteration 1 of 1 using IIC iic_lib
Slave 0x50 replied with data 0x00
Enabling MOD_SEL for QSFP port B
Probing QSFP port B 
Using design XCKU5P_DUAL_QSFP_ibert in device 0000:2d:00.0
Iteration 1 of 1 using IIC iic_lib

The SCL and SDA seem to read back, just no ACK on the 9th clock: For the above the QSFP B I2C constraints were the following, which enabled internal pull-ups and used the pin locations in the XCKU5P_pcie_qsfp_or_sfp_IO.xls supplied with the board:

set_property PACKAGE_PIN B10 [get_ports {QSFP_I2C_B_scl_io}]
set_property IOSTANDARD LVCMOS33 [get_ports {QSFP_I2C_B_scl_io}] 
set_property PULLUP true [get_ports {QSFP_I2C_B_scl_io}]

set_property PACKAGE_PIN B12 [get_ports {QSFP_I2C_B_sda_io}]
set_property IOSTANDARD LVCMOS33 [get_ports {QSFP_I2C_B_sda_io}] 
set_property PULLUP true [get_ports {QSFP_I2C_B_sda_io}]

The XCKU5P_DUAL_QSFP_input_monitor result suggest pin B10 is not connected externally to the FPGA.

Commented out the internal PULLUP on QSFP_I2C_B_scl_io and QSFP_I2C_B_sda_io which results in:

i2c_probe hanging on QSFP B, and have to Ctrl-C to abort.
ILA shows SLOT_1_IIC_scl_i can read back as 1 even when the IIC module has tri-stated the SCL pin.

Changed the QSFP_I2C_B_scl_io pin from B10 to B11 and now can probe the DAC cable fitted in QSFP B:

$ i2c_probe/i2c_probe -a 0x50:0x50 -m standard 
Opening device 0000:15:00.0 (10ee:7024) with IOMMU group 41
Enabled bus master for 0000:15:00.0
Opening device 0000:2d:00.0 (10ee:9038) with IOMMU group 25
Opening device 0000:36:00.0 (10ee:7011) with IOMMU group 86
Enabled bus master for 0000:36:00.0
Enabling MOD_SEL for QSFP port A
Probing QSFP port A 
Using design XCKU5P_DUAL_QSFP_ibert in device 0000:2d:00.0
Iteration 1 of 1 using IIC standard
Slave 0x50 replied with data 0x00
Enabling MOD_SEL for QSFP port B
Probing QSFP port B 
Using design XCKU5P_DUAL_QSFP_ibert in device 0000:2d:00.0
Iteration 1 of 1 using IIC standard
Slave 0x50 replied with data 0x0d

11. Reading of sysmon values broken

After attempting to change XCKU5P_DUAL_QSFP_dma_stream_loopback to be built using Vivado 2025.1 insead of 2024.2, found display_sensor_values was reporting incorrect values:

isplaying SYSMON values for design XCKU5P_DUAL_QSFP_dma_stream_loopback in PCI device 0000:2d:00.0 IOMMU group 25:
SYSMON samples using Continuous sequence mode
Number of samples averaged none
Current enabled channels in sequencer: Temp    Vccint  Vccaux  Vbram   Cal     Vuser0  Vuser1  Vuser2  Vuser3 
Analog Bus configuration 0x0E98
  Channel  Measurement     Min           Max
  Temp       -280.2309C           
  Vccint      0.0000V           
  Vccaux      0.0000V           
  Vbram       0.0000V           
  Vuser0      0.0000V           
  Vuser1      0.0000V           
  Vuser2      0.0000V           
  Vuser3      0.0000V

On investigating:

The issue isn't in re-built bitstream, can repeat the same issue if load the previous bitstream built with Vivado 2024.2
Can repeat the issue with other designs which use the sysmon.
The configuration registers still read back the same values as when the measured values were correct.
In the debugger:
- All raw measurement values read as zero
- All min values read as 1023
- All max values read as zero
If power cycle the PC, the Vivado 2024.2 Hardware Manager displays the same incorrect measurement values over JTAG as the software reading sysmon over PCIe.
With the Vivado 2024.2 Hardware Manager after power cycling the PC:
- The device is reported as not programmed
- Attempting Boot from Configuration Memory Device reports a failure after waiting tens of seconds

Point 6) above suggests the as-delivered configuration memory might have been corrupted.

Used Vivado 2024.2 Hardware Manager to save the current configuration memory contents to a file. parse_bitstream_file run on the file saved over JTAG, and quad_spi_flasher with the XCKU5P_DUAL_QSFP_dma_stream_loopback design loaded over JTAG both produced the same output. Albeit these programs only summarise the configuration frames and don't check that the bitstream has a valid CRC.

To see if re-programming the configuration memory would resolve the issue:

Started with the XCKU5P_DUAL_QSFP_dma_stream_loopback_wrapper.bit file built with Vivado 2024.2
The Vivado Hardware Manager is unable to read .bit files when programming configuration memory.

Used hexdump -C on the .bit file to find the start offset of the bitstream. Then created a .bin file by remove the .bit file header:

$ dd ibs=1 skip=161 if=XCKU5P_DUAL_QSFP_dma_stream_loopback_wrapper.bit of=XCKU5P_DUAL_QSFP_dma_stream_loopback_wrapper.bin

Used Hardware Manager to program the XCKU5P_DUAL_QSFP_dma_stream_loopback_wrapper.bin file into the configuration memory.
Power cycled the PC and the bitstream loaded, but the sysmon measurements were still incorrect.
Power cycled the PC and the bitstream didn't load. Hardware Manager reported a failure trying to boot from the memory configuration device.
Left the PC powered off for at least 1 hour.
Powered on the PC and the bitstream did't load.
Used the Vivado Hardware Manager to readback the configuration bitstream, which was successful.

After loading XCKU5P_DUAL_QSFP_dma_stream_loopback_wrapper.bit over JTAG:

display_identified_pcie_fpga_designs displays the expected information.
display_sensor_values displays the correct configuration, but the incorrect measurements.
test_dma_bridge_parallel_streams runs at 1483.706 Mbytes/sec for each stream, and the final test pattern verification passes.
quad_spi_flasher is able to read the configuration flash.

After loading XCKU5P_DUAL_QSFP_qdma_ram_wrapper.bit over JTAG:

quad_spi_flasher is able to read the configuration flash.
display_identified_pcie_fpga_designs displays the expected information.
identify_uarts identifies the expected 16550A UART.
display_sensor_values displays the correct configuration, but the incorrect measurements.

After loading XCKU5P_DUAL_QSFP_ibert_4_166_wrapper.bit over JTAG:

display_identified_pcie_fpga_designs displays the expected information.
quad_spi_flasher is able to read the configuration flash.
display_sensor_values displays the correct configuration, but the incorrect measurements.
qsfp_management_menu can control the LEDs and read a QSFP+ DAC cable fitted in both ports.
Serial I/O Links in the Vivado 2025.1 Hardware Manager detects all 8 links, when the QSFP+ DAC cable in use.

As for trying to boot from the configuration device:

Start with a design loaded over JTAG.
In the Vivado Hardware Manager the FGPA status is Programmed.

Select Boot from Configuration Memory Device in the Viado Hardware Manager. The Tcl Console shows

boot_hw_device  [lindex [get_hw_devices xcku5p_0] 0]
INFO: [Labtoolstcl 44-664] Will wait up to 180 seconds for booting to complete.
ERROR: [Labtools 27-2254] Booting from configuration memory device unsuccessful.
boot_hw_device: Time (s): cpu = 00:00:06 ; elapsed = 00:03:02 . Memory (MB): peak = 10082.836 ; gain = 0.000 ; free physical = 7630 ; free virtual = 21571
ERROR: [Common 17-39] 'boot_hw_device' failed due to earlier errors.

Refreshing the device changes the status to Not programmed.
In the hardware manager add a configuration device mt25qu256-spi-x1_x2_x4 and then readback the configuration device. The readback contents is binary match of that programmed above (allowing for the readback being the entire configuration memory contents).

Tried:

Editing quadspi to reduce BITSTREAM.CONFIG.CONFIGRATE from 56.7 (the value in an example) to 2.7 (minimum value specified in Artix, Virtex, and Kintex UltraScale+ Bitstream Settings
Used the Hardware Manager to program the configuration device, which was successful.
Used the Hardware Manager to Boot from Configuration Memory Device, which was successful with an elapsed time of 9 seconds.
display_identified_pcie_fpga_designs displayed the expected build timestamp. display_sensor_values still showed the incorrect measurements.
Power cycled the PC.
The design enumerated on the PCIe bus, but display_sensor_values still showed the incorrect measurements.
Power cycled the PC and got the same behaviour.
Power cycled the PC and got the same behaviour.
Used Hardware Manager to erase the configuration memory, then power cycled the PC.
The Hardware Manager still showed invalid measurements.
Loaded XCKU5P_DUAL_QSFP_dma_stream_loopback_wrapper.bit over JTAG. display_sensor_values displays the correct configuration, but the incorrect measurements.

Therefore, appears to be two issues:

sysmon doesn't perform any measurements, regardless of if there is a bitstream loaded or not.
Loading of bitstream from the configuration memory seems unreliable at a 56.7 config rate.

The bitstream was re-built with some different BITSTREAM.CONFIG.CONFIGRATE values:

Tried 10.6, and failed to boot from the configuration device. This was an attempt to determine if a higher rate worked.
Tried the minimum 2.7 again, and failed to boot from the configuration device. With this programmed, quad_spi_flasher gives the COR0 register value as 38003FE5. The OSCFSEL field is b000000 (i.e. mininum).
For a bitstream with the BITSTREAM.CONFIG.CONFIGRATE set to 56.7, COR0 is 383C3FE5. The OSCFSEL field is b011110. Can't seem to find the documentation for the encoding of the OSCFSEL field.

The SYSMON has a dedicated V_CCADC supply pin, so perhaps that supply is no longer working / connected.

The bistream BSPI register value is 0000066C which is:

SPI_32BIT_ADDR : 1: 32-bit address
SPI bus width : 10: x4
SPI_READ_OPCODE : 6C "Read quad out (4 byte address)"

The quad_spi_flasher software uses opcode EC "Quad I/O read (4 byte address), High freq,4 lines, addr on 2 lines"

Commented out the following in the contstraints file, to try and use defaults:

#set_property BITSTREAM.CONFIG.CONFIGRATE 56.7 [current_design]
#set_property BITSTREAM.CONFIG.SPI_32BIT_ADDR YES [current_design]
#set_property BITSTREAM.CONFIG.SPI_BUSWIDTH 4 [current_design]
#set_property BITSTREAM.CONFIG.SPI_FALL_EDGE YES [current_design]

The start of the bitstream changed from:

  Type 1 packet opcode NOP
  Type 1 packet opcode write register BSPI words 0000066C
  Type 1 packet opcode write register CMD command BSPI_READ
  Type 1 packet opcode NOP (2 consecutive)
  Type 1 packet opcode write register TIMER words 00000000
  Type 1 packet opcode write register WBSTAR words 00000000
  Type 1 packet opcode write register CMD command NULL
  Type 1 packet opcode NOP
  Type 1 packet opcode write register CMD command RCRC
  Type 1 packet opcode NOP (2 consecutive)
  Type 1 packet opcode write register AXSS 3B337498 - 07/06/2025 23:18:24

To:

  Type 1 packet opcode NOP (2 consecutive)
  Type 1 packet opcode write register TIMER words 00000000
  Type 1 packet opcode write register WBSTAR words 00000000
  Type 1 packet opcode write register CMD command NULL
  Type 1 packet opcode NOP
  Type 1 packet opcode write register CMD command RCRC
  Type 1 packet opcode NOP (2 consecutive)
  Type 1 packet opcode write register AXSS 7B32B177 - 15/06/2025 11:05:55

Which removed the BSPI register write and BSPI_READ command to modfify the SPI settings. However, the bitstream still failed to load from the configuration memory.

Powered up when didn't load the bitstream from the configuration memory. Attached JTAG and the following register value read from the Vivado Hardware Manager:

REGISTER.BOOT_STATUS	0x00000000
REGISTER.BOOT_STATUS.BIT[00]_0_STATUS_VALID	0
REGISTER.BOOT_STATUS.BIT[01]_0_FALLBACK	0
REGISTER.BOOT_STATUS.BIT[02]_0_INTERNAL_PROG	0
REGISTER.BOOT_STATUS.BIT[03]_0_WATCHDOG_TIMEOUT_ERROR	0
REGISTER.BOOT_STATUS.BIT[04]_0_ID_ERROR	0
REGISTER.BOOT_STATUS.BIT[05]_0_CRC_ERROR	0
REGISTER.BOOT_STATUS.BIT[06]_0_WRAP_ERROR	0
REGISTER.BOOT_STATUS.BIT[07]_0_SECURITY_ERROR	0
REGISTER.BOOT_STATUS.BIT[08]_1_STATUS_VALID	0
REGISTER.BOOT_STATUS.BIT[09]_1_FALLBACK	0
REGISTER.BOOT_STATUS.BIT[10]_1_INTERNAL_PROG	0
REGISTER.BOOT_STATUS.BIT[11]_1_WATCHDOG_TIMEOUT_ERROR	0
REGISTER.BOOT_STATUS.BIT[12]_1_ID_ERROR	0
REGISTER.BOOT_STATUS.BIT[13]_1_CRC_ERROR	0
REGISTER.BOOT_STATUS.BIT[14]_1_WRAP_ERROR	0
REGISTER.BOOT_STATUS.BIT[15]_1_SECURITY_ERROR	0

REGISTER.COR0	0x38003fec
REGISTER.COR0.BIT[02:00]_GWE_CYCLE	100
REGISTER.COR0.BIT[05:03]_GTS_CYCLE	101
REGISTER.COR0.BIT[08:06]_LOCK_CYCLE	111
REGISTER.COR0.BIT[11:09]_MATCH_CYCLE	111
REGISTER.COR0.BIT[14:12]_DONE_CYCLE	011
REGISTER.COR0.BIT[16:15]_RESERVED	00
REGISTER.COR0.BIT[22:17]_OSCFSEL	000000
REGISTER.COR0.BIT[23]_RESERVED	0
REGISTER.COR0.BIT[24]_DRIVE_DONE	0
REGISTER.COR0.BIT[25]_RESERVED	0
REGISTER.COR0.BIT[26]_ECLK_EN	0
REGISTER.COR0.BIT[31:27]_RESERVED	00111

REGISTER.COR1	0x00000000
REGISTER.COR1.BIT[01:00]_BPI_PAGE_SIZE	00
REGISTER.COR1.BIT[03:02]_BPI_1ST_READ_CYCLE	00
REGISTER.COR1.BIT[07:04]_RESERVED	0000
REGISTER.COR1.BIT[08]_RBCRC_EN	0
REGISTER.COR1.BIT[09]_RBCRC_NO_PIN	0
REGISTER.COR1.BIT[14:10]_RESERVED	00000
REGISTER.COR1.BIT[16:15]_RBCRC_ACTION	00
REGISTER.COR1.BIT[17]_PERSIST_DEASSERT_AT_DESYNC	0
REGISTER.COR1.BIT[18]_RESERVED	0

REGISTER.CONFIG_STATUS	0x1080190c
REGISTER.CONFIG_STATUS.BIT[00]_CRC_ERROR	0
REGISTER.CONFIG_STATUS.BIT[01]_DECRYPTOR_ENABLE	0
REGISTER.CONFIG_STATUS.BIT[02]_PLL_LOCK_STATUS	1
REGISTER.CONFIG_STATUS.BIT[03]_DCI_MATCH_STATUS	1
REGISTER.CONFIG_STATUS.BIT[04]_END_OF_STARTUP_(EOS)_STATUS	0
REGISTER.CONFIG_STATUS.BIT[05]_GTS_CFG_B_STATUS	0
REGISTER.CONFIG_STATUS.BIT[06]_GWE_STATUS	0
REGISTER.CONFIG_STATUS.BIT[07]_GHIGH_STATUS	0
REGISTER.CONFIG_STATUS.BIT[08]_MODE_PIN_M[0]	1
REGISTER.CONFIG_STATUS.BIT[09]_MODE_PIN_M[1]	0
REGISTER.CONFIG_STATUS.BIT[10]_MODE_PIN_M[2]	0
REGISTER.CONFIG_STATUS.BIT[11]_INIT_B_INTERNAL_SIGNAL_STATUS	1
REGISTER.CONFIG_STATUS.BIT[12]_INIT_B_PIN	1
REGISTER.CONFIG_STATUS.BIT[13]_DONE_INTERNAL_SIGNAL_STATUS	0
REGISTER.CONFIG_STATUS.BIT[14]_DONE_PIN	0
REGISTER.CONFIG_STATUS.BIT[15]_IDCODE_ERROR	0
REGISTER.CONFIG_STATUS.BIT[16]_SECURITY_ERROR	0
REGISTER.CONFIG_STATUS.BIT[17]_SYSTEM_MONITOR_OVER-TEMP_ALARM_STATUS	0
REGISTER.CONFIG_STATUS.BIT[20:18]_CFG_STARTUP_STATE_MACHINE_PHASE	000
REGISTER.CONFIG_STATUS.BIT[23:21]_SECURITY_STATUS	100
REGISTER.CONFIG_STATUS.BIT[24]_RESERVED	0
REGISTER.CONFIG_STATUS.BIT[25]_CFG_BUS_WIDTH_DETECTION	0
REGISTER.CONFIG_STATUS.BIT[26]_CFG_BUS_WIDTH_DETECTION	0
REGISTER.CONFIG_STATUS.BIT[27]_SECURITY_AUTH_ERROR	0
REGISTER.CONFIG_STATUS.BIT[28]_PUDC_B_PIN	1
REGISTER.CONFIG_STATUS.BIT[29]_BAD_PACKET_ERROR	0
REGISTER.CONFIG_STATUS.BIT[30]_CFGBVS_PIN	0
REGISTER.CONFIG_STATUS.BIT[31]_RESERVED	0

REGISTER.WBSTAR	0x00000000
REGISTER.WBSTAR.BIT[28:00]_START_ADDR	00000000000000000000000000000
REGISTER.WBSTAR.BIT[29]_RS_TS_B	0
REGISTER.WBSTAR.BIT[31:30]_RS	00

Readback registers after had loaded a bitstream using JTAG:

REGISTER.BOOT_STATUS	0x00000001
REGISTER.BOOT_STATUS.BIT[00]_0_STATUS_VALID	1
REGISTER.BOOT_STATUS.BIT[01]_0_FALLBACK	0
REGISTER.BOOT_STATUS.BIT[02]_0_INTERNAL_PROG	0
REGISTER.BOOT_STATUS.BIT[03]_0_WATCHDOG_TIMEOUT_ERROR	0
REGISTER.BOOT_STATUS.BIT[04]_0_ID_ERROR	0
REGISTER.BOOT_STATUS.BIT[05]_0_CRC_ERROR	0
REGISTER.BOOT_STATUS.BIT[06]_0_WRAP_ERROR	0
REGISTER.BOOT_STATUS.BIT[07]_0_SECURITY_ERROR	0
REGISTER.BOOT_STATUS.BIT[08]_1_STATUS_VALID	0
REGISTER.BOOT_STATUS.BIT[09]_1_FALLBACK	0
REGISTER.BOOT_STATUS.BIT[10]_1_INTERNAL_PROG	0
REGISTER.BOOT_STATUS.BIT[11]_1_WATCHDOG_TIMEOUT_ERROR	0
REGISTER.BOOT_STATUS.BIT[12]_1_ID_ERROR	0
REGISTER.BOOT_STATUS.BIT[13]_1_CRC_ERROR	0
REGISTER.BOOT_STATUS.BIT[14]_1_WRAP_ERROR	0
REGISTER.BOOT_STATUS.BIT[15]_1_SECURITY_ERROR	0

REGISTER.CONFIG_STATUS	0x109079fc
REGISTER.CONFIG_STATUS.BIT[00]_CRC_ERROR	0
REGISTER.CONFIG_STATUS.BIT[01]_DECRYPTOR_ENABLE	0
REGISTER.CONFIG_STATUS.BIT[02]_PLL_LOCK_STATUS	1
REGISTER.CONFIG_STATUS.BIT[03]_DCI_MATCH_STATUS	1
REGISTER.CONFIG_STATUS.BIT[04]_END_OF_STARTUP_(EOS)_STATUS	1
REGISTER.CONFIG_STATUS.BIT[05]_GTS_CFG_B_STATUS	1
REGISTER.CONFIG_STATUS.BIT[06]_GWE_STATUS	1
REGISTER.CONFIG_STATUS.BIT[07]_GHIGH_STATUS	1
REGISTER.CONFIG_STATUS.BIT[08]_MODE_PIN_M[0]	1
REGISTER.CONFIG_STATUS.BIT[09]_MODE_PIN_M[1]	0
REGISTER.CONFIG_STATUS.BIT[10]_MODE_PIN_M[2]	0
REGISTER.CONFIG_STATUS.BIT[11]_INIT_B_INTERNAL_SIGNAL_STATUS	1
REGISTER.CONFIG_STATUS.BIT[12]_INIT_B_PIN	1
REGISTER.CONFIG_STATUS.BIT[13]_DONE_INTERNAL_SIGNAL_STATUS	1
REGISTER.CONFIG_STATUS.BIT[14]_DONE_PIN	1
REGISTER.CONFIG_STATUS.BIT[15]_IDCODE_ERROR	0
REGISTER.CONFIG_STATUS.BIT[16]_SECURITY_ERROR	0
REGISTER.CONFIG_STATUS.BIT[17]_SYSTEM_MONITOR_OVER-TEMP_ALARM_STATUS	0
REGISTER.CONFIG_STATUS.BIT[20:18]_CFG_STARTUP_STATE_MACHINE_PHASE	100
REGISTER.CONFIG_STATUS.BIT[23:21]_SECURITY_STATUS	100
REGISTER.CONFIG_STATUS.BIT[24]_RESERVED	0
REGISTER.CONFIG_STATUS.BIT[25]_CFG_BUS_WIDTH_DETECTION	0
REGISTER.CONFIG_STATUS.BIT[26]_CFG_BUS_WIDTH_DETECTION	0
REGISTER.CONFIG_STATUS.BIT[27]_SECURITY_AUTH_ERROR	0
REGISTER.CONFIG_STATUS.BIT[28]_PUDC_B_PIN	1
REGISTER.CONFIG_STATUS.BIT[29]_BAD_PACKET_ERROR	0
REGISTER.CONFIG_STATUS.BIT[30]_CFGBVS_PIN	0
REGISTER.CONFIG_STATUS.BIT[31]_RESERVED	0

REGISTER.COR0	0x383c3fe5
REGISTER.COR0.BIT[02:00]_GWE_CYCLE	101
REGISTER.COR0.BIT[05:03]_GTS_CYCLE	100
REGISTER.COR0.BIT[08:06]_LOCK_CYCLE	111
REGISTER.COR0.BIT[11:09]_MATCH_CYCLE	111
REGISTER.COR0.BIT[14:12]_DONE_CYCLE	011
REGISTER.COR0.BIT[16:15]_RESERVED	00
REGISTER.COR0.BIT[22:17]_OSCFSEL	011110
REGISTER.COR0.BIT[23]_RESERVED	0
REGISTER.COR0.BIT[24]_DRIVE_DONE	0
REGISTER.COR0.BIT[25]_RESERVED	0
REGISTER.COR0.BIT[26]_ECLK_EN	0
REGISTER.COR0.BIT[31:27]_RESERVED	00111

REGISTER.COR1	0x00400000
REGISTER.COR1.BIT[01:00]_BPI_PAGE_SIZE	00
REGISTER.COR1.BIT[03:02]_BPI_1ST_READ_CYCLE	00
REGISTER.COR1.BIT[07:04]_RESERVED	0000
REGISTER.COR1.BIT[08]_RBCRC_EN	0
REGISTER.COR1.BIT[09]_RBCRC_NO_PIN	0
REGISTER.COR1.BIT[14:10]_RESERVED	00000
REGISTER.COR1.BIT[16:15]_RBCRC_ACTION	00
REGISTER.COR1.BIT[17]_PERSIST_DEASSERT_AT_DESYNC	0
REGISTER.COR1.BIT[18]_RESERVED	0

REGISTER.WBSTAR	0x00000000
REGISTER.WBSTAR.BIT[28:00]_START_ADDR	00000000000000000000000000000
REGISTER.WBSTAR.BIT[29]_RS_TS_B	0
REGISTER.WBSTAR.BIT[31:30]_RS	00

The M[2:0] pins are sampled as 001, which is the expected Master SPI mode.

Tried power cycling with the JTAG adapter disconnected from the board (rather than just the USB cable unplugged), in case the connected and un-powered JTAG adapter was causing the issue. The bitstream still didn't load at power on, and the SYSMON still didn't perform any measurements.

Subsequently the board appears completely dead:

Bitstream not loading at power up.
FPGA no longer found over JTAG.

Tried moving board to a different HP Pavilion 590-p0053na but:

Didn't enumerate on the PCIe bus
Can't detect the FPGA over the PCIe bus

The output from the root port when the board failed to enumerate:

$ sudo lspci -nn -vvv -s 00:01.1
[sudo] password for mr_halfword: 
00:01.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Raven/Raven2 PCIe GPP Bridge [6:0] [1022:15d3] (prog-if 00 [Normal decode])
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin ? routed to IRQ 26
	IOMMU group: 1
	Bus: primary=00, secondary=10, subordinate=10, sec-latency=0
	I/O behind bridge: 0000f000-00000fff [disabled]
	Memory behind bridge: fff00000-000fffff [disabled]
	Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff [disabled]
	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
	BridgeCtl: Parity- SERR+ NoISA- VGA- VGA16+ MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: [50] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
		Status: D3 NoSoftRst- PME-Enable+ DSel=0 DScale=0 PME-
	Capabilities: [58] Express (v2) Root Port (Slot+), MSI 00
		DevCap:	MaxPayload 512 bytes, PhantFunc 0
			ExtTag+ RBE+
		DevCtl:	CorrErr+ NonFatalErr+ FatalErr+ UnsupReq+
			RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+
			MaxPayload 512 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
		LnkCap:	Port #247, Speed 8GT/s, Width x8, ASPM L1, Exit Latency L1 <64us
			ClockPM- Surprise- LLActRep+ BwNot+ ASPMOptComp+
		LnkCtl:	ASPM Disabled; RCB 64 bytes, Disabled- CommClk-
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 2.5GT/s (downgraded), Width x16 (strange)
			TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
		SltCap:	AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surprise-
			Slot #0, PowerLimit 0.000W; Interlock- NoCompl+
		SltCtl:	Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg-
			Control: AttnInd Unknown, PwrInd Unknown, Power- Interlock-
		SltSta:	Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet+ Interlock-
			Changed: MRL- PresDet- LinkState-
		RootCap: CRSVisible+
		RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna+ CRSVisible+
		RootSta: PME ReqID 0000, PMEStatus- PMEPending-
		DevCap2: Completion Timeout: Range ABCD, TimeoutDis+ NROPrPrP- LTR+
			 10BitTagComp- 10BitTagReq- OBFF Not Supported, ExtFmt+ EETLPPrefix+, MaxEETLPPrefixes 1
			 EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
			 FRS- LN System CLS Not Supported, TPHComp- ExtTPHComp- ARIFwd+
			 AtomicOpsCap: Routing- 32bit+ 64bit+ 128bitCAS-
		DevCtl2: Completion Timeout: 65ms to 210ms, TimeoutDis- LTR- OBFF Disabled, ARIFwd-
			 AtomicOpsCtl: ReqEn- EgressBlck-
		LnkCap2: Supported Link Speeds: 2.5-8GT/s, Crosslink- Retimer- 2Retimers- DRS-
		LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete- EqualizationPhase1-
			 EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest-
			 Retimer- 2Retimers- CrosslinkRes: unsupported
	Capabilities: [a0] MSI: Enable+ Count=1/1 Maskable- 64bit+
		Address: 00000000fee00000  Data: 0000
	Capabilities: [c0] Subsystem: Hewlett-Packard Company Device [103c:8433]
	Capabilities: [c8] HyperTransport: MSI Mapping Enable+ Fixed+
	Capabilities: [100 v1] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?>
	Capabilities: [150 v2] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UESvrt:	DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr-
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
		AERCap:	First Error Pointer: 00, ECRCGenCap- ECRCGenEn- ECRCChkCap- ECRCChkEn-
			MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
		HeaderLog: 00000000 00000000 00000000 00000000
		RootCmd: CERptEn+ NFERptEn+ FERptEn+
		RootSta: CERcvd- MultCERcvd- UERcvd- MultUERcvd-
			 FirstFatal- NonFatalMsg- FatalMsg- IntMsg 0
		ErrorSrc: ERR_COR: 0000 ERR_FATAL/NONFATAL: 0000
	Capabilities: [270 v1] Secondary PCI Express
		LnkCtl3: LnkEquIntrruptEn- PerformEqu-
		LaneErrStat: 0
	Capabilities: [2a0 v1] Access Control Services
		ACSCap:	SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans+
		ACSCtl:	SrcValid+ TransBlk- ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans-
	Capabilities: [370 v1] L1 PM Substates
		L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+ L1_PM_Substates+
			  PortCommonModeRestoreTime=0us PortTPowerOnTime=10us
		L1SubCtl1: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1-
			   T_CommonMode=0us LTR1.2_Threshold=0ns
		L1SubCtl2: T_PwrOn=10us
	Kernel driver in use: pcieport

The slot status is reporting PresDet+, which is different to a previous Attempting to debug why a PCIe card doesn't enumerate investigation using the same PC.

Chester-Gillon/Notes_about_XCKU5P_PCIE_3.0_QSFP_X2.md

Select an option

No results found