Notes about 100G Multirate Ethernet MAC (MRMAC)

0. Introduction

Contains notes about using the Versal™ Devices Integrated 100G Multirate Ethernet MAC Subsystem IP.

1. Documentation

Differences between Designing with UltraScale+ CMAC and Versal MRMAC has some useful links.

1.1. Currently can't find user guide for MRMAC 3.2

With Viavdo 2025.2 adding the MRMAC IP to a block design for a Versal AI Edge gets IP v3.2. However, attempting to select Documentation -> Product Guide on the Re-customize IP dialog get the following error:

This document was not found in the current document catalog: https://www.xilinx.com/cgi-bin/docs/ipdoc?c=mrmac;v=v3_2;d=pg314-versal-mrmac.pdf

Please make sure the catalog is up to date. The program will now try to open the document on the Xilinx web site.

Also can't find PG314 v3.2 on the Xilinx web site.

Most recent version of https://docs.amd.com/r/en-US/pg314-versal-mrmac/ available is 3.1.

The Vivado 2025.2 Documentation -> Change Log has the following for 3.2:

2025.2:

Version 3.2

General: Versal New GT Wiz support added for Mixed mode.

General: Enable visibility of hierarchy, cells and schematics in Vivado GUI

General: Device support added

General: Removed quiet switch from create_waiver command in XDC files.

1.2. Can't find documentation for gtpowergood_in port

Both MRMAC 3.1 and 3.2 IP have a gtpowergood_in port, which find described in PG314.

On running Block Automation:

A gtpowergood_in External Input port is created, connected to the mrmac block.
A gtpowergood External Output port is crested, connected to the gt_quad_base Versal Adaptive SoC Transeivers Wizard.

For now, assume OK to connect the gt_quad_base gtpowergood output to the MRMAC gtpowergood_in input.

2. Register Space

MRMAC Memory Map gives the base address of the Configuration registers, Status registers, and Statistics counters for each port.

Based upon that at least 15 address bits are needed to decode the registers.

The MRMAC on the block diagram shows the s_axi interface has s_axi_araddr[31:0] and s_axi_awaddr[31:0]. I.e. apparently 32 address bits.

Looking at a schematic shows only 16 address bits out of the axi_apb_bridge. The hard MRMAC block has a APB3_PADDR[15:0] address input.

The auto assignment of address segments allocated 64K for the MRMAC s_axi.

3. Block automation

3.1. Block Automation only supported with Legacy GT Wizard

The Customize the IP step in the PG314 MRMAC Example Design Simulation and Validation Steps has:

Select the required GT Wizard. For the legacy GT Wizard, enable Use Legacy GT Wizard in Example Design and disable it for the New GT wizard selection. This option will be deprecated in a future release.

If try and run block automation with the New GT wizard no block automation occurs and the following is reported in the TCL console:

INFO:: Block Automation is not supported with Versal Adaptive SoC GT Wizard Subsystem

Occurs with Vivado 2025.1 and 2025.2.

If run the block automation with the Legacy GT Wizzard then the following MRMAC clocks get connected to external interfaces, rather than to the Utility Buffer clock buffer outputs:

rx_alt_serdes_clk[3:0]
rx_core_clk[3:0]
rx_serdes_clk[3:0]
tx_alt_serdes_clk[3:0]
tx_core_clk[3:0]

3.2. Block Automation fails in a hierarchy sub-block

Added a MRMAC block in a hierarchy sub-block. Wanted to use a sub-block to hide the GT blocks and connections on the top-level diagram. Attempting to run Block Automation fails with:

apply_bd_automation -rule xilinx.com:bd_rule:mrmac -config { DataPath_Interface_Connection {Auto} Lane0_selection {NULL} Lane1_selection {NULL} Lane2_selection {NULL} Lane3_selection {NULL} Quad0_selection {NULL} Quad1_selection {NULL} Quad2_selection {NULL} Quad3_selection {NULL}}  [get_bd_cells mrac_10G_dual/mrmac_0]
WARNING: [IP_Flow 19-3374] An attempt to modify the value of disabled parameter 'PROT0_ADD_CONFIG_EN' from 'false' to 'true' has been ignored for IP 'mrac_10G_dual/gt_quad_base'
WARNING: [IP_Flow 19-3374] An attempt to modify the value of disabled parameter 'PROT0_ADD_CONFIG_FILE' from 'no_addn_file_loaded' to 'undef' has been ignored for IP 'mrac_10G_dual/gt_quad_base'
WARNING: [IP_Flow 19-3374] An attempt to modify the value of disabled parameter 'PROT1_ADD_CONFIG_EN' from 'false' to 'true' has been ignored for IP 'mrac_10G_dual/gt_quad_base'
WARNING: [IP_Flow 19-3374] An attempt to modify the value of disabled parameter 'PROT1_ADD_CONFIG_FILE' from 'no_addn_file_loaded' to 'undef' has been ignored for IP 'mrac_10G_dual/gt_quad_base'
WARNING: [IP_Flow 19-3374] An attempt to modify the value of disabled parameter 'PROT2_ADD_CONFIG_EN' from 'false' to 'true' has been ignored for IP 'mrac_10G_dual/gt_quad_base'
WARNING: [IP_Flow 19-3374] An attempt to modify the value of disabled parameter 'PROT2_ADD_CONFIG_FILE' from 'no_addn_file_loaded' to 'undef' has been ignored for IP 'mrac_10G_dual/gt_quad_base'
WARNING: [IP_Flow 19-3374] An attempt to modify the value of disabled parameter 'PROT3_ADD_CONFIG_EN' from 'false' to 'true' has been ignored for IP 'mrac_10G_dual/gt_quad_base'
WARNING: [IP_Flow 19-3374] An attempt to modify the value of disabled parameter 'PROT3_ADD_CONFIG_FILE' from 'no_addn_file_loaded' to 'undef' has been ignored for IP 'mrac_10G_dual/gt_quad_base'
INFO: [BD_TCL-6] Checking if the following IPs exist in the project's IP catalog:  
    xilinx.com:ip:mrmac:3.2 xilinx.com:ip:bufg_gt:1.0 xilinx.com:ip:gt_quad_base:1.1 xilinx.com:inline_hdl:ilconstant:*  .
ERROR: [Common 17-55] 'get_property' expects at least one object.
Resolution: If [get_<value>] was used to populate the object, check to make sure this command returns at least one valid object.
ERROR: [BD 41-2168] Errors found in procedure apply_rule:


INFO: [BD 5-145] Automation rule xilinx.com:bd_rule:mrmac was not applied to object mrmac_0

The error messages don't specify where in the block automation the problem occured.

Seen trying a xcve2302 in Vivado 2025.2 and a xcvm1802 in Vivado 2025.1.

4. Maximum packet size

PG314 doesn't seem to explicity define the maximum packet size which can be transmitted or received.

In mrmac-registers-v3.0.xlsx the CONFIGURATION_RX_MTU_x registers have:

Attribute	Contents
Bits	30:16
Default	9600
Description	Port 0 Maximum Packet Length. Any packet longer than this value is considered to be oversized. The allowed value for this register can range from 64 to 16,383.

Given the value for the maximum packet length being 16,383 initially assumed that was the maximum possible. Albeit given there are 15 bits in the maximum length register, that would put the maxium at 32727. TBD what the actual maximum limit is.

Both PG314 and mrmac-registers-v3.0.xlsx define the histogram registers indicate statistics are mainainted for packets sized between:

64
65-127
128-255
256-511
512-1023
1024-1518
1519-1522
1523-1548
1549-2047
2048-4095
4096-8191
8192-9215

mrmac-registers-v3.0.xlsx also shows statistics registers which count:

less than 64
more than 9215

If have a working design with two MRMAC ports could try exchanging packets between the ports to test the actual maximum, without being limited by the maximum in a switch / other host.

5. Unexpected ports when enabled only client0 and client1

Configured MRMAC to set client0 and client1 as 10GE, with client2 and client3 as N/U.

The tdata and tkeep ports are only present for client0 and clint1 as expected.

However, the tlast, tready and tvalid ports are present for client0, client1 and client2. I.e. unexpected client2 signals. The PG314 Table 14: 10G Non-Segmented Signaling for 32 Bits shows the client2 signals should not be needed. Therefore, will try leaving the tlast, tready and tvalid ports for client2 as unconnected.

6. Tests using VD100_10G_ether_dual

For these initial tests just checking the port status when connected to a T1700G-28TQ 3.0 switch via 10GBase-SR fibre optic modules.

When only port 0 connected to the switch, and the correponding port on the switch was up:

$ identify_pcie_fpga_design/display_identified_pcie_fpga_designs 
Opening device 0000:01:00.0 (10ee:b044) with IOMMU group 12
Enabled bus master for 0000:01:00.0

Design VD100_10G_ether_dual:
  PCI device 0000:01:00.0 rev 00 IOMMU group 12
  DMA bridge bar 2 AXI Stream
  Channel ID  addr_alignment  len_granularity  num_address_bits
       H2C 0               1                1                64
       H2C 1               1                1                64
       C2H 0               1                1                64
       C2H 1               1                1                64
  IIC registers at bar 0 offset 0x11000
  MRMAC configuration revision: 0x00000001
  Port 0:
    Data rate: 10GE
    GT Quad Operating mode: 10GE Wide, 10.3125 Gb/s, 32 bits, 322.2656 MHz, NRZ
    AXI4-Stream Mode: Independent, 32 bits, Non-Segmented
    Rx min packet len: 64
    Rx max packet len: 9600
    FEC Operating Mode: FEC Disabled
    TX realtime status: 0x00000000
      Port TX local fault            : 0
      Port TX axis underflow         : 0 
      Port TX axis error             : 0
      Port TX flexif error           : 0
      Port TX pcs bad code           : 0
      Port TX CL82 CL49 convert error: 0
      Port TX flex fifo overflow     : 0
      Port TX flex fifo underflow    : 0
    RX realtime status: 0x00000003
      Port RX Status: 1
      Port RX Block Lock: 1
      Port RX aligned: 0
      Port RX misaligned: 0
      Port RX aligned error: 0
      Port RX High BER: 0
      Port RX remote fault: 0
      Port RX local fault: 0
      Port RX internal local fault: 0
      Port RX received local fault: 0
      Port RX bad code: 0
      Port RX bad preamble: 0
      Port RX bad SFD: 0
      Port RX got signal ordered set: 0
      Port RX flex if error: 0
      Port RX Framing Error: 0
      Port RX Synced: 0
      Port RX Synced Error: 0
      Port RX BIP Error: 0
      Port RX CL49_82 convert error: 0
      Port RX pcs bad code: 0
      Port RX AXIS fifo overflow: 0
      Port RX AXIS error: 0
      Port RX invalid start: 0
      Port RX flex fifo overflow: 0
      Port RX flex fifo underflow: 0
  Port 1:
    Data rate: 10GE
    GT Quad Operating mode: 10GE Wide, 10.3125 Gb/s, 32 bits, 322.2656 MHz, NRZ
    AXI4-Stream Mode: Independent, 32 bits, Non-Segmented
    Rx min packet len: 64
    Rx max packet len: 9600
    FEC Operating Mode: FEC Disabled
    TX realtime status: 0x00000000
      Port TX local fault            : 0
      Port TX axis underflow         : 0 
      Port TX axis error             : 0
      Port TX flexif error           : 0
      Port TX pcs bad code           : 0
      Port TX CL82 CL49 convert error: 0
      Port TX flex fifo overflow     : 0
      Port TX flex fifo underflow    : 0
    RX realtime status: 0x00000180
      Port RX Status: 0
      Port RX Block Lock: 0
      Port RX aligned: 0
      Port RX misaligned: 0
      Port RX aligned error: 0
      Port RX High BER: 0
      Port RX remote fault: 0
      Port RX local fault: 1
      Port RX internal local fault: 1
      Port RX received local fault: 0
      Port RX bad code: 0
      Port RX bad preamble: 0
      Port RX bad SFD: 0
      Port RX got signal ordered set: 0
      Port RX flex if error: 0
      Port RX Framing Error: 0
      Port RX Synced: 0
      Port RX Synced Error: 0
      Port RX BIP Error: 0
      Port RX CL49_82 convert error: 0
      Port RX pcs bad code: 0
      Port RX AXIS fifo overflow: 0
      Port RX AXIS error: 0
      Port RX invalid start: 0
      Port RX flex fifo overflow: 0
      Port RX flex fifo underflow: 0

When only port 1 connected to the switch, and the correponding port on the switch was up:

$ identify_pcie_fpga_design/display_identified_pcie_fpga_designs 
Opening device 0000:01:00.0 (10ee:b044) with IOMMU group 12
Enabled bus master for 0000:01:00.0

Design VD100_10G_ether_dual:
  PCI device 0000:01:00.0 rev 00 IOMMU group 12
  DMA bridge bar 2 AXI Stream
  Channel ID  addr_alignment  len_granularity  num_address_bits
       H2C 0               1                1                64
       H2C 1               1                1                64
       C2H 0               1                1                64
       C2H 1               1                1                64
  IIC registers at bar 0 offset 0x11000
  MRMAC configuration revision: 0x00000001
  Port 0:
    Data rate: 10GE
    GT Quad Operating mode: 10GE Wide, 10.3125 Gb/s, 32 bits, 322.2656 MHz, NRZ
    AXI4-Stream Mode: Independent, 32 bits, Non-Segmented
    Rx min packet len: 64
    Rx max packet len: 9600
    FEC Operating Mode: FEC Disabled
    TX realtime status: 0x00000000
      Port TX local fault            : 0
      Port TX axis underflow         : 0 
      Port TX axis error             : 0
      Port TX flexif error           : 0
      Port TX pcs bad code           : 0
      Port TX CL82 CL49 convert error: 0
      Port TX flex fifo overflow     : 0
      Port TX flex fifo underflow    : 0
    RX realtime status: 0x00000180
      Port RX Status: 0
      Port RX Block Lock: 0
      Port RX aligned: 0
      Port RX misaligned: 0
      Port RX aligned error: 0
      Port RX High BER: 0
      Port RX remote fault: 0
      Port RX local fault: 1
      Port RX internal local fault: 1
      Port RX received local fault: 0
      Port RX bad code: 0
      Port RX bad preamble: 0
      Port RX bad SFD: 0
      Port RX got signal ordered set: 0
      Port RX flex if error: 0
      Port RX Framing Error: 0
      Port RX Synced: 0
      Port RX Synced Error: 0
      Port RX BIP Error: 0
      Port RX CL49_82 convert error: 0
      Port RX pcs bad code: 0
      Port RX AXIS fifo overflow: 0
      Port RX AXIS error: 0
      Port RX invalid start: 0
      Port RX flex fifo overflow: 0
      Port RX flex fifo underflow: 0
  Port 1:
    Data rate: 10GE
    GT Quad Operating mode: 10GE Wide, 10.3125 Gb/s, 32 bits, 322.2656 MHz, NRZ
    AXI4-Stream Mode: Independent, 32 bits, Non-Segmented
    Rx min packet len: 64
    Rx max packet len: 9600
    FEC Operating Mode: FEC Disabled
    TX realtime status: 0x00000000
      Port TX local fault            : 0
      Port TX axis underflow         : 0 
      Port TX axis error             : 0
      Port TX flexif error           : 0
      Port TX pcs bad code           : 0
      Port TX CL82 CL49 convert error: 0
      Port TX flex fifo overflow     : 0
      Port TX flex fifo underflow    : 0
    RX realtime status: 0x00000003
      Port RX Status: 1
      Port RX Block Lock: 1
      Port RX aligned: 0
      Port RX misaligned: 0
      Port RX aligned error: 0
      Port RX High BER: 0
      Port RX remote fault: 0
      Port RX local fault: 0
      Port RX internal local fault: 0
      Port RX received local fault: 0
      Port RX bad code: 0
      Port RX bad preamble: 0
      Port RX bad SFD: 0
      Port RX got signal ordered set: 0
      Port RX flex if error: 0
      Port RX Framing Error: 0
      Port RX Synced: 0
      Port RX Synced Error: 0
      Port RX BIP Error: 0
      Port RX CL49_82 convert error: 0
      Port RX pcs bad code: 0
      Port RX AXIS fifo overflow: 0
      Port RX AXIS error: 0
      Port RX invalid start: 0
      Port RX flex fifo overflow: 0
      Port RX flex fifo underflow: 0

When both ports 0 and 1 were connected to the switch and were up at the switch end:

$ identify_pcie_fpga_design/display_identified_pcie_fpga_designs 
Opening device 0000:01:00.0 (10ee:b044) with IOMMU group 12
Enabled bus master for 0000:01:00.0

Design VD100_10G_ether_dual:
  PCI device 0000:01:00.0 rev 00 IOMMU group 12
  DMA bridge bar 2 AXI Stream
  Channel ID  addr_alignment  len_granularity  num_address_bits
       H2C 0               1                1                64
       H2C 1               1                1                64
       C2H 0               1                1                64
       C2H 1               1                1                64
  IIC registers at bar 0 offset 0x11000
  MRMAC configuration revision: 0x00000001
  Port 0:
    Data rate: 10GE
    GT Quad Operating mode: 10GE Wide, 10.3125 Gb/s, 32 bits, 322.2656 MHz, NRZ
    AXI4-Stream Mode: Independent, 32 bits, Non-Segmented
    Rx min packet len: 64
    Rx max packet len: 9600
    FEC Operating Mode: FEC Disabled
    TX realtime status: 0x00000000
      Port TX local fault            : 0
      Port TX axis underflow         : 0 
      Port TX axis error             : 0
      Port TX flexif error           : 0
      Port TX pcs bad code           : 0
      Port TX CL82 CL49 convert error: 0
      Port TX flex fifo overflow     : 0
      Port TX flex fifo underflow    : 0
    RX realtime status: 0x00000003
      Port RX Status: 1
      Port RX Block Lock: 1
      Port RX aligned: 0
      Port RX misaligned: 0
      Port RX aligned error: 0
      Port RX High BER: 0
      Port RX remote fault: 0
      Port RX local fault: 0
      Port RX internal local fault: 0
      Port RX received local fault: 0
      Port RX bad code: 0
      Port RX bad preamble: 0
      Port RX bad SFD: 0
      Port RX got signal ordered set: 0
      Port RX flex if error: 0
      Port RX Framing Error: 0
      Port RX Synced: 0
      Port RX Synced Error: 0
      Port RX BIP Error: 0
      Port RX CL49_82 convert error: 0
      Port RX pcs bad code: 0
      Port RX AXIS fifo overflow: 0
      Port RX AXIS error: 0
      Port RX invalid start: 0
      Port RX flex fifo overflow: 0
      Port RX flex fifo underflow: 0
  Port 1:
    Data rate: 10GE
    GT Quad Operating mode: 10GE Wide, 10.3125 Gb/s, 32 bits, 322.2656 MHz, NRZ
    AXI4-Stream Mode: Independent, 32 bits, Non-Segmented
    Rx min packet len: 64
    Rx max packet len: 9600
    FEC Operating Mode: FEC Disabled
    TX realtime status: 0x00000000
      Port TX local fault            : 0
      Port TX axis underflow         : 0 
      Port TX axis error             : 0
      Port TX flexif error           : 0
      Port TX pcs bad code           : 0
      Port TX CL82 CL49 convert error: 0
      Port TX flex fifo overflow     : 0
      Port TX flex fifo underflow    : 0
    RX realtime status: 0x00000003
      Port RX Status: 1
      Port RX Block Lock: 1
      Port RX aligned: 0
      Port RX misaligned: 0
      Port RX aligned error: 0
      Port RX High BER: 0
      Port RX remote fault: 0
      Port RX local fault: 0
      Port RX internal local fault: 0
      Port RX received local fault: 0
      Port RX bad code: 0
      Port RX bad preamble: 0
      Port RX bad SFD: 0
      Port RX got signal ordered set: 0
      Port RX flex if error: 0
      Port RX Framing Error: 0
      Port RX Synced: 0
      Port RX Synced Error: 0
      Port RX BIP Error: 0
      Port RX CL49_82 convert error: 0
      Port RX pcs bad code: 0
      Port RX AXIS fifo overflow: 0
      Port RX AXIS error: 0
      Port RX invalid start: 0
      Port RX flex fifo overflow: 0
      Port RX flex fifo underflow: 0

Chester-Gillon/Notes_about_100G_Multirate_Ethernet_MAC_(MRMAC).md

Select an option

No results found