Contains notes about reading information from QSFP modules.
See SFF Specifications for the related standards:
- SFF-8636 Management Interface for 4-lane Modules and Cables : This specification defines a common management interface for 4-lane pluggable transceiver modules and cable assemblies. Physical layer and mechanical details of the connector interface are outside the scope of this document.
- SFF-8024 SFF Module Management Reference Code Tables : This specification provides codes for module identifiers, encoding values, connector types, extended compliance codes, host electrical interfaces and module media interfaces. This specification is the reference source for identifiers assigned to interpret the memory maps of self-identifying modules.
- SFF-8679 QSFP+ 4X Hardware and Electrical Specification : This specification defines the contact pads, the electrical, power supply, ESD and thermal characteristics of the pluggable QSFP+ module or cable plug.
- SFF-8472 Management Interface for SFP+ : This specification defines an enhanced digital diagnostic monitoring interface for optical transceivers which allows real time access to device operating parameters, control and status registers.
For I2C addresses:
- SFF-8636 specifies target address as
10100000for write and10100001as read. That is a 7-bit I2C address of 0x50. - SFF-8472 specifies bus address as:
1010000Xfor common information, which is a 7-bit I2C address of 0x501010001Xfor diagnostic information, which is a 7-bit I2C address of 0x51
This is running AlmaLinux 9.5.
This results in a error:
$ sudo ethtool -m enp16s0
netlink error: Input/output error
DAC cable P/N MC2206130-002 on label. Product brief 56Gb/s and 40GbE QSFP+ Direct Attach Copper Cables
Description is Mellanox® passive copper cable, VPI, up to 40 Gb/s, QSFP, 2m. With this DAC cable fitted to both ports the link is up.
Decoded information:
$ sudo ethtool -m enp16s0
Identifier : 0x0d (QSFP+)
Extended identifier : 0x00
Extended identifier description : 1.5W max. Power consumption
Extended identifier description : No CDR in TX, No CDR in RX
Extended identifier description : High Power Class (> 3.5 W) not enabled
Power set : Off
Power override : Off
Connector : 0x23 (No separable connector)
Transceiver codes : 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00
Encoding : 0x00 (unspecified)
BR, Nominal : 0Mbps
Rate identifier : 0x00
Length (SMF,km) : 0km
Length (OM3 50um) : 0m
Length (OM2 50um) : 0m
Length (OM1 62.5um) : 0m
Length (Copper or Active cable) : 2m
Transmitter technology : 0xa0 (Copper cable unequalized)
Attenuation at 2.5GHz : 5db
Attenuation at 5.0GHz : 8db
Attenuation at 7.0GHz : 0db
Attenuation at 12.9GHz : 0db
Vendor name : Mellanox
Vendor OUI : 00:02:c9
Vendor PN : MC2206130-002
Vendor rev : A2
Vendor SN : MT1309VS00213
Date code : 130312
Revision Compliance : Revision not specified
Module temperature : 0.00 degrees C / 32.00 degrees F
Module voltage : 0.0000 V
Hex decode:
$ sudo ethtool -m enp16s0 hex on
Offset Values
------ ------
0x0000: 0d 00 06 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0040: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0050: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0060: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0070: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0080: 0d 00 23 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0090: 00 00 02 a0 4d 65 6c 6c 61 6e 6f 78 20 20 20 20
0x00a0: 20 20 20 20 07 00 02 c9 4d 43 32 32 30 36 31 33
0x00b0: 30 2d 30 30 32 20 20 20 41 32 05 08 00 00 46 b7
0x00c0: 00 00 00 00 4d 54 31 33 30 39 56 53 30 30 32 31
0x00d0: 33 20 20 20 31 33 30 33 31 32 20 20 00 00 00 d7
0x00e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x00f0: 00 00 00 00 00 00 00 00 00 00 02 00 00 00 00 00
Attempting to use the page argument results in netlink error: Invalid argument.
Using a pair of INNOLIGHT TR-FC13T-NFB 100Gb/s Optical Transceivers with a a OS2 single-mode fibre. The link doesn't come up. Probably due to an incompatability with a 100Gbase module and a 40Gbase card.
Attempting a decode results in an error:
$ sudo ethtool -m enp16s0
[sudo] password for mr_halfword:
netlink error: Invalid argument
Whereas a hex dump works:
$ sudo ethtool -m enp16s0 hex on
Offset Values
------ ------
0x0000: 11 07 00 ff 00 ff 00 00 00 00 00 55 00 00 00 00
0x0010: 00 00 00 00 00 00 13 a2 00 00 82 e3 00 00 00 00
0x0020: 00 00 00 01 00 01 00 01 00 01 00 00 00 00 00 00
0x0030: 00 00 00 01 00 01 00 01 00 01 00 00 00 00 00 00
0x0040: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0050: 00 00 00 00 00 00 00 aa aa 00 00 00 00 03 00 00
0x0060: 00 00 ff 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0070: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0080: 11 cc 07 80 00 00 00 00 00 00 00 07 ff 02 02 00
0x0090: 00 00 00 40 49 4e 4e 4f 4c 49 47 48 54 20 20 20
0x00a0: 20 20 20 20 00 44 7c 7f 54 52 2d 46 43 31 33 54
0x00b0: 2d 4e 46 42 20 20 20 20 31 41 65 a4 39 08 46 12
0x00c0: 06 07 ff fa 49 4e 49 41 46 37 34 31 33 38 39 37
0x00d0: 20 20 20 20 31 38 30 33 30 31 20 20 0c 18 67 5c
0x00e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x00f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
A pair of 100BASE-FX modules connected by a multi mode fibre. These as SFP modules fitted into Mellanox QSFP+ to SFP+ adapters. The link doesn't come up.
Decoded information:
$ sudo ethtool -m enp16s0
[sudo] password for mr_halfword:
Identifier : 0x03 (SFP)
Extended identifier : 0x04 (GBIC/SFP defined by 2-wire interface ID)
Connector : 0x07 (LC)
Transceiver codes : 0x00 0x00 0x00 0x20 0x00 0x00 0x00 0x00 0x00
Transceiver type : Ethernet: 100BASE-FX
Encoding : 0x02 (4B/5B)
BR, Nominal : 100MBd
Rate identifier : 0x00 (unspecified)
Length (SMF,km) : 0km
Length (SMF) : 0m
Length (50um) : 2000m
Length (62.5um) : 2000m
Length (Copper) : 0m
Length (OM3) : 0m
Laser wavelength : 1310nm
Vendor name : AVAGO
Vendor OUI : 00:17:6a
Vendor PN : HFBR-57E0PZ-3C
Vendor rev :
Option values : 0x00 0x12
Option : RX_LOS implemented
Option : TX_DISABLE implemented
BR margin, max : 0%
BR margin, min : 0%
Vendor SN : LRZP020038556
Date code : 090106
Attrempt for hex a decode without any additional parameters fails with an error:
$ sudo ethtool -m enp16s0 hex on
Cannot get Module EEPROM data: Unknown error 2332
A hex decode for bank 1 works:
$ sudo ethtool -m enp16s0 hex on bank 1
Offset Values
------ ------
0x0000: 03 04 07 00 00 00 20 00 00 00 00 02 01 00 00 00
0x0010: c8 c8 00 00 41 56 41 47 4f 20 20 20 20 20 20 20
0x0020: 20 20 20 20 00 00 17 6a 48 46 42 52 2d 35 37 45
0x0030: 30 50 5a 2d 33 43 20 20 20 20 20 20 05 1e 00 70
0x0040: 00 12 00 00 4c 52 5a 50 30 32 30 30 33 38 35 35
0x0050: 36 20 20 20 30 39 30 31 30 36 20 20 00 00 00 f7
0x0060: 33 43 6f 6d 33 43 53 46 50 38 31 20 20 20 20 20
0x0070: ba 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20
While these SFP modulues were labelled as 3COM parts, the decode identifies them as AVAGO HFBR-57E0PZ-3C.
The HFBR-57E0LZ/ALZ/PZ/APZ datahseet Table 3. EEPROM Serial ID Memory Contents matches the hex output for bank 1.
A pair of 4.25 Gb/s fibre optic transceivers connected by a multi mode fibre. These as SFP modules fitted into Mellanox QSFP+ to SFP+ adapters.
The part number of the Mellanox QSFP+ to SFP+ adapters is CVR-QSFP-SFP10G, for which can't find a schematic. MAM1Q00A-QSA is a different part number from NVIDIA which the linked document does contain a schematic which shows how the QSFP+ and SFP+ module control signals are connected.
The link comes up at 1000Mb/s full duplex.
Decoded output:
$ sudo ethtool -m enp16s0
Identifier : 0x03 (SFP)
Extended identifier : 0x04 (GBIC/SFP defined by 2-wire interface ID)
Connector : 0x07 (LC)
Transceiver codes : 0x00 0x00 0x00 0x01 0x20 0x40 0x0c 0x15 0x00
Transceiver type : Ethernet: 1000BASE-SX
Transceiver type : FC: intermediate distance (I)
Transceiver type : FC: Shortwave laser w/o OFC (SN)
Transceiver type : FC: Multimode, 62.5um (M6)
Transceiver type : FC: Multimode, 50um (M5)
Transceiver type : FC: 400 MBytes/sec
Transceiver type : FC: 200 MBytes/sec
Transceiver type : FC: 100 MBytes/sec
Encoding : 0x01 (8B/10B)
BR, Nominal : 4200MBd
Rate identifier : 0x00 (unspecified)
Length (SMF,km) : 0km
Length (SMF) : 0m
Length (50um) : 150m
Length (62.5um) : 70m
Length (Copper) : 0m
Length (OM3) : 0m
Laser wavelength : 850nm
Vendor name : FINISAR CORP.
Vendor OUI : 00:90:65
Vendor PN : FTLF8524P2BNV
Vendor rev : A
Option values : 0x00 0x32
Option : RX_LOS implemented
Option : TX_DISABLE implemented
Option : RATE_SELECT implemented
BR margin, max : 0%
BR margin, min : 0%
Vendor SN : PFA48HS
Date code : 090306
Optical diagnostics support : Yes
Laser bias current : 6.904 mA
Laser output power : 0.3997 mW / -3.98 dBm
Receiver signal average optical power : 0.4399 mW / -3.57 dBm
Module temperature : 28.99 degrees C / 84.19 degrees F
Module voltage : 3.3044 V
Alarm/warning flags implemented : Yes
Laser bias current high alarm : Off
Laser bias current low alarm : Off
Laser bias current high warning : Off
Laser bias current low warning : Off
Laser output power high alarm : Off
Laser output power low alarm : Off
Laser output power high warning : Off
Laser output power low warning : Off
Module temperature high alarm : Off
Module temperature low alarm : Off
Module temperature high warning : Off
Module temperature low warning : Off
Module voltage high alarm : Off
Module voltage low alarm : Off
Module voltage high warning : Off
Module voltage low warning : Off
Laser rx power high alarm : Off
Laser rx power low alarm : Off
Laser rx power high warning : Off
Laser rx power low warning : Off
Laser bias current high alarm threshold : 17.000 mA
Laser bias current low alarm threshold : 1.000 mA
Laser bias current high warning threshold : 14.000 mA
Laser bias current low warning threshold : 2.000 mA
Laser output power high alarm threshold : 0.6310 mW / -2.00 dBm
Laser output power low alarm threshold : 0.0670 mW / -11.74 dBm
Laser output power high warning threshold : 0.6310 mW / -2.00 dBm
Laser output power low warning threshold : 0.0790 mW / -11.02 dBm
Module temperature high alarm threshold : 95.00 degrees C / 203.00 degrees F
Module temperature low alarm threshold : -25.00 degrees C / -13.00 degrees F
Module temperature high warning threshold : 90.00 degrees C / 194.00 degrees F
Module temperature low warning threshold : -20.00 degrees C / -4.00 degrees F
Module voltage high alarm threshold : 3.9000 V
Module voltage low alarm threshold : 2.7000 V
Module voltage high warning threshold : 3.7000 V
Module voltage low warning threshold : 2.9000 V
Laser rx power high alarm threshold : 1.2590 mW / 1.00 dBm
Laser rx power low alarm threshold : 0.0100 mW / -20.00 dBm
Laser rx power high warning threshold : 0.7940 mW / -1.00 dBm
Laser rx power low warning threshold : 0.0158 mW / -18.01 dBm
Hex decode with no page specified:
$ sudo ethtool -m enp16s0 hex on
Offset Values
------ ------
0x0000: 03 04 07 00 00 00 01 20 40 0c 15 01 2a 00 00 00
0x0010: 0f 07 00 00 46 49 4e 49 53 41 52 20 43 4f 52 50
0x0020: 2e 20 20 20 00 00 90 65 46 54 4c 46 38 35 32 34
0x0030: 50 32 42 4e 56 20 20 20 41 20 20 20 03 52 00 71
0x0040: 00 32 00 00 50 46 41 34 38 48 53 20 20 20 20 20
0x0050: 20 20 20 20 30 39 30 33 30 36 20 20 68 d8 01 e3
0x0060: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0070: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0080: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0090: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x00a0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x00b0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x00c0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x00d0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x00e0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x00f0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0100: 5f 00 e7 00 5a 00 ec 00 98 58 69 78 90 88 71 48
0x0110: 21 34 01 f4 1b 58 03 e8 18 a6 02 9e 18 a6 03 16
0x0120: 31 2e 00 64 1f 04 00 9e 00 00 00 00 00 00 00 00
0x0130: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0140: 00 00 00 00 3f 80 00 00 00 00 00 00 01 00 00 00
0x0150: 01 00 00 00 01 00 00 00 01 00 00 00 00 00 00 52
0x0160: 1d 3b 80 f8 0d 7b 0f 9c 11 34 00 00 00 00 10 00
0x0170: 00 00 00 00 00 00 00 00 ff ff ff ff ff ff ff 01
0x0180: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0190: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x01a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x01b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x01c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x01d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x01e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x01f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Can retreive pages 1 and 2:
$ sudo ethtool -m enp16s0 page 1
Offset Values
------ ------
0x0080: 5f 00 e7 00 5a 00 ec 00 98 58 69 78 90 88 71 48
0x0090: 21 34 01 f4 1b 58 03 e8 18 a6 02 9e 18 a6 03 16
0x00a0: 31 2e 00 64 1f 04 00 9e 00 00 00 00 00 00 00 00
0x00b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x00c0: 00 00 00 00 3f 80 00 00 00 00 00 00 01 00 00 00
0x00d0: 01 00 00 00 01 00 00 00 01 00 00 00 00 00 00 52
0x00e0: 1d 6c 81 04 0d 89 0f 9b 11 38 00 00 00 00 10 00
0x00f0: 00 00 00 00 00 00 00 00 ff ff ff ff ff ff ff 01
$ sudo ethtool -m enp16s0 page 2
Offset Values
------ ------
0x0080: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0090: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x00a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x00b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x00c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x00d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x00e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x00f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Looking at the Optical diagnostics under different conditions, with a short 0.5m multi-mode fibre:
- No fibre connected:
Laser bias current : 6.942 mA Laser output power : 0.3999 mW / -3.98 dBm Receiver signal average optical power : 0.0000 mW / -inf dBm - LC connection not fully inserted, and link LEDs flashing and mostly off:
Laser bias current : 6.982 mA Laser output power : 0.4000 mW / -3.98 dBm Receiver signal average optical power : 0.0018 mW / -27.45 dBm - LC connection not fully inserted, but link LEDs fully lit:
Laser bias current : 6.942 mA Laser output power : 0.3998 mW / -3.98 dBm Receiver signal average optical power : 0.0089 mW / -20.51 dBm - LC connection fully inserted:
Laser bias current : 6.916 mA Laser output power : 0.3997 mW / -3.98 dBm Receiver signal average optical power : 0.4491 mW / -3.48 dBm
This is running AlmaLinux 8.10.
Decoded output:
$ sudo ethtool -m ens1f0
Identifier : 0x03 (SFP)
Extended identifier : 0x04 (GBIC/SFP defined by 2-wire interface ID)
Connector : 0x21 (Copper pigtail)
Transceiver codes : 0x00 0x00 0x00 0x00 0x00 0x04 0x00 0x00 0x00
Transceiver type : Passive Cable
Encoding : 0x00 (unspecified)
BR, Nominal : 10300MBd
Rate identifier : 0x00 (unspecified)
Length (SMF,km) : 0km
Length (SMF) : 0m
Length (50um) : 0m
Length (62.5um) : 0m
Length (Copper) : 3m
Length (OM3) : 0m
Passive Cu cmplnce. : 0x01 (SFF-8431 appendix E) [SFF-8472 rev10.4 only]
Vendor name : OEM
Vendor OUI : 00:40:20
Vendor PN : SFP-H10GB-CU3M
Vendor rev : R
Option values : 0x00 0x00
BR margin, max : 0%
BR margin, min : 0%
Vendor SN : CSC210403850189
Date code : 210426
Hex output:
$ sudo ethtool -m ens1f0 hex on
Offset Values
------ ------
0x0000: 03 04 21 00 00 00 00 00 04 00 00 00 67 00 00 00
0x0010: 00 00 03 00 4f 45 4d 20 20 20 20 20 20 20 20 20
0x0020: 20 20 20 20 00 00 40 20 53 46 50 2d 48 31 30 47
0x0030: 42 2d 43 55 33 4d 20 20 52 20 20 20 01 00 00 f7
0x0040: 00 00 00 00 43 53 43 32 31 30 34 30 33 38 35 30
0x0050: 31 38 39 20 32 31 30 34 32 36 20 20 00 00 00 d1
0x0060: 81 00 11 3d 6f 15 24 9c 21 00 31 ad 1e c4 4e 68
0x0070: 6d ad 07 00 00 00 00 00 00 00 00 00 a1 f7 52 aa
0x0080: 43 4f 50 51 41 41 36 4a 41 42 33 37 2d 30 39 36
0x0090: 31 2d 30 33 56 30 33 20 01 00 46 00 00 00 00 cf
0x00a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x00b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x00c0: 53 46 50 2d 48 31 30 47 42 2d 43 55 33 4d 20 20
0x00d0: 20 20 20 20 30 39 00 00 00 00 00 00 00 00 00 b6
0x00e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x00f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0100: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0110: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0120: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0130: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0140: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0150: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0160: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0170: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0180: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0190: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x01a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x01b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x01c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x01d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x01e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x01f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
The dpdk_information program was modified to test using rte_eth_dev_get_module_info and rte_eth_dev_get_module_eeprom. With a 10G Ethernet DAC cable between each port and a switch, the results were:
$ dpdk_switch_test/dpdk_information --no-huge -m 1024m
EAL: Detected CPU lcores: 8
EAL: Detected NUMA nodes: 1
EAL: Detected shared linkage of DPDK
EAL: Multi-process socket /run/user/1000/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'VA'
EAL: VFIO support initialized
EAL: Probe PCI driver: mlx5_pci (15b3:1015) device: 0000:21:00.0 (socket 0)
mlx5_net: No available register for sampler.
EAL: Probe PCI driver: mlx5_pci (15b3:1015) device: 0000:21:00.1 (socket 0)
mlx5_net: No available register for sampler.
TELEMETRY: No legacy callbacks, legacy socket not created
Non EAL arguments: dpdk_switch_test/dpdk_information
rte_lcore_count = 8
rte_eal_iova_mode : DMA using virtual address
rte_eth_dev_count_total = 2
rte_eth_dev_count_avail = 2
port_id=0 driver_name=mlx5_pci
if_index=4 if_name=ens1f0
Rx Offloading Capabilities:
Per Queue : VLAN_STRIP IPV4_CKSUM UDP_CKSUM TCP_CKSUM TCP_LRO SCATTER TIMESTAMP KEEP_CRC RSS_HASH BUFFER_SPLIT
Per Port : VLAN_FILTER
Tx Offloading Capabilities:
Per Queue :
Per Port : VLAN_INSERT IPV4_CKSUM UDP_CKSUM TCP_CKSUM TCP_TSO OUTER_IPV4_CKSUM VXLAN_TNL_TSO GRE_TNL_TSO GENEVE_TNL_TSO MULTI_SEGS MBUF_FAST_FREE UDP_TNL_TSO IP_TNL_TSO
rte_eth_dev_get_eeprom_length() failed : Operation not supported
rte_eth_dev_get_module_info() module type=SFF_8472 eeprom_len=512
Offset Values
------ ------
0x0000: 03 04 21 00 00 00 00 00 04 00 00 00 67 00 00 00
0x0010: 00 00 03 00 4f 45 4d 20 20 20 20 20 20 20 20 20
0x0020: 20 20 20 20 00 00 40 20 53 46 50 2d 48 31 30 47
0x0030: 42 2d 43 55 33 4d 20 20 52 20 20 20 01 00 00 f7
0x0040: 00 00 00 00 43 53 43 32 31 30 34 30 33 38 35 30
0x0050: 31 38 39 20 32 31 30 34 32 36 20 20 00 00 00 d1
0x0060: 81 00 11 3d 6f 15 24 9c 21 00 31 ad 1e c4 4e 68
0x0070: 6d ad 07 00 00 00 00 00 00 00 00 00 a1 f7 52 aa
0x0080: 43 4f 50 51 41 41 36 4a 41 42 33 37 2d 30 39 36
0x0090: 31 2d 30 33 56 30 33 20 01 00 46 00 00 00 00 cf
0x00a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x00b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x00c0: 53 46 50 2d 48 31 30 47 42 2d 43 55 33 4d 20 20
0x00d0: 20 20 20 20 30 39 00 00 00 00 00 00 00 00 00 b6
0x00e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x00f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0100: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0110: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0120: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0130: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0140: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0150: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0160: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0170: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0180: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0190: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x01a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x01b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x01c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x01d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x01e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x01f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
adjusted nb_rxd=1 nb_txd=1
rx_qinfo nb_desc=1 rx_buf_size=0 scattered_rx=0
rx_qinfo rx_free_thresh=1 rx_drop_en=1 pthresh=0 hthresh=0 wthresh=0
mlx5_net: port 0 number of descriptors requested for Tx queue 0 must be higher than MLX5_TX_COMP_THRESH, using 33 instead of 1
mlx5_net: port 0 increased number of descriptors in Tx queue 0 to the next power of two (64)
tx_qinfo nb_desc=64 pthresh=0 hthresh=0 wthresh=0 tx_rs_thresh=0 tx_free_thresh=0
link_speed=10000 Mbps link_duplex=Full link_autoneg=Autonegotiated link_status=Up
Flow control mode : Enable flow control on both side
Flow control high_water=0 low_water=0 pause_time=0 send_xon=0 mac_ctrl_frame_fwd=0 autoneg=0
rx_desc_lim nb_max=65535 nb_min=0 nb_align=1 nb_seg_max=65535 nb_mtu_seg_max=65535
tx_desc_lim nb_max=65535 nb_min=0 nb_align=1 nb_seg_max=40 nb_mtu_seg_max=40
nb_rx_queues=0 nb_tx_queues=0
default_rxportconf burst_size=64 ring_size=256 nb_queues=8
default_txportconf burst_size=64 ring_size=256 nb_queues=8
rx_pkt_burst=0x7f407c74a0e0 rx_queue_count=0x7f407c569980 rx_descriptor_status=0x7f407c569730
tx_pkt_burst=0x7f407c67fdc0 tx_pkt_prepare=(nil) tx_descriptor_status=0x7f407c57fed0
port_id=1 driver_name=mlx5_pci
if_index=5 if_name=ens1f1
Rx Offloading Capabilities:
Per Queue : VLAN_STRIP IPV4_CKSUM UDP_CKSUM TCP_CKSUM TCP_LRO SCATTER TIMESTAMP KEEP_CRC RSS_HASH BUFFER_SPLIT
Per Port : VLAN_FILTER
Tx Offloading Capabilities:
Per Queue :
Per Port : VLAN_INSERT IPV4_CKSUM UDP_CKSUM TCP_CKSUM TCP_TSO OUTER_IPV4_CKSUM VXLAN_TNL_TSO GRE_TNL_TSO GENEVE_TNL_TSO MULTI_SEGS MBUF_FAST_FREE UDP_TNL_TSO IP_TNL_TSO
rte_eth_dev_get_eeprom_length() failed : Operation not supported
rte_eth_dev_get_module_info() module type=SFF_8472 eeprom_len=512
Offset Values
------ ------
0x0000: 03 04 21 00 00 00 00 00 04 00 00 00 67 00 00 00
0x0010: 00 00 03 00 4f 45 4d 20 20 20 20 20 20 20 20 20
0x0020: 20 20 20 20 00 00 40 20 53 46 50 2d 48 31 30 47
0x0030: 42 2d 43 55 33 4d 20 20 52 20 20 20 01 00 00 f7
0x0040: 00 00 00 00 43 53 43 32 31 30 34 30 33 38 35 30
0x0050: 31 33 39 20 32 31 30 34 32 36 20 20 00 00 00 cc
0x0060: 81 00 11 e7 f8 d6 b5 d4 9c 30 17 40 51 ee df 9c
0x0070: e7 30 1c 00 00 00 00 00 00 00 00 00 63 12 44 33
0x0080: 43 4f 50 51 41 41 36 4a 41 42 33 37 2d 30 39 36
0x0090: 31 2d 30 33 56 30 33 20 01 00 46 00 00 00 00 cf
0x00a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x00b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x00c0: 53 46 50 2d 48 31 30 47 42 2d 43 55 33 4d 20 20
0x00d0: 20 20 20 20 30 39 00 00 00 00 00 00 00 00 00 b6
0x00e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x00f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0100: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0110: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0120: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0130: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0140: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0150: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0160: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0170: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0180: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0190: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x01a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x01b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x01c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x01d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x01e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x01f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
adjusted nb_rxd=1 nb_txd=1
rx_qinfo nb_desc=1 rx_buf_size=0 scattered_rx=0
rx_qinfo rx_free_thresh=1 rx_drop_en=1 pthresh=0 hthresh=0 wthresh=0
mlx5_net: port 1 number of descriptors requested for Tx queue 0 must be higher than MLX5_TX_COMP_THRESH, using 33 instead of 1
mlx5_net: port 1 increased number of descriptors in Tx queue 0 to the next power of two (64)
tx_qinfo nb_desc=64 pthresh=0 hthresh=0 wthresh=0 tx_rs_thresh=0 tx_free_thresh=0
link_speed=10000 Mbps link_duplex=Full link_autoneg=Autonegotiated link_status=Up
Flow control mode : Enable flow control on both side
Flow control high_water=0 low_water=0 pause_time=0 send_xon=0 mac_ctrl_frame_fwd=0 autoneg=0
rx_desc_lim nb_max=65535 nb_min=0 nb_align=1 nb_seg_max=65535 nb_mtu_seg_max=65535
tx_desc_lim nb_max=65535 nb_min=0 nb_align=1 nb_seg_max=40 nb_mtu_seg_max=40
nb_rx_queues=0 nb_tx_queues=0
default_rxportconf burst_size=64 ring_size=256 nb_queues=8
default_txportconf burst_size=64 ring_size=256 nb_queues=8
rx_pkt_burst=0x7f407c74a0e0 rx_queue_count=0x7f407c569980 rx_descriptor_status=0x7f407c569730
tx_pkt_burst=0x7f407c67fdc0 tx_pkt_prepare=(nil) tx_descriptor_status=0x7f407c57fed0
The raw hex decode from dpdk_information has the same output as from ethtool.
DPDK isn't suitable for attempting to modify the CDR settings when the INNOLIGHT TR-FC13T-NFB is fitted to the ConnectX-3 Pro since:
- Under AlmaLinux 8 and 9 the dpdk package no longer seems to include the mlx4 driver.
- Looking at the dpdk source the mlx4 driver doesn't include support for
rte_eth_dev_get_module_infoorrte_eth_dev_get_module_eeprom. - The DPDK API doesn't contain a function to set the module EEPROM, only to get the contents.
- The dpdk mlx5 driver issues
ETHTOOL_GMODULEINFOandETHTOOL_GMODULEEEPROMcommands to the Kernel driver viaioctl. I.e. doesn't show how to perform register access using I2C to the actual module.
This is running AlmaLinux 9.5
This is unmaintained hardware in AlmaLinux 9.5:
$ dmesg|grep Unmain
[ 2.262033] Warning: Unmaintained Hardware is detected: mlx4_core:15B3:1003 @ 0000:10:00.0
[ 34.184649] Warning: Unmaintained driver is detected: ip_set
The kernel/rh_messages.h Kernel source file contains the array rh_unmaintained_pci_devices with the list of the devices considered unmaintained.
At boot the ports are Infiniband, and therefore the ConnectX-3 wasn't showing up as an Ethernet devices. Switched the ports to Ethernet:
$ echo eth | sudo tee /sys/module/mlx4_core/drivers/pci\:mlx4_core/00*/mlx4_port1
eth
$ echo eth | sudo tee /sys/module/mlx4_core/drivers/pci\:mlx4_core/00*/mlx4_port2
eth
For some reason the card can read from a DAC cable fitted to enp16s0, but returns Input/output error attempting to read a fibre module in that port.
This results in a error:
$ sudo ethtool -m enp16s0
[sudo] password for mr_halfword:
netlink error: Input/output error
DAC cable P/N MC2206130-002 on label. With this cable fitted to both ports the link is up.
Decoded information:
$ sudo ethtool -m enp16s0
Identifier : 0x0d (QSFP+)
Extended identifier : 0x00
Extended identifier description : 1.5W max. Power consumption
Extended identifier description : No CDR in TX, No CDR in RX
Extended identifier description : High Power Class (> 3.5 W) not enabled
Power set : Off
Power override : Off
Connector : 0x23 (No separable connector)
Transceiver codes : 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00
Encoding : 0x00 (unspecified)
BR, Nominal : 0Mbps
Rate identifier : 0x00
Length (SMF,km) : 0km
Length (OM3 50um) : 0m
Length (OM2 50um) : 0m
Length (OM1 62.5um) : 0m
Length (Copper or Active cable) : 2m
Transmitter technology : 0xa0 (Copper cable unequalized)
Attenuation at 2.5GHz : 5db
Attenuation at 5.0GHz : 8db
Attenuation at 7.0GHz : 0db
Attenuation at 12.9GHz : 0db
Vendor name : Mellanox
Vendor OUI : 00:02:c9
Vendor PN : MC2206130-002
Vendor rev : A2
Vendor SN : MT1309VS00213
Date code : 130312
Revision Compliance : Revision not specified
Module temperature : 0.00 degrees C / 32.00 degrees F
Module voltage : 0.0000 V
Hex decode:
$ sudo ethtool -m enp16s0 hex on
Offset Values
------ ------
0x0000: 0d 00 06 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0040: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0050: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0060: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0070: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0080: 0d 00 23 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0090: 00 00 02 a0 4d 65 6c 6c 61 6e 6f 78 20 20 20 20
0x00a0: 20 20 20 20 07 00 02 c9 4d 43 32 32 30 36 31 33
0x00b0: 30 2d 30 30 32 20 20 20 41 32 05 08 00 00 46 b7
0x00c0: 00 00 00 00 4d 54 31 33 30 39 56 53 30 30 32 31
0x00d0: 33 20 20 20 31 33 30 33 31 32 20 20 00 00 00 d7
0x00e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x00f0: 00 00 00 00 00 00 00 00 00 00 02 00 00 00 00 00
Using a pair of INNOLIGHT TR-FC13T-NFB 100Gb/s Optical Transceivers with a a OS2 single-mode fibre. The link doesn't come up. Probably due to an incompatability with a 100Gbase module and a 40Gbase card.
Attempting a decode results in an error:
$ sudo ethtool -m enp16s0d1
netlink error: Invalid argument
Whereas a hex dump works:
$ sudo ethtool -m enp16s0d1 hex on
Offset Values
------ ------
0x0000: 11 07 00 ff 00 ff 00 00 00 00 55 00 00 55 00 00
0x0010: 00 00 00 00 00 00 16 f2 00 00 81 c5 00 00 00 00
0x0020: 00 00 00 01 00 01 00 01 00 01 00 00 00 00 00 00
0x0030: 00 00 00 01 00 01 00 01 00 01 00 00 00 00 00 00
0x0040: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0050: 00 00 00 00 00 00 00 aa aa 00 00 00 00 03 00 00
0x0060: 00 00 ff 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0070: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0080: 11 cc 07 80 00 00 00 00 00 00 00 07 ff 02 02 00
0x0090: 00 00 00 40 49 4e 4e 4f 4c 49 47 48 54 20 20 20
0x00a0: 20 20 20 20 00 44 7c 7f 54 52 2d 46 43 31 33 54
0x00b0: 2d 4e 46 42 20 20 20 20 31 41 65 a4 39 08 46 12
0x00c0: 06 07 ff fa 49 4e 49 41 46 37 34 31 31 35 31 30
0x00d0: 20 20 20 20 31 38 30 32 32 37 20 20 0c 18 67 4f
0x00e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x00f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
A pair of 4.25 Gb/s fibre optic transceivers connected by a multi mode fibre. These as SFP modules fitted into Mellanox QSFP to SFP adapters.
The link doesn't come, potential issue with one port not accepting fibre modules
Decoded output:
$ sudo ethtool -m enp16s0d1
[sudo] password for mr_halfword:
Identifier : 0x03 (SFP)
Extended identifier : 0x04 (GBIC/SFP defined by 2-wire interface ID)
Connector : 0x07 (LC)
Transceiver codes : 0x00 0x00 0x00 0x01 0x20 0x40 0x0c 0x15 0x00
Transceiver type : Ethernet: 1000BASE-SX
Transceiver type : FC: intermediate distance (I)
Transceiver type : FC: Shortwave laser w/o OFC (SN)
Transceiver type : FC: Multimode, 62.5um (M6)
Transceiver type : FC: Multimode, 50um (M5)
Transceiver type : FC: 400 MBytes/sec
Transceiver type : FC: 200 MBytes/sec
Transceiver type : FC: 100 MBytes/sec
Encoding : 0x01 (8B/10B)
BR, Nominal : 4200MBd
Rate identifier : 0x00 (unspecified)
Length (SMF,km) : 0km
Length (SMF) : 0m
Length (50um) : 150m
Length (62.5um) : 70m
Length (Copper) : 0m
Length (OM3) : 0m
Laser wavelength : 850nm
Vendor name : FINISAR CORP.
Vendor OUI : 00:90:65
Vendor PN : FTLF8524P2BNV
Vendor rev : A
Option values : 0x00 0x32
Option : RX_LOS implemented
Option : TX_DISABLE implemented
Option : RATE_SELECT implemented
BR margin, max : 0%
BR margin, min : 0%
Vendor SN : PFA4HFX
Date code : 090306
Optical diagnostics support : Yes
Laser bias current : 6.980 mA
Laser output power : 0.4046 mW / -3.93 dBm
Receiver signal average optical power : 0.0000 mW / -inf dBm
Module temperature : 31.69 degrees C / 89.04 degrees F
Module voltage : 3.3093 V
Alarm/warning flags implemented : Yes
Laser bias current high alarm : Off
Laser bias current low alarm : Off
Laser bias current high warning : Off
Laser bias current low warning : Off
Laser output power high alarm : Off
Laser output power low alarm : Off
Laser output power high warning : Off
Laser output power low warning : Off
Module temperature high alarm : Off
Module temperature low alarm : Off
Module temperature high warning : Off
Module temperature low warning : Off
Module voltage high alarm : Off
Module voltage low alarm : Off
Module voltage high warning : Off
Module voltage low warning : Off
Laser rx power high alarm : Off
Laser rx power low alarm : On
Laser rx power high warning : Off
Laser rx power low warning : On
Laser bias current high alarm threshold : 17.000 mA
Laser bias current low alarm threshold : 1.000 mA
Laser bias current high warning threshold : 14.000 mA
Laser bias current low warning threshold : 2.000 mA
Laser output power high alarm threshold : 0.6310 mW / -2.00 dBm
Laser output power low alarm threshold : 0.0670 mW / -11.74 dBm
Laser output power high warning threshold : 0.6310 mW / -2.00 dBm
Laser output power low warning threshold : 0.0790 mW / -11.02 dBm
Module temperature high alarm threshold : 95.00 degrees C / 203.00 degrees F
Module temperature low alarm threshold : -25.00 degrees C / -13.00 degrees F
Module temperature high warning threshold : 90.00 degrees C / 194.00 degrees F
Module temperature low warning threshold : -20.00 degrees C / -4.00 degrees F
Module voltage high alarm threshold : 3.9000 V
Module voltage low alarm threshold : 2.7000 V
Module voltage high warning threshold : 3.7000 V
Module voltage low warning threshold : 2.9000 V
Laser rx power high alarm threshold : 1.2590 mW / 1.00 dBm
Laser rx power low alarm threshold : 0.0100 mW / -20.00 dBm
Laser rx power high warning threshold : 0.7940 mW / -1.00 dBm
Laser rx power low warning threshold : 0.0158 mW / -18.01 dBm
Hex decode:
$ sudo ethtool -m enp16s0d1 hex on
Offset Values
------ ------
0x0000: 03 04 07 00 00 00 01 20 40 0c 15 01 2a 00 00 00
0x0010: 0f 07 00 00 46 49 4e 49 53 41 52 20 43 4f 52 50
0x0020: 2e 20 20 20 00 00 90 65 46 54 4c 46 38 35 32 34
0x0030: 50 32 42 4e 56 20 20 20 41 20 20 20 03 52 00 71
0x0040: 00 32 00 00 50 46 41 34 48 46 58 20 20 20 20 20
0x0050: 20 20 20 20 30 39 30 33 30 36 20 20 68 d8 01 f6
0x0060: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0070: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0080: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0090: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x00a0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x00b0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x00c0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x00d0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x00e0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x00f0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0100: 5f 00 e7 00 5a 00 ec 00 98 58 69 78 90 88 71 48
0x0110: 21 34 01 f4 1b 58 03 e8 18 a6 02 9e 18 a6 03 16
0x0120: 31 2e 00 64 1f 04 00 9e 00 00 00 00 00 00 00 00
0x0130: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0140: 00 00 00 00 3f 80 00 00 00 00 00 00 01 00 00 00
0x0150: 01 00 00 00 01 00 00 00 01 00 00 00 00 00 00 52
0x0160: 1f de 81 40 0d 78 0f 8d 00 00 00 00 00 00 12 00
0x0170: 00 40 00 00 00 40 00 00 ff ff ff ff ff ff ff 01
0x0180: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0190: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x01a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x01b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x01c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x01d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x01e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x01f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00