Chester-Gillon

0. Introduction

Describes notes made about installing AlmaLinux 8, and making customisations as required. The installation was performed on a HP Z640 workstation, and to avoid changing the Windows and Ubuntu installations on the existing RAID arrays used a USB external disk as the target for the AlmaLinux installation.

1. Creation of install media and initial installation.

The full DVD ISO image was downloaded from AlmaLinux-8.4-x86_64-dvd.iso

Followed the instructions at Installation instructions to verify the checksum of the downloaded ISO which was succesful in that the SHA256 of the ISO file matched the expected value:

Chester-Gillon

0. Introduction

As a result of TMDSEMU110-U: TMDSEMU110-U - XDS110 JTAG Debug Probe : Galvanic isolation looking into the performance of different options.

1. USB enumeration speed

For this checking the speed at which the USB interface enumerates at, by looking at the output from lsusb -t under Linux.

For a reference saved the initial lsbusb output when no debug probe connected:

Chester-Gillon

The problem

ibv_controller_worker_messages.h has the following C structures which have fields which are arrays:

/** Contents of the CW_WORKER_READY message sent from a worker to the controller */
typedef struct
{
    /** The number of valid characters in worker_executable_pathname */
    uint32_t worker_executable_pathname_len;
    /** The absolute pathname of the worker executable */

Chester-Gillon

0. Introduction

Notes about using RDMA under Windows 10 21H1, with a Mellanox ConnectX-4 Lx EN

What is the replacement of libibverbs/librdmacm for Windows? suggested verbs isn't actively supported under Windows, and instead to use NetworkDirect which provides as similar API.

1. Install MLNX_OFED for Windows - WinOF / WinOF-2

NVIDIA WinOF-2: Firmware - Driver Compatibility Matrix says for a ConnectX-4 Lx EN under Windows 10 Client 21H1 driver version 2.80 is supported. The firmware on the ConnectX-4 Lx EN has already been updated to 14.32.1010 which is the required version.

Chester-Gillon

0. Introduction

Performing a test of the ibv_message_bw program from ibv_message_passing using:

1. A Mellanox ConnectX-2 VPI card set to Ethernet link-layer in one PC running Ubuntu 18.04.
2. A Mellanox ConnectX-4 Lx card in a different PC running AlmaLinux 8.5.
3. Both ports of each ConnectX card connected to the 4 10G SFP+ ports in a tplink T1700G-28TQ Ethernet switch.

1. Intial test with single port used on the ConnectX-4 and both ports on the ConnectX-2

Was running the tests where two instances of ibv_message_passing were run on each PC where the flow was such that on the ConnectX-2 a different port was used for transmit and receive, and on the ConnectX-4 the same port was used for transmit and receive. I.e.:

Chester-Gillon

0. Introduction

This contains notes about using Data Plane Development Kit (DPDK) with a Mellanox ConnectX-4 Lx under AlmaLinux 8.5

1. Installation

Downloaded the latest DPDK 22.03.0

Uncompressed the source with:

Chester-Gillon

0. Introduction

Contains notes about encryption support in Mellanox ConnectX adapters.

With a ConnectX-4 Lx or ConnectX-5 noticed that the ethtool statistics had some counters with tls in their names, so wasn't sure if that meant the adapters has hardware offload support for encryption.

1. Mellanox product highlights

Starting at ConnectX SmartNICs and looking at the product features:

1. ConnectX-4 Lx says ENCRYPTION N/A

Chester-Gillon

0. Introduction

Notes about checking for the effect of the nohz_full parameter, NO_HZ: Reducing Scheduling-Clock Ticks notes that for the nohz_full parameter to be able to turn CPUs into adaptive-ticks CPUs the kernel must have been built with CONFIG_NO_HZ_FULL=y

1. No obvious effect with a kernel which doesn't have CONFIG_NO_HZ_FULL

Used a 3.10.33-rt32.33.el6rt.x86_64 kernel with Scientific Linux 6.6.

This kernel doesn't have CONFIG_NO_HZ_FULL set:

Chester-Gillon

0. Introduction

The following source file defines the ethercat_frame_t structure where the expected size is 1518 bytes, and the program displays the overall size of the structure and size and offset of it's fields:

/*
 * Program to evalute structure padding
 */

#include <stddef.h>
#include <stdint.h>

Chester-Gillon

0. Introduction

Hardware packet-pacing could be useful for the switch test to get a deterministic transmit rate avoiding software variation, and potentially allow the software to queue a number of packets in one go that are then allow the hardware to output at the requested rate.

Mellanox documentation:

1. Raw Ethernet Programming: Packet Pacing - Code Example
2. HowTo Configure Packet Pacing on ConnectX-4. Says for a ConnectX-4 and ConnectX-4 Lx.
3. Supported Non-Volatile Configurations from relese notes from the 14.32.1010 firmware in use. Nothing in the list appears related to packet pacing.
4. [Rate Limit](https://docs.nvidia.com/networking/pages/viewpage.action?pageId=61869430#QualityofService(QoS)-RateLimitR