MinIO AIStor RDMA RPM Installation Guide (EDGE)

Installing MinIO AIStor RDMA RPM Package

Modes of Operation

MinIO AIStor has two independent communication layers, each with an HTTP and RDMA mode. They are controlled separately and can be mixed in any combination.

                          ┌──────────────────────────────────┐
  S3 Clients              │         MinIO Node 1             │         MinIO Node 2
  (GPU / CPU)             │                                  │
                          │  ┌────────────┐  ┌────────────┐  │  ┌────────────────────┐
  PutObject ─────────────►│  │  S3 HTTP   │  │ Internode  │──┼─►│  Internode HTTP    │
  GetObject  (HTTP/RDMA)  │  │  or        │  │ HTTP or    │  │  │  or                │
  Multipart               │  │  S3 RDMA   │  │ RDMA       │  │  │  Internode RDMA    │
                          │  └────────────┘  └────────────┘  │  └────────────────────┘
                          └──────────────────────────────────┘

S3 HTTP (Client-to-Server)

Standard S3 protocol over TCP. All S3 clients use this by default. Data flows through kernel TCP stack with memory copies between userspace and kernel.

Works with any S3-compatible client
No special hardware required
Always available regardless of RDMA configuration

S3 RDMA / GPU-Direct (Client-to-Server)

Zero-copy data transfer between NVIDIA GPU memory and MinIO storage over RDMA fabric. The client sends an x-amz-rdma-token header with its GPU buffer descriptor; MinIO performs a direct RDMA READ/WRITE bypassing the CPU entirely.

Requires NVIDIA GPU with GPUDirect capability and RDMA NIC
Client must use NVIDIA's aws-c-s3 SDK with GPUDirect support
Enabled automatically when the RDMA binary detects GPU-Direct capable hardware
Falls back to S3 HTTP transparently if the server responds with x-amz-rdma-reply: 501
No environment variable needed to enable -- the binary advertises capability if built with RDMA

Internode HTTP (Server-to-Server)

Default internode communication. Erasure coding shards, healing data, and replication traffic flow over HTTP/TCP between MinIO nodes. With multi-NIC config (v3), traffic is round-robin distributed across NICs for aggregate bandwidth.

Always available as the baseline transport
Multi-NIC support multiplies bandwidth (e.g., 2 NICs = 2x aggregate)
Used for all internode traffic when MINIO_RDMA_INTERNODE=off (default)

Internode RDMA (Server-to-Server)

Replaces HTTP for erasure shard transfers between nodes with RDMA. MinIO opens a dedicated RDMA listener (default port 5555) on each node and performs zero-copy RDMA WRITE/SEND for shard data. Transfers below MINIO_RDMA_MIN_SIZE (default 1MB) still use HTTP.

Requires MINIO_RDMA_INTERNODE=on
Requires RDMA-capable NICs on all nodes (no GPU needed)
With multi-NIC config, RDMA binds a listener per NIC and routes outbound connections to the matching remote NIC
Falls back to internode HTTP for transfers below the minimum size threshold
Control plane (Grid RPC, health checks) remains on HTTP

Combination Matrix

S3 Layer	Internode Layer	Config Required
HTTP	HTTP	Default -- no RDMA config needed
HTTP	RDMA	`MINIO_RDMA_INTERNODE=on`
GPU-Direct	HTTP	RDMA binary + GPU client (internode off)
GPU-Direct	RDMA	RDMA binary + GPU client + `MINIO_RDMA_INTERNODE=on`

GPU-Direct S3 RDMA and Internode RDMA are independent. You can run GPU-Direct without internode RDMA (useful when only the client-to-server path has RDMA fabric) or internode RDMA without GPU-Direct (useful when nodes have RDMA NICs but clients are CPU-based).

Prerequisites

RHEL 10+, Rocky 10+, Alma 10+, or Fedora 42+
Kernel 6.8+ with RDMA subsystem enabled
InfiniBand (HDR/NDR) or RoCE v2 network fabric
RDMA-capable NICs (NVIDIA/Mellanox ConnectX-6+)

Verify RDMA capability before proceeding:

ibv_devinfo   # Must show PORT_ACTIVE

Install

sudo dnf install ./minio-20260327215607.0.0-1.x86_64.rpm

This installs:

Path	Description
`/usr/local/bin/minio`	MinIO AIStor binary (RDMA)
`/usr/lib/minio/libcuobjserver.so.1.0.0`	NVIDIA cuObject server library
`/usr/lib/minio/libcuobjserver.so.1`	soname symlink
`/usr/lib/minio/libcuobjserver.so`	development symlink
`/lib/systemd/system/minio.service`	systemd unit

Dependencies pulled automatically: libibverbs, rdma-core, numactl-libs, shadow-utils, coreutils, util-linux, glibc-common, which.

The post-install script:

Creates minio-user system user and group
Writes /etc/ld.so.conf.d/minio-rdma.conf and runs ldconfig for RDMA library discovery
Creates /etc/default/minio (systemd environment file)
Creates /etc/minio/config.env (MinIO configuration)
Sets up /etc/minio/certs/CAs/

Verify the install:

ldd /usr/local/bin/minio | grep -E 'ibverbs|rdmacm|cuobj'
ldconfig -p | grep cuobjserver

OS Configuration

Kernel Modules

Load required RDMA kernel modules:

# /etc/modules-load.d/rdma.conf
ib_core
ib_uverbs
rdma_cm
mlx5_core

Load immediately without reboot:

sudo modprobe ib_core ib_uverbs rdma_cm mlx5_core

Locked Memory Limits

RDMA buffer registration requires unlimited locked memory. The installed minio.service does not set LimitMEMLOCK, so create a systemd drop-in override:

sudo mkdir -p /etc/systemd/system/minio.service.d
cat <<'EOF' | sudo tee /etc/systemd/system/minio.service.d/rdma.conf
[Service]
LimitMEMLOCK=infinity
EOF
sudo systemctl daemon-reload

Note: /etc/security/limits.conf does not apply to systemd services. The drop-in override above is required.

Firewall

Open the RDMA peer port between cluster nodes:

sudo firewall-cmd --add-port=5555/tcp --permanent
sudo firewall-cmd --reload

Configuration

The systemd unit sources /etc/default/minio and runs:

ExecStart=/usr/local/bin/minio server $MINIO_OPTS $MINIO_VOLUMES

There are two ways to configure the cluster: environment variables in /etc/default/minio or a YAML config file.

Option A: Environment Variables

Edit /etc/default/minio:

MINIO_VOLUMES="https://server{1...4}:9000/mnt/disk{1...8}/"
MINIO_OPTS="-S /etc/minio/certs/ --address :9000 --console-address :9001"
MINIO_CONFIG_ENV_FILE=/etc/minio/config.env

Edit /etc/minio/config.env:

MINIO_ROOT_USER=minioadmin
MINIO_ROOT_PASSWORD=minioadmin

Option B: YAML Config File (v3)

Edit /etc/default/minio:

MINIO_OPTS="--config /etc/minio/config.yaml"
MINIO_CONFIG_ENV_FILE=/etc/minio/config.env

Edit /etc/minio/config.env:

MINIO_ROOT_USER=minioadmin
MINIO_ROOT_PASSWORD=minioadmin

Single-NIC config.yaml

version: v3
rootUser: "minioadmin"
rootPassword: "minioadmin"
address: ":9000"
pools:
  - nodes:
      - addresses:
          - "https://server{1...4}:9000"
        path: "/mnt/disk{1...8}/"

Multi-NIC config.yaml (Aggregate Bandwidth)

Multiple addresses per node enable round-robin internode traffic across NICs. Each address template must expand to the same node count.

version: v3
rootUser: "minioadmin"
rootPassword: "minioadmin"
address: ":9000"
pools:
  - nodes:
      - addresses:
          - "https://192.168.10.{11...18}:9000" # NIC1
          - "https://192.168.20.{11...18}:9000" # NIC2
        path: "/mnt/disk{1...8}/"

Deploy the same config file to all nodes. Each node auto-detects itself by matching NIC IPs against local interfaces.

GPU-Direct RDMA Address Resolution

GPU-Direct S3 RDMA requires binding to a specific NIC IP -- RDMA cannot bind to 0.0.0.0 because the rdma_cm layer needs to locate the exact InfiniBand/RoCE device backing the NIC.

Auto-detection (recommended): When --address uses a wildcard host (--address :9000 or --address 0.0.0.0:9000), MinIO automatically resolves the GPU-Direct bind address using:

v3 config nodes: list -- the first NIC IP from the addresses: matching this node. These are typically the RDMA-capable NICs on the data-plane fabric.
Resolved local hostname -- falls back to globalLocalNodeHost if no v3 multi-NIC config.

This means the standard v3 config is sufficient -- no special --address override is needed for GPU-Direct:

# config.yaml -- GPU-Direct auto-detects the RDMA NIC from addresses:
version: v3
address: ":9000"
pools:
  - nodes:
      - addresses:
          - "https://192.168.10.{11...18}:9000"
        path: "/mnt/disk{1...8}/"

Explicit bind (override): If GPU clients connect through a different NIC than the internode fabric (e.g., a dedicated GPU-to-storage network), specify the exact IP:

MINIO_OPTS="--config /etc/minio/config.yaml --address 192.168.30.11:9000"

The resolved address is logged at startup:

MinIO RDMA Server listening on 192.168.10.11:9000

Enabling Internode RDMA

Add RDMA environment variables to /etc/default/minio:

MINIO_RDMA_INTERNODE=on
MINIO_RDMA_PEER_PORT=5555
MINIO_RDMA_MIN_SIZE=1048576

Environment Variables

Core

Variable	Default	Description
`MINIO_RDMA_INTERNODE`	`off`	Enable internode RDMA (`on` / `off`)
`MINIO_RDMA_PEER_PORT`	`5555`	RDMA listen port
`MINIO_RDMA_MIN_SIZE`	`1048576`	Minimum transfer size in bytes for RDMA

Performance Tuning

Variable	Default	Description
`MINIO_RDMA_CHANNELS`	`256`	Concurrent RDMA channels per node
`MINIO_RDMA_NUM_DCIS`	`256`	DC Initiators (must be >= `MINIO_RDMA_CHANNELS`)
`MINIO_RDMA_CQ_DEPTH`	`4096`	Completion queue depth
`MINIO_RDMA_TIMEOUT`	`14`	Timeout in 4.096us units (14 = ~67ms)
`MINIO_RDMA_RETRY_COUNT`	`7`	Max retries (0-7)
`MINIO_RDMA_MAX_SGE`	`16`	Scatter-gather entries per operation
`MINIO_RDMA_DELAY_INTERVAL`	`500`	Polling delay in nanoseconds
`MINIO_RDMA_DELAY_MODE`	`1`	0=none, 1=batch, 2=entry, 3=adaptive
`MINIO_RDMA_QP_RESET_ON_FAILURE`	`on`	Auto-reset queue pairs on errors
`MINIO_RDMA_SKIP_FDATASYNC`	`off`	Skip fdatasync for RDMA part writes

Advanced

Variable	Default	Description
`MINIO_RDMA_DC_KEY`	`0xffeeddcc`	Dynamic Connection key
`MINIO_RDMA_HOP_LIMIT`	`4`	Hop limit for RDMA packets
`MINIO_RDMA_SERVICE_LEVEL`	`0`	QoS service level (0-15)
`MINIO_RDMA_PKEY_INDEX`	`0`	Partition key index

Tuning Profiles

High-throughput (400Gb/s fabric):

MINIO_RDMA_INTERNODE=on
MINIO_RDMA_CHANNELS=256
MINIO_RDMA_CQ_DEPTH=4096
MINIO_RDMA_NUM_DCIS=256
MINIO_RDMA_DELAY_INTERVAL=500

Low-latency:

MINIO_RDMA_INTERNODE=on
MINIO_RDMA_DELAY_INTERVAL=100
MINIO_RDMA_TIMEOUT=10
MINIO_RDMA_RETRY_COUNT=7

Conservative (start here):

MINIO_RDMA_INTERNODE=on
MINIO_RDMA_CHANNELS=64

Start the Service

sudo systemctl daemon-reload
sudo systemctl enable --now minio

Verify RDMA activation:

journalctl -u minio | grep -i rdma
# "RDMA: GPU-Direct initialized successfully"
# "RDMA: peer-to-peer initialized on port 5555"

Disabling Internode RDMA

Set in /etc/default/minio:

MINIO_RDMA_INTERNODE=off

Restart:

sudo systemctl restart minio

When disabled:

All internode traffic falls back to HTTP over TCP
GPU-Direct S3 client RDMA (PutObject/GetObject) remains available -- only internode peer transfers are affected
No data loss or downtime beyond the restart window
RDMA libraries stay installed but are unused

Verify:

journalctl -u minio --since "5 min ago" | grep -i "rdma.*peer"
curl -s http://localhost:9000/minio/metrics/v3/system/network/internode/rdma

Monitoring

# S3-over-RDMA (GPU-Direct)
curl http://localhost:9000/minio/metrics/v3/api/rdma

# Internode RDMA
curl http://localhost:9000/minio/metrics/v3/system/network/internode/rdma

# InfiniBand fabric
perfquery -x
ibstat | grep -i error

Troubleshooting

Symptom	Check
RDMA not activating	`ibv_devinfo` (PORT_ACTIVE?), `grep MINIO_RDMA /etc/default/minio`
Buffer registration fails	`systemctl show minio -p LimitMEMLOCK` (must show `infinity`)
Connection timeouts	`nc -zv <peer> 5555`, increase `MINIO_RDMA_TIMEOUT`
Low performance	`ibstat \| grep MTU` (should be 4096), tune `MINIO_RDMA_DELAY_INTERVAL`

Uninstall

sudo systemctl disable --now minio
sudo dnf remove minio

Configuration in /etc/minio/ and /etc/default/minio is preserved.

harshavardhana/INSTALL-RDMA-PACKAGES.md

Select an option

No results found