Resizable BAR on SuperMicro H11SSL-I with Intel Arc B580

README.md

Resizable BAR on SuperMicro H11SSL-I with Intel Arc B580

This might work other mainboards and graphic adapters as well, but this article assumes H11SSL and B580 running on a Linux operating system.

Tip

UEFI hacks is possible ¹ using ReBarUEFI, but this is more invasive and requires firmware modifications. In worst case it can brick your system and require recovery with a chip programmer. If you're running Windows then this is probably the only solution.

The Issue

The SuperMicro H11SSL-I board does not have an option in the BIOS to enable "Resizable BAR" ("Resizable Base Address Register", or "ReBAR"), aka SAM ("Smart Access Memory") for AMD Platforms. It was not introduced until the H12 series boards.

Without ReBAR the kernel will report the following:

[    6.177256] xe 0000:08:00.0: [drm] Attempting to resize bar from 256MiB -> 16384MiB
[    6.177273] xe 0000:08:00.0: BAR 0 [mem 0xec000000-0xecffffff 64bit]: releasing
[    6.177284] xe 0000:08:00.0: BAR 2 [mem 0x1ffe0000000-0x1ffefffffff 64bit pref]: releasing
[    6.177462] xe 0000:08:00.0: BAR 2 [mem size 0x400000000 64bit pref]: can't assign; no space
[    6.177470] xe 0000:08:00.0: BAR 2 [mem size 0x400000000 64bit pref]: failed to assign
[    6.177480] xe 0000:08:00.0: BAR 0 [mem 0xec000000-0xecffffff 64bit]: assigned
[    6.177503] xe 0000:08:00.0: BAR 2 [mem size 0x400000000 64bit pref]: can't assign; no space
[    6.177511] xe 0000:08:00.0: BAR 2 [mem size 0x400000000 64bit pref]: failed to assign
[    6.177520] xe 0000:08:00.0: BAR 0 [mem 0xec000000-0xecffffff 64bit]: assigned
[    6.177658] xe 0000:08:00.0: [drm] Failed to resize BAR2 to 16384M (-ENOSPC). Consider enabling 'Resizable BAR' support in your BIOS
[    6.177671] xe 0000:08:00.0: BAR 2 [mem 0x1ffe0000000-0x1ffefffffff 64bit pref]: assigned
[    6.179509] xe 0000:08:00.0: [drm] VISIBLE VRAM: 0x000001ffe0000000, 0x0000000010000000
[    6.179571] xe 0000:08:00.0: [drm] Small BAR device
[    6.179577] xe 0000:08:00.0: [drm] VRAM[0]: Actual physical size 0x0000000300000000, usable size exclude stolen 0x00000002fb800000, CPU accessible size 0x0000000010000000
[    6.179587] xe 0000:08:00.0: [drm] VRAM[0]: DPA range: [0x0000000000000000-300000000], io range: [0x000001ffe0000000-1fff0000000]
[    6.179597] xe 0000:08:00.0: [drm] VRAM: 0x0000000300000000 is larger than resource 0x0000000010000000
[    6.179605] xe 0000:08:00.0: [drm] Small BAR device
[    6.179611] xe 0000:08:00.0: [drm] VRAM[0]: Actual physical size 0x0000000300000000, usable size exclude stolen 0x00000002fb800000, CPU accessible size 0x0000000010000000
[    6.179620] xe 0000:08:00.0: [drm] VRAM[0]: DPA range: [0x0000000000000000-300000000], io range: [0x000001ffe0000000-1fff0000000]

We can see a total VRAM size if 0x0000000300000000 (12 GiB), but CPU can only access 0x0000000010000000 (256 MiB) windows.

This is reflected when we inspect the PCI device memory regions:

        Region 0: Memory at ec000000 (64-bit, non-prefetchable) [size=16M]
        Region 2: Memory at 1ffe0000000 (64-bit, prefetchable) [size=256M]

Workaround

Turns out it's possible to apply ReBAR under certain conditions by overriding PCI registers. No invasive hacks and easily reversible.

Note

This was written for Ubuntu², but it can be adopted to any distro.

Prerequisites

• Latest revision (3.4) BIOS recommended ³
• Above 4G Decoding must be enabled in BIOS
• Linux Kernel version 6.12 or later with intel xe driver is required ²
• GPU must be in bottom x16 slot ⁴⁵

Step 1: Update bootloader

In /etc/default/grub enable the realloc flag to allow kernel to reorganize PCI address space:

GRUB_CMDLINE_LINUX_DEFAULT="pci=realloc"

Apply the changes:

sudo update-grub

Reboot and confirm this is applied using cat /proc/cmdline.

Step 2: Update initramfs

Next, a couple of initramfs⁶ scripts is required to perform the workaround when the system boots.

First create a hook:

sudo tee /etc/initramfs-tools/hooks/b580-rebar > /dev/null << 'EOF'
#!/bin/sh

# initramfs dependency check
PREREQ=""
prereqs() { echo "$PREREQ"; }
case "$1" in
  prereqs) prereqs; exit 0 ;;
esac

# Load api
. /usr/share/initramfs-tools/hook-functions

# Install binaries required for resize script
copy_exec /usr/bin/setpci /usr/bin
copy_exec /usr/bin/lspci  /usr/bin
EOF

Then the resize script:

sudo tee /etc/initramfs-tools/scripts/init-premount/b580-rebar > /dev/null << 'EOF'
#!/bin/sh

# initramfs dependency check
PREREQ=""
prereqs() { echo "$PREREQ"; }
case "$1" in
  prereqs) prereqs; exit 0 ;;
esac

# Find relevant PCI addresses from vendor device id lookups
GPU=$(lspci -D -d 8086:e20b | cut -d' ' -f1)
[ -z "$GPU" ] && exit 0
AUDIO=$(lspci -D -d 8086:e2f7 | cut -d' ' -f1)

# Already 16G? nothing to do
lspci -vv -s "$GPU" 2>/dev/null | grep -q 'size=16G' && exit 0

# Walk UP to the root port (parent is the root complex pci0000:NN)
dev=$(readlink -f /sys/bus/pci/devices/$GPU)
parent=$(dirname "$dev")
while :; do
  case "$(basename "$parent")" in
    pci0000:*) break ;;
  esac
  dev="$parent"; parent=$(dirname "$dev")
done
PORT=$(basename "$dev")

# Release whatever grabbed the card during coldplug
echo "$GPU" > /sys/bus/pci/drivers/xe/unbind 2>/dev/null || true
[ -n "$AUDIO" ] && echo "$AUDIO" > /sys/bus/pci/drivers/snd_hda_intel/unbind 2>/dev/null || true

# Resizable BAR control -> size field 14 (= 16G). ECAP_REBAR+0x08 is the control reg for the first resizable BAR
setpci -s "$GPU" COMMAND=0000
old=$(setpci -s "$GPU" ECAP_REBAR+0x08.l)
new=$(printf '%08x' $(( (0x$old & 0xFFFFE0FF) | (14 << 8) )))
setpci -s "$GPU" ECAP_REBAR+0x08.l=$new

# Re-lay the bridge window and re-enumerate; xe rebinds against the 16G BAR
echo 1 > /sys/bus/pci/devices/$PORT/remove
sleep 1
echo 1 > /sys/bus/pci/rescan

exit 0
EOF

Finalize and install:

sudo chmod +x /etc/initramfs-tools/hooks/b580-rebar
sudo chmod +x /etc/initramfs-tools/scripts/init-premount/b580-rebar
sudo update-initramfs -u

Testing

After a reboot sudo dmesg | grep -iE "xe.*(BAR|VRAM|resize)" should result in output that confirms the changes:

[    7.474451] xe 0000:08:00.0: [drm] VISIBLE VRAM: 0x0000019400000000, 0x0000000400000000
[    7.474666] xe 0000:08:00.0: [drm] VRAM[0]: Actual physical size 0x0000000300000000, usable size exclude stolen 0x00000002fb800000, CPU accessible size 0x00000002fb800000
[    7.474680] xe 0000:08:00.0: [drm] VRAM[0]: DPA range: [0x0000000000000000-300000000], io range: [0x0000019400000000-196fb800000]
[    7.474694] xe 0000:08:00.0: [drm] VRAM[0]: Actual physical size 0x0000000300000000, usable size exclude stolen 0x00000002fb800000, CPU accessible size 0x00000002fb800000
[    7.474705] xe 0000:08:00.0: [drm] VRAM[0]: DPA range: [0x0000000000000000-300000000], io range: [0x0000019400000000-196fb800000]

Here we can see that the CPU can access the entire VRAM: 0x00000002fb800000 (~11.93 GiB) excluding reserved firmware which would total to 0x0000000300000000(12 GiB)

Next, in lspci -vv -s 08:00.0 | grep -i region:

        Region 0: Memory at ea000000 (64-bit, non-prefetchable) [size=16M]
        Region 2: Memory at 19400000000 (64-bit, prefetchable) [size=16G]

Here we can see a large region of 0x0000000400000000 (16 GiB) which was exactly what the driver originally failed to resize to before the workaround!

Footnotes

1. I can't really find any other sources of successful attempts ↩

2. If you run Ubuntu 24.04 you need to install Ubuntu Hardware Enablement (HWE) Kernel using: sudo apt install --install-recommends linux-generic-hwe-24.04 ↩ ↩²

3. A SuperMiro account is required to download BIOS images. If your BMC requires a serial number unlock to flash, simply use supermicro-product-key to generate one. ↩

4. This is the only PCIe port wired directly to the CPU (no router) and large enough >4G window ↩

5. Annoyingly long GPUs overlaps most SATA ports on this motherboard. If you use these ports, low profile cables is a requirement ↩

6. It is possible to do this in userspace with a systemd unit, but initramfs is cleaner beacuse it runs before drivers are loaded ↩