Resizable BAR on Supermicro H11SSL-i with Intel Arc B580

README.md

Resizable BAR on Supermicro H11SSL-i with Intel Arc B580

This might work on other mainboards and graphic adapters as well ¹, but this article assumes H11SSL-i and B580 running on a Linux operating system.

Tip

UEFI hacks are possible ² using ReBarUEFI, but this is more invasive and requires firmware modifications. In worst case it can brick your system and require recovery with a chip programmer. If you're running Windows then this is probably the only solution.

The Issue

The Supermicro H11SSL-i board does not have an option in the BIOS to enable "Resizable BAR" ("Resizable Base Address Register", or "ReBAR"), aka SAM ("Smart Access Memory") for AMD Platforms. It was not introduced until the H12 series boards ³.

Without ReBAR the kernel will report the following:

[    6.177256] xe 0000:08:00.0: [drm] Attempting to resize bar from 256MiB -> 16384MiB
[    6.177273] xe 0000:08:00.0: BAR 0 [mem 0xec000000-0xecffffff 64bit]: releasing
[    6.177284] xe 0000:08:00.0: BAR 2 [mem 0x1ffe0000000-0x1ffefffffff 64bit pref]: releasing
[    6.177462] xe 0000:08:00.0: BAR 2 [mem size 0x400000000 64bit pref]: can't assign; no space
[    6.177470] xe 0000:08:00.0: BAR 2 [mem size 0x400000000 64bit pref]: failed to assign
[    6.177480] xe 0000:08:00.0: BAR 0 [mem 0xec000000-0xecffffff 64bit]: assigned
[    6.177503] xe 0000:08:00.0: BAR 2 [mem size 0x400000000 64bit pref]: can't assign; no space
[    6.177511] xe 0000:08:00.0: BAR 2 [mem size 0x400000000 64bit pref]: failed to assign
[    6.177520] xe 0000:08:00.0: BAR 0 [mem 0xec000000-0xecffffff 64bit]: assigned
[    6.177658] xe 0000:08:00.0: [drm] Failed to resize BAR2 to 16384M (-ENOSPC). Consider enabling 'Resizable BAR' support in your BIOS
[    6.177671] xe 0000:08:00.0: BAR 2 [mem 0x1ffe0000000-0x1ffefffffff 64bit pref]: assigned
[    6.179509] xe 0000:08:00.0: [drm] VISIBLE VRAM: 0x000001ffe0000000, 0x0000000010000000
[    6.179571] xe 0000:08:00.0: [drm] Small BAR device
[    6.179577] xe 0000:08:00.0: [drm] VRAM[0]: Actual physical size 0x0000000300000000, usable size exclude stolen 0x00000002fb800000, CPU accessible size 0x0000000010000000
[    6.179587] xe 0000:08:00.0: [drm] VRAM[0]: DPA range: [0x0000000000000000-300000000], io range: [0x000001ffe0000000-1fff0000000]
[    6.179597] xe 0000:08:00.0: [drm] VRAM: 0x0000000300000000 is larger than resource 0x0000000010000000
[    6.179605] xe 0000:08:00.0: [drm] Small BAR device
[    6.179611] xe 0000:08:00.0: [drm] VRAM[0]: Actual physical size 0x0000000300000000, usable size exclude stolen 0x00000002fb800000, CPU accessible size 0x0000000010000000
[    6.179620] xe 0000:08:00.0: [drm] VRAM[0]: DPA range: [0x0000000000000000-300000000], io range: [0x000001ffe0000000-1fff0000000]

We can see a total VRAM size of 0x0000000300000000 (12 GiB), but CPU can only access 0x0000000010000000 (256 MiB) windows.

This is reflected when we inspect the PCI device memory regions:

        Region 0: Memory at ec000000 (64-bit, non-prefetchable) [size=16M]
        Region 2: Memory at 1ffe0000000 (64-bit, prefetchable) [size=256M]

Why this happens

Without ReBAR support the BIOS never selects the larger BAR sizes the card reports.

Since B580 defaults to 256MiB, the firmware builds the memory map based on this. The kernel tries to resize the BAR at boot, but there's not enough space in the window the firmware allotted -- and it won't redo this map because it would have to re-enumerate the PCIe tree, which would disrupt devices in use.

The workaround below selects 16GiB directly and re-enumerates the device which lets the kernel lay out the address space the firmware would have if it understood ReBAR.

Workaround

Turns out it's possible to apply ReBAR under certain conditions by overriding PCI registers. No invasive firmware modifications and easily reversible.

Note

This was written for Ubuntu⁴, but it can be adapted to any distro.

Prerequisites

• Latest revision (3.4) BIOS recommended ⁵
• Above 4G Decoding must be enabled in BIOS
• Linux Kernel version 6.12 or later with intel xe driver is required ⁴
• GPU must be in bottom x16 slot ⁶⁷

Step 1: Update bootloader

In /etc/default/grub enable the realloc flag to allow kernel to reorganize PCI address space:

GRUB_CMDLINE_LINUX_DEFAULT="pci=realloc"

Apply the changes:

sudo update-grub

Reboot and confirm this is applied using cat /proc/cmdline.

Step 2: Update initramfs

Next, a couple of initramfs⁸ scripts are required to perform the workaround when the system boots.

First create a hook:

sudo tee /etc/initramfs-tools/hooks/b580-rebar > /dev/null << 'EOF'
#!/bin/sh

# initramfs dependency check
PREREQ=""
prereqs() { echo "$PREREQ"; }
case "$1" in
  prereqs) prereqs; exit 0 ;;
esac

# Load api
. /usr/share/initramfs-tools/hook-functions

# Install binaries required for resize script
copy_exec /usr/bin/setpci /usr/bin
copy_exec /usr/bin/lspci  /usr/bin
EOF

Then the resize script:

sudo tee /etc/initramfs-tools/scripts/init-premount/b580-rebar > /dev/null << 'EOF'
#!/bin/sh

# initramfs dependency check
PREREQ=""
prereqs() { echo "$PREREQ"; }
case "$1" in
  prereqs) prereqs; exit 0 ;;
esac

# Find relevant PCI addresses from vendor device id lookups
GPU=$(lspci -D -d 8086:e20b | cut -d' ' -f1)
[ -z "$GPU" ] && exit 0
AUDIO=$(lspci -D -d 8086:e2f7 | cut -d' ' -f1)

# Already 16G? nothing to do
lspci -vv -s "$GPU" 2>/dev/null | grep -q 'size=16G' && exit 0

# Walk UP to the root port (parent is the root complex pci0000:NN)
dev=$(readlink -f /sys/bus/pci/devices/$GPU)
parent=$(dirname "$dev")
while :; do
  case "$(basename "$parent")" in
    pci0000:*) break ;;
  esac
  dev="$parent"; parent=$(dirname "$dev")
done
PORT=$(basename "$dev")

# Release whatever grabbed the card during coldplug
echo "$GPU" > /sys/bus/pci/drivers/xe/unbind 2>/dev/null || true
[ -n "$AUDIO" ] && echo "$AUDIO" > /sys/bus/pci/drivers/snd_hda_intel/unbind 2>/dev/null || true

# Resizable BAR control -> size field 14 (= 16G). ECAP_REBAR+0x08 is the control reg for the first resizable BAR
setpci -s "$GPU" COMMAND=0000
old=$(setpci -s "$GPU" ECAP_REBAR+0x08.l)
new=$(printf '%08x' $(( (0x$old & 0xFFFFE0FF) | (14 << 8) )))
setpci -s "$GPU" ECAP_REBAR+0x08.l=$new

# Re-lay the bridge window and re-enumerate; xe rebinds against the 16G BAR
echo 1 > /sys/bus/pci/devices/$PORT/remove
sleep 1
echo 1 > /sys/bus/pci/rescan

exit 0
EOF

Finalize and install:

sudo chmod +x /etc/initramfs-tools/hooks/b580-rebar
sudo chmod +x /etc/initramfs-tools/scripts/init-premount/b580-rebar
sudo update-initramfs -u

Testing

After a reboot sudo dmesg | grep -iE "xe.*(BAR|VRAM|resize)" should result in output that confirms the changes:

[    7.474451] xe 0000:08:00.0: [drm] VISIBLE VRAM: 0x0000019400000000, 0x0000000400000000
[    7.474666] xe 0000:08:00.0: [drm] VRAM[0]: Actual physical size 0x0000000300000000, usable size exclude stolen 0x00000002fb800000, CPU accessible size 0x00000002fb800000
[    7.474680] xe 0000:08:00.0: [drm] VRAM[0]: DPA range: [0x0000000000000000-300000000], io range: [0x0000019400000000-196fb800000]
[    7.474694] xe 0000:08:00.0: [drm] VRAM[0]: Actual physical size 0x0000000300000000, usable size exclude stolen 0x00000002fb800000, CPU accessible size 0x00000002fb800000
[    7.474705] xe 0000:08:00.0: [drm] VRAM[0]: DPA range: [0x0000000000000000-300000000], io range: [0x0000019400000000-196fb800000]

Here we can see that the CPU can access the entire VRAM: 0x00000002fb800000 (~11.93 GiB) excluding reserved firmware which would total to 0x0000000300000000(12 GiB)

Next, in lspci -vv -s 08:00.0 | grep -i region:

        Region 0: Memory at ea000000 (64-bit, non-prefetchable) [size=16M]
        Region 2: Memory at 19400000000 (64-bit, prefetchable) [size=16G]

Here we can see a large region of 0x0000000400000000 (16 GiB) which was exactly what the driver originally failed to resize to before the workaround!

Found this useful? Consider leaving a tip.

Footnotes

1. See comment section for other working combinations ↩

2. There are other successful attempts documented in the comments of List of working motherboards issue on ReBarUEFI's Github ↩

3. Officially stated by Supermicro in a FAQ article ↩

4. If you run Ubuntu 24.04 you need to install Ubuntu Hardware Enablement (HWE) Kernel using: sudo apt install --install-recommends linux-generic-hwe-24.04 ↩ ↩²

5. A Supermicro account is required to download BIOS images. If your BMC requires a product serial/key unlock to flash functionality, simply use supermicro-product-key to generate one. ↩

6. The slot furthest away from the CPU (CPU SLOT2 PCI-E 3.0 X16 in manual). This was the only slot where the workaround succeeded with my configuration -- it's wired directly to the CPU with enough >4G address headroom for the 16 GiB window ↩

7. Annoyingly, long GPUs overlaps most SATA ports on this motherboard using the lower slots. If you use these ports, low profile cables is a requirement ↩

8. It is possible to do this in userspace with a systemd unit, but initramfs is cleaner beacuse it runs before any software tries to use the driver ↩

Same situation, different board — hoping you can sanity-check whether your
approach applies.

Supermicro X9DRH-7TF (C602 / Xeon E5 v2, AMI BIOS 3.2, no ReBAR support),
Arc B580, kernel 7.1.8, xe driver. Identical error:
"Failed to resize BAR2 to 16384MiB (-ENOSPC)" -> Small BAR device, 256MB of 12GB.

Above 4G Decoding is enabled and working (BAR sits at 0x381fe0000000 inside a
64GB _CRS window with 63.5GB free). pci=realloc alone does nothing here — the
release walk goes 86:00.0 -> 85:01.0 -> 84:00.0 and then stops, so the root port
80:03.0's firmware-assigned 264MB prefetchable window is never released:

pcieport 0000:84:00.0: bridge window [mem size 0x400000000 64bit pref]: can't assign; no space

Before I try your setpci + remove/rescan approach: on H11SSL-i, did the
re-enumeration actually re-size the root port window, or was yours already
large enough that only the downstream bridges needed to grow? That's the part
I can't tell from the outside, and it's the difference between this working and
hitting the same wall.

Thanks for writing this up either way — it's the only documented case I found of
a B580 on a Supermicro board without ReBAR firmware.

Hi @RodBurlamaqui

With this workaround the address moved from 0x1ffe0000000 to 0x19400000000, so technically it was a reallocation I suppose. But the end result is that the root port got resized (not grow in-place).

hi @andersevenrud , yes thank you , and for your reply - i ended up writing a fix, well multiple, ended up with 3 fixes in total and well documented below ( i hope ). There is a .deb package ( to save time compiling ) fpr both for ubuntu/deb x64 that should cover automated install /fix.

https://github.com/RodBurlamaqui/Intel-ARC-Rebar
https://github.com/RodBurlamaqui/Intel-ARC-B580-vaapi-sigbus-fix

15. Credits
    andersevenrud — the setpci + remove/rescan method, in a gist for the H11SSL-i + B580. ( Intel-ARC-Rebar)
    Ilpo Järvinen and Lucas De Marchi (Intel) — the kernel-side analysis and the stalled quirk that confirmed the mechanism.
    The kernel's own log line, was not released (still contains assigned resources), which was the whole answer once someone read it.