Somebody recently nudged me into the direction of NVIDIAs Bluefield 2 cards being a weird but fun system to play around with, so I did exactly that and I wanted to get rid of the vendor kernel and instead use a modern(ish) distro with a mainline kernel. Funnily enough that wasn't actually too hard.
In my case I got some DELL cards, part number 0JNDCM. They have 32GB of memory and 128GB of EMMC storage. Unfortunately the EMMC is highly redundant, which means that only about 40GB of storage is actually usable, and it's definitively not the fastest.