Custom obs-backgroundremoval build with support for NVIDIA cuda 12 on linux

README.md

The below CMakeUserPreset.json can be used to successfully build and install obs-backgroundremoval with a version of the ONNXRUNTIME that supports current NVIDIA CUDA 12.

It makes use of changes introduced in my fork with commit 2c658d8a. EDIT: The PR has now been merged into master

Due to changes upstream with ONNXRUNTIME, the plugin now depends on libcudnn for CUDA which is provided by nvidia-cudnn.

To make use of this:

1. Clone from my fork on the feature/custom-onnxruntime branch. Clone locaal-ai/obs-backgroundremoval
2. Place the below CMakeUserPresets.json file at the projects root
3. Run the following:

   rm -rf build_x86_64 release \
     && cmake --preset cuda12 \
     && cmake --build --preset cuda12 \
     && cmake --build build_x86_64 -t package \
     && sudo apt install ./release/obs-backgroundremoval-1.2.0-x86_64-linux-gnu.deb

4. Profit

Update 9/15/2025: Changed debian package dependencies for more accurate cuda selection

CMakeUserPresets.json

{
  "version": 3,
  "cmakeMinimumRequired": {
    "major": 3,
    "minor": 28,
    "patch": 0
  },
  "configurePresets": [
    {
      "name": "cuda12",
      "displayName": "Linux x86_64 w/ cuda12",
      "description": "Build for Linux x86_64 against CUDA 12",
      "inherits": ["template"],
      "binaryDir": "${sourceDir}/build_x86_64",
      "condition": {
        "type": "equals",
        "lhs": "${hostSystemName}",
        "rhs": "Linux"
      },
      "generator": "Ninja",
      "warnings": {"dev": true, "deprecated": true},
      "cacheVariables": {
        "QT_VERSION": "6",
        "CMAKE_INSTALL_PREFIX": "/usr",
        "CMAKE_BUILD_TYPE": "Release",
        "CPACK_DEBIAN_PACKAGE_DEPENDS": "obs-studio, libcuda1, nvidia-cuda-toolkit (>= 12~) | cuda-toolkit-12-9 | cuda-toolkit-12-8 | cuda-toolkit-12-6 | cuda-toolkit-12-5 | cuda-toolkit-12-4 | cuda-toolkit-12-3, nvidia-cuda-toolkit (<< 13~) | cuda-toolkit-12-9 | cuda-toolkit-12-8 | cuda-toolkit-12-6 | cuda-toolkit-12-5 | cuda-toolkit-12-4 | cuda-toolkit-12-3, nvidia-cudnn | cudnn",
        "CUSTOM_ONNXRUNTIME_VERSION": "1.21.0",
        "CUSTOM_ONNXRUNTIME_URL": "https://github.com/microsoft/onnxruntime/releases/download/v1.21.0/onnxruntime-linux-x64-gpu-1.21.0.tgz",
        "CUSTOM_ONNXRUNTIME_HASH": "SHA256=ef37a33ba75e457aebfd0d7b342ab20424aa6126bc5f565d247f1201b66996cf"
      }
    }
  ],
  "buildPresets": [
    {
      "name": "cuda12",
      "configurePreset": "cuda12",
      "displayName": "Linux x86_64 w/ cuda12",
      "description": "Linux build for x86_64 against CUDA 12",
      "configuration": "Release"
    }
  ]
}

@mercster The CMAKE_INSTALL_PREFIX guess was just an off the cuff guess that is a common gotcha when mixing compiled plugin debs with packaged obs. Admittedly I was on the go while responding and missed that you weren't creating a deb. I'm not that familiar with the rpm side of things if you're creating one or are you just installing directly to the dirs?

Regardless, your OBS log file might be helpful in chasing down what's happening since it appears to be loading into OBS.

Also I thought the ONNX runtime was merged quite awhile ago

This CMakeUserPresets.json preset is primarily to ensure required CUDA libraries are installed and override the default ONNX with a newer version that works with CUDA 12. Back in April my PR was accepted to have the flags for custom ONNX to work, but the actual runtime versions in the repo are still the older ones that don't work with CUDA 12.

Happy to try and troubleshoot it further with you!

Managed to use this, but with small changes: "CPACK_DEBIAN_PACKAGE_DEPENDS": "obs-studio, nvidia-driver-580, cudnn9-cuda-12, nvidia-cuda-toolkit", on my Ubuntu 24.04 PC

Managed to use this, but with small changes: "CPACK_DEBIAN_PACKAGE_DEPENDS": "obs-studio, nvidia-driver-580, cudnn9-cuda-12, nvidia-cuda-toolkit", on my Ubuntu 24.04 PC

Hey @tomekjarosik thanks for that! Ubuntu packages their nvidia drivers in a non-debian way so that is a good catch.

Can you try building with:

"CPACK_DEBIAN_PACKAGE_DEPENDS": "obs-studio, libcuda1, (nvidia-cuda-toolkit (>= 12~) | cuda-toolkit-12), (nvidia-cuda-toolkit (<< 13~) | cuda-toolkit-12), (libcudnn9 | cudnn9-cuda-12)"

This in theory will gate the package to specifically require cuda 12 (which can be installed alongside 13). I had to reinstall 12 with the latest nvidia drivers from the developer repo to have the package work.

I believe switching the nvidia-driver requirement to libcuda1 should be platform agnostic

EDIT:

Actually I think cudnn can be based on cuda 13 and work just fine as long as cuda 12 is installed....

Try this:

"CPACK_DEBIAN_PACKAGE_DEPENDS": "obs-studio, libcuda1, (nvidia-cuda-toolkit (>= 12~) | cuda-toolkit-12), (nvidia-cuda-toolkit (<< 13~) | cuda-toolkit-12), nvidia-cudnn | cudnn"

Unfortunately, I got:

tomek@uextremepc:~/src/background-removal/obs-backgroundremoval$ sudo apt install ./release/obs-backgroundremoval-1.2.0-x86_64-linux-gnu.deb
Reading package lists... Error!
E: Problem parsing dependency 20 of obs-backgroundremoval:amd64=1.2.0
E: Error occurred while processing obs-backgroundremoval (NewVersion2)
E: Problem with MergeList /home/tomek/src/background-removal/obs-backgroundremoval/release/obs-backgroundremoval-1.2.0-x86_64-linux-gnu.deb
E: The package lists or status file could not be parsed or opened.

I tried some other variations and broke my plugin setup. Do you have any recommendation for best quality? So far I tried with GPU - CUDA + rvm_resnet50_fp32.onnx model + support with base 640x384 (https://github.com/tomekjarosik/obs-backgroundremoval/commits/dev/tjarosik/rvm50/). Not sure if I should aim for some other model + TensorRT

@tomekjarosik sorry about that, I had some extra parenthesis in there. I have now tested with the following which I tested on debian. The major change is that in order to specifically gate cuda to version 12 without forcing the removal of cuda 13 I also needed to specify the specific minor versions. What a mess!

"CPACK_DEBIAN_PACKAGE_DEPENDS": "obs-studio, libcuda1, nvidia-cuda-toolkit (>= 12~) | cuda-toolkit-12-9 | cuda-toolkit-12-8 | cuda-toolkit-12-6 | cuda-toolkit-12-5 | cuda-toolkit-12-4 | cuda-toolkit-12-3, nvidia-cuda-toolkit (<< 13~) | cuda-toolkit-12-9 | cuda-toolkit-12-8 | cuda-toolkit-12-6 | cuda-toolkit-12-5 | cuda-toolkit-12-4 | cuda-toolkit-12-3, nvidia-cudnn | cudnn"

I honestly haven't explored more onnx models beyond the v1.21.0 in the above config as it works solid for me!

I should note, I haven't tested if the nvidia-cuda-toolkit package will force a downgrade from cuda 13 but I don't think there is actually a cuda 13 version released for this package yet. So it should be a non-issue for now but might need to be adjusted in the future.