New tested approach: Build Nvidia driver for rt kernel with installed driver on non rt OS. (tested on July 19, 2024 on zorin17 (equivalent to Ubuntu23.10))
Important
One thing before start the below process is to make sure your non-rt kernel and installed rt-kernel has same or close major version, in my most recent test case, it is 6.5.0 for generic vs 6.6.40 for rt.
Step. 0: First, install the rt kernel but dont boot into the rt version OS right now. Install the Nvidia driver first in your non rt OS. When you installed your OS this has probably been done, but check if you got the nvidia-kernel-source, if not,