Might be working now, oddly I thought I used the format modules-load= and no luck but it is at the moment. Only issue, when I rebooted I had to reset my dual monitor config. Any issues with kernel modules listing other drivers, or is the only thing that matters the kernel driver in use?
02:00.0 VGA compatible controller : NVIDIA Corporation GF110 [GeForce GTX 560 Ti 448 Cores] [10de:1087] (rev a1)
Subsystem: NVIDIA Corporation Device [10de:0000]
Kernel driver in use: vfio-pci
Kernel modules: nouveau, nvidia_drm, nvidia