This completely solves the problem and all is well doing passthrough on my Skylake system.Since efifb can't be compiled as a module, and I'd rather not turn it off, here's what I did:Thanks!That was exactly what was going wrong. efifb had attached to some of the nvidia device's memory.
echo 0 > /sys/class/vtconsole/vtcon0/bind
echo 0 > /sys/class/vtconsole/vtcon1/bind
echo efi-framebuffer.0 > /sys/bus/platform/drivers/efi-framebuffer/unbindHopefully now that there's a solution with the right magic words in it on the Internet, others will find their answer here.