What you suggest is something that previously happened to me in Xen around two years ago. Xen 4.3 worked fine with PCI and VGA Passthrough, but when 4.4 was released, they somehow wrecked it. The Passthroughed Sound Card audio was horrendous, and the Video Card was a no-go - good thing is that magically everything was working again for Xen 4.5, but the real culprit was never found. Old Thread here:
After some experimentation, I found out by coincidence looking at older solved issues, that if instead of using xen-pciback as a Kernel Parameter to hide the Sound Card from the host (Dom0), I used xl pci-assignable-add to detach it from the host and attach it to xen-pciback after the host booted (Akin to vfio-pci if manually using modprobe), it worked properly. Whatever was the case, the workaround is still the same: Letting the host grab the Sound Card and initialize it with snd_hda_intel, then detaching it and attaching to xen-pciback/vfio-pci, instead of passing it fresh.
I didn't did a lot of experimentation with audio in QEMU-KVM-VFIO (Its in the TO DO list, after I debug why my Radeon 5770 doesn't works in a Q35 VM) with the emulated Intel HDA Sound Card, but I noticed that there are WAY TOO MANY ways to configure it, both host and guest side. In Arch Linux, I couldn't get QEMU to work with ALSA without producing crackling audio in a W10 VM, but I didn't tried that much since there are a lot of enviromental variables to be set and I only used the QEMU_AUDIO_DRV. However, installing Pulseaudio and telling QEMU to use it magically fixed everything with absolutely no configuration, and sound quality was pretty much the same as in Xen when passing through the Sound Card. What I recall is that with the the audio also breaks if I set the Intel HDA to use higher sample rates like 24-Bit 192 KHz instead of default 24 Bit 48 KHz (Which I think is also Pulseaudio and ALSA default), but it gives me the impression that host and guest needs to have a matching config.
I think that people are underestimating these quircks and config issues and instead blaming audio issues on latency or heavy CPU usage. After these experiences, I would point to quircks first. Sadly, I don't recall that anyone did a deep dive about how to configure the audio so ALSA/Pulseaudio and the Windoes guest match, so there is at least a standard test setup to use as reference before troubleshooting further.