Not really. There seems to be a certain pattern that always the first query to a certain name server may time out. Are you sure that the v6 default route _really_ makes a difference, or is that just coincidence? Can you reproduce results by re-adding the default etc?
I seriously have not the slightest clue why v6 should make ANY difference, given that all I'm ssing from these outputs is pure IPv4 stuff. And it's not some antipathy of your ISP's DNS cache against AAAA RRs as (if I'm not mistaken) even A RR requests timed out.
Can you try installing a local BIND as local cache and use 127.0.0.1 as resolver in /etc/resolv.conf, and try that?