I don't think we should make it a hard physical block on PR merging. Setting aside the occasional infrastructure issues, we also have some unit tests (in pulp core, at least) that rely on e.g. non-expired certificates, and fixing those once they break would require circumventing the process or disabling the checks. Maybe that scenario is infrequent enough that we don't care, though - I can see that being the case.
I _do_ think we need to formalize a set of rules about merging, though, and decide how strict we want to be about it. There are a few possibilities:
Option 1: Nothing merges without passing PR runner tests, ever, even if the issue is rooted in the configuration or infrastructure of the test runners or an expired certificate etc. This would light a fire to get these issues resolved ASAP because nothing can happen without them.
Option 2: Every PR must pass all the unit tests, have a clean docs build and no PEP8 errors, but if the automated runners are not working correctly it is fine to just run those tests offline and merge if they pass.
Option 3: Every PR must pass all the unit tests, but if something is wrong with the automated docs or PEP8 runners, we disregard them until they are functional again and fix anything that got through in the interim.