Test servers

[Mike McGrath]

So we're in kind of an awkward but probably beneficial spot right now for
our test server infrastructure.

Its _ALL_ down right now.  And we need to build it back up.  We've talked
about another layer of test servers and I think its probably time for that
so here's what I propose.

Rebuild our test servers external to PHX.  There is an open ticket for
this already, we're just going to be doing it finally.

Create a staging environment in the publictest and test areas in PHX.

Staging will be identical to our production environment where possible
with the exception of production data and where new features are being
implemented and are not in production yet.

Staging will be in puppet.  All of it.  Just like production is and
they'll get rebuilt regularly.  We can use puppet's environments to do
this but I'm going to talk with kanarip about implementation and such.  We
actually have a very basic testing environment setup in puppet now but its
not really tested or documented yet.  Staging will be different.

What to know about the old pt servers:

1) AFAIK at this point in time, no data has been lost or anything.  The
old servers are just off.
2) PT10 was kind of a mess and needed to be rebuilt anyway.
3) If you need to recover something ask but we're not going to blanket
replace everyone's home dirs.  Hopefully you weren't developing on the
public test servers but were simply using it to show others.

This isn't all a done deal yet.  I want to see what others say since this
affects a lot of people, we have a lot of testers.

	-Mike