[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

Re: Apache tweak

Karsten Wade wrote:
On Sun, 2007-05-27 at 17:41 +0100, Damian Myerscough wrote:
Yea, it would. However mod_easvie is able to detect if users a
continuously hitting refresh
to consume bandwidth.

Would it make sense to have a few tricks ready to go?  Untested or
unproven items to pull out in response to what happens.  Sort of like
what Egon Spengler might pull out in Ghostbusters ...

I know, it could be *worse* than whatever happens, but there is a slim
chance we'll survive.

Actually we've got lots of plans as it is. I've been meaning to send this out but have been busy so here goes (all, comment and become familiar as necessary, the more people that know what is going on, the better)

This is only things related to the F7 launch:

   1)  The static pages
   2)  The wiki
   3) Mirror manager
4) Mirrors (which should be taken care of and synced long before the actual bit flip)

Presently the static pages are fully redundant and the wiki and mirror manager are redundant up to their data layers. (netapp and database respectively)

Plan A)
We currently have 2 proxy servers and 4 application servers in place. 2 app servers are FC6, 2 are RHEL. At present the two RHEL boxes load balance the wiki (moin), the two app servers are for mirror manager (TurboGears). The static pages are on the actual proxy servers. On release day we'll track the load trend on these 6 machines. As long as release related content is being served, we hold.

Plan B)
If any one of the above pieces begins to fail we can add capacity as required as well as limiting traffic with iptables. FC6 servers run mirror manager. RHEL servers run the wiki (though the wiki could technically be run on FC6 boxes, it just hasn't been done yet, no reason) and the static pages are run directly on the proxy servers. Adding application servers is easy, just kick start the box, have it connect to puppet. Proxy servers are trickier. They exist on a different network, but the same principal applies. We just contact the network guys to switch a port to the correct vlan. I've got xen6 so it has one nic on the app network and one nic on the proxy network.

Plan C)
Start redirecting all traffic to the mirrors (those that have our web content). The theory is the most efficient transaction our web servers can do is to take a get and re-direct to a different server. The big 'gotcha' here is mirror manager. Mirror manager won't know which mirrors have Fedora 7 until they have it. So we'll have to serve the public server list at PHX. We're featuring the torrents a bit more this time compared to last time. Hopefully that will help keep load on the mirrors lighter, there's little way for us to track this AFAIK.

Unfortunately we don't have full trends from last year because a different team was running fedora.redhat.com during the last release (and f.rh.c doesn't even exist this release) We'll get a much better idea of what we're facing on release day.

In general I'd like to encourage that we hold this configuration unless there's any simple upgrades or obvious flaws. We're not really in an 'infrastructure change freeze' but I'd encourage everyone needing to make changes to wait until a day or so after the release. Exceptions, of course, would include something like bodhi which is of a high priority. I'll be flying to Germany for Linux Tag tomorrow (Monday) and will remain there for the release. I'll probably be on German time but will try to be around as needed or if any problems arise. My cell phone may be limited though.



[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]