Apache tweak

Mike McGrath mmcgrath at redhat.com
Sun May 27 23:53:17 UTC 2007


Karsten Wade wrote:
> On Sun, 2007-05-27 at 17:41 +0100, Damian Myerscough wrote:
>   
>> Yea, it would. However mod_easvie is able to detect if users a
>> continuously hitting refresh
>> to consume bandwidth.
>>     
>
> Would it make sense to have a few tricks ready to go?  Untested or
> unproven items to pull out in response to what happens.  Sort of like
> what Egon Spengler might pull out in Ghostbusters ...
>
> I know, it could be *worse* than whatever happens, but there is a slim
> chance we'll survive.
>   

Actually we've got lots of plans as it is.  I've been meaning to send 
this out but have been busy so here goes (all, comment and become 
familiar as necessary, the more people that know what is going on, the 
better)

This is only things related to the F7 launch:

Critical:
    1)  The static pages
    2)  The wiki
    3) Mirror manager
    4) Mirrors  (which should be taken care of and synced long before 
the actual bit flip)

Presently the static pages are fully redundant and the wiki and mirror 
manager are redundant up to their data layers.  (netapp and database 
respectively)

Plan A)
    We currently have 2 proxy servers and 4 application servers in 
place.  2 app servers are FC6, 2 are RHEL.  At present the two RHEL 
boxes load balance the wiki (moin), the two app servers are for mirror 
manager (TurboGears).  The static pages are on the actual proxy 
servers.  On release day we'll track the load trend on these 6 
machines.  As long as release related content is being served, we hold.

Plan B)
    If any one of the above pieces begins to fail we can add capacity as 
required as well as limiting traffic with iptables.  FC6 servers run 
mirror manager.  RHEL servers run the wiki (though the wiki could 
technically be run on FC6 boxes, it just hasn't been done yet, no 
reason) and the static pages are run directly on the proxy servers.  
Adding application servers is easy, just kick start the box, have it 
connect to puppet.  Proxy servers are trickier.  They exist on a 
different network, but the same principal applies.  We just contact the 
network guys to switch a port to the correct vlan.  I've got xen6 so it 
has one nic on the app network and one nic on the proxy network.

Plan C)
    Start redirecting all traffic to the mirrors (those that have our 
web content).  The theory is the most efficient transaction our web 
servers can do is to take a get and re-direct to a different server.  
The big 'gotcha' here is mirror manager.  Mirror manager won't know 
which mirrors have Fedora 7 until they have it.  So we'll have to serve 
the public server list at PHX.  We're featuring the torrents a bit more 
this time compared to last time.  Hopefully that will help keep load on 
the mirrors lighter, there's little way for us to track this AFAIK.

Unfortunately we don't have full trends from last year because a 
different team was running fedora.redhat.com during the last release 
(and f.rh.c doesn't even exist this release)  We'll get a much better 
idea of what we're facing on release day.

In general I'd like to encourage that we hold this configuration unless 
there's any simple upgrades or obvious flaws.  We're not really in an 
'infrastructure change freeze' but I'd encourage everyone needing to 
make changes to wait until a day or so after the release.  Exceptions, 
of course, would include something like bodhi which is of a high 
priority.  I'll be flying to Germany for Linux Tag tomorrow (Monday) and 
will remain there for the release.  I'll probably be on German time but 
will try to be around as needed or if any problems arise.  My cell phone 
may be limited though.

Comments?

    -Mike




More information about the Fedora-infrastructure-list mailing list