[Fedora-infrastructure-list] release days docs/instructions/faq

[Stacy J. Brandenburg]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Add HW.

You need a dedicated load balanced env if you want to ever run the
primary bits.  We have 5 Dell servers for PRX, 4 APP that fell over.  On
top of that the Load Balancers died one time due to load - and we were
not even serving pages, just redirects!

These are Cisco CSM blades that failed today.  Not small Load Balancers.

Probably need to define well, the roles people will play for
administration, so that you do not have a free for all.  Maybe even
shifts of people so Seth can take a shower or get food :)





seth vidal wrote:
> Hey guys,
>  So some of you have heard that the load from today was pretty close to
> crippling for a while. We got it under 'control' for a bit but it wasn't
> happy. Mike suggested making some of our pages in the wiki static so we
> pulled them out using firefox and then put them onto the site using
> apache redirects. That worked out okay to bring the load down. We're
> still get beaten pretty badly but at least the load is manageable.
> 
> Let's assume for now that the next N releases will be just like this.
> That the world will melt down. I think we should probably put together
> some docs on what steps to take. Here are some items we did today:
> 
> 1. iptables rate limiting on fpserv: 2-4 new connection per ip for every
> 10-20 seconds.
> 
> 2. look at the logs, figure out where all the hits are going and make
> those pages static. You can do this by using firefox to save the page
> (it grabs images and css, too)
> 
> 3. turn off any unnecessary bits: zope (if its not being used, rsyncd)
> 
> 4. trim out your log outputs. Decrease the number of things writing out
> to disk anywhere
> 
> 5. get comfortable with /etc/init.d/httpd restart - just to get the load
> back under control
> 
> 
> 
> What else should we be adding?
> 
> -sv
> 
> 
> _______________________________________________
> Fedora-infrastructure-list mailing list
> Fedora-infrastructure-list at redhat.com
> https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list

- --
========================================================
= Stacy J. Brandenburg                    Red Hat Inc. =
= Manager, Network Operations      sbranden at redhat.com =
= 919-754-4313                   http://www.redhat.com =
========================================================

Fingerprint
03F7 43BE 1150 CCFA F57B 54DD AEDB 1C27 1828 D94D

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.5 (GNU/Linux)
Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org

iD8DBQFFPoNnrtscJxgo2U0RAp6rAKDGX11u0J2onLKYfyXlsC1MUhjC/wCg1Tqp
Y/E2QQi7Lwv8NKm2lOktLk4=
=6+yT
-----END PGP SIGNATURE-----