Restart TG apps for high mem-usage

[Toshio Kuratomi]

Mike McGrath wrote:
> Bill Nottingham wrote:
>> Toshio Kuratomi (a.badger at gmail.com) said:  
>>> Here's a short script to test our TG apps run via supervisor for 
>>> excessive memory usage and restart them if necessary.  We could run 
>>> this via cron in alternate hours on each app server.  Does this seem 
>>> like a good or bad idea to people?
>>>     
>>
>> It's a good idea if it's needed, but it's a bad idea that it is 
>> needed. What's
>> wrong with TG that it leads to this situation?
>>   
> 
> I was wondering this myself, I know smolt recently had some major 
> changes to keep memory usage down.  Which TG apps are having this issue 
> and how often?  I know MM uses a lot of memory but, AFAIK, it was 
> determined that there's not much of a leak if there is one and that all 
> of that memory is actually used.
> 
Looks like smolt was upgraded just before Thanksgiving so it could be 
that we've plugged the leaks we had to deal with that inspired me to 
write this.  Would it be a good idea to have this in place anyways? 
With it periodically checking, we would find out that we had problems 
when cron emails us a notice that the script had to restart a process. 
Without it, we'll be notified when nagios or a user tells us they're 
getting timeouts.

I noticed that mirrormanager is currently at 761MB of RSS.  If that's 
steady-state for mm we'd want to bump the value the script checks for a 
bit higher before deploying it or set different values per app.

-Toshio