[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

Re: Restart TG apps for high mem-usage



Yaakov Nemoy wrote:
On Nov 25, 2007 4:00 PM, Toshio Kuratomi <a badger gmail com> wrote:
Here's a short script to test our TG apps run via supervisor for
excessive memory usage and restart them if necessary.  We could run this
via cron in alternate hours on each app server.  Does this seem like a
good or bad idea to people?

-Tosjio

+1, but does it make sure all transactions are finished?  I know smolt
does not have good transaction protection.  If a transaction fails
halfway through, we might have a mess.

Not if the app doesn't.  From a brief test, TG apps do not do this.

The script is asking supervisor to shutdown the application. supervisor sends a TERM to the TG app (we can configure it to send something other than TERM if we want but I don't see any documentation that leads me to believe it will be different with a HUP or QUIT). At that point it looks like a TG app will immediately shutdown and rollback any current transactions.

smolt is on shaky ground if it's not using transactions correctly... At the beginning of the month when smolt was getting hit hard we did pretty much this same thing except manually instead of via a script when we noticed that smolt was giving timeouts and taking up 1G+ of RAM. I think the current smolt code is using SQLAlchemy, correct? It's pretty easy to use transactions so that you don't leave the db in an inconsistent state with that configuration. Using the session's implicit transaction flushed just before the return should do the safe thing. You can look through the code later and find additional places where you can safely flush the transaction if there's a need.

-Toshio


[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]