[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

Re: [Cluster-devel] cluster/logging settings



On Thu, 2008-11-06 at 19:42 -0700, Steven Dake wrote:
> On Tue, 2008-11-04 at 14:58 -0600, David Teigland wrote:
> > On Thu, Oct 30, 2008 at 11:26:14PM -0700, Steven Dake wrote:
> > > There are two types of messages.  Those intended for users/admins and
> > > those intended for developers.
> > > 
> > > Both of these message types should always be recorded *somewhere*.  The
> > > entire concept of "LOG_LEVEL_DEBUG" is dubious to me.  If you want to
> > > stick with that symanetic and definition that is fine, but really a
> > > LOG_LEVEL_DEBUG means "this message is for the developer".  These
> > > messages should be recorded and stored when a process segfaults, aborts
> > > due to assertion, or at administrative request.  Since the frequency of
> > > these messages is high there is no other option for recording them since
> > > they must _always_ be recorded for the purposes of debugging a field
> > > failure.  Recording to disk or syslog has significant performance
> > > impact.
> > > 
> > > The only solution for these types of messages is to record them into a
> > > flight recorder buffer which can be dumped:
> > > 1) at segv
> > > 2) at sigabt
> > > 3) at administrative request
> > > 
> > > This is a fundamental difference in how we have approached logging
> > > debugging messages in the past but will lead to the ability to ensure we
> > > _always_ have debug trace data available instead of telling the
> > > user/admin "Go turn on debug and hope you can reproduce that error and
> > > btw since 100000k messages are logged your disk will fill up with
> > > irrelevant debug messages and your system will perform like mud".
> > > 
> > > Logging these in memory is the only solution that I see as suitable and
> > > in all cases they should be filtered from any output source such as
> > > stderr, file, or syslog.
> > 
> > There's a difference between high volume trace debug data stored in
> > memory, and low volume informational debug data that can be easily written
> > to a file.  Both kinds of data can be useful.
> > 
> > My programs are simple enough that low volume informational debug data is
> > enough for me to identify and fix a problem.  So, low volume informational
> > data is all I produce.  It can be useful to write this data to a file.
> > 
> > Your program is complex enough that high volume trace debug data is
> > usually needed for you to identify and fix a problem.  So, high volume
> > trace data is all you produce.  This is too much data to write to a file
> > (by the running program).
> > 
> > So, we're using "DEBUG" to refer to different things.  We need to define
> > two different levels (just for clarity in this discussion):
> > . DEBUGLO is low volume informational data like I use
> > . DEBUGHI is high volume trace data like you use
> > 
> > DEBUGHI messages wouldn't ever be logged to files by the program while
> > running.  DEBUGLO messages could be, though, if the user configured it.
> > So, circling back around, how should a user configure DEBUGLO messages to
> > appear in syslog or a logfile?   In particular, what would they enter in
> > the cluster.conf <logging/> section?  My suggestion is:
> > 
> >   syslog_level=foo
> >   logfile_level=bar
> > 
> > where foo and bar are one of the standard priority names in syslog.h.
> > So, if a user wanted DEBUGLO messages to appear in daemon.log, they'd set
> > 
> >   logging/<daemon>/logfile_level=debug
> > 
> > and if they wanted DEBUGLO messages to appear in /var/log/messages,
> > 
> >   logging/<daemon>/syslog_level=debug
> > 
> > (Note that "debug" means DEBUGLO here because DEBUGHI messages are only
> > saved in memory, not to files by a running program.)
> > 
> > There's another separate question I have about corosync, and that's
> > whether you could identify some limited number of messages that would be
> > appropriate for DEBUGLO?  They would be used by non-experts to do some
> > rough debugging of problems, and by experts to narrow down a problem
> > before digging into the high volume trace data.  I'd suggest that a good
> > starting point for DEBUGLO would be the data that openais has historically
> > put in /var/log/messages.  Data that helps you quickly triage a problem
> > (or verify that things are happening correctly) without stepping through
> > all the trace data.
> > 
> 
> I thought about this a few days and this seems to make sense.  The
> priority (ie: specifically LOG_LEVEL_DEBUG messages) would be selectable
> per output medium (syslog, file, or stderr) and the high performance
> tracing would use the trace macros or log_rec and only ever go to
> memory.  Each output medium if configured would filter based upon a
> filtering priority which is LOG_LEVEL_DEBUG or others.
> 
> This does require further API changes that Fabio would have to manage
> with.

Ok good that we are heading towards the same goal.

Before you fully dive into this code, let's finalize the config options
for logging across the board. The API change at this point is no problem
since cluster.git didn't move to the logsys v2 API yet.

I'll send a summary of the different thread in another email to the
different mailing lists. Users might have different ideas as well.

Fabio


[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]