[sos-devel] addCopySpecLimit

[Bryn M. Reeves]

On 03/13/2012 04:10 PM, Jesse Jaggars wrote:
> There is a plugin method called addCopySpecLimit with the following
> signature:
> 
> def addCopySpecLimit(self, fname, sizelimit=None, sub=None)
> 
> It's a fairly complicated function, as far as I can tell it does about
> three things:

Right; it's not such a bad interface but the implementation is lacking
at the moment. The intent is clear: copy files matching the copySpec up
to the limit but as you noticed things aren't quite written that way today.

> 1. If fname is a single file it will attempt to copy the file.
>   - if the file is under sizelimit it adds it via addCopySpec
>   - if the file is over sizelimit it tails the last sizelimit bytes and
>     writes that as an extCommand (via shelling out to tail)
> 
> 2. If fname is a glob it will attempt to copy (via addCopySpec) all the
> files, stopping before it gets to sizelimit bytes total.
> 
> 3. If fname is a glob, but the first file it tries to copy is larger
> than sizelimit bytes then nothing is added via addCopySpec.
> 
> There may be some problems with the above (especially #3) so I wanted to
> start a thread on what the expectations for this function are and what
> folks think this thing ought to do.

I think it's also worth considering whether addCopySpecLimit() is the
best way to approach this. It complicates things even more but we would
get more robust behaviour if we kept track of how much data we've
accumulated and checked against a limit (either global or per copySpec).

> Here are my ideas:
> 
> * control the sorting
> 
> Currently files found via glob.glob are sorted alphabetically, which is
> probably not what you want in some cases. It would be nice to be able to
> sort the files however you wanted, or at least some parameter of
> os.stat.

There are some complicated-ish rules to think about if we do something
like this. For e.g. determining what is the "best" set of data to
capture if we can't get everything. For log files it's not too hard -
always get the latest (and they are the only current user of limits).

> * trim the first file in a glob if it is too big
> 
> Rather than copying nothing, it might be better to try to copy the first
> file (post sort) according to the 'only one file passed in rule'.

The current code should attempt this:

        # Truncate the first file (others would likely be
        # compressed),
        # ensuring we get at least some logs
        if flog == files[0] and limit_reached:
            self.collectExtOutput("tail -c%d %s" % (sizelimit, flog),
                "tail_" + os.path.basename(flog))

It works OK in my testing.

Regards,
Bryn.