[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

Re: [Pulp-list] Minor usability problem with stale cached login credentials

On 03/27/2012 08:39 PM, Nick Coghlan wrote:
On 03/26/2012 03:33 PM, Nick Coghlan wrote:
That sounds promising. For the moment I'm just using Build 0.267 with a
httpd based workaround for sync log access while a job is still running,
and I will probably stay with that approach for PulpDist 0.1.0, but
updating Pulp and replacing my current workaround with something more
robust based on set_progress() is very high on the todo list for 0.2.0.

It's taken a while, but I finally have PulpDist up to a point where I'm
running some real jobs through the plugins, and I discovered a major
flaw in my logging design: the full sync logs can simply get too big (on
the order of ~20 MB each when initialising a tree from scratch) for it
to be sensible to store them directly in the sync results. While this
was enough to cause a BSON serialisation failure in the pymongo DB
access layer, even if that problem could be eliminated, it would still
be a serious performance issue when it comes to retrieving the sync

For now, I'm going to remove the logs from the Pulp data store entirely,
and consider my options for the next version of PulpDist. The two main
possibilities are:
- continue to refine my current workaround (where sync log files are
written directly to the filesystem and published independently of Pulp)
- revert to a design I considered earlier, where I define a custom
"sync_log" content type and store the logs in the repo itself

Would this work? If you hit a DB failure storing it in the report, I'm not sure it'd necessarily fare any better as its own content type. But this is an area I'm not hugely familiar with in Mongo. I've worked with high amounts of fields, but never large data in a single field. If there's some sort of extra DB configuration you find to support it better and you think it makes sense to be part of the type definition let me know.

The latter has the nice advantage of keeping the server side of things
fully contained within Pulp rather than relying on additional web server
configuration to publish the sync logs directly over http(s).


Jay Dobies
Freenode: jdob @ #pulp
http://pulpproject.org | http://blog.pulpproject.org

[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]