[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

Re: [Pulp-list] MongoDB document size errors



On 08/02/2010 11:00 PM, Mike McCune wrote:
I spent some time today tracking down the exception that a few people were seeing:

InvalidDocument: document too large - BSON documents are limited to 4 MB

I wrote a unit test to create a repo and add 5,000 packages to the repository and easily hit the above error:

File "/home/mmccune/devel/pulp/test/unit/test_api.py", line 615, in test_sync_large_repo
    self.rapi.update(repo)
File "/home/mmccune/devel/pulp/test/unit/../../src/pulp/auditing.py", line 166, in _audit
    _record_event()
File "/home/mmccune/devel/pulp/test/unit/../../src/pulp/auditing.py", line 141, in _record_event
    _objdb.insert(event, safe=False, check_keys=False)
File "build/bdist.linux-x86_64/egg/pymongo/collection.py", line 245, in insert
    message.insert(self.__full_name, docs, check_keys, safe), safe)
InvalidDocument: document too large - BSON documents are limited to 4 MB

Note the trace is contained within the auditing insert() call.

Turns out the Event class has both a 'params' field and a field called 'action' but the action field also contained a copy of the params so for a large Repo document you get an Event document that is 2-3X the size of the actual parameter being audited.

Jason is working to trim this down but the short story is, we aren't hitting the 4MB limit because of the structure of our actual Repo document, we are hitting it because of the way we were auditing the parameters of a method call.

Jason also in his latest update took out the repo from the param logging in RepoApi.update()

    @audit(params=[])
    def update(self, repo_data):

Also, to make people feel more confortable, with the fix to the audit() call I was able to create a repo with 20,000 packages without error whereas before it would bail with a repo with 5,000.



Ah nice find Mike. Thanks for tracking this down.

~ Prad


[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]