[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

Re: [Pulp-list] MongoDB document size errors



On Mon, 2010-08-02 at 20:00 -0700, Mike McCune wrote:
> I spent some time today tracking down the exception that a few people 
> were seeing:
> 
>    InvalidDocument: document too large - BSON documents are limited to 4 MB
> 
> I wrote a unit test to create a repo and add 5,000 packages to the 
> repository and easily hit the above error:
> 
> File "/home/mmccune/devel/pulp/test/unit/test_api.py", line 615, in 
> test_sync_large_repo
>      self.rapi.update(repo)
>    File "/home/mmccune/devel/pulp/test/unit/../../src/pulp/auditing.py", 
> line 166, in _audit
>      _record_event()
>    File "/home/mmccune/devel/pulp/test/unit/../../src/pulp/auditing.py", 
> line 141, in _record_event
>      _objdb.insert(event, safe=False, check_keys=False)
>    File "build/bdist.linux-x86_64/egg/pymongo/collection.py", line 245, 
> in insert
>      message.insert(self.__full_name, docs, check_keys, safe), safe)
> InvalidDocument: document too large - BSON documents are limited to 4 MB
> 
> Note the trace is contained within the auditing insert() call.
> 
> Turns out the Event class has both a 'params' field and a field called 
> 'action' but the action field also contained a copy of the params so for 
> a large Repo document you get an Event document that is 2-3X the size of 
> the actual parameter being audited.
> 
> Jason is working to trim this down but the short story is, we aren't 
> hitting the 4MB limit because of the structure of our actual Repo 
> document, we are hitting it because of the way we were auditing the 
> parameters of a method call.
> 
> Jason also in his latest update took out the repo from the param logging 
> in RepoApi.update()
> 
>      @audit(params=[])
>      def update(self, repo_data):
> 
> Also, to make people feel more confortable, with the fix to the audit() 
> call I was able to create a repo with 20,000 packages without error 
> whereas before it would bail with a repo with 5,000.
> 
> 

I believe (read 'hope') that the fix is in...

-- 
Jason L Connor
Senior Software Engineer
Systems Management and Cloud Enablement
Red Hat, Inc.
+1.919.890.8331
Freenode: linear

Attachment: signature.asc
Description: This is a digitally signed message part


[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]