[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

Re: [Pulp-list] pulp v1 vs pulp v2 rpm repo sync times

On Mon, 25 Feb 2013, Jay Dobies wrote:
Touch base with Randy. He tweaked a bunch of those numbers before v2 released and should be able to point you to the best places to start playing with.

I had done some lab testing with the --num-threads parameter in
December. I had learned that our Grinder code was using threads for CPU
intensive work during downloads. Due to the Python GIL, this was
actually causing the threads to thrash each other, which significantly
lowered performance for synchronization.

For my test, I used traffic control to limit my bandwidth to 20 Mbps, 10
Mpbs, and 1 Mbps between myself and a LAN reachable CentOS repository,
and in all three cases I found that having one thread resulted in the
best performance. Due to this finding, I set the default number of
threads to 1. The --num-threads flag can be used to override the

One thing I did not simulate in my testing was network latency. If there
were high network latency, I would guess that adding more threads might
eventually lead to better performance, as more of them would be in a
waiting state instead of thrashing each other. I didn't much time to
simulate this scenario, so if you find that adding threads improves
performance, it might be due to latency. I'd like to know if that does
help, as it would warrant another test.


Randy Barlow

[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]