[Pulp-list] Bandwidth Limit pulp collectively over all syncs

[ben.stanley at exemail.com.au]

I have researched the bandwidth limiting problem a bit, and I have come up
with a solution that works for me.

Note that I only have one pulp server, and I access the internet through a
proxy server. This simplifies my problem considerably.

I use the following iptables commands to perform rate limiting:

# Insert the numeric address of your proxy server here
PROXY_IP=XXX.XXX.XXX.XXX
# insert the port used by your proxy server here
PROXY_PORT=8080
iptables --append INPUT --protocol tcp \
    --src $PROXY_IP --source-port $PROXY_PORT \
    --match hashlimit \
    --hashlimit-upto 200kb/sec \
    --hashlimit-mode srcip \
    --hashlimit-name proxy \
    --jump ACCEPT
iptables --append INPUT --protocol tcp \
    --src $PROXY_IP --source-port $PROXY_PORT \
    --jump REJECT

I found that the actual data rate appearing on the wire was higher than
the limit, due to the reject traffic. The reject traffic was up to 50% of
the accept traffic, so this method does not appear to be very efficient.

I also found that I had to limit the number of pulp workers
(/etc/defaults/pulp_workers, PULP_CONCURRENCY=4), or the number of
downloads caused the REJECT rate to overwhelm the ACCEPT rate, and the
wire traffic becomes too high. I monitored the traffic using nload -u H,
but note that this measures all the traffic on an interface.

I found that the Firefox extension "Down Them All" implements a rate
limiting feature that does what it says. It is simpler to do in a single
process. To implement useful bandwidth limiting in pulp, there must be
communication between workers requesting a bandwidth allocation, and
relinquishing it when finished.

Ben.