[Pulp-list] Parallel repo sync gets stuck

[ben.stanley at exemail.com.au]

I am having trouble synching my pulp repos.

I am using pulp 2.5.0-1 on RHEL 7.

I have configured 570 repos (I used a script). If I want to synchronise
them right now, I run this bash script:

----------- SyncAllRepos.sh --------------
AllRepoNames=( $(pulp-admin rpm repo list --fields=Id | sed --quiet -e
"s/Id: \+\([^ \t]\+\)/\1/p") )
for (( i=0; i<${#AllRepoNames[@]}; ++i ))
do
    pulp-admin rpm repo sync run --repo-id=${AllRepoNames[i]} --bg
done
--------------------------------------------------------
Prior to running this script, pulp-admin must be properly logged in:
pulp-admin login -u admin

Running this script seems to queue all the syncs ok. But, while
processing, pulp seems to get stuck. The bandwidth consumption is zero.
There are tasks in the Running state, but they don't seem to be doing
anything.

Stopping pulp and restarting it doesn't fix things.
systemctl stop pulp_resource_manager
systemctl stop pulp_celerybeat
systemctl stop pulp_workers

systemctl start pulp_workers
systemctl start pulp_celerybeat
systemctl start pulp_resource_manager

The result is that tasks that were marked as Running are changed to
Cancelled. Some other jobs that were marked Cancelled before the restart,
with Incomplete Finish Time, have their finish time filled in. All jobs
marked Waiting before the restart are cancelled at the restart. Some of
these have a finish time of Incomplete.

At this point no talks are marked as Running, and I try running my sync
script again. Things start out OK, but then it gets stuck again.

I don't know how to dig into the cause of the blockage. I do know that if
I change my SyncAllRepos.sh script and remove the --bg, that it no longer
gets stuck (but then it takes more than 1 day to do the sync).

Could I please have some help to track down what is getting stuck?