[Pulp-list] Children resource usage

[Sean Waite]

By children, I'm referring to child nodes - the subservers that can sync
from a "parent" node.

Looking again at the resources, below is what I have. It does look like the
1.7g proc is actually a worker.

Some statistics on what I have here (resident memory):
2 celery__main__worker procs listed as "resource_manager"  - 41m memory each
2 celery__main__worker procs listed as "reserved_resource_worker" - 42m and
1.7g respectively
1 mongo process - 972m
1 celerybeat - 24m
a pile of httpd procs - 14m each
1 qpid -  21m

For disk utilization, the mongo db is around 3.8G and my directory
containing all of the rpms etc is around 95G.

We're on a system with only 3.5G available memory, which is probably part
of the problem. We're looking at expanding it, I'm just trying to figure
out how much to expand it by. From your numbers above, we'd need 6-7G of
memory + 2*N gigs for the workers. Should I expect maybe 3-4 workers at any
one time? I've got 2 now, but that is at an idle state.


On Mon, Jun 22, 2015 at 9:24 AM, Brian Bouterse <bbouters at redhat.com> wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
>
> Hi Sean,
>
> I'm not really sure what you mean by the term 'children'. Maybe you
> mean process or consumer?
>
> I expect pulp_resource_manager to use less than 1.7G of memory, but
> its possible. Memory analysis can be a little bit tricky so more
> details are needed about how this is being measured to be sure.
>
> The biggest memory process within Pulp by far is mongodb. If you can,
> ensure that at least 4G of RAM is available on that machine that you
> are running mongodb on.
>
> I looked into the docs and we don't talk much about the memory
> requirements. Feel free to file a bug on that if you want. Roughly I
> expect the following amounts of RAM to be available per process:
>
> pulp_celerybeat, 256MB - 512MB
> pulp_resource_manager, 256MB - 512MB
> pulp_workers. This process spawns N workers. Each worker could use
> 256MB - 2GB depending on what its doing.
> httpd, 1GB
> mongodb, 4GB
> qpidd/rabbitMQ, ???
>
> Note all the pulp_*, processes have a parent and child process, for
> the numbers above I consider each parent/child together. I usually
> show the inheritance using `sudo ps -awfux`.
>
> I'm interested to see what others think about these numbers too.
>
> - -Brian
>
>
> On 06/22/2015 08:46 AM, Sean Waite wrote:
> > Hi,
> >
> > I've got a pulp server running, and I'd like to add some children.
> > The server itself is a bit hard up on resources, so we're going to
> > rebuild with a larger one. How much resources would the children
> > use? Is it a fairly beefy process/memory hog?
> >
> > We've got a large number of repositories. pulp-resource-manager
> > seems to be using 1.7G of memory, with a .7G of mongodb.
> >
> > Any pointers on how much I might be able to expect?
> >
> > Thanks
> >
> > -- Sean Waite
> > swaite at tracelink.com <mailto:swaite at tracelink.com> Cloud Operations
> > Engineer                GPG 3071E870 TraceLink, Inc.
> >
> > Be Excellent to Each Other
> >
> >
> > _______________________________________________ Pulp-list mailing
> > list Pulp-list at redhat.com
> > https://www.redhat.com/mailman/listinfo/pulp-list
> >
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v2
>
> iQEcBAEBCAAGBQJViAx4AAoJEK48cdELyEfyjeUH/j06u2ERqrvTogSW+T3ZNYgI
> 4xnkypN6/oIv87BhaVysif1adYI4R/egiIHlqHGxO0HBWm/AKQygMWJBvMMK3Dlg
> PtGdDdD7BBnGEwuTeFm0qJMlofk3PKmRaPRrFhwFe6DD/UaYgM7FSVsVbyn4zZpf
> HSMSk+j77FoEH8ExUX4i43UJOjkp1vfFgyynKMwxIHi6vLY0VDnmIS3iISlfroIA
> T+ZmS5t2u2NBU3dgTSHNlQsWP4BT2JH8VRWatoVoMc/vwlIJv+fzYn+tMAjNwKu+
> Lepcowq7sXLQzmlqGgYpVMofcBy4Mv3V0z2tjZOzqySF7omIG5YK7uFDLISxu1g=
> =AhJm
> -----END PGP SIGNATURE-----
>
> _______________________________________________
> Pulp-list mailing list
> Pulp-list at redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-list
>



-- 
Sean Waite                                      swaite at tracelink.com
Cloud Operations Engineer                GPG 3071E870
TraceLink, Inc.

Be Excellent to Each Other
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/pulp-list/attachments/20150622/9f635b0a/attachment.htm>