[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

Re: [Rdo-list] Quick rabbitmq question....



Totally fixed it. Thanks man. I learn something new about systemd every day....

On 11/19/15 7:55 PM, Jeff Weber wrote:
I struggled with this as well until I found out the limits.conf entries
don't apply to systemd managed services.

If you create a /etc/systemd/system/rabbitmq-server.service.d directory
and place a limits.conf file in there with contents similar to

[Service]
LimitNOFILE=4096

Then reload + restart

systemctl daemon-reload
systemctl restart rabbitmq-server

https://fedoraproject.org/wiki/Systemd#How_do_I_customize_a_unit_file.2F_add_a_custom_unit_file.3F



On Thu, Nov 19, 2015 at 10:43 PM, Erich Weiler <weiler soe ucsc edu
<mailto:weiler soe ucsc edu>> wrote:

    Hi Y'all,

    I'm sure someone has encountered this issue...  basically my
    rabbitmq instance on my controller node is running out of file
    descriptors, this is on RHEL 7.  I've upped the max file descriptors
    (nofile) to 1000000 in /etc/security/limits.conf, and my sysctl
    limit for file descriptors is equally huge.  Yet, I can't get my
    rabbitmq process to get it's limit's past 1000 or so:

    [root os-con-01 ~]# ps -afe | grep rabbit
    rabbitmq  4989     1  4 16:42 ?        00:07:10
    /usr/lib64/erlang/erts-5.10.4/bin/beam.smp -W w -K true -A30 -P
    1048576 -- -root /usr/lib64/erlang -progname erl -- -home
    /var/lib/rabbitmq -- -pa
    /usr/lib/rabbitmq/lib/rabbitmq_server-3.3.5/sbin/../ebin -noshell
    -noinput -s rabbit boot -sname rabbit os-con-01 -boot start_sasl
    -kernel inet_default_connect_options [{nodelay,true}] -sasl
    errlog_type error -sasl sasl_error_logger false -rabbit error_logger
    {file,"/var/log/rabbitmq/rabbit os-con-01 log"} -rabbit
    sasl_error_logger
    {file,"/var/log/rabbitmq/rabbit os-con-01-sasl log"} -rabbit
    enabled_plugins_file "/etc/rabbitmq/enabled_plugins" -rabbit
    plugins_dir
    "/usr/lib/rabbitmq/lib/rabbitmq_server-3.3.5/sbin/../plugins"
    -rabbit plugins_expand_dir
    "/var/lib/rabbitmq/mnesia/rabbit os-con-01-plugins-expand" -os_mon
    start_cpu_sup false -os_mon start_disksup false -os_mon start_memsup
    false -mnesia dir "/var/lib/rabbitmq/mnesia/rabbit os-con-01"
    -kernel inet_dist_listen_min 25672 -kernel inet_dist_listen_max 25672
    rabbitmq  5004     1  0 16:42 ?        00:00:00
    /usr/lib64/erlang/erts-5.10.4/bin/epmd -daemon
    rabbitmq  5129  4989  0 16:42 ?        00:00:00 inet_gethost 4
    rabbitmq  5130  5129  0 16:42 ?        00:00:00 inet_gethost 4
    root     17470 17403  0 19:34 pts/0    00:00:00 grep --color=auto rabbit

    [root os-con-01 ~]# cat /proc/4989/limits
    Limit                     Soft Limit           Hard Limit
      Units
    Max cpu time              unlimited            unlimited
    seconds
    Max file size             unlimited            unlimited
    bytes
    Max data size             unlimited            unlimited
    bytes
    Max stack size            8388608              unlimited
    bytes
    Max core file size        0                    unlimited
    bytes
    Max resident set          unlimited            unlimited
    bytes
    Max processes             127788               127788 processes
    Max open files            1024                 4096
      files
    Max locked memory         65536                65536
    bytes
    Max address space         unlimited            unlimited
    bytes
    Max file locks            unlimited            unlimited
    locks
    Max pending signals       127788               127788
      signals
    Max msgqueue size         819200               819200
      bytes
    Max nice priority         0                    0
    Max realtime priority     0                    0
    Max realtime timeout      unlimited            unlimited            us
    [root os-con-01 ~]#

    This is causing huge problems in my OpenStack cluster (Kilo
    Release). I've read that you can set this limit in
    /etc/rabbitmq/rabbitmq-env.conf or /etc/rabbitmq/rabbitmq.config but
    no matter what I do there I get nothing, after restarting rabbitmq
    many times.  Does this have something to do with systemd?

    [root os-con-01 ~]# rabbitmqctl status
    Status of node 'rabbit os-con-01' ...
    [{pid,4989},
      {running_applications,[{rabbit,"RabbitMQ","3.3.5"},
                             {os_mon,"CPO  CXC 138 46","2.2.14"},
                             {mnesia,"MNESIA  CXC 138 12","4.11"},
                             {xmerl,"XML parser","1.3.6"},
                             {sasl,"SASL  CXC 138 11","2.3.4"},
                             {stdlib,"ERTS  CXC 138 10","1.19.4"},
                             {kernel,"ERTS  CXC 138 10","2.16.4"}]},
      {os,{unix,linux}},
      {erlang_version,"Erlang R16B03-1 (erts-5.10.4) [source] [64-bit]
    [smp:32:32] [async-threads:30] [hipe] [kernel-poll:true]\n"},
      {memory,[{total,645523200},
               {connection_procs,32257624},
               {queue_procs,48513416},
               {plugins,0},
               {other_proc,15448376},
               {mnesia,1209984},
               {mgmt_db,0},
               {msg_index,292800},
               {other_ets,1991744},
               {binary,517865992},
               {code,16698259},
               {atom,602729},
               {other_system,10642276}]},
      {alarms,[]},
      {listeners,[{clustering,25672,"::"},{amqp,5672,"::"}]},
      {vm_memory_high_watermark,0.4},
      {vm_memory_limit,13406973132},
      {disk_free_limit,50000000},
      {disk_free,37354610688},
      {file_descriptors,[{total_limit,924}, <-----  ??????
                         {total_used,831},
                         {sockets_limit,829},
                         {sockets_used,829}]},
      {processes,[{limit,1048576},{used,8121}]},
      {run_queue,0},
      {uptime,10537}]
    ...done.

    Anyone know how to get the file descriptor limits up for rabbitmq?
    I've only got like 40 nodes in my OpenStack cluster, and it's
    choking, and I need to add several hundred more nodes...

    Any help much appreciated!!!  I looked around the list and couldn't
    find anything on this, and I've RTFM'd as much as I could...

    cheers,
    erich

    _______________________________________________
    Rdo-list mailing list
    Rdo-list redhat com <mailto:Rdo-list redhat com>
    https://www.redhat.com/mailman/listinfo/rdo-list

    To unsubscribe: rdo-list-unsubscribe redhat com
    <mailto:rdo-list-unsubscribe redhat com>




[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]