[libvirt] [PATCH] Increase default file handle limits for daemons

[Guido Günther]

Hi,
On Wed, Mar 15, 2017 at 04:55:04PM +0000, Daniel P. Berrange wrote:
> Linux still defaults to a 1024 open file handle limit. This causes
> scalability problems for libvirtd / virtlockd / virtlogd on large
> hosts which might want > 1024 guest to be running. In fact if each
> guest needs > 1 FD, we can't even get to 500 guests. This is not
> good enough when we see machines with 100's of physical cores and
> TBs of RAM.
> 
> In comparison to other memory requirements of libvirtd & related
> daemons, the resource usage associated with open file handles
> is essentially line noise. It is thus reasonable to increase the
> limits unconditionally for all installs.
> 
> Signed-off-by: Daniel P. Berrange <berrange at redhat.com>
> ---
>  daemon/libvirtd.service.in       | 7 +++++--
>  src/locking/virtlockd.service.in | 4 ++++
>  src/logging/virtlogd.service.in  | 5 +++++
>  3 files changed, 14 insertions(+), 2 deletions(-)
> 
> diff --git a/daemon/libvirtd.service.in b/daemon/libvirtd.service.in
> index c72dde5..22fc156 100644
> --- a/daemon/libvirtd.service.in
> +++ b/daemon/libvirtd.service.in
> @@ -24,8 +24,11 @@ ExecStart=@sbindir@/libvirtd $LIBVIRTD_ARGS
>  ExecReload=/bin/kill -HUP $MAINPID
>  KillMode=process
>  Restart=on-failure
> -# Override the maximum number of opened files
> -#LimitNOFILE=2048
> +# At least 1 FD per guest, often 2 (eg qemu monitor + qemu agent).
> +# If we want to support 2048 guests, we'll typically need 4096 FDs

4096 FDs here…

> +# If changing this, also consider virtlogd.service & virtlockd.service
> +# limits which are also related to number of guests
> +LimitNOFILE=8192

…but 8192 here. So we're looking at 4096 rather than 2048 guests (2 fds per guest)?

>  
>  [Install]
>  WantedBy=multi-user.target
> diff --git a/src/locking/virtlockd.service.in b/src/locking/virtlockd.service.in
> index 69b568f..c369591 100644
> --- a/src/locking/virtlockd.service.in
> +++ b/src/locking/virtlockd.service.in
> @@ -13,6 +13,10 @@ ExecReload=/bin/kill -USR1 $MAINPID
>  # cause the machine to be fenced (rebooted), so make
>  # sure we discourage OOM killer
>  OOMScoreAdjust=-900
> +# Needs to allow for max guests * average disks per guest
> +# libvirtd.service written to expect 4096 guests, so if we
> +# allow for 4 disks per guest, we get:
> +LimitNOFILE=16384

Correct if we're looking at 4095 guests above.

>  
>  [Install]
>  Also=virtlockd.socket
> diff --git a/src/logging/virtlogd.service.in b/src/logging/virtlogd.service.in
> index 09e0740..be039c6 100644
> --- a/src/logging/virtlogd.service.in
> +++ b/src/logging/virtlogd.service.in
> @@ -13,6 +13,11 @@ ExecReload=/bin/kill -USR1 $MAINPID
>  # cause the machine to be fenced (rebooted), so make
>  # sure we discourage OOM killer
>  OOMScoreAdjust=-900
> +# Need to have at least one file open per guest (eg QEMU
> +# stdio log), but might be more (eg serial console logs)
> +# libvirtd.service written to expect 2048 guests, so if we

Rather 4096 as above?

> +# guess at 2 log files per guest here (stdio + 1 serial):
> +LimitNOFILE=8192
>  
>  [Install]
>  Also=virtlogd.socket

Cheers,
 -- Guido