[libvirt] [PATCH 0/9] Resolve libvirtd hang on termination with connected long running client

Marc Hartmayer mhartmay at linux.vnet.ibm.com
Tue Jan 23 16:41:43 UTC 2018


On Tue, Jan 23, 2018 at 04:01 PM +0100, John Ferlan <jferlan at redhat.com> wrote:
> On 01/23/2018 04:21 AM, Marc Hartmayer wrote:
>> On Fri, Jan 19, 2018 at 06:23 PM +0100, John Ferlan <jferlan at redhat.com> wrote:
>>> RFC:
>>> https://www.redhat.com/archives/libvir-list/2018-January/msg00318.html
>>>
>>> Adjustments since RFC...
>>>
>>> Patches 1&2: No change, were already R-B'd
>>> Patch 3: Removed code as noted in code review, update commit message
>>> Patch 4: From old series removed, see below for more details
>>> Patch 9: no change
>>> NB: Patches 5-8 and 10 from Nikolay Shirokovskiy <nshirokovskiy at virtuozzo.com>
>>>     are removed as they seemed to not be necessary
>>>
>>> Replaced the former patch 4 with series of patches to (slowly) provide
>>> support to disable new connections, handle removing waiting jobs, causing
>>> the waiting workers to quit, and allow any running jobs to complete.
>>>
>>> As it turns out, waiting for running jobs to complete cannot be done
>>> from the virNetServerClose callbacks because that means the event loop
>>> processing done during virNetServerRun will not allow any currently
>>> long running worker job thread a means to complete.
>>>
>>> So when virNetDaemonQuit is called as a result of the libvirtd signal
>>> handlers for SIG{QUIT|INT|TERM}, instead of just causing virNetServerRun
>>> to quit immediately, alter to using a quitRequested flag and then use
>>> that quitRequested flag to check for long running worker threads before
>>> causing the event loop to quit resulting in libvirtd being able to run
>>> through the virNetDaemonClose processing.
>>
>> Gave a quick test:
>>  + Didn't get a segmentation fault at the end of libvirtd (at least for
>>    my quick test)
>>  - a single SIGTERM doesn’t always lead to the termination of libvirtd
>>    now (debugged it: main thread is waiting for poll()). This behavior
>>    can be easily reproduced: Start libvirtd on the CLI, wait some
>>    seconds for the first initialization -> CTRL + C -> libvirtd doesn’t
>>    terminate, but also doesn’t accept new connections.
>>
>
> Does this include the "update" in my response to patch 7?  It should be
> extract-able and apply-able.

No, I forgot this one :( I’ll try it tomorrow.

>
> John
>
>>>
>>> John Ferlan (9):
>>>   libvirtd: Alter refcnt processing for domain server objects
>>>   libvirtd: Alter refcnt processing for server program objects
>>>   netserver: Remove ServiceToggle during ServerDispose
>>>   util: Introduce virThreadPoolDrain
>>>   rpc: Introduce virNetServerQuitRequested
>>>   rpc: Introduce virNetServerWorkerCount
>>>   rpc: Alter virNetDaemonQuit processing
>>>   docs: Add news article for libvirtd issue
>>>   APPLY ONLY FOR TESTING PURPOSES
>>>
>>>  daemon/libvirtd.c        | 43 +++++++++++++++++++++++---------
>>>  docs/news.xml            | 12 +++++++++
>>>  src/libvirt_private.syms |  1 +
>>>  src/libvirt_remote.syms  |  2 ++
>>>  src/qemu/qemu_driver.c   |  5 ++++
>>>  src/rpc/virnetdaemon.c   | 45 +++++++++++++++++++++++++++++++++-
>>>  src/rpc/virnetserver.c   | 52 ++++++++++++++++++++++++++++++++++++---
>>>  src/rpc/virnetserver.h   |  4 +++
>>>  src/util/virthreadpool.c | 64 ++++++++++++++++++++++++++++++++++++++++--------
>>>  src/util/virthreadpool.h |  2 ++
>>>  10 files changed, 204 insertions(+), 26 deletions(-)
>>>
>>> --
>>> 2.13.6
>>>
>>> --
>>> libvir-list mailing list
>>> libvir-list at redhat.com
>>> https://www.redhat.com/mailman/listinfo/libvir-list
>>>
>> --
>> Beste Grüße / Kind regards
>>    Marc Hartmayer
>>
>> IBM Deutschland Research & Development GmbH
>> Vorsitzende des Aufsichtsrats: Martina Koederitz
>> Geschäftsführung: Dirk Wittkopp
>> Sitz der Gesellschaft: Böblingen
>> Registergericht: Amtsgericht Stuttgart, HRB 243294
>>
>
--
Beste Grüße / Kind regards
   Marc Hartmayer

IBM Deutschland Research & Development GmbH
Vorsitzende des Aufsichtsrats: Martina Koederitz
Geschäftsführung: Dirk Wittkopp
Sitz der Gesellschaft: Böblingen
Registergericht: Amtsgericht Stuttgart, HRB 243294





More information about the libvir-list mailing list