[Date Prev][Date Next] [Thread Prev][Thread Next] [Thread Index] [Date Index] [Author Index]

*From*: "Daniel P. Berrange" <berrange redhat com>*To*: Daniel Veillard <veillard redhat com>*Cc*: libvir-list redhat com*Subject*: Re: [libvirt] PATCH: 0/28: Thread safety for libvirtd daemon and drivers*Date*: Wed, 3 Dec 2008 19:44:02 +0000

On Wed, Dec 03, 2008 at 08:20:36PM +0100, Daniel Veillard wrote: > On Wed, Dec 03, 2008 at 06:52:32PM +0000, Daniel P. Berrange wrote: > > On Wed, Dec 03, 2008 at 04:18:21PM +0100, Daniel Veillard wrote: > > > On Mon, Dec 01, 2008 at 12:26:48AM +0000, Daniel P. Berrange wrote: > > > > This is a diffstat summary for the combined series of 28 patches > > > > > > Okay, my take at this point is that those should be commited with > > > the few fix found my manual examination, maybe extend the documentation > > > a bit, and start testing it as much as prossible. > > > Some locking debug facility might be a good addition, > > > > I wrote some an OCaml program using CIL to check driver method exit paths > > and validate that all objects were left in an unlocked state. This found > > some real bugs ! > > > > So here's the incremental fixes for those > [...] > > Damn, I missed them :-) > > Is the CIL code too specific or too limited to share ? I must admit I > looked for a bit on the CIL website and Rich details from his previous > work on it: > http://et.redhat.com/~rjones/cil-analysis-of-libvirt/ > but the Ocaml language barrier somehow blocked me. I started off from Rich's example, and then bashed my head again a brick for 2 days and the attached file is the result :-) Basically CIL is able to parse the GCC intermediate files - the .i files generated from -save-temps. It can compute the control flow information - eg various paths from if/while/switch statements. There's a handy module called 'DataFlow' which traverses the graph and invokes an instance of what they call the 'ForwardsTransfer' module interface. This is what you implement todo your analysis on the code. I've tried to stuff a bunch of comments in my code to explain what I'm doing. The key thing is that I'm not attempting to solve the general problem of checking thread safety in arbitrary code. That would justbe insane & impossible. Instead I'm only trying to analyse the internal driver API entry points for each driver we have and so far just checking everything is unlocked upon exit. Its fairly easy to extend to track other rules we define. eg validate that you lock the 'qemud_driver *' object before locking the virDomainObjPtr instance. These libvirt specific rules I apply are what makes the analysis even remotely practical. I'll document this more fully & try and figure out how we can integrate it into our build system, but here's my quick notes on how I use it - Edit gnulib/lib/c-ctype.h, gnulib/lib/c-ctype.c, src/xmlrpc.c src/xmlrpc.h, src/cgroup.c and replace all references to the datatype 'bool' with 'char' CIL wants the compiler temporary files (eg -save-tmps) but thse include 'bool' data types, and it doesn't know how to handle them so I have to remove them by editing the source. - Run 'make distclean' and 'find -name *.i | xargs rm' - Run autogen.sh as normal - Run make CFLAGS="-save-temps -g -O0" - Install 'ocaml-devel', 'ocaml-cil-devel', 'ocaml-findlib' RPMS - Edit cilvirtflow.ml and find where it says '/home/berrange/....' and make that point to your libvirt directory. - Compile the program ocamlfind ocamlopt -package unix,str,cil -c cilvirtflow.ml ocamlfind ocamlopt -package unix,str,cil cilvirtflow.cmx -linkpkg -o cilvirtflow - Run the analysis ./cilvirtflow If all went to plan it should print out a huge report along the lines of this - this is without my fixes from a few minutes ago - showing the kind of info it gives for identifying bugs Function: qemuDriverLock - Total exit points with locked vars: 0 Function: qemuDriverUnlock - Total exit points with locked vars: 0 Function: qemudSetCloseExec - Total exit points with locked vars: 0 Function: qemudSetNonBlock - Total exit points with locked vars: 0 Function: qemudAutostartConfigs - Total exit points with locked vars: 0 Function: qemudStartup - Total exit points with locked vars: 0 Function: qemudNotifyLoadDomain - Total exit points with locked vars: 0 Function: qemudReload - Total exit points with locked vars: 0 Function: qemudActive - Total exit points with locked vars: 0 Function: qemudShutdown - Total exit points with locked vars: 0 Function: qemudReadMonitorOutput - Total exit points with locked vars: 0 Function: qemudCheckMonitorPrompt - Total exit points with locked vars: 0 Function: qemudOpenMonitor - Total exit points with locked vars: 0 Function: qemudExtractMonitorPath - Total exit points with locked vars: 0 Function: qemudFindCharDevicePTYs - Total exit points with locked vars: 0 Function: qemudWaitForMonitor - Total exit points with locked vars: 0 Function: qemudDetectVcpuPIDs - Total exit points with locked vars: 0 Function: qemudInitCpus - Total exit points with locked vars: 0 Function: qemudNextFreeVNCPort - Total exit points with locked vars: 0 Function: qemudStartVMDaemon - Total exit points with locked vars: 0 Function: qemudVMData - Total exit points with locked vars: 0 Function: qemudShutdownVMDaemon - Total exit points with locked vars: 0 Function: qemudDispatchVMEvent - Total exit points with locked vars: 0 Function: qemudMonitorCommand - Total exit points with locked vars: 0 Function: qemudProbe - Total exit points with locked vars: 0 Function: qemudOpen - Total exit points with locked vars: 0 Function: qemudClose - Total exit points with locked vars: 0 Function: qemudSupportsFeature - Total exit points with locked vars: 0 Function: qemudGetType - Total exit points with locked vars: 0 Function: qemudGetMaxVCPUs - Total exit points with locked vars: 0 Function: qemudGetNodeInfo - Total exit points with locked vars: 0 Function: qemudGetCapabilities - Total exit points with locked vars: 0 Function: qemudNodeGetCellsFreeMemory - Total exit points with locked vars: 0 Function: qemudNodeGetFreeMemory - Total exit points with locked vars: 0 Function: qemudGetProcessInfo - Total exit points with locked vars: 0 Function: qemudDomainLookupByID - Total exit points with locked vars: 0 Function: qemudDomainLookupByUUID - Total exit points with locked vars: 0 Function: qemudDomainLookupByName - Total exit points with locked vars: 0 Function: qemudGetVersion - Total exit points with locked vars: 0 Function: qemudGetHostname - Total exit points with locked vars: 0 Function: qemudListDomains - Total exit points with locked vars: 0 Function: qemudNumDomains - Total exit points with locked vars: 0 Function: qemudDomainCreate - Total exit points with locked vars: 0 Function: qemudDomainSuspend - Total exit points with locked vars: 0 Function: qemudDomainResume - Total exit points with locked vars: 0 Function: qemudDomainShutdown - Total exit points with locked vars: 0 Function: qemudDomainDestroy - Total exit points with locked vars: 0 Function: qemudDomainGetOSType - Total exit points with locked vars: 0 Function: qemudDomainGetMaxMemory - Total exit points with locked vars: 0 Function: qemudDomainSetMaxMemory - Total exit points with locked vars: 0 Function: qemudDomainSetMemory - Total exit points with locked vars: 0 Function: qemudDomainGetInfo - Total exit points with locked vars: 0 Function: qemudEscape - Total exit points with locked vars: 0 Function: qemudEscapeMonitorArg - Total exit points with locked vars: 0 Function: qemudEscapeShellArg - Total exit points with locked vars: 0 Function: qemudDomainSave - Total exit points with locked vars: 1 - Locked vars at exit #line 2131 "qemu_driver.c" return (-1); ^^^^^^^^^ >>>>Drivers < Var: driver 14905 >>>>Objects < Var: vm 14906 Function: qemudDomainSetVcpus - Total exit points with locked vars: 0 Function: qemudDomainPinVcpu - Total exit points with locked vars: 0 Function: qemudDomainGetVcpus - Total exit points with locked vars: 0 Function: qemudDomainGetMaxVcpus - Total exit points with locked vars: 0 Function: qemudDomainRestore - Total exit points with locked vars: 0 Function: qemudDomainDumpXML - Total exit points with locked vars: 0 Function: qemudListDefinedDomains - Total exit points with locked vars: 0 Function: qemudNumDefinedDomains - Total exit points with locked vars: 0 Function: qemudDomainStart - Total exit points with locked vars: 0 Function: qemudDomainDefine - Total exit points with locked vars: 0 Function: qemudDomainUndefine - Total exit points with locked vars: 0 Function: qemudDiskDeviceName - Total exit points with locked vars: 0 Function: qemudDomainChangeEjectableMedia - Total exit points with locked vars: 0 Function: qemudDomainAttachPciDiskDevice - Total exit points with locked vars: 0 Function: qemudDomainAttachUsbMassstorageDevice - Total exit points with locked vars: 0 Function: qemudDomainAttachHostDevice - Total exit points with locked vars: 0 Function: qemudDomainAttachDevice - Total exit points with locked vars: 0 Function: qemudDomainDetachPciDiskDevice - Total exit points with locked vars: 0 Function: qemudDomainDetachDevice - Total exit points with locked vars: 0 Function: qemudDomainGetAutostart - Total exit points with locked vars: 0 Function: qemudDomainSetAutostart - Total exit points with locked vars: 0 Function: qemudDomainBlockStats - Total exit points with locked vars: 0 Function: qemudDomainInterfaceStats - Total exit points with locked vars: 0 Function: qemudDomainBlockPeek - Total exit points with locked vars: 0 Function: qemudDomainMemoryPeek - Total exit points with locked vars: 0 Function: qemudDomainEventRegister - Total exit points with locked vars: 0 Function: qemudDomainEventDeregister - Total exit points with locked vars: 0 Function: qemuDomainEventDispatchFunc - Total exit points with locked vars: 1 - Locked vars at exit #line 3777 return; ^^^^^^^^^ >>>>Drivers < Var: driver 16606 >>>>Objects < Function: qemuDomainEventFlush - Total exit points with locked vars: 0 Function: qemuDomainEventQueue - Total exit points with locked vars: 0 Function: qemudDomainMigratePrepare2 - Total exit points with locked vars: 0 Function: qemudDomainMigratePerform - Total exit points with locked vars: 0 Function: qemudDomainMigrateFinish2 - Total exit points with locked vars: 0 Function: qemuRegister - Total exit points with locked vars: 0 Function: umlDriverLock - Total exit points with locked vars: 0 Function: umlDriverUnlock - Total exit points with locked vars: 0 Function: umlSetCloseExec - Total exit points with locked vars: 0 Function: umlAutostartConfigs - Total exit points with locked vars: 0 Function: umlIdentifyOneChrPTY - Total exit points with locked vars: 0 Function: umlIdentifyChrPTY - Total exit points with locked vars: 0 Function: umlInotifyEvent - Total exit points with locked vars: 3 - Locked vars at exit #line 232 "uml_driver.c" return; ^^^^^^^^^ >>>>Drivers < >>>>Objects < Var: dom 30814 - Locked vars at exit #line 239 return; ^^^^^^^^^ >>>>Drivers < >>>>Objects < Var: dom 30814 - Locked vars at exit #line 283 return; ^^^^^^^^^ >>>>Drivers < >>>>Objects < Var: dom 30814 Function: umlStartup - Total exit points with locked vars: 0 Function: umlReload - Total exit points with locked vars: 0 Function: umlActive - Total exit points with locked vars: 0 Function: umlShutdown - Total exit points with locked vars: 0 Function: umlReadPidFile - Total exit points with locked vars: 0 Function: umlMonitorAddress - Total exit points with locked vars: 0 Function: umlOpenMonitor - Total exit points with locked vars: 0 Function: umlMonitorCommand - Total exit points with locked vars: 0 Function: umlStartVMDaemon - Total exit points with locked vars: 0 Function: umlShutdownVMDaemon - Total exit points with locked vars: 0 Function: umlOpen - Total exit points with locked vars: 0 Function: umlClose - Total exit points with locked vars: 0 Function: umlGetType - Total exit points with locked vars: 0 Function: umlGetNodeInfo - Total exit points with locked vars: 0 Function: umlGetCapabilities - Total exit points with locked vars: 0 Function: umlNodeGetCellsFreeMemory - Total exit points with locked vars: 0 Function: umlNodeGetFreeMemory - Total exit points with locked vars: 0 Function: umlGetProcessInfo - Total exit points with locked vars: 0 Function: umlDomainLookupByID - Total exit points with locked vars: 0 Function: umlDomainLookupByUUID - Total exit points with locked vars: 0 Function: umlDomainLookupByName - Total exit points with locked vars: 0 Function: umlGetVersion - Total exit points with locked vars: 0 Function: umlGetHostname - Total exit points with locked vars: 0 Function: umlListDomains - Total exit points with locked vars: 0 Function: umlNumDomains - Total exit points with locked vars: 0 Function: umlDomainCreate - Total exit points with locked vars: 0 Function: umlDomainShutdown - Total exit points with locked vars: 0 Function: umlDomainDestroy - Total exit points with locked vars: 0 Function: umlDomainGetOSType - Total exit points with locked vars: 0 Function: umlDomainGetMaxMemory - Total exit points with locked vars: 0 Function: umlDomainSetMaxMemory - Total exit points with locked vars: 0 Function: umlDomainSetMemory - Total exit points with locked vars: 0 Function: umlDomainGetInfo - Total exit points with locked vars: 0 Function: umlDomainDumpXML - Total exit points with locked vars: 0 Function: umlListDefinedDomains - Total exit points with locked vars: 0 Function: umlNumDefinedDomains - Total exit points with locked vars: 0 Function: umlDomainStart - Total exit points with locked vars: 0 Function: umlDomainDefine - Total exit points with locked vars: 0 Function: umlDomainUndefine - Total exit points with locked vars: 0 Function: umlDomainGetAutostart - Total exit points with locked vars: 0 Function: umlDomainSetAutostart - Total exit points with locked vars: 0 Function: umlDomainBlockPeek - Total exit points with locked vars: 0 Function: umlRegister - Total exit points with locked vars: 0 Function: lxcDriverLock - Total exit points with locked vars: 0 Function: lxcDriverUnlock - Total exit points with locked vars: 0 Function: lxcProbe - Total exit points with locked vars: 0 Function: lxcOpen - Total exit points with locked vars: 0 Function: lxcClose - Total exit points with locked vars: 0 Function: lxcDomainLookupByID - Total exit points with locked vars: 0 Function: lxcDomainLookupByUUID - Total exit points with locked vars: 0 Function: lxcDomainLookupByName - Total exit points with locked vars: 0 Function: lxcListDomains - Total exit points with locked vars: 0 Function: lxcNumDomains - Total exit points with locked vars: 0 Function: lxcListDefinedDomains - Total exit points with locked vars: 0 Function: lxcNumDefinedDomains - Total exit points with locked vars: 0 Function: lxcDomainDefine - Total exit points with locked vars: 0 Function: lxcDomainUndefine - Total exit points with locked vars: 0 Function: lxcDomainGetInfo - Total exit points with locked vars: 0 Function: lxcGetOSType - Total exit points with locked vars: 0 Function: lxcDomainDumpXML - Total exit points with locked vars: 0 Function: lxcVMCleanup - Total exit points with locked vars: 0 Function: lxcSetupInterfaces - Total exit points with locked vars: 0 Function: lxcMonitorClient - Total exit points with locked vars: 0 Function: lxcVmTerminate - Total exit points with locked vars: 0 Function: lxcMonitorEvent - Total exit points with locked vars: 0 Function: lxcControllerStart - Total exit points with locked vars: 0 Function: lxcVmStart - Total exit points with locked vars: 0 Function: lxcDomainStart - Total exit points with locked vars: 0 Function: lxcDomainCreateAndStart - Total exit points with locked vars: 0 Function: lxcDomainShutdown - Total exit points with locked vars: 0 Function: lxcDomainDestroy - Total exit points with locked vars: 0 Function: lxcCheckNetNsSupport - Total exit points with locked vars: 0 Function: lxcStartup - Total exit points with locked vars: 0 Function: lxcShutdown - Total exit points with locked vars: 0 Function: lxcActive - Total exit points with locked vars: 0 Function: lxcVersion - Total exit points with locked vars: 0 Function: lxcGetSchedulerType - Total exit points with locked vars: 0 Function: lxcSetSchedulerParameters - Total exit points with locked vars: 0 Function: lxcGetSchedulerParameters - Total exit points with locked vars: 0 Function: lxcRegister - Total exit points with locked vars: 0 Function: openvzDriverLock - Total exit points with locked vars: 0 Function: openvzDriverUnlock - Total exit points with locked vars: 0 Function: openvzDomainDefineCmd - Total exit points with locked vars: 0 Function: openvzDomainLookupByID - Total exit points with locked vars: 0 Function: openvzGetVersion - Total exit points with locked vars: 0 Function: openvzGetOSType - Total exit points with locked vars: 0 Function: openvzDomainLookupByUUID - Total exit points with locked vars: 0 Function: openvzDomainLookupByName - Total exit points with locked vars: 0 Function: openvzDomainGetInfo - Total exit points with locked vars: 0 Function: openvzDomainDumpXML - Total exit points with locked vars: 0 Function: openvzSetProgramSentinal - Total exit points with locked vars: 0 Function: openvzDomainShutdown - Total exit points with locked vars: 0 Function: openvzDomainReboot - Total exit points with locked vars: 0 Function: openvzGenerateVethName - Total exit points with locked vars: 0 Function: openvzGenerateContainerVethName - Total exit points with locked vars: 0 Function: openvzDomainSetNetwork - Total exit points with locked vars: 0 Function: openvzDomainSetNetworkConfig - Total exit points with locked vars: 0 Function: openvzDomainDefineXML - Total exit points with locked vars: 0 Function: openvzDomainCreateXML - Total exit points with locked vars: 0 Function: openvzDomainCreate - Total exit points with locked vars: 0 Function: openvzDomainUndefine - Total exit points with locked vars: 0 Function: openvzDomainSetAutostart - Total exit points with locked vars: 0 Function: openvzDomainGetAutostart - Total exit points with locked vars: 0 Function: openvzGetMaxVCPUs - Total exit points with locked vars: 0 Function: openvzDomainGetMaxVcpus - Total exit points with locked vars: 0 Function: openvzDomainSetVcpus - Total exit points with locked vars: 0 Function: openvzProbe - Total exit points with locked vars: 0 Function: openvzOpen - Total exit points with locked vars: 0 Function: openvzClose - Total exit points with locked vars: 0 Function: openvzGetType - Total exit points with locked vars: 0 Function: openvzGetNodeInfo - Total exit points with locked vars: 0 Function: openvzGetCapabilities - Total exit points with locked vars: 0 Function: openvzListDomains - Total exit points with locked vars: 0 Function: openvzNumDomains - Total exit points with locked vars: 0 Function: openvzListDefinedDomains - Total exit points with locked vars: 0 Function: openvzGetProcessInfo - Total exit points with locked vars: 0 Function: openvzNumDefinedDomains - Total exit points with locked vars: 0 Function: openvzRegister - Total exit points with locked vars: 0 Function: testDriverLock - Total exit points with locked vars: 0 Function: testDriverUnlock - Total exit points with locked vars: 0 Function: testBuildCapabilities - Total exit points with locked vars: 0 Function: testOpenDefault - Total exit points with locked vars: 0 Function: testBuildFilename - Total exit points with locked vars: 0 Function: testOpenFromFile - Total exit points with locked vars: 0 Function: testOpen - Total exit points with locked vars: 0 Function: testClose - Total exit points with locked vars: 0 Function: testGetVersion - Total exit points with locked vars: 0 Function: testGetHostname - Total exit points with locked vars: 0 Function: testGetMaxVCPUs - Total exit points with locked vars: 0 Function: testNodeGetInfo - Total exit points with locked vars: 0 Function: testGetCapabilities - Total exit points with locked vars: 0 Function: testNumOfDomains - Total exit points with locked vars: 0 Function: testDomainCreateXML - Total exit points with locked vars: 0 Function: testLookupDomainByID - Total exit points with locked vars: 0 Function: testLookupDomainByUUID - Total exit points with locked vars: 0 Function: testLookupDomainByName - Total exit points with locked vars: 0 Function: testListDomains - Total exit points with locked vars: 0 Function: testDestroyDomain - Total exit points with locked vars: 0 Function: testResumeDomain - Total exit points with locked vars: 0 Function: testPauseDomain - Total exit points with locked vars: 0 Function: testShutdownDomain - Total exit points with locked vars: 0 Function: testRebootDomain - Total exit points with locked vars: 0 Function: testGetDomainInfo - Total exit points with locked vars: 0 Function: testDomainSave - Total exit points with locked vars: 0 Function: testDomainRestore - Total exit points with locked vars: 0 Function: testDomainCoreDump - Total exit points with locked vars: 0 Function: testGetOSType - Total exit points with locked vars: 0 Function: testGetMaxMemory - Total exit points with locked vars: 0 Function: testSetMaxMemory - Total exit points with locked vars: 0 Function: testSetMemory - Total exit points with locked vars: 0 Function: testSetVcpus - Total exit points with locked vars: 0 Function: testDomainDumpXML - Total exit points with locked vars: 0 Function: testNumOfDefinedDomains - Total exit points with locked vars: 0 Function: testListDefinedDomains - Total exit points with locked vars: 0 Function: testDomainDefineXML - Total exit points with locked vars: 0 Function: testNodeGetCellsFreeMemory - Total exit points with locked vars: 0 Function: testDomainCreate - Total exit points with locked vars: 0 Function: testDomainUndefine - Total exit points with locked vars: 0 Function: testDomainGetAutostart - Total exit points with locked vars: 0 Function: testDomainSetAutostart - Total exit points with locked vars: 0 Function: testDomainGetSchedulerType - Total exit points with locked vars: 0 Function: testDomainGetSchedulerParams - Total exit points with locked vars: 0 Function: testDomainSetSchedulerParams - Total exit points with locked vars: 0 Function: testOpenNetwork - Total exit points with locked vars: 0 Function: testCloseNetwork - Total exit points with locked vars: 0 Function: testLookupNetworkByUUID - Total exit points with locked vars: 0 Function: testLookupNetworkByName - Total exit points with locked vars: 0 Function: testNumNetworks - Total exit points with locked vars: 0 Function: testListNetworks - Total exit points with locked vars: 0 Function: testNumDefinedNetworks - Total exit points with locked vars: 0 Function: testListDefinedNetworks - Total exit points with locked vars: 0 Function: testNetworkCreate - Total exit points with locked vars: 0 Function: testNetworkDefine - Total exit points with locked vars: 0 Function: testNetworkUndefine - Total exit points with locked vars: 0 Function: testNetworkStart - Total exit points with locked vars: 0 Function: testNetworkDestroy - Total exit points with locked vars: 0 Function: testNetworkDumpXML - Total exit points with locked vars: 0 Function: testNetworkGetBridgeName - Total exit points with locked vars: 0 Function: testNetworkGetAutostart - Total exit points with locked vars: 0 Function: testNetworkSetAutostart - Total exit points with locked vars: 0 Function: testStoragePoolObjSetDefaults - Total exit points with locked vars: 0 Function: testStorageOpen - Total exit points with locked vars: 0 Function: testStorageClose - Total exit points with locked vars: 0 Function: testStoragePoolLookupByUUID - Total exit points with locked vars: 0 Function: testStoragePoolLookupByName - Total exit points with locked vars: 0 Function: testStoragePoolLookupByVolume - Total exit points with locked vars: 0 Function: testStorageNumPools - Total exit points with locked vars: 0 Function: testStorageListPools - Total exit points with locked vars: 0 Function: testStorageNumDefinedPools - Total exit points with locked vars: 0 Function: testStorageListDefinedPools - Total exit points with locked vars: 0 Function: testStoragePoolStart - Total exit points with locked vars: 0 Function: testStorageFindPoolSources - Total exit points with locked vars: 0 Function: testStoragePoolCreate - Total exit points with locked vars: 1 - Locked vars at exit #line 2396 "test.c" return ((virStoragePool *)((void *)0)); ^^^^^^^^^ >>>>Drivers < Var: privconn 73413 >>>>Objects < Function: testStoragePoolDefine - Total exit points with locked vars: 0 Function: testStoragePoolUndefine - Total exit points with locked vars: 0 Function: testStoragePoolBuild - Total exit points with locked vars: 0 Function: testStoragePoolDestroy - Total exit points with locked vars: 0 Function: testStoragePoolDelete - Total exit points with locked vars: 0 Function: testStoragePoolRefresh - Total exit points with locked vars: 0 Function: testStoragePoolGetInfo - Total exit points with locked vars: 0 Function: testStoragePoolDumpXML - Total exit points with locked vars: 0 Function: testStoragePoolGetAutostart - Total exit points with locked vars: 0 Function: testStoragePoolSetAutostart - Total exit points with locked vars: 0 Function: testStoragePoolNumVolumes - Total exit points with locked vars: 0 Function: testStoragePoolListVolumes - Total exit points with locked vars: 0 Function: testStorageVolumeLookupByName - Total exit points with locked vars: 0 Function: testStorageVolumeLookupByKey - Total exit points with locked vars: 0 Function: testStorageVolumeLookupByPath - Total exit points with locked vars: 0 Function: testStorageVolumeCreateXML - Total exit points with locked vars: 0 Function: testStorageVolumeDelete - Total exit points with locked vars: 0 Function: testStorageVolumeTypeForPool - Total exit points with locked vars: 0 Function: testStorageVolumeGetInfo - Total exit points with locked vars: 0 Function: testStorageVolumeGetXMLDesc - Total exit points with locked vars: 0 Function: testStorageVolumeGetPath - Total exit points with locked vars: 0 Function: testDevMonOpen - Total exit points with locked vars: 0 Function: testDevMonClose - Total exit points with locked vars: 0 Function: testRegister - Total exit points with locked vars: 0 Function: networkDriverLock - Total exit points with locked vars: 0 Function: networkDriverUnlock - Total exit points with locked vars: 0 Function: networkAutostartConfigs - Total exit points with locked vars: 0 Function: networkStartup - Total exit points with locked vars: 0 Function: networkReload - Total exit points with locked vars: 0 Function: networkActive - Total exit points with locked vars: 0 Function: networkShutdown - Total exit points with locked vars: 0 Function: networkBuildDnsmasqArgv - Total exit points with locked vars: 0 Function: networkAddMasqueradingIptablesRules - Total exit points with locked vars: 0 Function: networkAddRoutingIptablesRules - Total exit points with locked vars: 0 Function: networkAddIptablesRules - Total exit points with locked vars: 0 Function: networkRemoveIptablesRules - Total exit points with locked vars: 0 Function: networkEnableIpForwarding - Total exit points with locked vars: 0 Function: networkStartNetworkDaemon - Total exit points with locked vars: 0 Function: networkShutdownNetworkDaemon - Total exit points with locked vars: 0 Function: networkLookupByUUID - Total exit points with locked vars: 0 Function: networkLookupByName - Total exit points with locked vars: 0 Function: networkOpenNetwork - Total exit points with locked vars: 0 Function: networkCloseNetwork - Total exit points with locked vars: 0 Function: networkNumNetworks - Total exit points with locked vars: 0 Function: networkListNetworks - Total exit points with locked vars: 0 Function: networkNumDefinedNetworks - Total exit points with locked vars: 0 Function: networkListDefinedNetworks - Total exit points with locked vars: 0 Function: networkCreate - Total exit points with locked vars: 0 Function: networkDefine - Total exit points with locked vars: 0 Function: networkUndefine - Total exit points with locked vars: 0 Function: networkStart - Total exit points with locked vars: 0 Function: networkDestroy - Total exit points with locked vars: 0 Function: networkDumpXML - Total exit points with locked vars: 0 Function: networkGetBridgeName - Total exit points with locked vars: 0 Function: networkGetAutostart - Total exit points with locked vars: 0 Function: networkSetAutostart - Total exit points with locked vars: 0 Function: networkRegister - Total exit points with locked vars: 0 Function: nodeDeviceLock - Total exit points with locked vars: 0 Function: nodeDeviceUnlock - Total exit points with locked vars: 0 Function: nodeNumOfDevices - Total exit points with locked vars: 0 Function: nodeListDevices - Total exit points with locked vars: 0 Function: nodeDeviceLookupByName - Total exit points with locked vars: 0 Function: nodeDeviceDumpXML - Total exit points with locked vars: 0 Function: nodeDeviceGetParent - Total exit points with locked vars: 0 Function: nodeDeviceNumOfCaps - Total exit points with locked vars: 0 Function: nodeDeviceListCaps - Total exit points with locked vars: 0 Function: nodedevRegister - Total exit points with locked vars: 0 Function: nodeDeviceWatchFree - Total exit points with locked vars: 0 Function: storageDriverLock - Total exit points with locked vars: 0 Function: storageDriverUnlock - Total exit points with locked vars: 0 Function: storageDriverAutostart - Total exit points with locked vars: 0 Function: storageDriverStartup - Total exit points with locked vars: 0 Function: storageDriverReload - Total exit points with locked vars: 0 Function: storageDriverActive - Total exit points with locked vars: 0 Function: storageDriverShutdown - Total exit points with locked vars: 0 Function: storagePoolLookupByUUID - Total exit points with locked vars: 0 Function: storagePoolLookupByName - Total exit points with locked vars: 0 Function: storagePoolLookupByVolume - Total exit points with locked vars: 0 Function: storageOpen - Total exit points with locked vars: 0 Function: storageClose - Total exit points with locked vars: 0 Function: storageNumPools - Total exit points with locked vars: 0 Function: storageListPools - Total exit points with locked vars: 0 Function: storageNumDefinedPools - Total exit points with locked vars: 0 Function: storageListDefinedPools - Total exit points with locked vars: 0 Function: storageFindPoolSources - Total exit points with locked vars: 0 Function: storagePoolCreate - Total exit points with locked vars: 0 Function: storagePoolDefine - Total exit points with locked vars: 0 Function: storagePoolUndefine - Total exit points with locked vars: 0 Function: storagePoolStart - Total exit points with locked vars: 0 Function: storagePoolBuild - Total exit points with locked vars: 0 Function: storagePoolDestroy - Total exit points with locked vars: 0 Function: storagePoolDelete - Total exit points with locked vars: 0 Function: storagePoolRefresh - Total exit points with locked vars: 0 Function: storagePoolGetInfo - Total exit points with locked vars: 0 Function: storagePoolDumpXML - Total exit points with locked vars: 0 Function: storagePoolGetAutostart - Total exit points with locked vars: 0 Function: storagePoolSetAutostart - Total exit points with locked vars: 0 Function: storagePoolNumVolumes - Total exit points with locked vars: 0 Function: storagePoolListVolumes - Total exit points with locked vars: 0 Function: storageVolumeLookupByName - Total exit points with locked vars: 0 Function: storageVolumeLookupByKey - Total exit points with locked vars: 0 Function: storageVolumeLookupByPath - Total exit points with locked vars: 0 Function: storageVolumeCreateXML - Total exit points with locked vars: 0 Function: storageVolumeDelete - Total exit points with locked vars: 0 Function: storageVolumeGetInfo - Total exit points with locked vars: 0 Function: storageVolumeGetXMLDesc - Total exit points with locked vars: 0 Function: storageVolumeGetPath - Total exit points with locked vars: 0 Function: storageRegister - Total exit points with locked vars: 0 There was one false positive there qemuDomainEventDispatchFunc which is defined to expect a locked input & output, thus violating the assumptions in my program Oh, and it'll dump .dot files for the control flow graphs which you can view with 'dotty' Daniel -- |: Red Hat, Engineering, London -o- http://people.redhat.com/berrange/ :| |: http://libvirt.org -o- http://virt-manager.org -o- http://ovirt.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|

(* * Analyse libvirt driver API methods for mutex locking mistakes * * Copyright 2008 Red Hat, Inc * * This library is free software; you can redistribute it and/or * modify it under the terms of the GNU Lesser General Public * License as published by the Free Software Foundation; either * version 2.1 of the License, or (at your option) any later version. * * This library is distributed in the hope that it will be useful, * but WITHOUT ANY WARRANTY; without even the implied warranty of * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU * Lesser General Public License for more details. * * You should have received a copy of the GNU Lesser General Public * License along with this library; if not, write to the Free Software * Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA * * Author: Daniel P. Berrange <berrange redhat com> *) open Pretty open Cil (* * Convenient routine to load the contents of a file into * a list of strings *) let input_file filename = let chan = open_in filename in let lines = ref [] in try while true; do lines := input_line chan :: !lines done; [] with End_of_file -> close_in chan; List.rev !lines module DF = Dataflow module UD = Usedef module IH = Inthash module E = Errormsg module VS = UD.VS let debug = ref false (* * This is the list of all libvirt methods which return * pointers to locked objects *) let lockedObjMethods = [ "virDomainFindByID"; "virDomainFindByUUID"; "virDomainFindByName"; "virDomainObjAssignDef"; "virNetworkFindByUUID"; "virNetworkFindByName"; "virNetworkObjAssignDef"; "virNodeDeviceFindByName"; "virNodeDeviceObjAssignDef"; "virStoragePoolFindByUUID"; "virStoragePoolFindByName"; "virStoragePoolFindByKey"; "virStoragePoolObjAssignDef" ] (* * This is the list of all libvirt methods which * can release an object lock. Technically we * ought to pair them up correctly with previous * ones, but the compiler can already complain * about passing a virNetworkObjPtr to a virDomainObjUnlock * method so lets be lazy *) let objectUnlockMethods = [ "virDomainObjUnlock"; "virNetworkObjUnlock"; "virStoragePoolObjUnlock"; "virNodeDevObjUnlock" ] (* * The data types that the previous two sets of * methods operate on *) let lockableObjects = [ "virDomainObjPtr"; "virNetworkObjPtr"; "virStoragePoolObjPtr"; "virNodeDevObjPtr" ] (* * The methods which globally lock an entire driver *) let driverLockMethods = [ "qemuDriverLock"; "openvzDriverLock"; "testDriverLock"; "lxcDriverLock"; "umlDriverLock"; "nodedevDriverLock"; "networkDriverLock"; "storageDriverLock" ] (* * The methods which globally unlock an entire driver *) let driverUnlockMethods = [ "qemuDriverUnlock"; "openvzDriverUnlock"; "testDriverUnlock"; "lxcDriverUnlock"; "umlDriverUnlock"; "nodedevDriverUnlock"; "networkDriverUnlock"; "storageDriverUnlock" ] (* * The data types that the previous two sets of * methods operate on. These may be structs or * typedefs, we don't care *) let lockableDrivers = [ "qemud_driver"; "lxc_driver_t"; "uml_driver"; "openvz_driver"; "testConnPtr" ] let isFuncCallLval lval methodList = match lval with Var vi, o -> List.mem vi.vname methodList | _ -> false let isFuncCallExp exp methodList = match exp with Lval lval -> isFuncCallLval lval methodList | _ -> false let isFuncCallInstr instr methodList = match instr with Call (retval,exp,explist,srcloc) -> isFuncCallExp exp methodList | _ -> false (* * Convenience methods which take a Cil.Instr object * and ask whether its associated with one of the * method sets defined earlier *) let isObjectFetchCall instr = isFuncCallInstr instr lockedObjMethods let isObjectUnlockCall instr = isFuncCallInstr instr objectUnlockMethods let isDriverLockCall instr = isFuncCallInstr instr driverLockMethods let isDriverUnlockCall instr = isFuncCallInstr instr driverUnlockMethods let isWantedType typ typeList = match typ with TNamed (tinfo, attrs) -> List.mem tinfo.tname typeList | TPtr (ptrtyp, attrs) -> let f = match ptrtyp with TNamed (tinfo2, attrs) -> List.mem tinfo2.tname typeList | TComp (cinfo, attrs) -> List.mem cinfo.cname typeList | _ -> false in f | _ -> false (* * Convenience methods which take a Cil.Varinfo object * and ask whether it matches a variable datatype that * we're interested in tracking for locking purposes *) let isLockableObjectVar varinfo = isWantedType varinfo.vtype lockableObjects let isLockableDriverVar varinfo = isWantedType varinfo.vtype lockableDrivers (* * Take a Cil.Exp object (ie an expression) and see whether * the expression corresponds to a check for NULL against * one of our interesting objects * eg * * if (!vm) ... * * For a variable 'virDomainObjPtr vm' *) let isLockableObjectNull exp = match exp with | UnOp (op,exp,typ) -> ( match op with LNot -> ( match exp with Lval (lhost, off) -> ( match lhost with Var vi -> isLockableObjectVar vi | _ -> false ) | _ -> false ) | _ -> false ) | _ -> false (* * Prior to validating a function, intialize these * to VS.empty * * They contain the list of driver and object variables * objects declared as local variables * *) let lockableObjs: VS.t ref = ref VS.empty let lockableDriver: VS.t ref = ref VS.empty (* * Given a Cil.Instr object (ie a single instruction), get * the list of all used & defined variables associated with * it. Then caculate intersection with the driver and object * variables we're interested in tracking and return four sets * * List of used driver variables * List of defined driver variables * List of used object variables * List of defined object variables *) let computeUseDefState i = let u, d = UD.computeUseDefInstr i in let useo = VS.inter u !lockableObjs in let defo = VS.inter d !lockableObjs in let used = VS.inter u !lockableDriver in let defd = VS.inter d !lockableDriver in (used, defd, useo, defo) (* Some crude helpers for debugging this horrible code *) let printVI vi = ignore(printf " Var: %s %d\n" vi.vname vi.vid) let printVS how vs = print_endline (" >>>>" ^ how ^ "<"); VS.iter printVI vs let prettyprint2 stmdat () (_, ld, ud, lo, ui) = text "" (* * This module implements the Cil.DataFlow.ForwardsTransfer * interface. This is what 'does the interesting stuff' * when walking over a function's code paths *) module Locking = struct let name = "Locking" let debug = debug (* * Our state currently consists of * * The set of driver variables that are locked * The set of driver variables that are unlocked * The set of object variables that are locked * The set of object variables that are unlocked * * Eventually we will add another 4 sets with error data from specific points * * Instrs using an unlocked driver variable * Instrs using an unlocked object variable * Instrs locking a object variable while not holding a locked driver variable * Instrs locking a driver variable while holding a locked object variable * * But for now we're just tracking still locked vars at exit *) type t = (unit * VS.t * VS.t * VS.t * VS.t) (* This holds an instance of our state data, per statement *) let stmtStartData = IH.create 32 let pretty = prettyprint2 stmtStartData let copy (_, ld, ud, lo, uo) = ((), ld, ud, lo, uo) let computeFirstPredecessor stm (_, ld, ud, lo, uo) = ((), ld, ud, lo, uo) (* * Merge existing state for a statement, with new state * * If new and old state is the same, this returns None, * If they are different, then returns the union. *) let combinePredecessors (stm:stmt) ~(old:t) ((_, ldn, udn, lon, uon):t) = match old with (_, ldo, udo, loo,uoo)-> begin let lde= (VS.equal ldo ldn) || ((VS.is_empty ldo) && (VS.is_empty ldn)) in let ude= VS.equal udo udn || ((VS.is_empty udo) && (VS.is_empty udn)) in let loe= VS.equal loo lon || ((VS.is_empty loo) && (VS.is_empty lon)) in let uoe= VS.equal uoo uon || ((VS.is_empty uoo) && (VS.is_empty uon)) in if lde && ude && loe && uoe then None else ( let ldret = VS.union ldo ldn in let udret = VS.union udo udn in let loret = VS.union loo lon in let uoret = VS.union uoo uon in Some ((), ldret, udret, loret, uoret) ) end (* * This handles a Cil.Instr object. This is sortof a C level statement. *) let doInstr i (_, ld, ud, lo, uo) = let transform (_, ld, ud, lo, uo) = let used, defd, useo, defo = computeUseDefState i in (* * XXX this method needs extending to check for * correctly locking order. ie take driver lock * before object lock. Don't try to take driver * lock again, if you still hold an object lock * * and any other interesting bugs you can think of *) if isDriverLockCall i then ( (* * A driver was locked, so add to the list of locked * driver variables, and remove from the unlocked list *) let retld = VS.union ld used in let retud = VS.diff ud used in ((), retld, retud, lo, uo); ) else if isDriverUnlockCall i then ( (* * A driver was unlocked, so add to the list of unlocked * driver variables, and remove from the locked list *) let retld = VS.diff ld used in let retud = VS.union ud used in ((), retld, retud, lo, uo); ) else if isObjectFetchCall i then ( (* * A object was fetched & locked, so add to the list of * locked driver variables. Nothing to remove from unlocked * list here. * * XXX, not entirely true. We should check if they're * blowing away an existing non-NULL value in the lval * really. *) let retlo = VS.union lo defo in ((), ld, ud, retlo, uo); ) else if isObjectUnlockCall i then ( (* * A object was unlocked, so add to the list of unlocked * driver variables, and remove from the locked list *) let retlo = VS.diff lo useo in let retuo = VS.union uo useo in ((), ld, ud, retlo, retuo); ) else ( (* * Nothing special happened, at best an assignment. * So add any defined variables to the list of unlocked * object or driver variables. * XXX same edge case as isObjectFetchCall about possible * overwriting *) let retud = VS.union ud defd in let retuo = VS.union uo defo in ((), ld, retud, lo, retuo) ); in DF.Post transform (* * This handles a Cil.Stmt object. This is sortof a C code block *) let doStmt stm (_, ld, ud, lo, uo) = DF.SDefault (* * This handles decision making for a conditional statement, * ie an if (foo). It is called twice for each conditional * ie, once per possible choice. *) let doGuard exp (_, ld, ud, lo, uo) = (* * If we're going down a branch where our object variable * is set to NULL, then we must remove it from the * list of locked objects. This handles the case of... * * vm = virDomainFindByUUID(..) * if (!vm) { * .... this code branch .... * } else { * .... leaves default handling for this branch ... * } *) if isLockableObjectNull exp then ( let lonull = UD.computeUseExp exp in let loret = VS.diff lo lonull in let uoret = VS.union uo lonull in DF.GUse ((), ld, ud, loret, uoret) ) else ( DF.GDefault ) (* * We're not filtering out any statements *) let filterStmt stm = true end module L = DF.ForwardsDataFlow(Locking) let () = (* Read the list of files from "libvirt-files". *) let files2 = input_file "libvirt-files" in (* * It is tediously slow to process all the .i files listed * in the auto-generated files2 list above * So for now, lets just hardcode list of driver files *) let files = [ "/home/berrange/src/xen/libvirt-work/src/qemu_driver.i"; "/home/berrange/src/xen/libvirt-work/src/uml_driver.i"; "/home/berrange/src/xen/libvirt-work/src/lxc_driver.i"; "/home/berrange/src/xen/libvirt-work/src/openvz_driver.i"; "/home/berrange/src/xen/libvirt-work/src/test.i"; "/home/berrange/src/xen/libvirt-work/src/network_driver.i"; "/home/berrange/src/xen/libvirt-work/src/node_device.i"; "/home/berrange/src/xen/libvirt-work/src/node_device_hal.i"; "/home/berrange/src/xen/libvirt-work/src/storage_driver.i" ] in (* Load & parse each input file. *) let files = List.map ( fun filename -> (* Why does parse return a continuation? *) let f = Frontc.parse filename in f () ) files in (* Merge them. *) let file = Mergecil.merge files "test" in (* Do control-flow-graph analysis. *) Cfg.computeFileCFG file; (* * Now comes our fun.... iterate over every global symbol * definition Cfg found..... but... *) List.iter ( function (* ....only care about functions *) | GFun (fundec, loc) -> (* function definition *) let name = fundec.svar.vname in (* * We ought to look for the * * "static virDriver qemuDriver" * * and extract the list of functions. * And also any functions passed to virEventAddHandle or * virEventAddTimeout * * For now I'm lazy so lets just pick them by prefix. This * will generate some false positives so we need to fix * this properly sometime *) let prefixes = [ "qem"; "uml"; "lxc"; "ope"; "tes"; "net"; "sto"; "nod" ] in if String.length name > 4 then let prefix = String.sub fundec.svar.vname 0 3 in let wantit = List.mem prefix prefixes in if wantit then ( ignore (printf "Function: %s\n" name); Cfg.printCfgFilename (name ^ ".dot") fundec; (* Initialize list of driver & object variables to be empty *) lockableDriver = ref VS.empty; lockableObjs = ref VS.empty; (* * Query all local variables, and figure out which correspond * to interesting driver & object variables we track *) List.iter ( fun var -> if isLockableDriverVar var then lockableDriver := VS.add var !lockableDriver else if isLockableObjectVar var then lockableObjs := VS.add var !lockableObjs; ) fundec.slocals; (* * Initialize the state for each statement (ie C code block) * to be empty *) List.iter ( fun st -> IH.add Locking.stmtStartData st.sid ((), VS.empty, VS.empty, VS.empty, VS.empty); ) fundec.sallstmts; (* * This walks all the code paths in the function building * up the state for each statement (ie C code block) * ie, this is invoking the "Locking" module we created * earlier *) L.compute fundec.sallstmts; (* * Find all statements (ie C code blocks) which have no * successor statements. This means they are exit points * in the function *) let exitPoints = List.filter ( fun st -> List.length st.succs = 0 ) fundec.sallstmts in (* * For each of the exit points, check to see if there are * any with locked driver or object variables & grab them *) let leaks = List.filter ( fun st -> let (_, ld, ud, lo, uo) = IH.find Locking.stmtStartData st.sid in let leakDrivers = not (VS.is_empty ld) in let leakObjects = not (VS.is_empty lo) in leakDrivers or leakObjects ) exitPoints in printf " - Total exit points with locked vars: %d\n" (List.length leaks); (* * Finally tell the user which exit points had locked varaibles * And show them the line number and code snippet for easy fixing *) List.iter ( fun st -> ignore (Pretty.printf " - Locked vars at exit %a\n^^^^^^^^^\n" d_stmt st); let (_, ld, ud, lo, uo) = IH.find Locking.stmtStartData st.sid in printVS "Drivers " ld; printVS "Objects " lo ) leaks; () ) | _ -> () ) file.globals;

**Follow-Ups**:**Re: [libvirt] PATCH: 0/28: Thread safety for libvirtd daemon and drivers***From:*Daniel P. Berrange

**Re: [libvirt] PATCH: 0/28: Thread safety for libvirtd daemon and drivers***From:*Richard W.M. Jones

**References**:**Re: [libvirt] PATCH: 0/28: Thread safety for libvirtd daemon and drivers***From:*Daniel P. Berrange

**Re: [libvirt] PATCH: 0/28: Thread safety for libvirtd daemon and drivers***From:*Daniel Veillard

**Re: [libvirt] PATCH: 0/28: Thread safety for libvirtd daemon and drivers***From:*Daniel P. Berrange

**Re: [libvirt] PATCH: 0/28: Thread safety for libvirtd daemon and drivers***From:*Daniel Veillard