2009/8/14 Dave Allan <dallan redhat com <mailto:dallan redhat com>>
Łukasz Mierzwa wrote:
I will try to answer your questions inline.
I'm trying to setup pool of machines (nodes) for virtual
machines hosting and I got few question about shared storage. My
main requirements are:
1. central management - I've got simple python app that stores
information about all virtual machines and all nodes, this app
needs to be able to manage volumes using libvirt API, so I need
libvirt volume pools
Storage pools will let you manage iSCSI LUNs as volumes. When you
start the iSCSI pool, the node will login to the iSCSI target and
libvirt will create a volume in the pool for each LUN on that target.
2. live migration - I got shared storage with HA, I want to use
it also for live migration in case one of nodes is dying or if I
want to do some load balancing
By HA you mean you have two targets presenting the same LUNs, or do
you mean something else?
Yes, both targets have the same LUNs and data stored in those LUNs is
synchronized using drdb. Like described in this howto
Right now I'm thinking about 2 machines with disks synchronized
using drdb, both acting as a identical iscsi targets, iscsi HA
will be provided by heartbeat. So I will end up with virtual IP
pointing to working iscsi target, drdb should keep storage is
1. I can't just use single iscsi LUN and export it as libvirt
storage pool to each node, because no pool type would work that
You could create a logical pool on the iSCSI LUN and libvirt can
create and destroy volumes in the logical pool, which I think is
what you're saying below.
2. http://libvirt.org/storage.html section "iSCSI volume pools"
"Volumes must be pre-allocated on the iSCSI server, and cannot
be created via the libvirt APIs."
So even if I got one LUN per node and set it as iscsi volume
pool I would need to create each volume on iscsi target. Libvirt
can't manage volumes in such pool, it can only assign already
created volumes to virtual machines, right?
What that means is that libvirt doesn't know how to tell an iSCSI
array to create a new LUN. You have to use whatever admin tools you
use to manage the array to create new LUNs, but then libvirt will
discover them with a pool refresh and you can assign them to guests.
3. So maybe my storage could be setup as LVM volume group and
this lvm group would be managed as libvirt lvm volume pool on
master (from heartbeat POV) iscsi target. I would create one
logical volume per virtual machine, export this volume as a
separate iscsi LUN, and use this LUN as iscsi volume for virtual
To create new virtual machine I would:
a) create lvm volume on iscsi target using libvirt
b) export this volume using iscsi
c) define virtual machine with this volume
But can I create iscsi volume without pool?
I think I'm following you, but let me see if I can restate what
a) With your array's management tools, not with with libvirt, create
a single iSCSI LUN (or several LUNs) on a target on your array.
b) With libvirt, create an iSCSI storage pool for that iSCSI target.
c) With libvirt, create a logical storage pool on that LUN, which
creates a LVM group on that LUN.
d) With libvirt, you create (and destroy) logical volumes in the LVM
group and assign them to guests.
Does it makes any sense? Are there better ways to *manage*
volumes for virtual machines using iscsi?
And just to make sure I'm clear on what you mean by manage, you mean
create and destroy, right?
Create, destroy and possibly resize (which means grow as shrinking
filesystems is complex and more dangerous) virtual machine volumes.
Let me make that little more clear:
1. Lets assume I have one storage node that is acting as a iscsi target.
2. This storage node has one big block device (RAID of some kind) that
will be used for storing virtual machines volumes
3. I have multiple nodes for hosting virtual machines.
4. Each of those hosting nodes needs to be able to access every volume
stored on the storage node (for live migration) using iscsi.
I'm looking for proper way to do this. Right now I'm thinking about this
1. On the storage node I will create lvm volume group using my RAID.
2. Each virtual machine volume is created as lvm logical volume, it
doesn't matter how this volume is created (by hand, using management app
or any other way) but it always happens on the storage node. I could
export whole RAID as a single LUN, create lvm volume group on top of
that, but to be able to work with this lvm volume group from every
hosting node at the same time I would need to use clustered lvm (or
something like that) and that would add some complexity.
3. Each lvm logical volume is exported as unique LUN (also out of
4. iscsi LUNs are discovered using libvirt iscsi volume pool, and
assigned to right virtual machine.