[Linux-cluster] Multicast for GFS?

[Lazar Obradovic]

Hi, 

can someone, please, give some advice about configuring multicast with
GFS? I know it might go out of topic, but it's perhaps useful for
others.

I'd use broadcast instead, but I have a problem that two groups of
servers sharing the same storage, but that are located in different
vlans, separated by router-on-a-stick, so I guess I have to use
multicast.

I've configured the router for multicast (config is right below), but it
doesn't seem to work. 

Here's ascii pic of what I'm trying to make: 

        +--------+
        | router |
        +--------+
         /       \
+----------+    +----------+
| switch A |    | switch B | 
| vlan 100 |    | vlan 200 | 
+----------+    +----------+
     |               |
+----------+    +----------+
| server A |    | server B |
+----------+    +----------+
     |               |
  +---------------------+
  |   san / storage     | 
  +---------------------+

and relevant config (that I made this far): 

router (cisco ios): 

ip multicast-routing 
!
interface FastEthernet0/0
 description Branch A
 ip address 1.1.1.1 255.255.255.0 
 ip pim sparse-dense-mode
 ip igmp version 1
 encapsulation dot1q 100
!
interface FastEthernet0/1
 description Branch B
 ip address 1.1.2.1 255.255.255.0
 ip pim sparse-dense-mode
 ip igmp version 1
 encapsulation dot1q 200
!
ip pim send-rp-announce FastEthernet0/0 scope 16
ip pim send-rp-discovery scope 16
!

switch A and switch B are manageable Intel switches (dunno the exact
model; they are bundled with my IBM Blades), but have IGMP Snooping
turned on on every interface, and show default cisco pim groups
(224.0.0.40 and 224.0.0.39) on upstream ports.

/etc/cluster/cluster.conf for each cluster node is same (only important
part of config is here, ask for more, if needed): 

<cluster name="test" config_version="1">
        <cman>
                <multicast addr="224.0.0.11"/>
        </cman>
        <nodes>
                <node name="node1" votes="1">
                        <multicast addr="224.0.0.11" interface="eth0"/>
                </node>
                <node name="node2" votes="1">
                        <multicast addr="224.0.0.11" interface="eth0"/>
                </node>
        </nodes>
</cluster>

hosts ping each other, so networking part, as far as basic ip and
unicast is concerned, is working properly.

When starting, cman_tool says: 
cluster # cman_tool join -d
multicast address 224.0.0.11
if eth0 for mcast address 224.0.0.11
setup up interface for address: node1

and as I can see from strace, "cman_tool join", this is what happens: 

socket(0x1f /* PF_??? */, SOCK_DGRAM, 2) = 3
ioctl(3, 0x780b, 0x2)                   = 0
setsockopt(3, 0x2 /* SOL_?? */, 109, [6516590], 4) = 0
[...]
socket(PF_INET, SOCK_DGRAM, IPPROTO_UDP) = 4
bind(4, {sa_family=AF_INET, sin_port=htons(6809), sin_addr=inet_addr("224.0.0.11")}, 16) = 0
socket(PF_INET, SOCK_DGRAM, IPPROTO_UDP) = 5
bind(5, {sa_family=AF_INET, sin_port=htons(6809), sin_addr=inet_addr("1.1.1.2")}, 16) = 0
setsockopt(3, 0x2 /* SOL_?? */, 100, "\4\0\0\0\0\0\0\0", 8) = 0
setsockopt(3, 0x2 /* SOL_?? */, 103, "\5\0\0\0\0\0\0\0", 8) = 0
setsockopt(3, 0x2 /* SOL_?? */, 101, "\1\344\377\277\3\0\0\0\0\0\0\0\1\0\0\0smtp\0\'\1@\210\0"..., 36) = 0
close(3)                                = 0
exit_group(0)                           = ?

I've checked some programming examples on multicast as well as code for
cman, and I thing cman_tool/join.c has two problems: 

- it never seems to issue setsockopt(..., IP_ADD_MEMBERSHIP...), thus,
never joins the group. I believe the problem is in if (!bcast) check,
which, if replaced with "if (bhe)" should work fine...

- it binds the socket with multicast address (fd = 4 in my case) instead
of local address. If the examples I looked are true, one should bind
local interface, and then specify mcast address in setsockopt call. 

Can someone comment this issue? Am I going in completly wrong direction,
or multicast support isn't ready yet?

-- 
Lazar Obradovic, System Engineer
----- 
laza at YU.net
YUnet International http://www.EUnet.yu
Dubrovacka 35/III, 11000 Belgrade
Tel: +381 11 3119901; Fax: +381 11 3119901
-----
This e-mail is confidential and intended only for the recipient.
Unauthorized distribution, modification or disclosure of its
contents is prohibited. If you have received this e-mail in error,
please notify the sender by telephone +381 11 3119901.
-----