Re: multiple network interfaces questions, RHEL 3 new install

[Rick Stevens <rstevens vitalstream com>]

Chuck Campbell wrote:
> Our new dual Opteron server (x86_64) running RHEL3, update 3 (Taroon)
> has a K8s pro motherboard with on board networking.  It also has two
> broadcom Gigabit Ethernet NICs.
>
> The system sees eth0 as the motherboard 100bT, and eth1 adn eth2 as the
> two Broadcom interfaces.  
>
> Questions:
> I want to use one GB interface to talk to the other cluster machines,
> the other GB interface to be the samba server, and not mix traffic between
> them.
>
> I'd like to use the 100bT interface for mail only.
>
> Where can I find info on setting things up this way (or is it impossible) ?
> I know that I can bind samba to a particular interface, but how do I control
> what interface talks to the local intranet and what interface only talks
> to other cluster machines?

Ideally, each interface should be on a separate subnet or VLAN.  It's
not absolutely necessary, but if they're on the same network each one
will see traffic from the others (unless your switch is clever with 
layer 3 switching--most cheap home units really aren't).

If you want to do this, there really isn't any valid reason to have
three interfaces in the first place.  You really aren't segregating
traffic.

> Will they need to be different sub-nets to make this work?

No, but you have to force routes to each host.  If you put them on
separate subnets, then the routing will take care of itself.

> The output of netstat -r is here:
> [root cluster_master etc]# netstat -r
> Kernel IP routing table
> Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
> 172.17.2.0      *               255.255.255.0   U         0 0          0 eth0
> 172.17.2.0      *               255.255.255.0   U         0 0          0 eth1
> 172.17.2.0      *               255.255.255.0   U         0 0          0 eth2
> 169.254.0.0     *               255.255.0.0     U         0 0          0 eth2
> default         172.17.2.1      0.0.0.0         UG        0 0          0 eth0

You have all interfaces talking to the same network segment.  It'll
work, but it's pretty lame and you're really not segregating traffic.
Since you're using non-routing addresses, why not use 172.17.2/24 for
mail on eth0, 172.17.3/24 for Samba on eth1 and 172.17.4/24 for cluster
on eth2?

As to making sure each interface does what you want, as long as you
don't change the order of the cards in your bus, the "ifcfg-ethX"
scripts found in "/etc/sysconfig/network-scripts" define the IP address
and netmask for each interface.  The "address & netmask" determine the
network and thereby the route for the interface.  If you move the cards
around in your bus, the assignments as to which is eth1 and eth2 may
change.

> I don't know what or where the 169.254.0.0 address is...

It's used for global DHCP.  Don't worry about it.

> The output of ifconfig -a is here:
> [root cluster_master etc]# ifconfig -a
> eth0      Link encap:Ethernet  HWaddr 00:E0:81:2C:74:33
>           inet addr:172.17.2.29  Bcast:172.17.2.255  Mask:255.255.255.0
>           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
>           RX packets:928092 errors:0 dropped:0 overruns:0 frame:0
>           TX packets:218612 errors:0 dropped:0 overruns:0 carrier:0
>           collisions:0 txqueuelen:1000
>           RX bytes:1134994584 (1082.4 Mb)  TX bytes:40338127 (38.4 Mb)
>           Interrupt:18 Base address:0xbc00 Memory:feafb000-feafb038
>  
> eth1      Link encap:Ethernet  HWaddr 00:E0:81:2C:73:86
>           inet addr:172.17.2.30  Bcast:172.17.2.255  Mask:255.255.255.0
>           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
>           RX packets:88001 errors:0 dropped:0 overruns:0 frame:0
>           TX packets:70 errors:0 dropped:0 overruns:0 carrier:0
>           collisions:0 txqueuelen:1000
>           RX bytes:6950013 (6.6 Mb)  TX bytes:4480 (4.3 Kb)
>           Interrupt:24
>  
> eth2      Link encap:Ethernet  HWaddr 00:E0:81:2C:73:87
>           inet addr:172.17.2.31  Bcast:172.17.2.255  Mask:255.255.255.0
>           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
>           RX packets:85921 errors:0 dropped:0 overruns:0 frame:0
>           TX packets:70 errors:0 dropped:0 overruns:0 carrier:0
>           collisions:0 txqueuelen:1000
>           RX bytes:6791141 (6.4 Mb)  TX bytes:4480 (4.3 Kb)
>           Interrupt:25
>  
> lo        Link encap:Local Loopback
>           inet addr:127.0.0.1  Mask:255.0.0.0
>           UP LOOPBACK RUNNING  MTU:16436  Metric:1
>           RX packets:539071 errors:0 dropped:0 overruns:0 frame:0
>           TX packets:539071 errors:0 dropped:0 overruns:0 carrier:0
>           collisions:0 txqueuelen:0
>           RX bytes:49468938 (47.1 Mb)  TX bytes:49468938 (47.1 Mb)

Fine.

> Here is the dmesg output for the network stuff:
> Intel(R) PRO/100 Network Driver - version 2.3.43-k1
> Copyright (c) 2004 Intel Corporation
>                                                                                 
> divert: allocating divert_blk for eth0
> e100: selftest OK.
> e100: eth0: Intel(R) PRO/100 Network Connection
>   Hardware receive checksums enabled
>   cpu cycle saver enabled
>                                                                                 
> tg3.c:v3.6RH (June 12, 2004)
> divert: allocating divert_blk for eth1
> eth1: Tigon3 [partno(BCM95704A7) rev 2003 PHY(5704)] (PCIX:100MHz:64-bit) 10/100/1000BaseT Ethernet 00:e0:81:2c:73:86
> eth1: HostTXDS[1] RXcsums[1] LinkChgREG[0] MIirq[0] ASF[0] Split[0] WireSpeed[1] TSOcap[0]
> divert: allocating divert_blk for eth2
> eth2: Tigon3 [partno(BCM95704A7) rev 2003 PHY(5704)] (PCIX:100MHz:64-bit) 10/100/1000BaseT Ethernet 00:e0:81:2c:73:87
> eth2: HostTXDS[1] RXcsums[1] LinkChgREG[0] MIirq[0] ASF[0] Split[0] WireSpeed[1] TSOcap[0]
> divert: freeing divert_blk for eth0
> divert: freeing divert_blk for eth1
> divert: freeing divert_blk for eth2
> ip_tables: (C) 2000-2002 Netfilter core team
> Intel(R) PRO/100 Network Driver - version 2.3.43-k1
> Copyright (c) 2004 Intel Corporation
>                                                                                 
> divert: allocating divert_blk for eth0
> e100: selftest OK.
> e100: eth0: Intel(R) PRO/100 Network Connection
>   Hardware receive checksums enabled
>   cpu cycle saver enabled
>                                                                                 
> ip_tables: (C) 2000-2002 Netfilter core team
> e100: eth0 NIC Link is Up 100 Mbps Full duplex
> tg3.c:v3.6RH (June 12, 2004)
> divert: allocating divert_blk for eth1
> eth1: Tigon3 [partno(BCM95704A7) rev 2003 PHY(5704)] (PCIX:100MHz:64-bit) 10/100/1000BaseT Ethernet 00:e0:81:2c:73:86
> eth1: HostTXDS[1] RXcsums[1] LinkChgREG[0] MIirq[0] ASF[0] Split[0] WireSpeed[1] TSOcap[0]
> divert: allocating divert_blk for eth2
> eth2: Tigon3 [partno(BCM95704A7) rev 2003 PHY(5704)] (PCIX:100MHz:64-bit) 10/100/1000BaseT Ethernet 00:e0:81:2c:73:87
> eth2: HostTXDS[1] RXcsums[1] LinkChgREG[0] MIirq[0] ASF[0] Split[0] WireSpeed[1] TSOcap[0]
> ip_tables: (C) 2000-2002 Netfilter core team
> tg3: eth1: Link is up at 100 Mbps, full duplex.
> tg3: eth1: Flow control is on for TX and on for RX.
> ip_tables: (C) 2000-2002 Netfilter core team
> tg3: eth2: Link is up at 100 Mbps, full duplex.
> tg3: eth2: Flow control is on for TX and on for RX.

Interesting, but not really necessary for the current discussion.
----------------------------------------------------------------------
- Rick Stevens, Senior Systems Engineer     rstevens vitalstream com -
- VitalStream, Inc.                       http://www.vitalstream.com -
-                                                                    -
-    "Hello. My PID is Inigo Montoya.  You `kill -9'-ed my parent    -
-                     process.  Prepare to vi."                      -
----------------------------------------------------------------------