[rhn-users] Network Receive/Send Traffic

[pbounds]

This may sound like a newbie question but it is perplexing at best.  I
recently reinstalled RH3ESup3 on one of our servers in Dallas. This system is
running a dual Xeon Intel SE7520JR2 mainboard, with (2) 3.0MHz processors. 
This system also has two Intel Pro/1000 Gigabit nics onboard.  The system is
also running an Adaptec 7211C iscsi card accessing an Intel iscsi storage
array.

When the system was originanlly configured at our office we had configured the
two onboard nics with a ###.###.###.100 and ###.###.###.101 on a private class
C lan.  The ###.###.###.100 nic is the one that is nat'd to the firewall for
incoming traffic. The outside world doesn't know about eth1.

Once the systems were in place in Dallas, I went in and edited the
/etc/sysconfig/network-scripts with the new private addresses on that network
there and ifconfig -a shows both configured and running properly.  One thing
to note is I have a duplicate system in Houston, with the same motherboard and
o/s that I cannot duplicate the issue.  And of further note, if I down eth1 to
force the tcpip traffic to only use eth0, I lose my network connection.

But what is happening is that if you run an ifconfig -a you see all the
incoming traffic on eth0 and all the outgoing traffic on eth1.  Even removing
both nic devices, rebooting the system , and adding back the nics still
produce this odd traffic.  The only way we were able to 'trick' tcpip was to
disable both of those devices, the configure eth0 only for a ###.###.###.120
IP, and restart the network.  Now you see all of the send/receive traffic
going in and out the same port.

Question is 'Am I totally missing something?' Checking arp, and resetting the
dlink gigaswitch did not solve the problem associated with both of those IPs.

Any help anyone can shed some light on this weird issue would be appreciated.