IBM x346 broadcom debacle.

Jimmy Stewpot squid at oranged.to
Fri Feb 24 17:16:33 UTC 2006


Hello,

It appears as though this problem may well be something a little more 
substantial. I have since disabled in the kernel both broadcom nics and 
the issue exists with ANY network interface card in the server when 
there are TWO.

Currently there are two intel eepro100 cards in the system. When I 
enable eth1 the system starts to get laggy. However if I disable it the 
performance goes back down to the normal fast system.

When I check the routes to make sure there are no routing issues (only 
one default gateway and route for our network etc).

The second interface is used for Veritas netbackups.

I am convinced that there are some irq/dma style issues occuring on this 
system. However I am not really sure about how I would go about 
diagnosing these issues more closely.

Any feedback would be fantastic.

Regards,

Jimmy.

Jimmy Stewpot wrote:
> Hello,
> 
> I have been working to diagnose some issues with the IBM x346 servers 
> when using the on board broadcom network cards. The issues occur when I 
> use both the bcm5700 driver provided by IBM and the tg3 driver provided 
> by Redhat, Broadcom or the vanilla kernel.
> 
> The issue is that when we enable our ICMP Polling service that sends out 
> several thousand packets over a period of 5minutes to several thousand 
> different hosts we see that the packets leave the system. When we go and 
> have a look at the Cisco Catalyst 6509 and do an rspan of the port we 
> see the ICMP packets returning to the switch port and on the server we 
> never see those packets with tcpdump on the system.
> 
> After some research I found that there are many other people 
> experiencing similar problems and after my previous post I have been 
> sent many people asking if I had a resolution. In one way I have a 
> resolution but it has been compounded further with this issue.
> 
> The resolution for the problem was to add an additional Intel 
> EtherExpress 100 network card into the system and make that the primary 
> interface. Since doing that and having BOTH broadcom network interfaces 
> disabled the problem goes away. No more lag, no issues at all.
> 
> One thing that I did on the side was update all of the firmware on the 
> server to the latest BIOS, network firmware and others. Since doing that 
> the problem was slightly reduced but it was by no means a complete 
> resolution to the problem.
> 
> Now that I am using an intel eepro 100 card in the server it is working 
> great. However we also have a Veritas netbackup client running on that 
> server which backs up the data across the second network interface. That 
> second network interface is again the broadcom on board. This time it is 
> the second on board interface rather than the first. When the interfaces 
> are both disabled the problem is non existant. However when we enable 
> the eth2 interface on board for the netbackup the system starts to lag 
> even though the routes are all via the eth0 (eepro100) interface.
> 
>  From seeing those symptoms I am wondering if there could be some sort 
> of memory interrupt/dma issue on the system that Linux is not reporting 
> correctly. I am interested to know if anyone else has had similar issues 
> in the past. Could it also be that the card is running 64bit rather than 
> in 32bit like the eepro100 card?
> 
> Here is a copy of my lspci -v output for any further hardware 
> investigation. Any ideas or concepts would be great to hear about so 
> that I can hopefully try and resolve these outstanding issues once and 
> for all.
> 
> 
> ===SNIP====
> 
> lspci -v
> 00:00.0 Host bridge: Intel Corp. Server Memory Controller Hub (rev 0a)
>         Subsystem: IBM: Unknown device 02dd
>         Flags: bus master, fast devsel, latency 0
>         Memory at <ignored> (32-bit, non-prefetchable) [size=4K]
>         Capabilities: [40] #09 [4105]
> 
> 00:00.1 Class ff00: Intel Corp. Memory Controller Hub Error Reporting 
> Register (rev 0a)
>         Subsystem: IBM: Unknown device 02dd
>         Flags: fast devsel
> 
> 00:02.0 PCI bridge: Intel Corp. Memory Controller Hub PCI Express Port 
> A0 (rev 0a) (prog-if 00 [Normal decode])
>         Flags: bus master, fast devsel, latency 0
>         Bus: primary=00, secondary=02, subordinate=04, sec-latency=0
>         I/O behind bridge: 00004000-00004fff
>         Memory behind bridge: ce000000-cfffffff
>         Capabilities: [50] Power Management version 2
>         Capabilities: [58] Message Signalled Interrupts: 64bit- 
> Queue=0/1 Enable-
>         Capabilities: [64] #10 [0041]
> 
> 00:04.0 PCI bridge: Intel Corp. Memory Controller Hub PCI Express Port 
> B0 (rev 0a) (prog-if 00 [Normal decode])
>         Flags: bus master, fast devsel, latency 0
>         Bus: primary=00, secondary=05, subordinate=05, sec-latency=0
>         Memory behind bridge: cc000000-cdffffff
>         Capabilities: [50] Power Management version 2
>         Capabilities: [58] Message Signalled Interrupts: 64bit- 
> Queue=0/1 Enable-
>         Capabilities: [64] #10 [0041]
> 
> 00:05.0 PCI bridge: Intel Corp. Memory Controller Hub PCI Express Port 
> B1 (rev 0a) (prog-if 00 [Normal decode])
>         Flags: bus master, fast devsel, latency 0
>         Bus: primary=00, secondary=06, subordinate=06, sec-latency=0
>         Memory behind bridge: ca000000-cbffffff
>         Capabilities: [50] Power Management version 2
>         Capabilities: [58] Message Signalled Interrupts: 64bit- 
> Queue=0/1 Enable-
>         Capabilities: [64] #10 [0041]
> 
> 00:06.0 PCI bridge: Intel Corp. Memory Controller Hub PCI Express Port 
> C0 (rev 0a) (prog-if 00 [Normal decode])
>         Flags: bus master, fast devsel, latency 0
>         Bus: primary=00, secondary=07, subordinate=09, sec-latency=0
>         I/O behind bridge: 00005000-0000ffff
>         Memory behind bridge: c8000000-c9ffffff
>         Capabilities: [50] Power Management version 2
>         Capabilities: [58] Message Signalled Interrupts: 64bit- 
> Queue=0/1 Enable-
>         Capabilities: [64] #10 [0041]
> 
> 00:08.0 System peripheral: Intel Corp. Memory Controller Hub Extended 
> Configuration Registers (rev 0a)
>         Subsystem: IBM: Unknown device 02dd
>         Flags: fast devsel
> 
> 00:1d.0 USB Controller: Intel Corp. 82801EB/ER (ICH5/ICH5R) USB UHCI #1 
> (rev 02) (prog-if 00 [UHCI])
>         Subsystem: IBM: Unknown device 02dd
>         Flags: bus master, medium devsel, latency 0, IRQ 16
>         I/O ports at 2200 [size=32]
> 
> 00:1d.1 USB Controller: Intel Corp. 82801EB/ER (ICH5/ICH5R) USB UHCI #2 
> (rev 02) (prog-if 00 [UHCI])
>         Subsystem: IBM: Unknown device 02dd
>         Flags: bus master, medium devsel, latency 0, IRQ 19
>         I/O ports at 2600 [size=32]
> 
> 00:1d.7 USB Controller: Intel Corp. 82801EB/ER (ICH5/ICH5R) USB2 EHCI 
> Controller (rev 02) (prog-if 20 [EHCI])
>         Subsystem: IBM: Unknown device 02dd
>         Flags: bus master, medium devsel, latency 0, IRQ 23
>         Memory at f0000000 (32-bit, non-prefetchable) [size=1K]
>         Capabilities: [50] Power Management version 2
>         Capabilities: [58] #0a [20a0]
> 
> 00:1e.0 PCI bridge: Intel Corp. 82801BA/CA/DB/EB/ER Hub interface to PCI 
> Bridge (rev c2) (prog-if 00 [Normal decode])
>         Flags: bus master, fast devsel, latency 0
>         Bus: primary=00, secondary=01, subordinate=01, sec-latency=32
>         I/O behind bridge: 00003000-00003fff
>         Memory behind bridge: de000000-dfffffff
>         Prefetchable memory behind bridge: d0000000-ddffffff
> 
> 00:1f.0 ISA bridge: Intel Corp. 82801EB/ER (ICH5/ICH5R) LPC Bridge (rev 02)
>         Flags: bus master, medium devsel, latency 0
> 
> 00:1f.1 IDE interface: Intel Corp. 82801EB/ER (ICH5/ICH5R) Ultra ATA 100 
> Storage Controller (rev 02) (prog-if 8a [Master SecP PriP])
>         Subsystem: IBM: Unknown device 02dd
>         Flags: bus master, medium devsel, latency 0, IRQ 17
>         I/O ports at <ignored>
>         I/O ports at <ignored>
>         I/O ports at <ignored>
>         I/O ports at <ignored>
>         I/O ports at 0480 [size=16]
>         Memory at d8000000 (32-bit, non-prefetchable) [size=1K]
> 
> 00:1f.3 SMBus: Intel Corp. 82801EB/ER (ICH5/ICH5R) SMBus Controller (rev 
> 02)
>         Subsystem: IBM: Unknown device 02dd
>         Flags: medium devsel, IRQ 17
>         I/O ports at 0440 [size=32]
> 
> 01:06.0 VGA compatible controller: ATI Technologies Inc Radeon RV100 QY 
> [Radeon 7000/VE] (prog-if 00 [VGA])
>         Subsystem: IBM: Unknown device 02c8
>         Flags: bus master, stepping, medium devsel, latency 64, IRQ 20
>         Memory at d0000000 (32-bit, prefetchable) [size=128M]
>         I/O ports at 3000 [size=256]
>         Memory at dfff0000 (32-bit, non-prefetchable) [size=64K]
>         Expansion ROM at <unassigned> [disabled] [size=128K]
>         Capabilities: [50] Power Management version 2
> 
> 02:00.0 PCI bridge: Intel Corp. PCI Bridge Hub A (rev 09) (prog-if 00 
> [Normal decode])
>         Flags: bus master, fast devsel, latency 0
>         Bus: primary=02, secondary=03, subordinate=03, sec-latency=48
>         I/O behind bridge: 00004000-00004fff
>         Memory behind bridge: ce000000-cfffffff
>         Capabilities: [44] #10 [0071]
>         Capabilities: [5c] Message Signalled Interrupts: 64bit+ 
> Queue=0/0 Enable-
>         Capabilities: [6c] Power Management version 2
>         Capabilities: [d8] PCI-X non-bridge device.
> 
> 02:00.2 PCI bridge: Intel Corp. PCI Bridge Hub B (rev 09) (prog-if 00 
> [Normal decode])
>         Flags: bus master, fast devsel, latency 0
>         Bus: primary=02, secondary=04, subordinate=04, sec-latency=64
>         Capabilities: [44] #10 [0071]
>         Capabilities: [5c] Message Signalled Interrupts: 64bit+ 
> Queue=0/0 Enable-
>         Capabilities: [6c] Power Management version 2
>         Capabilities: [d8] PCI-X non-bridge device.
> 
> 03:03.0 Ethernet controller: Intel Corp. 82557/8/9 [Ethernet Pro 100] 
> (rev 0c)
>         Subsystem: IBM 10/100 Ethernet Server Adapter
>         Flags: bus master, medium devsel, latency 64, IRQ 72
>         Memory at cffff000 (32-bit, non-prefetchable) [size=4K]
>         I/O ports at 4000 [size=64]
>         Memory at cffc0000 (32-bit, non-prefetchable) [size=128K]
>         Expansion ROM at <unassigned> [disabled] [size=64K]
>         Capabilities: [dc] Power Management version 2
> 
> 05:00.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5721 
> Gigabit Ethernet PCI Express (rev 01)
>         Subsystem: IBM: Unknown device 02c6
>         Flags: bus master, fast devsel, latency 0, IRQ 16
>         Memory at cdff0000 (64-bit, non-prefetchable) [size=64K]
>         Capabilities: [48] Power Management version 2
>         Capabilities: [50] Vital Product Data
>         Capabilities: [58] Message Signalled Interrupts: 64bit+ 
> Queue=0/3 Enable-
>         Capabilities: [d0] #10 [0001]
> 
> 06:00.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5721 
> Gigabit Ethernet PCI Express (rev 01)
>         Subsystem: IBM: Unknown device 02c6
>         Flags: bus master, fast devsel, latency 0, IRQ 16
>         Memory at cbff0000 (64-bit, non-prefetchable) [size=64K]
>         Capabilities: [48] Power Management version 2
>         Capabilities: [50] Vital Product Data
>         Capabilities: [58] Message Signalled Interrupts: 64bit+ 
> Queue=0/3 Enable-
>         Capabilities: [d0] #10 [0001]
> 
> 07:00.0 PCI bridge: Intel Corp. 80332 [Dobson] I/O processor (rev 07) 
> (prog-if 00 [Normal decode])
>         Flags: bus master, fast devsel, latency 0
>         Bus: primary=07, secondary=08, subordinate=08, sec-latency=64
>         Memory behind bridge: c8000000-c9ffffff
>         Capabilities: [44] #10 [0071]
>         Capabilities: [5c] Message Signalled Interrupts: 64bit+ 
> Queue=0/0 Enable-
>         Capabilities: [6c] Power Management version 2
>         Capabilities: [d8] PCI-X non-bridge device.
> 
> 07:00.2 PCI bridge: Intel Corp. 80332 [Dobson] I/O processor (rev 07) 
> (prog-if 00 [Normal decode])
>         Flags: bus master, fast devsel, latency 0
>         Bus: primary=07, secondary=09, subordinate=09, sec-latency=64
>         I/O behind bridge: 00005000-0000ffff
>         Capabilities: [44] #10 [0071]
>         Capabilities: [5c] Message Signalled Interrupts: 64bit+ 
> Queue=0/0 Enable-
>         Capabilities: [6c] Power Management version 2
>         Capabilities: [d8] PCI-X non-bridge device.
> 
> 08:0e.0 RAID bus controller: Adaptec ServeRAID Controller (rev 07)
>         Subsystem: IBM: Unknown device 028e
>         Flags: bus master, stepping, 66Mhz, medium devsel, latency 64, 
> IRQ 38
>         Memory at c9fff000 (32-bit, non-prefetchable) [size=4K]
>         Expansion ROM at <unassigned> [disabled] [size=512K]
>         Capabilities: [c0] Power Management version 2
>         Capabilities: [d0] Message Signalled Interrupts: 64bit+ 
> Queue=0/1 Enable-
>         Capabilities: [e0] PCI-X non-bridge device.
> 
> 
> ===SNIP===
> 
> Regards,
> 
> Jimmy Stewpot.
> 




More information about the redhat-list mailing list