Issues with Performance
Job Cacka
job at ccbmail.ccbox.com
Tue Feb 14 16:19:40 UTC 2006
Jimmy,
This looks like a tough question and you have done a lot of
troubleshooting already. One thing that I noticed you did not mention was
checking the physical cable. Is it possible your CAT 5, 5e, or 6 cable is
misrated or damaged? Also, if this is going through the wall you might try
an alternative cable run. I know this seems very basic, but to me it sounds
like you are running in half duplex mode even though you are set to full
duplex. If it is imposible to try a different cable run then call up someone
with some good CAT 5 testing equipment, and have them check it out. The guys
with HP or Agilent testers are usually the best, but maybe that is more so
in my area.
Job Cacka
-----Original Message-----
From: redhat-list-bounces at redhat.com
[mailto:redhat-list-bounces at redhat.com]On Behalf Of Jimmy Stewpot
Sent: Monday, February 13, 2006 7:45 AM
To: redhat-list at redhat.com
Subject: Issues with Performance
Hello,
I am currently trying to diagnose a particularly strange issue with a
Redhat Enterprise Linux v3.0 Update 5 system. The particular issue is as
follows.
I am attempting to run an ICMP monitoring package from the system to
monitor all of the devices around a network. (circa 10,000). When the
ICMP monitoring software is enabled the server becomes unusable via SSH.
By saying unusable I mean the system is very laggy often taking several
minutes to respond to a command. However if I am on the system console
the system is still usable and responds with NO delay.
Originally I suspected that the issue was related to system load or
network congestion. However I started to monitor the switch ports on the
catalyst 6509 ports with snmp and the utilisation never reaches above 1Mbps.
The system is an IBM x336 with dual 2.8Ghz XEONs, 4gb ram and SCSI
drives it has dual port broad com 10/100/1000 interfaces. If I monitor
iostat/vmstat or top I see that the system averages around 90% idle. If
I look at SAR for a long term performance report I see similar information.
On the server it directly connects to the 6509 @ 100Mbps Full duplex
there are no errors on the interface using the bcm5700 driver provided
by broadcom. Since looking into the issue I initially thought that the
problem was with the SSH but the problem appears to be more wide spread.
If I do a rspan on the interface that the server is connected to I see
that the Ethernet frames go both directions however If I do a tcpdump on
the server I am not seeing the packets with the syntax
'tcpdump -vvv -n icmp'.
The SSH sessions and the ICMP return packets not being seen I am sure
are related. In all cases the ports have been forced to 100-FULL Duplex
and I have tested at several other port link speeds and modes without
any success.
One thing that is slightly different to many configurations is that the
gateway is running CatOS, it then routes in hybrid mode and has VRRP
configured on the device's gateway. However I do not see why that would
cause any problems as the routing is not flipping and there are no layer
2/3 issues that I can detect.
What I am trying to diagnose/determine is why the Linux box would not
be seeing the packets returning even though we see them on an rspan as
leaving the switch interface. There are no iptables rules on the server
at all and from what I can see the sysctl stuff is all setup correctly.
When I turn on the debug syslog I still get no error messages or
notifications.
Where should I be looking now and is there any advice that you guys can
give me?
Regards,
Jimmy.
===System Information===
/sbin/lspci -v
00:00.0 Host bridge: Intel Corp. Server Memory Controller Hub (rev 0a)
Subsystem: IBM: Unknown device 02dd
Flags: bus master, fast devsel, latency 0
Memory at <ignored> (32-bit, non-prefetchable) [size=4K]
Capabilities: <available only to root>
00:00.1 Class ff00: Intel Corp. Memory Controller Hub Error Reporting
Register (rev 0a)
Subsystem: IBM: Unknown device 02dd
Flags: fast devsel
00:02.0 PCI bridge: Intel Corp. Memory Controller Hub PCI Express Port
A0 (rev 0a) (prog-if 00 [Normal decode])
Flags: bus master, fast devsel, latency 0
Bus: primary=00, secondary=02, subordinate=04, sec-latency=0
Capabilities: <available only to root>
00:04.0 PCI bridge: Intel Corp. Memory Controller Hub PCI Express Port
B0 (rev 0a) (prog-if 00 [Normal decode])
Flags: bus master, fast devsel, latency 0
Bus: primary=00, secondary=05, subordinate=05, sec-latency=0
Memory behind bridge: ce000000-cfffffff
Capabilities: <available only to root>
00:05.0 PCI bridge: Intel Corp. Memory Controller Hub PCI Express Port
B1 (rev 0a) (prog-if 00 [Normal decode])
Flags: bus master, fast devsel, latency 0
Bus: primary=00, secondary=06, subordinate=06, sec-latency=0
Memory behind bridge: cc000000-cdffffff
Capabilities: <available only to root>
00:06.0 PCI bridge: Intel Corp. Memory Controller Hub PCI Express Port
C0 (rev 0a) (prog-if 00 [Normal decode])
Flags: bus master, fast devsel, latency 0
Bus: primary=00, secondary=07, subordinate=09, sec-latency=0
I/O behind bridge: 00004000-0000ffff
Memory behind bridge: ca000000-cbffffff
Capabilities: <available only to root>
00:08.0 System peripheral: Intel Corp. Memory Controller Hub Extended
Configuration Registers (rev 0a)
Subsystem: IBM: Unknown device 02dd
Flags: fast devsel
00:1d.0 USB Controller: Intel Corp. 82801EB/ER (ICH5/ICH5R) USB UHCI #1
(rev 02) (prog-if 00 [UHCI])
Subsystem: IBM: Unknown device 02dd
Flags: bus master, medium devsel, latency 0, IRQ 16
I/O ports at 2200 [size=32]
00:1d.1 USB Controller: Intel Corp. 82801EB/ER (ICH5/ICH5R) USB UHCI #2
(rev 02) (prog-if 00 [UHCI])
Subsystem: IBM: Unknown device 02dd
Flags: bus master, medium devsel, latency 0, IRQ 19
I/O ports at 2600 [size=32]
00:1d.7 USB Controller: Intel Corp. 82801EB/ER (ICH5/ICH5R) USB2 EHCI
Controller (rev 02) (prog-if 20 [EHCI])
Subsystem: IBM: Unknown device 02dd
Flags: bus master, medium devsel, latency 0, IRQ 23
Memory at f0000000 (32-bit, non-prefetchable) [size=1K]
Capabilities: <available only to root>
00:1e.0 PCI bridge: Intel Corp. 82801BA/CA/DB/EB/ER Hub interface to PCI
Bridge (rev c2) (prog-if 00 [Normal decode])
Flags: bus master, fast devsel, latency 0
Bus: primary=00, secondary=01, subordinate=01, sec-latency=32
I/O behind bridge: 00003000-00003fff
Memory behind bridge: de000000-dfffffff
Prefetchable memory behind bridge: d0000000-ddffffff
00:1f.0 ISA bridge: Intel Corp. 82801EB/ER (ICH5/ICH5R) LPC Bridge (rev 02)
Flags: bus master, medium devsel, latency 0
00:1f.1 IDE interface: Intel Corp. 82801EB/ER (ICH5/ICH5R) Ultra ATA 100
Storage Controller (rev 02) (prog-if 8a [Master SecP PriP])
Subsystem: IBM: Unknown device 02dd
Flags: bus master, medium devsel, latency 0, IRQ 17
I/O ports at <ignored>
I/O ports at <ignored>
I/O ports at <ignored>
I/O ports at <ignored>
I/O ports at 0480 [size=16]
Memory at c8000000 (32-bit, non-prefetchable) [size=1K]
00:1f.3 SMBus: Intel Corp. 82801EB/ER (ICH5/ICH5R) SMBus Controller (rev 02)
Subsystem: IBM: Unknown device 02dd
Flags: medium devsel, IRQ 17
I/O ports at 0440 [size=32]
01:06.0 VGA compatible controller: ATI Technologies Inc Radeon RV100 QY
[Radeon 7000/VE] (prog-if 00 [VGA])
Subsystem: IBM: Unknown device 02c8
Flags: bus master, stepping, medium devsel, latency 64, IRQ 20
Memory at d0000000 (32-bit, prefetchable) [size=128M]
I/O ports at 3000 [size=256]
Memory at dfff0000 (32-bit, non-prefetchable) [size=64K]
Expansion ROM at <unassigned> [disabled] [size=128K]
Capabilities: <available only to root>
02:00.0 PCI bridge: Intel Corp. PCI Bridge Hub A (rev 09) (prog-if 00
[Normal decode])
Flags: bus master, fast devsel, latency 0
Bus: primary=02, secondary=03, subordinate=03, sec-latency=64
Capabilities: <available only to root>
02:00.2 PCI bridge: Intel Corp. PCI Bridge Hub B (rev 09) (prog-if 00
[Normal decode])
Flags: bus master, fast devsel, latency 0
Bus: primary=02, secondary=04, subordinate=04, sec-latency=64
Capabilities: <available only to root>
05:00.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5721
Gigabit Ethernet PCI Express (rev 01)
Subsystem: IBM: Unknown device 02c6
Flags: bus master, fast devsel, latency 0, IRQ 16
Memory at cfff0000 (64-bit, non-prefetchable) [size=64K]
Capabilities: <available only to root>
06:00.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5721
Gigabit Ethernet PCI Express (rev 01)
Subsystem: IBM: Unknown device 02c6
Flags: bus master, fast devsel, latency 0, IRQ 16
Memory at cdff0000 (64-bit, non-prefetchable) [size=64K]
Capabilities: <available only to root>
07:00.0 PCI bridge: Intel Corp. 80332 [Dobson] I/O processor (rev 07)
(prog-if 00 [Normal decode])
Flags: bus master, fast devsel, latency 0
Bus: primary=07, secondary=08, subordinate=08, sec-latency=64
Memory behind bridge: ca000000-cbffffff
Capabilities: <available only to root>
07:00.2 PCI bridge: Intel Corp. 80332 [Dobson] I/O processor (rev 07)
(prog-if 00 [Normal decode])
Flags: bus master, fast devsel, latency 0
Bus: primary=07, secondary=09, subordinate=09, sec-latency=64
I/O behind bridge: 00004000-0000ffff
Capabilities: <available only to root>
08:0e.0 RAID bus controller: Adaptec ServeRAID Controller (rev 07)
Subsystem: IBM: Unknown device 028e
Flags: bus master, stepping, 66Mhz, medium devsel, latency 64,
IRQ 38
Memory at cbfff000 (32-bit, non-prefetchable) [size=4K]
Expansion ROM at <unassigned> [disabled] [size=512K]
Capabilities: <available only to root>
Module Size Used by Tainted: P
bcm5700 105380 2
ide-cd 34016 0 (autoclean)
cdrom 32608 0 (autoclean) [ide-cd]
sg 37260 0 (autoclean)
autofs4 16984 0 (autoclean) (unused)
audit 90584 2
basp 62020 0 (unused)
microcode 6912 0 (autoclean)
keybdev 2976 0 (unused)
mousedev 5656 0 (unused)
hid 22308 0 (unused)
input 6176 0 [keybdev mousedev hid]
ehci-hcd 20776 0 (unused)
usb-uhci 26988 0 (unused)
usbcore 81088 1 [hid ehci-hcd usb-uhci]
ext3 89960 4
jbd 55060 4 [ext3]
ips 45252 5
sd_mod 13712 10
scsi_mod 114344 3 [sg ips sd_mod]
/sbin/mii-tool
eth0: 100 Mbit, full duplex, link ok
eth1: 100 Mbit, full duplex, link ok
--
redhat-list mailing list
unsubscribe mailto:redhat-list-request at redhat.com?subject=unsubscribe
https://www.redhat.com/mailman/listinfo/redhat-list
More information about the redhat-list
mailing list