[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

Huge amount of used inode handlers reported by sar -v (inode-sz)



Any help with this problem would be very much appreciated (even "it's not
7.3 or ext3 pointers, look somewhere else").

I've seen a similar post to ext3-users, but since that one received no reply
and I'm not convinced it's a ext3 problem (it only appears on our 7.3 hosts)
, I'm CCing to the valhalla list.

We have the same problem on ALL our Redhat 7.3 machines (various dual
processor Dell 2400, 2500, 2600 machines with RAID cards). There appears to
be no consistent time it starts (except that it's always out of business
hours 9-5) but once it does the machine eventually dies. At first we though
it was our amanda backups, but they're not always running when it starts.
We've tightened the machines down as far as we can afford and patched one to
the latest rpm and kernel versions available from redhat but no joy. Once it
starts the file opens become unreliable and syslog and other processes that
rely on sockets start to behave in strange ways. At one stage, on our amanda
host the amanda backup kept going long after everything else stopped
working, until it needed to rename the log files and then it died too
(amanda keeps its log file open the entire time until the end, unlike
syslog).

I've read through the archives for ext3 and valhalla and only found one
email concerning this problem (no reply) and looked through the Redhat
errata, and google etc. I've also checked the proc filesystem and can't find
any large numbers in inode-nr etc.

The only way I've found to get rid of the problem is a reboot.


Here's a copy of the sar output from one host. Note the interesting
dentunusd values at one stage.

00:01:01    dentunusd   file-sz  %file-sz  inode-sz  super-sz %super-sz
dquot-sz %dquot-sz  rtsig-sz %rtsig-sz
01:01:01       351831       506      0.24 4252446106         0      0.00
0      0.00         2      0.20
01:06:01       351832       469      0.22 4252446106         0      0.00
0      0.00         2      0.20
01:11:01       351832       507      0.24 4252446106         0      0.00
0      0.00         2      0.20
01:16:01       351833       507      0.24 4252446106         0      0.00
0      0.00         2      0.20
01:21:01       351834       467      0.22 4252446106         0      0.00
0      0.00         2      0.20
01:26:01       351835       508      0.24 4252446106         0      0.00
0      0.00         2      0.20
01:31:01    4294965457       461      0.22 4251971351         0      0.00
0      0.00         2      0.20
01:36:01    4294965457       460      0.22 4251971351         0      0.00
0      0.00         2      0.20
01:41:01    4294965461       459      0.22 4251971356         0      0.00
0      0.00         2      0.20
*
* deleted to save bandwidth
*
03:36:01    4294966740       509      0.24 4251971696         0      0.00
0      0.00         2      0.20
03:41:01    4294966741       508      0.24 4251971696         0      0.00
0      0.00         2      0.20
03:46:01    4294965710       468      0.22 4251971527         0      0.00
0      0.00         2      0.20
03:51:01    4294965736       507      0.24 4251971527         0      0.00
0      0.00         2      0.20
03:56:01    4294965752       509      0.24 4251971539         0      0.00
0      0.00         2      0.20
04:01:00    4294965763       508      0.24 4251971546         0      0.00
0      0.00         2      0.20
04:06:00       227450       470      0.22 4252135348         0      0.00
0      0.00         2      0.20
04:11:00       227935       470      0.22 4251950501         0      0.00
0      0.00         2      0.20
04:16:01       203080       472      0.23 4251887721


And another host (note how fast it happens, it's not a gradual build up).

00:01:00    dentunusd   file-sz  %file-sz  inode-sz  super-sz %super-sz
dquot-sz %dquot-sz  rtsig-sz %rtsig-sz
*
* boring stuff edited out
*
04:50:59        64932       992      0.95     61428         0      0.00
0      0.00         2      0.20
04:55:59        64947       992      0.95     61442         0      0.00
0      0.00         2      0.20
05:01:01        64970       992      0.95     61461         0      0.00
0      0.00         2      0.20
05:06:01        65098       983      0.94     61312         0      0.00
0      0.00         2      0.20
05:11:01        65121       983      0.94     61314         0      0.00
0      0.00         2      0.20
05:16:01           68       977      0.93 4294960298         0      0.00
0      0.00         3      0.29
05:21:01          622       992      0.95 4294960717         0      0.00
0      0.00         2      0.20
05:26:01         1252      1153      1.10 4294961116         0      0.00
0      0.00         1      0.10
05:31:01         1500      1175      1.12 4294961376         0      0.00
0      0.00         1      0.10
05:36:01         1499      1160      1.11 4294961380         0      0.00
0      0.00         1      0.10
05:41:01         1500      1160      1.11 4294961380         0      0.00
0      0.00         1      0.10
05:46:01         1503      1175      1.12 4294961376         0      0.00
0      0.00         1      0



One very strange thing, the average line from sar for the last one  is

Average:         9306       842      0.80      4298         0      0.00
0      0.00         1      0.10


But given that the sar file only has less than 50% of inode-sz values less
than 4 billion I'm a little perplexed by this line.






[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]