[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

Re: [Linux-cluster] Why GFS is so slow? What it is waiting for?

Ja S wrote:
Hi, Wendy:

Thanks for your so prompt and kind explanation. It is
very helpful. According to your comments, I did
another test. See below:
# stat abc/
  File: `abc/'
  Size: 8192            Blocks: 6024       IO Block:
4096   directory
Device: fc00h/64512d    Inode: 1065226     Links: 2
Access: (0770/drwxrwx---) Uid: ( 0/ root) Gid: ( 0/ root)
Access: 2008-05-08 06:18:58.000000000 +0000
Modify: 2008-04-15 03:02:24.000000000 +0000
Change: 2008-04-15 07:11:52.000000000 +0000

# cd abc/
# time ls | wc -l 31764

real    0m44.797s
user    0m0.189s
sys     0m2.276s

The real time in this test is much shorter than the
previous one. However, it is still reasonable long. As
you said, the ‘ls’ command only reads the single
directory file. In my case, the directory file itself
is only 8192 bytes. The time spent on disk IO should
be included in “sys 0m2.276s”. Although DLM needs time
to lookup the location of the corresponding master
lock resource and to process locking, the system
should not take about 42 seconds to complete the “ls”
command. So, what is the hidden issue or is there a
way to identify possible bottlenecks?
IIRC, disk IO wait time is excluded from "sys", so you really can't conclude the lion share of your wall (real) time is due to DLM locking. We don't know for sure unless you can provide the relevant profiling data (try to learn how to use OProfile and/or SystemTap to see where exactly your system is waiting at). Latency issues like this is tricky. It would be foolish to conclude anything just by reading the command output without knowing the surrounding configuration and/or run time environment.

If small file read latency is important to you, did you turn off storage device's readahead ? Did you try different Linux kernel elevator algorithms ? Did you make sure your other network traffic didn't block DLM traffic ? Be aware latency and bandwidth are two different things. A big and fat network link doesn't automatically imply a quick response time though it may carry more bandwidth.

-- Wendy

[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]