[Linux-cluster] Why GFS is so slow? What it is waiting for?

[Wendy Cheng]

Ja S wrote:
> Hi, All:
>
> I used to post this question before, but have not
> received any comments yet. Please allow me post it
> again.
>
> I have a subdirectory containing more than 30,000
> small files on a SAN storage (GFS1+DLM, RAID10). No
> user application knows the existence of the
> subdirectory. In other words, the subdirectory is free
> of accessing. 
>   
Short answer is to remember "ls" and "ls -la" are very different 
commands. "ls" is a directory read (that reads from one single file) but 
"ls -la" needs to get file attributes (file size, modification times, 
ownership, etc) from *each* of the files from the subject directory. In 
your case, it needs to read more than 30,000 inodes to get them. The "ls 
-la" is slower for *any* filesystem but particularly troublesome for a 
cluster filesystem such as GFS due to:

1. Cluster locking overheads (it needs readlocks from *each* of the 
files involved).
2. Depending on when and how these files are created. During file 
creation time and if there are lock contentions, GFS has a tendency to 
spread the file locations all over the disk.
3. You use iscsi such that dlm lock traffic and file block access are on 
the same fabric ?  If this is true, you will more or less serialize the 
lock access.

Hope above short answer will ease your confusion.

-- Wendy
> However, it took ages to list the subdirectory on an
> absolute idle cluster node. See below:
>
> # time ls -la | wc -l
> 31767
>
> real    3m5.249s
> user    0m0.628s
> sys     0m5.137s
>
> There are about 3 minutes spent on somewhere. Does
> anyone have any clue what the system was waiting for?
>
>
> Thanks for your time and wish to see your valuable
> comments soon.
>
> Jas
>
>
>       ____________________________________________________________________________________
> Be a better friend, newshound, and 
> know-it-all with Yahoo! Mobile.  Try it now.  http://mobile.yahoo.com/;_ylt=Ahu06i62sR8HDtDypao8Wcj9tAcJ
>
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster
>