[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

Re: [Linux-cluster] GFS Performance Problems (RHEL5)




Hi James,

Like I said in my last email, my M500i has been swell so far, but I'm only using one interface. In regards to your problems though, did you ever call Promise to get help? I haven't had a big need to call them in the past, but when I have, they've been extremely helpful.

My thinking is that this is related to the glock problem that I mentioned in my previous email. The reason I think this is because there are two large filesystems on two separate NAS's that I'm trying to combine into one. One NAS has a directory structure with a max of about 1000 files per directory. When I transferred those files via rsync I got great performance with no stalling.

The other NAS has directories with 5000+ files, and it was only after those directories started showing up that the system began stalling. The problem is, now that those directories are here, the whole filesystem stalls, whether I'm accessing those directories or not.

The thing that's confusing to me is that, according to the writeup I read (http://people.redhat.com/wcheng/Patches/GFS/readme.gfs_glock_trimming.R4) the problem is with glocks. From what I understand, a certain number of glocks are allocated and then the system needs to "garbage collect," which causes the pauses. This indicates to me that there is a glock table that must be cleaned up on a regular basis. Apparently I'm running out of lock space and that's why my system is pausing. My question is, how big is this table? Can it be increased? I've got a machine with 4G of ram and 2.5G are free. I would think that if I could allocate another gig for the glock table and simply let gfs_scand run constantly at 10% of CPU, I might be able to prevent the stalls.

(I'm not asking you, James, for the answer, but I'm wondering if anybody else can chime in.)

Paul


James Chamberlain wrote:
Hi Paul,

In my experience with the VTrak M500i, it didn't seem like it could handle active multipathing. When I tried to use both interfaces simultaneously rather than fail over between them, my throughput to the disks dropped to less than 1 MB/s. It looks like they've made some improvements in the VTrak M610i, such as link aggregation, so this may not be applicable for your newer hardware.

You might want to check the management interface of your VTrak and see if anything interesting is going on when everything freezes up on you. You might also want to check dmesg and your syslogs to see if you see anything about the iSCSI session being lost and reestablished.

Regards,

James Chamberlain


[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]