[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

Re: ext3 performance mystery



Just a "me too" post... I've noticed somewhat similar behavior, but with
a different test.

I ran an application that forked off two threads on dual processor
workstation. One thread tried to write data once every 2 seconds , and
the other tried to read data constantly (although it was writing a tiny
bit as well because of file atime updates). Both apps were contending
for the same physical device. When using ext3, the writes would be
delayed for a fairly substantial arbitrary amount of time, and then they
many of them would go through all at once. This pattern would then
repeat. When using reiserfs, each write went through about every two
seconds.  

I chalked it up to an overall bandwidth vs. latency optimization
tradeoff. Sometimes though, in real world conditions when CPU usage is
very high, I've noticed odd latency problems. Sometimes after quitting
vi I'll be hanging for 10 seconds waiting to close out the file and
fsync, while I can work normally in other terminals.

On Tue, 2002-09-10 at 16:06, Max Hailperin wrote:
> Help solving a performance mystery would be welcome.
> 
> I have two test programs:
> 
>  *  one totally CPU bound (it just infinitely loops) 
>  *  the other I/O bound (it loops, each time doing
>      * a write, 
>      * an fsync or fdatasync (I get similar behavior with either),
>      * an lseek to back up to the same position again
> 
> Under ext2, if I run these two together, they each run nearly as fast
> as they would alone, since when the disk completes a write, the I/O
> bound process has been sleeping long enough to have gained priority
> over the CPU bound process, preempts for long enough to submit the next
> write request, and then goes back to sleep, letting the CPU bound one
> run while the disk does its thing.
> 
> Under ext3, I get a totally different, and very mysterious, behavior.
> Sometimes the I/O bound process will make very low rates of progress
> for tens of seconds.  For example, this is evident in the following
> "vmstat 1" output.  I start with just the I/O bound process running.
> Then at some point I start the CPU bound process, and the "bo" column
> drops precipitously.  After tens of seconds, the "bo" column goes back
> up again, for no apparent reason.  How long this takes to happens
> varies considerably.  Sometimes the I/O rate never dips in the first
> place, or at most only very briefly.
> 
> Can anyone explain this phenomenon?  I'm wondering if it has something
> to do with the need to schedule two different threads -- my I/O-bound
> benchmark and the kjournald.  But I can't think why either of them
> wouldn't gain priority.
> 
> max 30% vmstat 1
>    procs                      memory    swap          io     system         cpu
>  r  b  w   swpd   free   buff  cache  si  so    bi    bo   in    cs  us  sy  id
>  2  0  0  11064   2316  27888  30260   0   3    22   816  788  1441  34   5  61
>  3  1  1  11196   2332  27936  30224   0 132     0  1156  480   948  93   7   0
>  1  1  1  11196   2324  28008  30152   0   0     0  1100  346   945  97   3   0
>  2  0  1  11196   2320  27956  30208   0   0     0  1120  348   772  97   3   0
>  2  1  1  11196   2304  28032  30144   0   0     0  1160  348   716  99   1   0
>  1  1  1  11196   2304  28048  30128   0   0     0  1116  366   728  99   1   0
>  1  1  1  11196   2304  28080  30096   0   0     0  1116  345   691  96   4   0
>  1  1  1  11196   2304  28000  30176   0   0     0  1148  350   717  98   2   0
>  1  1  1  11196   2304  28028  30148   0   0     0  1132  342   707  98   2   0
>  0  1  1  11196   2356  27904  30272   0   0     0  1136  351   758  39   3  58
>  1  0  0  11196   2356  27904  30272   0   0     0  1152  356   758   1   3  96
>  2  0  0  11196   2336  27840  30304   0   0     0   820  287   585  34   2  64
>  2  0  0  11196   2332  27840  30304   0   0     0    92  146   192  99   1   0
>  2  0  0  11196   2332  27804  30340   0   0     0    76  154   198 100   0   0
>  2  0  0  11196   2332  27804  30340   0   0     0    64  142   176  99   1   0
>  2  0  0  11196   2332  27704  30440   0   0     0    64  147   180 100   0   0
>  2  0  0  11196   2332  27704  30440   0   0     0    80  147   179 100   0   0
>  4  0  0  11196   2332  27676  30468   0   0     0    64  145   176  99   1   0
>  3  0  0  11196   2336  27676  30468   0   0     0    64  145   187  99   1   0
>  2  0  0  11196   2336  27648  30496   0   0     0    72  141   180 100   0   0
>  2  0  0  11196   2336  27648  30496   0   0     0    68  143   186  99   1   0
>  4  0  0  11196   2332  27464  30680   0   0     0    64  142   168 100   0   0
>  3  0  0  11196   2332  27464  30680   0   0     0    72  144   181 100   0   0
>  3  0  0  11196   2332  27420  30724   0   0     0    64  141   181  99   1   0
>  2  0  0  11196   2324  27428  30724   0   0     0    64  141   175  99   1   0
>  2  0  0  11196   2324  27432  30720   0   0     0    72  146   189  99   1   0
>  3  0  0  11196   2324  27440  30712   0   0     0    64  140   169  99   1   0
>  3  0  0  11196   2324  27204  30944   0   0     0    64  143   177  99   1   0
>  3  0  0  11196   2324  27204  30940   0   0     0    72  141   190  99   1   0
>  2  0  0  11196   2324  27068  31076   0   0     0    64  146   177  99   1   0
>  3  0  0  11196   2324  27068  31076   0   0     0    64  146   177  99   1   0
>  2  0  0  11196   2324  26972  31172   0   0     0    72  150   182  98   2   0
>  2  0  0  11196   2324  26972  31172   0   0     0    64  146   181  98   2   0
>  2  0  0  11196   2324  26852  31292   0   0     0    64  148   174 100   0   0
>  2  0  0  11196   2324  26700  31444   0   0     0    76  157   195  99   1   0
>  2  0  0  11196   2324  26700  31440   0   0     0    64  148   174  99   1   0
>  2  0  0  11196   2324  26276  31864   0   0     0    64  142   172  99   1   0
>  4  0  0  11196   2324  26284  31856   0   0     0    72  145   189  99   1   0
>  2  0  0  11196   2324  26256  31884   0   0     0    64  144   171  99   1   0
>  3  0  0  11196   2324  26256  31884   0   0     0    64  155   190  99   1   0
>  2  0  0  11196   2324  26256  31884   0   0     0    72  142   179 100   0   0
>  2  0  0  11196   2324  26256  31884   0   0     0    64  141   177  98   2   0
>  5  0  0  11196   2324  26276  31864   0   0     0    64  149   189 100   0   0
>  2  0  0  11196   2324  26292  31844   0   0     0    72  143   180 100   0   0
>  2  0  0  11196   2324  26308  31828   0   0     0    68  144   187  99   1   0
>  2  0  0  11196   2324  26320  31816   0   0     0    64  140   173  99   1   0
>  3  0  0  11196   2324  26332  31804   0   0     0    72  145   180 100   0   0
>  2  0  0  11196   2324  26336  31800   0   0     0    60  142   181  99   1   0
>  2  0  0  11196   2324  26340  31796   0   0     0    60  159   174  99   1   0
>  2  0  0  11196   2324  26352  31780   0   0     0    72  154   188  99   1   0
>  2  0  0  11196   2324  26364  31768   0   0     0    60  141   177  99   1   0
>  3  0  0  11196   2324  26372  31760   0   0     0    64  140   173  99   1   0
>  3  0  0  11196   2324  26376  31756   0   0     0    72  144   183  99   1   0
>  3  0  0  11196   2324  26392  31740   0   0     0    60  140   175  99   1   0
>  4  0  0  11196   2324  26400  31732   0   0     0    64  143   178  99   1   0
>  3  0  0  11196   2324  26404  31728   0   0     0    72  142   181  99   1   0
>  3  0  0  11196   2324  26424  31708   0   0     0    64  142   178  99   1   0
>  4  0  0  11196   2324  26432  31696   0   0     0    64  142   187  99   1   0
>  3  0  0  11196   2324  26460  31668   0   0     0    72  143   180  99   1   0
>  3  0  0  11196   2324  26480  31648   0   0     0    68  145   189  99   1   0
>  4  0  0  11196   2324  26496  31632   0   0     0    64  143   175 100   0   0
>  3  0  0  11196   2324  26512  31616   0   0     0    72  143   180  99   1   0
>  3  0  0  11196   2324  25848  32280   0   0     0    64  142   181  99   1   0
>  3  0  0  11196   2324  25852  32276   0   0     0    72  170   193 100   0   0
>  3  0  0  11196   2324  25852  32276   0   0     0    72  123   178  99   1   0
>  4  0  0  11196   2324  25852  32276   0   0     0    64  137   166  99   1   0
>  4  0  0  11196   2324  25868  32260   0   0     0    68  144   172  99   1   0
>  6  0  0  11196   2324  25876  32252   0   0     0    72  141   182  99   1   0
>  5  0  0  11196   2324  25900  32232   0   0     0    64  142   180  99   1   0
>  4  0  0  11196   2324  25920  32212   0   0     0    68  158   184  99   1   0
>  3  0  0  11196   2324  25924  32208   0   0     0    72  144   188  99   1   0
>  5  0  0  11196   2324  25932  32196   0   0     0    64  142   175  99   1   0
>  5  0  0  11196   2324  25952  32176   0   0     0    68  151   191  99   1   0
>  5  0  0  11196   2324  25972  32156   0   0     0    72  142   185  99   1   0
>  4  0  0  11196   2324  26000  32128   0   0     0    72  153   184  98   2   0
>  3  0  0  11196   2324  26012  32116   0   0     0    68  142   181 100   0   0
>  4  0  0  11196   2324  26028  32100   0   0     0    72  143   183  99   1   0
>  5  0  0  11196   2324  26048  32080   0   0     0    64  151   185 100   0   0
>  5  0  0  11196   2324  26064  32060   0   0     0    72  171   205  98   2   0
>  3  0  0  11196   2324  26096  32024   0   0     0    72  122   172  99   1   0
>  3  0  0  11196   2324  26116  32004   0   0     0    64  139   172  99   1   0
>  3  0  0  11196   2324  26128  31992   0   0     0    68  147   182  99   1   0
>  3  0  0  11196   2324  26136  31984   0   0     0    72  143   184  98   2   0
>  3  0  0  11196   2324  26148  31972   0   0     0    64  142   176  98   2   0
>  4  0  0  11196   2324  26156  31964   0   0     0    68  145   185  99   1   0
>  3  0  0  11196   2324  26156  31960   0   0     0    72  147   174  99   1   0
>  3  0  0  11196   2324  26156  31960   0   0     0    64  142   183  99   1   0
>  4  0  0  11196   2324  26156  31960   0   0     0    68  142   177  99   1   0
>  3  0  0  11196   2324  26172  31944   0   0     0    72  150   178  98   2   0
>  4  0  0  11196   2324  26180  31936   0   0     0    64  145   179  97   3   0
>  4  0  0  11196   2324  26196  31852   0   0   324    68  177   216  99   1   0
>  2  1  1  11196   2388  25732  31980  24   0   212   652  269   490  99   1   0
>  2  1  1  11196   2360  25960  31780   0   0     0  1060  332   653 100   0   0
>  3  0  1  11192   2660  26252  31460   0   0     0  1164  350   739  97   3   0
>  1  1  1  11192   2568  26488  31316   0   0     0  1172  350   725  99   1   0
>  1  1  1  11192   2424  26020  31928   0   0     0  1176  359   724  98   2   0
>  2  1  1  11192   2364  26324  31684   0   0     0  1116  346   704  97   3   0
>  1  1  1  11192   2364  26584  31424   0   0     0  1116  340   692  98   2   0
>  2  1  1  11192   2364  26924  31084   0   0     0  1116  541  1280  94   6   0
>  2  1  1  11192   2364  27192  30816   0   0     0  1148  459  1119  97   3   0
> 
> 
> 
> _______________________________________________
> Ext3-users mailing list
> Ext3-users redhat com
> https://listman.redhat.com/mailman/listinfo/ext3-users






[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]