[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

[redhat-lspp] LSPP kernel performance



Hi,

I was doing some testing this weekend to see how the performance of the lspp 
kernel is. Well...not so good. The short story is that I think some 
algorithms need to be changed.

I put the test program at:

http://people.redhat.com/sgrubb/files/lspp-perf.tar.gz

In all tests that use audit, I am using the lspp rules. The first test, I ran 
a normal FC-5 kernel without audit enabled. (To disable audit, you have to 
chkconfig --del audit and reboot.) This is what I got:

real    6m52.431s
user    0m1.436s
sys     6m50.754s

looping 25000000 times on /usr/include
102716 total                                      0.0433
 86155 check_poison_obj                         206.1124
  2770 kmem_cache_free                            7.4462
  1770 __d_lookup                                 6.5314
  1332 __link_path_walk                           0.3527
  1277 avc_has_perm_noaudit                       1.4561
  1248 memset                                     6.5000
   625 _raw_spin_lock                             2.4606
   622 do_path_lookup                             0.8556
   549 dput                                       1.2708

Then I turned on audit, rebooted, and got this:

real    7m10.989s
user    0m1.520s
sys     7m9.239s

looping 25000000 times on /usr/include
107334 total                                      0.0453
 85985 check_poison_obj                         205.7057
  2760 kmem_cache_free                            7.4194
  1683 __d_lookup                                 6.2103
  1598 avc_has_perm_noaudit                       1.8221
  1428 __link_path_walk                           0.3781
  1256 memset                                     6.5417
  1254 audit_filter_syscall                       7.4201
   837 _raw_spin_lock                             3.2953
   763 _atomic_dec_and_lock                       9.0833
   700 do_path_lookup                             0.9629

That works out to a 4.6% performance hit. This is about what I expected. next 
I ran the lspp.13 kernel without the audit system enabled and got this:

real    6m46.056s
user    0m1.664s
sys     6m44.125s

looping 25000000 times on /usr/include
101063 total                                      0.0423
 86093 check_poison_obj                         206.4580
  2200 kmem_cache_free                            5.9140
  1422 __d_lookup                                 5.2472
  1361 __link_path_walk                           0.3603
  1286 avc_has_perm_noaudit                       1.4664
  1174 memset                                     6.1146
   726 do_path_lookup                             0.9732
   632 _raw_spin_lock                             2.4882
   576 _atomic_dec_and_lock                       6.8571
   508 dput                                       1.1759

Its performing as good as the regular FC5 kernel if not slightly better. Next, 
I turned on auditing and rebooted.

real    8m36.692s
user    0m1.568s
sys     8m34.820s

looping 25000000 times on /usr/include
128681 total                                      0.0539
 87784 check_poison_obj                         210.5132
  5643 kfree                                     12.5400
  3297 context_struct_to_string                   8.2839
  2448 kmem_cache_free                            6.5806
  2344 vsnprintf                                  1.6255
  1878 __d_lookup                                 6.9299
  1681 memset                                     8.7552
  1300 __kmalloc                                  4.1401
  1108 __link_path_walk                           0.2934
  1026 audit_filter_syscall                       6.0000
   977 strnlen                                   37.5769
   972 avc_has_perm_noaudit                       1.1083
   970 do_path_lookup                             1.3003
   925 audit_syscall_exit                         0.9204
   787 _raw_spin_lock                             3.0984
   767 _atomic_dec_and_lock                       9.1310
   732 mls_sid_to_context                         1.2099

That took much longer to run. This is about a 25.5% performance hit. The thing 
to notice is that context_struct_to_string  is the bottleneck in the whole 
operation. I think it is also responsible for calling vsnprintf, strlen, and 
mls_sid_to_context.

Admittedly, I'm perf testing a kernel with slab debug turned on, but I 
question the wisdom of getting all the SE Linux contexts converted to strings 
during normal syscall processing. I think we have to change this. We cannot 
accept a 25% performance degradation.

I think the audit code needs to be changed so that it only collects sid's for 
subject and object and converts them to text only at the point where we are 
writing an audit message. I also wonder about calling all those SE Linux 
functions twice...one to get the length and then another to do it for real. I 
also looked at the context_struct_to_string  function and see that it calls 
strlen for each piece of the context to calculate the length. It seems the 
length could be part of the policydb structure so that its set at load and 
just referenced later.

Please try the program and see if you get similar results. With most of the 
kernel work done, I think we have to look at tuning it now. There is a README 
file in the test program explaining how to set up for the test. I will 
probably make a new lspp test kernel with slab debug turned off to get that 
out of the way.

-Steve


[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]