SELinux DoS?

[David]

Hey Guys!

Some colleagues and I tested the behavior of SELinux due to a project on
the university. So we wrote a little test program and the necessary
policies.
Overall it works fine, but we build in a bug in our test program which
offers the exploitation through a stack based buffer overflow.
When we tried to getting a root shell based on this bug (the demo tool
has set suid bit), SELinux prevents the execution of the shell, but the 
demo program will not be quitted.
It hangs at the point of trying to open the shell and SELinux writes 
endless log entries to /var/log/audit/audit.log.

We assumed that this behavior will occur due to following actions:

- demo tool tries to open a shell via shellcode, occurred through a 
buffer overflow.
- selinux prevents this execution.
- the function-call in demo tool tries to jump back to the return address,
- but the address is overwritten through the bof.
- so, it jumps to the buffer and tries to open a shell again.
All together in a endless loop.

This behavior seems to be alright from technical aspect, but should this 
be the behavior of selinux? Or is there an option which instructs 
selinux to kill processes which tries pass over there contexts too often?

For instance, this manner could be easy used for DoS Attacks. Our tests
exhibits that the execution of many demo program instances will make the
system unusable.

Any ideas about this behavior, or any solution?


David

(Tested on FC9.)