Security Technologies: ExecShield

This article was originally published on the Red Hat Customer Portal. The information may no longer be current.

The world of computer security has changed dramatically in the last few years. Keeping your operating system updated with the latest security patches is no longer sufficient. Operating system providers need to be more proactive in combating security problems. A majority of exploitable security flaws are due to memory corruption. ExecShield, a Red Hat-developed technology, included since Red Hat Enterprise Linux 3, aims to help protect systems from this type of exploitable security flaws.

Buffer Overflows

Buffer overflows are common mistakes found in programs written in the C or C++ programming languages and are generally very easy to exploit. In fact, there are semi-automated exploit-creation kits available on the Internet. When a function is called, the return address is first pushed on the stack. This is followed by any variables which are declared on the stack including buffers. On Intel and compatible processors, the stack grows in a downward direction over time (The image shows a simplistic, but inverted view of the memory, in which stack is showed to grow in the upward direction). This is why the buffer is stored before the return address. This buffer is stored on the stack and is located before the memory location containing the address of the program code that invoked the function. When the function is finished, the address is used to resume the program at the point of the function invocation.

If there is a stack-based buffer overflow flaw (CWE-121) in the application, the attacker can effectively write beyond the bounds of the buffer and could overwrite the return address. By overwriting the return address (which holds the address of the memory location of the code to execute when the function is complete), the exploit can control which code is executed when the function finishes. The simplest and most common approach is to make the return address point back into the buffer, which the attacker has filled with program code in the same step which caused the overflow. Such injected program code is often called "shellcode", as an attacker would typically start a shell such as bash, through which further commands can be ran.

Preventing Security flaws caused by buffer-overflow

An attacker can often change the return address of the function and point them to an area within the buffer which contains shellcode. The first logical step in countering buffer overflows is to ensure that return addresses only point to trusted program code and not to hostile externally injected program code. This is the approach that ExecShield and NX technology, provided by AMD and Intel, take. ExecShield approximates a separation of read and execute permissions by segment limits. The effect of applying segment limits is that the first N megabytes of the virtual memory of a process are executable, while the remaining virtual memory is not. The operating system kernel selects the value of N.

With such a segment limit in place, the operating system must make sure that all program code is located below this limit (the top side of the above picture) while data, especially the stack, should be located in the higher virtual memory addresses (the bottom side of the picture). When a violation of the execution permission happens, the program triggers a segmentation fault and terminates. This behavior is identical to when a program tries to violate read or write memory permissions or access unmapped memory addresses.

Intel and AMD NX Technology

Both AMD and Intel have recognized the lack of ability in separating read and execute permissions in the x86 architecture. In the AMD64 processor line, AMD extended the architecture in a backward compatible way by adding a No eXecute permission to the set of existing memory permissions. After AMD's decision to include this functionality in its AMD64 instruction set, Intel implemented the similar XD bit feature in x86 processors beginning with the Pentium 4 processors based on later iterations of the Prescott core.

Since NX is more fine grained than the previously described segment approach, for processors which have NX enabled, ExecShield attempts to use that.

Randomization

Since a typical buffer-overflow exploit works by overwriting the original return address with the address of the buffer containing the shell code, the attacker needs to know the exact address of the buffer containing this code. While this appears to be difficult, in practice it may not be very difficult especially when there is some sort of an address leak flaw as well.

Each system running the same version of the operating system has the same binaries and libraries. A person who is writing an exploit only has to examine their own system to determine the address that will be similar on all other such systems. Another approach in exploiting buffer overflows also involves overwriting the return address, however, rather than overwriting it with the address of the shellcode injected into the buffer, an attacker overwrites it with the address of a subroutine that is already present in the application, quite often the system() function from the glibc library. Since this type of attack does not depend on executing code in a data/stack area, but does depend on executing previously present and legitimate code with attacker-supplied data, it defeats the ExecShield approach of making the stack non-executable. Note, however, that this approach also depends on knowing the exact address of the function that is to be called.

In order to prevent the above, the ExecShield technology uses Address Space Layout Randomization, which gives randomized offsets to several key components like the stack, locations of shared libraries and the program heap. This randomization offers more system security by making it difficult to find the exact address needed for these exploits; the address is now different for every machine as well as being different each time a program starts.

Though ExecShield provides a decent level of protection against exploitation, it is not enough on its own. Several other technologies such as ASLR should be enabled in the operating system. Further, important binaries need to be compiled with exploitation-mitigation technologies like PIE, REPRO, FORTIFY_SOURCE etc, to provide an optimal level of protection, which is what Red Hat has been doing for years.