Getting any Docker image running in your own OpenShift cluster

For those who have been using stand-alone development environments, leveraging containers using pre-existing docker format images (either internal images or images from an external registry such as the Red Hat registry or docker hub) but have outgrown your single machine and want to leverage the power of OpenShift using Kubernetes, this blog post is for you!

My assumption is that readers are familiar with how to install OpenShift and how to setup projects with quotas. 

Recently I have been working with some organizations who are looking to quickly create and tear down projects to deliver small, quick and agile proof of concepts which are looking to be self-contained and might last a few days or weeks at most.
In a stand-alone environment, there are some prepackaged images such as GitLab-CE which can be utilized to great effect. In fact, most of the great open source projects have docker images maintained now.
For the post here let's assume that we just want to get a stand-alone GitLab-CE up and running; in this case the community edition image from docker hub.

Now you could go off and find the README for the docker image and then write an OpenShift manifest based on this (the manifest is a yaml or json file which describes how OpenShift should run this docker image such as where to pull the docker image from, any persistent storage volumes required, ports to expose and other deployment information).
So, let me introduce you to the “oc new-app” command and, using the gitlab/gitlab-ce:latest image from the Docker Hub, lets deploy GitLab in OpenShift to show how easy this is.

I am going to assume that you have a running OpenShift environment, so let's set-up a new project to use.
This project has been requested to be part of my temporary development environment for a new user, to which I have been asked to provide a GitLab community container image. As user with the role binding of admin, I am going to log onto OpenShift and run:

# oc new-project innovation-2016

Now using project "innovation-2016" on server "https://uki-ose3-master1.example.com:8443".

You can add applications to this project with the 'new-app' command…..[snip]

Note that we have moved into the new project automatically. I have asked my development team which image they want to use, they informed me that this was docker.io/gitlab/gitlab-ce:latest. Also note that you should exercise the appropriate level of inspection and diligence around selecting any images. I am going to run this through my usual security scanning (for example check out atomic scan from the projectatomic.io website as an example).

So far, so good. I like the image so let us deploy this into OpenShift:

# oc new-app gitlab/gitlab-ce

--> Found Docker image 91ae3a8 (5 days old) from Docker Hub for "gitlab/gitlab-ce"

* An image stream will be created as "gitlab-ce:latest" that will track this image

* This image will be deployed in deployment config "gitlab-ce"

* [WARNING] Image "gitlab-ce" runs as the 'root' user which may not be permitted by your cluster administrator

* Ports 22/tcp, 443/tcp, 80/tcp will be load balanced by service "gitlab-ce"

--> Creating resources with label app=gitlab-ce ...

ImageStream "gitlab-ce" created

DeploymentConfig "gitlab-ce" created

Service "gitlab-ce" created

--> Success

Run 'oc status' to view your app.

So OpenShift now starts the magic, it pulls the image from docker.io and examines the image and metadata; from which it writes us a definition for a pod, service, deploymentConfig and replication controller. It then tries to start the new container…. and here it falls over for which we need to take a quick aside. The [WARNING] above shows the reason!

# oc get pods

NAME                READY     STATUS             RESTARTS   AGE

gitlab-ce-1-kekx2   0/1       CrashLoopBackOff   4          5m

Looking at the logs from the container before it went into the failed state:

# oc logs -p gitlab-ce-1-kekx2

Thank you for using GitLab Docker Image!

Current version: gitlab-ce=8.4.3-ce.0

Configure GitLab for your system by editing /etc/gitlab/gitlab.rb file
And restart this container to reload settings.
To do it use docker exec:

docker exec -it gitlab vim /etc/gitlab/gitlab.rb
docker restart gitlab

For a comprehensive list of configuration options please see the Omnibus GitLab readme
https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/README.md

If this container fails to start due to permission problems try to fix it by executing:

docker exec -it gitlab update-permissions
docker restart gitlab

Generating ssh_host_rsa_key...
No user exists for uid 1000530000


By default, all containers that we try and launch within OpenShift, are set blocked from “RunAsAny” which basically means that they are not allowed to use a root user within the container. This prevents root actions such as chown or chmod from being run and is a sensible security precaution as, should a user be able to perform a local exploit to break out of the container, then they would not be running as root on the underlying container host. NB what about user-namespaces some of you are no doubt asking, these are definitely coming but the testing/hardening process is taking a while and whilst companies such as Red Hat are working hard in this space, there is still a way to go until they are ready for the mainstream.

So the failure, was because the GitLab-CE image attempted to run as root, and perform a chown on the filesystems mounted into the container. This of course fails as OpenShift prevented the container from running as root. Note we see an error from the OpenShift point of view, the counter argument is that, this is a failure of the underlying GitLab-CE docker image – as well all know we should run as little as possible as root and GitLab is perfectly capable of running as a non-root user so the image should be updated to do this.

So we know why it failed, how do we fix this? Well ideally we fix the original docker image to not run as root. If this is not possible then we can tell OpenShift to allow this project to run as root using the below command to change the security context constraints (see manual for these here):

# oadm policy add-scc-to-user anyuid -z default

Just to describe the above command, we are going to add a security constraint (or rather we are going to relax a default setting which would normally prohibit containers running as root). The -z in the command  indicates that we are going to add a capability to the service account (this is the user that by default is used to run containers within our current namespace - i.e. our project innovation-2016) and the command add the “run as any user” capability, i.e. we are no longer prohibited from running as root.

So the container starting account (the service account) has now been granted the “anyuid” capability which means it can start as root and therefore perform chown/chomd and other root level commands. Of course, I would much rather that the container was changed to be able to run without requiring root permissions, but I am trying to run a pre-packaged image here.

You can check this will the following command:

# oc edit scc anyuid

# Please edit the object below. Lines beginning with a '#' will be ignored,
......
users:
- system:serviceaccount:default:ci
- system:serviceaccount:ci:default
- system:serviceaccount:innovation-2016:default


The above has been truncated, but you can see the default serviceaccount (which is the one deploying pods in the project) is now granted the anyuid SCC. For more information on service account within projects, see here. Lets try and re-run that deployment and suddenly voila:

# oc deploy gitlab-ce --latest

Started deployment #2

Wait for the second deployment to succeed:

# oc logs gitlab-ce-2-k06r2

Thank you for using GitLab Docker Image!

Current version: gitlab-ce=8.4.3-ce.0

…….[snip]

And we can now curl the URL of gitlab:

curl http://172.30.220.10

You are being <a href="http://172.30.220.10/users/sign_in">redirected</a>.

We have a running Gitlab, we can look at this and customise this (for example I would suggest that you change the storage to persistent storage reservations if you want to use this container image in anger), but we'll save that for a subsequent blog post.

One word of caution though, we have taken the path of least resistance here and granted the anyuid permission to the default service account within the project innovation-2016. This means that ALL deployment configuration, replication controllers and pods will be able to run as root. Ideally no containers should need to run as root but should they need to then we can limit these permissions to just the deployment configuration.

As we mentioned, by default when a project is created a default account will be created which will be used to launch pods. What we can do is create an additional service account which is the only user with anyuid permissions, then we can assign the specific deployment configuration for the pods to this user. For example we could create a serviceaccount called “gitlab”, assign the anyuid to this account “oadm policy add-scc-to-user anyuid -z gitlab” and assign this user to the deployment configuration so all pods use this service account, for example see the seviceAccountName section we can add to the DeploymentConfig section which you can set:

{

"kind": "DeploymentConfig",

"apiVersion": "v1",

"metadata": {

...

},

"spec": {

"strategy": {

...},

"triggers": [

...

],

"replicas": 1,

"selector": {

...

},

"template": {

"metadata": {

...

},

"spec": {

"containers": [

...

],

...

"serviceAccountName": "gitlab"

}

}

}

}

For more information on this and other configuration settings, you can see the full rest api specification here.

So there you have it, from stand alone docker format, imported to OpenShift and up and running in a few minutes. Of course I will note that by no means this is the image you'll want to run as gitlab in production. As for that, many customizations to the image itself should be provided such as using persistent storage of course.

See you next time.

NB For reference, to get some idea of what oc new-app has generated for us so that you can tailor this to provide things like persistent storage, you can dump out the information into template format using the following command:

# oc export all --as-template=gitlab --selector='app=gitlab-ce'

{{cta('1ba92822-e866-48f0-8a92-ade9f0c3b6ca')}}