Many modern developers have learned that ‘sticking with HEAD’ (the most recent stable release) can be the best way to keep their application more secure. In this new ‘devops’ world there’s a fine line between using the latest and greatest, and breaking changes introduced by an upgrade. In this post we’ll explore some configuration options in Red Hat OpenShift which can make keeping up with the latest release easier, while reducing the impact of breaking changes. For more information on image streams I encourage you to read the source-to-image FAQ by Maciej Szulik.

Background

Many people, when they deploy an application on OpenShift, use source-to-image. It can provide a lot of functionality out of the box, including base images for language runtimes. Those base images install the runtime itself and usually some sort of tool to build the application which runs on top.

When building new versions of your image you pick up new libraries. You might want the runtime and build tool to update as well. By default, OpenShift sticks with whatever source-to-image base image was the latest at the time the image streams were installed. The default image streams won’t pull a new image from an external registry unless an image stream is tagged to a new version, or if they are ‘scheduled’. By default, an OpenShift upgrade, using the upgrade ansible playbooks, updates the Image Streams.

In this example I’m going to use the upstream Centos images so that I can push them to Docker Hub without worrying about license restrictions placed on RHEL images, and because I want to be able to push an image update to the ‘latest’ tag without waiting for that to happen naturally on the official Red Hat Container Catalogue registry. However if you find the concepts outlined here compelling, I encourage you to modify the existing image streams to be scheduled.

Setting up the test environment

Let’s explore this topic by creating our own image stream from a registry we control. We use a registry we control so that we can push a new image there, however these steps should work with any external registry. The image streams which come installed with OpenShift Container Platform (OCP) 3.10 use images from the Red Hat Container Catalogue. I’d recommend using that registry for Red Hat official images because it can offer security features, such as Deep Container Inspection.

When experimenting with this feature I used OCP 3.10. I then modified the configuration so that we don’t have to wait so long for new images to be pulled from the external registry. The default is to import new images every 15 minutes.


$ ssh root@master.example.com
[master.example.com] $ vi /etc/origin/master/master-config.yaml

After:


imagePolicyConfig:
 internalRegistryHostname: docker-registry.default.svc:5000

Add:


 scheduledImageImportMinimumIntervalSeconds: 60
[master.example.com] $ master-restart controllers
2

I created an image in an external repository by first pulling the bucharestgold/centos7-s2i-nodejs:10.9.0 image locally, tagging it with my own namespace and pushing it using the ‘latest’ tag.


$ docker pull docker.io/bucharestgold/centos7-s2i-nodejs:10.9.0 

$ docker tag docker.io/bucharestgold/centos7-s2i-nodejs:10.9.0 \
docker.io/jazinner/centos7-s2i-nodejs:latest

$ docker push docker.io/jazinner/centos7-s2i-nodejs:latest

The push refers to a repository [docker.io/jazinner/centos7-s2i-nodejs]
...
latest: digest: ba8f35ef3b854b1dab87c20d2b9e8a1b15c219717526970c4aa196ca2fc8d3ad

Creating an Image Stream

Then, as an administrator I create a new image stream in the ‘openshift’ project. I use the ‘openshift’ project because then my image stream is available to other users, but these steps should also work with an imagestream created in a developer project.



$ oc import-image docker.io/jazinner/centos7-s2i-nodejs:latest \
--confirm --scheduled=true


The import completed successfully.
Name:centos7-s2i-nodejs
Namespace:openshift
Created:1 second ago
Labels:<none>
Annotations:openshift.io/image.dockerRepositoryCheck=2018-09-12T02:54:11Z
Docker Pull Spec:172.30.1.1:5000/openshift/centos7-s2i-nodejs
Image Lookup:local=false
Unique Images:1
Tags:1
latest
 updates automatically from registry docker.io/jazinner/centos7-s2i-nodejs:latest
 prefer registry pullthrough when referencing this tag
 * docker.io/jazinner/centos7-s2i-nodejs@sha256:ba8f35ef3b854b1dab87c20d2b9e8a1b15c219717526970c4aa196ca2fc8d3ad
 1 second ago
Image Name:centos7-s2i-nodejs:latest
Docker Image:docker.io/jazinner/centos7-s2i-nodejs@sha256:ba8f35ef3b854b1dab87c20d2b9e8a1b15c219717526970c4aa196ca2fc8d3ad
Name:sha256:ba8f35ef3b854b1dab87c20d2b9e8a1b15c219717526970c4aa196ca2fc8d3ad
...

Using the Image Stream

We can then build a new image which uses the imported source-to-image image stream. For this, switch to a developer account, and use a new project called ‘myproject’.


$ oc login https://192.168.42.110:8443 -u developer 

Logged into "https://192.168.42.110:8443" as "developer" using existing credentials.
You have one project on this server: "myproject"
Using project "myproject".

$ oc new-app centos7-s2i-nodejs~https://github.com/openshift/nodejs-ex.git

--> Found image a3effc2 (2 days old) in image stream "openshift/centos7-s2i-nodejs" under tag "latest" for "centos7-s2i-nodejs"
...
--> Creating resources ...
 imagestream "nodejs-ex" created
 buildconfig "nodejs-ex" created
 deploymentconfig "nodejs-ex" created
 service "nodejs-ex" created
--> Success
 Build scheduled, use 'oc logs -f bc/nodejs-ex' to track its progress.
 Application is not exposed. You can expose services to the outside world by executing one or more of the commands below:
 'oc expose svc/nodejs-ex' 
 Run 'oc status' to view your app.

Running the container pod allows us to verify the NodeJS version in use:


$ oc get pods -w

NAME READY STATUS RESTARTS AGE
nodejs-ex-1-build 1/1 Running 0 40s
nodejs-ex-1-deploy 0/1 Pending 0 0s
..
nodejs-ex-1-deploy 0/1 ContainerCreating 0 0s
nodejs-ex-1-build 0/1 Completed 0 2m
...
nodejs-ex-1-q8dsg 0/1 ContainerCreating 0 0s
nodejs-ex-1-deploy 1/1 Running 0 3s
nodejs-ex-1-q8dsg 1/1 Running 0 2s

$oc rsh nodejs-ex-1-q8dsg node --version
V10.9.0


Verifying scheduled imports

If we push a new image to the docker.io/jazinner/centos7-s2i-nodejs:latest tag a new source-to-image build of our application picks up the new image.



$ docker pull docker.io/bucharestgold/centos7-s2i-nodejs:10.10.0

$ docker tag docker.io/bucharestgold/centos7-s2i-nodejs:10.10.0 \
docker.io/jazinner/centos7-s2i-nodejs:latest

$ docker push docker.io/jazinner/centos7-s2i-nodejs:latest


The push refers to a repository [docker.io/jazinner/centos7-s2i-nodejs]
...
latest: digest: sha256:de0c8f343594c096d209e492f1e694886280f53e2997a50c9e9ab57ae1aad02b size: 1583

Once the Scheduled import minimum interval has expired the new 10.10.0 image will be pulled from the external registry. You can verify that a new image has been pulled using the ‘oc describe’ feature. Notice this time we use the ‘-n’ flag to reference resources in a different project.


$ oc describe is/centos7-s2i-nodejs -n openshift

Name:centos7-s2i-nodejs
Namespace:openshift
Created:About an hour ago
Labels:<none>
Annotations:openshift.io/image.dockerRepositoryCheck=2018-09-12T03:12:54Z
Docker Pull Spec:172.30.1.1:5000/openshift/centos7-s2i-nodejs
Image Lookup:local=false
Unique Images:2
Tags:1
latest
 updates automatically from registry docker.io/jazinner/centos7-s2i-nodejs:latest
 prefer registry pullthrough when referencing this tag
 * docker.io/jazinner/centos7-s2i-nodejs@sha256:d25befaa1961d8b8634fb6afe4e1d74c6b1d9d03253027c264bd89f1e1b0b86a
 About an hour ago
 docker.io/jazinner/centos7-s2i-nodejs@sha256:de0c8f343594c096d209e492f1e694886280f53e2997a50c9e9ab57ae1aad02b
 About an hour ago

The Annotation field shows us the timestamp the image was updated:

openshift.io/image.dockerRepositoryCheck=2018-09-12T03:12:54Z

If you prefer not to schedule image imports from an external registry, you can do it on demand for an image stream by tagging it. You can manually tag a new version using a command like this:


$ oc import-image openshift/centos7-s2i-nodejs:latest

The import completed successfully.
Name:centos7-s2i-nodejs
Namespace:myproject
Created:47 hours ago
Labels:<none>
Annotations:openshift.io/image.dockerRepositoryCheck=2018-09-13T01:50:26Z
Docker Pull Spec:docker-registry.default.svc:5000/myproject/centos7-s2i-nodejs
Image Lookup:local=false
Unique Images:3
Tags:1
latest
 tagged from docker.io/jazinner/centos7-s2i-nodejs:latest
 * docker.io/jazinner/centos7-s2i-nodejs@sha256:ba8f35ef3b854b1dab87c20d2b9e8a1b15c219717526970c4aa196ca2fc8d3ad
 4 seconds ago
...

By default a build configuration created by ‘oc new-app’ is setup to trigger a build when a new base image is imported. We can see this by describing the build configuration


$ oc describe buildconfigs/nodejs-ex

Name:nodejs-ex
Namespace:myproject
Created:2 hours ago
Labels:app=nodejs-ex
Annotations:openshift.io/generated-by=OpenShiftNewApp
Latest Version:2
Strategy:Source
URL:https://github.com/openshift/nodejs-ex.git
From Image:ImageStreamTag openshift/centos7-s2i-nodejs:latest
Output to:ImageStreamTag nodejs-ex:latest
Build Run Policy:Serial
Triggered by:Config, ImageChange
…
BuildStatusDurationCreation Time
nodejs-ex-2 complete 16s 2018-09-12 13:12:54 +1000 AEST
nodejs-ex-1 complete 1m54s 2018-09-12 12:54:43 +1000 AEST


From this output we can also see that a new build ‘nodejs-ex-2’ was kicked off at the same time as the import happened. Note that the 2 times have a different timezones however.

You could change this behaviour by updating the build configuration triggers. Here’s a command you can use to remove the ImageChange build trigger. It uses an index to refer to the ImageChange option, so we first check that the type is correct before removing it.


$ oc set triggers bc nodejs-ex --remove --from-image='openshift/centos7-s2i-nodejs:latest'
buildconfig "nodejs-ex" updated

Now if a new image is pushed to the external registry it will be updated in the centos7-s2i-nodejs image stream. However when it’s updated a new build won’t be triggered for the application. A developer will have to trigger a build in some other way in order to pick up the new base image.

With this configuration of source-to-image base images being updated automatically, and developers manually triggering new builds which use those base images we have a good balance of security and compatibility. Breaking changes made to the base image should be picked up when a developer builds a source-to-image application and tests it. Also, the cluster continues to get updates to base images without requiring a full upgrade.

Conclusion

Using scheduled source-to-image base image streams, along with a build configuration which disables ImageChange triggers, we can strike a nice balance between “sticking with head”, and avoiding breaking changes. Consider updating the pre-installed image streams in the ‘openshift’ project to allow your developers get the latest security updates in language runtimes and build tools.

While I used CentOS images for demonstration purposes in this post, I’d recommend using RHEL images for your production applications. The Red Hat Container Catalogue contains regularly updated and certified container images, fully supported by Red Hat.


About the author

Specializing in Kubernetes, container runtimes, and web applications, Jason Shepherd is a principal security engineer in Red Hat's Product Security team. With a passion for open source and dedication to client success, Shepherd is your go-to guy for security assessment and data for security audits.

Read full bio