Many modern developers have learned that ‘sticking with HEAD’ (the most recent stable release) can be the best way to keep their application more secure. In this new ‘devops’ world there’s a fine line between using the latest and greatest, and breaking changes introduced by an upgrade. In this post we’ll explore some configuration options in Red Hat OpenShift which can make keeping up with the latest release easier, while reducing the impact of breaking changes. For more information on image streams I encourage you to read the source-to-image FAQ by Maciej Szulik.
Many people, when they deploy an application on OpenShift, use source-to-image. It can provide a lot of functionality out of the box, including base images for language runtimes. Those base images install the runtime itself and usually some sort of tool to build the application which runs on top.
When building new versions of your image you pick up new libraries. You might want the runtime and build tool to update as well. By default, OpenShift sticks with whatever source-to-image base image was the latest at the time the image streams were installed. The default image streams won’t pull a new image from an external registry unless an image stream is tagged to a new version, or if they are ‘scheduled’. By default, an OpenShift upgrade, using the upgrade ansible playbooks, updates the Image Streams.
In this example I’m going to use the upstream Centos images so that I can push them to Docker Hub without worrying about license restrictions placed on RHEL images, and because I want to be able to push an image update to the ‘latest’ tag without waiting for that to happen naturally on the official Red Hat Container Catalogue registry. However if you find the concepts outlined here compelling, I encourage you to modify the existing image streams to be scheduled.
Setting up the test environment
Let’s explore this topic by creating our own image stream from a registry we control. We use a registry we control so that we can push a new image there, however these steps should work with any external registry. The image streams which come installed with OpenShift Container Platform (OCP) 3.10 use images from the Red Hat Container Catalogue. I’d recommend using that registry for Red Hat official images because it can offer security features, such as Deep Container Inspection.
When experimenting with this feature I used OCP 3.10. I then modified the configuration so that we don’t have to wait so long for new images to be pulled from the external registry. The default is to import new images every 15 minutes.
$ ssh email@example.com [master.example.com] $ vi /etc/origin/master/master-config.yaml
imagePolicyConfig: internalRegistryHostname: docker-registry.default.svc:5000
scheduledImageImportMinimumIntervalSeconds: 60 [master.example.com] $ master-restart controllers 2
I created an image in an external repository by first pulling the bucharestgold/centos7-s2i-nodejs:10.9.0 image locally, tagging it with my own namespace and pushing it using the ‘latest’ tag.
$ docker pull docker.io/bucharestgold/centos7-s2i-nodejs:10.9.0 $ docker tag docker.io/bucharestgold/centos7-s2i-nodejs:10.9.0 \ docker.io/jazinner/centos7-s2i-nodejs:latest $ docker push docker.io/jazinner/centos7-s2i-nodejs:latest The push refers to a repository [docker.io/jazinner/centos7-s2i-nodejs] ... latest: digest: ba8f35ef3b854b1dab87c20d2b9e8a1b15c219717526970c4aa196ca2fc8d3ad
Creating an Image Stream
Then, as an administrator I create a new image stream in the ‘openshift’ project. I use the ‘openshift’ project because then my image stream is available to other users, but these steps should also work with an imagestream created in a developer project.
$ oc import-image docker.io/jazinner/centos7-s2i-nodejs:latest \ --confirm --scheduled=true The import completed successfully. Name:centos7-s2i-nodejs Namespace:openshift Created:1 second ago Labels:<none> Annotations:openshift.io/image.dockerRepositoryCheck=2018-09-12T02:54:11Z Docker Pull Spec:172.30.1.1:5000/openshift/centos7-s2i-nodejs Image Lookup:local=false Unique Images:1 Tags:1 latest updates automatically from registry docker.io/jazinner/centos7-s2i-nodejs:latest prefer registry pullthrough when referencing this tag * docker.io/jazinner/centos7-s2i-nodejs@sha256:ba8f35ef3b854b1dab87c20d2b9e8a1b15c219717526970c4aa196ca2fc8d3ad 1 second ago Image Name:centos7-s2i-nodejs:latest Docker Image:docker.io/jazinner/centos7-s2i-nodejs@sha256:ba8f35ef3b854b1dab87c20d2b9e8a1b15c219717526970c4aa196ca2fc8d3ad Name:sha256:ba8f35ef3b854b1dab87c20d2b9e8a1b15c219717526970c4aa196ca2fc8d3ad ...
Using the Image Stream
We can then build a new image which uses the imported source-to-image image stream. For this, switch to a developer account, and use a new project called ‘myproject’.
$ oc login https://192.168.42.110:8443 -u developer Logged into "https://192.168.42.110:8443" as "developer" using existing credentials. You have one project on this server: "myproject" Using project "myproject". $ oc new-app centos7-s2i-nodejs~https://github.com/openshift/nodejs-ex.git --> Found image a3effc2 (2 days old) in image stream "openshift/centos7-s2i-nodejs" under tag "latest" for "centos7-s2i-nodejs" ... --> Creating resources ... imagestream "nodejs-ex" created buildconfig "nodejs-ex" created deploymentconfig "nodejs-ex" created service "nodejs-ex" created --> Success Build scheduled, use 'oc logs -f bc/nodejs-ex' to track its progress. Application is not exposed. You can expose services to the outside world by executing one or more of the commands below: 'oc expose svc/nodejs-ex' Run 'oc status' to view your app.
Running the container pod allows us to verify the NodeJS version in use:
$ oc get pods -w NAME READY STATUS RESTARTS AGE nodejs-ex-1-build 1/1 Running 0 40s nodejs-ex-1-deploy 0/1 Pending 0 0s .. nodejs-ex-1-deploy 0/1 ContainerCreating 0 0s nodejs-ex-1-build 0/1 Completed 0 2m ... nodejs-ex-1-q8dsg 0/1 ContainerCreating 0 0s nodejs-ex-1-deploy 1/1 Running 0 3s nodejs-ex-1-q8dsg 1/1 Running 0 2s $oc rsh nodejs-ex-1-q8dsg node --version V10.9.0
Verifying scheduled imports
If we push a new image to the docker.io/jazinner/centos7-s2i-nodejs:latest tag a new source-to-image build of our application picks up the new image.
$ docker pull docker.io/bucharestgold/centos7-s2i-nodejs:10.10.0 $ docker tag docker.io/bucharestgold/centos7-s2i-nodejs:10.10.0 \ docker.io/jazinner/centos7-s2i-nodejs:latest $ docker push docker.io/jazinner/centos7-s2i-nodejs:latest The push refers to a repository [docker.io/jazinner/centos7-s2i-nodejs] ... latest: digest: sha256:de0c8f343594c096d209e492f1e694886280f53e2997a50c9e9ab57ae1aad02b size: 1583
Once the Scheduled import minimum interval has expired the new 10.10.0 image will be pulled from the external registry. You can verify that a new image has been pulled using the ‘oc describe’ feature. Notice this time we use the ‘-n’ flag to reference resources in a different project.
$ oc describe is/centos7-s2i-nodejs -n openshift Name:centos7-s2i-nodejs Namespace:openshift Created:About an hour ago Labels:<none> Annotations:openshift.io/image.dockerRepositoryCheck=2018-09-12T03:12:54Z Docker Pull Spec:172.30.1.1:5000/openshift/centos7-s2i-nodejs Image Lookup:local=false Unique Images:2 Tags:1 latest updates automatically from registry docker.io/jazinner/centos7-s2i-nodejs:latest prefer registry pullthrough when referencing this tag * docker.io/jazinner/centos7-s2i-nodejs@sha256:d25befaa1961d8b8634fb6afe4e1d74c6b1d9d03253027c264bd89f1e1b0b86a About an hour ago docker.io/jazinner/centos7-s2i-nodejs@sha256:de0c8f343594c096d209e492f1e694886280f53e2997a50c9e9ab57ae1aad02b About an hour ago
The Annotation field shows us the timestamp the image was updated:
If you prefer not to schedule image imports from an external registry, you can do it on demand for an image stream by tagging it. You can manually tag a new version using a command like this:
$ oc import-image openshift/centos7-s2i-nodejs:latest The import completed successfully. Name:centos7-s2i-nodejs Namespace:myproject Created:47 hours ago Labels:<none> Annotations:openshift.io/image.dockerRepositoryCheck=2018-09-13T01:50:26Z Docker Pull Spec:docker-registry.default.svc:5000/myproject/centos7-s2i-nodejs Image Lookup:local=false Unique Images:3 Tags:1 latest tagged from docker.io/jazinner/centos7-s2i-nodejs:latest * docker.io/jazinner/centos7-s2i-nodejs@sha256:ba8f35ef3b854b1dab87c20d2b9e8a1b15c219717526970c4aa196ca2fc8d3ad 4 seconds ago ...
By default a build configuration created by ‘oc new-app’ is setup to trigger a build when a new base image is imported. We can see this by describing the build configuration
$ oc describe buildconfigs/nodejs-ex Name:nodejs-ex Namespace:myproject Created:2 hours ago Labels:app=nodejs-ex Annotations:openshift.io/generated-by=OpenShiftNewApp Latest Version:2 Strategy:Source URL:https://github.com/openshift/nodejs-ex.git From Image:ImageStreamTag openshift/centos7-s2i-nodejs:latest Output to:ImageStreamTag nodejs-ex:latest Build Run Policy:Serial Triggered by:Config, ImageChange … BuildStatusDurationCreation Time nodejs-ex-2 complete 16s 2018-09-12 13:12:54 +1000 AEST nodejs-ex-1 complete 1m54s 2018-09-12 12:54:43 +1000 AEST
From this output we can also see that a new build ‘nodejs-ex-2’ was kicked off at the same time as the import happened. Note that the 2 times have a different timezones however.
You could change this behaviour by updating the build configuration triggers. Here’s a command you can use to remove the ImageChange build trigger. It uses an index to refer to the ImageChange option, so we first check that the type is correct before removing it.
$ oc set triggers bc nodejs-ex --remove --from-image='openshift/centos7-s2i-nodejs:latest' buildconfig "nodejs-ex" updated
Now if a new image is pushed to the external registry it will be updated in the centos7-s2i-nodejs image stream. However when it’s updated a new build won’t be triggered for the application. A developer will have to trigger a build in some other way in order to pick up the new base image.
With this configuration of source-to-image base images being updated automatically, and developers manually triggering new builds which use those base images we have a good balance of security and compatibility. Breaking changes made to the base image should be picked up when a developer builds a source-to-image application and tests it. Also, the cluster continues to get updates to base images without requiring a full upgrade.
Using scheduled source-to-image base image streams, along with a build configuration which disables ImageChange triggers, we can strike a nice balance between “sticking with head”, and avoiding breaking changes. Consider updating the pre-installed image streams in the ‘openshift’ project to allow your developers get the latest security updates in language runtimes and build tools.
While I used CentOS images for demonstration purposes in this post, I’d recommend using RHEL images for your production applications. The Red Hat Container Catalogue contains regularly updated and certified container images, fully supported by Red Hat.
About the author
Specializing in Kubernetes, container runtimes, and web applications, Jason Shepherd is a principal security engineer in Red Hat's Product Security team. With a passion for open source and dedication to client success, Shepherd is your go-to guy for security assessment and data for security audits.