You’ve just installed Red Hat OpenShift Container Platform 3 - now what?
Red Hat OpenShift Container Platform 3 is a complex product with a lot of components. This article is going to go over steps to validate your installation was successful, that your applications are responsive, and what to look for if things aren’t working as expected.
Step 1: Authenticate
Depending on what installation method you used, you may or may not already have some form of external authentication enabled. If you don’t, you’ll want to set one up, with HTPasswd being the fastest and easiest method to install and configure.
Logging from the command line is as simple as running "oc login", and supplying the name of the master (single master environment) or master cluster name.
# oc login https://ocp.paas.home.lab:8443 Authentication required for https://ocp.paas.home.lab:8443 (openshift) Username: jritenour Password: Login successful. You have access to the following projects and can switch between them with 'oc project <projectname>': * default kube-system management-infra openshift openshift-infra Using project "default".
Authentication issues against an external provider can be challenging to troubleshoot. First, you need to verify you can actually authenticate against the provider outside the context of Red Hat OpenShift Container Platform. For example, if you’re using IdM/FreeIPA as an LDAP source, verify you can login with your credentials on a host joined to the IdM domain. If so, verify you are using the correct parameters in your master-config.yaml for all OpenShift masters - the most common issue I tend to see is trying to use secure LDAP, but without a trusted certificate.
If everything looks correct in the config, then you need to verify your authentication request is actually reaching the external provider. Check to see if there any failed authentication attempts on the provider’s logs. If the attempt isn’t even getting to the provider, then you might have roll up your sleeves and do some network troubleshooting with utilities such as tcpdump, nmap, and traceroute. That’s a bit outside the scope of this article, however.
Running "oc get nodes" will return information about the nodes in an OpenShift cluster.
# oc get nodes NAME STATUS AGE jr-ose001.home.lab Ready,SchedulingDisabled 19m jr-ose002.home.lab Ready,SchedulingDisabled 2m jr-ose003.home.lab Ready,SchedulingDisabled 19m jr-ose004.home.lab Ready 19m jr-ose005.home.lab Ready 19m jr-ose006.home.lab Ready 19m jr-ose007.home.lab Ready 19m jr-ose008.home.lab Ready 19m
All nodes should be listed as "Ready", and masters should also have "SchedulingDisabled" in their status. If any nodes are in any state other than "Ready", verify the host is up and responsive. Some quick checks you can perform are:
Is the host responding on the correct IP address? Does it match up with the hostname configured in DNS?
Is the atomic-openshift-node service running?
Is the firewall properly configured? Sample output:
Chain OS_FIREWALL_ALLOW (1 references) target prot opt source destination ACCEPT tcp — anywhere anywhere state NEW tcp dpt:10250 ACCEPT tcp — anywhere anywhere state NEW tcp dpt:http ACCEPT tcp — anywhere anywhere state NEW tcp dpt:https ACCEPT tcp — anywhere anywhere state NEW tcp dpt:10255 ACCEPT udp — anywhere anywhere state NEW udp dpt:10255 ACCEPT udp — anywhere anywhere state NEW udp dpt:4789
Validate default project status
The "default" project is where all the essential containerized OpenShift services run, specifically the router and registry. Verifying this namespace is in a healthy state is essential. Run ‘oc status’ while in the default project (oc project default). You should have:
A Kubernetes service
A Docker registry service with at least one pod
A router service with at least one pod.
#oc status In project default on server https://ocp.paas.home.lab:8443 svc/docker-registry - 172.30.24.180:5000 dc/docker-registry deploys registry.access.redhat.com/openshift3/ose-docker-registry:v188.8.131.52 deployment #1 deployed 25 minutes ago - 1 pod svc/kubernetes - 172.30.0.1 ports 443, 53->8053, 53->8053 svc/router - 172.30.253.41 ports 80, 443, 1936 dc/router deploys docker.io/openshift3/ose-haproxy-router:v184.108.40.206 deployment #1 deployed 2 minutes ago - 2 pods View details with 'oc describe <resource>/<name>' or list everything with 'oc get all'.
You can also run "oc get all" to show the current state of the deployment configs, replication configs, services, and pods in this project.
# oc get all NAME REVISION DESIRED CURRENT TRIGGERED BY dc/docker-registry 1 1 1 config dc/router 1 2 2 config NAME DESIRED CURRENT AGE rc/docker-registry-1 1 1 29m rc/router-1 2 2 5m NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE svc/docker-registry 172.30.24.180 <none> 5000/TCP 29m svc/kubernetes 172.30.0.1 <none> 443/TCP,53/UDP,53/TCP 1h svc/router 172.30.253.41 <none> 80/TCP,443/TCP,1936/TCP 5m NAME READY STATUS RESTARTS AGE po/docker-registry-1-on709 1/1 Running 0 28m po/router-1-baety 1/1 Running 0 4m po/router-1-bmdy2 1/1 Running 0 5m
Now that we’ve verified the overall state of the default project, we’ll want to verify the registry is working, and allows pushes and pulls with an authenticated user. You’ll need to login to test the registry, and for that you need to know what the registry’s service IP is, and the value of your authentication token.
# oc get svc docker-registry NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE docker-registry 172.30.24.180 <none> 5000/TCP 35m # oc whoami -t *redacted token string*
With that info, I can now login to my registry service.
# docker login -u jritenour -e firstname.lastname@example.org -p *redacted token string* 172.30.24.180:5000 WARNING: login credentials saved in /root/.docker/config.json Login Succeeded
Now I can try to push an image. Pull the "busybox" image down from docker.io, as it’s small and easy to test with. Tag it with your registry ip/port and put it in the "openshift" namespace, then push.
# docker images REPOSITORY TAG IMAGE ID CREATED SIZE docker.io/busybox latest e02e811dd08f 2 weeks ago 1.093 MB # docker tag docker.io/busybox 172.30.24.180:5000/openshift/busybox # docker push 172.30.24.180:5000/openshift/busybox The push refers to a repository [172.30.24.180:5000/openshift/busybox] e88b3f82283b: Pushed latest: digest: sha256:b321c7c9c643778fbe22de13a01fdfbac0f21c6c5452d8164de9367d96235d0c size: 2099 # docker images REPOSITORY TAG IMAGE ID CREATED SIZE 172.30.24.180:5000/openshift/busybox latest e02e811dd08f 2 weeks ago 1.093 MB docker.io/busybox latest e02e811dd08f 2 weeks ago 1.093 MB
A successful push means all is well. If it fails, then you likely have a problem with whatever storage your registry is using on the backend. Verify the storage is accessible, writable, and mounted by the docker registry pods.
Create an application
To continue, we’ll next create application. First, let’s create a project/namespace to run this app in:
oc new-project jr-test Now using project "jr-test" on server "https://ocp.paas.home.lab:8443". You can add applications to this project with the 'new-app' command. For example, try: oc new-app centos/ruby-22-centos7~https://github.com/openshift/ruby-ex.git to build a new example application in Ruby.
And to keep it simple, we’ll deploy the Ruby application that the "new-project" command suggested.
# oc new-app centos/ruby-22-centos7~https://github.com/openshift/ruby-ex.git —> Found Docker image 0449712 (3 days old) from Docker Hub for "centos/ruby-22-centos7" Ruby 2.2 ———— Platform for building and running Ruby 2.2 applications Tags: builder, ruby, ruby22 * An image stream will be created as "ruby-22-centos7:latest" that will track the source image * A source build using source code from https://github.com/openshift/ruby-ex.git will be created * The resulting image will be pushed to image stream "ruby-ex:latest" * Every time "ruby-22-centos7:latest" changes a new build will be triggered * This image will be deployed in deployment config "ruby-ex" * Port 8080/tcp will be load balanced by service "ruby-ex" * Other containers can access this service through the hostname "ruby-ex" —> Creating resources with label app=ruby-ex ... imagestream "ruby-22-centos7" created imagestream "ruby-ex" created buildconfig "ruby-ex" created deploymentconfig "ruby-ex" created service "ruby-ex" created —> Success Build scheduled, use 'oc logs -f bc/ruby-ex' to track its progress. Run 'oc status' to view your app
I can run "oc status" to see what stage the deployment is in, or log into the Web UI to actually watch the console output of the build process.
When the build is complete, I can then run "oc get service" to obtain the service IP, and verify I can connect to it using the curl command.
# oc get service NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE ruby-ex 172.30.138.103 <none> 8080/TCP 5m # curl 172.30.138.103:8080 -I HTTP/1.1 200 OK Content-Type: text/html Content-Length: 39590
Now I know that I can connect to the application by the service IP, but in the real world, we need to route hostnames to applications, and that’s where our router comes in. In most cases, you’re going to have a wildcard DNS record for your OpenShift/PaaS subdomain pointed at your infrastructure node(s) that your router pod(s) live on. For example, I have "*.paas.home.lab" pointing at jr-ose005, and jr-ose006, with router pods running on poth. So any name that lives in the .paas.home.lab domain that isn’t explicitly defined in DNS will get directed to these nodes. From there, the OpenShift router checks its exposed services, and send traffic to the appropriate pod/node.
Let’s expose the ruby-ex service we just created:
#oc expose svc/ruby-ex —hostname ruby.paas.home.lab
Now, we can curl against that hostname, just as we did with the IP.
curl ruby.paas.home.lab -I HTTP/1.1 200 OK Content-Type: text/html Content-Length: 39590 Set-Cookie: 46835afb0f3cb981bcb9d80703f10156=8a5b6b851a46307c10c5a52a2859aad5; path=/; HttpOnly Cache-control: private
Another "200 OK" status response - this application is healthy, and we can get traffic to it.
Finally, you can run "oadm diagnostics" on the cluster. This is more or less the "kitchen sink" as far as Red Hat OpenShift Container Platform health checks go, and will examine the running environment, along with your master/node config files. It’s verbose, so I won’t include the full output here, but here is a small sample in which I’m warning about the fact that I haven’t configured metrics or aggregated logging.
WARN: [DH0005 from diagnostic MasterConfigCheck@openshift/origin/pkg/diagnostics/host/check_master_config.go:52] Validation of master config file '/etc/origin/master/master-config.yaml' warned: assetConfig.loggingPublicURL: Invalid value: "": required to view aggregated container logs in the console assetConfig.metricsPublicURL: Invalid value: "": required to view cluster metrics in the console
In this post, we went through several steps to validate that a Red Hat OpenShift Container Platform environment has been deployed correctly, and verify basic operation.