OpenShift Ansible Broker - Proxy Considerations for an APB Developer

Introduction

With the OpenShift Container Platform 3.9 release of OpenShift Ansible Broker (OAB), we've added support to run the broker and Ansible Playbook Bundles (APBs) behind an HTTP/HTTPS proxy such as Squid. This makes it possible to run the broker in secured network environments where outgoing traffic must pass through a proxy.

OAB + Proxy: Use Cases

The diagram below illustrates three different cases where data might be sent through an HTTP proxy at various points of the OAB application management lifecycle.

Use Case 1: Broker Bootstrap (image metadata retrieval)

The broker's bootstrap action connects to container image registries and retrieves APB metadata. In order for the OAB to connect to the Internet to access image registries such as the Red Hat Container Catalog or Docker Hub, a proxy must be used when a direct connection isn't available. Without a successful bootstrap of image metadata retrieval, APB-based services won't show up in the OpenShift Service Catalog.

Use Case 2: APB action pods (provision, bind, update)

APB action pods such as provision, deprovision, bind, unbind, and update may require Internet access to successfully execute (e.g. an APB provision that creates an API key over the Internet). In OCP 3.9, any proxy settings provided to the broker will be inherited by APB action containers.

Use Case 3: Pods resulting from APBs

Provisioned app/service pods created by APBs may require Internet access to execute successfully. APBs responsible for deploying app/service pods can be easily modified to inject proxy configuration from the broker. Steps for doing this are detailed later in this post.

If modifying APB source files is undesirable or impossible, app/service pods can be manually modified with appropriate proxy settings.

Configuring the OAB with a Proxy

Steps for configuring the OAB with a proxy differ depending on whether you're working with an existing cluster or starting from scratch.

Setup on an existing OpenShift Cluster

The process of configuring the OAB pod with a proxy follows the OpenShift convention. We'll need to set the HTTP_PROXY, HTTPS_PROXY, and NO_PROXY environment variables in the broker's deployment config.

To begin, edit the OAB deployment config:

$ oc edit dc asb -n <broker_namespace>

Navigate to the section in the deployment config defining the OAB container, and edit the env section and add the following three environment variables. An excerpt is shown below with placeholder proxy values.

[...]

containers:

- env:

  - name: "HTTP_PROXY"

    value: "http://<user>:<password>@<ip_addr>:<port>"

  - name: "HTTPS_PROXY"

    value: "http://<user>:<password>@<ip_addr>:<port>"

  - name: "NO_PROXY"

    value: ".cluster.local,.svc,.default,<serviceNetworkCIDR>,<masterHostname>,<masterIP>"

[...]

The first two variables, HTTP_PROXY and HTTPS_PROXY, should point at a proxy host capable of relaying requests to both: 1) configured APB image registries, and 2) any external APIs that will be needed when running APB actions.

The third variable, NO_PROXY, specifies network locations for which the proxy shouldn't be used. Be sure to include ".cluster.local", ".svc", ".default", the value of serviceNetworkCIDR (e.g. "172.30.0.0/16", see the OpenShift documentation for more detail), and the IP address and hostname of the OpenShift master.

The same excerpt from above with placeholder values populated looks something like this:

[...]

containers:

- env:

  - name: "HTTP_PROXY"

    value: "http://proxyhost:3128"

  - name: "HTTPS_PROXY"

    value: "http://proxyhost:3128"

  - name: "NO_PROXY"

    value: ".cluster.local,.svc,.default,172.30.0.0/16,master.foo.com,192.168.120.4"

[...]

Once you've edited the deployment config, verify that the broker pod is using the new settings by looking for a successful APB bootstrap in the broker logs.

$ oc logs <asb_pod_name> -n <broker_namespace>

Setup on a new OpenShift Cluster using openshift-ansible

As of OCP 3.9, if you're setting up an OpenShift cluster using the openshift-ansible installer, global proxy settings defined in openshift_http_proxy, openshift_https_proxy, and openshift_no_proxy variables will be passed through to the OAB during the installation process.

The value for openshift_no_proxy is auto-generated by default, so in most cases, you'll only need to set openshift_http_proxy and openshift_https_proxy. Example definitions of the required variables are shown below.

openshift_http_proxy=http://USER:PASSWORD@IPADDR:PORT

openshift_https_proxy=https://USER:PASSWORD@IPADDR:PORT

Once the OpenShift installation process has finished, check the broker logs to ensure APB bootstrap ran successfully.

$ oc logs <broker_pod_name> -n <broker_namespace>

Developing APBs for Use in Proxied Environments

The OAB will pass its proxy settings to APB action pods (e.g. provision, deprovision, bind, unbind, update) as environment variables. We have found that there is little consensus on proxy settings being read from uppercase vs. lowercase environment variables (e.g. http_proxy vs. HTTP_PROXY), so the OAB assigns the same values to both within each APB action pod, as shown below.

http_proxy="<http_proxy>:<port>"

https_proxy="<https_proxy>:<port>"

no_proxy="<no_proxy_list>"

HTTP_PROXY="<http_proxy>:<port>"
HTTPS_PROXY="<https_proxy>:<port>"
NO_PROXY="<no_proxy_list>"


As an APB developer, you can access any of these environment variables from an Ansible Playbook using a lookup.

set_fact:

  http_proxy: {{ lookup('env', 'http_proxy') }}

  https_proxy: {{ lookup('env', 'https_proxy') }}

  no_proxy: {{ lookup('env', 'no_proxy') }}

Passing Proxy Settings to Child Pods

You might want to pass proxy settings through to child pods created by an APB action pod. Edit the provision action of your APB, navigating to the section defining the deployment config that will be created for the child pod. Copy the APB action pod proxy vars to the env section of the container definition as shown below.

- openshift_v1_deployment_config:

    name: demo-app

    namespace: '{{ namespace }}'

    containers:

    - name: demo-app

      env:

        - name: http_proxy

          value: "{{ lookup('env','http_proxy') }}"

        - name: https_proxy

          value: "{{ lookup('env','https_proxy') }}"

        - name: no_proxy

          value: "{{ lookup('env','no_proxy') }}"

        - name: HTTP_PROXY

          value: "{{ lookup('env','http_proxy') }}"

        - name: HTTPS_PROXY

          value: "{{ lookup('env','https_proxy') }}"

        - name: NO_PROXY

          value: "{{ lookup('env','no_proxy') }}"

   [...]

If more fine-grained control over proxy settings is desired at provision time, consider adding a boolean parameter to apb.yml giving the APB user control over whether broker proxy settings should pass through to the APBs child pods.

Manual configuration of child pods is possible as well. See the above section on configuring the broker for use with proxy being sure to include lowercase proxy variables as well.

If you're using APBs that haven't been adapted to deploy child pods with proxy settings attached, consider provisioning the proxy-config-apb. This APB copies the broker's proxy configuration to a bind credential, which can be attached to any application pod.

Completing the Chain

Having completed these steps, the chain of proxy setting inheritance is complete:

1. Proxy settings are defined in the openshift-ansible installer (or applied manually)
2. Proxy settings are passed to the OAB as environment variables
3. Proxy settings are passed to APB action pod as environment variables
4. Proxy settings are (optionally) passed to the provisioned service as environment variables

Recap: Actions required to use the OAB behind a Proxy

As an OpenShift Cluster Administrator:
Enabling proxy support requires editing the OAB deployment config, or using openshift-ansible in OCP 3.9 to deploy a new cluster with proxy settings defined.

As an APB Developer:
Consuming proxy settings from the OAB in deployed services requires editing APBs to pass proxy settings from APB Action Pod environment variables to the deployment config of services that will be created.