Polling triggers in OpenShift Pipeline

Webhooks can be triggered whenever specific events on GitHub send change information to OpenShift Pipelines for integrating a CI/CD workflow. In the process, you should set up your server to receive and manage the payload from the Internet by adding triggers to a pipeline. If your pipeline is in a restricted network and does not allow access from the Internet (GitHub),  consider poll-based change detection for integrating GitHub on the Internet instead.

Poll-based change detection is a kind of polling trigger, which is an event that periodically makes a call to your Git repository to look for new changes. This article will take you through configuring polling triggers in OpenShift Pipelines (Tekton) in a restricted network.

Workflow of pull-based triggers

Basically, a polling trigger is a pull-based workflow, while webhooks are push-based. In other words, polling triggers periodically initiate an event over an interval to determine if there is any change, whereas webhooks respond to a push of changes from the repository.

If you have a private GitLab repository in your disconnected network, you can configure the polling triggers for your pipeline CI/CD using Repository Mirroring (which mirrors all changes over an interval with another repository). A push event usually triggers webhooks. The following diagram shows the workflow based on GitLab:

However, GitLab is overkill for some use cases, so you should implement polling triggers using CronJob with Bash scripts. The diagram above illustrates this in Option 2.

Polling triggers using CronJob

I prepared a sample application for testing a pipeline run through Creating CI/CD solutions for applications using OpenShift Pipelines as follows. In this section, I skipped a secret token configuration in the Trigger YAML for simpler tests. In a restricted network, security would not be a problem.

// Create a project for the sample application
$ oc new-project pipelines-tutorial

// Creating pipeline tasks
$ oc create -f https://raw.githubusercontent.com/openshift/pipelines-tutorial/pipelines-1.9/01_pipeline/01_apply_manifest_task.yaml
$ oc create -f https://raw.githubusercontent.com/openshift/pipelines-tutorial/pipelines-1.9/01_pipeline/02_update_deployment_task.yaml

// Assembling a pipeline
$ oc create -f https://raw.githubusercontent.com/openshift/pipelines-tutorial/pipelines-1.9/01_pipeline/04_pipeline.yaml

// Adding triggers to a pipeline
$ oc create -f https://raw.githubusercontent.com/openshift/pipelines-tutorial/pipelines-1.9/03_triggers/01_binding.yaml
$ oc create -f https://raw.githubusercontent.com/openshift/pipelines-tutorial/pipelines-1.9/03_triggers/02_template.yaml
$ oc create -f - <<EOF
apiVersion: triggers.tekton.dev/v1beta1
kind: Trigger
metadata:
  name: vote-trigger
spec:
  serviceAccountName: pipeline
  interceptors:
    - ref:
        name: "github"
      params:
        - name: "eventTypes"
          value: ["push"]
  bindings:
    - ref: vote-app
  template:
    ref: vote-app
EOF

$ oc create -f https://raw.githubusercontent.com/openshift/pipelines-tutorial/pipelines-1.9/03_triggers/04_event_listener.yaml

// Running a pipeline
$ tkn pipeline start build-and-deploy \
    -w name=shared-workspace,volumeClaimTemplateFile=https://raw.githubusercontent.com/openshift/pipelines-tutorial/pipelines-1.9/01_pipeline/03_persistent_volume_claim.yaml \
    -p deployment-name=pipelines-vote-ui \
    -p git-url=https://github.com/openshift/pipelines-vote-ui.git \
    -p IMAGE='image-registry.openshift-image-registry.svc:5000/$(context.pipelineRun.namespace)/pipelines-vote-ui' \
    --use-param-defaults

$ oc get pod -n pipelines-tutorial
NAME                                               READY   STATUS      RESTARTS   AGE
build-and-deploy-run-tbks8-apply-manifests-pod     0/1     Completed   0          3m38s
build-and-deploy-run-tbks8-build-image-pod         0/1     Completed   0          4m11s
build-and-deploy-run-tbks8-fetch-repository-pod    0/1     Completed   0          4m23s
build-and-deploy-run-tbks8-update-deployment-pod   0/1     Completed   0          3m24s
el-vote-app-68fcd9675f-hdw6j                       1/1     Running     0          10m
pipelines-vote-ui-676bccc85-m4rsp                  1/1     Running     0          3m20s

After confirming that the sample application pipelines-vote-ui is working well, replace the above GitHub repository with the one you have forked from the front-end pipelines-vote-ui for your tests (in my case, https://github.com/bysnupy/pipelines-vote-ui). I will apply a change to my repository to check whether the pipeline kicks off or not in the following demonstration.

Let’s look at the details of the following resources to implement simple polling triggers with scripts. Simply comparing the previous and current revision sha256 hash values using the git command is the main logic to determine triggering the pipeline, and the latest sha256 hash would be stored in a PV volume for the next checks. The CronJob requests a push event using the curl command to an EventListener on behalf of the GitHub repository webhook for triggering a pipeline run.

This approach allows us to reuse existing TriggerBinding and TriggerTemplate resources. Regardless, you can consider using tkn CLI instead of the curl command in CronJob for your needs. The tkn CLI can customize more than the curl command in your pipeline flow.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: repolist-pvc
  namespace: pipelines-tutorial
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
---
apiVersion: batch/v1
kind: CronJob
metadata:
  name: polling-triggers
  namespace: pipelines-tutorial
spec:
  schedule: "*/1 * * * *"
  jobTemplate:
    spec:
      template:
        spec:
          containers:
          - name: polling-triggers
            volumeMounts:
            - name: repolist
              mountPath: /repolist
            image: registry.redhat.io/rhel8/toolbox:latest
            env:
            - name: REPONAME
              value: pipelines-vote-ui
            - name: REPOBASEURL
              value: https://github.com/bysnupy
            - name: REPOBRANCH
              value: master
            - name: REPOURL
              value: https://github.com/bysnupy/pipelines-vote-ui
            - name: BASEDIR
              value: /repolist
            - name: EVENTLISTENERSVC
              value: "http://el-vote-app.pipelines-tutorial.svc.cluster.local:8080"
            - name: JSONTEMPLATE
              value: '{"object_kind": "push","event_name": "push","head_commit": {"id": "GITHUBREPOREV"},"repository": {"name": "GITHUBREPONAME","url": "GITHUBREPOURL"}}'
            command:
            - /bin/sh
            args:
            - -c
            - |
              set -eu

              # revision initialization
              _current_revision=$(git ls-remote --heads ${REPOBASEURL}/${REPONAME} ${REPOBRANCH} | awk '{print $1}')
              _prev_revision=${_current_revision}

              # if there is not existing previous revision data file, creating new revision data file with current revision
              test -f ${BASEDIR}/${REPONAME}.sha256 && _prev_revision=$(cat ${BASEDIR}/${REPONAME}.sha256) || echo ${_current_revision} > ${BASEDIR}/${REPONAME}.sha256

              # generating JSON data
              _jsondata=$(echo ${JSONTEMPLATE} | sed -e "s=GITHUBREPOREV=${_current_revision}=" -e "s=GITHUBREPONAME=${REPONAME}=" -e "s=GITHUBREPOURL=${REPOURL}=")

              # check if there are any changes through comparing previous and current revisions.

              # If there are any changes, trigger a new pipeline using curl and json data.
              test "${_current_revision}" != "${_prev_revision}" && echo ${_current_revision} > ${BASEDIR}/${REPONAME}.sha256 &&
              curl -s -X POST -H 'Content-Type: application/json' -H 'X-GitHub-Event: push' \
              -d "${_jsondata}" ${EVENTLISTENERSVC} ||
              echo "No changes"
          restartPolicy: Never
          volumes:
          - name: repolist
            persistentVolumeClaim:
              claimName: repolist-pvc

For more details of GitHub webhook HTTP POST payloads for modification of curl options, please refer to Webhook events and payloads.

Triggering a pipeline run using polling triggers

When a change event occurs in the Git repository, the aforementioned polling triggers CronJob checks it periodically. After becoming aware of new data, send an event payload (JSON data) to the privately exposed EventListener service using the curl command. The EventListener service of the application processes the payload and passes it to the relevant TriggerBinding and TriggerTemplate resource pairs. The TriggerBinding resource extracts the parameters, and the TriggerTemplate resource uses these parameters and specifies how to create the resources. This process may rebuild and redeploy the application.

In this section, you push an empty commit to the front-end pipelines-vote-ui repository, triggering the pipeline run.

// Check the polling triggers pod logs before changes
$ oc get pod -n pipelines-tutorial
 NAME                                 READY   STATUS              RESTARTS   AGE
el-vote-app-68fcd9675f-hdw6j         1/1     Running             0          135m
pipelines-vote-ui-54fcb9658d-tkqjp   1/1     Running             0          9m29s
polling-triggers-28174084-hrdth      0/1     Completed           0          33s

$ oc logs -f polling-triggers-28174084-hrdth
No changes

// Clone your forked Git repository pipelines-vote-ui, and push an empty commit
$ git clone https://github.com/bysnupy/pipelines-vote-ui.git -b master
$ git commit -m "empty-commit" --allow-empty && git push origin master

// Check if the new pipeline run would be triggered by the polling triggers
$ oc get pod -n pipelines-tutorial
 NAME                                 READY   STATUS              RESTARTS   AGE
el-vote-app-68fcd9675f-hdw6j         1/1     Running             0          136m
pipelines-vote-ui-54fcb9658d-tkqjp   1/1     Running             0          9m59s
polling-triggers-28174084-hrdth      0/1     Completed           0          63s
polling-triggers-28174085-s8hzh      0/1     ContainerCreating   0          3s

$ oc logs -f polling-triggers-28174085-s8hzh
{"eventListener":"vote-app","namespace":"pipelines-tutorial","eventListenerUID":"b71d09d8-245f-4126-9c3b-53f399c7c24c","eventID":"169d6f94-29c1-4f65-92fe-1985d6faa3fb"}

$ oc get pod -n pipelines-tutorial
NAME                                                        READY   STATUS      RESTARTS   AGE
build-deploy-pipelines-vote-ui-8bxfv-fetch-repository-pod   0/1     Init:0/2    0          23s <--- New pipeline tasks started
el-vote-app-68fcd9675f-hdw6j                                1/1     Running     0          136m
pipelines-vote-ui-54fcb9658d-tkqjp                          1/1     Running     0          10m
polling-triggers-28174084-hrdth                             0/1     Completed   0          89s
polling-triggers-28174085-s8hzh                             0/1     Completed   0          29

Wrap up

This post demonstrated polling triggers using CronJob and verified how the process works. You can also customize the logic for your own use cases; for instance, directly running the pipeline without EventListener using the tkn command. Using your own implementation,  you can do anything that meets your needs. I hope this information helps deepen your knowledge of the OpenShift Pipeline.