4.1 Tekton Pipelines

It is time to automate the deployment of our Quarkus application to OpenShift by using OpenShift Pipelines. OpenShift Pipelines are based on Tekton. In this example we will use the namespace to follow along the previous chapters. If you don’t have your Kafka cluster, data-producer and data-consumer up and running please repeat the previous chapter to be ready to continue! We will create another microservice which consumes the same stream of data and transform the data to calculate a simple average count.

Task 4.1.1: Basic Concepts

Tekton makes use of several Kubernetes custom resources (CRD).

These CRDs are:

• Task: A collection of steps that perform a specific task.
• Pipeline: A series of tasks, combined to work together in a defined (structured) way
• TaskRun: The execution and result of running an instance of a task
• PipelineRun: The actual execution of a whole Pipeline, containing the results of the pipeline (success, failed…)

Pipelines and tasks should be generic and must never define possible variables - such as ‘input git repository’ - directly in their definition. The concrete PipelineRun will get the parameters, that are being used inside the pipeline.

Workspaces are used to share the data between Tasks and Steps.

Static definition of a Pipeline

For each task, a pod will be allocated and for each step inside this task, a container will be used.

Runtime view of a Pipeline showing mapping to pods and containers

Task 4.1.2: Check project setup

We first check that the project is ready for the lab.

Ensure that the LAB_USER environment variable is set.

echo $LAB_USER

If the result is empty, set the LAB_USER environment variable.

export LAB_USER=<username>

Change to your main Project.

oc project $LAB_USER

The OpenShift Pipeline operator automatically creates a pipeline ServiceAccount with all required permissions to build and push an image. This service account is used by PipelineRuns. List the service accounts of your project.

oc get ServiceAccount

oc get sa

Output listing the pipeline service account:

NAME       SECRETS   AGE
builder    2         11s
default    2         11s
deployer   2         11s
pipeline   2         11s
...

Task 4.1.3: Tekton CLI tkn

For additional features, we are going to add another CLI that eases access to the Tekton resources and gives you more direct access to the OpenShift Pipeline semantics:

Verify tkn version by running:

tkn version

Client version: 0.10.0
Pipeline version: unknown
Triggers version: unknown

Task 4.1.4: Create Pipeline tasks

A Task is the smallest block of a Pipeline, which by itself can contain one or more steps. These steps are executed to process a specific element. For each task, a pod is allocated and each step is running in a container inside this pod. Tasks are reusable by other Pipelines. Input and Output specifications can be used to interact with other tasks.

Let’s examine the task that does a deployment. Create the local file <workspace>/deploy-task.yaml with the following content:

apiVersion: tekton.dev/v1beta1
kind: Task
metadata:
  name: apply-manifests
spec:
  workspaces:
  - name: source
  params:
    - name: manifest-dir
      description: The directory in source that contains yaml manifests
      type: string
      default: 'openshift/templates'
  steps:
    - name: apply
      image: appuio/oc:v4.5
      workingDir: $(workspaces.source.path)
      command: ["/bin/bash", "-c"]
      args:
        - |-
          echo Applying manifests in $(inputs.params.manifest-dir) directory
          oc apply -f $(inputs.params.manifest-dir)
          echo -----------------------------------          

source

Let’s create the task.

oc apply -f deploy-task.yaml

Expected output:

tkn task.tekton.dev/apply-manifests created

Using the Tekton CLI, verify that the task has been created:

tkn task ls

NAME              DESCRIPTION   AGE
apply-manifests                 19 seconds ago

Task 4.1.5: Create a Pipeline

A pipeline is a set of tasks, which should be executed in a defined way to achieve a specific goal.

It first uses the Task git-clone, which is a default task the OpenShift operator created automatically. This task will check out the defined git repository with the needed Dockerfile. The next task buildah will build the image. The resulted image is pushed to an image registry, defined by the image-name parameter. After that, the created tasks apply-manifest is executed. The execution order of these tasks is defined with the runAfter Parameter in the YAML definition.

Create the following pipeline <workspace>/deploy-pipeline.yaml:

apiVersion: tekton.dev/v1beta1
kind: Pipeline
metadata:
  name: build-and-deploy
spec:
  params:
    - name: git-url
      type: string
      description: git repo url
    - name: git-revision
      type: string
      description: git repo revision
    - name: deployment-name
      type: string
      description: name of the deployment to be patched
    - name: docker-file
      description: Path to the Dockerfile
      default: 'src/main/docker/Dockerfile.binary'
    - name: image-name
      description: name of the resulting image (inclusive registry)
    - name: manifest-dir
      description: location of the OpenShift templates
      default: 'src/main/openshift/templates'
  tasks:
    - name: git-checkout
      params:
        - name: deleteExisting
          value: 'true'
        - name: url
          value: $(params.git-url)
        - name: revision
          value: $(params.git-revision)
      taskRef:
        kind: ClusterTask
        name: git-clone
      workspaces:
        - name: output
          workspace: source-workspace
    - name: build-image
      taskRef:
        name: buildah
        kind: ClusterTask
      params:
        - name: TLSVERIFY
          value: 'false'
        - name: DOCKERFILE
          value: $(params.docker-file)
        - name: IMAGE
          value: $(params.image-name)
      runAfter:
        - git-checkout
      workspaces:
        - name: source
          workspace: source-workspace
    - name: apply-manifests
      taskRef:
        name: apply-manifests
      params:
        - name: manifest-dir
          value: $(params.manifest-dir)
      runAfter:
        - build-image
      workspaces:
        - name: source
          workspace: source-workspace
  workspaces:
    - name: source-workspace

source

Create the Pipeline.

oc apply -f deploy-pipeline.yaml

which will result in: pipeline.tekton.dev/build-and-deploy created

Using the Tekton CLI, verify that the Pipeline has been created:

tkn pipeline ls

NAME               AGE              LAST RUN   STARTED   DURATION   STATUS
build-and-deploy   19 seconds ago   ---        ---       ---        ---

Task 4.1.6: Prepare persistent workspace

The data for the tasks is shared by a common workspace. We use a Persistent Volume, short PV, to back our workspace and make it persistent. The PV is requested by a Persistent Volume Claim, short PVC.

Create the following resource definition for a PVC inside <workspace>/workspaces-pvc.yaml:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: pipeline-workspace
spec:
  resources:
    requests:
      storage: 2Gi
  volumeMode: Filesystem
  accessModes:
    - ReadWriteOnce

source

Create the PVC.

oc apply -f workspaces-pvc.yaml

which will result in: persistentvolumeclaim/pipeline-workspace created

Now we are ready to build and deploy our new microservice using a Tekton pipeline.

Task 4.1.7: Trigger Pipeline

After the Pipeline has been created, it can be triggered to execute the tasks. Since the Pipeline is generic, we have to provide the concrete values.

Following parameters are needed to configure the pipeline to deploy the data-transformer component:

• git-url: Git repository of the transformer
• git-revision: revision (branch/tag) of the repository
• docker-file: path to the Dockerfile inside the repository
• image-name: image name (incl. registry) of the resulting image
• manifest-dir: path to directory inside the repository that contains the yaml manifests

Create PipelineRun Resources

Creating PipelineRun Resources will trigger the pipeline.

Create the following openshift template <workspace>/pipeline-run-template.yaml:

apiVersion: template.openshift.io/v1
kind: Template
metadata:
  name: pipeline-run-template
  annotations:
    description: 'Template to create project specific PipelineRuns.'
objects:
- apiVersion: tekton.dev/v1beta1
  kind: PipelineRun
  metadata:
    generateName: build-and-deploy-run-
    labels:
      tekton.dev/pipeline: build-and-deploy
  spec:
    params:
    - name: deployment-name
      value: data-transformer
    - name: docker-file
      value: src/main/docker/Dockerfile.binary
    - name: git-revision
      value: master
    - name: git-url
      value: https://github.com/puzzle/quarkus-techlab-data-transformer.git
    - name: image-name
      value: image-registry.openshift-image-registry.svc:5000/${PROJECT_NAME}/data-transformer:latest
    - name: manifest-dir
      value: src/main/openshift/templates
    pipelineRef:
      name: build-and-deploy
    serviceAccountName: pipeline
    timeout: 1h0m0s
    workspaces:
    - name: source-workspace
      persistentVolumeClaim:
        claimName: pipeline-workspace
parameters:
- description: OpenShift Project Name
  name: PROJECT_NAME
  required: true

Create the PipelineRun by processing the template and creating the generated resources:

oc process -f pipeline-run-template.yaml \
  --param=PROJECT_NAME=$(oc project -q) \
| oc create -f-

which will result in: pipelinerun.tekton.dev/build-and-deploy-run-5wxbf created

This will create and execute a PipelineRun. Use the command tkn pipelinerun logs build-and-deploy-run-5wxbf -f -n $LAB_USER to display the logs.

The PipelineRuns can be listed with:

tkn pipelinerun ls

NAME                         STARTED         DURATION   STATUS
build-and-deploy-run-5wxbf   9 seconds ago   ---        Running

Moreover, the logs can be viewed with the following command and selecting the appropriate Pipeline and PipelineRun:

tkn pipeline logs

Execute Pipelines using tkn

Alternatively we can also trigger a Pipeline using the tkn cli.

Start the Pipeline for the data-transformer:

tkn pipeline start build-and-deploy \
  -p git-url='https://github.com/puzzle/quarkus-techlab-data-transformer.git' \
  -p git-revision='master' \
  -p docker-file='src/main/docker/Dockerfile.binary' \
  -p image-name="image-registry.openshift-image-registry.svc:5000/$(oc project -q)/data-transformer:latest" \
  -p manifest-dir='src/main/openshift/templates' \
  -p deployment-name=data-transformer \
  -s pipeline \
  -w name=source-workspace,claimName=pipeline-workspace

This will create and execute a PipelineRun. Use the same commands as listed above to check the progress of the run.

Task 4.1.8: OpenShift WebUI

Go to the developer view of the WebUI of OpenShift and select your pipeline project.

Do you remember that you did not create any Deployment for your application? That has been done by your Tekton pipeline.

With the OpenShift Pipeline operator, a new menu item is introduced to the WebUI of OpenShift named Pipelines. All Tekton CLI commands, which are used above, could be replaced with the web interface. The big advantage is the graphical presentation of Pipelines and their lifetime.

Checking your application

Check the logs of your data-transformer microservice. You will see that he will start to log average data consumed from the data stream.

Example log output:

16:05:03 INFO  traceId=, spanId=, sampled= [ch.pu.qu.re.bo.ReactiveDataTransformer] (vert.x-eventloop-thread-1) Received reactive message
16:05:03 INFO  traceId=, spanId=, sampled= [ch.pu.qu.re.bo.ReactiveDataTransformer] (vert.x-eventloop-thread-1) Current average: 0.5402286381615338
16:05:05 INFO  traceId=, spanId=, sampled= [ch.pu.qu.re.bo.ReactiveDataTransformer] (vert.x-eventloop-thread-1) Received reactive message
16:05:05 INFO  traceId=, spanId=, sampled= [ch.pu.qu.re.bo.ReactiveDataTransformer] (vert.x-eventloop-thread-1) Current average: 0.5443646390092566

High quality and secure Pipeline

This was just an example for a pipeline, that builds and deploys a container image to OpenShift. There are lots of security features missing.

Check out the Puzzle delivery pipeline concept for further information.

Links and Sources

• Tekton
• Understanding OpenShift Pipelines
• Creating CI/CD solutions for applications using OpenShift Pipelines
• Pipeline-Tutorial
• Interactive OpenShift Pipelines tutorialon learn.openshift.com

Solution

The needed resource files are available inside the folder manifests/04.0/4.1/ of the techlab github repository.

If you weren’t successful, you can update your project with the solution by cloning the Techlab Repository git clone https://github.com/puzzle/amm-techlab.git and executing this command:

oc apply -f manifests/04.0/4.1/