Kubeflow Pipelines Tekton Backend Optimizations

10/23/2023

This post details optimizations for Kubeflow Pipelines V2.0 on the Tekton backend. Key contributions include the development of a dedicated Tekton custom task controller to handle repetitive driver and publisher tasks more efficiently, reducing latency and improving throughput. Additionally, it describes integrating Kubeflow Pipelines V2.0 driver and task construction logic into a single Tekton reconciliation cycle, which reduces the number of queued tasks by 50% and streamlines processing.

9/19/2021

Introduces the `finally` clause in Tekton Pipelines (v0.14.0) for executing tasks after pipeline completion, regardless of success or failure. Details use cases such as sending notifications and cleaning up resources. Explains how `finally` tasks affect the overall `pipelineRun` status. Introduces features for conditional execution of `finally` tasks using `when` expressions and accessing task execution status (`$(tasks.<task-name>.status)`, `$(tasks.status)`). Describes how to pass task results to `finally` tasks (`$(tasks.acquire.results.leased-resource)`). Details graceful termination of pipeline runs with `CancelledRunFinally` and `StoppedRunFinally` statuses.

4/29/2020

This post details the process of adding Python support to Tekton Pipelines, building upon existing work for Java and JavaScript. It outlines three key tasks: installing app dependencies using a virtual environment, packaging the application source code and libraries into a zip file, and modifying the Dockerfile for the OpenWhisk runtime using Kaniko Executor. Specific technical challenges and solutions are discussed, such as handling long lines in Dockerfiles with `fold` and `sed` commands for base64 encoded zip files.