Kubeflow Pipelines Tekton Backend Optimizations

3 posts · Apr 2020 – Oct 2023

What this trail covers

Kubeflow Pipelines V2.0 introduces a platform-neutral Intermediate Representation (IR) for ML workflows, improving portability and metadata richness. Optimizations for the Tekton backend include a dedicated custom task controller to reduce latency and throughput for drivers and publishers, and integration of driver and task construction logic into a single reconciliation cycle to decrease the number of queued tasks. The `finally` clause in Tekton Pipelines allows for guaranteed execution of tasks. This post details adding Python support to Tekton Pipelines, enabling OpenWhisk Python Actions to run on Knative by creating a CI/CD pipeline that packages Python applications and their dependencies into Docker images.

All IBM trails

Oct 2023Tekton Optimizations for Kubeflow Pipelines 2.0This post details optimizations for the Tekton backend of Kubeflow Pipelines V2.0. It introduces a dedicated custom task controller to improve latency and throughput for drivers and publishers, and integrates driver and task construction logic into a single reconciliation cycle to reduce queued tasks. It also highlights performance improvements for cached tasks compared to the Argo backend.
Sep 2021Add finally tasks to Tekton pipelinesIntroduced the `finally` clause in Tekton Pipelines (v0.14.0) to execute tasks after all main tasks complete, regardless of success or failure. Detailed use cases for `finally` tasks including sending notifications, cleaning up resources, and graceful termination. Explained how `finally` tasks affect `pipelineRun` status. Introduced features for conditional execution of `finally` tasks using `when` expressions and accessing task execution status (`$(tasks.<task-name>.status)`, `$(tasks.status)`). Described how to pass task results to `finally` tasks. Detailed graceful termination mechanisms for `pipelineRun`s using `CancelledRunFinally` and `StoppedRunFinally` statuses.
Apr 2020Add Python support to Tekton PipelinesThis post details the addition of Python support to Tekton Pipelines, building upon existing Java and JavaScript pipeline work. It outlines the architecture of Tekton Pipelines (Tasks, Steps, Pipelines, PipelineRuns) and describes the three main tasks involved in creating a Python CI/CD pipeline: 1. Installing a virtual environment and necessary app dependencies using pip. 2. Packaging the app source code and filtered virtual environment into a zip file. 3. Modifying the OpenWhisk runtime Dockerfile to include the zipped code as an environment variable and building the image using Kaniko Executor. It also provides an example PipelineRun and Knative service deployment.
Advertisement