Volumes, Tasks and PipelineResources overlap

[vtereso]

Background

Tekton Pipelines/Tasks separate functionality from configuration by acting as reusable components for work. Work is actualized/instantiated in the correspondingPipelineRun/TaskRun objects (noted as Run objects onward for the sake of brevity). Run objects store the work configuration through both PipelineResources and parameters.

Goals

• Understand the true intent/purpose behind PipelineResources
• Maybe remove PipelineResources
• Potentially augment volume handling in Tekton

Introduction

According to the documentation:

PipelinesResources in a pipeline are the set of objects that are going to be used as inputs to a Task and can be output by a Task.

This does not not explain their purpose, but rather their current embodiment.
To the case of resuability, it makes a lot of sense to structure things in a functional way (e.g. f(x)=y) where Pipelines/Tasks are the functions that ingest configuration. In contrast, it seems that PipelineResources seem to fall somewhere in-between.

PipelineResource Role

Although each kind of PipelineResource does something different, I believe it is reasonable to consider them as syntactically sweet mounts. In general, mounts are really helpful because they allow some foreign information to be attached to a pod/container.

Without mounts, there would only be two options:

• Create wrapper images around base images that expect configuration to be injected
• Have functionality and configuration coupled, which is less than ideal

In actuality, PipelineResources append steps to tasks and handle volumes. Since Tasks are supposed to be reusable (e.g. the catalog), it seems strange to add more carpentry to manipulate their definition. More strange, each PipelineResource does something unique rather than being a normalized operation, which also lends itself to an extensibility problem. PipelineResources would likely be more clear if their responsilities were divided between appending steps and handling volumes. It seems like the orchestration of the volume is the important piece, where the step appending is somewhat less so. PipelineResources can be reorganized as Tasks that could be composed together.

Let's look at a few different PipelineResources:

Git Resource

Git resource represents a git repository, that contains the source code to be built by the pipeline. Adding the git resource as an input to a Task will clone this repository and allow the Task to perform the required actions on the contents of the repo.

Pull Request Resource

Adding the Pull Request resource as an input to a Task will populate the workspace with a set of files containing generic pull request related metadata such as base/head commit, comments, and labels.

...

Adding the Pull Request resource as an output of a Task will update the source control system with any changes made to the pull request resource during the pipeline.

In just these two use cases (although there are more), PipelineResources do help simplify mounting. However, there is a bit of quirkiness between PipelineResources. Some resources like GitResources seem to only be input resources, while PullRequestResources are both, but likely never as inputs to other Tasks. In any case, they could very operate as Tasks rather than being shipped around between Tasks. This sort of behavior is outlined here.

All of the currently supported PipelineResources can be seen here, which is likely to grow especially with consideration for integrations with the notifications proposal. I think it's important to make a distinction on the overlap of responsibilities between Volumes, Tasks, and PipelineResources before we invest further.

PipelineResources Problems

As mentioned in the background section, Run objects take parameters and PipelineResources.
Since these are distinct objects, PipelineResources need to be created ahead of time (separately), which is an inconvenience.

As a byproduct, there is currently a proposal to allow for PipelineResource embedding into the PipelineRun (although it remains a distinct k8s resource) to address this as well as resource littering. The aforementioned extensability problem is also a concern. Further, PipelineResources can be tampered with/deleted. This introduces the cookie-cutter/templating problem, where Run objects utilizing PipelineResources (at least as current) cannot determine whether they have been modified during or between runs.

Potential Solution

Add some logic into the Tekton API to declaratively handle volumes to facilitate data between steps/Tasks in a reusable way.

There are multiple ways to do this, but ultimately this gets as making Tekton much more simple in a few different ways:

• Simplified yaml
• Cutting down on interpolation to only params.
• Reduce load on reconciler
• Clear separation of resources and more catalog contributions

This also has me wonder: If images and other fields coulds be overrided by Runs, would interpolation be necessary at all? With no interpolation (maybe this is a stretch), tools like Kustomize (not that I am familiar with it) or the sort would be able to be the engine to edit resources rather than it being done internally.

One of the benefits of PipelineResources is that they declare the expected types of inputs into and outputs out of a Task. If we were to get rid of them how could a Task describe what it operates on and what it produces?

[vtereso]

One of the benefits of PipelineResources is that they declare the expected types of inputs into and outputs out of a Task. If we were to get rid of them how could a Task describe what it operates on and what it produces?

The above approach does everything with volumes. I imagine the PipelineRun effectively mapping volumes to Tasks where there is no such thing as an input/output, but rather a volume that was previously mounted and modified. Rather than a producer/consumer paradigm, I see this sort of pipeline like an assembly line where each piece just expects it has the necessary tools (more or less the same as current).

Since execution is a DAG, the path of the volume is also known. Of course, mounting would be optional since not all tasks need to share their state. The only weird situations seems like when the DAG fans out and multiple Tasks want access to the volume. In this case, volume cloning seems like a decent solution? Alternatively, this could be restricted or they could be run sequentially in lockstep stages. This has been mentioned similarly here were restricting seems to have been the suggestion.

This approach is less structured than PipelineResources (where a certain form is expected), but I think it has considerable merit. Users will likely be more familiar with volumes, there is less black magic/obfuscation, it is natively "extendable" and it seems easy enough to debug. Thoughts @sbwsg?

Couple thoughts from my pov. I'm coming at this from a somewhat selfish point of view - I've written and run CI pipelines for a few different projects and so my reaction is based on some bad experiences with existing tools rather than necessarily from the perspective of someone writing a brand new platform on top of Tekton. I can see value in the idea of having easy generic storage passed between Tasks but I think there are considerable benefits to Tekton's declarative resources too. Anyway, here are my immediate thoughts:

1. I'm generally in favor of anything that can help users or integrating systems validate input/output, detect errors, or otherwise fail early. CI/CD pipelines can take hours - days in some cases - and discovering halfway through that you referenced a file on a volume with the wrong path and just wasted time on a run can be frustrating. Using generic volumes of data in place of PipelineResources feels like we'd be losing a lot of opportunity to do some kind of helpful linting and validating. Would you push that kind of work into Tasks or Conditions instead, or something else? I see a risk that we end up burdening a potential user more with this approach, even if they have less YAML.

2. I'm not clear which kinds of users we're aiming to help here. Do you have a sense of who benefits most from this change and why? And of the groups that it doesn't help, are there possibilities that we end up hurting their use cases instead? We've attempted to codify some of the user types here if it helps to frame the conversation: https://github.com/tektoncd/community/blob/master/user-profiles.md

3. Is there an opportunity to have both instead of one-or-other? It may be that there are really strong use cases for improved volume support and also for typed resources. I'd personally always prefer to see a feature implemented non-destructively and then naturally adopted and approved by the community as the better choice before removing the thing it's replacing.

[vtereso]

I'm generally in favor of anything that can help users or integrating systems validate input/output, detect errors, or otherwise fail early.

I think this is a really strong point and the main concern with this sort of approach. I wonder if the pipeline step/process had already been going for a long time, would the failure at this particular section be intrinsic to the volume? Maybe in the same way, a PipelineResource would have failed to be output? For concerns about mounting location, something like a volume diff of a step or restricting/recommendation all data to be stored in a single dir are my initial ideas. This way, if there is a failure relating to a volume, it is the fault of the previous step for succeeding.

Somewhat similar, if a contract needed to be fulfilled between Tasks, maybe a ConfigMap would be another option. As I see it, PipelineResources are mostly ConfigMaps wrappers anyway, albeit ConfigMaps have "arbitrary" data. Ultimately, I think it's always up to the user to correctly or incorrectly pass data around where constraining the pass data to a type like GitResource or PullRequestResource isn't infallible either (they could have empty/bogus values).

Originally, I put some consideration to using ConfigMaps to replace PipelineResources, but they still share some of the same issues such as having to be generated ahead of time of PipelineRuns/TaskRuns. However, if they were generated within PipelineRuns/TaskRuns (acting sort of like a volume), that might be more advantageous?

Is there an opportunity to have both instead of one-or-other?

I was just looking to start a discussion to consider this approach since there any many issues and conversations about improving PipelineResources and this is some sort of alternative. The release schedule includes items like PipelineResources are fixed, so I suppose this was an attempt to potentially forgo that work (many ifs), but that probably isn't realistic. My only concern is that it seems like having multiple resources encapsulate logic will always carry some caveats, where this seems better at face value. I don't claim to know everything necessary to make this happen or other potential problems this introduces. I'm not opinionated one way or the other, but I feel that if one design was satisfactory it would make sense to favor one in the long run. If considered, I agree it would only make sense to gauge adoption before any sort of removing of features.

[vtereso]

To the point about who benefits from this:

Pipeline Distributor

I think broadly speaking it makes the catalog more well defined since you don't need to create PipelineResources ahead of time, where catalog Task with only params encapsulate everything (assuming dynamic volume generation/etc.)

Platform Builder

Instead of having to fork the code base to add their own PipelineResource, they could use this new paradigm that enables anything. Currently (and AFAIK), any use case that is not covered by the PipelineResource would have to be handled by volumes/secrets in any case?

Tekton Developer

Instead of having to deal with new PipelineResources, new first-party Tasks could be added that would make things more agile than otherwise. Also, assuming this was a success and adopted (a big if), the codebase could be consolidated somewhat.

I think other roles also get indirect benefits from the above.

[vtereso]

I haven't looked https://github.com/tektoncd/pipeline/pull/1184/files over extensively, but I suppose my thinking is that this is not just a particular PipelineResource kind, but could effectively model any inter-step communication. Then taking this one step forward, do so in a more flexible way that doesn't have the pain points as current. The above ConfigMap idea is more or less a generic way of passing around params between tasks that is a hybrid/middleground between volumes and current PipelineResources.

I wonder if #1285 alleviates some of the same pain. A FileSet is essentially an "untyped" group of files to be output from one Task and input to another but a benefit I see there is that the paths of those files is declared as part of the resource.

[vtereso]

To the same point of PipelineResources being dicey, I don't feel particularly strong about one PipelineResource being able to reference another as this further compounds/complicates things. I do like the idea between being able to select between raw data and config maps. Would the raw data support a compressed directory? That would essentially satisfy the "volume" approach. In any case, I think the volume PipelineResource and FileSet PipelineResource are a step toward being more generic, where at the very least, extensibility would be a non issue.

[imjasonh]

@vtereso I'm having trouble imagining a world without PipelineResources.

Can you give me an example pipeline (perhaps a version of this canonical example pipeline) that doesn't define any PipelineResources? How would a Task know what a previous Task produced, and how would the overall PipelineRun report on the new state of things updated during its run?