[Initiative]: Specification for declaring application integration dependencies

[colinjlacy]

Name

Specification for declaring application integration dependencies

Short description

Spec that lists integration dependencies needed for an application to run successfully, e.g. DBs, message bus, etc.

Responsible group

TAG Developer Experience

Does the initiative belong to a subproject?

No

Subproject name

No response

Primary contact

Colin Lacy (colinjlacy@gmail.com)

Additional contacts

Graziano Casto (cst.grzn@gmail.com)
Alexandra Aldershaab (alexandra@housa.dk)

Initiative description

One of the common things we hear in software development is that if we package an application a certain way, it can run anywhere. While that may be true in terms of executing the command that runs it, it doesn’t guarantee that the application won’t exit 1 within a few milliseconds when it can’t connect to a database or cache, or start throwing 500s when it tries to call to a peer service that hasn’t been deployed. Yes, it can run, but can it run successfully?

This initiative would set out to create a project-agnostic specification that for declaring a service’s dependencies, so that it could indicate to an application deployment team what other services must be readily available for the application to run successful - e.g. APIs, databases, caches, blob stores, message buses, etc. This is not meant to replace conversations between application engineers and production support. Rather, it’s meant to give them a standardized reference that they can use to enable their discussion.

I say project agnostic specification because there have been attempts to build this as part of a solution in the past, e.g. Radius. Since the Radius approach packaged the spec in with the solution, the spec has only evolved to the degree that the solution can support it, most notably with a strong focus on Azure services. Making the spec project-agnostic would create a solution-independent language that can be implemented however necessary by conforming projects, including Radius. In fact, this effort would likely look to Radius, Porter, and other projects like them for inspiration. But the benefit of a spec is that it could also be leveraged for other use cases, such as guaranteeing abstraction coverage in Dapr, or for integration mocking in Microcks. Furthermore, it can support security efforts that are focused on expected application behavior, such as Software Bill of Behavior (SBOB) initiatives.

Deliverable(s) or exit criteria

• first deliverable is research and report on existing specs and previous attempts that have failed (and maybe could be resurrected) to better understand the problem space
  • the outcome of this first step will dictate the direction of subsequent steps, but the following should still apply regardless
  • this is the first potential exit point if, for whatever reason, the conclusion in the report is that this is too complex and we recommend that a spec not be created
• after the report has been delivered to the TAG, identify (and potentially extend) and existing spec, or as a last resort create a specification that indicates what services must be available to an application to ensure that it can run successfully
  • First release will be scoped to a specific integration set, e.g. external APIs, or required data stores (DB, cache)
    • TBD based on group discussion
  • The specification must exist in a format that can be consumed programmatically by spec consumers, e.g. JSON, YAML, etc.
• A plan and project description for automatic generation of dependency declarations that adhere to the spec during application development
  • to be clear, this is the description or recommendation of a project, not the project itself
• Guidance for consumers of dependency declarations for how to consume the spec and interpret it for downstream use
  • likely in the form of a short white paper

Expected timeline is 3-6 months.

[salaboy]

My two cents @colinjlacy ..

I really like the topic of this initiative, but I see a few of things that might help us to be more focused:

• Creating a specification, is hard and usually I would recommend to figure out which projects can directly benefit from something like this. You mentioned Radius and Porter, but would this be useful for CD projects? Can they use this to validate before deploy? What about policy engines like Kyverno or OPA?
• What about creating an example for this?, with just plain Kubernetes to demonstrate the problem to be solved.
• Replace Dapper for Dapr

[colinjlacy]

@salaboy thanks for the feedback. My thoughts:

• It's pretty open, so I'd say that yes, policy engines like OPA and Kyverno, as well as Cilium with its CiliumNetworkPolicy spec that could - in theory - be autogenerated from a clear spec to match expected behavior. CD projects could likely leverage this spec to ensure that dependencies exist before deploying. Both are solid points, thanks for bringing them up.
• I'm not sure about the implementation yet, which is why I'd start with the spec. Maybe it could influence existing implementations - that is, for example, influence how Radius consumes a standardized dependency set. It's something that I think is certainly doable, but I'd look for help in putting together an implementation.
• Sorry! It's corrected above. Thanks for pointing that out.

[salaboy]

Ok, so my suggestions (two cents) as we discussed in the meeting:

• Create a list of possible projects interested in this idea. This will help us to find the right people to have a scoped discussion
• Create some use cases description to be able to drive conversations and guide discussions.
• Having these list of projects and people can help us to define and scope the initiative with concrete outputs for the projects.

When I think about specs, I think about CDEvents Spec: https://cdevents.dev, which was created as an extension for CloudEvents but specifically for CI/CD use cases. For that case, the target projects were obvious, but Application Integration Dependencies can affect a wide range of projects and identifying which ones of those will be more interested in the idea will help us to prioritise work.

[mathieu-benoit]

Hi @colinjlacy and team, I would love to hear your thoughts and feedback about where the existing Score (score.dev) as CNCF Sandbox does fit within this initiative. Score is first, a Workload spec, and then have 2 implementations (score-compose and score-k8s).

Looking forward to seeing what this initiative will generate, and the learnings and feedback that the Score project could benefit from it.

The positioning of other existing projects like Radius and KubeVela would be also interesting to see.

[colinjlacy]

@mathieu-benoit thanks for taking a look! Ultimately the goal would be a tool-agnostic spec, but I'd love to see it build on the learning points from projects that have already solved problems in this space. My thought is that if it's tool agnostic, it can benefit app deployment, integration mocking, and security in a single declarative model. Tools like Score, Microcks, and Cilium could then consume that declaration file to power their own interpretations of what should happen downstream for specific use cases.

[colinjlacy]

Slack channel for this issue: https://cloud-native.slack.com/archives/C09S7A5T3GF
Running Google doc for notes: https://docs.google.com/document/d/1oz_1K1l-VuLuy-JPOh7oWrDqPzoImpYH5racSMJ5Br0/edit

[danieloh30]

Looks good to me! This should be ready to vote!

@salaboy @SwEngin @julsemaan @kdubois @cloudmelon @joshuabezaleel Feel free to add your thoughts too!

[colinjlacy]

Made additional clarifications on the deliverables per recommendation made by Yacine in the Slack channel.

[yada]

Made additional clarifications on the deliverables per recommendation made by Yacine in the Slack channel.

Thanks, @colinjlacy, and I agree regarding your comments and clarifications based on the pre-work we all need to do regarding this topic.

[danieloh30]

@riaankleinhans can you please move forward to the vote status?

[kdubois]

Looks like a great initiative! As mentioned by a few others, I'm not sure about wether we'll be able to get enough projects on board to get such a spec nailed down and broadly adopted but I think it's definitely worth a shot and I'm 100% in favor of this initiative.