Allow defining payloads with non-JSON standards (XSD, Protobuf, etc.)

[jonaslagoni]

I choose to start a new issue to map out more precisely what is needed to support this, without any specific sight on one approach or the other. This is a generic solution that would work with all non-JSON standards regardless of which one.

This was first pursued for V3 of the spec, but I don't believe that it will be possible for that deadline. Therefore I gathered the current progress to have one place to discuss it so it can be removed from the progress list.

Steps currently being taken:

• Figure out if we need a new referencing standard or not.
  • This is not explicitly agreed upon, but so far it seems like it's not sufficient with the current standard.
• If we do, which one should we go with?
  • Alternatives here: https://github.com/json-schema-org/referencing
    • Overall perspective: An AsyncAPI perspective json-schema-org/referencing#14
    • JRI perspective: JRI from an AsyncAPI perspective json-schema-org/referencing#17
    • JRef perspective: JRef from an AsyncAPI perspective json-schema-org/referencing#16
  • Custom alternative: feat: introduce new schema referencing standard #825
• Implement a reference handling tool in TypeScript (this is bare minimum) that can handle multiple format types and be context dependant.
  • Look through the requirement discussion here: https://github.com/orgs/asyncapi/discussions/485

Related resources:

• Proposal to allow defining schema format other than default one (AsyncAPI Schema) #622
• Proposal for Unparsable Remote References (mainly to support XML.) #624
• [2.0.0 REVIEW] Clarify usage of JSON References (/Pointers) for non-JSON data structures #216
• Support more than one schema #434
• https://github.com/orgs/asyncapi/discussions/485
• feat: introduce new schema referencing standard #825
• https://github.com/json-schema-org/referencing
• Proposal to enable support for alternate schemas OAI/OpenAPI-Specification#764
• Proposal: x-oas-draft-alternativeSchemas  OAI/OpenAPI-Specification#1532

[derberg]

I looked into the linked draft PR and the long discussion there. It was hard to follow as there was a lot of OpenAPI and JSON Schema "history" involved and I got lost.

just to make it clear: for me this is clearly only about the message payload referencing so referencing of schemas only, not referencing other parts of AsyncAPI

what I understand is:

• the community needs a way to reference the payload of schema in xsd or proto
• I assume we will need also something for GraphQL at some point of time
• sometimes it is about referencing just a file, sometimes part of the file, as both in xsd and proto you can have some specific parts to reference too, like you can have multiple messages in one proto file
• $ref cannot be used as it is for JSON

How about solving it by having another way of referencing things:

• so alongside Schema Object, Reference Object we could have something like Schema Reference Object
• ServerlessWorkflow spec has it already (different structure, pointing them out as example of custom pointers) and it makes lots of sense -> https://github.com/serverlessworkflow/specification/blob/main/specification.md#workflow-functions and the structure is pretty clear
• long term Reference Object could be deprecated (AsyncAPI 5.0 - although I do not have strong opinion on deprecation need) and people would always use Schema Reference Object

So in short $ref is just for bundling multiple JSON/YAML AsyncAPI structures into one, referencing inside document different structures. It is better and more intuitive for users to reference schemas in a more defined way when you can say that whatever comes after # is a proto message:

"reference": "https://gist.githubusercontent.com/iamsurajgharat/1d43d8181f286a58617adebce1714c6f/raw/343d91d8e3f09ab717f50bffa3213a7fd8fb0e13/sumservice.proto#SumInput"

or xsd element:

"reference": "https://gist.githubusercontent.com/iamsurajgharat/1d43d8181f286a58617adebce1714c6f/raw/343d91d8e3f09ab717f50bffa3213a7fd8fb0e13/sumservice.proto#ldml"

the only problem I see is with the schemaFormat. I'm not 100% sure yet how that would play. The more I think about it, my view is that schemaFormat should not be defined outside payload as it is now.

PS. I do not worry that we will not have out-of-the-box tooling support as we have for $ref. We will just make sure to have current Studio and IDEs plugins support it and that should be really enough.

[jonaslagoni]

just to make it clear: for me this is clearly only about the message payload referencing so referencing of schemas only, not referencing other parts of AsyncAPI

Agreed, I think it will just over complicate things otherwise.

So in short $ref is just for bundling multiple JSON/YAML AsyncAPI structures into one, referencing inside document different structures. It is better and more intuitive for users to reference schemas in a more defined way when you can say that whatever comes after # is a proto message:

That would definitely be possible yes.

Do you know if serverlessworkflow has a core tool for solving references we could maybe use if this became the case?

PS. I do not worry that we will not have out-of-the-box tooling support as we have for $ref. We will just make sure to have current Studio and IDEs plugins support it and that should be really enough.

That is a contradicting sentence, cause Studio and IDE plugins can't support it without the core tool in place 🤷 Unless they implement the functionality themself of course, but then it might as well be in a separate tool everyone can integrate. Or did I misunderstand you? 🤔

[jonaslagoni]

So started specific discussions for the 2 proposals in JSON Schema referencing repository and answered the outstanding clarification questions in separate issues for each standard

• JRI: JRI from an AsyncAPI perspective json-schema-org/referencing#17
• JRef: JRef from an AsyncAPI perspective json-schema-org/referencing#16

They seem to both support what we want, and leave it up to the context specification (AsyncAPI) to dictate the behavior necessary for referencing non-JSON Standards.

[derberg]

Do you know if serverlessworkflow has a core tool for solving references we could maybe use if this became the case?

I do not think they have any standard there. Their needs are also super specific, like for example in case of referencing AsyncAPI, reference points to operationId, so they for sure just pull out the message payload of that operation "manually" without any standard in use.

Regarding JRI or JRef -> I could not find any reference on internet, what are these standards all about? and are they useful if there are no reference implementations anyway?

[GreenRover]

+1
I am interested in protobuff support as well.

[GreenRover]

@derberg i dont see GraphQL support.
Because as i understand GraphQL it is HTTP based only. And dont know concepts like channels.
Or differently sayed: channels/operations are all parts of the GraphQL schema.
So GraphQL is more a competitor of AsyncAPI as something that could be integreated

[derberg]

@GreenRover well, maybe not competitor, but yeah it is different spec for different use cases, and it has 2 things in one, synchronous queries and mutations and asynchronous subscriptions.

What I meant by mentioning GraphQL is specifically GraphQL schemas. Just like with OpenAPI, we support that someone can reference OpenAPI Schema in AsyncAPI document. Can be that someone already has schemas provided for GraphQL API, and they could reference them in AsyncAPI docs for other service. But there is nothing custom here, whatever solution we will have for XSD or Proto will work with GraphQL as it is very similar to Proto as far as I remember (didn't use it for over 3 year 😅 )

[rober15]

I think we have as well to consider another type of async messaging added now as well to Open API:
https://swagger.io/docs/specification/callbacks/

So i wonder how to handle that Callbacks which can be used for webhooks but shown under Open API which most people associate more with synchronous messaging and not as well async.

For the message other message formats a fast help could be a reference to other schemas.
That at least would allow companies to show all async apis in one place no matter what format they use to exchange.

Maybe similar to #ref we could have #extref

[derberg]

@rober15 we will "soon" release AsyncAPI 3.0 where one of features is support for request/reply in EDA, so you can specify that operation B is a reply to operation A or specify that reply to operation A should go to dynamically specified target, through the header info, for example