Support `referenceIntoComponents` for other components than `message`

[thake]

Reason/Context

Currently, the referenceIntoComponents option only seems to work for message components. However, it can also be useful for all other component types (see https://www.asyncapi.com/docs/reference/specification/v2.5.0#componentsObject).

Description

Changes:

• Extend https://github.com/asyncapi/bundler/blob/master/src/parser.ts so that it detects all kinds of references.

Although the bundled spec will change, I would not consider this feature as a breaking change. Implementing this feature would only mean the correct implementation of the current documentation of the referenceIntoComponents flag:

Pass true to resolve external references to components.

[github-actions]

Welcome to AsyncAPI. Thanks a lot for reporting your first issue. Please check out our contributors guide and the instructions about a basic recommended setup useful for opening a pull request.
Keep in mind there are also other channels you can use to interact with AsyncAPI community. For more details check out this issue.

[derberg]

I think if maybe we should plug in https://github.com/asyncapi/optimizer/ or not even plug in but simply describe how both should be used 🤔

cc @KhudaDad414 @Souvikns

[Souvikns]

While building reference-into-components feature I thought I could use optimiser for resolving references but that was not possible, what we can do is optimize the document after all the referencing is done, will open a proposal for this soon.

[derberg]

@Souvikns do you remember the reason? it would be good to have one mechanism for things like this. But of course only if it makes sense and will help and not complicated life

[aeworxet]

While trying to find out what would be better architecturally:

• searching for objects that can be moved to components, in predefined places (like crawlChannelPropertiesForRefs()
  https://github.com/asyncapi/bundler/blob/42a2b189e8303b57f8fcbc14c721637838ae504f/src/parser.ts#L33..L35
  that searches for a known in advance object in a predefined place)

• or going from list of objects that CAN exist in Components Object

schemas
servers
channels
operations       <-- will be added in specification 3.0.0
messages
securitySchemes
serverVariables
parameters
correlationIds
externalDocs     <-- will be added in specification 3.0.0
tags             <-- will be added in specification 3.0.0
operationTraits
messageTraits
serverBindings
channelBindings
operationBindings
messageBindings

and search, say, for message through all schema (iterate through all properties of the schema object), and if it finds a message (with a $ref which is an external $ref), then this $ref is dereferenced, and full dereferenced message object is moved to components: { messages: {} },

I found that there exists one big pro in the second approach, which will make the solution more generic (@magicmatatjahu): if objects move around the schema as specification changes with time, nothing would change for Bundler who in search of suitable objects was iterating through all of the schema object from the start.

And this led me to using @derberg's and @Souvikns's idea of utilizing Optimizer as much as possible.

'As much as possible' I see as adding to Optimizer the ability to dereference external references before starting the optimization of a given schema, and then Bundler will only merge schemas Optimizer will give out to it (because by logic, Optimizer is meant to OPTIMIZE ONE given schema, and Bundler is meant to BUNDLE SEVERAL schemas into one).
@KhudaDad414

@jonaslagoni do you have additions or comments?

[Souvikns]

@Souvikns do you remember the reason? it would be good to have one mechanism for things like this. But of course only if it makes sense and will help and not complicated life

optimizer currently does not resolve external references, and only works with internal references, so to build the reference-into-components we could not use optimizer. But we can use optimizer after the bundling step when all the external references are resolved and files are merged, then optimizer can optmize the document since there is only internal references to work with. (currently only true for messages)

'As much as possible' I see as adding to Optimizer the ability to dereference external references before starting the optimization of a given schema, and then Bundler will only merge schemas Optimizer will give out to it (because by logic, Optimizer is meant to OPTIMIZE ONE given schema, and Bundler is meant to BUNDLE SEVERAL schemas into one).
@KhudaDad414

Are you suggesting that we move the external reference logic to optimizer?

[aeworxet]

Are you suggesting that we move the external reference logic to optimizer?

Not move but copy. Extend Optimizer's functionality. And Bundler will use it after extension.

I can even take such a task.

The question is whether there is a need in such functionality, and if it doesn't go against Optimizer's concept (it might be intended for local use only.)
For now AFAIK external references dereference parser-js and bundler. Those are only the ones that I know. So such functionality might become available in all AsyncAPI Initiative tools by default. (it might be even time already to create a library that holds all reusable code 😆)

[KhudaDad414]

TBH, I don't think moving components to the components section is ideal unless we gain some specific benefits. why? since it adds complexity and makes the specification file harder to read for humans.
In the optimizer, we accept this complexity since it provides reusability by only moving components to the components section if there is more than one copy of the same component.

If I only have one copy of a message why would I want to move it to the components section? I want to have it where it belongs.

I think we should let each of these tools do its thing. let the bundler bundle the specification into one file and let the optimizer check the output of the bundler and move components to the components section if there are reusability benefits.

[derberg]

The thing is that it is a recommendation to put as much as possible in components. At least I always recommend this way.

Reading huge YAML is hard, but when things are moved into components it is easier. As long as you read YAML in editor that supports $ref that allows you to click it and go into a given reffed component.

But of course moving something to components, if there is just one ref that will use it, do not make much sense either 😄

---

anyway 😄

bundler -> optimizer user flow makes much more sense to me really.

At the library level, we just explain what people should do, and we make sure that these libraries are not heavy.
At CLI or Studio level, we can abstract it in a way that you can still bundle or optimize separately but we also enable that when you bundle, you can also pass a flag like --moveToComponents (just quick idea, to be discussed once work on CLI happens) that will trigger optimization after bundling (same can be done in Studio).

[magicmatatjahu]

Yes, the bundler is supposed to do one thing and the optimizer something else, combining it should only be done in integrations like CLI or Studio. As for the approach, as you wrote yourself, the generic approach is better. I'm just wondering whether we should use the API from the parser to bundle external references more easily, or rather write all the logic from scratch. Of course, the parser (for now) does not provide all the data needed for proper bundling (e.g. the original reference of the de-referenced object), but if it did, it would make the work on the bundler much easier.

Currently, you can create this generic function and try to bundle only the message as the bundler currently works, and in other PRs we can add other components to bundle - to do it iteratively.

[magicmatatjahu]

On the other hand, bundler could work as a combination of parser + optimizer, because at the end, the bundled spec should look identical if you use a parser and then an optimizer - but for now let's not deal with it because it has some problems in implementation.

[thake]

I just want to highlight a potential problem if moving to components is done in a separate step after bundling:

The $ref usually also carries a semantical meaning to understand easier what it is (example "$ref : financial-system.yaml#/components/schemas/bankAccountIdentifier"). If the bundling just resolves this ref inline, the semantical meaning of the $ref pointer gets lost and cannot be recovered in later steps. The optimizer would need to invent an artificial component name for the "bankAccountIdentifier" when moving it to the components section.

Because of this, I think moving to the components section needs to be done in the bundler itself.

[aeworxet]

Conversion of this high-level discussion into a technical task.

Bundler does only one job: it bundles several specifications into one file (dereferencing external $refs), optionally aggregating all AsyncAPI Specification-legal components in the components object. Anything more than this is the concern of other tools.

Bundler accepts three options:

referenceIntoComponentsAll: true | false
referenceIntoComponentsOptimize: true | false
saveOriginsOfRefs: true | false

referenceIntoComponentsAll || referenceIntoComponentsOptimize

ONE, and ONLY ONE, of options referenceIntoComponentsAll and referenceIntoComponentsOptimize can equal true.

• If both options equal true, referenceIntoComponentsAll takes higher priority, and the referenceIntoComponentsOptimize is forced to equal false.
• If both options equal false, Bundler only bundles and dereferences external $refs, without searching for AsyncAPI Specification-legal components and performing any optimization on the resulting specification.
• If none of two options is specified, they are both considered false.
• If one of two options is not specified, the non-specified option is considered false.

referenceIntoComponentsAll (optional) (referenceIntoComponents is also queried, to provide backward compatibility with existing referenceIntoComponents, issuing a warning of future deprecation)

If equals true, 'Bundler only' workflow is used.
ALL, and EVERY, AsyncAPI Specification-legal component is moved into components object, regardless of optimization considerations.
Bundler's internal generic mechanism to find every AsyncAPI Specification-legal component throughout all of the schema object is used in this case.

referenceIntoComponentsOptimize (optional)

If equals true, 'Bundler -> Optimizer' workflow is used.
Bundler bundles several specifications into one file (dereferencing external $refs) and lets the Optimizer check the output.
Optimizer moves components to the components object if there is more than one copy of the same component (if there are reusability benefits).

saveOriginsOfRefs (optional)

If equals true, Bundler creates list of origins of external $refs via Specification Extensions, in form

{
  "x-origins-of-refs": {
    "#/components/schemas/bankAccountIdentifier": "./financial-system.yaml#/components/schemas/bankAccountIdentifier"
  }
}

to keep traceability of where each resolved external $ref came from.

Please react with like/dislike/seen or add comments.