Publishing events via Push mechanism

[michael-danko-passport]

Is your feature request related to a problem? Please describe.

Currently the CDS Events API defines a query-able API to search for events that have occurred. Since this API is defined as a GET request, consumers of the API will need to parse through the data and compare the event's event_id to those already ingested and deduplicate any events that have already been received.

With a high volume of events occurring at the curb within a city, this could become a computationally intensive process to iterate through the array of events and identify which events are the new events that need to be ingested.

Describe the solution you'd like

A method to receive the data defined in our events as a "push" where the source system that creates the event can initiate the integration and send the data to a receiving API endpoint. Events could still be provided in batches to support any preprocessing of data.

Is this a breaking change

This would not be a breaking change since it is a net-new addition to the specification.

Impacted Spec

For which spec is this feature being requested?

• Events

Describe alternatives you've considered

I believe the only alternative is the existing API endpoint, events that provides a method to query the data as a consuming system. In my experience, this can be difficult to scale as consuming applications will consistently want the latest data and decreasing the time between queries will increase the load placed on the source system.

[michael-danko-passport]

@schnuerle Here is the issue that I opened following our discussion in today's WGSC. Feel free to add any additional notes with context you can share.

[schnuerle]

In MDS, it has always had complete different APIs for push and pull. Endpoints in Provider for pull and Agency for push. They've always been separate and different in endpoints and fields, but over time they have converged and aligned more. In MDS 2.0 there is a push to align them totally (same endpoints and fields with only difference being push/pull calls) or make them new identical (same fields and data, but keeping them in Provider and Agency).

See details of this work here. The two specs could learn from each other on how to implement this. openmobilityfoundation/mobility-data-specification#759

[michael-danko-passport]

This was discussed in today's CDS WG meeting (7/26). It was highlighted that there could be advantages to supporting both a push based and pull based API within the spec. A push mechanism could better support near-real time data transfers and a pull mechanism would be able to help with the less frequent reconciliation of data and solve for concerns where a push based system intermittently fails to deliver events.

In both formats, the schemas for what is sent should be aligned to help with the interoperability of both formats.

There were also discussions that referenced the Curb Architectural Decisions document on this subject.
https://github.com/openmobilityfoundation/curb-data-specification/wiki/Curb-Architectural-Decisions#push-vs-pull

[schnuerle]

I would like to explore this and see how we could keep the same data structures, but have a push and a pull option. We are diving into this in MDS this week, so maybe we can take some learnings and decisions from there to use in CDS.

[jacobmalleau]

@schnuerle looks like there is already an open issue for having Events having a Push option - we should discuss in the next meeting.

[schnuerle]

We could implement this in a similar way to how we do push events in MDS with a new endpoint.

[schnuerle]

We will be talking about this at tomorrow's public Curb Working Group meeting.