[Feature Request] Pull-based ingestion source APIs

[yupeng9]

Is your feature request related to a problem? Please describe

Introduce a set of APIs that allow for the pulling of the messages from a streaming source for ingestion.

Describe the solution you'd like

The API has the following components:

• An IngestionShardConsumer interface that can be implemented by different streaming technologies like KafkaPartitionConsumer and KinesisShardConsumer
• An IngestionShardPointer interface that points to a message in the stream, implemented as KafkaOffset, KinesisSequenceNumber and PulsarCursor
• A Message interface that is implemented based on the technology, such as KafkaMessage and KinesisMessage
• An IngestionConsumerFactory that is able to create an IngestionShardConsumer based on the technology, such as KafkaConsumerFactory and KinesisConsumerFactory
• An IngestionConsumerPlugin that is able to construct the IngestionConsumerFactory, such as KafkaPlugin and KinesisPlugin
• A ReadResult that holds both the message and the pointer to the message

More details can be found in this design doc

Related component

Indexing

Describe alternatives you've considered

No response

Additional context

No response

[soosinha]

[Triage - attendees 1 2 3 4]