[Feature Request] Allow custom document parsing by skipping OS default mapping framework

[varunbharadwaj]

Is your feature request related to a problem? Please describe

The indexing request today is parsed in IndexShard to generate a ParsedDocument instance containing Lucene field instances along with the _source. This is then handed off to the Engine for indexing. For custom plugins such as the tsdb plugin (#19461), we don't really need OpenSearch indexing flow to parse the document. The parsing can be handled within the plugin (MetricsEngine) for better throughput and CPU utilization.

Benchmarks show ~18% improved ingestion throughput (wps/core) and P95 CPU utilization by skipping the default parsing and using customized parsing logic that suits the metrics use case. Under extreme load, this optimization further shows 57% improvement in wps/core and 25% P95 core utilization.

This feature request is to support such custom use cases by allowing them to bypass OS default mapping/parsing framework.

Describe the solution you'd like

Introduce a new setting skip_default_document_parsing to skip the default index mapping and document parsing in indexing flow, and instead do the following in IndexShard.

operation = new Engine.Index(
            new Term(IdFieldMapper.NAME, Uid.encodeId(sourceToParse.id())),
            new ParsedDocument(null, null, sourceToParse.id(), null, null, sourceToParse.source(), sourceToParse.getMediaType(), null),
            seqNo,
            opPrimaryTerm,
            version,
            versionType,
            origin,
            System.nanoTime(),
            autoGeneratedTimeStamp,
            isRetry,
            ifSeqNo,
            ifPrimaryTerm
        );

        return index(engine, operation);

This creates a ParsedDocument containing only the _source and other required information. The custom engine implementations can handle document parsing by looking up _source.

Related component

Indexing

Describe alternatives you've considered

No response

Additional context

No response

[msfroh]

@varunbharadwaj -- I don't like the idea of introducing this as a setting, since that essentially makes it part of the user-visible API surface. This is definitely something that only plugin developers (not users) need to worry about.

What do you think of the idea of changing the IndexShard.prepareIndex method from a static method on IndexShard to an object method on Engine? It would still be visible from the two places it's called (IndexShard#applyIndexOperation and TranslogHandler#convertToEngineOp), but custom Engine implementations could choose to override it. I think that could solve your problem in a cleaner way.

A few people (like @mch2) have put far more thought than I have into detangling the IndexShard/Engine mess. I feel like moving prepareIndex would be a step in the right direction.

[varunbharadwaj]

Good idea. That should solve the problem. 👍

[mch2]

yeah while I don't love adding more to engine...it makes sense to move this. we can do some more refactoring here for the NRT implementations where no parsing is required and remove those blocks from IndexShard. This also works with the ongoing effort to support writes to alternate formats outside of lucene where parsing now becomes engine specific and doesn't need to rely on a lucene Document as input. An alternative is to pull some abstraction in DocumentMapper/Parser but that would still need to tie with the engine implementation and will take more thought.

[mgodwan]

This is a step in the right direction towards the work being done for the proposed issue. I’ve been exploring some complementary ideas in my PR (#19475) that I think could potentially enhance the foundational idea you've established with the PR.

It introduces high-level abstractions to help decouple the shard from the Engine through bridge interfaces—still very much in progress, and I'd love to hear your thoughts and feedback!

One aspect I'm particularly covering is how we might leverage the existing Mapper code (tied to OpenSearch data types) while evolving beyond Lucene-specific field creation. This approach would let us preserve all the valuable functionality like dynamic mappings, OpenSearch parameter validation that the currently opensearch data types relies on, etc. while opening doors to supporting multiple backing formats (Lucene, Parquet, etc.).

The key idea is introducing a DocumentInput abstraction that works to support different intermediate data representation creation — the mapper would handle field type parsing at the OpenSearch layer (just as it does today), then pass those parsed values to format-specific implementations. Each format can then handle it appropriately (buffering in Lucene fields, converting to Arrow, etc.). Would love to hear thoughts on the same.

[varunbharadwaj]

I like the idea of introducing the intermediate state from the mapper layer, and letting individual engine implementations handle the data format specific logic (I might be over simplifying here). @mgodwan Nice proposal!

For the tsdb plugin specifically, the idea is similar by maintaining an intermediate state, and later apply tsdb specific logic (as of now). But the underlying core is still Lucene.