[PROPOSAL] Neural Search field type

[asfoorial]

Can we mimic this feature in OpenSearch https://www.elastic.co/search-labs/blog/semantic-search-simplified-semantic-text

I know that a lot has been done recently in OpenSearch projects to make things headache free. I think a neural-search field type in OpenSearch would be an interesting addition. However, it should account for synonyms to avoid any fine-tuning headache.

[navneet1v]

@asfoorial from Opensearch side we do this via a combination of ingestion processor and vector field. As there are multiple use-cases for semantic search including multi-model, this would be an interesting field to have.

But is there any specific reason you are looking for the field as compared to what is present currently. My main motive here is to know the advantages of a new field vs what is currently present in opensearch.

[asfoorial]

The main reason is simplifying the process and keep the focus on the business. In fact elasticsearch had the same reason when they introduced the field.

Another reason is alignment of new features across multiple OpenSearch projects. I have noticed over the past number of releases we get new features in ml-commons and kNN. But it takes a while until we see their benefits reflected in neural-search. If they become one component (neural-search field), then that would sort of guarantee that any new feature in ml-common or kNN must be reflected in the neural-search field type before their release.

[navneet1v]

If they become one component (neural-search field), then that would sort of guarantee that any new feature in ml-common or kNN must be reflected in the neural-search field type before their release.

@asfoorial thanks for providing the details. I want to know little bit more on what features added in ML/k-NN doesn't make into Neural. May be there is something missing.

But I really like the idea of having a field which can encapsulate the processor information.

[navneet1v]

One place where having the field will be useful is nested fields. I see putting this information in the processor is very painful and not intutive.

[navneet1v]

@minalsha please take a look into this and please add your thoughts

[heemin32]

I think this is a good idea as it simplifies the use of neural search significantly. By defining a neural field, all other processes, such as the neural search pipeline, neural ingestion pipeline, KNN index creation, chunking, and more, will be handled behind the scenes.

[navneet1v]

@heemin32 any reason for closing this gh issue?