feat: Created DocEmbedder class by patelchaitany · Pull Request #5973 · feast-dev/feast

patelchaitany · 2026-02-16T11:28:33Z

What this PR does / why we need it:

This PR adds a Document Embedder capability to Feast, allowing users to go from raw documents to embeddings stored in the online vector store in a single step. It handles chunking, embedding generation, and writing the results to the online store — providing an end-to-end ingestion pipeline for RAG workflows within Feast.

What changed:

sdk/python/feast/chunker.py

Defines the document chunking layer. Provides:

A BaseChunker abstract class that users can extend with custom chunking strategies
A built-in TextChunker that splits plain text by word count with configurable size, overlap, and thresholds
A chunk_dataframe() convenience method to chunk all documents in a DataFrame

Currently only basic text chunking is implemented. There is room for improvement — future iterations can support more advanced strategies like semantic chunking, sentence-aware splitting, or format-specific chunkers (PDF, HTML, etc.).

sdk/python/feast/embedder.py

Defines the embedding generation layer. Provides:

A BaseEmbedder abstract class with a modality-routing registry for extensibility
A MultiModalEmbedder with built-in support for text (via sentence-transformers) and image (via CLIP) embeddings
Lazy model loading and configurable batch processing

sdk/python/feast/doc_embedder.py

The high-level orchestrator that coordinates chunking, embedding, and storage. Provides:

A DocEmbedder class that runs the full pipeline: chunk -> embed -> logical layer -> write to online store
A user-defined logical layer function to map output to the FeatureView schema
Auto-generation of a default FeatureView if the user does not define their own (controlled by create_feature_view). This is a basic implementation intended as a starting point

sdk/python/feast/init.py

Updated to export DocEmbedder, LogicalLayerFn, BaseChunker, TextChunker, ChunkingConfig, BaseEmbedder, MultiModalEmbedder, and EmbeddingConfig as part of Feast's public API.

Which issue(s) this PR fixes:

Create DocEmbedder class along with RAGRetriever #5426

Misc

patelchaitany · 2026-02-16T11:53:20Z

@ntkathole @jyejare can You pls review this PR and let me know if any changes is needed.

jyejare

Great Addition @patelchaitany , this is a milestone for Feast in RAG. Glad to see multiple types of data are being supported by Embedder.

Few comments and we should be good to go.

jyejare · 2026-02-23T05:16:03Z

sdk/python/tests/unit/test_doc_embedder.py

+        chunker = TextChunker()
+        text = " ".join([f"word{i}" for i in range(200)])
+
+        chunks = chunker.load_parse_and_chunk(source=text, source_id="doc1")


I think chunker should decide the text in source to chunk the text, we should not need to manually feed that.

Yes, Chunker can decide the which field is text in the DataFrame We Just need to pass the column name in the chunk_dataframe function of the Chunker class.

This test only for testing that load_parse_and_chunk function return the match the required return type.

jyejare · 2026-02-23T05:33:27Z

sdk/python/tests/unit/test_doc_embedder.py

+    def test_supported_modalities(self):
+        """After init, supported_modalities returns text and image."""
+        embedder = MultiModalEmbedder()
+        modalities = embedder.supported_modalities()


Supported modalities can be set as a property

sdk/python/tests/unit/test_doc_embedder.py

jyejare · 2026-02-23T10:46:44Z

sdk/python/tests/unit/test_doc_embedder.py

+        assert embedder._image_model is None
+        assert embedder._image_processor is None
+
+    def test_custom_modality_registration(self):


Repeat test ?

This is for when we register new Modality then It will correctly route to the New Modality.

But I agree we can remove this test.

ntkathole · 2026-03-02T10:58:02Z

@patelchaitany filename typo - examples/rag-retriever/rag_feast_docebedder.ipynb should be rag_feast_docembedder.ipynb

sdk/python/feast/doc_embedder.py

sdk/python/feast/embedder.py

sdk/python/feast/chunker.py

ntkathole · 2026-03-02T11:18:17Z

sdk/python/feast/embedder.py

+from dataclasses import dataclass
+from typing import Any, Callable, List, Optional
+
+import numpy as np


Consider lazy-loading numpy too

we cannot do the lazy of numpy as it is required for the Type checking.

Gate this behind type check

from typing import TYPE_CHECKING if TYPE_CHECKING: import numpy as np

importing feast will fail if numpy is not installed

sdk/python/feast/embedder.py

…ng them into the FeatureView schema. - Added BaseChunker and TextChunker classes for document chunking. - Updated pyproject.toml to include sentence-transformers dependency. - Created a new Jupyter notebook example for using the RAG retriever with document embedding. Signed-off-by: Chaitany patel <patelchaitany93@gmail.com>

ntkathole · 2026-03-06T15:15:34Z

sdk/python/feast/doc_embedder.py

+        id_column: str,
+        source_column: str,
+        type_column: Optional[str] = None,
+        column_mapping: Optional[dict[str, tuple[str, str]]] = None,


column_mapping I think simply can be tuple, column_mapping[source_column] is only used

ntkathole · 2026-03-06T15:17:25Z

sdk/python/feast/doc_embedder.py

+{feature_view_name.replace(" ", "_").replace("-", "_")}_source = FileSource(
+    name="{feature_view_name}_source",
+    path="data/{feature_view_name}.parquet",
+    timestamp_field="event_timestamp",
+)
+
+# FeatureView
+{feature_view_name.replace(" ", "_").replace("-", "_")} = FeatureView(
+    name="{feature_view_name}",


@patelchaitany this is valid

ntkathole · 2026-03-06T15:17:34Z

sdk/python/feast/doc_embedder.py

+    name="{feature_view_name}_source",
+    path="data/{feature_view_name}.parquet",
+    timestamp_field="event_timestamp",
+)
+
+# FeatureView
+{feature_view_name.replace(" ", "_").replace("-", "_")} = FeatureView(
+    name="{feature_view_name}",


needs to resolve

ntkathole · 2026-03-06T15:18:43Z

sdk/python/feast/embedder.py

+from dataclasses import dataclass
+from typing import Any, Callable, List, Optional
+
+import numpy as np


Gate this behind type check

from typing import TYPE_CHECKING if TYPE_CHECKING: import numpy as np

importing feast will fail if numpy is not installed

ntkathole · 2026-03-06T15:22:04Z

sdk/python/feast/doc_embedder.py

+                feature_view_name=feature_view_name,
+                vector_length=resolved_vector_length,
+            )
+        self.apply_repo()


Consider making this explicit, may be a separate setup() method or a auto_apply=True parameter. The constructor always calls apply_repo(), which invokes apply_total, a heavy operation that touches the registry

ntkathole · 2026-03-06T15:25:09Z

sdk/python/feast/doc_embedder.py

+        self.embedder = embedder or MultiModalEmbedder()
+        self.store: Optional[FeatureStore] = None
+
+        if isinstance(logical_layer_fn, LogicalLayerFn):


what is isinstance check for ? it's not validating anything

ntkathole · 2026-03-06T15:28:51Z

pyproject.toml

 rag = [
    "transformers>=4.36.0",
    "datasets>=3.6.0",
+    "sentence-transformers>=3.0.0"


Suggested change

"sentence-transformers>=3.0.0"

"sentence-transformers>=3.0.0",

ntkathole · 2026-03-06T15:32:44Z

sdk/python/feast/chunker.py

+                source_column,
+                row[type_column] if type_column else None,
+            ),
+            axis=1,


def.apply with axis=1 is going to be slow for large dataframes, instead use simple list compression

all_chunks = [] type_values = df[type_column] if type_column else [None] * len(df) for src, doc_id, doc_type in zip(df[source_column], df[id_column], type_values): chunks = self.load_parse_and_chunk(src, str(doc_id), source_column, doc_type) all_chunks.extend(chunks) if not all_chunks: return pd.DataFrame( columns=["chunk_id", "original_id", source_column, "chunk_index"] ) return pd.DataFrame(all_chunks)

patelchaitany requested a review from a team as a code owner February 16, 2026 11:28

This comment was marked as resolved.

Sign in to view

patelchaitany changed the title ~~feat: Created DocEmbedder class #5972~~ feat: Created DocEmbedder class Feb 16, 2026

This comment was marked as resolved.

Sign in to view

jyejare reviewed Feb 23, 2026

View reviewed changes

patelchaitany requested a review from jyejare February 24, 2026 10:01

This comment was marked as resolved.

Sign in to view

ntkathole force-pushed the fet-DocEmbedder branch from 1ef9d8e to 083eadb Compare March 2, 2026 05:15

ntkathole added the ok-to-test label Mar 2, 2026

ntkathole force-pushed the fet-DocEmbedder branch from 083eadb to bb74079 Compare March 2, 2026 07:52

ntkathole reviewed Mar 2, 2026

View reviewed changes

sdk/python/feast/doc_embedder.py Outdated Show resolved Hide resolved

ntkathole reviewed Mar 2, 2026

View reviewed changes

sdk/python/feast/doc_embedder.py Outdated Show resolved Hide resolved

ntkathole reviewed Mar 2, 2026

View reviewed changes

sdk/python/feast/doc_embedder.py Outdated Show resolved Hide resolved

ntkathole reviewed Mar 2, 2026

View reviewed changes

sdk/python/feast/embedder.py Show resolved Hide resolved

ntkathole reviewed Mar 2, 2026

View reviewed changes

sdk/python/feast/chunker.py Outdated Show resolved Hide resolved

ntkathole reviewed Mar 2, 2026

View reviewed changes

sdk/python/feast/embedder.py Show resolved Hide resolved

ntkathole removed the ok-to-test label Mar 2, 2026

jyejare approved these changes Mar 3, 2026

View reviewed changes

patelchaitany force-pushed the fet-DocEmbedder branch from daf292f to fcc85cd Compare March 4, 2026 05:38

This comment was marked as resolved.

Sign in to view

patelchaitany force-pushed the fet-DocEmbedder branch 2 times, most recently from e00ee22 to ee663f5 Compare March 5, 2026 06:31

ntkathole force-pushed the fet-DocEmbedder branch from ee663f5 to 282243b Compare March 6, 2026 15:06

ntkathole reviewed Mar 6, 2026

View reviewed changes

	"sentence-transformers>=3.0.0"
	"sentence-transformers>=3.0.0",

Conversation

patelchaitany commented Feb 16, 2026 • edited by devin-ai-integration bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What this PR does / why we need it:

What changed:

sdk/python/feast/chunker.py

sdk/python/feast/embedder.py

sdk/python/feast/doc_embedder.py

sdk/python/feast/init.py

Which issue(s) this PR fixes:

Misc

Uh oh!

This comment was marked as resolved.

Uh oh!

This comment was marked as resolved.

Uh oh!

patelchaitany commented Feb 16, 2026

Uh oh!

This comment was marked as resolved.

Uh oh!

jyejare left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

This comment was marked as resolved.

Uh oh!

ntkathole commented Mar 2, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

This comment was marked as resolved.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

patelchaitany commented Feb 16, 2026 •

edited by devin-ai-integration bot

Loading