GSoC 2026: OpenVINO Deep Search AI Assistant on Multimodal Personal Database for AI PC

[Rahuldrabit]

Hi @zhaohb and @18582088138,

My name is Rahul Drabit Chowdhury. I am currently pursuing a Master of Science (Thesis) in Computer Science and Engineering. I have been working in AI for the past two years, with hands-on experience in multimodal RAG systems and running optimized local inference using OpenVINO.

In my workflow, I typically:

• Download models in ONNX format
• Optimize and run them locally using OpenVINO
• Build RAG pipelines on top of them

Recently, I have:

• Built a simple RAG backend system:
  -- https://github.com/Rahuldrabit/AI-Chatbot-Backend-with-RAG-Pipeline

• Developed a multi-agent deep research system using LangGraph as part of coursework

I have also contribution experience .
I am very interested in contributing to this project and would love to discuss how I can align my experience with your roadmap and current needs.

Looking forward to your guidance.

Best regards,
Rahul Drabit Chowdhury

[zhaohb]

Hi @Rahuldrabit ,
Excellent work. I have a couple of questions: how is the metadata for images and text stored in this demo? Additionally, at scale, what are the best ways to improve retrieval accuracy, and are there any new directions we should consider?

Thank you very much.

[Rahuldrabit]

Hi @zhaohb,

Thank you for the feedback and the questions.

Currently, in the existing demo I implemented text-based RAG only, but the architecture can be extended to multimodal retrieval (images, audio, video).

1. Metadata Storage (Current Demo)

In the current implementation, documents are recursively split into token-aware chunks, and each chunk is stored with structured metadata.

Example schema for text chunks:

{
  "chunk_id": "doc123::chunk_4",
  "doc_id": "doc123",
  "title": "How billing works",
  "source": "knowledge_base/documents.json",
  "url": "https://example.com/billing",
  "category": "billing",
  "lang": "en",
  "chunk_text": "The text here...",
  "chunk_pos": 4,
  "token_count": 210,
  "embedding_id": 4567,
  "created_at": "2026-03-08T12:00:00Z",
  "updated_at": null,
  "quality_score": 0.92,
  "tags": ["pricing","faq"],
  "checksum": "sha256:...",
  "canonical": true
}

Important fields for scalability:

• category → enables metadata filtering before vector search
• tags → improves semantic grouping
• quality_score → useful for ranking and filtering
• embedding_id → links metadata with vector database entries

For scalability, metadata can be stored in a dedicated PostgreSQL database, while embeddings are stored in a vector database.

Example filtering before vector search:

category = "finance"

This reduces the search space and improves retrieval efficiency.

2. Image Metadata (Multimodal Extension)

Although the demo currently focuses on text, for multimodal systems I typically store image metadata like this:

{
  "image_id": "img_20260308_001",
  "file_path": "/uploads/img_20260308_001.jpg",
  "caption": "Dashboard showing billing summary",
  "alt_text": "billing dashboard",
  "uploader": "admin",
  "created_at": "2026-03-08T12:05:00Z",
  "mime": "image/jpeg",
  "width": 1200,
  "height": 800,
  "ocr_text": "Total due: $123.45 ...",
  "object_labels": ["chart","table"],
  "tags": ["billing","dashboard"],
  "embedding_id_clip": 83421
}

This supports dual-index multimodal retrieval:

Text-based retrieval

• captions
• OCR text
• tags

Image similarity retrieval

• CLIP embeddings

Both indexes can be queried in parallel and merged during reranking.

3. Retrieval Architecture

The system follows a hybrid RAG architecture:

User
 │
 ▼
API Gateway
 │
 ▼
Query Processor
 │
 ▼
Hybrid Retriever
 │
 ├─ Vector DB cluster
 ├─ Metadata DB
 └─ Cache
 │
 ▼
Reranker
 │
 ▼
Context Builder
 │
 ▼
LLM Service
 │
 ▼
Response

The retriever combines multiple retrieval signals:

• Semantic search (vector similarity)
• Keyword search (BM25)
• Exact match
• Metadata filtering

4.How I Improving Retrieval Accuracy at Scale

Hybrid Retrieval

Combine:

• vector similarity
• keyword search
• metadata filters

Query Generation and Expansion

Use techniques like:

• Query rewriting
• Hypothetical Document Embeddings (HyDE)

HyDE generates a synthetic document representing the expected answer to improve recall.

Agentic RAG for Complex Queries

For complex queries, agent-based retrieval strategies can help:

• Query planning
• Self-RAG
• Corrective RAG
• multi-step retrieval

Knowledge Graph RAG

For structured datasets (e.g., banking or relational data), KG-RAG can improve reasoning by retrieving entities and relationships.

5. Re-ranking Layer

A multi-stage retrieval pipeline improves precision.

Typical workflow:

retrieve top 50 candidates
↓
cross-encoder reranking
↓
send top 5 chunks to the LLM

Cross-encoder models such as bge-reranker-large can significantly improve ranking quality.

---

6. Scaling Data Ingestion

Parallel Ingestion

worker1 → docs 1–1000
worker2 → docs 1001–2000
worker3 → docs 2001–3000

---

Streaming Ingestion

Kafka
 → ingestion workers
 → embedding pipeline
 → vector database

---

Incremental Indexing

Instead of rebuilding the entire index:

• only new or updated documents are embedded
• indexes update asynchronously

7. Context Management for LLM Limits

Because LLMs have token limits, several techniques are useful:

Top-K selection

retrieve 50
rerank
send top 5

Context compression

• summarize retrieved chunks before passing to the LLM

Chunk reranking

• cross-encoder models

8. Caching Layer

Caching improves latency and throughput.

Types of caching:

Query cache

query → final answer

Retrieval cache

query → retrieved documents

Embedding cache

text → embedding

9. Continuous Improvement

To maintain system quality over time:

• A/B testing for retrieval strategies
• async index updates
• continuous learning pipelines
• feedback-driven improvement

I would be happy to work in this project .

Thank you again for the discussion.

Best regards,
Rahul Drabit Chowdhury

[Rahuldrabit]

Hi @zhaohb and @18582088138,

I would like to mention that I contributed to Google Summer of Code 2025, where I worked on the MBDyn project. Where I implement Genetic Algorithm Module for multi-body simulations.
Project link: https://summerofcode.withgoogle.com/archive/2025/projects/iemnSvFQ

In addition, I have contributed fixes and improvements to several open-source projects:

• frontend/pytorch: add aten::cross_entropy_loss translator and tests (fixes #29691, relates #28584) #32554
• OpenVINO backend: add bitwise_left_shift and bitwise_right_shift keras-team/keras#22146 (successfully merged )
• Fix Prometheus Helm.status crash due to invalid chart_path argument microsoft/AIOpsLab#126 (comment) (successfully merged )
• MNT: Replace live server calls with mocks in tests/test_utils/test_utils.py openml/openml-python#1679
• Frontend(Pytorch): add support for quantized::relu6 #34343

I enjoy contributing to open-source projects and collaborating with the community.

Best regards,
Rahul Drabit Chowdhury

[18582088138]

@Rahuldrabit
Thank you very much for your positive feedback recently; your enthusiasm for the project has left a strong impression on us. During our recent interactions, it became clear to me that you possess a very deep understanding of the project. Over the past few weeks, we have been engaged in internal discussions regarding the selection of suitable candidates; if possible, I would appreciate the opportunity to see some of your personal reflections on this specific project.

In this current era dominated by AI, many project tasks—as well as the conceptualization of solutions to problems—can be rapidly facilitated through AI tools. This makes it increasingly difficult to accurately assess a developer's true capabilities. My hope is that the GSoC program will serve to genuinely enhance your personal technical skills and your business-oriented thinking, rather than merely becoming a means to pad your resume.

Regarding this "AI PC" project, I would like you to approach it from the perspective of a real-world product. Consider how to design and optimize applications that are best suited for the Intel hardware platform—specifically, how to deploy these applications effectively within resource-constrained environments. This presents a significant contrast to deploying agents on servers, as you must contend with challenges such as limited memory resources, insufficient local computing power, and the computational limitations of running local LLMs.

Furthermore, a purely local AI PC experience can sometimes be less than user-friendly. If you are able to adopt a "Cloud + Client" hybrid approach for key components of the application, you could significantly enhance the overall user experience. However, doing so would require you to carefully address issues regarding user data security as well as the operational costs associated with calling Cloud LLM APIs. (Please note: the "Cloud + Client" development approach is an optional requirement, not a mandatory one.)

I look forward to reading your thoughts on this project and await your reply. Thank you very much.

[Rahuldrabit]

Thank you @18582088138 sincerely for your kind words and for sharing your expectations so openly. I want to share my honest thinking about this project — including both my ideas and the challenges I see.

1. SEMANTIC SEARCH AS THE CORE FOUNDATION

For a truly useful personal search assistant, keyword search is fundamentally insufficient. A user asking "why did he seem nervous?" should retrieve a document that says "he was visibly anxious" — something BM25 entirely misses. I plan to build the retrieval layer entirely around dense semantic embeddings, using a lightweight OpenVINO-optimized sentence transformer (e.g., bge-small-en, ~33M params) for text, and CLIP-based embeddings for images. This keeps the embedding pipeline fast enough to run on the NPU in the background without disrupting user workflows.

2. MULTIMODAL INDEXING — VIDEO AS THE HARD CASE

Video is the most challenging modality. Running a large video-language model end-to-end is not feasible on an AI PC. My proposed approach decomposes the problem:

• Keyframe extraction using scene-change detection (PySceneDetect), not uniform sampling, to avoid wasting compute on static frames.
• A small VLM (2–4B params, e.g., MiniCPM-V via OpenVINO) generates natural language captions per keyframe.
• Whisper (OpenVINO-quantized) transcribes the audio track.
• A local small LLM synthesizes the captions + transcript into a structured video summary stored alongside the embeddings.

This means complex video queries like "find the clip where the package was dropped" work through semantic caption retrieval, not raw frame comparison.

3. CLOUD + CLIENT HYBRID — MY CORE ARCHITECTURAL PROPOSAL

You raised the Cloud+Client approach as an optional enhancement for UX. I believe it is architecturally central, but only if done correctly with respect to privacy.

My proposed flow:

[User query]
→ Local embedding model converts query to vector
→ Local vector store retrieves top-k relevant chunks (text/caption summaries only — never raw files)
→ Privacy gate: local NER model (spaCy) strips PII from retrieved chunks
→ [Query + sanitized chunk summaries] sent to Cloud LLM
→ Cloud LLM synthesizes answer

The raw file never leaves the device. The cloud only ever sees semantic summaries.

Further, I want to propose a Hierarchical Agent Architecture that I think is novel for this context:

Cloud LLM acts as the PLANNER: it decomposes complex queries into sub-queries and sends them back to the local system.
Local LLM acts as the EXECUTOR/SUB-AGENT: it searches the local knowledge base using these sub-queries and returns results.
The Cloud LLM then synthesizes the final answer from the local results.

For example:
User: "Why did he run?"
Local retrieves: ["drops bag at 5s", "runs away at 10s"]
Cloud receives: query + these two chunks → answers without ever touching the raw video.

This also means that if local retrieval is poor, the cloud LLM can issue a refined sub-query rather than failing silently — solving what I see as a key failure mode of naive RAG systems.

For cost control, I plan to implement a local confidence scorer: only escalate to the cloud LLM when the local model's self-assessed confidence is below a threshold. This prevents unnecessary API calls for simple queries the local LLM can handle alone.

import openvino_genai as ov_genai
import spacy, anthropic, json

# Local NPU LLM (sub-agent / executor)
local_llm = ov_genai.LLMPipeline(
    "Qwen2.5-3B-int4-ov", "NPU",
    {
        "GENERATE_HINT":   "BEST_PERF",
        "NPUW_CACHE_DIR":  ".npucache",
        "MAX_PROMPT_LEN":  1024,         # static shape — keep ≤1024 on 16GB RAM
        "MIN_RESPONSE_LEN": 256,
    }
)

 # privacy gate — strip PII before any cloud cal
nlp = spacy.load("en_core_web_sm")   # lightweight local NER (12MB)

def strip_pii(text: str) -> str:
    doc = nlp(text)
    for ent in reversed(doc.ents):
        if ent.label_ in {"PERSON", "ORG", "GPE", "EMAIL", "PHONE"}:
            text = text[:ent.start_char] + f"[{ent.label_}]" + text[ent.end_char:]
    return text

#  decomposes query into sub-queries
cloud = anthropic.Anthropic() 

def cloud_plan(user_query: str) -> list[str]:
    resp = cloud.messages.create(
        model="claude-sonnet-4-6",
        max_tokens=200,
        messages=[{"role": "user", "content":
            f"Decompose into 1-3 specific sub-queries for local document search.\n"
            f"Return ONLY a JSON array.\nQuery: {user_query}"}]
    )
    return json.loads(resp.content[0].text)

 # runs each sub-query against the knowledge base
def local_execute(sub_query: str) -> dict:
    chunks = semantic_search(sub_query, k=4)          # vector store lookup
    safe   = [strip_pii(c[0]) for c in chunks]        # PII-stripped summaries
    prompt = f"Context:\n{chr(10).join(safe)}\n\nAnswer: {sub_query}"
    answer = local_llm.generate(prompt, max_new_tokens=200, do_sample=False)
    return {"sub_query": sub_query, "chunks": safe, "local_answer": answer}

 # Cloud synthesiser
def cloud_synthesise(user_query: str, results: list[dict]) -> str:
    context = "\n\n".join(
        f"Sub-query: {r['sub_query']}\nResult: {r['local_answer']}"
        for r in results
    )
    resp = cloud.messages.create(
        model="claude-sonnet-4-6",
        max_tokens=500,
        messages=[{"role": "user", "content":
            f"Using these local search results (PII removed):\n{context}\n\n"
            f"Answer: {user_query}"}]
    )
    return resp.content[0].text

CONFIDENCE_THRESHOLD = 0.65   # tune per use case

#  full outline
def deep_search(user_query: str) -> str:
    local = answer_locally(user_query)
    if local["confidence"] >= CONFIDENCE_THRESHOLD:
        return local["answer"]

      sub_queries = cloud_plan(user_query)

    results = [local_execute(sq) for sq in sub_queries]

    return cloud_synthesise(user_query, results)

4. LOCAL KNOWLEDGE GRAPH FOR MULTI-HOP REASONING

Vector similarity retrieval has a known weakness: it cannot answer relational, multi-hop queries like "all notes related to Project X that also mention Alice." I propose maintaining a lightweight local knowledge graph (SQLite-backed or neo4j, scoped to ~50K nodes for MVP) that links cross-modal entities — people, projects, dates, locations — extracted via the local LLM at indexing time. This graph augments vector retrieval for structured reasoning without requiring a heavyweight graph database.

5. COLD START & GRACEFUL DEGRADATION

A system that only works after hours of indexing will never be adopted. I plan to address this with:

• A background file watcher (watchdog) that triggers incremental indexing on idle, prioritizing recently modified files first.
• Graceful degradation: while embeddings are being built, the system falls back to BM25 keyword search over file metadata and extracted text, so the user always gets some result.
• A file hash → version map to detect modified files and re-index only changed content.

6. HARDWARE-AWARE SCHEDULING (OpenVINO-specific)

I see OpenVINO's device plugin abstraction as the key enabler here:
• NPU: background embedding generation (low power, sustained throughput)
• iGPU: interactive LLM inference during active queries (lower latency)
• CPU: orchestration, OCR, pre/post-processing

This is not just a performance optimization — it is what makes the application feel native to the Intel AI PC platform.

Thank you again for this opportunity, and I look forward to your thoughts.

Best regards,
Rahul

[18582088138]

@Rahuldrabit
The application period for GSoC contributors has officially begun. Please pay close attention to the application submission deadline.
If you encounter any uncertainties or wish to discuss specific details during the application process, please feel free to reach out to us via email.
- hongbo.zhao@intel.com
- kunda.xu@intel.com

I would like to add a specific reminder: the application system performs AI-generation checks on submitted content. While you may use AI tools as an aid in drafting your application, we place a much greater emphasis on your personal understanding of—and insights into—the project.

I am optimistic about your proposal; I encourage you to prepare your application with diligence. Please note that the final list of selected participants is subject to the official announcement by GSoC.

A quick reminder about the dates:
- March 16 – March 31 - GSoC contributor application period
- April 21 - 18:00 UTC - GSoC contributor slot requests due from Org Admins
- April 30 - 18:00 UTC - Accepted GSoC contributor projects announced
- May 1 - May 24 - Community Bonding Period
- May 25 - Coding officially begins

[Rahuldrabit]

I acknowledge , Thank You .

[18582088138]

@Rahuldrabit
I have already reviewed the proposal you submitted for GSoC.

Additionally, I would like to remind you that the selection of the model/pipeline is a critical aspect of this project. In application scenarios where hardware resources are constrained, you must consider not only the model's functionality and capabilities but also its hardware resource footprint and response speed.
While AI-assisted programming can certainly facilitate the initial setup of a Proof of Concept (POC), a great deal of work remains to be done to evolve it into a full-fledged product—for instance:

• How to select the appropriate model? The latest LLMs often require specific enablement steps and cannot be directly enabled by openvino.
• How to strike a balance between model performance and accuracy? While optimization techniques—such as model compression—may loss the accuracy, they enable the deployment of models with significantly larger parameter model.
• How to accelerate the application startup and model loading times?
• When a task requires the simultaneous orchestration of multiple modules/models, how can the processing pipeline be optimized effectively to enhance the user experience?
• And so on…
  While AI tools can typically provide a generalized design framework for such issues, addressing them effectively in real-world scenarios ultimately requires your own personal experience and a deep understanding of the specific problem at hand.
  Furthermore, for this AI-PC project, the objective is not simply to maximize the number of features or the complexity of the application; rather, you must make strategic trade-offs regarding functionality and model selection to satisfy user requirements while utilizing the fewest possible resources.
  The points I have raised here are not intended as formal task requirements for your project; rather, my hope is that by reflecting on these questions, you will gain a deeper and more profound understanding of what is involved in developing a real-world project.

[Rahuldrabit]

Hi @18582088138,

Thank you for reviewing my proposal and for the thoughtful feedback.

From my experience, I have seen that quantized models, especially INT4, can produce hallucinations or unstable behavior in some outlier cases. That is why I have focused more on retrieval techniques in my proposal. For faster queries, I use a lightweight retrieval method, and for better accuracy, I use a hybrid retrieval method. This is already included in my design.
as example
``
Task: "Find all documents about X and summarize"
→ Acceptable with INT4 (retrieval does the heavy lifting)
→ Model just needs to synthesize, not reason deeply

Task: "Compare the budget figures across my 3 financial reports"
→ Risky with INT4 — numerical reasoning degrades faster than text quality
→ May need INT8 or a larger model

Task: "Explain the relationship between concept X in doc A and concept Y in doc B"
→ Very risky with very small models even at FP16
→ This is where model SIZE matters more than precision
``

For the UI, I would like to ask whether it would be acceptable to let users choose between a quantized model and a normal model, depending on their preference and hardware capability.

I have also included the technical analysis in my proposal. I wanted to ask whether there is any strict page limit for the proposal. Because I added the technical research and code-level suggestions for each part of the project, the full document became quite long. The architecture and main proposal are within 17 pages, while the technical analysis and report extend to 56 pages.

This is the part that concerned me a little, because I wanted to make sure the proposal remains clear while also showing the technical depth behind my design choices. If there is a preferred length, I will try to shorten and refine it further.

Also, I will add your technical questions and detailed analysis be included directly in the proposal.

Best regards,
Rahul Drabit Chowdhury

[Rahuldrabit]

Hi @18582088138 ,
I have a question regarding model selection.

Since many of the latest LLMs require specific enablement steps and cannot be directly supported by OpenVINO, what criteria would you recommend for selecting appropriate models for this project?

Also, I noticed the proposed project “Automated Agent System for Adding New Model Support to Optimum-Intel.” Would it be possible or encouraged to collaborate or align with that project, especially for handling model onboarding and compatibility?

Thanks!

[18582088138]

Hi @Rahuldrabit
Our primary focus is on your understanding of the project and your individual development capabilities. While the use of AI tools for assistance is permitted, please avoid relying on them excessively.
Do not worry on the format of the proposal; it serves primarily as a reference.
Of course, if you wish to elaborate on it further, that would be even better.

[Rahuldrabit]

Thank you.

I have used AI mainly for research and learning concepts. I have collected all code portions from official documentation and properly cited all the sources from which I obtained them.

I have around 3 years development experience, and I am quite concerned about AI model hallucination. Therefore, most of the time I rely on documentation for coding. I also apply my own logic and reasoning, because AI models cannot always correctly implement my ideas and sometimes produce highly inaccurate outputs.

As a beginner developer, I avoid relying on AI for coding, as I believe it may gradually kill my skills.

[18582088138]

The follow link is the OpenVINO release LLM:

• https://huggingface.co/OpenVINO
  you can reference the provided LLMs

[Rahuldrabit]

Thank you.
Another question: Intel AI PCs include NPUs, but many Intel-based PCs only have shared GPUs. Do I need to consider those cases as well?
I ask this because , I don't AI PC's laptop .

[Rahuldrabit]

Hi @zhaohb and @18582088138,

Thank you for your feedback and suggestions. I have submitted my proposal; can you review it and give suggestions? I will be very grateful.

In the project, I have added a post-GSoC plan due to the time constraints of GSoC to implement my proposed full architecture. I can change that, I mean (swap cloud with full deep search to post-GSoC plan), if you suggest.

I am waiting to hear from you.
Best regards,
Rahul Drabit Chowdhury