Answer Engine Quality - Ideas

[wsxiaoys]

Under Construction

The Answer Engine, released in version 0.13, provides a Q&A interface for Tabby's users to interact with the LLM, optionally within the context of a connected repository. The implementation is quite naive thus not really an performing one, we'd like to record ideas and improve the quality upon current baseline.

Prompt construction

Currently, the implementation is relatively simple. We collect snippets(instead of full article) from various sources and input them into a single LLM inference call to generate both the answer and the grounding (citation) information simultaneously.

1. collect references: Query the index, which includes both code and documents, to gather chunks. Deduplication is carried out at the document level, making sure that from each document, only the chunk with the highest score is chosen.

2. answer generation: Construct the prompt by combining the question and the content of each chunk to create a single LLM inference prompt.

3. relevant questions: Generate relevant questions for the request, the prompt is also based on question and chunk

Ideas

• [Answer Engine]For code references, when the file content is reasonble(e.g <200 LoC), include whole file content directly, instead of only including the chunk. #3096
• include commit context provider to inject commit information (e.g commit message) to prompt #3122
• include directory tree context provider to inject directory information to prompt, so it can answer questions like: which file is implementing the embedding model inferences? #3123

Integrations

• Integration with Notion Database #2894
• [Answer Engine] switching between models (or select different models to answer the question) #3097
• An cli inspector for tabby's tantivy index #2984

[shinohara-rin]

I always though it's a good idea to not rely solely on embedding models for the retrival of relevant code.

In reality, human developers relies a lot on language servers, it would make sense to let LLMs make use of the language servers too (autocompletion, retriving def/refs, type inference, arguments hinting, docs, etc.)

For instance, extracting all symbols inside the user's selection and feeding the definitions and/or references to the LLM. This is of course a rather naive approach, but it should facilitate code understanding of the LLM.

[sananand007]

how exactly currently folks are testing out indexed data through a local repository that is indexed ? Are there any legit ways to do that ? Any apiCalls etc .. ?

[wsxiaoys]

Hi! Please check https://demo.tabbyml.com/search/tabby-ml-tabby-how-to-add-new-implementation-of-h-zGLyqw for an example on how code context is affecting the answer engine's output

[imotai]

Have you considered using a reranker model? @wsxiaoys