Add empirical measurements of the search quality of the indexes

[sam-hey]

Hi @NohTow,

I was wondering if it's possible to include recall information in the evaluation when using an index. I’m not sure if I’ve overlooked it or if it’s not included.

Since recall is a valuable metric for comparing index-based evaluations, it would be great to have it included. Is this something that could be added?

Thanks!
Sam

[NohTow]

Hey @sam-hey,

The evaluate function leverages ranx, so you should be able to get all the metrics you want.
You can see an example in the documentation, where you can select which metrics you want to be returned.

Am I missing something?

If no metrics argument is provided, it default to ndcg@10 and hit@k, you want to include more things by default?

[sam-hey]

Hey, thanks for the response! I should probably rephrase my question a bit. What I mean is the recall for the index—because, as I understand it, with the index approach, many MaxSim operations aren't performed on pairs, as doing so would trigger a re-ranking operation. So my question is: would the results differ if those pairs were actually compared, potentially changing their retrieval ranking?

As for PLAID: "PLAID ColBERTv2 is able to match the MRR@10 and Recall@100 achieved by vanilla ColBERTv2" (https://arxiv.org/pdf/2205.09707).

I am interested in the performance metrics for Voyager HNSW:
https://www.marqo.ai/blog/understanding-recall-in-hnsw-search

[NohTow]

Let me be sure I understand what you mean.
You want a value that illustrates how much the performances "degrade" while using our indexes rather than directly comparing the query and all the documents using MaxSim?

I think we cannot give theoretical values, but only empirical tests that actually compare the performance you get without index and with indexes. My main concern with that is that, to me, it depends on a a lot of factors, namely the different parameters of the indexes as well as the distribution itself, which includes the dataset as well as the embedding model.

Though you are right, maybe an empirical study would be helpful for people to navigate and know how to select the different parameters depending on their setup...
That being said, I am not really happy with the index we propose right now (only HNSW actually), as the original purpose was more to at least provide something for people to play easily rather than actually being proper indexes (which could be achieved through e.g, Vespa or Qdrant).

I was planning of adding at the very least PLAID (hence my PLAID branch), as well as maybe XTR and MUVERA, so maybe it would be a good thing to add when we do so. Main issue is I am lacking time lately, but someone proposed to help a bit back then, so it will eventually come, just not sure about the timeline.

[sam-hey]

Yes, this is exactly what I mean. No problem, I totally understand the time aspect—just knowing that it will come eventually is reassuring.

My experiments showed a small difference, but it wasn’t significant as far as I can tell for now. (I only tested with NFCorpus.)

[NohTow]

Yeah I defined very loose base parameters for HNSW, so it's not as fast as it could be, but the perf should be ok-ish.
I'll let the issue open and just change a bit the name to better match what we need to add as an enhancement.

Thanks for the input!