Filter documents for searching

[jainaayush05]

One of the biggest drawbacks of libs like annoy or Faiss is that it doesn't allow filtering of documents to be searched before finding nearest neighbours. This is something that can be solved using the changes I have suggested

[alexklibisz]

@jainaayush05 I don't think the cosineDistance part will work. I'm pretty sure that the LSH hashing implementation would have to change in order for cosine distance to make sense. Do you have results to suggest otherwise?

[sai-prasanna]

@jainaayush05 Any thoughts on the cosine distance not working with this implementation of LSH?
@alexklibisz If the filtering of documents part of this pull request work, can that be added to your library?

[alexklibisz]

The implementation of lsh is different for cosine and euclidean similarity. If you are interested, this video series is a fantastic deep dive: https://youtu.be/tlGocuyQ7F0 It's certainly possible to implement but it's not plug and play with the current implementation. Also, the implementation for euclidean in this plugin is actually more complicated than it needs to be. I'm not maintaining any of this code anymore, so I'd rather not merge any PRs. I don't really consider it a library so much as it's a proof of concept. I think there are one or two forks floating around that other people have extended and might be maintaining. Would love to maintain it but there's only so much time in a day, and my current daily work is unrelated.

…

On Wed, Aug 28, 2019, 02:47 Sai ***@***.***> wrote: @jainaayush05 <https://github.com/jainaayush05> Any thoughts on the cosine distance not working with this implementation of LSH? @alexklibisz <https://github.com/alexklibisz> If the filtering of documents part of this pull request work, can that be added to your library? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#10?email_source=notifications&email_token=AB5E27GP5FNZGVSQNAIWZF3QGYNO5A5CNFSM4GJHRQD2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD5KCFVI#issuecomment-525607637>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AB5E27BQLJ3RVOG7CIBQ363QGYNO5ANCNFSM4GJHRQDQ> .