Add KMP_DUPLICATE_OK workaround for Faiss OMP: Error #15 #1009
Description
The default faiss integration continues to cause issues primarily on Windows and macOS. An non-exhaustive list is:
- [Python 3.9, Mac OS] Code hangs while building embedding index #62
- GitHub Actions build error with torch 1.12 on macOS #300
- Segmentation fault on arm64 macOS while searching embeddings index #350
- 38210 segmentation fault #463
- Add topk functionality to qa pipeline #480
- Random Segmentation fault with embedding.search after upserting #498
- embeddings.search hangs indefinitely on Mac #576
- Cannot get it to work on M2 Mac #685
- Segmentation fault #689
- Segmetation fault on macOS Sonoma 14.4.1 (MacBook Pro Apple M3 chip) #717
- Segmentation fault when calling embeddings.index #744
- Embeddings.search leads to segfault error #813
- Add text extraction with Docling #814
- Add programmatic workaround for Faiss + macOS #818
- OMP: Error #15 on Windows 11 #830
This all stems from faiss-wheels/faiss-wheels#73
This change will add the KMP_DUPLICATE_OK workaround for Windows and macOS. It will also limit the number of threads to 1 on Windows in addition to macOS.
Linux won't include either of these workarounds.
Removing the faiss library from the default txtai install and moving the default ann to numpy is another option if these issues continue to persist. This underlying issue is destructive to the adoption of txtai.