Bugs in dataset download script

[xiaowu0162]

Thank you for the great work! I met two issues when running scripts/download_compactds.sh:

• The output_dir variable is not used
• The shard combination pattern also included the file index_IVFPQ.100000000.768.65536.64.faiss.meta, resulting in an error when the index is used later.

Here is the working script after some minor changes:

#!/bin/bash

# Check if an argument is passed
if [ $# -eq 0 ]; then
  echo "Usage: $0 <argument>"
  exit 1
fi

output_dir=$1

# Download the sharded index files
python scripts/download_index.py --output_path $output_dir

# Combine the shards
cat $output_dir/embeddings/index_IVFPQ/index_IVFPQ.100000000.768.65536.64.faiss_* > index_IVFPQ.100000000.768.65536.64.faiss

# Remove shard files
rm $output_dir/embeddings/index_IVFPQ/index_IVFPQ.100000000.768.65536.64.faiss_*

[shallinan1]

+1 - thanks for the updated script, worked for me

[onestardao]

just stumbled on this ~ your patch actually fixes one of the sneakiest faiss pitfalls i've seen around. solid catch.

for others wondering why this matters:
the *.faiss.meta file is metadata, not part of the actual shards. if you cat it along with the index shards, you'll silently corrupt the final index. no warning, just runtime chaos later. brutal.

this exact trap is now officially part of a RAG + indexing failure map i’ve been working on (open source, mit licensed, backed by the tesseract.js guy — yeah, that one). happy to share the link if you’re interested. trying to document these silent gotchas before they eat someone’s weekend.

either way, thanks for fixing this upstream. you just saved someone hours of debugging.