bloom filter for record -> fileid mapping

[HenryCaiHaiying]

In your blog post, https://eng.uber.com/hoodie/, you mentioned using bloom filter to store the recordKey->fileid mapping.

But bloom filter can only answer whether this key is definitely not in the set, does that you mean you would still need to linearly scan the files if the answer is possibly in?

[prazanna]

Yes, once bloom filter test passes, we ensure that the key is actually in the parquet file. Columnar access of a single field is usually very effective in parquet. Bloom filter false positive rate is configured by default as 0.000000001 (very low).

-Prasanna

[vinothchandar]

Note that this fp rate is much much lower than what is used in LSM based key value stores, if you are thinking about that..

Just wondering, do you have a potential use for hoodie?

[prazanna]

Please re-open if you have any follow ups/ Thanks.