add new experimental xxh3 hash algorithm

[flier]

Add new experimental xxh3 hash algorithm which was optimized for small key and also support AVX2/SSE2 for large key.

• hash with seed (64bits) or secret (&[u8])
• hash to 64 or 128 bits output
• support SIMD with AVX2/SSE2
• sanity check from upstream
• implements core::hash::Hasher
• implements core::hash::BuildHasher
• implements digest::Digest
• implements serde::{Deserialize, Serialize}
• benchmark with criterion

What am I missing?

Thanks

[shepmaster]

Thanks!

Heh, I've had the XXH3 tab open in my browser ever since I read about it on Hacker News, but I've been too busy to act on it. 😉

Can you tell me a little about the process you followed to port the code over? When I did the original version, I basically copied over the C code into a Rust file and iterated until the code compiled (then iterated until the tests passed!). Based on some of the names, it looks like you might have followed the same process.

I'm also interested in knowing what your usecase(s) are for XXH3, if you are able to share about them.

The code is quite dense, so it will probably take me a while to review it. Please feel free to ping me every week or so if I haven't otherwise shown any signs of life.

/cc @quark-zju, in case you are interested.

[quark-zju]

Thanks for tagging me. We're interested. However, it is not quite ready for on-disk format (our main usecase) yet, according to README:

It can be used for ephemeral data, and for tests, but avoid storing long-term hash values yet.

We definitely want to switch once it gets stabilized 😃

[flier]

Sure, I use similar process to port the code, just rename some C style name or extract some Trait as Rust style. So, you can review the code base on C code side by side.

I'm working in a network security company, so we have a lot of scenes which need high performance (high throughput and low latency) hash algorithm, that's why I'm trying to evaluate and port T1HA and XXH3, I also maintain some experimental fasthash with C binding.

Thanks

[NotSqrt]

I just compared twox_hash::Xxh3Hash64 from https://github.com/flier/twox-hash/tree/xxh3 to twox_hash::XxHash64 :
twox_hash::XxHash64 is 3 times faster on small keys (my use case is HashMap with small strings as keys : 7 to 10 chars).

[That3Percent]

You asked what the use-cases are for using XXH3. My primary need for using this over XxHash64 would be that it has a 128-bit output.

[shepmaster]

I think it's time to go ahead and merge this. I'll be honest and say that I haven't fully grokked the new algorithm (to the level I'd like), but my skimming seemed fine.

[shepmaster]

@flier @quark-zju please let me know if either of you would be interested in becoming a co-maintainer!

[flier]

@shepmaster Sure, I could handle issues about xxh3, thanks.

[mjhostet]

FWIW the "stable" xxh3 algorithm was just released, suitable for on-disk use etc. https://github.com/Cyan4973/xxHash/releases/tag/v0.8.0

It's worth verifying that this implementation matches the final version.