Possible statistical flaws in biski generators #8
Description
Hello!
I know that your generators were tested by BigCrush and PractRand batteries. However, during developing of my own battery of statistical tests, SmokeRand (https://github.com/alvoskov/SmokeRand), I probably found unknown statistical flaws in biski16, biski32, biski64 generators. In the case of biski16 the "full" battery of SmokeRand the hamming_distr test systematically give "suspicious" p-values around 1e-5 - 1e-6. In the case of biski32 and biski64 all four batteries ("express", "brief", "default" and "full"; their design philosophy is similar to TestU01 but includes some ideas from PractRand, gjrand, and some new tests too) are unable to detect the same flaw but if the sample is large enough, around 1 TiB, the test will be failed.
The details can be found in the biski64 plugin (shared library) source code (https://github.com/alvoskov/SmokeRand/blob/main/generators/biski64.c). The test vectors are taken from your reference implementation but the initialization procedure was changed: SplitMix and warmup were removed. SmokeRand uses ChaCha20 seeded from system CSPRNG instead of SplitMix for seeding. It also includes cfg file for the custom battery required to detect the flaw, and it may take about 1 hour for this run for biski64.
An idea of the hamming_distr test is very simple: it takes non-overlapping blocks of bits and processes them in two different modes:
- Calculates the Hamming weight for each block and compares the obtained distribution with the binomial one.
- XORs neighbour blocks (non-overlapping pairs) and then calculates Hamming weights and compares the distribution with theoretical again. In this case your generators fail. This mode was designed to catch counter-based PRNG with bad avalanche effect such as SplitMix with gamma = 1.
I've tried to solve this problem in "_alt" versions of your generators but it slows them down a bit.
SmokeRand test suite includes a lot of generators. Most of them are high-performance bithacks such as LCG, xoroshiro++, MWC, MT19937, PCG, KISS, SFC64. But some robust general purpose PRNGs such as AES, ChaCha, Speck128/128, LEA, HC256, ISAAC are included too, and you can use it to recheck the test itself (I've probably made it several months ago but lost the logs).
Best regards,
Alexey