crc checksum for header (key + value sizes)

[funny-falcon]

If data will be corrupted, then key+value size could be decoded into large value: unneccessary allocations and unneccessary disk read will follow.

It is better to add checsum for key+value size header to early detect such corruption.

And then crc32 checksum for data path could be resided in a header as well, therefore there will be no need to allocate buffer for both header and data.

I suppose, header could have following structure:

keySize = 2 bytes
typeAndValueSize = 4 bytes
dataCRC = 4 bytes
headerCRC = 4 bytes
Therefore headerCRC will check dataCRC as well.

[akrylysov]

It is better to add checsum for key+value size header to early detect such corruption.

Read/write performance and space amplification will take a hit - adding another checksum will increase the single key-value pair overhead.

If data will be corrupted, then key+value size could be decoded into large value: unneccessary allocations and unneccessary disk read will follow.

I'm not too worried about unnecessary reads and allocations - it may happen only on data corruption. The worst case scenario is Pogred allocates 4GB of memory and then discards the corrupted record - it won't prevent recovery.

I'm leaving the issue open, but I have no plans fixing it in the near future.

[akrylysov]

Checksumming the header (key size + value size) can be done without adding another checksum. The sizes could be checksummed together with the data.