Support Serialization of objects larger than 2^32 bytes

[crockeea]

The SealHeader size field in serialization.h is a uint32_t, which restricts serialized objects to 2^32 bytes. Galois keys can be much larger than that for large parameters (e.g., when using large rings). Please add support for serialization of larger objects.

[WeiDaiWD]

Thanks, Eric. A fix will be part of the upcoming release, along with some other fixes to the issues you met when using very large parameters.

[crockeea]

There's a separate issue when compression is enabled.

[WeiDaiWD]

@crockeea Could you please elaborate on the separate issue you have seen with compression on?

[crockeea]

The problem can be seen around the SEAL_USE_ZLIB block of Serialization::save():

#ifdef SEAL_USE_ZLIB
                case compr_mode_type::deflate:
                {
                    // First save_members to a temporary byte stream; set the size
                    // of the temporary stream to be right from the start to avoid
                    // extra reallocs.
                    SafeByteBuffer safe_buffer(
                        raw_size - static_cast<streamoff>(sizeof(SEALHeader)));
                   ...

This passes in the raw_size (uncompressed size) to the SafeByteBuffer constructor. The constructor requires the size to fit into an int:

            SafeByteBuffer(streamsize size = 1) : size_(size)
            {
                if (!fits_in<int>(add_safe(size_, streamsize(1))))
                {
                    throw invalid_argument("size is too large1");
                }

but for large keys, the uncompressed size exceeds 2^32 bytes.

[WeiDaiWD]

Much appreciated. I have made this fix internally (soon to release).

I'm testing basic functionalities for 262144 with ~7000-bit coefficient modulus. I cannot guarantee that this version leaves no issue with large parameters. It is hard to create a thorough test for such a large parameter set, left alone running it. I'm happy to fix future issues if you witness any.

Such a big parameter selection will still be marked as invalid in SEAL. But you know how to bypass it in "defines.h".

[dnat112]

What is the reasoning behind using "!fits_in" in line above instead of using "!fits_in<uint32_t>? The former restricts sizes even further to fit in 2^31 instead of 2^32.

[dnat112]

Upon further investigation, it looks like simply changing this line to "!fits_in" does not seem to fix the problem for saving Galois keys between 2^31 and 2^32 bytes, as tellp() ends up failing (returns -1) just after save_members writes more than 2^31 bytes. I am not sure how to get around this issue and I think there is a problem with writing more than 2GB at once on many machines.

I'll add that currently I am not able to use the deflate compression mode with Galois keys between 2^31 and 2^32 bytes at all and can only write the keys to a file using the galois_keys_save() key generator function with no compression. Generating the galois keys and then writing them with the "save()" functionality does not work, even with no compression mode enabled. It would be helpful to enable compression even for this just this slightly larger size.

[WeiDaiWD]

@crockeea @dnat112 Serializing objects larger than 4 GB work now with or without ZLIB. You will see that in the coming release. Thanks for your helpful issue reports.