Document the protocol used to serialise/deserialise (serde)

[mccorby]

Is your feature request related to a problem? Please describe.
If we consider the serde process as the protocol used by PySyft to communicate with other systems, it would be very useful to keep a documentation of what this protocol looks like for the different operations

Describe the solution you'd like
Examples of different calls and operations with workers to start with.

Describe alternatives you've considered
Print the result of the serialisation process before compressing.

[hereismari]

Probably some tutorials for developers could be useful: "how serialization and deserializations works in syft"

[Ankit-Dhankhar]

Hi @mari-linhares I would like to work on this issue. Are there are pointer where I can start from like some paper or blogs?

[Ankit-Dhankhar]

I was going through implementation in serde. I found implementation for torch and numpy_tensor_serializer/deserialize using TemporaryFile() which could be done with io.BytesIO too.
e.g.
For serialization

def numpy_tensor_serializer(tensor: torch.Tensor) -> bin:
    """Strategy to serialize a tensor using numpy npy format.
    If tensor requires to calculate gradients, it will detached.
    """
    if tensor.requires_grad:
        warnings.warn(
            "Torch to Numpy serializer can only be used with tensors that do not require grad. "
            "Detaching tensor to continue"
        )
        tensor = torch.detach()

    np_tensor = tensor.numpy()
    outfile = io.BytesIO()
    numpy.save(outfile, np_tensor)
    return outfile.getvalue()

For deserialization

def torch_tensor_deserializer(tensor_bin) -> torch.Tensor:
    """Strategy to deserialize a binary input in npy format into Torch tensor"""
    bin_tensor_stream = io.BytesIO(tensor_bin)
    return torch.from_numpy(numpy.load(bin_tensor_stream))

Are there some differnce in use-case of efficiency issue for using two different I/O streams, which I am missing.

[mccorby]

@Ankit-Dhankhar Please feel free to open a PR and change it. It does look like an improvement.

[Ankit-Dhankhar]

I was going through serde.py and I found

PySyft/syft/serde/serde.py

Line 372 in e4044e6

method (such as JSON or msgpack). The reason we have this function

but only msgpack is used for serialization. What is a benefit as compared to JSON or protobuf which are more widely used libraries(having a larger community support)? I also had a conversation with @kamathhrishi where he mentioned there was a plan to use these.

[karlhigley]

@mccorby Are the examples in the test helpers sufficient to document the msgpack serialization format? Wondering if we can now close this issue.

[mccorby]

Lots of work around serde makes this issue not relevant anymore.