Send & Get datasets

[LaRiffle]

We can currently send and get tensors, we would like to have the ability to directly fetch datasets with all the data inputs and targets in an easy way.

Things to do:

• Change sy.BaseDataset to inherit from AbstractObject to have tagging and description and to be registrable by workers: this allows datasets to be referenced for the search functionality
• Extend BaseDataset with a simplify and detail method (see other objects with a simplify and detail method like the FixedPrecisionTensor to understand the behaviour): this allows to send and receive datasets. (+ reference BaseDataset in serde.py just like FixedPrecisionTensor). You need to update syft-proto as well
• Support pointing to BaseDataset: create a new type of Pointer for BaseDataset which inherits from ObjectPointer (get inspiration for example from PointerProtocol)
• Add the .send() and .get() functionality to BaseDataset (for .send() you can get inspiration from Protocol and for .get() native.py)
• Allow search of datasets in .search to fetch remote datasets: make sure the BaseWorker.search method correctly retrieve datasets

Each of this point can be addressed in a separate PR if needed.

[NovaVic]

Hi Theo,
I can take this one.
I see similar coding structure (with tag, description, simplify, detail methods etc.) in protocol.py file - so can to some extent imagine what the "things to do" are referring to but I think I still need more context as a new person on this.
Thanks in advance Theo!

[LaRiffle]

Hi @NovaVic I just updated the description with more details, does it looks good to you? :)

[NovaVic]

Hi Theo @LaRiffle, Thanks for the updated description - that definitely helps! And yes I feel it would be better manageable if I can address change related to each of the points as separate PR.

You can assign it to me. Thanks Theo!

[LaRiffle]

Hi @NovaVic, any updates on how the project is going? :)

[Guillem96]

@LaRiffle any updates on this issue? If you want I can give it a try

[LaRiffle]

Please go ahead :)

[Guillem96]

@LaRiffle When I try to send a Dataset I get this error:

syft.exceptions.UndefinedProtocolTypeError: syft.frameworks.torch.fl.dataset.BaseDataset is not defined in the protocol file

Should I modify syst proto package too?

[LaRiffle]

@LaRiffle When I try to send a Dataset I get this error:

syft.exceptions.UndefinedProtocolTypeError: syft.frameworks.torch.fl.dataset.BaseDataset is not defined in the protocol file

Should I modify syst proto package too?

Yes you should!