(WIP) Add Protobuf serialization#2826
Conversation
|
Tons of work left to do on this, but starting a PR so y'all can see what it looks like and provide feedback along the way. |
karlhigley
force-pushed
the
feature/protobuf
branch
2 times, most recently
from
a58129e to
8031138
Compare
|
There's a lot of machinery to set up to make serialization and deserialization work! This PR probably, very nearly, almost makes a single round-trip Python -> Protobuf -> Python test pass (for (I also haven't figured out a good way to select between msgpack and Protobuf yet. I think it can be done, but I'm still trying to get the basic functionality working right now.) |
karlhigley
force-pushed
the
feature/protobuf
branch
from
8031138 to
67cfda6
Compare
I'd like to make this configurable, but haven't quite figured out how that's going to work yet. This is a convenience commit to be undone later.
karlhigley
force-pushed
the
feature/protobuf
branch
from
67cfda6 to
b0e11eb
Compare
|
CI tests are failing because it can't find the new modules in |
Updating requirements.txt to use direct git url like |
| @@ -0,0 +1,355 @@ | |||
| from collections import OrderedDict | |||
There was a problem hiding this comment.
_generate_bufferizers_and_unbufferizers, _bufferize, _unbufferize, seem very similar to msgpack's serde.
Although, it seems there's no point to extract that to common code at this time.
There was a problem hiding this comment.
They're intentionally very, very similar. IIRC, there are a few tweaks to how the data structures are created and what they contain, but there's almost certainly shared code that could be factored out.
|
Closing this in favor of a series of smaller PRs with round-trip serialization tests. Thanks for the review, @vvmnnnkv! |
2 participants
This PR will add
bufferizeandunbufferizeto the classes that currently definesimplifyanddetailin order to provide de/serialization for Protobuf (instead of msgpack.)Depends on OpenMined/syft-proto#12.