Asynchronous FL on MNIST: call WebsocketClientWorker evaluate() method causes error

[9sashafr]

Hello!
I am trying to run the example from tutorials. Everything works great until the models' evaluation. The script crashes with the error:

Client message

RuntimeError Traceback (most recent call last)
in async-def-wrapper()
37 )
38
---> 39 # Federate models (note that this will also change the model in models[0]
40 for worker_id, worker_model, worker_loss in results:
41 if worker_model is not None:

in evaluate_model_on_worker(model_identifier, worker, dataset_key, model, nr_bins, batch_size, print_target_hist)
67 nr_bins=nr_bins,
68 return_loss=True,
---> 69 return_raw_accuracy=True
70 )
71 test_loss = result["loss"]

~/Library/Python/3.7/lib/python/site-packages/syft/workers/websocket_client.py in evaluate(self, dataset_key, return_histograms, nr_bins, return_loss, return_raw_accuracy)
221 nr_bins=nr_bins,
222 return_loss=return_loss,
--> 223 return_raw_accuracy=return_raw_accuracy,
224 )
225

~/Library/Python/3.7/lib/python/site-packages/syft/workers/websocket_client.py in _send_msg_and_deserialize(self, command_name, *args, **kwargs)
113 # Send the message and return the deserialized response.
114 serialized_message = sy.serde.serialize(message)
--> 115 response = self._send_msg(serialized_message)
116 return sy.serde.deserialize(response)
117

~/Library/Python/3.7/lib/python/site-packages/syft/workers/websocket_client.py in _send_msg(self, message, location)
81
82 def _send_msg(self, message: bin, location=None) -> bin:
---> 83 return self._recv_msg(message)
84
85 def _forward_to_websocket_server_worker(self, message: bin) -> bin:

~/Library/Python/3.7/lib/python/site-packages/syft/workers/websocket_client.py in _recv_msg(self, message)
102 if not self.ws.connected:
103 raise RuntimeError(
--> 104 "Websocket connection closed and creation of new connection failed."
105 )
106 return response

RuntimeError: Websocket connection closed and creation of new connection failed.

Server message:

2020-02-18 17:54:56,388 | Task exception was never retrieved
future: <Task finished coro=<WebsocketServerWorker._producer_handler() done, defined at /opt/conda/lib/python3.6/site-packages/syft/workers/websocket_server.py:95> exception=AttributeError("'dict' object has no attribute 'owner'",)>
Traceback (most recent call last):
File "/opt/conda/lib/python3.6/site-packages/syft/workers/websocket_server.py", line 113, in _producer_handler
response = self._recv_msg(message)
File "/opt/conda/lib/python3.6/site-packages/syft/workers/websocket_server.py", line 124, in _recv_msg
return self.recv_msg(message)
File "/opt/conda/lib/python3.6/site-packages/syft/workers/base.py", line 310, in recv_msg
response = self._message_routertype(msg)
File "/opt/conda/lib/python3.6/site-packages/syft/workers/base.py", line 457, in execute_command
command_name, response, list(return_ids), self
File "/opt/conda/lib/python3.6/site-packages/syft/generic/frameworks/hook/hook_args.py", line 665, in register_response
new_response = register_response_function(response, response_ids=response_ids, owner=owner)
File "/opt/conda/lib/python3.6/site-packages/syft/generic/frameworks/hook/hook_args.py", line 766, in
return lambda x, **kwargs: f(lambdas, x, **kwargs)
File "/opt/conda/lib/python3.6/site-packages/syft/generic/frameworks/hook/hook_args.py", line 522, in two_fold
return lambdas[0](args[0], **kwargs), lambdas[1](args[1], **kwargs)
File "/opt/conda/lib/python3.6/site-packages/syft/generic/frameworks/hook/hook_args.py", line 744, in
else lambda i, **kwargs: register_tensor(i, **kwargs)
File "/opt/conda/lib/python3.6/site-packages/syft/generic/frameworks/hook/hook_args.py", line 712, in register_tensor
tensor.owner = owner
AttributeError: 'dict' object has no attribute 'owner'

Exactly the same issue is described here

Version of the lib is '0.2.3a1'

[brandonhee]

See #2948

[karlhigley]

Sounds like we might need to release a new version with recent fixes.

[karlhigley]

Wait, #2948 was included in 0.2.3.a1...

[9sashafr]

I checked 2948. It is not the source of the problem.

[9sashafr]

The source of the problem is incorrect type of the function output. According to Operation class (from syft/messaging/message.py) the output must be Tensor. But evaluate() function (from syft/federated/federated_client.py) returns dictionary of floats.

Facing the same Issue with recently published docker image also @karlhigley

[fermat97]

@karlhigley I have more or less the same problem running the same tutorial. @9sashafr could you find any workaround?

[karlhigley]

I'm in the process of doing some clean-up deep in the guts of Plans and Operations, starting with #3078. It may take a while to resolve this issue; not sure if making the output type more flexible or adding separate messages to the Syft protocol will turn out to be a preferable approach. Will keep this issue in mind as we sort through issues with the core abstractions.

[brandonhee]

I was able to workaround by changing the generic/frameworks/hook/hook_args.py file so the register_tensor function is:

def register_tensor(
    tensor: FrameworkTensorType, owner: AbstractWorker, response_ids: List = list()
):
    """
    Registers a tensor.

    Args:
        tensor: A tensor.
        owner: The owner that makes the registration.
        response_ids: List of ids where the tensor should be stored
            and each id is pop out when needed.
    """

    #tensor.owner = owner
    #try:
    #    tensor.id = response_ids.pop(-1)
    #except IndexError:
    #    raise exceptions.ResponseSignatureError

    owner.register_obj(tensor, response_ids.pop(-1))

[Dhrumilsoni]

@brandonhee After doing, what you suggested I got another error saying dict object has no attribute 'id'.

Any ideas?

[fermat97]

@Dhrumilsoni I have tried with the master and I think the bug has been fixed and it works now.

[karlhigley]

Fixed in master and should work in the next release.