CUDA Support

[wingter562]

Feature desired
Looks like sending models to GPU is not allowed if Pytorch is hooked with PySyft.
For example:

hook = sy.TorchHook(torch)  # no problem occurs if comment this line
model.to(torch.device('cuda')) 

incurs: RuntimeError: Expected object of backend CPU but got backend CUDA for argument #2 ‘source’
But that never happens if I don't hook PySyft with Pytorch.

Describe the solution you'd like
Does PySyft provide Cuda support, both on Windows and Linux?

[wingter562]

More
Actually I strictly followed @LaRiffle 's tutorial on CNN+FL at here

I copied the code and ran locally on my windows 1050ti PC with python3.7+pytorch1.1.0+PySyft0.1.23.

Still got the same error:

RuntimeError: Expected object of backend CPU but got backend CUDA for argument #2 ‘source’

when executing model = Net().to(device) where device = torch.device('cuda')

So I doubt the availability of CUDA in case Pytorch is hooked with PySyft.

[Williamongh]

Try this to bypass this bug:

torch.set_default_tensor_type('torch.cuda.FloatTensor')
model = Net().to(device)
torch.set_default_tensor_type('torch.FloatTensor')

Some tutorials don't work. You have to face one bug after another.

[wingter562]

Thanks! the problem solved but looks like the efficiency is not decent on GPU with syft.

[Williamongh]

If you are working with websocket-based asynchronous(i.e. parallel) federated learning, I'd be glad to discuss it with you.

[fermat97]

@Williamongh I would like to try the asynchronous websocket-based scenario on GPU. I was wandering how one can provide the information to use the GPU on a remote worker. Is that enough to just use Net.to(device), since there is no option on the sy.TrainConfig. Thank you.