Manual layer distribution per device

[leanzero-srl]

The rationale around asking for something like this would be to burden the main machine I am actually from a bit less compared to the machine I use only for AI usage. I am sure this is something other users are thinking too and want.

Would be a great feature to optimize / let the user choose the number of layers per device or at least some % of usage per device.

[Evanev7]

agreed!

[AlexCheema]

This is actually very easy to do. The main bottleneck here is we need to rethink the EXO Admin dashboard to give users more control.

It's great that we have a simple UX for new users who just want to quickly get up and running, but a lot of users want more control, e.g. setting memory limits on each node, controlling context length, controlling inference parameters like temperature.

This should be fairly easy to do once we know how it should work, so it's more of a design problem.