Inquiry regarding MuxServe++ implementation and model support for reproduction

[Rabenda]

Hi authors,

Thank you for the impressive work presented in your paper. We are very interested in your research and are currently working on reproducing the experimental results mentioned in the paper.

In the text, you referenced MuxServe++ as part of your methodology. However, when we attempted to run the publicly available version of MuxServe, we encountered limitations regarding support for newer models like Llama3-8B.

To ensure our reproduction aligns with your findings, we would appreciate it if you could clarify:
How were the experiments involving newer models conducted in your study?
Is the MuxServe++ implementation or the specific code used for these experiments available (or planned to be released)?

Any guidance or access to the relevant codebase would be greatly appreciated.

[jiarong0907]

@Rabenda Thanks for your interests in our project.

@shanyu-sys Could you provide some guidance here? Thanks!

[shanyu-sys]

@Rabenda Thanks again for your interest in our project!

For MuxServe++, we use kvcached to support more recent model architectures. We found that the original MuxServe requires substantial engineering effort to support the diverse and newer model architectures we evaluated, due to its design limitations.
Using our own kvcached makes Muxserve++ a stronger and more flexible baseline.

[Rabenda]

@shanyu-sys Thanks for the clarification! It's great to know that MuxServe++ leverages kvcached to overcome the limitations of the original version.
Since I am currently working on reproducing the results mentioned in your study, could you please let me know if the MuxServe++ codebase is publicly accessible now? If not, would it be possible to provide a private preview/branch for research purposes?

[shanyu-sys]

Hi @Rabenda, we are still working on cleaning up the research code and making it publicly accessible. We will let you know you when it is ready for access. In the meantime, welcome to try out colocating multiple LLMs with our kvcached (see example). Thanks for your interest!