[RoadMap] Mooncake Roadmap Q1 & Q2 2025

[stmatengss]

We categorized our roadmap into two major themes: New Component (Mooncake Managed Object Store) and New Features of Mooncake. As we are seeing more.

---

New Component: Mooncake Managed Object Store

25Q1

• Object Store Interfaces
  • vLLM adapter for PUT/GET interface #45
    • Multiple Language support
  • Prefix KVCache Management
• Object Store Features
  • Support M-to-N (XpYd) KVCache Sharing between P/D instances
    • Support building multiple connections between Xp and Yd
    • Object Store Master to manage metadata of KVcache
    • Asynchronous KV cache transfer with Layer-by-layer pipeline
  • GDR-based VRAM-to-VRAM implementation

25Q2+

• Advanced Features
  • Cluster reconfiguration
  • Better cache eviction strategy Add LRU in MasterService, complexity O(1) #287
  • Multi-tier caching
    • support 3FS as secondary storage @JasonZhang517

New Features of Mooncake

Transfer Engine

• 25Q1
  • Revise APIs to support more protocols/media #52
  • Check & revise error handling (problems from device/connection/software)
  • Metadata Management and OOB Communication
    • Plugin-based metadata client (etcd, Redis, HTTP, ...)
    • Extend current's naive handshake mechanism [TransferEngine] feature: support p2phandshake without metadata server #240
  • Add support of transports
    • CXL/Shared memory #48
    • NVMe-of
    • Libfabric transport layer support #76
• 25Q2+
  • Transfer data from GPU memory
    • More features integrated into GDS, including direct RDMA-based VRAM-to-DRAM transfers
    • VRAM-to-VRAM fast transfers based on NVLink
    • Support for GPU of other vendors (domestic...)
  • Path Selection
    • Automatically generate Topology Matrix according to hardware conditions [Feature] support auto-detect topo for both GPUs and CPUs #244
    • Optimize multi-NIC scheduling strategy based on request load and hardware characteristics
  • Performance
    • Working-set based QP management, avoiding RNIC cache thrashing
  • Security Enhancement
    • Remote memory protection based on RDMA keys
    • Auth
  • Support Nvidia Nixl Add Mooncake backend in NIXL ai-dynamo/nixl#169

P2P Store

• 25Q1
  • Support python interface Support python interface for p2p store #326
  • Network bandwidth allocation

LLM Framework Integration

• 25Q1
  • ZeroCopy from vLLM to RDMA memory
  • Support More LLM Framework
    • TensorRT-LLM
    • SGLang [MooncakeAdaptor] Init sglang_adaptor to support SGLang using transfer engine #181
    • LMDeploy LMDeploy Distserve InternLM/lmdeploy#3304
  • Support Asynchronous KV cache transfer for vLLM
    • Layer-by-layer pipeline
    • API: transferAsync in class VLLMAdaptor
  • Support KVCache Prefetch

---

If any of the items you wanted is not on the roadmap, your suggestion and contribution are still welcomed! Please feel free to comment in this thread, open feature request, or create an RFC.

[VegetaPn]

The roadmap looks fantastic! What does "Cluster reconfiguration" mean? Does it refer to dynamically adjusting the P/D role type?

[ziqianliu423]

Hello, I am interested in the Transport support CXL/shared memory and maybe I can do this in one month.

[stmatengss]

The roadmap looks fantastic! What does "Cluster reconfiguration" mean? Does it refer to dynamically adjusting the P/D role type?

Yes, it has two meanings. Firstly, any GPU server can freely join or leave the Mooncake KVCache pool. Secondly, a Prefill or Decoding Instance can change its role type.

[stmatengss]

Hello, I am interested in the Transport support CXL/shared memory and maybe I can do this in one month.

Thank you for your contribution. I look forward to seeing the pull request on GitHub.

[doujiang24]

Check & revise error handling (problems from device/connection/software)
Hello, I'm happy to take this one in Q1.

[alogfans]

Check & revise error handling (problems from device/connection/software)
Hello, I'm happy to take this one in Q1.

Thank you for your contribution! Looking forward to seeing the pull request on GitHub.

[ANormalMan12]

I think it quite hard to implement "ZeroCopy from vLLM to RDMA memory" and "Layer-by-layer pipeline" without modifying core components of vllm a lot. Is there an easier way to implement "ZeroCopy" and "Layer-by-layer Pipeline"?

[cherhh]

Is the functionality provided by this Mooncake Managed Object Store similar to that of an in-memory database like Redis?