[refactor] Simplify initialization and improve API usability

[0oshowero0]

Background

In previous versions of TransferQueue, we lacked an easy-to-use initialization process. Users had to manually initialize and configure TQ, leading to verbose boilerplate code.

Reference for previous complex usage:

• TQ tutorial：

  TransferQueue/tutorial/01_core_components.py

  Line 59 in c2bb0fa

  def demonstrate_basic_setup():

• verl integration：https://github.com/verl-project/verl/blob/e4915bd8859e75a8634ec48c20a0bbc781d66eb8/verl/experimental/transfer_queue/ray_trainer.py#L340

Key Changes

• Initialization: Introduced a one-click initialization via transfer_queue.init(conf).
• Interface: Wrapped the core TransferQueueClient APIs directly into the top-level transfer_queue namespace (e.g., tq.put, tq.get_meta).
• Config: Added a default config.yaml to support zero-config initialization and formalize the config structure.
• Backend:
  • Simplified backend class names for better readability:
    - AsyncSimpleStorageManager -> SimpleStorage
    - YuanrongStorageManager -> Yuanrong
    - MooncakeStorageManager -> MooncakeStore
  • Simplified the initialization logic of StorageManager. [Enhancement] A suggestion regarding StorageManager initialization #6

Usage Example

# In process 0, node A
import transfer_queue as tq
tq.init()   # Initialize the TransferQueue
tq.put(...) # then you can use tq for data operations

# In process 1, node B (with Ray connected to node A)
import transfer_queue as tq
tq.init()   # This will only initialize a TransferQueueClient and link with existing TQ
metadata = tq.get_meta(...)
data = tq.get_data(metadata)

For more details, please refer to the updated tutorials.

Config Structure

# This is the default configuration of TransferQueue. Users may modify the default value
# and use transfer_queue.init(conf) to overwrite the config entries.

controller:
  # User-defined sampler. User can pass sampler instance to overwrite this string config.
  sampler: SequentialSampler
  # Whether return an empty BatchMeta to prevent request blocking when no enough data is available
  polling_mode: False
  # ZMQ Server IP & Ports (automatically generated during init)
  zmq_info: null


backend:
  # Pluggable storage/transport backend of TransferQueue. Choose from:
  # SimpleStorage, Yuanrong, MooncakeStore, ...
  storage_backend: SimpleStorage

  # For SimpleStorage:
  SimpleStorage:
    # Total number of samples
    total_storage_size: 100000
    # Number of distributed storage units for SimpleStorage backend
    num_data_storage_units: 2
    # ZMQ Server IP & Ports (automatically generated during init)
    zmq_info: null

  # For Yuanrong:
  # TODO

  # For MooncakeStore:
  # TODO

TODO

• Provide Yuanrong backend default config (and initialization) @dpj135
• Provide MooncakeStore backend default config (and initialization)

---

CC: @wuxibin89 @dpj135

[Copilot]

Pull request overview

This PR optimizes the TransferQueue initialization process by introducing a simplified init() function that provides a more user-friendly API. The changes consolidate the initialization logic and rename storage manager types to be more concise.

Changes:

• Added a new transfer_queue.init() function that simplifies TransferQueue initialization with automatic controller detection and configuration management
• Renamed storage manager types: AsyncSimpleStorageManager → SimpleStorage, MooncakeStorageManager → MooncakeStore, YuanrongStorageManager → Yuanrong with backward compatibility through deprecation warnings
• Moved process_zmq_server_info function from client.py to zmq_utils.py for better organization and added config.yaml for default configuration

Reviewed changes

Copilot reviewed 15 out of 16 changed files in this pull request and generated 10 comments.

Show a summary per file

File	Description
transfer_queue/interface.py	New file introducing the simplified init(), get_meta(), get_data(), put(), and related wrapper functions for easier API usage
transfer_queue/controller.py	Added store_config() and get_config() methods to store/retrieve global TransferQueue configuration
transfer_queue/config.yaml	New default configuration file defining controller and backend settings
transfer_queue/init.py	Updated exports to include new interface functions and process_zmq_server_info
transfer_queue/storage/managers/factory.py	Added deprecation warnings for old manager type names with automatic fallback to new names
transfer_queue/storage/managers/simple_backend_manager.py	Updated registration name and added deprecation warning for storage_unit_infos config key
transfer_queue/storage/managers/base.py	Changed config parameter type annotation to DictConfig
transfer_queue/storage/managers/yuanrong_manager.py	Updated registration name from YuanrongStorageManager to Yuanrong
transfer_queue/storage/managers/mooncake_manager.py	Updated registration name from MooncakeStorageManager to MooncakeStore
transfer_queue/utils/zmq_utils.py	Added process_zmq_server_info() function moved from client.py
transfer_queue/client.py	Removed process_zmq_server_info() function and unused imports
tutorial/01_core_components.py	Simplified tutorial to use new tq.init() and module-level functions
tests/test_client.py	Updated test cases to use new SimpleStorage manager type name
requirements.txt	Added omegaconf dependency
pyproject.toml	Added *.yaml to package data for config file inclusion
README.md	Updated storage backend names in documentation

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 22 out of 23 changed files in this pull request and generated 15 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.