[PD] Add custom gpu id to device topo support

[stmatengss]

Motivation

Currently, PD doesn't support indicating specified IB devices for different GPU id (tp/dp rank).
This PR supports this feature by introducing a new json format, and it consists of a mapping between gpu id and IB devices.

Usage

# Support both methods.
--disaggregation-ib-device ib0,ib1,ib2
--disaggregation-ib-device {0: "ib0, ib1", 1: "ib2, ib3", 2: "ib4"}

Modifications

Accuracy Tests

Benchmarking and Profiling

Checklist

• Format your code according to the Format code with pre-commit.
• Add unit tests according to the Run and add unit tests.
• Update documentation according to Write documentations.
• Provide accuracy and speed benchmark results according to Test the accuracy and Benchmark the speed.

[gemini-code-assist]

Warning

You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again!

[stmatengss]

@ShangmingCai PTAL. thx

[ShangmingCai]

LGTM. But we definitely should fix auto-discovery.

nit: if ib_device_str is None has already been checked before the try block, so it is impossible to be None here, we can just return ib_device_str.