Dependency on depth

[Germany321]

Thanks for your excellent work! I noticed that your model takes depth as input and includes a specific stage to adapt depth information. I'm curious—what would happen if we removed the depth input? Specifically, if I train a model using only RGB images on your dataset, would the performance be comparable to the version that uses depth?

[Zhoues]

@Germany321

Thank you for your attention to the dependency on depth inputs!

Please refer to Table 7 in the main paper, where we conduct an ablation study on depth information and find that it enhances spatial understanding.

RoboBrain v2.0 has also used this dataset as an RGB-only dataset and still achieves SOTA performance across several benchmarks about spatial understanding and reasoning.

[Germany321]

Thanks for your quick response! It appears that RGB inputs also yield promising performance.

[Germany321]

I'm also curious about the scaling effect. What is the minimal scale required to achieve good performance (e.g., >80)?

[Zhoues]

@Germany321

Sorry for keeping you waiting. Actually, each part of the RefSpatial dataset serves a different purpose.

RefSpatial/
├── 2D/
│   ├── choice_qa.json (Enhances basic spatial multiple-choice question abilities; large volume)
│   └── reasoning_template_qa.json (Enhances understanding of basic spatial relationships, concepts, and spatial referring; large volume)
├── 3D/
│   ├── choice_qa.json (Enhances basic spatial multiple-choice question abilities with more spatial concepts and relationships; more precise)
│   ├── multi_view_qa.json (Enhances multi-view comprehension ability)
│   ├── reasoning_template_qa.json (Enhances deeper understanding of spatial concepts, relationships, and spatial referring; more precise)
│   ├── vacant_qa.json (Enhances ability to refer to vacant positions)
│   └── visual_choice_qa.json (Enhances visual prompt understanding in benchmarks, such as different colored boxes or point expressions; reduces the gap between benchmark and training)
├── Simulator/
│   └── metadata.json (Enhances multi-step spatial referring ability)

You can choose the corresponding dataset based on your needs. For example, during RoboBrain v2.0 training, only a subset of these was sampled (mainly 2D/3D choice_qa and reasoning_template_qa, which help in learning various spatial concepts and relationships as well as basic object location referencing, complemented by vacant_qa for learning spatial references to vacant positions).

[dohyun1411]

Hi,

Thanks for the nice work!
Is there any plan for releasing checkpoints of the RGB-input models?
It would be helpful when we don't have depth information!

[Zhoues]

Hi,

Thanks for the nice work! Is there any plan for releasing checkpoints of the RGB-input models? It would be helpful when we don't have depth information!

@dohyun1411

First of all, RoboRefer natively supports RGB-only input, as it was trained using a mix of RGB and RGB-D data.

Specifically, to use it this way, simply set enable_depth to 0 in use_api.py.

RoboRefer/API/use_api.py

Line 61 in 15f93d4

enable_depth=1

Of course, there's no need to worry if you don't have depth data, because our depth is inferred from RGB images using DepthAnything V2 (when enable_depth=1).