Regarding trajectory waypoints prediction

[lucamodica]

Hi, thank you a lot for the impressive work, especially in terms of capturing corner case scenarios with the Impromptu dataset!

My question is more of a curiosity: in many VLA works, other than predicting future trajectory as waypoints in text format (as in your work or in EMMA for example), there are alternatives such as:

• clustering the trajectories and expanding the VLM vocabulary to predict specialised action tokens that represent either waypoint trajectories or low-level controls (as in AutoVLA)
• using separate planning modules, such as a generative module (diffusion head, VAE, ...) as in the case of ORION

I was wondering if any of these 2 alternatives were explored, and if the case, how they perform after the fine-tuning stage with your dataset. Thanks!

[ahydchh]

Hi,

Thanks for the support and the great question!

regarding your query, we haven't explored these two alternatives (action token clustering or separate generative planning modules) in our current setting. We mainly focused on the text-based waypoint prediction approach similar to existing VLA baselines.

Thank you for bringing up these references!

[lucamodica]

@ahydchh Thank you for the answer! Based on your experience, do you think that using one of those alternatives could benefit more or less out of the corner case scenarios contained in the Impromptu dataset?

[ahydchh]

Based on recent trends in the field, I do think that exploring alternative action representations on the Impromptu dataset is a very promising direction. The rich corner case data we provide should theoretically help these models learn more robust distributions compared to standard datasets.

We haven't verified this experimentally yet, but we would be thrilled to see future works leveraging our data to test these architectures!

[lucamodica]

thanks a lot for the answer!