About the metrics of the official model.

[mengtan00]

Thanks for your great work!
I encountered a problem when evaluating the model provided in the Google Drive by running:

CUDA_VISIBLE_DEVICES=0 python tools/test.py projects/configs/SSR/SSR_e2e.py ssr.pth --launcher none --eval bbox --tmpdir tmp

and got the output as follows:

While plan_L2_stp3_xs matches the L2 accuracy well, I can't find the metrics matching the Collision Rate.
Was there anything wrong with my evaluation?

[PeidongLi]

-------------- Planning --------------

plan_L2_1s:0.18560029235750863 
plan_L2_2s:0.3552287642032173
plan_L2_3s:0.6150584226519661
plan_obj_col_1s:0.0
plan_obj_col_2s:0.0
plan_obj_col_3s:6.511688674886516e-05
plan_obj_box_col_1s:0.0001953506544246923
plan_obj_box_col_2s:0.00034186364524321154
plan_obj_box_col_3s:0.0012697792857809676
fut_valid_flag:1.0
plan_L2_stp3_1s:0.24661295420432905
plan_obj_col_stp3_1s:0.0
plan_obj_box_col_stp3_1s:0.0
plan_L2_stp3_2s:0.6393535662851461
plan_obj_col_stp3_2s:0.0
plan_obj_box_col_stp3_2s:0.0007814026176987692
plan_L2_stp3_3s:1.3278388364209877
plan_obj_col_stp3_3s:0.0001953506544246923
plan_obj_box_col_stp3_3s:0.0042977143973432315

This is my original result. Can you check the log for specific environment?

[mengtan00]

Thanks for your response! My environment is as follows. Are there packages that may influence the metrics?

absl-py==2.1.0
addict==2.4.0
anyio==4.5.2
argon2-cffi==23.1.0
argon2-cffi-bindings==21.2.0
arrow==1.3.0
asttokens==3.0.0
async-lru==2.0.4
attrs==25.1.0
babel==2.17.0
backcall==0.2.0
beautifulsoup4==4.13.3
black==24.8.0
bleach==6.1.0
cachetools==5.5.1
certifi==2025.1.31
cffi==1.17.1
charset-normalizer==3.4.1
click==8.1.8
comm==0.2.2
contourpy==1.1.1
cycler==0.12.1
debugpy==1.8.12
decorator==5.1.1
defusedxml==0.7.1
descartes==1.1.0
exceptiongroup==1.2.2
executing==2.2.0
fastjsonschema==2.21.1
filelock==3.16.1
fire==0.7.0
flake8==7.1.1
fonttools==4.56.0
fqdn==1.5.1
fsspec==2025.2.0
grpcio==1.70.0
h11==0.14.0
httpcore==1.0.7
httpx==0.28.1
huggingface-hub==0.28.1
idna==3.10
imageio==2.35.1
importlib_metadata==8.5.0
importlib_resources==6.4.5
iniconfig==2.0.0
ipykernel==6.29.5
ipython==8.12.3
ipywidgets==8.1.5
isoduration==20.11.0
jedi==0.19.2
Jinja2==3.1.5
joblib==1.4.2
json5==0.10.0
jsonpointer==3.0.0
jsonschema==4.23.0
jsonschema-specifications==2023.12.1
jupyter==1.1.1
jupyter-console==6.6.3
jupyter-events==0.10.0
jupyter-lsp==2.2.5
jupyter_client==8.6.3
jupyter_core==5.7.2
jupyter_server==2.14.2
jupyter_server_terminals==0.5.3
jupyterlab==4.3.5
jupyterlab_pygments==0.3.0
jupyterlab_server==2.27.3
jupyterlab_widgets==3.0.13
kiwisolver==1.4.7
lazy_loader==0.4
llvmlite==0.31.0
lyft-dataset-sdk==0.0.8
Markdown==3.7
MarkupSafe==2.1.5
matplotlib==3.3.0
matplotlib-inline==0.1.7
mccabe==0.7.0
mistune==3.1.1
mmcv-full==1.4.0
mmdet==2.14.0
-e git+https://github.com/open-mmlab/mmdetection3d.git@f1107977dfd26155fc1f83779ee6535d2468f449#egg=mmdet3d
mmsegmentation==0.14.1
mypy-extensions==1.0.0
narwhals==1.26.0
nbclient==0.10.1
nbconvert==7.16.6
nbformat==5.10.4
nest-asyncio==1.6.0
networkx==2.2
notebook==7.3.2
notebook_shim==0.2.4
numba==0.48.0
numpy==1.19.5
nuscenes-devkit==1.1.9
opencv-python==4.11.0.86
overrides==7.7.0
packaging==24.2
pandas==1.4.4
pandocfilters==1.5.1
parso==0.8.4
pathspec==0.12.1
pexpect==4.9.0
pickleshare==0.7.5
pillow==10.4.0
pkgutil_resolve_name==1.3.10
platformdirs==4.3.6
plotly==6.0.0
pluggy==1.5.0
plyfile==1.0.1
prettytable==3.11.0
prometheus_client==0.21.1
prompt_toolkit==3.0.50
protobuf==6.30.0rc1
psutil==6.1.1
ptyprocess==0.7.0
pure_eval==0.2.3
pycocotools==2.0.7
pycodestyle==2.12.1
pycparser==2.22
pyflakes==3.2.0
Pygments==2.19.1
pyparsing==3.1.4
pyquaternion==0.9.9
pytest==8.3.4
python-dateutil==2.9.0.post0
python-json-logger==3.2.1
pytz==2025.1
PyWavelets==1.4.1
PyYAML==6.0.2
pyzmq==26.2.1
referencing==0.35.1
requests==2.32.3
rfc3339-validator==0.1.4
rfc3986-validator==0.1.1
rpds-py==0.20.1
safetensors==0.5.2
scikit-image==0.19.3
scikit-learn==1.0.2
scipy==1.7.3
Send2Trash==1.8.3
Shapely==1.8.5
similaritymeasures==1.0.0
six==1.17.0
sniffio==1.3.1
soupsieve==2.6
stack-data==0.6.3
tensorboard==2.18.0
tensorboard-data-server==0.7.2
termcolor==2.4.0
terminado==0.18.1
terminaltables==3.1.10
threadpoolctl==3.5.0
tifffile==2023.7.10
timm==1.0.14
tinycss2==1.2.1
tomli==2.2.1
torch==1.9.1+cu111
torchaudio==0.9.1
torchstat==0.0.7
torchvision==0.10.1+cu111
tornado==6.4.2
tqdm==4.67.1
traitlets==5.14.3
trimesh==2.35.39
types-python-dateutil==2.9.0.20241206
typing_extensions==4.12.2
tzdata==2025.1
uri-template==1.3.0
urllib3==2.2.3
wcwidth==0.2.13
webcolors==24.8.0
webencodings==0.5.1
websocket-client==1.8.0
Werkzeug==3.1.3
widgetsnbextension==4.0.13
yapf==0.40.1
zipp==3.20.2

[SEU-zxj]

@mengtan00 I got the same result with you! By the way, do you solve the problem now?
I use A800 to run the test.py.

-------------- Planning --------------
plan_L2_1s:0.18561602474882635
plan_L2_2s:0.35524758928265565
plan_L2_3s:0.6150620685221184
plan_obj_col_1s:0.0
plan_obj_col_2s:0.0
plan_obj_col_3s:6.511688674886516e-05
plan_obj_box_col_1s:0.0009767532721234617
plan_obj_box_col_2s:0.0010255909357296347
plan_obj_box_col_3s:0.0023767663605116754
fut_valid_flag:1.0
plan_L2_stp3_1s:0.2466333914444972
plan_obj_col_stp3_1s:0.0
plan_obj_box_col_stp3_1s:0.0007814026176987692
plan_L2_stp3_2s:0.6393702675243328
plan_obj_col_stp3_2s:0.0
plan_obj_box_col_stp3_2s:0.001172103926548154
plan_L2_stp3_3s:1.327791417212411
plan_obj_col_stp3_3s:0.0001953506544246923
plan_obj_box_col_stp3_3s:0.007227974213713616

[SEU-zxj]

@PeidongLi I encountered the same problem with @mengtan00 , it seems that the Collision Rate is not accurate as you list in the paper......
Do you test the model on RTX 3090? It is strange that L2 is mostly the same, but Collision rates are not......

[SEU-zxj]

I list these metrics in a table, hope this helps observe the difference. @PeidongLi @mengtan00

[PeidongLi]

Sorry for the late reply. I'm very busy recently and have no bandwidth to re-check the environment. Actually, our internal images didn't follow the readme from VAD strictly. So I guess some difference lead to the result.

Thanks the effort from @SEU-zxj to list the results. The provided ckpt is trained on the released code which refactored from our internal code, so it has minor differences with the paper. It looks the results of second row of your table is close to our evaluated results of the provided ckpt.

I will update the Installation once I have some time to do it.

[SEU-zxj]

@PeidongLi Thanks for your reply! I totally understand—no worries at all.

Yes, the collision rate in the second row (evaluated on A100) matches your reported results quite well. However, for the third row (evaluated on A800), while the L2 error aligns with your results, the collision rate does not, which is a bit unexpected.

It might be due to differences in the metric calculation code—we're still investigating this. If we find anything useful, we’ll share it here for discussion.

Looking forward to your updated installation instructions when you have time. Thanks again for your great work!

[mengtan00]

I list these metrics in a table, hope this helps observe the difference. @PeidongLi @mengtan00

@SEU-zxj Hi, thank you for showing more evaluation results. Does that mean different GPUs will also influence the metrics?
I evaluated the official model on RTX3090 and got the same metrics as your evaluation on A800.

[SEU-zxj]

@mengtan00 Actually, I am not sure......
I think different computing nodes should not affect the result......
So now we have these conclusion:

1. your result on TRX3090 aligns with our results on A800
2. but our results on A100 aligns with the authors' results which does not aligns with the result in 1 (This one needs to be checked)
3. all L2 error align well (no matter we test on what GPU), only the collision rate does not align (hope this can be solved when the author update new installation method)

In the table, the result is evaluated by @Ziyuan-Yang, I need to valid that result with him......
It is strange that you use RTX3090 and obtain a totally same result with me on A800!

[BigBoomDream]

I list these metrics in a table, hope this helps observe the difference. @PeidongLi @mengtan00

@SEU-zxj @mengtan00
I have a few questions regarding the metrics used in our planning module:

1. Could you clarify the difference between plan_obj_col and plan_obj_box_col metrics?
2. what do fut_valid_flag and plan_obj_col_stp3 mean?
   Thanks a lot for your help!

[SEU-zxj]

@BigBoomDream To be honest, I do not familiar with the metric computation detail now.
You can find the details in the following code blocks:

SSR/projects/mmdet3d_plugin/SSR/SSR.py

Lines 596 to 603 in 9b17910

	metric_dict['plan_L2_{}s'.format(i+1)] = traj_L2
	metric_dict['plan_obj_col_{}s'.format(i + 1)] = obj_coll.mean().item()
	metric_dict['plan_obj_box_col_{}s'.format(i + 1)] = obj_box_coll.mean().item()
	# metric_dict['plan_obj_col_plus_{}s'.format(i + 1)] = obj_coll_plus.mean().item()
	# metric_dict['plan_obj_box_col_plus_{}s'.format(i + 1)] = obj_box_coll_plus.mean().item()
	metric_dict['plan_L2_stp3_{}s'.format(i+1)] = traj_L2_stp3
	metric_dict['plan_obj_col_stp3_{}s'.format(i + 1)] = obj_coll[-1].item()
	metric_dict['plan_obj_box_col_stp3_{}s'.format(i + 1)] = obj_box_coll[-1].item()

you can find that col_stp3 only use the final frame of col (with/without box)

And the logical of plan_obj_col and plan_obj_box_col, you can find their implementation as follows:

SSR/projects/mmdet3d_plugin/SSR/planner/metric_stp3.py

Line 355 in 9b17910

def evaluate_coll(

SSR/projects/mmdet3d_plugin/SSR/planner/metric_stp3.py

Line 279 in 9b17910

def evaluate_single_coll(self, traj, segmentation, input_gt):

But I think in the paper of the author, the collision rate refers to plan_obj_box_col (VAD AVG) and plan_obj_box_col_stp3 (UniAD MAX)?
@PeidongLi What do you think of that?

[SEU-zxj]

@PeidongLi @mengtan00 @BigBoomDream
We trained SSR on our A800 server, and test the ssr_epoch_12_ema.pth on both A800 and A100 server, we list the experiment result at the last two lines, it seems that the collision rate is still not aligned......

[SEU-zxj]

@mengtan00 @PeidongLi Hello!
When I trying to install the environment, I encountered a bug with endless stream of the following warning appears during training:

/opt/nvme0/zhengxj/softwares/anaconda3/envs/world_model/lib/python3.8/site-packages/shapely/set_operations.py:133: RuntimeWarning: invalid value encountered in intersection
  return lib.intersection(a, b, **kwargs)

I raised a issue here: #18

And I suspect that this may contribute the misaligned result of the collision rate above......

[mengtan00]

@mengtan00 @PeidongLi Hello! When I trying to install the environment, I encountered a bug with endless stream of the following warning appears during training:

/opt/nvme0/zhengxj/softwares/anaconda3/envs/world_model/lib/python3.8/site-packages/shapely/set_operations.py:133: RuntimeWarning: invalid value encountered in intersection
  return lib.intersection(a, b, **kwargs)

I raised a issue here: #18

And I suspect that this may contribute the misaligned result of the collision rate above......

Hi, have you figured out this problem now?

[SEU-zxj]

@mengtan00 no, I think that bug is raised because I wrongly build mmdetion3d...... It's not the root reason of our results misaligned with the results of SSR.