PointPillars ONNX/TensorRT Export

[AliT1369]

Prerequisite

• I have searched Issues and Discussions but cannot get the expected help.
• I have read the FAQ documentation but cannot get the expected help.
• The bug has not been fixed in the latest version (dev-1.x) or latest version (dev-1.0).

Task

I'm using the official example scripts/configs for the officially supported tasks/models/datasets.

Branch

main branch https://github.com/open-mmlab/mmdetection3d

Environment

11/18 10:00:43 - mmengine - INFO -

11/18 10:00:43 - mmengine - INFO - Environmental information
11/18 10:01:00 - mmengine - INFO - sys.platform: win32
11/18 10:01:00 - mmengine - INFO - Python: 3.8.20 | packaged by conda-forge | (default, Sep 30 2024, 17:44:03) [MSC v.1929 64 bit (AMD64)]
11/18 10:01:00 - mmengine - INFO - CUDA available: True
11/18 10:01:00 - mmengine - INFO - MUSA available: False
11/18 10:01:00 - mmengine - INFO - numpy_random_seed: 2147483648
11/18 10:01:00 - mmengine - INFO - GPU 0: NVIDIA GeForce RTX 4070 SUPER
11/18 10:01:00 - mmengine - INFO - CUDA_HOME: C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.1
11/18 10:01:00 - mmengine - INFO - NVCC: Cuda compilation tools, release 12.1, V12.1.66
11/18 10:01:00 - mmengine - INFO - MSVC: Microsoft (R) C/C++ Optimizing Compiler Version 19.44.35217 for x64
11/18 10:01:00 - mmengine - INFO - GCC: n/a
11/18 10:01:00 - mmengine - INFO - PyTorch: 2.1.2
11/18 10:01:00 - mmengine - INFO - PyTorch compiling details: PyTorch built with:

• C++ Version: 199711
• MSVC 192930151
• Intel(R) Math Kernel Library Version 2020.0.2 Product Build 20200624 for Intel(R) 64 architecture applications
• Intel(R) MKL-DNN v3.1.1 (Git Hash 64f6bcbcbab628e96f33a62c3e975f8535a7bde4)
• OpenMP 2019
• LAPACK is enabled (usually provided by MKL)
• CPU capability usage: AVX512
• CUDA Runtime 12.1
• NVCC architecture flags: -gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_61,code=sm_61;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86;-gencode;arch=compute_90,code=sm_90
• CuDNN 8.8.1 (built against CUDA 12.0)
• Magma 2.5.4
• Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=12.1, CUDNN_VERSION=8.8.1, CXX_COMPILER=C:/cb/pytorch_1000000000000/work/tmp_bin/sccache-cl.exe, CXX_FLAGS=/DWIN32 /D_WINDOWS /GR /EHsc /bigobj /FS -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOCUPTI -DLIBKINETO_NOROCTRACER -DUSE_FBGEMM -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE /utf-8 /wd4624 /wd4068 /wd4067 /wd4267 /wd4661 /wd4717 /wd4244 /wd4804 /wd4273, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_DISABLE_GPU_ASSERTS=OFF, TORCH_VERSION=2.1.2, USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=OFF, USE_NNPACK=OFF, USE_OPENMP=ON, USE_ROCM=OFF,

11/18 10:01:00 - mmengine - INFO - TorchVision: 0.16.2
11/18 10:01:00 - mmengine - INFO - OpenCV: 4.12.0
11/18 10:01:00 - mmengine - INFO - MMEngine: 0.10.7
11/18 10:01:00 - mmengine - INFO - MMCV: 2.1.0
11/18 10:01:00 - mmengine - INFO - MMCV Compiler: MSVC 192930148
11/18 10:01:00 - mmengine - INFO - MMCV CUDA Compiler: 12.1
11/18 10:01:00 - mmengine - INFO - MMDeploy: 1.3.1+3f8604b
11/18 10:01:00 - mmengine - INFO -

11/18 10:01:00 - mmengine - INFO - Backend information
11/18 10:01:00 - mmengine - INFO - tensorrt: None
11/18 10:01:00 - mmengine - INFO - ONNXRuntime: 1.16.0
11/18 10:01:00 - mmengine - INFO - ONNXRuntime-gpu: None
11/18 10:01:00 - mmengine - INFO - ONNXRuntime custom ops: Available
11/18 10:01:00 - mmengine - INFO - pplnn: None
11/18 10:01:00 - mmengine - INFO - ncnn: None
11/18 10:01:00 - mmengine - INFO - snpe: None
11/18 10:01:00 - mmengine - INFO - openvino: None
11/18 10:01:00 - mmengine - INFO - torchscript: 2.1.2
11/18 10:01:00 - mmengine - INFO - torchscript custom ops: NotAvailable
11/18 10:01:00 - mmengine - INFO - rknn-toolkit: None
11/18 10:01:00 - mmengine - INFO - rknn-toolkit2: None
11/18 10:01:00 - mmengine - INFO - ascend: None
11/18 10:01:00 - mmengine - INFO - coreml: None
11/18 10:01:00 - mmengine - INFO - tvm: None
11/18 10:01:00 - mmengine - INFO - vacc: None
11/18 10:01:00 - mmengine - INFO -

11/18 10:01:00 - mmengine - INFO - Codebase information
11/18 10:01:01 - mmengine - INFO - mmdet: 3.2.0
11/18 10:01:01 - mmengine - INFO - mmseg: None
11/18 10:01:01 - mmengine - INFO - mmpretrain: 1.2.0
11/18 10:01:01 - mmengine - INFO - mmocr: None
11/18 10:01:01 - mmengine - INFO - mmagic: None
11/18 10:01:01 - mmengine - INFO - mmdet3d: 1.4.0
11/18 10:01:01 - mmengine - INFO - mmpose: None
11/18 10:01:01 - mmengine - INFO - mmrotate: None
11/18 10:01:01 - mmengine - INFO - mmaction: None
11/18 10:01:01 - mmengine - INFO - mmrazor: None
11/18 10:01:01 - mmengine - INFO - mmyolo: None

Reproduces the problem - code sample

Reproduces the problem - command or script

/tools/deploy.py configs/mmdet3d/voxel-detection/voxel-detection_onnxruntime_dynamic.py ../mmdetection3d/configs/pointpillars/pointpillars_hv_secfpn_sbn-all_8xb4-2x_nus-3d.py ../mmdetection3d/checkpoints/hv_pointpillars_secfpn_sbn-all_4x8_2x_nus-3d_20210826_225857-f19d00a3.pth ../mmdetection3d/demo/data/nuscenes/n015-2018-07-24-11-22-45+0800__LIDAR_TOP__1532402927647951.pcd.bin --work-dir onnx_pp_nus_secfpn --show

Reproduces the problem - error message

Additional information

Hi, I’m trying to export the PointPillars model from MMDetection3D using MMDeploy, and I’m facing inconsistencies and documentation gaps. I’d appreciate clarification on a few key points.

Background

• I can run the model normally through MMDetection3D, and the inference results look correct.
• I then export the same model using MMDeploy to ONNX.
• The deploy.py tool shows that the PyTorch output and ONNX output match closely (as expected).
• However, both differ noticeably from the original MMDetection3D inference results, which I assume is due to missing pre-processing/post-processing steps (voxelization, anchor generation, NMS, box decoding, etc.).

Documentation Confusion

The MMDeploy documentation states that several operations from MMDetection3D—specifically voxelize preprocessing and postprocessing for 3D detectors—are not supported in ONNX.

My questions:

1. If preprocessing/postprocessing ops are not supported in ONNX, are they supported when exporting to TensorRT?

It is unclear whether the unsupported ONNX operators areactually supported in TensorRT or not

A clear table or confirmation would help a lot.

2. For ONNX export: what is the correct way to decode the model outputs?

The exported ONNX model outputs three tensors, for example:

(1, 140, 200, 200)
(1, 126, 200, 200)
(1, 28, 200, 200)

I assume they correspond to:

• classification scores
• bounding box predictions
• direction predictions

…but the exact structure is undocumented.
To run inference in my own C# code, I need to replicate the same decoding logic used by MMDeploy.

Is there an official reference or expected approach for decoding ONNX outputs into:

• final 3D bounding boxes,
• orientation,
• class scores,
• and applying NMS?

3. Expected workflow

If ONNX/TensorRT do not include voxelization and postprocessing, is the expected workflow:

• Run voxelization manually (outside ONNX/TensorRT).
• Feed voxel tensors into ONNX/TensorRT.
• Run box decoding + NMS manually (based on MMDetection3D code).

If so:

• Which exact MMDetection3D modules should be ported?
• Is there a minimal reference implementation for postprocessing?

4. Goal

I want to load and run the exported ONNX (or TensorRT) model inside a C# application.
To do that, I need:

• a clear description of the ONNX output format
• confirmation about which pre/post ops MMDeploy expects the caller to implement
• clarification on whether TensorRT supports ops that ONNX does not

Thanks in advance. Any clarification or pointers to the correct documentation would be very helpful.