Skip to content

README.md

Latest commit

 

History

History
270 lines (221 loc) · 12.7 KB

README.md

File metadata and controls

270 lines (221 loc) · 12.7 KB

Logo

123D: Unifying Multi-Modal Autonomous Driving Data at Scale

Paper | Video | Documentation

PyPI Version Python Versions License

One library for autonomous driving datasets. 123D converts raw data from Argoverse 2, nuScenes, nuPlan, KITTI-360, PandaSet, and Waymo into a unified Apache Arrow format, and then gives you a single API to read cameras, lidar, HD maps, and labels across all of them.

✨ Features

  • Dataset download: Fetch supported datasets from their official sources via the CLI, and optionally convert directly into the unified format.
  • Hydra-based conversion CLI: YAML configs to manage your data pipelines.
  • Apache Arrow storage: columnar, memory-mapped, zero-copy reads. Fast and memory efficient.
  • Multiple sensor codecs: MP4/JPEG/PNG for cameras; LAZ/Draco/Arrow IPC for lidar.
  • No sensor duplication: Converted logs reference source camera/lidar files via relative paths, so you don't store sensors twice.
  • Unified API: Read cameras, lidar, maps, and labels through a single interface, regardless of the source dataset.
  • Built-in visualization: interactive 3D viewer (Viser), and matplotlib plotting.

📦 Installation

pip install py123d

Per-dataset extras (e.g. py123d[av2], py123d[nuscenes], py123d[waymo]) install the parser dependencies for each dataset on demand. See the Demo below for an example.

🚀 Demo

Demo using the Argoverse 2 Sensor dataset, which is publicly readable from S3 and requires no cloud authentication.

The av2-sensor-stream config downloads the requested logs/maps into a managed temp directory, converts them into our self-contained Arrow format, and cleans up the source files afterwards. PY123D_DATA_ROOT controls where the converted logs/maps are written. The script below installs the Av2 extra, converts the first 3 validation logs (~250 MB each), and launches the Viser viewer:

# 1. Install
pip install py123d[av2]

export PY123D_DATA_ROOT=/path/to/py123d_data

# 2. Download + Convert
py123d-conversion dataset=av2-sensor-stream \
  dataset.parser.splits='[av2-sensor_val]' \
  dataset.parser.downloader.num_logs=3

# 3. Launch Viewer
py123d-viser scene_filter=av2-sensor

Open http://localhost:8080 to browse the converted scenes interactively.

🖼️ Viewer

Viser 3D Viewer

📊 Supported Datasets

Scale Sensors [#/Hz] Annotations [✓/Hz]
Dataset Year Dur. [h] Dist. [km] Logs [#] Cam. LiDAR 3D Box Tls. Map
Manual nuScenes 20205.6100.91,000 6 / 121 / 20 ✓ / 2✗✓
WOD-Perc. 20206.4154.01,150 5 / 105 / 10 ✓ / 10✗✓
AV2-Sens. 20214.487.51,000 9 / 202 / 10 ✓ / 10✗✓
PandaSet 20210.28.3103 6 / 102 / 10 ✓ / 10✗✗
KITTI-360 20222.773.79 4 / 101 / 10 ✓ / 10✗✓
Auto-labeled WOD-Mot. 2021574.110,323.5*103,354 ✗✗ ✓ / 10✓ / 10✓
nuPlan 20241,174.317,808.615,910 8 / 10†5 / 20† ✓ / 20✓ / 20✓
  – nuPlan-mini 20247.2103.064 8 / 10†5 / 20† ✓ / 20✓ / 20✓
PAI-AV 20251,707.069,265.7307,332 7 / 301 / 10 ✓ / 10✗✗
  – NCore 20266.3167.61,147 7 / 301 / 10 ✓ / 10✗✗
Synth. CARLA 2017var.var.var. var.var. ✓✓✓
  – L3AD 20267.3138.7789 6 / 102 / 10 ✓ / 10✓ / 10✓

* Computed only from the non-overlapping 20 s training files.   † Released for a 120 h subset; full coverage on mini.

📝 Changelog

v0.4.0 (2026-05-19)
  • Added L3AD dataset support (CARLA-derived) with a HuggingFace-hosted downloader that fetches pre-converted Arrow logs directly into $PY123D_DATA_ROOT.
  • New SceneAPI.get_modality_between_timestamps() for time-windowed retrieval.
  • nuPlan route and lidar tokens exposed through the custom-modality interface.
  • Visualization: expanded viser color and rendering options, with configurable output resolution and frame rate.
  • Parser fixes for PAI-AV and ncore: bounding box alignment and file-derived vehicle dimensions.

No breaking changes to the public API, Arrow schema, or CLI entry points.

v0.3.0 (2026-04-28)
  • Refactored dataset download interface, with new download/stream options for nuScenes, PandaSet (HF mirror), AV2-sensor, WOD-perception, WOMD (#126), and nuPlan.
  • Added ncore dataset support with parser, downloader, and on-the-fly conversion (#125).
  • Waymo Open Motion: remaining WOMD splits, WOMD-specific fields via custom modalities, and skip-logs / skip-map options for storage-constrained runs.
  • Map improvements: speed bumps as surface map objects in Waymo (#130); align_road_edges_to_traffic for traffic-aligned road edges (#123).
  • Parser fixes across PandaSet (lidar/ego poses, extrinsics), nuPlan (#128), KITTI-360 labels, nuScenes map path in stream mode, WOD motion streaming, and pai-av / ncore labels.
  • Runtime and packaging: corrected sync-table entries when inferring dynamics; Ray executor compatibility; google-cloud-storage moved to the Waymo extra.

Includes all fixes from v0.2.1 and v0.2.2. No breaking changes to the public API, Arrow schema, or CLI entry points.

v0.2.0 (2026-04-14)

  • Transferred repository to KE:SAI.

  • Aligned ego and agent dynamics to a unified global/ego-frame convention, with velocity/acceleration inference in LogWriter from poses for EgoState and BoxDetectionsSE3 (#119, #120).

  • Improved OpenDRIVE maps: 3D road-edge lifting, lane-boundary reconstruction, and cleaner map-metadata location handling (#121).

  • Parser and visualization fixes: NuScenes interpolated parser defaults to 10 Hz sync with camera-pose interpolation; PandaSet extrinsic/undistortion fixes; viser fixes (#117); new matplotlib camera-rig and lidar-reprojection utilities.

No breaking changes to the public API, Arrow schema, or CLI entry points.

v0.1.0 (2026-03-22)
  • Asynchronous (native-rate) data storage: modalities are now written at their original capture rate, not just at the a frame-wise rate.
  • New parser architecture with BaseLogParser.iter_modalities_async for native-rate iteration alongside the existing synchronized path.
  • Added NVIDIA Physical AI AV dataset support (experimental).
  • Added standalone OpenDRIVE / CARLA map parser.
  • Refactored conversion/ module into parser/ with consistent naming across all dataset parsers.
  • Refactored Viser 3D viewer. Adds more control and dark mode.
  • Added LaneType, IntersectionType, StopZoneType to map data structure.
  • Replaced Waymo heavy dependencies with lightweight protobufs.
  • Various fixes to camera-to-global transforms across all datasets.
v0.0.9 (2026-02-09)
  • Added Waymo Open Motion Dataset support.
  • Replaced gpkg map implementation with Arrow-based format for improved performance.
  • Added sensor names and timestamps to camera and Lidar data across all datasets.
  • Added ego-to-camera transforms in static metadata.
  • Implemented geometry builders for PoseSE2/PoseSE3 from arbitrary rotation/translation representations.
  • Added support for loading merged point clouds in API.
  • Improved map querying speed and OpenDrive lane connectivity handling.
  • Added recommended conversion options to dataset YAML configuration files.
  • Fixed PandaSet static extrinsics and KITTI-360 timestamp handling.
  • Fixed memory issues when converting large datasets (e.g., nuPlan).
v0.0.8 (2025-11-21)
  • Release of package and documentation.
  • Demo data for tutorials.

📚 Citation

@article{Dauner2026ARXIV,
  title={123D: Unifying Multi-Modal Autonomous Driving Data at Scale},
  author={Dauner, Daniel and Charraut, Valentin and Berle, Bastian and Li, Tianyu and Nguyen, Long and Wang, Jiabao and Jing, Changhui and Igl, Maximilian and Caesar, Holger and Ivanovic, Boris and Geiger, Andreas and Chitta, Kashyap},
  journal={arXiv preprint arXiv:2605.08084},
  year={2026}
}

⚖️ License

123D is released under the Apache License 2.0.