Skip to content

Evaluation Code for Scenario Mining Challenge Release#295

Merged
benjaminrwilson merged 22 commits intoargoverse:mainfrom
CainanD:main
Mar 12, 2025
Merged

Evaluation Code for Scenario Mining Challenge Release#295
benjaminrwilson merged 22 commits intoargoverse:mainfrom
CainanD:main

Conversation

@CainanD
Copy link
Copy Markdown
Contributor

@CainanD CainanD commented Mar 10, 2025

PR Summary

Added scenario mining evaluation code in preparation for the new challenge release. Since the tracking is the primary evaluation metric, overall code changes are pretty minimal. Some additional logic was added to evaluate how the tracker performs at classifying time steps. This requires no additional dependencies. Included in this pull request are two small .pkl files that contain the predicted and ground truth track data to test the evaluation function.

Testing

This pull request includes a unit test and sample data files for the added evaluation script. The evaluation script works as intended.

In order to ensure this PR works as intended, it is:

  • [✅] unit tested.
  • other or not applicable (additional detail/rationale required)

Compliance with Standards

As the author, I certify that this PR conforms to the following standards:

  • [✅] Code changes conform to PEP8 and docstrings conform to the Google Python style guide.
  • [✅] A well-written summary explains what was done and why it was done.
  • [✅] The PR is adequately tested and the testing details and links to external results are included.

Testing details showing complete coverage of av2/evaluation/scenario_mining/eval.py:
logs_35428725033.zip
Visualization of ground truth data:
https://github.com/user-attachments/assets/a0ef5497-7672-4365-ae95-e37e5597d975

Copy link
Copy Markdown
Collaborator

@benjaminrwilson benjaminrwilson left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, @CainanD! Looks good. Left a few comments. Also, was map_tutorial.ipynb meant to change?

@benjaminrwilson
Copy link
Copy Markdown
Collaborator

Awesome, looks good. Thank you for making the changes, @CainanD. I still see a diff on the map_tutorial.ipynb - let's reset that unless that's intended, otherwise I can go ahead and merge!

@benjaminrwilson benjaminrwilson merged commit af47ea5 into argoverse:main Mar 12, 2025
54 of 55 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants