Link to test images?

[kristinbranson]

Thank you for creating your datasets. I am trying to follow the instructions listed here:
"Your model should export predictions for all images in the test datasets. Package these predictions in a format of your choice."
https://benchmark.deeplabcut.org/submission.html
As I understand it, the data set can be downloaded here:
Tri-Mouse https://zenodo.org/record/5851157#.YeHC23vMJhE
only contains the training images. I understand that the annotations for the test data are not available, but I cannot find the images for them. I see 112 images trimice-dlc-2021-06-22/labeled_data/*/*png and 112 labels in the training data, so I'm assuming there are some test images somewhere?

Thanks again,
Kristin

[MMathisLab]

Hi @kristinbranson ! Sorry for the delay, I missed your post; @AlexEMG can chip in here, but that link includes the test data - the splits are noted in dlc_shuffle1_train.json or just another format in Documentation_data-trimice_70shuffle1.pickle (under training-datasets)!

[SkepticRaven]

That json only has 336 annotations present (3*112) and don't appear to contain any information about train/test splits (I'm assuming based on the filename this is the train split). The remaining 49 frames as listed on the dataset description page are not present. I even tried importing the dataset into both DLC and SLEAP -- they both claim 112 frames.

This appears to be the same case for other datasets (eg parenting-mouse only has 379 of 542 frames/annotations present).

[MMathisLab]

That's correct -- you need to submit results via our benchmark API to test

[SkepticRaven]

This still doesn't answer where the test images are to make predictions on.

@AlexEMG can chip in here, but that link includes the test data - the splits are noted in dlc_shuffle1_train.json

That link does not contain the test data. There are only 112 png files in the link and I confirmed in the my last post that the json labels all of them. The remaining 49 png files are not present.

[MMathisLab]

@stes @AlexEMG can you comment please?

[AlexEMG]

I'm really sorry for this, we will resolve this asap!!

[stes]

Hello @kristinbranson,

sorry for the inconvenience and the slow response from my end. We double checked, and can confirm that the zenodo link indeed only contains the train images, which is of course not intended. We decided to keep the existing zenodo records unmodified (for training and validation), and added new records to share the test images for every dataset.

These will be added to the website, I'll open a PR.

For your convenience, here are the links directly for the various datasets:

• Mouse: https://zenodo.org/record/8437103
• Parenting: https://zenodo.org/record/8437111
• Fish: https://zenodo.org/record/8437115
• Marmosets: https://zenodo.org/record/8437121

Based on this, you can generate predictions in any suitable format. For compatibility with the benchmark, please then follow the instructions on https://benchmark.deeplabcut.org/submission.html to package your prediction files (CSV, H5, Pickle etc.) and return them to our API. You submit this as a PR to this repo, and your submission will then appear in the public leaderboard.

Let me know if you have any questions.