draft exporter for animal cohort data

[TaranRallings]

Description

This is the full version of the animal cohort data exporter. It follows the structure of the plant community exporter. I have reworked it quite a bit from the original draft version and now it actually has tests.

Basically what this is is an on/off switch to dump out the data of every animal cohort at every timestep into a csv. Which attributes of the cohorts you want outputted is configurable. I have run the ve with it and checked that it does, in fact, produce the desired csv file.

Fixes # (issue)

Type of change

• New feature (non-breaking change which adds functionality)
• Optimization (back-end change that speeds up the code)
• Bug fix (non-breaking change which fixes an issue)

Key checklist

• Make sure you've run the pre-commit checks: $ pre-commit run -a
• All tests pass: $ poetry run pytest

Further checks

• Code is commented, particularly in hard-to-understand areas
• Tests added that prove fix is effective or that feature works
• Relevant documentation reviewed and updated

[codecov-commenter]

Codecov Report

❌ Patch coverage is 96.22642% with 4 lines in your changes missing coverage. Please review.
✅ Project coverage is 94.38%. Comparing base (8e07f18) to head (80e8907).

Files with missing lines	Patch %	Lines
virtual_ecosystem/models/animal/exporter.py	95.69%	4 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##           develop    #1056      +/-   ##
===========================================
+ Coverage    94.36%   94.38%   +0.02%     
===========================================
  Files           70       71       +1     
  Lines         7058     7162     +104     
===========================================
+ Hits          6660     6760     +100     
- Misses         398      402       +4     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[davidorme]

LGTM.

You maybe don't need to have _dump_cohort_data as a separate thing. I do in plants because I've got more than one output to write, and dump would be huge. But it does no harm - arguably more expandable in future - although I don't know what else you'd write?

Oh - hang on. You could want to write the actual territory cells? That would be a separate file because different number of lines per cohort. I'm not saying you should do that now, but argues to keep the separate method to stick in a new one for _dump_ranges.

[TaranRallings]

LGTM.

Ok awesome. In an effort to get as us ready as possible for borneo, I'm going to park this until I'm finished with the plant piping, then I'll get the testing worked out.

[nickwctan]

Looks good to me. As of now, I assume the exporter will dump all the output in a single CSV file containing spatial (cell_ids), temporal (timestamps) variables and subsequent subsets of cohort attributes (as the user specifies - which relates to my next comment). How similar would it look to the dataframe I whipped out below?

cell_id	time	cohort_id	functional_group
1	timestamp1	cam1	carnivorous_arboreal_mammal
2	timestamp1	htm1	herbivorous_terrestrial_mammal
2	timestamp1	htm2	herbivorous_terrestrial_mammal
1	timestamp2	aa1	arboreal_amphibian
2	timestamp2	htb1	herbivorous_terrestrial_beetle
2	timestamp2	htb2	herbivorous_terrestrial_beetle

[nickwctan]

A next question from my end, as a user (who looks foward to seeing some output data), how and where would I specify the subset of attributes I would like to the exporter to dump? Could we add it in a .toml file in /config folder?

The only place I see currently where it can be specified is in model_config.py line 404-406.

    enabled: bool = False
    cohort_attributes: tuple[str, ...] = ()
    float_format: str = "%0.5f"

I understand if running the example_data might not be possible currently as the config has just changed but I would be curious to know the output data pipeline. Thanks for including me in this PR.

[TaranRallings]

Looks good to me. As of now, I assume the exporter will dump all the output in a single CSV file containing spatial (cell_ids), temporal (timestamps) variables and subsequent subsets of cohort attributes (as the user specifies - which relates to my next comment). How similar would it look to the dataframe I whipped out below?
cell_id time cohort_id functional_group
1 timestamp1 cam1 carnivorous_arboreal_mammal
2 timestamp1 htm1 herbivorous_terrestrial_mammal
2 timestamp1 htm2 herbivorous_terrestrial_mammal
1 timestamp2 aa1 arboreal_amphibian
2 timestamp2 htb1 herbivorous_terrestrial_beetle
2 timestamp2 htb2 herbivorous_terrestrial_beetle

Yes, the output looks very much like this. I will send you an example output file and show you how it all works in our meeting tomorrow.

[TaranRallings]

A next question from my end, as a user (who looks foward to seeing some output data), how and where would I specify the subset of attributes I would like to the exporter to dump? Could we add it in a .toml file in /config folder?

The only place I see currently where it can be specified is in model_config.py line 404-406.

    enabled: bool = False
    cohort_attributes: tuple[str, ...] = ()
    float_format: str = "%0.5f"

I understand if running the example_data might not be possible currently as the config has just changed but I would be curious to know the output data pipeline. Thanks for including me in this PR.

Right now, the easiest way is through example_data/config/animal_config.toml. When you generate the example data through ve_run --install-example /tmp/ you can go into the config there and fiddle with which attributes you want before actually running the simulation.

[davidorme]

This looks fine.

The only thing I question is that the animal model testing uses a lot of "no-op" and "minimal example" objects that are intended to duplicate the real objects. These definitely can be useful for computationally expensive tasks but they also make the code more bloaty and add an extra possible layer of failure. Particularly with the dummy exporter (which does nothing by default) it seems unnecessary, and it also seems like running the exporter testing on a real model with actual cohorts is more direct. We have minimal model for testing anyway - just run the "update" test setup and then export?

[TaranRallings]

This looks fine.

The only thing I question is that the animal model testing uses a lot of "no-op" and "minimal example" objects that are intended to duplicate the real objects. These definitely can be useful for computationally expensive tasks but they also make the code more bloaty and add an extra possible layer of failure. Particularly with the dummy exporter (which does nothing by default) it seems unnecessary, and it also seems like running the exporter testing on a real model with actual cohorts is more direct. We have minimal model for testing anyway - just run the "update" test setup and then export?

Good point! I revised the testing.