Revamp the existing "Using the VE" documentation

[davidorme]

Description

This PR is tidying up the existing "Using the VE" documentation. The main changes are:

• I've created an overview of the main pages as a landing page for using the VE rather than diving straight into installation.

• I've separated the structure out a bit so that we now have specific pages for "Installing VE" and then "The example data".

• The example data variable tables were wildly out of date. I've moved to a system where each netCDF file has dataset description and then variable descriptions and units in the file attributes. That's good practice for NetCDF files anyway but additionally the example data page now has a function to automatically create the variable tables from the files, so everything stays updated automatically as long as the files are annotated correctly.

  The example data page now basically avoids showing or mentioning the data generation scripts except in passing at the end. I think the file contents are the key thing and those generation scripts are distracting.

Important

This bit isn't complete - some of the NetCDF files need to have units and descriptions added to get the information into the tables. Each file needs to have a dataset_description attribute and then each variable needs units and description attributes. The code below shows these being set in the climate data generation file.

virtual_ecosystem/virtual_ecosystem/example_data/generation_scripts/climate_example_data.py

Lines 161 to 188 in f90ec84

	# Add attributes
	dataset_xyt.attrs["dataset_description"] = """The dummy climate data for the example
	simulation provides reference data for the climatic conditions above the canopy for the
	time steps in the model, along with climatological data on the mean annual temperature.
	"""
	dataset_xyt.air_temperature_ref.attrs = dict(
	units="°C", description="Air temperature above canopy"
	)
	dataset_xyt.relative_humidity_ref.attrs = dict(
	units="unitless", description="Relative humidity above canopy"
	)
	dataset_xyt.precipitation.attrs = dict(
	units="mm month", description="Total monthly precipitation"
	)
	dataset_xyt.atmospheric_pressure_ref.attrs = dict(
	units="kPa", description="Atmospheric pressure above canopy"
	)
	dataset_xyt.atmospheric_co2_ref.attrs = dict(
	units="ppm", description="Atmospheric CO2 concentration"
	)
	dataset_xyt.wind_speed_ref.attrs = dict(
	units="???", description="Wind speed above the canopy"
	)
	dataset_xyt.mean_annual_temperature.attrs = dict(
	units="°C", description="Mean annual temperature"
	)

• The Using the VE page has changed quite a bit:
  • It now points to those previous pages so avoids repeating stuff about the example data structure
  • It conceals basically all of the bash stuff - it shows command syntax but the execution of the model runs on code that is entirely hidden.
  • I've fixed some display glitches, although for some goddam reason there are still what I think are buffering issues with showing the ve_run command help.
  • I've moved this single page over into an ipynb file to capture the outputs and hopefully allow it to be built to html using the existing outputs.

I've built the docs - this is the key section:
https://virtual-ecosystem.readthedocs.io/en/1050-revamp-the-existing-using-the-ve-documentation/using_the_ve/overview.html

Fixes #1050

Type of change

• New feature (non-breaking change which adds functionality)
• Optimization (back-end change that speeds up the code)
• Bug fix (non-breaking change which fixes an issue)

Key checklist

• Make sure you've run the pre-commit checks: $ pre-commit run -a
• All tests pass: $ poetry run pytest

Further checks

• Code is commented, particularly in hard-to-understand areas
• Tests added that prove fix is effective or that feature works
• Relevant documentation reviewed and updated

[codecov-commenter]

Codecov Report

❌ Patch coverage is 0% with 27 lines in your changes missing coverage. Please review.
✅ Project coverage is 95.38%. Comparing base (c1a73b9) to head (4688172).

Files with missing lines	Patch %	Lines
...le_data/generation_scripts/climate_example_data.py	0.00%	23 Missing ⚠️
..._data/generation_scripts/elevation_example_data.py	0.00%	4 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##           develop    #1055      +/-   ##
===========================================
+ Coverage    95.20%   95.38%   +0.18%     
===========================================
  Files           81       79       -2     
  Lines         6889     6876      -13     
===========================================
  Hits          6559     6559              
+ Misses         330      317      -13     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[arne-exe]

Well done @davidorme, the structure is much clearer and it's a lot easier to understand how different files of the VE link together.

[vgro]

Suggested change

	not need a time axis, but other forcing variables (like temperature and
	not need a time axis, but other forcing variables (like reference temperature and

[vgro]

I agree with @arne-exe , it looks much clearer now. I have made some abiotic/hydrology specific changes in #1064 and wait with approving until the other parts are complete

[annarallings]

@davidorme this is a really fantastic update - thank you for doing this! @Baizurah @nickwctan @lelavathy @hrlai I definitely recommend you read through it, though I know you're quite familiar with the model at this point. This is very nicely and logically laid out. It would be worthwhile to get your feedback on clarity.

[vgro]

Happy for you to merge once the stalled tests pass

[davidorme]

@davidorme this is a really fantastic update - thank you for doing this! @Baizurah @nickwctan @lelavathy @hrlai I definitely recommend you read through it, though I know you're quite familiar with the model at this point. This is very nicely and logically laid out. It would be worthwhile to get your feedback on clarity.

Thanks @annarallings.

This has been reviewed by you, @arne-exe and @vgro, so we're going to merge this down into develop.

@Baizurah @nickwctan @lelavathy @hrlai - that doesn't mean this is the final version - just that we want to bank what we have and not have the PR sitting around getting out of sync! Please do read the new section and let us know if there are any other things to improve.

[Baizurah]

Great work on this PR 🙌 Really nice to see the clearer flow, which is super helpful. I’ll need to go through this again since there’s so much more information here compared to the previous version. I agree on the idea of documenting NetCDF vs CSV split, which has been mentioned in other issues/discussion. It will be helpful to know which inputs belong to NetCDF (assuming this will be for spatial/gridded ie climate) vs CSV (categorical/functional definitions ie: animal traits).

Might be worth considering:

-As a visual person, I think adding a simple folder layout/structure diagram would go a long way where it is relevant, for example, folder structure/showing how NetCDF vs CSV files fit together.
-As a new user, I will also be interested to know on pre-processing/requirements needed before I can use VE, if that makes sense.
-A short troubleshooting section for common setup issues (missing netCDF4, Poetry/venv problems, file path mix-ups, etc.). At the moment the docs assume everything installs smoothly. A quick checklist could be really handy for new users. I know that first-hand :D

I will give the doc deeper read and be back if anything relevant.
As a fresh pair of eyes, I can still comment here even if it’s closed, right? @davidorme

[davidorme]

I will give the doc deeper read and be back if anything relevant. As a fresh pair of eyes, I can still comment here even if it’s closed, right? @davidorme

You absolutely can - although closed issues get less attention, so tagging people is useful to get eyes on your comment. The other thing you can do is to add new issues to this repo to suggest the changes you'd like, in this case ideally as new sub-issues of the doc revamp meta issue (#1035) - that gives us specific things to action and they don't go away until we've closed them.

-As a visual person, I think adding a simple folder layout/structure diagram would go a long way where it is relevant, for example, folder structure/showing how NetCDF vs CSV files fit together.

This is actually less important than the example data makes it look. All of the data files are loaded from paths that are set in the configuration, so they could be organised in any structure. Probably better to have them organised, but the configured paths are the key bit!

-As a new user, I will also be interested to know on pre-processing/requirements needed before I can use VE, if that makes sense.
-A short troubleshooting section for common setup issues (missing netCDF4, Poetry/venv problems, file path mix-ups, etc.). At the moment the docs assume everything installs smoothly. A quick checklist could be really handy for new users. I know that first-hand :D

Yup - that would be new pages on preparing data files and configuration, probably with R and Python subpages.

[hrlai]

Thanks for doing this.

👍🏼 Reading flow is much better
👍🏼 Appreciate tips on virtual environment for python beginners like me
👍🏼 Cleaner example data in a table view, with units!
💭 We will continue to use the Model inputs page for the input data projects and feedback
😆 Appreciate the poetic opening of the static model, will feedback once the above is kicked off