Implement sensitivity analysis preparation

[dalonsoa]

VR depends on many parameters that, in isolation or collectively, define the evolution of the simulation. A sensitivity analysis will help to identify which parameters have more impact in the simulation, which ones result in small changes when vary, and which might be correlated with each other.

This umbrella issue implements the necessary tooling and configuration files to create a collection of input files that can be run massively in parallel at Imperial's HPC using array jobs.

• Define the format and contents of the sensitivity analysis (SA) input file #240
• Create a python script produces a range of VR input files out of the SA file and a template input #241
• Facilitate producing a pbs submission script for Imperial's HPC #242

[davidorme]

I think the sensitivity analysis tool is likely to be widely used by end users and is going to be an important component of many of the end user stories we anticipate for vr. Not all of those end users are going to have access to an HPC (or alternatively the specific flavour of HPC we use, although differences there are likely cosmetic). I think we want to support SA runs on a personal computer (ideally allowing parallel runs on different cores) so I think we also need to provide an approach to batch run and compile SA runs on a local machine.

I think that also alters how we approach configuring the runs? If we're using array jobs on the HPC, then having a potentially large folder of config files and then just chunking them into array job sized blocks and then looping over a set of inputs to vr_run makes sense. If this is something running on a personal computer, I worry about the sea of largely identical TOML files - which could get very large if more than one parameter is varying and we need combinations.

Could we have a SensitivityAnalysisManager, with methods to:

• ingest the SA input file and work out a list of different permutations of the varying parameters.
• outputs the permutation list as TOML, optionally divided into n chunks.

Then we have a SensitivityAnalysisRunner, which takes a broader config file and a (possibly divided) permutation list and runs vr, overwriting the broader config with each setting in the permutation list in turn.

This still works with the HPC - we can divide into job array chunks and then parallelise those chunks, but it also provides a framework to run locally. We could potentially then add a parallelise method to SensitivityAnalysisRunner which passes jobs to different cores?

[dalonsoa]

The actual implementation is something we will need to consider, but I agree that running a SA in the perfectly capable, multicore computers we all have is a perfectly sensible use case. We can certainly implement something that can serve both of them.

About the the sea of largely identical TOML files, I feel that's the least of your problems: if you are running a sensitivity analysis with many permutations in your laptop, the killing factor is going to be the collection of output data that will be produced. The TOML files can be discarded after using them, as you suggest, but the output files will need to stay, and that might represent a barrier.

[davidorme]

the killing factor is going to be the collection of output data that will be produced

Yes - that's certainly a problem. I think it is conceivable that many or even most sensitivity analyses will look at effects on particular output variables, so might have slimmed down output variable lists but that isn't generally true.

[alexdewar]

So as you've probably seen, @dalonsoa has assigned these issues to me as I'll be the one working on this.

I've just spent the last few hours playing around with Snakemake and I think it could be really useful here. It's a Python-based tool which uses makefile-like config files to define data processing pipelines and then execute them in parallel. You can run it locally (on multiple cores) or on a number of backends, one of which provides a generic interface for running jobs in parallel on a cluster. I've managed to get a toy example working on Imperial's HPC, so it certainly seems possible, at least in principle.

From my (admittedly limited) understanding, it seems like we might be able to define the parameter space directly in a Snakefile without having to come up with our own format for doing this (e.g. based on TOML, see #240). It would also make #242 unnecessary as the jobscript generation would be done by Snakemake (though there will be some Imperial-specific quirks we have to account for somewhere in the config).

Anyone have any thoughts?

[dalonsoa]

I didn't thought about it back when I wrote these issues, but I agree with @alexdewar in that Snakemake - or similar pipeline tooling - is precisely the sort of thing made to solve this issues. So, unless you have some specific concerns about that, I would go for it.

Alex, how complex is to get it installed locally?

[alexdewar]

It's not bad at all. You guys are using Python anyway, so you should just be able to add it as a dependency to your pyproject.toml (or install with conda/pip etc.).

[davidorme]

I think snakemake does look interesting but - at a cursory glance - suits some parts of the SA process more than others.

The (very large) bit that I think would work well is the ability to execute multiple vr_run on different platforms and chain those with subsequent steps. We could, for example, provide a directory of sensitivity_analysis_scripts that would extract particular measures of interest and then have the snakemake file contain rules to run vr and then run a particular summary script on the outputs. Being able to parallelise that locally or on an HPC is very attractive.

The bit that seems slightly trickier is how to create the input files that vary the parameters of interest. snakemake does have a params namespace and config files that can be used to populate the params with sets of values. The things that are tricky here is that for SA we need to adjust vr model configs, which have complex nested namespaces, and we probably want to be able to provide (start, stop, step) style values, rather than having to type an exhaustive list into a config file. Those values then need to be injected into the different vr configs.

So - I think snakemake looks like a powerful tool for running and summarizing vr runs, but it isn't clear whether it is well suited to configuring vr.

[davidorme]

I can imagine something within vr that is a config generator? So, something that would take an existing fixed config (generated using the normal process) and that is then fed an additional TOML file that stipulates one or more configuration options with ranges of values:

[plant_model.config.PlantConfig]
option_a = [1,2,3,4,5,6]
option_b.start = 1
option_b.stop = 6
option_b.step = 1

And then we could do cfg_gen = ConfigGenerator(fixed_config, path_to_toml). That could be used to write out all possible combinations to create a set of input files (although that seems clumsy) or possibly iterate over the configs. I guess - if we could ensure the order of the resulting configs - we could also make it indexable so then different sections of the same generator could be executed on different nodes.

[alexdewar]

I think snakemake does look interesting but - at a cursory glance - suits some parts of the SA process more than others.

The (very large) bit that I think would work well is the ability to execute multiple vr_run on different platforms and chain those with subsequent steps. We could, for example, provide a directory of sensitivity_analysis_scripts that would extract particular measures of interest and then have the snakemake file contain rules to run vr and then run a particular summary script on the outputs. Being able to parallelise that locally or on an HPC is very attractive.

Totally agree. This is the feature that I think is most useful to us. As a bonus, it would let users run VR in parallel with another executor, e.g. a cloud service or their own HPC cluster. They might have to do some (hopefully minor) configuring of snakemake to make it work with their own HPC, but I think it's much better to have this layer of abstraction, because trying to write this code ourselves in a suitably generic fashion will be extremely difficult.

The other difficulty with working with HPC systems is resource management, i.e. how many CPUs and how much memory you request etc. and it seems like snakemake could help us here too (though, again, we would likely need a shim to handle the Imperial HPC's quirks).

The bit that seems slightly trickier is how to create the input files that vary the parameters of interest. snakemake does have a params namespace and config files that can be used to populate the params with sets of values. The things that are tricky here is that for SA we need to adjust vr model configs, which have complex nested namespaces, and we probably want to be able to provide (start, stop, step) style values, rather than having to type an exhaustive list into a config file. Those values then need to be injected into the different vr configs.

So - I think snakemake looks like a powerful tool for running and summarizing vr runs, but it isn't clear whether it is well suited to configuring vr.

I'm a bit more optimistic than you here here -- my guess is that we probably could find a way to define the parameter space in a Snakefile -- but I think we'll only know for sure once we try it. In the worst case, if snakemake turns out to be useless for figuring out what the input parameters should be, then we can always just use it in the dumbest way possible: we generate a bunch of input files with our own tool and use snakemake to run VR and produce the corresponding output files. The benefits of using it for job control still make it worthwhile, IMO.

That said, snakemake does seem to be quite a flexible tool. You can include snippets of Python directly in your Snakefile, which means you can express parameters as e.g. range(0, 10, 2) etc. And there's no reason you can't just include a line like like this in your Snakefile:

from virtual_rainforest import AFancyParameterSpaceGenerator

with AFancyParameterSpaceGenerator doing whatever you want. With some kind of wrangling like this, I feel like we might be able to just invoke VR directly with the desired params without having to generate a bunch of intermediate input files, though if it's not up to scratch, we could do something like you suggest above.

As I say though, I'm not 100% certain that this will work, or work easily, but I think it's worth pursuing because of the flexibility that the executor part of it gives us.

So in summary, my current plan is:

1. Use snakemake to invoke VR in parallel on either the HPC or a local machine
2. Try to see if it can also be used to configure these simulations directly

If no. 2 doesn't come off, then we can just write our own scripts to do that bit.

Sound sensible?

[davidorme]

Sounds sensible to me.