✨ Project Class

[joernweissenborn]

This PR adds a basic Project class.

The Project class is intended to serve as a standard API to pyglotaran.

Add the moment, using pyglotaran as a python package requires alot of imports from different sub packages. This ok because our current API is kind of a "super user" API. The nature of it is that it changes alot, which is not user friendly. On the other hand side, we now have a deprecation framework, which is very useful but slows down development.

The Project class mitigates this. It is designed as a user API, escpeicially with having GUIs and stuff in mind. It is meant to be a representation of the actual iterative process of global target analysis.

Details

The Project class as implemented in this PR is basically a kind of "API aggregator" plus a KISS principle based "database" called ProjectRegistry.

It is essentially a collection of a project file plus

• a model folder
• a parameters folder
• a data folder

This folders are represented by ProjectRegistry objects. They work by leveraging the io plugin system and pathlib.

A worwflow in looks like this:

from glotaran.project import Project

EXAMPLE_DIR = "my_project"
EXAMPLE_DATA = "testdata.nc"

Project.create(EXAMPLE_DIR) # create the project

gtap = Project.open(EXAMPLE_DIR) # open a project
gtap.import_data(EXAMPLE_DATA, name="dataset_1")

gtap.generate_model(
    generator_name="simple_model",
    generator="decay_sequential",
    generator_arguments={"nr_compartments": 3, "irf": True},
)

gtap.generate_parameters("simple_model", name="simple_params")

# Note: For now, you need to open the generated models in an editor and change the starting values.

gtap.optimize(
    model="simple_model",
    parameters="simple_params",
    maximum_number_function_evaluations=1,
    name="simple_result",
) # optimize the model parameters

result = gtap.load_result("simple_result") # load the result

result_path = gtap.get_result_path("simple_result") # convenience for e.g. plotting

Checklist

• ✔️ Passing the tests (mandatory for all PR's)
• 👌 Closes issue (mandatory for ✨ feature and 🩹 bug fix PR's)
• 🧪 Adds new tests for the feature (mandatory for ✨ feature and 🩹 bug fix PR's)
• 📚 Adds documentation of the feature

Closes issues

closes #XXXX

[github-actions]

👈 Launch a binder notebook on branch joernweissenborn/pyglotaran/feature/project/class

[sonarqubecloud]

SonarCloud Quality Gate failed.   

0 Bugs
0 Vulnerabilities
0 Security Hotspots
6 Code Smells

No Coverage information
4.5% Duplication

[codecov]

Codecov Report

Merging #869 (ea9a5d5) into main (b7921d4) will increase coverage by 0.8%.
The diff coverage is 94.1%.

@@           Coverage Diff           @@
##            main    #869     +/-   ##
=======================================
+ Coverage   86.0%   86.9%   +0.8%     
=======================================
  Files         92      99      +7     
  Lines       4981    5314    +333     
  Branches     939     980     +41     
=======================================
+ Hits        4287    4618    +331     
+ Misses       547     544      -3     
- Partials     147     152      +5     

Impacted Files	Coverage Δ
glotaran/builtin/io/folder/folder_plugin.py	100.0% <ø> (ø)
glotaran/project/result.py	90.9% <ø> (ø)
glotaran/utils/io.py	93.1% <40.0%> (-6.9%)	⬇️
glotaran/model/model.py	84.7% <58.0%> (-0.9%)	⬇️
glotaran/project/project_registry.py	94.5% <94.5%> (ø)
glotaran/project/project_parameter_registry.py	95.8% <95.8%> (ø)
glotaran/builtin/io/yml/utils.py	100.0% <100.0%> (ø)
glotaran/builtin/io/yml/yml.py	90.6% <100.0%> (+0.4%)	⬆️
glotaran/model/item.py	95.7% <100.0%> (+0.2%)	⬆️
glotaran/model/property.py	93.9% <100.0%> (+1.4%)	⬆️
... and 11 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b7921d4...ea9a5d5. Read the comment docs.

[github-actions]

Benchmark is done. Checkout the benchmark result page.
Benchmark differences below 5% might be due to CI noise.

Benchmark diff v0.5.1 vs. main

Parametrized benchmark signatures:

BenchmarkOptimize.time_optimize(index_dependent, grouped, weight)

All benchmarks:

       before           after         ratio
     [96b42630]       [ea9a5d51]
     <v0.5.1>                   
       60.6±0.5ms         60.1±1ms     0.99  BenchmarkOptimize.time_optimize(False, False, False)
       66.3±0.8ms        70.2±30ms     1.06  BenchmarkOptimize.time_optimize(False, False, True)
       61.7±0.5ms         60.9±1ms     0.99  BenchmarkOptimize.time_optimize(False, True, False)
         67.6±2ms         65.3±1ms     0.97  BenchmarkOptimize.time_optimize(False, True, True)
         75.1±2ms         73.5±2ms     0.98  BenchmarkOptimize.time_optimize(True, False, False)
        79.6±30ms        88.8±30ms    ~1.11  BenchmarkOptimize.time_optimize(True, False, True)
         75.4±2ms       73.6±0.6ms     0.98  BenchmarkOptimize.time_optimize(True, True, False)
         81.4±7ms        79.7±30ms     0.98  BenchmarkOptimize.time_optimize(True, True, True)
             205M             204M     0.99  IntegrationTwoDatasets.peakmem_optimize
       2.07±0.06s       1.92±0.06s     0.93  IntegrationTwoDatasets.time_optimize

Benchmark diff main vs. PR

Parametrized benchmark signatures:

BenchmarkOptimize.time_optimize(index_dependent, grouped, weight)

All benchmarks:

       before           after         ratio
     [b7921d43]       [ea9a5d51]
       61.5±0.4ms         60.1±1ms     0.98  BenchmarkOptimize.time_optimize(False, False, False)
        69.5±30ms        70.2±30ms     1.01  BenchmarkOptimize.time_optimize(False, False, True)
       61.1±0.6ms         60.9±1ms     1.00  BenchmarkOptimize.time_optimize(False, True, False)
        66.7±30ms         65.3±1ms     0.98  BenchmarkOptimize.time_optimize(False, True, True)
       73.6±0.3ms         73.5±2ms     1.00  BenchmarkOptimize.time_optimize(True, False, False)
        78.9±10ms        88.8±30ms    ~1.12  BenchmarkOptimize.time_optimize(True, False, True)
       74.1±0.6ms       73.6±0.6ms     0.99  BenchmarkOptimize.time_optimize(True, True, False)
         110±40ms        79.7±30ms    ~0.72  BenchmarkOptimize.time_optimize(True, True, True)
             203M             204M     1.00  IntegrationTwoDatasets.peakmem_optimize
       1.97±0.02s       1.92±0.06s     0.97  IntegrationTwoDatasets.time_optimize

[sonarqubecloud]

SonarCloud Quality Gate failed.   

0 Bugs
0 Vulnerabilities
0 Security Hotspots
1 Code Smell

No Coverage information
9.2% Duplication

[jsnel]

After rebasing, please sit together @s-weigand and @joernweissenborn and try to turn one or two of the examples into a project. e.g. this one

[s-weigand]

Took a while but I fixed all issues I still had with this PR (we will leave the test poisoning for a different PR).

@jsnel for a quick usability test in binder
You can use the following:

Save the dataset from the testing data to the file which gets imported

from glotaran.io import save_dataset
from glotaran.testing.simulated_data.sequential_spectral_decay import (
    DATASET as dummy_data,
)

save_dataset(dummy_data, "testdata.nc", allow_overwrite=True)

Create project skeleton

from glotaran.project import Project

EXAMPLE_DIR = "my_project"
EXAMPLE_DATA = "testdata.nc"

gtap = Project.open(EXAMPLE_DIR)  # open a project
gtap.import_data(EXAMPLE_DATA, name="dataset_1", ignore_existing=True)

gtap.generate_model(
    model_name="simple_model",
    generator_name="decay_sequential",
    generator_arguments={"nr_compartments": 3, "irf": True},
    ignore_existing=True,
)

gtap.generate_parameters("simple_model", parameters_name="simple_params", ignore_existing=True)

Edit the parameters

label,value,expression,minimum,maximum,non-negative,vary,standard-error
irf.center,0.3,None,,,False,True,None
irf.width,0.1,None,,,False,True,None
rates.species_1,0.5,None,,,False,True,None
rates.species_2,0.3,None,,,False,True,None
rates.species_3,0.1,None,,,False,True,None

Optimize the model

# Note: For now, you need to open the generated models in an editor and change the starting values.

gtap.optimize(
    model_name="simple_model",
    parameters_name="simple_params",
    maximum_number_function_evaluations=1,
    result_name="simple_result",
)  # optimize the model parameters

result = gtap.load_result("simple_result")  # load the result

result_path = gtap.get_result_path("simple_result")  # convenience for e.g. plotting

@joernweissenborn I also fixed some naming consistency issues please also check my changes.
Since the kwargs changed maybe you also want to update your instructions in the PR text.

[joernweissenborn]

lgtm, if the tests are working again

[sonarqubecloud]

Kudos, SonarCloud Quality Gate passed!   

0 Bugs
0 Vulnerabilities
0 Security Hotspots
2 Code Smells

No Coverage information
0.0% Duplication

[s-weigand]

Should be fine to merge now