Skip to content

Bug: Score of multiple models fails if observations are not in each comparer#526

Merged
ecomodeller merged 13 commits intomainfrom
fix/score_multiple_models
Oct 6, 2025
Merged

Bug: Score of multiple models fails if observations are not in each comparer#526
ecomodeller merged 13 commits intomainfrom
fix/score_multiple_models

Conversation

@ecomodeller
Copy link
Copy Markdown
Member

@ecomodeller ecomodeller commented Sep 23, 2025

import modelskill as ms
import pandas as pd

time = pd.date_range("2000", periods=2)
cmp1 = ms.match(
    obs=ms.PointObservation(pd.Series([0, 0], index=time), name="foo"),
    mod=ms.PointModelResult(pd.Series([0, 0], index=time), name="bar"),
)
cmp2 = ms.match(
    obs=ms.PointObservation(pd.Series([0, 0], index=time), name="baz"),
    mod=ms.PointModelResult(pd.Series([0, 0], index=time), name="qux"),
)
# TODO should it be allowed to combine two comparers with different models, maybe not?
cc = ms.ComparerCollection([cmp1, cmp2])
cc.skill(metrics="rmse")
image

So far so good, but if I call

cc.score() 😟

it fails with a cryptical message since the weights vector is of the wrong length, if each comparers doesn't contain each observation.

        if len(data) != len(index):
>           raise ValueError(
                "Length of values "
                f"({len(data)}) "
                "does not match length of index "
                f"({len(index)})"
            )
E           ValueError: Length of values (4) does not match length of index (2)

This PR fixes this problem, and does some more refactoring related to weighted scoring.

@ecomodeller ecomodeller marked this pull request as ready for review September 23, 2025 20:55
@ecomodeller ecomodeller changed the title Fix score of multiple models Bug: Score of multiple models fails if observations are not in each comparer Sep 24, 2025
Copy link
Copy Markdown
Collaborator

@ryan-kipawa ryan-kipawa left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good :)

I assume "should it be allowed to combine two comparers with different models, maybe not?" is just a general comment outside the scope of the PR.

@ecomodeller
Copy link
Copy Markdown
Member Author

Looks good :)

I assume "should it be allowed to combine two comparers with different models, maybe not?" is just a general comment outside the scope of the PR.

Indeed, thinking out "loud"...

@ecomodeller ecomodeller merged commit 7ea4e4f into main Oct 6, 2025
3 checks passed
@ecomodeller ecomodeller deleted the fix/score_multiple_models branch October 6, 2025 11:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants