Challenge 22 - HydroML-Verify: Verification of physical consistency and hydrological signatures in ML-based streamflow forecasts and observations #5
Description
Stream 2 - Machine Learning
Mentors
• Maliko Tanguy
• Gwyneth Matthews
• Kenza Tazi
• Maria Luisa Taccari
• Nikolaos Mastrantonas
Skill Required
• Python (scientific stack: numpy, pandas, xarray)
• Time series analysis
• Hydrology fundamentals (streamflow behaviour, hydrographs)
• Machine learning evaluation and diagnostics
• Data visualisation (matplotlib, seaborn, or similar)
• Writing clean, documented, reusable open-source code
Goal
To develop an open-source verification toolkit that automatically detects physically inconsistent and hydrologically implausible behaviour in machine-learning-based streamflow forecasts and in observational streamflow datasets, using physical constraints, hydrological signatures, and diagnostic visualisations.
Description of the Challenge
Current problem / limitation
Machine-learning-based (ML-based) streamflow forecasts are increasingly used in research and pre-operational contexts. While they can show good performance according to traditional verification metrics, they may still violate basic hydrological constraints, such as producing negative discharge values, unrealistic response times, implausible hydrograph shapes, or abnormal flow variability.
Current verification workflows focus primarily on skill scores and pointwise errors and do not systematically detect or characterise these physically inconsistent behaviours. As a result, such issues are difficult to track, compare across models, or diagnose during model development and evaluation.
Similar issues are also present in observational streamflow datasets, where data artefacts such as negative values, unrealistic jumps, or inconsistent temporal behaviour can occur. This means that many of the diagnostics developed for ML-based verification are equally relevant for identifying quality-control issues in observational data.
Data / system
The project will use existing ECMWF-hosted or publicly available datasets of ML-based streamflow forecasts, together with open-source observational streamflow datasets. The methods should be model-agnostic and applicable to both gridded and point-based discharge time series.
Proposed solution
The challenge is to design and implement a modular verification component, consisting of a set of checks and diagnostics, that complements traditional skill metrics by focusing on physical consistency and hydrological realism. This will include:
• Basic physical constraint checks (e.g. non-negativity, continuity, timing consistency)
• Detection of anomalous or implausible hydrographs using simple statistical diagnostics
• Comparison of forecasts against expected ranges of hydrological signatures, such as flow distribution, timing, variability, recession behaviour, and response characteristics
• Clear diagnostic plots and summary indicators to flag problematic behaviour and support interpretation
The emphasis is on robustness, interpretability, and ease of integration into existing verification workflows, rather than on modifying or retraining ML models. While the primary focus is on ML-based forecasts, selected diagnostics will also be applicable to observational datasets, providing added value for data quality assessment.
Ideas for implementation (indicative)
• Modular Python package with clearly defined physical checks and hydrological signature computations
• Configurable thresholds and reference ranges
• Summary tables and visual diagnostics at station or catchment level
• Example notebooks demonstrating usage on ML forecast outputs and observational data
Evaluation criteria
- Feasibility
- Transferability
- Easy to maintain / Future-proof approach
- Comprehensibility
- Matching requirements;