Eval breaks for fine-tuning data

[clessig]

What happened?

Eval breaks with config below:

lessig@nid005239:~/santis/WeatherGenerator> uv run evaluate --config ./config/config_eval.yml 
/users/lessig/santis/WeatherGenerator/.venv/lib/python3.12/site-packages/mlflow/__init__.py:41: UserWarning: Versions of mlflow (3.9.0) and child packages mlflow-skinny (3.5.0) are different. This may lead to unexpected behavior. Please install the same version of all MLflow packages.
  mlflow.mismatch._check_version_mismatch()
Logging set up. No log files created.
Detected 1 runs
Using main process for evaluation
Loading config for run lelosmh6 from model directory: None
Loading private config from platform-env.py: /users/lessig/santis/WeatherGenerator-private/hpc/platform-env.py.
Detected HPC: santis.
Loading private config from platform-env.py output: /users/lessig/santis/WeatherGenerator-private/hpc/santis/config/paths.yml.
Loading config from specified run_id and mini_epoch: /capstor/store/cscs/userlab/ch17/shared_work/models/lelosmh6/model_lelosmh6_chkpt00000.json
Results directory obtained from private config: /capstor/store/cscs/userlab/ch17/shared_work/results/lelosmh6
2026-02-23 20:08:43,445 [MainProcess] INFO: Loading config for run lelosmh6 from model directory: None
2026-02-23 20:08:43,446 [MainProcess] INFO: Loading config from specified run_id and mini_epoch: /capstor/store/cscs/userlab/ch17/shared_work/models/lelosmh6/model_lelosmh6_chkpt00000.json
2026-02-23 20:08:43,508 [MainProcess] INFO: Results directory obtained from private config: /capstor/store/cscs/userlab/ch17/shared_work/results/lelosmh6
2026-02-23 20:08:43,508 [MainProcess] INFO: Opening zipstore, read-only: True
2026-02-23 20:08:43,513 [MainProcess] INFO: Opening zipstore, read-only: True
2026-02-23 20:08:43,516 [MainProcess] INFO: Taking cartopy paths from /capstor/store/cscs/userlab/ch17/shared_work/assets/cartopy
2026-02-23 20:08:43,516 [MainProcess] INFO: Opening zipstore, read-only: True
2026-02-23 20:08:43,520 [MainProcess] INFO: Opening zipstore, read-only: True
2026-02-23 20:08:43,533 [MainProcess] INFO: Opening zipstore, read-only: True
2026-02-23 20:08:43,535 [MainProcess] INFO: Opening zipstore, read-only: True
2026-02-23 20:08:43,957 [MainProcess] INFO: Requested sample(s) {2} is unavailable. Removing missing sample(s) for plotting.
2026-02-23 20:08:43,957 [MainProcess] INFO: Opening zipstore, read-only: True
2026-02-23 20:08:43,959 [MainProcess] INFO: Opening zipstore, read-only: True
2026-02-23 20:08:43,962 [MainProcess] INFO: RUN lelosmh6: Processing stream METOPB_IASI...
2026-02-23 20:08:43,962 [MainProcess] INFO: RUN lelosmh6 - METOPB_IASI: Processing fstep 0...
Processing lelosmh6 - METOPB_IASI - 0: 100%|██████████████████████████████████| 1/1 [00:00<00:00,  1.93it/s]
2026-02-23 20:08:44,481 [MainProcess] INFO: Opening zipstore, read-only: True
Plotting lelosmh6 - METOPB_IASI - fstep 0:   0%|                                      | 0/1 [00:00<?, ?it/s]2026-02-23 20:08:55,285 [MainProcess] INFO: Creating maps for all valid times of obsvalue_avhrr_mean_vis_0 - targets
Plotting lelosmh6 - METOPB_IASI - fstep 0:   0%|                                      | 0/1 [00:00<?, ?it/s]
Traceback (most recent call last):
  File "/users/lessig/santis/WeatherGenerator/.venv/bin/evaluate", line 10, in <module>
    sys.exit(evaluate())
             ^^^^^^^^^^
  File "/users/lessig/santis/WeatherGenerator/packages/evaluate/src/weathergen/evaluate/run_evaluation.py", line 105, in evaluate
    evaluate_from_args(sys.argv[1:], log_queue)
  File "/users/lessig/santis/WeatherGenerator/packages/evaluate/src/weathergen/evaluate/run_evaluation.py", line 157, in evaluate_from_args
    evaluate_from_config(cf, mlflow_client, log_queue)
  File "/users/lessig/santis/WeatherGenerator/packages/evaluate/src/weathergen/evaluate/run_evaluation.py", line 338, in evaluate_from_config
    results = [_process_stream(**task) for task in tasks]
               ^^^^^^^^^^^^^^^^^^^^^^^
  File "/users/lessig/santis/WeatherGenerator/packages/evaluate/src/weathergen/evaluate/run_evaluation.py", line 234, in _process_stream
    plot_data(reader, stream, global_plotting_opts)
  File "/users/lessig/santis/WeatherGenerator/packages/evaluate/src/weathergen/evaluate/utils/utils.py", line 489, in plot_data
    plotter.create_maps_per_sample(
  File "/users/lessig/santis/WeatherGenerator/packages/evaluate/src/weathergen/evaluate/plotting/plotter.py", line 406, in create_maps_per_sample
    f"({format_datetime(valid_time)})"
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/users/lessig/santis/WeatherGenerator/packages/evaluate/src/weathergen/evaluate/plotting/plotter.py", line 1947, in format_datetime
    return dt.astype("datetime64[m]").astype(datetime.datetime).strftime("%Y-%m-%d T%H:%M:%S")
           ^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'astype'

What are the steps to reproduce the bug?

Eval config:

verbose: true
results_dir : "./results/"
output_plotting_dir: "./plots/"
output_scores_dir : "./jsons/"
image_format : "png" #options: "png", "pdf", "svg", "eps", "jpg" ..
dpi_val : 300
summary_plots : true
print_summary: false

evaluation:
  metrics  : ["rmse"]
  regions: ["global", "nhem", "tropics", "shem"]

run_ids :

  lelosmh6 :
    label: "O96, reference"
    epoch: 0
    rank: 0
    streams:
      METOPB_IASI:
        evaluation:
          forecast_step: "all"
          sample: "all"
        plotting:
          sample: [1, 2]
          forecast_step: [0] 
          plot_maps: true
          plot_histograms: false 

Hedgedoc link to logs and more information. This ticket is public, do not attach files directly.

No response

[clessig]

This is also where the integration test is currently failing:

================================================= FAILURES =================================================
_____________________________ test_train_multi_stream[test_multi_stream_39be6] _____________________________

setup = None, test_run_id = 'test_multi_stream_39be6'

    @pytest.mark.parametrize("test_run_id", ["test_multi_stream_" + commit_hash])
    def test_train_multi_stream(setup, test_run_id):
        """Test training with multiple streams including gridded and observation data."""
        logger.info(f"test_train_multi_stream with run_id {test_run_id} {WEATHERGEN_HOME}")
    
        main(
            [
                "train",
                f"--base-config={WEATHERGEN_HOME}/integration_tests/small_multi_stream.yaml",
                "--run-id",
                test_run_id,
            ]
        )
    
        infer_multi_stream(test_run_id)
>       evaluate_multi_stream_results(test_run_id)

integration_tests/small_multi_stream_test.py:71: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
integration_tests/small_multi_stream_test.py:167: in evaluate_multi_stream_results
    evaluate_from_config(cfg, None, None)
packages/evaluate/src/weathergen/evaluate/run_evaluation.py:338: in evaluate_from_config
    results = [_process_stream(**task) for task in tasks]
packages/evaluate/src/weathergen/evaluate/run_evaluation.py:234: in _process_stream
    plot_data(reader, stream, global_plotting_opts)
packages/evaluate/src/weathergen/evaluate/utils/utils.py:489: in plot_data
    plotter.create_maps_per_sample(
packages/evaluate/src/weathergen/evaluate/plotting/plotter.py:406: in create_maps_per_sample
    f"({format_datetime(valid_time)})"
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

dt = None

    def format_datetime(dt):
>       return dt.astype("datetime64[m]").astype(datetime.datetime).strftime("%Y-%m-%d T%H:%M:%S")
E       AttributeError: 'NoneType' object has no attribute 'astype'

packages/evaluate/src/weathergen/evaluate/plotting/plotter.py:1947: AttributeError

[SavvasMel]

@clessig is your purpose to also plot subtime steps?

[clessig]

@clessig is your purpose to also plot subtime steps?

Not on purpose. The fine-tuning data is not time-stepped.

[SavvasMel]

@clessig is your purpose to also plot subtime steps?

Not on purpose. The fine-tuning data is not time-stepped.

I know what the problem is but on the other hand, I do not what the normal behavior should be. Would you have an old plot so that can I have an idea?

[SavvasMel]

I am just wondering which of those times should be appear in the plot. But your comment above about not "time-stepped" data confused me a bit.

da_t
<xarray.DataArray 'target' (ipoint: 4)> Size: 16B
array([465826.25, 165968.4 , 353633.28, 161834.27], dtype=float32)
Coordinates:
  * ipoint                 (ipoint) int64 32B 58636 58771 58773 58835
    valid_time             (ipoint) datetime64[ns] 32B 2022-10-02T00:00:01 .....
    lat                    (ipoint) float32 16B 33.32 33.68 33.69 33.83
    lon                    (ipoint) float32 16B 141.9 138.9 140.7 139.7
    source_interval_start  datetime64[ns] 8B 2022-10-01T18:00:00
    source_interval_end    datetime64[ns] 8B 2022-10-02
    stream                 <U11 44B 'METOPB_IASI'
    forecast_step          int64 8B 0
    channel                <U25 100B 'obsvalue_avhrr_mean_vis_0'
    sample                 int64 8B 1
    lead_time              timedelta64[ns] 8B 06:00:01

[clessig]

This would be a natural time: source_interval_start datetime64[ns] 8B 2022-10-01T18:00:00 . Does it break because source_interval_end has no hours?

[SavvasMel]

No, the valid_time is None and the function below which I made to remove the nanoseconds does not have a check for that:

def format_datetime(dt):
    return dt.astype("datetime64[m]").astype(datetime.datetime).strftime("%Y-%m-%d T%H:%M:%S")

If I turn this off I am getting for example the plot below. I am wondering what time should be plotted. Because the valid_times are around 70000 in the xarray so I am not sure what exactly we want here.