[Bug] dbt retry should use the original failure date as reference for microbatch models

[SGA-taichi-kato]

Is this a new bug in dbt-core?

• I believe this is a new bug in dbt-core
• I have searched the existing issues, and I could not find an existing issue for this bug

Current Behavior

When a microbatch model fails during dbt run, and we later execute dbt retry, the command uses the current date as reference instead of the original failure date. This results in processing different batches than those that originally failed.

Expected Behavior

When executing dbt retry, it should use the original failure date (2025-03-21 in this case) as the reference date for microbatch processing, retrying the batches that originally failed (2025-03-18 through 2025-03-21).

Steps To Reproduce

1. On 2025-03-21, executed the following microbatch model

Command:

dbt run --target dev --log-level info --target-path /target-path/job --select target_table

Model configuration:

{{
    config(
        materialized='incremental',
        incremental_strategy='microbatch',
        event_time='_event_date',
        begin='2024-12-01',
        lookback=3,
        batch_size='day'
    )
}}

Result:

07:55:33  1 of 8 START sql microbatch model schema.target_table ................. [RUN]
07:55:33  Batch 1 of 4 START batch 2025-03-18 of schema.target_table .................. [RUN]
07:55:35  Batch 1 of 4 ERROR creating batch 2025-03-18 of schema.target_table ......... [ERROR in 2.56s]
07:55:35  Batch 2 of 4 SKIP batch 2025-03-19 of schema.target_table ................... [SKIPPED in 0.00s]
07:55:35  Batch 3 of 4 SKIP batch 2025-03-20 of schema.target_table ................... [SKIPPED in 0.00s]
07:55:35  Batch 4 of 4 SKIP batch 2025-03-21 of schema.target_table ................... [SKIPPED in 0.00s]
07:55:35  1 of 8 ERROR creating sql microbatch model schema.target_table ........ [ERROR in 2.61s]

2. On 2025-03-25, executed dbt retry

Command:

dbt retry --target dev --log-level info --state /target-path/job

Result:

01:04:21  1 of 1 START sql microbatch model schema.target_table ................. [RUN]
01:04:21  Batch 1 of 4 START batch 2025-03-22 of schema.target_table .................. [RUN]
01:04:30  Batch 1 of 4 OK created batch 2025-03-22 of schema.target_table ............. [OK in 9.19s]
01:04:30  Batch 2 of 4 START batch 2025-03-23 of schema.target_table .................. [RUN]
01:04:37  Batch 2 of 4 OK created batch 2025-03-23 of schema.target_table ............. [OK in 6.75s]
01:04:37  Batch 3 of 4 START batch 2025-03-24 of schema.target_table .................. [RUN]
01:04:44  Batch 3 of 4 OK created batch 2025-03-24 of schema.target_table ............. [OK in 6.53s]
01:04:44  Batch 4 of 4 START batch 2025-03-25 of schema.target_table .................. [RUN]
01:04:50  Batch 4 of 4 OK created batch 2025-03-25 of schema.target_table ............. [OK in 6.25s]
01:04:50  1 of 1 OK created sql microbatch model schema.target_table ............ [SUCCESS in 28.78s]

Relevant log output

Environment

- OS:Ubuntu24.04
- Python:3.12.0
- dbt:1.9.3

Which database adapter are you using with dbt?

No response

Additional Context

The condition here is suspicious.
https://github.com/dbt-labs/dbt-core/blob/v1.9.3/core/dbt/task/retry.py#L144

[QMalcolm]

Hi @SGA-taichi-kato thank you for reporting this ❤ I agree with you that this is a bug.

Interestingly, this was done intentionally... by me 😬 Sorry 🙃 There are two cases when this happens. When the batches weren't even computed for a microbatch model on the previous run (i.e. the whole microbatch model was skipped due to an upstream failure) or when all the batches for a microbatch model either failed or skipped. In the second case we do this because, and I quote myself:

# We need this so that re-running of a microbatch model will only rerun
# batches that previously failed. Note _explicitly_ do no pass the
# batch info if there were _no_ successful batches previously. This is
# because passing the batch info _forces_ the microbatch process into
# _incremental_ model, and it may be that we need to be in full refresh
# mode which is only handled if previous_batch_results _isn't_ passed for a node

Boy howdy-doody.

As I see it, there are two possible solutions

1. make it such that having previous batches doesn't force incremental AND make it so that skipped microbatch model compute and populate their batch results
2. begin tracking the original invocation time in someway AND use this time during the computation of batches on retry

My gut reaction was we should do (1), but I think we should do (2). Option (1) feels more "stateless" which is generally how we've gone about things in core, but this is retry we are talking about which inherently has some state. Additionally, it has dawned on me that this probably affects sample mode as well. That is, if one is using a "relative time filter" for their sample run, then on retry we likely want to sample relative to the original invocation time. Also, I'm fairly option (2) will require a smaller changeset to fix the problem (more piping work, less logical refactoring work).

[github-actions]

This issue has been marked as Stale because it has been open for 180 days with no activity. If you would like the issue to remain open, please comment on the issue or else it will be closed in 7 days.

[SGA-taichi-kato]

remain