Reminders V2: Improve Reminder Implementation Semantics

[ElanHasson]

This is a DRAFT

Summary

This proposal is a high-level design for Reminders V2 that will allow the utmost flexibility in Reminders.

Reminder Middleware

The concept of Reminder Middleware allows users to insert functionality into the Reminder processing pipeline. The behavior would mimic ASP.NET Core's Middleware pipeline.

Reminders would have a persistent, shared data context, that is local to the Reminder instance. This can be used to store arbitrary data related to the reminder's execution. This would allow middleware to maintain internal state and share data with each other, much like middleware can mutate headers and the body of the response.

This ReminderState would behave like a normal GrainState and would remove the current need to have a different schema for persisting reminders, and allow it to be extensible. This context could be implemented as a Dictionary<string, object?> on the base of the ReminderState object.

Standard Middleware

Some standard middleware that would come with the new Microsoft.Orleans.Reminders package (#7410):

• Supporting unbounded dueDatess and intervals (internal default middleware)
• Supporting one-shot
• Late Reminder Handling
• Missed Reminder Handling
• Limit the number of reminder ticks
• Determining the next dueTime based on a complex cron expression
  • 0/5 2,3,8,9,17-23 ? 2-8,7,10 MON-FRI = Every 5 minutes, at 02:00 AM, 03:00 AM, 08:00 AM, 09:00 AM, and 05:00 PM through 11:59 PM, Monday through Friday, only in February through August, July, and October
• Implementing Reminder history by allowing users to create their own middleware and registering it in a specific reminder

Each reminder would be able to specify it's own, ordered set of reminder middleware:

Example 1: starting in 3 days, run every minute, max of 100 ticks, with additional options

this.CreateReminder(name: "MyAwesomeReminder")
  .UseWait(o=> o.NextTickAt = DateTime.UtcNow.AddDays(3)) // Wait 3 days until our first Tick
  .WithInterval(o=> o.NextTickAt = context.CurrentTick.Add(TimeSpan.FromMinutes(1)) // Run every minute
  .WithLimitTicks(o=> o.MaxTicks == 100)
  .UseSkipLateReminder(TimeSpan.FromMinutes(2)) // Do not deliver the reminder if later than 2 minutes (give a 2 minute grace-period), sets `context.ShouldSkip` = true if late
  .UseMissedReminderHandler(async (dueTime) => await MissedThis(dueTime))
 .UseReminderHistoryRecorder<MyCustomReminderHistoryGrain>(options {
    options.GrainId = $"{this.GetPrimaryKey()}-MyAwesomesReminder"; // This could be derived as well but there would be cases where we would want to manually control this
  });

Example 2: Cron Expression based reminder

await this.RegisterOrUpdateReminder("MyAwesomeReminder", builder => builder
  .UseWait(o=> o.NextTickAt= DateTime.UtcNow.AddMinutes(5))
  .UseCronExpression("0/5 2,3,8,9,17-23 ? 2-8,7,10 MON-FRI") 
  .UseSkipLateReminder(TimeSpan.FromMinutes(2))
  .UseMissedReminderHandler(async (dueTime) => await MissedThis(dueTime))
  .UseReminderHistoryRecorder<MyCustomReminderHistoryGrain>(options {
    options.GrainId = $"{this.GetPrimaryKey()}-MyAwesomesReminder"; // This could be derived as well but there would be cases where we would want to manually control this
  })
);

Bypassing Current Limits on Reminders

We are currently limited to ReminderRegistry.MaxSupportedTimeout, which is 0xfffffffe or approximately 49 days.

The way we bypass this limit is to intercept the reminder prior to it being delivered to the CallingGrainReference associated with the reminder and determine if it is time for the reminder to be triggered. If we've met dueTime, we deliver it, if it is still early, we update the reminder with the new dueTime up to a max of 0xfffffffe.

This logic would be registered as default Reminder Middleware and added to the pipeline by Reminders V2. If the reminder isn't ready to be fired, it would stop the pipeline before executing any other configured middleware.

For a user-land implementation, see https://github.com/web-scheduler/web-scheduler/blob/main/Source/WebScheduler.Grains/Scheduler/ScheduledTaskGrain.cs#L104-L196.

Persistence

Reminders V2 state data will be treated like any other Grain State and will not require any changes to grain storage providers.

Migrating Existing Timers

Two options exist for a path forward:

1. Leave support for Reminders V1 so this doesn't become a breaking change and introduce Reminders V2 as an opt-in feature.
2. Support Reminders V2 only, and migrate V1 to V2. This can be problematic and complex, see: Migrating v3.x to v4.x Grains.

Personally, I prefer Option 1 as this makes it removes a lot of engineering work and complexity. This puts the onus of migrating on users within their existing grain code. At most, this would take ~49 days for all reminders to be migrated without any downtime. Additionally, we'd publish a deprecation schedule for Reminders V1 or leave them until we're ready to remove them.

Reminders V2 will most likely have a separate processing pipeline all together due to the work in #7238 and #947, so it would make sense for it to be a fully separate, opt-in feature. This would also allow us to release Reminders V2 in preview mode alongside regular releases to get feedback and iterate.

[suraciii]

The way we bypass this limit is to intercept the reminder prior to it being delivered to the CallingGrainReference associated with the reminder and determine if it is time for the reminder to be triggered. If we've met dueTime, we deliver it, if it is still early, we update the reminder with the new dueTime up to a max of 0xfffffffe.

I think we won't need to do this if #947 resolved, only recent (2h e.g.) reminders will be queried and scheduled with .NET timers in memory

[ElanHasson]

@suraciii

I think we won't need to do this if #947 resolved

I think you may be correct here. I've been thinking about having the ability to support different ReminderService providers (read: not Reminder Storage providers as we have today) which will all have different pros and cons: This would allow users to select the algorithm and way reminders are managed within the silo, and even allow for a mix-and-match approach for different types.

I don't have anything more to post on this as I'm still iterating on the concept internally at the moment.

Perhaps this story is split into a reminders v1.5 and a v2, where v1.5 removes these limitations. It's not clear to me yet what the approach should be here.

#7410 will give us lots of flexibility here.

[koenbeuk]

I like this proposal, though I think the API needs some work. As reminder registration happens asynchronously, the reminder registration builder needs to be awaited. In addition, we need to consider how we would Update reminders. Therefore, I would propose something in the shape of the following API:

await this.RegisterOrUpdateReminder("MyAwesomeReminder", builder => builder
  .UseWait(o=> o.NextTickAt= DateTime.UtcNow.AddMinutes(5))
  .UseCronExpression("0/5 2,3,8,9,17-23 ? 2-8,7,10 MON-FRI") 
  .UseSkipLateReminder(TimeSpan.FromMinutes(2))
  .UseMissedReminderHandler(async (dueTime) => await MissedThis(dueTime))
  .UseReminderHistoryRecorder<MyCustomReminderHistoryGrain>(options {
    options.GrainId = $"{this.GetPrimaryKey()}-MyAwesomesReminder"; // This could be derived as well but there would be cases where we would want to manually control this
  })
);

In this example, I simplified the registration of the middleware by consuming extension methods in a hypothetical IReminderBuilder. In addition, MissedReminderHandler now refers to a Lambda instead of a separate handler class and ReminderHistoryRecorder is referring to a different grain with a custom GrainId.

[ElanHasson]

@koenbeuk this is great feedback! I love it! Will update the proposal to incorporate it.

Thank you!

[jsteinich]

One pattern that we use in the current API is a durable retry (on both one shot and periodic). Adding retry logic might be expanding scope a bit much, but it's definitely a bit clunky in the current API.

Could start to venture down the path of Polly which has very extensive logic available, but something along the lines of the ClusterClient connection retry would probably suffice.