Transactions usage in production environment, also stateless grains.

[luckyycode]

Good day/night.
Continuing this old issue #5692 , I'd like to know if anybody has it running stable. The information about orleans transactions is quite a lack, can't even find some real-usage talks for this actually perfect technology.

I couldn't adopt Orleans Transactions back then, due to project tech-debt-limitations or some stuff. Now currently I am implementing it into a project that contains around of ~300k grains in production, mostly persistent ones, non-azure storage provider.

When migrating from IPersistentState to ITransactionalState I ran into issues, but those are solved already at the testing environment. I don't know if the github issues section is right for this, maybe I should join some chat.

The issues are the following:

1. [Proposal] Add an option to exclusively lock Transactional State #9898 - this one is a huge hit after migrating from IPersistentState. Currently solved by my comment in the issue linked. R-V-W model in large scale of grains fails here with PerformRead due to shared/exclusive lock. Other async work is done aside PerformUpdate.

2. There is actually no information how would transaction context behave when using stateless worker grains (with no state), e.g.: client calls a stateless grain, meanwhile this stateless grain processes some data and then calls a transactional grain. Partially solved by hints from Starting transactions from non-statelessworker grains throws lock timeout exception a lot #7393 . By long term testing, turns out to be that stateless worker grains best to have TransactionOption.Supported attribute parameter, this brings no time-out or broken lock exceptions.
   Also, following the issue, I found out that every grain, even stateless that will participate in a transaction, must have Reentrant attribute. But due to the nature of processing, I had to use AsyncSerialExecutor in such stateless grains. My scenario is that a stateless grain firstly processes some data, and then can call a transactional grain. Due to reentrancy, I perform the work using AsyncSerialExecutor and when it's completed, I call my transactional grains. AsyncSerialExecutor is not documented anywhere in details, but that helped me, and I'd like to see more information about this.

tl;dr: silo client -> stateless grain processing, also may call some transactional grain -> the outcome is random: hangs or broken-lock using TransactionOption.CreateOrJoin/Join. TransactionOption.Supported is fine.

3. Currently using TransactionOption.Supported for stateless workers, TransactionOption.Join for transactional grains, but if it's an entry call, I use ITransactionClient.RunTransaction with TransactionOption.Create from the start. This is a bit weird, because if I just use TransactionOption.CreateOrJoin that brings me broken-lock or lock upgrade errors. Otherwise it works fine even in mid-load with no "random" lock errors.

4. For some reason, no matter the transaction storage provider, Store of ITransactionalStateStorage gets called each time when I try to read using PerformRead. Even if my transactional grain does no write at all, has ReadOnly attribute, I still get some activity in Store. I don't know if it is by design, or a bug.

But back to my main question: does anybody use orleans transactions in a high-load, production environment? If so, what issues you had? I'd really like to see some practical usage recommendations, not just for some small examples, but how you handled it, how it goes.

And my one cent, thank you for such great product, I love Orleans.

[jsteinich]

We've been using transactions since they were first released. I wouldn't say we've ever hit all that high of load, but we've probably had half the number of grains that you mentioned.

We use transactions pretty much everywhere. The vast majority of requests involve a transaction. Frequently transactions involve more than 2 grains (unlike the examples).

Given our widespread usage, one of the biggest challenges we've faced has been avoiding deadlocks. At one point a colleague actually worked on adding better deadlock detection. We've generally move more to better rules and analysis about grain access order and have retries to cover cases that we miss, but we've also needed to move some data outside of transactional state. We also have a bit of runtime IL generation to access the latest committed state without invoking a transaction (would be nice if that was built in).

General notes:

• Getting occasional OrleansBrokenTransactionLockExceptions when a transaction simply takes enough longer than normal is a bit of an annoyance.
• Having the PerformRead/Update boilerplate isn't ideal, but you get used to it
• We've also built a custom ITransactionalStateStorage implementation to create hooks so that certain operations (mostly external, but also stream publishing) only happens on transaction completion.
• We're using DynamoDB storage, so there is some extra overhead. I've thought about building a transaction specific provider, but we use a custom provider with additional tweaks so likely wouldn't use the official one.

Regarding the R-V-W pattern:

• We have encountered this specific issue.
• Generally we just treat it as an unnecessary optimization, but we do have some spots where we want to avoid a bigger initialization step.
  • When we know we're going to be updating anyways, we just do a PerformUpdate from the start
  • Created a flag outside of the transactional state (just in grain memory) that we check instead for read operations
  • The flag is just used to switch between PerformRead vs PerformUpdate since we still need to check under transaction to be certain of the state.

Regarding stateless workers:

• It's fine to call a transactional grain method that has the Create (+/- Join) option from a stateless worker
• If a stateless worker call is in the middle of a transactions, then Suppported is needed in order to pass the transaction context onto the next call in the chain
• We've actually added something that we call "SemiStatelessGrains" which is really just a per-silo pool of ids to roughly get silo local scalable grains which are able to participate more directly in a transaction (typically coupled with custom transactional storage)

I believe the store call is expected on read only transactions. It's not known until the end if a transaction will stay read only. There's also the possibility of writes coming from other transactions. Those combine to mean information about the transaction participants needs to be kept track of.