Introduce `surfnet_streamAccount`

[lgalabru]

With this cheat code, developers would start a routine updating a given account by pulling from the remote, either on a regular basis (we update every second) or lazily (the account is being updated every time it's being read accessed).

[MicaiahReid]

I have some concerns about this that I'd love to get community feedback on.

Whenever an account is pulled from the remote and inserted in the local state, in many cases this implies that the pulled account will be mutated in some way.

If we are regularly pulling an account from the remote and using the remote data to update our local state, there is a high risk that we will be overwriting state changes that the user is applying to the SVM via their own transactions.

For example:

1. A user has a program deployed to mainnet
2. Every time the program is invoked, the program's state is updated (say the program tracks calls to the protocol, or more likely, the program tracks the protocol's NAV)
3. The user configures their surfnet to surfnet_streamAccount(<program>), to pull updates for the program from mainnet
4. The user sends a transaction to their surfnet which calls the program
   a. Surfpool sees a transaction referencing the program, which it is always supposed to update from mainnet
   b. Surfpool pulls the program from mainnet, getting the current state
   c. The user's transaction alters the state of the program
5. The user sends another transaction to the program, expecting to have the state changes that were made in the last transaction be reflected in the program state
   a. Surfpool sees a transaction referencing the program, which it is always supposed to update from mainnet
   b. Surfpool pulls the program from mainnet, getting the current state
   c. The program state no longer reflects the changes made by the user's transaction
   d. The user's transaction alters the state of the program again
6. The user does not find expected changes to program state

This would be a difficult bug to track in the user's expected behavior, so a major footgun.

---

Rather, I think the approach we will have to take is to stream all transactions that invoke an account, and replay those transactions on a surfnet.

To play out the same example:

1. A user has a program deployed to mainnet
2. Every time the program is invoked, the program's state is updated (say the program tracks calls to the protocol, or more likely, the program tracks the protocol's NAV)
3. The user sends a transaction to their surfnet which calls the program
   a. Surfpool pulls the program from mainnet, getting the current state
   b. The user's transaction alters the state of the program
4. Every time the program is invoked on mainnet, the mainnet transactions are forwarded to the surfnet to be played out
   a. Each of these transactions also alters the program's state, according the the program's logic
5. The user sends another transaction to the program, expecting to have the state changes that were made in the last transaction be reflected in the program state
   a. Surfpool already has the program locally, so it does not pull it from mainnet
   b. The program's state reflects the user's local transactions + remote transactions
   c. The user's transaction alters the state of the program again
6. The user does finds expected changes to program state

This, however, is a much more difficult problem - we can no longer just fetch current state from the remote RPC; we need to be notified of every transaction referencing an account, so we'll need a geyser plugin indexing the remote for this purpose.

[AzothZephyr]

Whenever an account is pulled from the remote and inserted in the local state, in many cases this implies that the pulled account will be mutated in some way.

not true

If we are regularly pulling an account from the remote and using the remote data to update our local state, there is a high risk that we will be overwriting state changes that the user is applying to the SVM via their own transactions.

this is a reasonable concern but i don't expect an informed user to use the streaming of an entire account in the scenario they require modifications to that account. i think this should be generally advised to be used on accounts in surfpool which are read-only, such as pyth price feeds or other oracles.

one idea is to give a more flexible option and stream full accounts but also have a method that allows streaming of changes to specific values within an account as they change on mainnet. so, if you need to pull specific values from an account on mainnet, a separate method could be defined to keep that value of the account in surfpool updated vs the entire account. this would allow me to work in surfpool with other values in that account while surfpool updates the portion i've told it to with mainnet values.

tl;dr: streaming account updates is valuable for explicitly read-only accounts, so i wouldn't shit can the entire idea, and another approach would be to allow both full and partial account streaming.

[AzothZephyr]

also, i don't know if i agree with a fixed cadence / lazily. i'd update the account every time it updates by listening to account changes via grpc / ws and apply in real time

[MicaiahReid]

Thanks for the feedback @AzothZephyr!

Whenever an account is pulled from the remote and inserted in the local state, in many cases this implies that the pulled account will be mutated in some way.

not true

I think you're correct, this isn't always true. However, we don't really have a way of knowing the user's intent (do they expect this account that they pulled to be read from only and never written to? Or do they expect it to be updated by their transactions?). I think in many cases, the user hasn't actually thought through what they expect to happen when they send a transaction.

With this surfnet_streamAccount approach, I think it's too easy for a user to accidentally overwrite accounts they've written to locally, though they didn't know they were writing to that account locally.

I see the value in being able to continually pull data from mainnet, as your point about oracles makes clearly. However, do you think a more robust approach like surfnet_streamAccountTransactions or something, which forwards mainnet transactions targetting a specified account from mainnet to your surfnet would still work? This would:

• keep read-only accounts like oracles updated with mainnet data (the user isn't writing to this account, but the oracle is updated via txs [unless I'm misunderstanding oracles], so any upstream txs being applied to this account should keep the oracle updated)
• prevent users from shooting themselves in the foot by writing over accounts that they've written to locally.

Thoughts?

---

BTW, if we do go with surfnet_streamAccount, I love the idea you proposed of having a filter to set which account fields to stream

[lgalabru]

Streaming and re-applying all the transactions locally has a pretty high cost, both in terms of complexity and bandwidth.

How about we keep fetching the latest values as long as the streamed account was not locally updated. Otherwise, we emit a warn log, and stop fetching - the mutated local version became the source of truth.

[MicaiahReid]

Agreed, my idea introduces complexity while your initial take is a quick way to support many use-cases. Emitting a warn and stopping fetching on a mutation of one of these accounts sounds like a great middle ground

[mubarizkyc]

pub async fn update_accounts(addresses: Vec<Pubkey>, rpc_client: &Arc<RpcClient>,http_client:&Client) {
    let accounts = rpc_client
        .get_multiple_accounts(&addresses)
        .expect("Failed to fetch accounts");
    for (i, account_opt) in accounts.into_iter().enumerate() {
        if let Some(account) = account_opt {
            let pubkey_str = addresses[i].to_string();
            let owner_str = account.owner.to_string();
            let data_hex = hex::encode(account.data);

            let payload = json!({
                "jsonrpc": "2.0",
                "id": 1,
                "method": "surfnet_setAccount",
                "params": [
                    pubkey_str,
                    {
                        "lamports": account.lamports,
                        "owner": owner_str,
                        "executable": account.executable,
                        "rent_epoch": account.rent_epoch,
                        "data": data_hex,
                    }
                ]
            });
            let response = http_client
                .post("http://127.0.0.1:8899")
                .header("Content-Type", "application/json")
                .json(&payload)
                .send()
                .await;
            match response {
                Ok(resp) => {
                    if let Ok(text) = resp.text().await {
                        //  account updated
                    }
                }
                Err(e) => {
                }
            }
        } else {
        }
    }
}

[mubarizkyc]

@MicaiahReid i used this setup for my trading bot ,before each tx i updated the complete list of accounts i am going to use

[mubarizkyc]

i think for the time being we should impl surfnet_setAccounts ,which should work for some 100 accounts at a time.

[lgalabru]

Addressed with #371