.NET: [Bug]: How do you in RC1 get the messages in a session?

[rwjdk]

Description

Before RC1 you could do the following to get the messages in a session:

AgentSession session = await agent.CreateSessionAsync();
IList<ChatMessage> messagesInSession = session.GetService<IList<ChatMessage>>()!;

I can't for the life of me see how you can do something similar in the RC1 version :-(

Code Sample

Error Messages / Stack Traces

Package Versions

RC1

.NET Version

.NET10

Additional Context

No response

[rwjdk]

Think I found the way

InMemoryChatHistoryProvider? provider = agent.GetService<InMemoryChatHistoryProvider>();
List<ChatMessage> messagesInSession = provider?.GetMessages(session) ?? [];

Personal opinion: This is quite an unintuitive and cumbersome way to get these

• You need to know these things (not discoverable)
• You need to parse the agent around all over the place and not just the AgentSession)
• You need to ensure that an agent has a ChatHistoryProvider; otherwise, this will not work when you resume a session from a JSON-Element before you run the first .RunAsync that auto-creates the ChatHistoryProvider ... Is it not a bug that agent.DeserializeSessionAsync(jsonElement) does not initialize the ChatHistoryProvider on the agent if missing?

What is the primary reason for there not being a session.GetMessages() or the old way does not work (I understand there can be multiple providers, but still, will they not all. in some form, have a list of messages?)

[westey-m]

Hi @rwjdk, this is indeed the best way to do this now and I agree that it is not the most intuitive. This is certainly the main negative side-affect of the change we made recently.

The change was to decouple ChatHistoryProvider and AIContextProvider from the AgentSession. Previously the ChatHistoryProvider was stored as a property on ChatClientAgentSession, which meant that GetService could delegate to the ChatHistoryProvider. Also InMemoryChatHistoryProvider had the messages inside itself in a field. Having such a stateful object on the AgentSession complicates serialization and deserialization though, especially if it needs a reference to some db client or service client. It also means that ChatHistoryProvider and AIContextProvider cannot be used in other parts of the pipeline (e.g. as a decorator on a non ChatClientAgent AIAgent), since there's no easy way to contribute state in that case at session serialization time.

All *Providers are now essentially stateless and store their state in a StateBag on the AgentSession. *Providers are part of an Agent, while state lives on the session.
This has some benefits:

• We can now attach specialized AIContextProviders as decorators on AIAgent (including A2A) and all AIContextProviders to ChatCient (when used by an agent).
  • See here and here
• it fixes some weirdness where if you had used the same ChatClientAgentSession with multiple different ChatClientAgents that have different AIContextProviders, the AIContextProviders of the invoked Agent is now used rather than the ones from the agent that the session was created from. While we don't typically recommend someone doing this, it's clearly a bug if you have an agent that requires an attached RAG AIContextProvider to function correctly, and it doesn't get used, since the session has a different AIContextProvider on it.

There is a shortcut to getting the *Provider state from the AgentSession, but it has some caveats. You can read the state of the *Provider directly from the session StateBag if you know its key and type. For the built-in *Provider, the default keys used are the type name of the *Provider, and the state types are all *Provider.State. Note that the key can be overridden for each provider using an option. If you know what key the provider is using, you can of course read the value, but if you have an opaque AIAgent, using the approach you have above is safer, since it may have been configured with a custom state key.

// Getting provider state from the session.
if (session.StateBag.TryGetValue<InMemoryChatHistoryProvider.State>(nameof(InMemoryChatHistoryProvider), out var state))
{
    Console.WriteLine(state?.Messages.Count);
}

// If you want a custom state key you can change it like so:
    .AsAIAgent(new ChatClientAgentOptions()
    {
        ChatOptions = new() { Instructions = "You are a helpful personal assistant." },
        ChatHistoryProvider = new InMemoryChatHistoryProvider(new InMemoryChatHistoryProviderOptions
        {
            // Override StateBag Key name to something other than `nameof(InMemoryChatHistoryProvider)`
            StateKey = "CustomKey"
        })
    });

What is the primary reason for there not being a session.GetMessages() or the old way does not work (I understand there can be multiple providers, but still, will they not all. in some form, have a list of messages?)

The reason for having ChatHistoryProvider configurable is that you may want to store messages somewhere else than in the session state. These would have an id that points to e.g. Cosmos DB, where the messages are stored and they wouldn't have messages. Calling session.GetService<IList<ChatMessage>>() was also not previously supported for these providers.

Definitely open to ideas on making all this simpler to use, so let me know if you have any ideas.

[rwjdk]

@westey-m : Here are 4 options for making it more discoverable (here as Extension Methods, but directly on AgentSession in the real source code)

In order I like Option4 the most, then 1, then 3 and finally 2

Let me know if it is something you think you will implement... Else I'll add it to my AgentFrameworkToolk Nugets instead

public static class AgentSessionExtensions
{
    //Option 1: Make a method on AgentSession that if given a inMemoryChatHistory Provider can get the messages
    //- Pros: A: make things more discoverable - B: Is safe as input is known - C: There could be overloads for other out of the box ChatHistory providers
    //- Cons: Only for InMemoryChatHistory
    public static IList<ChatMessage> GetMessages(this AgentSession session, InMemoryChatHistoryProvider inMemoryChatHistoryProvider)
    {
        return inMemoryChatHistoryProvider.GetMessages(session) ?? [];
    }

    //Option 2: Make a method on AgentSession that if given an Agent get the messages
    //- Pros: More discoverable than option 1
    //- Cons: Confusing if you use a custom HistoryProvider and don't get any values back
    public static IList<ChatMessage> GetMessages(this AgentSession session, AIAgent agent)
    {
        ChatHistoryProvider? chatHistoryProvider = agent.GetService<ChatHistoryProvider>();
        if (chatHistoryProvider is InMemoryChatHistoryProvider inMemoryChatHistoryProvider)
        {
            return inMemoryChatHistoryProvider.GetMessages(session) ?? [];
        }
        //todo other out-of-the-box providers support

        return [];
    }

    //Option 3: Make a method on AgentSession that find data via state bag
    //- Pros: Most discoverable
    //- Cons: Confusing if you use a custom HistoryProvider and don't get any values back
    public static IList<ChatMessage> GetMessages(this AgentSession session)
    {
        if (session.StateBag.TryGetValue<InMemoryChatHistoryProvider.State>(nameof(InMemoryChatHistoryProvider), out var state))
        {
            return state?.Messages ?? [];
        }
        //todo other out-of-the-box providers support

        return [];
    }

    //Option 4: Make a method on AgentSession that if given a generic ChatHistory Provider can get the messages (will need a new virtual GetMessages() on ChatHistoryProvider that custom solutions can override)
    //- Pros: A: make things more discoverable - B: Is safe as input is known - C: There could be overloads for other out of the box ChatHistory providers
    //- Cons: Need a new virtual method on ChatHistoryProvider
    public static IList<ChatMessage> GetMessages(this AgentSession session, ChatHistoryProvider chatHistoryProvider)
    {
        return chatHistoryProvider.GetMessages(session) ?? []; //NB: THis can't compile in current code 
    }
}

Usage

//Option 1
IList<ChatMessage> chatMessages1 = session.GetMessages(agent.GetService<InMemoryChatHistoryProvider>()!);

//Option 2
IList<ChatMessage> chatMessages2 = session.GetMessages(agent);

//Option 3
IList<ChatMessage> chatMessages3 = session.GetMessages();

//Option 4
IList<ChatMessage> chatMessages4 = session.GetMessages(myCustomProviderOrDefaultInMemoryProvider);

[rwjdk]

@westey-m : But in any case, should

AgentSession = await agent.DeserializeSessionAsync(jsonElement);

..not always on the fly create an InMemoryChatHistoryProvider on the agent, if the agent used do not already have a chat-history provider... because if it is not there, you can't restore the conversation prior to the first agent.RunAsync(...). (See this sample for more context (line 27))

[westey-m]

@rwjdk, here is another option. I think in most cases people will not be customizing the statebag key for InMemoryChatHistoryProvider, and in cases that they do, they can choose between propagating the key around or using the provider to read messages.

Note as well TryGet, since there is no guarantee that the session will store any messages.

// Option 5
// *********
// Default
bool found = session.TryGetInMemoryChatHistory(out var messages);
session.SetInMemoryChatHistory(messages);
// With custom key
bool found = session.TryGetInMemoryChatHistory(out var messages, myKey);
session.SetInMemoryChatHistory(messages, myKey);
// With custom key and serialization options (important for Native AOT/Trimming)
bool found = session.TryGetInMemoryChatHistory(out var messages, myKey, myJso);
session.SetInMemoryChatHistory(messages, myKey, myJso);

should AgentSession = await agent.DeserializeSessionAsync(jsonElement); not always on the fly create an InMemoryChatHistoryProvider on the agent

This comes down to supporting the two core chat history storage scenarios:

1. In the AI Service, e.g. with Responses (store = true)
2. External to the AI Service, and therefore in a ChatHistoryProvider, e.g. with Chat Completion

Unfortunately, we do not know which mode we are in (1 or 2), when a ChatClientAgent is constructed. The ChatClient has no metadata to declare this. We only find out after the first ChatClient invocation.
The current logic is:

1. Did the user provide a ChatHistoryProvider? OK, set it on the ChatClientAgent property.
2. On first run:
   1. Did the ChatClient return a conversation Id? If so, we are in service stored chat history mode.
   2. If we are in service stored chat history mode AND the user provided a ChatHistoryProvider, throw since the two cannot be used together, to avoid users thinking that their ChatHistoryProvider will be used when it will not be used.
   3. If we are NOT in service stored chat history mode AND the user provided a ChatHistoryProvider, use it.
   4. If we are NOT in service stored chat history mode AND the user did NOT provide a ChatHistoryProvider, create an InMemoryChatHistoryProvider, set it on the ChatClientAgent property and use it.

We have a few options here:

1. Keep it the way it is.
2. Always set a ChatHistoryProvider and don't throw if we are in service stored chat history mode. This can be confusing for users, since they might expect the ChatHistoryProvider on the agent to be used.
3. Always set a ChatHistoryProvider but set it to null as soon as we realise we are in service stored chat history mode. Again, may be confusing, since it's essentially a silent failure, and the property might switch to null after the first run.
4. Same as 3 but also logging a warning.

Coming back to the original point, if we always have a ChatHistoryProvider on the agent, it doesn't mean that it is actually being used, and using it to read values from session state is only meaningful if it is being used.