Multi-Agent Support

[hayescode]

Multi-Agent frameworks have become the gold standard for robust high-value LLM orchestration. Chainlit was not built with this in mind and is behind the curve.

Proposal

• Introduce a new configuration pane to Chainlit to support the configuration of teams and sub-agents.
• Reconnect/Rethink tasklist for multi-agent observability.
• Improve nested steps - The input/output is not in the correct order by default. Maybe we should rethink using Steps for sub-agents/nested steps for sub-agent tool calls?
• Team configuration panel to the right side of the screen.
• Add a model picker to the chat input similar to commands but persistent and a drop-down that shows the selected model and icon when collapsed.

Please share your thoughts and opinions below so we can shape this to benefit all users, especially if you have multi-agent implemented today your input is highly valuable!

Discord Discussion Thread

Related Issues:

• Add "@" parsing to invoke agents #2580
• Tool Steps always below Answer #2365
• Idea: Allow profiles & starters to be set in on_chat_start #2355
• Support for Project Folders in ChatGPT Plus #2117
• Req support for A2A and agent cards #2099
• TaskList is disconnected from messages #2058
• Agent profile description does not disappear after quick mouse hover (UI freeze) #2015
• Easy way to make messages not be nested within a step #1913
• Send/Stream messages from multiple different processes #1491
• Integration with crewAI #1478
• Add human-in-the-loop capabilities #1129
• Nested steps for tools etc. #1077
• Possible to nest AskUserMessage under a step? #724
• Chainlit + Autogen + OpenSource LLM API locally hosted #662
• I need your help creating an example with chainlit and MiniAutoGen: Lightweight and Flexible Agents for Multi-Agent Chats #641

[munday-tech]

We have a multi-agent setup deployed within Langgraph & Chainlit now so I'll shed some light on how we currently manage things.

Firstly, I think its important to consider two types of multi-agent/agentic AI deployments here. One where you can "host" shareable/usable agents that people could select from a panel or @ etc. Secondly is agentic workflows, that exist as a single entity from the users perspective.

In our scenario we currently utilize option two. We have an internal AI chatbot that spans all internal business domains (marketing, sales, tech etc etc) this is built with a hierarchical structure where we have "departments" (A graph in langgraph) "teams" (a subgraph within a department) and agents, the workers sitting within teams, orchestrated by a supervisor.

When an employee loads a new chat on chainlit, we build a custom version of the overall graph for them, including only teams, departments, agents & tools they have permissions to use. This means there may be a "marketing internal" team that only marketing employees can access. This is stored in the user session and used throughout the chat.

To do this, we just use on_chat_start for building & storing the graph with message handling as normal. For "reasoning/thinking" we leverage with cl.Step() in agents, tools, supervisors etc. We run our main graph invoke under a step, and then all agents/teams/tools steps collapse under this parent so employees can view all steps taken by differing agents. The only other note we do differently is our slack sits outside of chainlit, so we build http_context within chainlit for slack to leverage the data layer & store chats in both slack & web.

Another note is consent, a lot of tools, especially those interacting with end user data/PII requires explicit consent from a user. We currently do this by building custom JSX elements and passing them in via AskElementMessage. The main issue here is its POD time gated, so you setup a timeout, wait and if the user fails to consent you don't continue. Is there a reasonable way to remove this timer, but prevent further execution until the consent is provided?

The other note on the Custom JSX elements for consent, expanding available imports here would be great as right now you cannot add calendars and a few other critical aspects for forms.

From my perspective, a few nice to haves would be;

1. A configurable page for users to "view" the setup they have access to. Allowing the ability to pass in a graph like structure to allow for transparency in single pane multi-agent setups. Challenge here is finding a framework agnostic way to display this data
2. Configurable/changeable chat starters - as raised in previous issue this is quite important as you're currently locked to starters on build, when these could potentially change per user. Offshoot of this could be when you @ an agent in option one, it loads that agents starters, allowing custom setups per agent for common tasks
3. "Re-run" capabilities. In Langgraph specifically, you can go back in time docs it would be great to be able to trigger this in chainlit from a step (if it stayed in steps) where a user could "try again from here" and pass in modified params if required. There should be a way to disable this for specific steps though for security (tool calls etc)

Overall multiagent works okay currently but could see some nice QOL and improvements. Potentially updating the step side so we can have a better way to display to the user whats happening would be great.

Happy to provide more insight!

[hayescode]

#2632

[hayescode]

#2734 for model picker. I made it flexible so you can also do modes or starting agents or whatever you need.

[hayescode]

Team configuration panel to the right side of the screen.

I implemented this in #2759

I have now added everything I can think of and it's multi agent is working much better for me using OpenAI Agents SDK and PydanticAI.