Skip to content

Commit dac0559

Browse files
Many fixes: load state, config, settings save, ... (#304)
1 parent f238e28 commit dac0559

14 files changed

Lines changed: 298 additions & 81 deletions

File tree

README.md

Lines changed: 29 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -64,6 +64,8 @@ pip install magentic-ui[azure]
6464
pip install magentic-ui[ollama]
6565
```
6666

67+
You can then pass a config file to the `magentic-ui` command (<a href="#model-client-configuration"> client config</a>) or change the model client inside the UI settings.
68+
6769
For further details on installation please read the <a href="#️-installation">🛠️ Installation</a> section. For common installation issues and their solutions, please refer to the [troubleshooting document](TROUBLESHOOTING.md). See advanced usage instructions with the command `magentic-ui --help`.
6870

6971

@@ -117,14 +119,15 @@ What differentiates Magentic-UI from other browser use offerings is its transpar
117119
▶️ <em> Click to watch a video and learn more about Magentic-UI </em>
118120
</div>
119121

122+
120123
### Autonomous Evaluation
121124

122125
To evaluate its autonomous capabilities, Magentic-UI has been tested against several benchmarks when running with o4-mini: [GAIA](https://huggingface.co/datasets/gaia-benchmark/GAIA) test set (42.52%), which assesses general AI assistants across reasoning, tool use, and web interaction tasks ; [AssistantBench](https://huggingface.co/AssistantBench) test set (27.60%), focusing on realistic, time-consuming web tasks; [WebVoyager](https://github.com/MinorJerry/WebVoyager) (82.2%), measuring end-to-end web navigation in real-world scenarios; and [WebGames](https://webgames.convergence.ai/) (45.5%), evaluating general-purpose web-browsing agents through interactive challenges.
123126
To reproduce these experimental results, please see the following [instructions](experiments/eval/README.md).
124127

125128

126129

127-
If you're interested in reading more checkout our [blog post](https://www.microsoft.com/en-us/research/blog/magentic-ui-an-experimental-human-centered-web-agent/).
130+
If you're interested in reading more checkout our [technical report](https://www.microsoft.com/en-us/research/wp-content/uploads/2025/07/magentic-ui-report.pdf) and [blog post](https://www.microsoft.com/en-us/research/blog/magentic-ui-an-experimental-human-centered-web-agent/).
128131

129132
## 🛠️ Installation
130133
### Pre-Requisites
@@ -139,7 +142,7 @@ If using Docker Desktop, make sure it is set up to use WSL2:
139142

140143

141144

142-
2. During the Installation step, you will need to set up your `OPENAI_API_KEY`. To use other models, review the [Custom Client Configuration](#configuration) section below.
145+
2. During the Installation step, you will need to set up your `OPENAI_API_KEY`. To use other models, review the [Model Client Configuration](#model-client-configuration) section below.
143146

144147
3. You need at least [Python 3.10](https://www.python.org/downloads/) installed.
145148

@@ -190,8 +193,31 @@ Once the server is running, you can access the UI at <http://localhost:8081>.
190193

191194
#### Model Client Configuration
192195

193-
If you want to use a different OpenAI key, or if you want to configure use with Azure OpenAI or Ollama, you can do so inside the UI by navigating to settings (top right icon) and changing model configuration.
196+
If you want to use a different OpenAI key, or if you want to configure use with Azure OpenAI or Ollama, you can do so inside the UI by navigating to settings (top right icon) and changing model configuration. Another option is to pass a yaml config file when you start Magentic-UI which will override any settings in the UI:
197+
198+
```bash
199+
magentic-ui --port 8081 --config config.yaml
200+
```
201+
202+
Where the `config.yaml` should look as follows with an AutoGen model client configuration:
194203

204+
```yaml
205+
gpt4o_client: &gpt4o_client
206+
provider: OpenAIChatCompletionClient
207+
config:
208+
model: gpt-4o-2024-08-06
209+
api_key: null
210+
base_url: null
211+
max_retries: 5
212+
213+
orchestrator_client: *gpt4o_client
214+
coder_client: *gpt4o_client
215+
web_surfer_client: *gpt4o_client
216+
file_surfer_client: *gpt4o_client
217+
action_guard_client: *gpt4o_client
218+
plan_learning_client: *gpt4o_client
219+
```
220+
You can change the client for each of the agents using the config file and use AzureOpenAI (`AzureOpenAIChatCompletionClient`), Ollama and other clients.
195221

196222
#### MCP Server Configuration
197223

frontend/src/components/settings/tabs/agentSettings/AgentSettingsTab.tsx

Lines changed: 66 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
import React, { useEffect, useState } from "react";
2-
import { Tooltip, Typography, Flex, Collapse, Switch } from "antd";
2+
import { Tooltip, Typography, Flex, Collapse, Switch, Alert } from "antd";
33
import ModelSelector, {
44
PROVIDER_FORM_MAP,
55
} from "./modelSelector/ModelSelector";
@@ -8,6 +8,11 @@ import { SettingsTabProps } from "../../types";
88
import { ModelConfig } from "./modelSelector/modelConfigForms/types";
99
import MCPAgentsSettings from "./mcpAgentsSettings/MCPAgentsSettings";
1010
import { SwitchChangeEventHandler } from "antd/es/switch";
11+
import { settingsAPI } from "../../../views/api";
12+
import { Prism, SyntaxHighlighterProps } from "react-syntax-highlighter";
13+
import { tomorrow } from "react-syntax-highlighter/dist/esm/styles/prism";
14+
15+
const SyntaxHighlighter = Prism as any as React.FC<SyntaxHighlighterProps>;
1116

1217
export const MODEL_CLIENT_CONFIGS = {
1318
orchestrator: {
@@ -31,7 +36,7 @@ export const MODEL_CLIENT_CONFIGS = {
3136
label: "Action Guard",
3237
defaultValue:
3338
PROVIDER_FORM_MAP[DEFAULT_OPENAI.provider].presets[
34-
"gpt-4.1-nano-2025-04-14"
39+
"gpt-4.1-nano-2025-04-14"
3540
],
3641
},
3742
};
@@ -45,6 +50,9 @@ const AgentSettingsTab: React.FC<SettingsTabProps> = ({
4550
const [advanced, setAdvanced] = useState<boolean>(
4651
(config as any).advanced_agent_settings ?? false
4752
);
53+
const [hasConfigFile, setHasConfigFile] = useState<boolean>(false);
54+
const [configFilePath, setConfigFilePath] = useState<string | null>(null);
55+
const [configContent, setConfigContent] = useState<any>(null);
4856

4957
// Initialize defaultModel from config or detect common model
5058
const initializeDefaultModel = () => {
@@ -105,6 +113,21 @@ const AgentSettingsTab: React.FC<SettingsTabProps> = ({
105113
}
106114
}, [defaultModel]);
107115

116+
// Fetch config info on component mount
117+
useEffect(() => {
118+
const fetchConfigInfo = async () => {
119+
try {
120+
const configInfo = await settingsAPI.getConfigInfo();
121+
setHasConfigFile(configInfo.has_config_file);
122+
setConfigFilePath(configInfo.config_file_path);
123+
setConfigContent(configInfo.config_content);
124+
} catch (error) {
125+
console.error("Failed to fetch config info:", error);
126+
}
127+
};
128+
fetchConfigInfo();
129+
}, []);
130+
108131
// Handle advanced toggle changes
109132
const handleAdvancedToggle = (value: boolean) => {
110133
setAdvanced(value);
@@ -119,6 +142,47 @@ const AgentSettingsTab: React.FC<SettingsTabProps> = ({
119142

120143
return (
121144
<Flex vertical gap="small" justify="start">
145+
{hasConfigFile && (
146+
<Alert
147+
message="LLM Configuration Override"
148+
description={
149+
<div>
150+
<Typography.Text>
151+
Magentic-UI was started with an LLM config file ({configFilePath}).
152+
LLM configurations set here will be ignored as they are overridden by the config file.
153+
</Typography.Text>
154+
{configContent && (
155+
<Collapse
156+
ghost
157+
size="small"
158+
style={{ marginTop: 8 }}
159+
items={[{
160+
key: 'config',
161+
label: 'Show Config Content',
162+
children: (
163+
<div style={{ maxHeight: '300px', overflow: 'auto' }}>
164+
<SyntaxHighlighter
165+
language="json"
166+
style={tomorrow}
167+
customStyle={{
168+
fontSize: '12px',
169+
margin: 0,
170+
}}
171+
>
172+
{JSON.stringify(configContent, null, 2)}
173+
</SyntaxHighlighter>
174+
</div>
175+
)
176+
}]}
177+
/>
178+
)}
179+
</div>
180+
}
181+
type="warning"
182+
showIcon
183+
closable
184+
/>
185+
)}
122186
<Flex gap="small" justify="space-between">
123187
<Flex gap="small" justify="start" align="center">
124188
<Typography.Text>{header}</Typography.Text>

frontend/src/components/settings/tabs/agentSettings/modelSelector/modelConfigForms/OpenAIModelConfigForm.tsx

Lines changed: 11 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -37,31 +37,36 @@ function normalizeConfig(config: any, hideAdvancedToggles?: boolean) {
3737

3838
export const OpenAIModelConfigForm: React.FC<ModelConfigFormProps> = ({ onChange, onSubmit, value, hideAdvancedToggles }) => {
3939
const [form] = Form.useForm();
40+
const baseUrl = Form.useWatch(['config', 'base_url'], form);
41+
42+
const shouldHideAdvanced = (config: any) => {
43+
return hideAdvancedToggles && !(config?.base_url && config.base_url.trim() !== '');
44+
};
4045

4146
const handleValuesChange = (_: any, allValues: any) => {
4247
const mergedConfig = { ...DEFAULT_OPENAI.config, ...allValues.config };
43-
const normalizedConfig = normalizeConfig(mergedConfig, hideAdvancedToggles);
48+
const normalizedConfig = normalizeConfig(mergedConfig, shouldHideAdvanced(mergedConfig));
4449
const newValue = { ...DEFAULT_OPENAI, config: normalizedConfig };
4550
if (onChange) onChange(newValue);
4651
};
4752
const handleSubmit = () => {
4853
const mergedConfig = { ...DEFAULT_OPENAI.config, ...form.getFieldsValue().config };
49-
const normalizedConfig = normalizeConfig(mergedConfig, hideAdvancedToggles);
54+
const normalizedConfig = normalizeConfig(mergedConfig, shouldHideAdvanced(mergedConfig));
5055
const newValue = { ...DEFAULT_OPENAI, config: normalizedConfig };
5156
if (onSubmit) onSubmit(newValue);
5257
};
5358

5459

5560
useEffect(() => {
5661
if (value) {
57-
form.setFieldsValue(normalizeConfig(value, hideAdvancedToggles))
62+
form.setFieldsValue(normalizeConfig(value, shouldHideAdvanced(value.config)))
5863
}
59-
}, [value, form]);
64+
}, [value, form, hideAdvancedToggles]);
6065

6166
return (
6267
<Form
6368
form={form}
64-
initialValues={normalizeConfig(value, hideAdvancedToggles)}
69+
initialValues={normalizeConfig(value, shouldHideAdvanced(value?.config))}
6570
onFinish={handleSubmit}
6671
onValuesChange={handleValuesChange}
6772
layout="vertical"
@@ -81,7 +86,7 @@ export const OpenAIModelConfigForm: React.FC<ModelConfigFormProps> = ({ onChange
8186
<Form.Item label="Max Retries" name={["config", "max_retries"]} rules={[{ type: "number", min: 1, max: 20, message: "Enter a value between 1 and 20" }]}>
8287
<Input type="number" />
8388
</Form.Item>
84-
{!hideAdvancedToggles && (
89+
{!shouldHideAdvanced({ base_url: baseUrl }) && (
8590
<Flex gap="small" wrap justify="space-between">
8691
<Form.Item label="Vision" name={["config", "model_info", "vision"]} valuePropName="checked">
8792
<Switch />

frontend/src/components/views/api.ts

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -443,6 +443,19 @@ export class SettingsAPI {
443443
if (!data.status)
444444
throw new Error(data.message || "Failed to update settings");
445445
}
446+
447+
async getConfigInfo(): Promise<{ has_config_file: boolean; config_file_path: string | null; config_content: any }> {
448+
const response = await fetch(
449+
`${this.getBaseUrl()}/settings/config-info`,
450+
{
451+
headers: this.getHeaders(),
452+
}
453+
);
454+
const data = await response.json();
455+
if (!data.status)
456+
throw new Error(data.message || "Failed to fetch config info");
457+
return data.data;
458+
}
446459
}
447460

448461
export const teamAPI = new TeamAPI();

src/magentic_ui/agents/_coder.py

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -641,6 +641,7 @@ async def load_state(self, state: Mapping[str, Any]) -> None:
641641
"""
642642
# Create message factory for deserialization.
643643
message_factory = MessageFactory()
644+
self._chat_history = []
644645
for msg_data in state["chat_history"]:
645646
msg = message_factory.create(msg_data)
646647
assert isinstance(msg, BaseChatMessage)

src/magentic_ui/agents/file_surfer/_file_surfer.py

Lines changed: 86 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,13 +1,14 @@
11
import json
22
from pathlib import Path
33
import traceback
4-
from typing import List, Sequence, Tuple, AsyncGenerator, Optional, Dict, Any
4+
from typing import List, Sequence, Tuple, AsyncGenerator, Optional, Dict, Any, Mapping
55
from datetime import datetime
66
from loguru import logger
77
import asyncio
88

99
from autogen_agentchat.agents import BaseChatAgent
1010
from autogen_agentchat.base import Response
11+
from autogen_agentchat.state import BaseState
1112
from autogen_agentchat.messages import (
1213
BaseChatMessage,
1314
TextMessage,
@@ -25,7 +26,7 @@
2526
UserMessage,
2627
)
2728
from autogen_core.model_context import TokenLimitedChatCompletionContext
28-
from pydantic import BaseModel
29+
from pydantic import BaseModel, Field
2930
from typing_extensions import Self
3031
from autogen_core.code_executor import CodeExecutor
3132
from autogen_ext.code_executors.docker import DockerCommandLineCodeExecutor
@@ -69,6 +70,27 @@ class FileSurferConfig(BaseModel):
6970
use_local_executor: bool = False
7071

7172

73+
class FileSurferState(BaseState):
74+
"""
75+
State class for saving and loading the FileSurfer's state.
76+
77+
Attributes:
78+
chat_history (List[LLMMessage]): List of chat messages exchanged with the model.
79+
type (str): The type of the state. Default: FileSurferState.
80+
browser_path (str, optional): The current path of the file browser.
81+
browser_title (str, optional): The current title/filename of the file browser.
82+
viewport_current_page (int): The current page in the viewport.
83+
approved_files (List[str]): List of previously approved file paths.
84+
"""
85+
86+
chat_history: List[LLMMessage] = []
87+
type: str = Field(default="FileSurferState")
88+
browser_path: str | None = None
89+
browser_title: str | None = None
90+
viewport_current_page: int = 0
91+
approved_files: List[str] = []
92+
93+
7294
class FileSurfer(BaseChatAgent, Component[FileSurferConfig]):
7395
"""
7496
An agent that can handle files using Markitdown and a code executor.
@@ -608,6 +630,68 @@ def _get_browser_state(self) -> Tuple[str, str]:
608630

609631
return (header, self._browser.viewport)
610632

633+
async def save_state(self) -> Mapping[str, Any]:
634+
"""
635+
Save the current state of the FileSurfer.
636+
637+
Returns:
638+
A dictionary containing the chat history and browser state
639+
"""
640+
if not self.did_lazy_init:
641+
state = FileSurferState(
642+
chat_history=self._chat_history,
643+
browser_path=None, # No browser state when not initialized
644+
browser_title=None,
645+
viewport_current_page=0,
646+
approved_files=list(self._approved_files),
647+
)
648+
else:
649+
state = FileSurferState(
650+
chat_history=self._chat_history,
651+
browser_path=self._browser.path,
652+
browser_title=self._browser.page_title,
653+
viewport_current_page=self._browser.viewport_current_page,
654+
approved_files=list(self._approved_files),
655+
)
656+
return state.model_dump()
657+
658+
async def load_state(self, state: Mapping[str, Any]) -> None:
659+
"""
660+
Load a previously saved state.
661+
662+
Args:
663+
state: Dictionary containing the state to load
664+
"""
665+
# Validate and convert the state to a FileSurferState
666+
file_surfer_state = FileSurferState.model_validate(state)
667+
668+
# Update the chat history
669+
self._chat_history = file_surfer_state.chat_history
670+
671+
# Update approved files
672+
self._approved_files = set(file_surfer_state.approved_files)
673+
674+
# Restore browser state if available and agent is initialized
675+
if self.did_lazy_init and file_surfer_state.browser_path is not None:
676+
# Restore browser path and page if possible
677+
try:
678+
await self._browser.open_path(file_surfer_state.browser_path)
679+
# Try to navigate to the correct page in viewport
680+
if file_surfer_state.viewport_current_page > 0:
681+
target_page = file_surfer_state.viewport_current_page
682+
current_page = self._browser.viewport_current_page
683+
if target_page != current_page:
684+
# Navigate to the correct page
685+
diff = target_page - current_page
686+
if diff > 0:
687+
for _ in range(diff):
688+
self._browser.page_down()
689+
else:
690+
for _ in range(-diff):
691+
self._browser.page_up()
692+
except Exception as e:
693+
logger.warning(f"Failed to restore browser state: {e}")
694+
611695
async def close(self) -> None:
612696
"""Close the FileSurfer agent."""
613697
logger.info("Closing FileSurfer...")

src/magentic_ui/agents/web_surfer/_web_surfer.py

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2090,6 +2090,7 @@ async def load_state(self, state: Mapping[str, Any]) -> None:
20902090

20912091
# Load the browser state if it exists
20922092
if web_surfer_state.browser_state is not None:
2093+
await self.lazy_init()
20932094
assert self._context is not None
20942095
await load_browser_state(self._context, web_surfer_state.browser_state)
20952096

0 commit comments

Comments
 (0)