Feature request: adopt A0 to self-hosted LLM's

[ABLomas]

When running A0 with self-hosted LLM's (llama.cpp on strix-halo, 124GB unified VRAM) i encountered many issues, most of them making A0 unusable for various reasons. I try to do summary of what is required:

1. Context length limitation is completely ignored in all possible ways (this is actually regression, was better in 1.4 or so):
   By all logic (including discord support bot, which, i believe, runs gemini-flash) - utility LLM should have small context window (it gets just small pieces of work), but instead - if gets called first, then gets entire question, memory context and tries to summarize. So if we set context window on model to 8K and cap usage to 0.5 in UI - it still overflows immediately, leading to memory errors.
   Same situation occurs on late phase, when memorisation is called - entire chat context is feeded into utility worker, independent on context settings and memorizing fails
   The only way to get around this "initial call" is to set utility worker context to 256K (64K, 128K works with smaller work but still fails on later phase)
   Entire thing works well with commercial API's as most of them have huge context window (which is wrapped asap later), but on local LLM's it fails.
   Proposal: make context limit cap actually matter, do not feed entire chat history and prompts to utility agent - let chat agent do that work.
2. There's no way to trace "which agent executed what". UI does not make remarks which agent is doing query/job and logfile (HTML!!!) does not contains timestamps (to match with litellm or llama.cpp logs). All we get is "something failed somewhere", with no actual query executed.
   Proposal:
   add symbol to UI next to query to indicate which agent is doing job (for example, "USE A0**-U**: Using tool 'text_editor<...>"
   add timestamps to log file
   if error occured - display LLM response or prompt, not only stack trace (it is in log file, but again, searching HTML log file isn't easiest thing to do)
3. fix tool usage (already addressed in Bug: ValueError crash when LLM outputs "args" instead of "tool_args" #1512 )
4. expose prompt configuration to UI, or at least - make easier to insert basic instructions. Proposal: add one field on agent configuration with text input - contents will be prepended to constructed system prompt (for each agent individually)

My test case to check how it's working - LLM box, VM nearby with docker containers. Each time starting new project, adding only IP's to memories. Test query:

create "wordpress" installation on docker containers:

• 1 container for database + wordpress webserver
• 1 container for reverse proxy
  expose reverse proxy outside to port 8233, leave other container unexposed.
  Keep resources at minimum, we aim for 10 visitors/day.
  Wordpress should run at default settings, show it's website (not configuration dialog), show administrator password in chat
  Verify result with curl before returning with success status.
  Mandatory tasks:
  a) plan task execution and list it in chat. Draw ASCII diagram how everything is connected
  b) verify that content is actually served before confirming to user

It's really trivial for hosted work, but currently impossible to do on self-hosted - i tested 32 models in total. Best combo which was working on anything smaller so far:

1. NvidiaNemotronCascade2-30B (utility LLM, context size varies)
2. Qwen3-Coder-Next-80B (chat LLM - 256K context)
3. qwen3-embeddings-06B

This combo can do most smaller projects, but fails on query above with various conditions:

1. if i give 8K context window (on llama.cpp side and in GUI) to utility worker - it fails immediately, due to oversized context (feeding memory with all container paths, keys + task prompt + prepared system prompt = easy overflow)
2. if i give 256K context window to utility worker - it can process all prompts, all fine - but fails on later phases due to tool usage (let's call this technical limitation - model can't "focus attention" with huge context window)
3. it's possible to give ~64K of context window, reduce temperature and top-k - to go until last phase (verification) where it fails due to mistake in reverse proxy config. Then it enters endless loop where utility worker (? - again, it's unclear due to missing info which is executing what) is writing plan and chat agent rejects it because it's the same as previously and not enough temperature to break out of loop.

Bot hallucinations about context question, not sure if this matters: https://discord.com/channels/1255926983745998948/1493854452900429824

[GozerZeGozerian]

I second this experience on a self-hosted setup with llama.cpp as the backend. I have tried several things to make the utility model actually honor the context limit or percentage of I set for it but haven't found a working solution yet. I've spent way too much time messing with it, tbh. Anything that involves heavy utility use is basically non-functional for me, which is a shame. Oh, and this utility issue also breaks the native memory plugin for me, so I had to disable it completely and create my own memory plugin. It's nice I can create plugins to fix core issues, but it seems to me like that shouldn't be necessary.

The, "no way to trace "which agent executed what", thing is a frustration for me as well. I resort to running commands in my host terminal to pull related logs so I can diagnose and attempt to resolve. Would be much more efficient and user-friendly if all relevant info was printed inline.

For the prompt configuration I just ask Agent0 to do it for me, that has worked fairly well and might be what the devs intended.

It's currently a bit clunky for self-hosted users, I hope to see some improvements for us soon. Good luck with yours!

[ABLomas]

Yes, logging is probably worst part. Just re-ran task above with slightly different params:

OK, something somewhere timeouted, maybe logs will help to trace root cause?

  File "/a0/agent.py", line 551, in prepare_prompt
    await extension.call_extensions_async(
  File "/a0/helpers/extension.py", line 235, in call_extensions_async
    await result
  File "/a0/plugins/_memory/extensions/python/message_loop_prompts_after/_91_recall_wait.py", line 30, in execute
    await task
  File "/opt/pyenv/versions/3.12.4/lib/python3.12/asyncio/futures.py", line 287, in __await__
    yield self  # This tells Task to wait for completion.
    ^^^^^^^^^^
  File "/opt/pyenv/versions/3.12.4/lib/python3.12/asyncio/tasks.py", line 385, in __wakeup
    future.result()
  File "/opt/pyenv/versions/3.12.4/lib/python3.12/asyncio/futures.py", line 203, in result
    raise self._exception.with_traceback(self._exception_tb)
  File "/opt/pyenv/versions/3.12.4/lib/python3.12/asyncio/tasks.py", line 316, in __step_run_and_handle_result
    result = coro.throw(exc)
             ^^^^^^^^^^^^^^^
  File "/opt/pyenv/versions/3.12.4/lib/python3.12/asyncio/tasks.py", line 519, in wait_for
    async with timeouts.timeout(timeout):
  File "/opt/pyenv/versions/3.12.4/lib/python3.12/asyncio/timeouts.py", line 115, in __aexit__
    raise TimeoutError from exc_val
TimeoutError


A0:

Error: Memory consolidation timeout for area fragments

Error: Memory consolidation timeout for area solutions

Not even single mentioning what happens. It text_editor:write did it's job, configs are here, but then something timeouted on something and everything stops, nudging does not help.
If we had timestamps, we could at least mix-match with llama logs:

srv  proxy_reques: proxying request to model NvidiaNemotronCascade2-30B on port 45895
<...>
reasoning-budget: deactivated (natural end)
slot print_timing: id  2 | task 34010 |
prompt eval time =     778.50 ms /   595 tokens (    1.31 ms per token,   764.29 tokens per second)
eval time =  125485.26 ms /  5910 tokens (   21.23 ms per token,    47.10 tokens per second)
total time =  126263.76 ms /  6505 tokens
slot      release: id  2 | task 34010 | stop processing: n_tokens = 6599, truncated = 0
srv  update_slots: all slots are idle
srv  log_server_r: done request: POST /v1/chat/completions 127.0.0.1 200
srv    operator(): http client error: Connection handling canceled

(all lines prefixed with datetime, i skipped for clarity)
...but without precise date is stil unclear what done who - utility agent finished responding then something failed? But why?
For debugging i had llama in very verbose mode, so i can dump queries, but concurrency (-np 3) makes this impractical anyway.

So, like i posted above in point 2 - maybe just very trivial fix - adding letter indicating model type to log would help to debug issues. At least, it would be clear which model failed it's job (in last case i don't think model failed, there's something else happening).

[ABLomas]

Also, as proof that context size is ignored:

srv  proxy_reques: proxying request to model NvidiaNemotronCascade2-30B on port 45895
<...>
Apr 15 11:48:20 tb distrobox[3773900]: [45895] decode: failed to find a memory slot for batch of size 1
Apr 15 11:48:20 tb distrobox[3773900]: [45895] srv  update_slots: Context size has been exceeded. i = 0, n_batch = 1, ret = 1
Apr 15 11:48:20 tb distrobox[3773900]: [45895] srv    send_error: task id = 41852, error: Context size has been exceeded.
Apr 15 11:48:20 tb distrobox[3773900]: [45895] slot      release: id  1 | task 41852 | stop processing: n_tokens = 33912, truncated = 0
Apr 15 11:48:20 tb distrobox[3773900]: [45895] slot prompt_clear: id  1 | task -1 | clearing prompt with 33912 tokens
Apr 15 11:48:20 tb distrobox[3773900]: [45895] srv          stop: cancel task, id_task = 41852

So, with 32K context window if failed on third utility call (first on was to make system prompt for chat agent i guess, then "config write" and then third one to check config again after nudging ended with overflow) Increasing model context size just moves issue to lates phase, but does not solve it completely.

[edhooksjr-ai]

Local LLM hosting is the dream, and i am blaming Ollama for ignoring all Context Limits in API Calling. Until i manually loaded qwen3-coder... model into ollama and /set parameter num_ctx 48000 incould not successfully complete an A0 - hello, what model is this? Check. If the many LLM
Parameter Control Settings can work then keep them, otherwise i vote to Fix them. If Ollama is breaking the parameters, i would suggest remove all the complexity from the Agent Settings. 🙏🏻