Skip to content

feat(tools): add LRU cache for read-only tool results#3305

Open
Mattheinrichs wants to merge 2 commits intoHKUDS:nightlyfrom
Mattheinrichs:feat/tool-result-cache
Open

feat(tools): add LRU cache for read-only tool results#3305
Mattheinrichs wants to merge 2 commits intoHKUDS:nightlyfrom
Mattheinrichs:feat/tool-result-cache

Conversation

@Mattheinrichs
Copy link
Copy Markdown

Summary

Introduces opt-in, in-session LRU caching for read-only tool results. When enabled, repeated identical calls to idempotent tools (file reads, glob, grep, web fetch, MCP resources/prompts) return the cached result rather than re-executing the tool. This reduces redundant work and cuts token usage when the same resources are accessed multiple times in a session.

The feature is off by default and has no effect on existing behaviour unless explicitly enabled.

Changes

File What changed
nanobot/config/schema.py Added cacheToolResults (bool, default false) and cacheToolResultsMaxSize (int, default 128) to ToolsConfig
nanobot/agent/tools/registry.py Added execute_cached() (LRU via OrderedDict) and clear_result_cache(); only caches read_only=True tools; never caches error results
nanobot/agent/runner.py Route _run_tool() through execute_cached() instead of calling tool.execute() directly
nanobot/agent/loop.py Pass cache_results/cache_max_size from ToolsConfig into ToolRegistry constructor
tests/tools/test_tool_registry_cache.py 9 async unit tests: disabled-by-default, cache hit, param isolation, key-order normalisation, mutable-tool bypass, error-not-cached, LRU eviction, LRU order refresh, cache clear
README.md New Tool Result Cache section with config table and JSON example

Design decisions

  • Opt-in - zero behaviour change unless cacheToolResults: true is set in config.
  • read_only gate - only tools that declare read_only = True are eligible; write/exec tools are never cached.
  • Errors not cached - if a tool returns a string starting with 'Error:' the result is discarded so a retry can succeed.
  • LRU eviction - OrderedDict.move_to_end / popitem(last=False) gives O(1) LRU with no extra dependency.
  • Cache key - (tool_name, json.dumps(params, sort_keys=True)) so parameter order from the LLM does not affect hit rate.
  • In-memory, per-session - cache lives with the ToolRegistry instance; no persistence or cross-session leakage.

Testing

uv run pytest tests/tools/test_tool_registry_cache.py -v  # 9 passed
uv run pytest tests/ -q                                    # 1924 passed, 12 skipped

One pre-existing test failure (test_exec_timeout_parameter) uses the sleep shell command which does not exist on Windows; unrelated to this PR.

Introduces opt-in in-session caching of results from read-only tools
(file reads, glob, grep, web fetch, MCP resources/prompts) to reduce
redundant calls and cut context-window token usage.

Changes:
- config/schema.py: add cacheToolResults (bool, default false) and
  cacheToolResultsMaxSize (int, default 128) to ToolsConfig
- agent/tools/registry.py: add execute_cached() with LRU eviction via
  OrderedDict; only caches when tool.read_only is True and result is
  not an error string; add clear_result_cache() helper
- agent/runner.py: route _run_tool() through execute_cached() instead of
  calling tool.execute() directly
- agent/loop.py: wire cache config from ToolsConfig into ToolRegistry

Tests:
- tests/tools/test_tool_registry_cache.py: 9 async tests covering
  disabled-by-default, cache hits, param isolation, key-order
  normalisation, mutable-tool bypass, error-not-cached, LRU eviction,
  LRU order refresh, and cache clear

Docs:
- README.md: new 'Tool Result Cache' section with config table and
  JSON example
Follow-up to Stage 2B (feat/tool-result-cache).

Adds observability for the in-session tool result cache:

- _cache_hits and _cache_misses counters incremented inside
  execute_cached() for calls that are eligible (cache enabled +
  read-only tool). Non-eligible calls (mutable tools, cache disabled)
  are excluded.
- cache_stats property returns a dict with hits, misses, eligible,
  and hit_rate (0.0-1.0).
- clear_result_cache() now logs an INFO-level summary
  (e.g. 'Tool cache stats: 14/20 hits (70%)') before clearing, and
  resets both counters.

Closes the 'cache hit rate' success metric from the token-efficiency spec.
@Re-bin Re-bin deleted the branch HKUDS:nightly April 19, 2026 16:22
@Re-bin Re-bin closed this Apr 19, 2026
@Re-bin Re-bin reopened this Apr 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants