feat(tools): add LRU cache for read-only tool results#3305
Open
Mattheinrichs wants to merge 2 commits intoHKUDS:nightlyfrom
Open
feat(tools): add LRU cache for read-only tool results#3305Mattheinrichs wants to merge 2 commits intoHKUDS:nightlyfrom
Mattheinrichs wants to merge 2 commits intoHKUDS:nightlyfrom
Conversation
Introduces opt-in in-session caching of results from read-only tools (file reads, glob, grep, web fetch, MCP resources/prompts) to reduce redundant calls and cut context-window token usage. Changes: - config/schema.py: add cacheToolResults (bool, default false) and cacheToolResultsMaxSize (int, default 128) to ToolsConfig - agent/tools/registry.py: add execute_cached() with LRU eviction via OrderedDict; only caches when tool.read_only is True and result is not an error string; add clear_result_cache() helper - agent/runner.py: route _run_tool() through execute_cached() instead of calling tool.execute() directly - agent/loop.py: wire cache config from ToolsConfig into ToolRegistry Tests: - tests/tools/test_tool_registry_cache.py: 9 async tests covering disabled-by-default, cache hits, param isolation, key-order normalisation, mutable-tool bypass, error-not-cached, LRU eviction, LRU order refresh, and cache clear Docs: - README.md: new 'Tool Result Cache' section with config table and JSON example
Follow-up to Stage 2B (feat/tool-result-cache). Adds observability for the in-session tool result cache: - _cache_hits and _cache_misses counters incremented inside execute_cached() for calls that are eligible (cache enabled + read-only tool). Non-eligible calls (mutable tools, cache disabled) are excluded. - cache_stats property returns a dict with hits, misses, eligible, and hit_rate (0.0-1.0). - clear_result_cache() now logs an INFO-level summary (e.g. 'Tool cache stats: 14/20 hits (70%)') before clearing, and resets both counters. Closes the 'cache hit rate' success metric from the token-efficiency spec.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Introduces opt-in, in-session LRU caching for read-only tool results. When enabled, repeated identical calls to idempotent tools (file reads, glob, grep, web fetch, MCP resources/prompts) return the cached result rather than re-executing the tool. This reduces redundant work and cuts token usage when the same resources are accessed multiple times in a session.
The feature is off by default and has no effect on existing behaviour unless explicitly enabled.
Changes
nanobot/config/schema.pycacheToolResults(bool, defaultfalse) andcacheToolResultsMaxSize(int, default128) toToolsConfignanobot/agent/tools/registry.pyexecute_cached()(LRU viaOrderedDict) andclear_result_cache(); only cachesread_only=Truetools; never caches error resultsnanobot/agent/runner.py_run_tool()throughexecute_cached()instead of callingtool.execute()directlynanobot/agent/loop.pycache_results/cache_max_sizefromToolsConfigintoToolRegistryconstructortests/tools/test_tool_registry_cache.pyREADME.mdDesign decisions
cacheToolResults: trueis set in config.read_only = Trueare eligible; write/exec tools are never cached.'Error:'the result is discarded so a retry can succeed.OrderedDict.move_to_end/popitem(last=False)gives O(1) LRU with no extra dependency.(tool_name, json.dumps(params, sort_keys=True))so parameter order from the LLM does not affect hit rate.ToolRegistryinstance; no persistence or cross-session leakage.Testing
One pre-existing test failure (
test_exec_timeout_parameter) uses thesleepshell command which does not exist on Windows; unrelated to this PR.