Skip to content

feat: Token budget aware fresh tail#236

Open
Bikkies wants to merge 1 commit intoMartian-Engineering:mainfrom
Bikkies:feat/token-budget-tail
Open

feat: Token budget aware fresh tail#236
Bikkies wants to merge 1 commit intoMartian-Engineering:mainfrom
Bikkies:feat/token-budget-tail

Conversation

@Bikkies
Copy link
Copy Markdown

@Bikkies Bikkies commented Apr 2, 2026

This basically just adds a way for us to force compact older messages when we actually reach the configured compaction threshold, instead of the blunt "keep last 64 messages no matter what" behaviour that could lead to overflows when using models with lower context lengths. Now we TRY to keep the last 64 (or whatever you configure) messages, but if they really don't fit, we gotta compact.

freshTailCount protects the last N messages from compaction regardless of size. With the default of 64, a handful of large tool results in the tail can exceed the context window and there's no way to compact them out, even when context is about to overflow.

This is common in agentic workflows where tool results vary wildly in size. One message might be 50 tokens and the next one might be 30k.

This adds a token-count cap to the fresh tail using the existing contextThreshold config. Walks backward from the newest message, accumulating tokens. Once the budget (contextThreshold * contextWindow) is hit, older messages lose protection and become compactable regardless of how many you try to protect with freshTailCount

@Bikkies Bikkies force-pushed the feat/token-budget-tail branch 2 times, most recently from faa57b9 to 8891a7e Compare April 2, 2026 10:59
@Bikkies
Copy link
Copy Markdown
Author

Bikkies commented Apr 2, 2026

2 tests fail because they use token budgets small enough that the fresh tail exceeds the compaction threshold which is the scenario this PR addresses.

- Fixed two failing tests by disabling token-budget tail for those specific cases so they test stock compaction behavior
- Added companion tests that verify the token-budget tail feature works as expected

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@Bikkies Bikkies force-pushed the feat/token-budget-tail branch from 8891a7e to 155b7bd Compare April 7, 2026 01:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant