Skip to content

feat: OTel budget circuit-breaker for AWS AgentCore integration#2

Open
up2itnow0822 wants to merge 2 commits intomainfrom
feat/otel-budget-circuit-breaker
Open

feat: OTel budget circuit-breaker for AWS AgentCore integration#2
up2itnow0822 wants to merge 2 commits intomainfrom
feat/otel-budget-circuit-breaker

Conversation

@up2itnow0822
Copy link
Copy Markdown
Owner

Summary

Adds an OpenTelemetry Budget Circuit-Breaker module that fills the gap left by AWS AgentCore Policy Controls (GA March 2026): observability without spend enforcement.

New MCP Tools

Tool Purpose
otel_register_budget_policy Set per-agent/per-task spend limits with warn/block/kill breach actions
otel_evaluate_spend Evaluate OTel span cost data against policies, returns enforcement decisions
otel_budget_status Query accumulated spend, remaining budget, utilization %, and decision history

How It Works

  1. Agent runs on AWS AgentCore emit OTel spans with cost attributes (agentcore.cost.usd, gen_ai.usage.cost)
  2. agentpay-mcp reads these spans and evaluates against registered budget policies
  3. Decisions (allow/warn/block/kill) are emitted as OTel-compatible events for AgentCore dashboard visibility
  4. Circuit-breaker pattern: kill action supports webhook callbacks to terminate runaway agent runs

Architecture

AWS AgentCore (observability) → OTel spans with cost data
                                     ↓
agentpay-mcp otel-budget module → evaluate against policies
                                     ↓
                              Budget decision (allow/warn/block/kill)
                                     ↓
                              OTel event emission → AgentCore dashboard

Test Coverage

Comprehensive test suite covering:

  • Policy registration (agent-level and task-level)
  • Budget evaluation lifecycle (allow → warn at 90% → block/kill at 100%)
  • Task-level policy precedence
  • OTel event attribute generation
  • Decision history audit trail

Context

  • AWS AgentCore Policy Controls GA'd in March 2026 with NO native per-agent spend caps
  • This positions agentpay-mcp as the budget enforcement layer above AgentCore's telemetry pipeline
  • Refs: INTEL-CYCLE-196, INTEL-CYCLE-198

Adds three new MCP tools for budget enforcement above AgentCore's
observability layer:

- otel_register_budget_policy: Set per-agent/per-task spend limits
  with configurable breach actions (warn/block/kill)
- otel_evaluate_spend: Evaluate OTel span cost data against policies,
  returns enforcement decisions as OTel-compatible events
- otel_budget_status: Query accumulated spend and policy utilization

Why: AWS AgentCore Policy Controls (GA March 2026) provide observability
but no native per-agent spend cap APIs. This module fills that gap by
reading OTel span data and applying budget policies with circuit-breaker
patterns.

Includes comprehensive test suite covering:
- Policy registration (agent-level and task-level)
- Budget evaluation (allow/warn/block/kill decisions)
- 90% utilization warning threshold
- Task-level policy precedence over agent-level
- OTel event attribute generation
- Decision history and audit trail

Refs: INTEL-CYCLE-196, INTEL-CYCLE-198
Copy link
Copy Markdown

@chatgpt-codex-connector chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 696a4aa897

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/tools/otel-budget.ts Outdated
Comment thread src/tools/otel-budget.ts
Copy link
Copy Markdown

@chatgpt-codex-connector chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 0e66c3c964

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/tools/otel-budget.ts
Comment on lines +118 to +124
return _spendLedger
.filter(
(r) =>
r.agentId === agentId &&
(taskId === undefined || r.taskId === taskId) &&
r.timestamp >= cutoff
)
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Bound spend ledger growth to active policy windows

Each call to evaluateSpan appends to _spendLedger, and getAccumulatedSpend then scans the full array for every evaluation. Because old records are never pruned, long-running/high-throughput agents will see steadily increasing latency and memory usage, and can eventually OOM even when most entries are already outside every policy window and no longer relevant.

Useful? React with 👍 / 👎.

Comment thread src/tools/otel-budget.ts
Comment on lines +165 to +166
budgetLimitUsd: Infinity,
remainingUsd: Infinity,
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Avoid non-JSON numeric sentinels in allow decisions

The no-policy branch sets budgetLimitUsd and remainingUsd to Infinity, but handleOTelEvaluateSpend serializes decisions with JSON.stringify, which converts Infinity to null. Clients expecting numeric budget fields will receive null and can mis-handle arithmetic or schema validation in the common case where no policy is registered.

Useful? React with 👍 / 👎.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant