[EPIC][AUTH]: Agent and tool authentication and authorization plugin

[imolloy]

Description

Implement plugins to handle authentication and delegation flows and credential injection. These are between agents and tools and agents and other agents, and are not gateway administrative actions. Ensure compliance with the latest MCP standard around authorization.

Tasks

• ContextForge
  • Create JWT claim extraction plugin leveraging the gateway's auth hooks [FEATURE][PLUGIN]: Create JWT claims and metadata extraction plugin #1439
  • Extract claims from JWTs from transport router to be passed to plugins via context [FEATURE][AUTH]: Propagate end user identity and context through the CF workflow #1436
  • Implement authorization server metadata (RFC8414) to identity (and passthrough) the necessary authorization server [FEATURE][AUTH]: Infer identity provider info for onboarded MCP servers #1435
  • Secure secret storage for secrets and keys for registered MCP servers
  • Create convenience library for oauth flows [FEATURE][AUTH]: Comprehensive OAuth2 base library with helper functions #1434
• Plugins
  • Implement token exchange oauth flow, with least privilege scopes [FEATURE][AUTH]: Comprehensive OAuth2 base library with helper functions #1434
  • Inject authorization credentials using http hooks [FEATURE][PLUGIN]: Create IAM pre-tool plugin #1437
  • Human-in-the-loop authorization flows [FEATURE][PLUGIN]: Enhance the IAM pre-tool plugin #1438
  • Enhanced oauth flows and RFC [FEATURE][PLUGIN]: Enhance the IAM pre-tool plugin #1438

[VRamakrishna]

Here is our envisioned augmentation to the MCP CF gateway system architecture to fulfil the tasks listed above.

graph TD
    
    classDef highlight fill:#79D25A,stroke:#46A228,stroke-width:3px
    classDef noWrapStyle white-space:nowrap;
    classDef noWrapStylehighlight fill:#aaddaa,stroke:#668866,stroke-width:3px,white-space:nowrap;
    classDef exthighlightg fill:#FFB6C1,stroke:#FF69B4,stroke-width:3px
    classDef exthighlightb fill:#FF69B4,stroke-width:3px

    subgraph Clients
        ui["Admin UI (Browser)"]
        cli["CLI Tools"]
        sdk["SDK / Scripts"]
    end

    subgraph IdentityProviders
        service["IdP"]
    end

    subgraph Gateway
        app["FastAPI App"]
        auth["Auth Middleware<br/>(JWT + Basic)"]
        subgraph router["Transport Router (HTTP / WS / SSE / STDIO)"]
            idpc[IdP Connectors]
            th[Transport Handler]
        end
        subgraph services["Service Layer (Tool / Resource / Prompt / Server)"]
            handler["Callback Handler"]
        end
        subgraph plugins["Plugin Framework"]
            rcaplugin["Rich Context Authorization"]
            hilplugin["Human-in-loop Handler"]
            authplugin["Auth Injection"]
        end
        subgraph cim["Credential and Identity Manager"]
            oauthflowmgr["OAuth Flow Manager"]
            samlflowmgr["SAML Flow Manager"]
            csam["Credentials Storage and Management"]
            adminapi["Idp Admin API Support"]
            tv["Token Validations"]
            cmapi["Credentials Management API"]
        end
        subgraph db["Async DB (SQLAlchemy + Alembic)"]
            toolinfo["Tool Resource Augmenter"]
            idpinfo["IdP Info"]
            authinfo["Tool Auth Info"]
            credinfo["Credentials"]
        end
        cache["Cache Backend<br/>(memory / redis / db)"]
        metrics["Metrics Exporter<br/>(/metrics Prometheus)"]
    end
    
    vault["Vault"]

    subgraph Federation
        discovery["Discovery Service<br/>(DNS-SD + Static Peers)"]
        peers["Remote Gateways"]
    end

    ui --> app
    cli --> router:::noWrapStyle
    sdk --> router
    app --> auth --> router
    router --> services
    services:::noWrapStyle --> db:::noWrapStyle
    services --> cache
    services --> metrics
    services --> discovery
    discovery --> peers
    idpc:::highlight --> service:::exthighlightb
    th:::highlight
    csam:::highlight --> vault:::exthighlightg
    services --> cmapi:::highlight
    services --> adminapi:::highlight
    services --> plugins
    handler:::highlight
    oauthflowmgr:::highlight
    samlflowmgr:::highlight
    tv:::highlight
    rcaplugin:::highlight
    hilplugin:::highlight
    authplugin:::highlight
    toolinfo:::highlight
    idpinfo:::highlight
    authinfo:::highlight
    credinfo:::highlight
    cim:::noWrapStylehighlight
    IdentityProviders:::exthighlightg

Loading

The following are independent components we need to add:

• Credential and Identity Manager
• Plugins for token acquisition and policy enforcement

The following CF components need to be augmented:

• Transport router
• Service layer
• Async DB

[VRamakrishna]

The following set of issues track individual features required to fulfil this epic:

• [FEATURE][AUTH]: Comprehensive OAuth2 base library with helper functions #1434
• [FEATURE][AUTH]: Infer identity provider info for onboarded MCP servers #1435
• [FEATURE][AUTH]: Propagate end user identity and context through the CF workflow #1436
• [FEATURE][PLUGIN]: Create IAM pre-tool plugin #1437

[aksharkaul]

The following set of issues track individual features required to fulfil this epic:

• [FEATURE][PLUGIN]: Enhance the IAM pre-tool plugin #1438

[araujof]

Looping @madhav165 here. He's looking into creating a plugin that registers to our auth hooks to extract token claims/metadata and map those into a reserved context key. This will enable custom and fine grained access control policies via our RBAC and OPA plugins.

[araujof]

Created #1439

@imolloy I'm not sure if #1436 is a duplicate or required for the internal auth flows in the gateway.

[terylt]

Awesome! Let me know if you need anything. These plugins may or may not be useful as references:

https://github.com/IBM/mcp-context-forge/tree/main/plugins/examples
https://github.com/IBM/mcp-context-forge/tree/main/plugins/vault

Here are the docs on the authentication hooks:

https://ibm.github.io/mcp-context-forge/using/plugins/http-auth-hooks/