AI Gateway and MCP security handoff

This is the review entry point for the AI Security team. It documents the production contract between Truffles Keycloak, Portkey AI Gateway (AIRS), the Clippy MCP registration, and the independent authorization check inside clippy-mcp.

Production was verified on 2026-08-19. Start with the architecture, execute the E2E test plan, compare with the production evidence, then use the operations runbook.

:::info Repository authority The canonical repository, issues, pull requests, CI, and container builds are in Forgejo. GitHub is a read-only push mirror used to publish this Docusaurus site. :::

Decision summary

Clippy supports the customer-requested OAuth model: a customer IdP mints an identity JWT with permission to invoke one MCP server; AIRS validates it; AIRS forwards it as a bearer identity; the destination MCP server independently verifies it.

The production permission is mcp.invoke. This value is our policy, not a universal Portkey scope. A customer may use another value if the IdP, AIRS jwt_validation.claimValues, and MCP server authorization policy all agree.

Two authentication modes

Mode	Gateway credential	Per-server identity	Result
Org-level JWKS	Org-claim JWT in x-portkey-api-key	Same or separate JWT in X-Auth-Token	Replaces a Portkey API key when JWT carries Portkey org/workspace claims
JWT Validator Guardrail	Workspace/gateway security key in x-portkey-api-key	IdP JWT in X-Auth-Token	Gateway authentication happens first; server-specific validation happens second

The required production header shape is:

x-portkey-api-key: <gateway credential>
X-Auth-Token: Bearer <external identity JWT>

For the production Org-level JWKS path, the Clippy JWT carries the organization and workspace claims, so the same JWT can fill both logical roles:

x-portkey-api-key: <Clippy org-claim JWT>
X-Auth-Token: Bearer <same Clippy JWT>

For a workspace key plus JWT Validator Guardrail, the values are different:

x-portkey-api-key: <gateway security key>
X-Auth-Token: Bearer <Clippy identity JWT>

Do not put the external identity JWT in the client-facing Authorization header for this registration. X-Auth-Token avoids collision with gateway authentication. After validation, AIRS identity forwarding creates Authorization: Bearer <JWT> on the upstream request to clippy-mcp.

Portkey documents the two-stage model in Bring Your Own Auth and the custom bearer header in JWT Validation.

Production policy

Control	Clippy value	Enforcement
Algorithm	RS256	AIRS and clippy-mcp
Issuer	https://auth.dev.cdot.io/realms/truffles	Exact
JWKS	https://auth.dev.cdot.io/realms/truffles/protocol/openid-connect/certs	Org and per-server validation
Audience	Contains clippy	AIRS and clippy-mcp
Authorized party	Exactly clippy-mcp-client	AIRS and clippy-mcp
Scope	Contains mcp.invoke	AIRS and clippy-mcp
Workspace	Exactly ws-produc-985697	AIRS and clippy-mcp
Required claims	sub, aud, azp, scope, portkey_workspace	AIRS; server also requires exp, iat

The Keycloak token also carries portkey_oid, allowing the JWT to authenticate to the gateway without a separate workspace API key.

User-bound gateway tokens

Every chat turn started from a browser login runs on the person's identity, not on a shared service account. The app never sends the clippy-web login token to AIRS; instead it performs an RFC 8693 standard token exchange (Keycloak 26.2 "standard token exchange") once per route:

Route	Requesting client	Subject token	Result
Inference	clippy-inference-client	the user's clippy-web access token	sub = the user, scope completions.write, aud inference, Portkey org/workspace claims, azp clippy-inference-client, 15-minute lifetime
MCP	clippy-mcp-client	the user's clippy-web access token	sub = the user, scope mcp.invoke, aud clippy, Portkey org/workspace claims, azp clippy-mcp-client, 15-minute lifetime

Keycloak only permits the exchange because clippy-web names both requesting clients in the subject token's aud (two audience mappers) and both clients have standard token exchange enabled. The exchanged token is what fills x-portkey-api-key (inference) and x-portkey-api-key + X-Auth-Token: Bearer (MCP), so the header contract, the registration policy (azp exactly clippy-mcp-client) and clippy-mcp's own verification are unchanged; the gateway and the MCP server now see the user's sub. No refresh token is ever requested from the exchange — the user's own refresh token, encrypted in the session row, is the only long-lived credential, and a refresh that Keycloak rejects ends the Clippy session (reauth).

Permission is a stack role, minted from group membership into resource_access.stack-clippy.roles:

Role	Granted by	Enforced where
completions.write	/stacks/clippy/users, /stacks/clippy/admins	app: a user without it is refused (403) before any token is exchanged
mcp.invoke	/stacks/clippy/users, /stacks/clippy/admins	app: no tools are offered without it; clippy-mcp: MCP_OIDC_REQUIRED_ROLE=stack-clippy:mcp.invoke rejects a forwarded token that lacks it

Removing a person from /stacks/clippy/users therefore revokes their tool access at the server on their next exchange (at most 15 minutes), without a deploy.

Callers with no user token to delegate keep the pre-existing service identity: the local bootstrap admin, and machine callers on the clippy-api bearer path such as the red-team adapter. Those run client_credentials on the same two clients (service-account sub), which is what every turn used before this change. The app log records which kind ran each turn.

Authorization boundaries

• AIRS rejects missing gateway credentials before JWT validation.
• The Clippy registration accepts only the Clippy audience, client, scope, and workspace.
• The Agent Gateway registration accepts only audience agent-gateway and an azp matching ^ag-[0-9a-f]{32}$, with the same scope/workspace/issuer.
• An inference token has completions.write; it cannot call either MCP server.
• A Clippy token cannot call Agent Gateway, and an Agent Gateway token cannot call Clippy.
• clippy-mcp verifies the forwarded bearer independently. A gateway misconfiguration therefore fails closed at the server.
• The direct in-cluster adapter route remains supported, but it bypasses AIRS and must supply a valid Clippy bearer directly. See MCP Tool Adapter.
• The chat API's own clippy-api bearer path is a separate contract and does not authorize MCP.

Chat API bearer path

verifyBearer guards /api/chat for service accounts and the red-team runner: RS256, the realm issuer, and scope containing clippy-api. Audience was historically unchecked there, so any clippy-api token in the realm authenticated regardless of who it was minted for.

M2M_AUDIENCE closes that. When set, aud joins the required claims and must contain the configured value. Production runs M2M_AUDIENCE=stack-clippy. When unset the scope gate stands alone and the app logs one M2M_AUDIENCE unset warning per process. It ships disarmed so the realm and the app can change in either order — arming it against a value the caller does not carry 401s every machine caller, and there is no warn-on-mismatch mode. Armed in production 2026-08-22.

truffles issues a per-stack audience and, for MCP, a per-service one: clippy-m2m carries the bare string stack-clippy, while clippy-mcp-client carries ["clippy", "stack-clippy"]. So stack-clippy would enforce today with no realm change, while clippy-api needs one audience mapper on clippy-m2m and completes the per-service pattern — the recommended option, since it makes aud and scope fail independently across the two paths.

Rollout order is mapper → prove the claim in a freshly minted token → arm the env var, in k8s/m2m-audience-rollout.md. The recommended value is clippy-api, deliberately distinct from the MCP audience clippy, so aud and scope fail independently across the two paths. See Authentication.

Auth0 and other customer IdPs

Auth0, Okta, and other IdPs are supported by the mechanism, but are not automatically trusted. For each issuer, configure:

1. a public JWKS URI and allowed signature algorithms;
2. exact issuer validation;
3. an audience representing the target MCP server;
4. a permission/scope claim such as mcp.invoke;
5. workspace and tenant claims needed for gateway routing;
6. claim-value checks in AIRS;
7. the same checks in the destination MCP server when end-to-end authorization is required.

Current production evidence covers Truffles Keycloak only. A customer Auth0 rollout needs its own registration, token projection tests, positive request, isolated negative-claim tests, and post-restart proof before approval.

Stack and vault boundary

Clippy Chat and AI Security Academy are separate stacks — ships in the night. They share the one truffles Keycloak realm, distinguished by the stack-clippy audience, but nothing for Clippy belongs in an Academy vault, namespace, or Keycloak stack.

Earlier revisions of this handoff said the opposite: they required clippy-mcp-client in AI Security Academy - Runtime, and Forgejo issue #22 tracked moving it there. That was backwards. #22 is closed as invalid.

The genuine drift ran the other way — clippy-postgres, clippy-app, and clippy-mcp-secrets were sourced from the shared AI Security Academy vault. As of 2026-08-22 they reconcile from the dedicated Clippy Chat vault, along with the delivery credentials (Harbor robots, Forgejo PAT and SSH key) that had been filed in AI Security Academy - Automation, and the Keycloak client credentials that had been in the shared realm vault Truffles. Every Clippy secret now has one owner.

Changing an itemPath is not a one-line edit: clippy-app is consumed via envFrom and its secretKeyRefs are non-optional, so a vault the Connect server cannot resolve puts every clippy-chat pod into CreateContainerConfigError — and takes clippy-postgres with it. Prove Connect access with a throwaway canary first. See Operations.

Review acceptance criteria

• Positive token initializes Clippy, lists the exact tool set, and calls a safe tool.
• Missing, malformed, wrong-audience, wrong-scope, wrong-workspace, and cross-server identities fail closed.
• Gateway and identity credentials can be distinct.
• Token values never appear in argv, logs, evidence files, or documentation.
• Tool parity survives AIRS and MCP workload restart.
• Source clients remain disabled only after destination-client configuration parity and token expiry are proven.
• The chat API bearer path enforces aud (M2M_AUDIENCE=stack-clippy), so a clippy-api token minted outside the Clippy stack no longer authenticates.
• Clippy secrets are isolated from AI Security Academy vaults: application, Postgres, and MCP secrets and Keycloak client credentials all source from the dedicated Clippy Chat vault.

See Production evidence for the recorded results.