AI Gateway MCP operations

Production inventory

Item	Value
Realm	`truffles`
Token endpoint	`https://auth.dev.cdot.io/realms/truffles/protocol/openid-connect/token`
JWKS endpoint	`https://auth.dev.cdot.io/realms/truffles/protocol/openid-connect/certs`
Workspace	`ws-produc-985697`
Clippy AIRS route	`https://mcp-airs.cdot.io/ws-produc-985697/clippy/mcp`
Agent Gateway AIRS route	`https://mcp-airs.cdot.io/ws-produc-985697/agent-gateway/mcp`
Clippy upstream	`http://clippy-mcp.clippy.svc.cluster.local:8080/mcp`
Clippy OAuth client	`clippy-mcp-client`
Permission	`mcp.invoke`

AIRS registration contract

The Clippy registration must contain equivalent policy to:

{
  "jwt_validation": {
    "jwksUri": "https://auth.dev.cdot.io/realms/truffles/protocol/openid-connect/certs",
    "algorithms": ["RS256"],
    "headerKey": "X-Auth-Token",
    "requiredClaims": ["sub", "aud", "azp", "scope", "portkey_workspace"],
    "claimValues": {
      "iss": {"values": "https://auth.dev.cdot.io/realms/truffles", "matchType": "exact"},
      "aud": {"values": ["clippy"], "matchType": "contains"},
      "azp": {"values": "clippy-mcp-client", "matchType": "exact"},
      "scope": {"values": ["mcp.invoke"], "matchType": "containsAll"},
      "portkey_workspace": {"values": "ws-produc-985697", "matchType": "exact"}
    }
  },
  "user_identity_forwarding": {
    "method": "bearer"
  }
}

Treat this as a conceptual projection. The Talos cluster repository is authoritative for the exact Portkey/AIRS registration schema used by the installed chart/version.

Destination enforcement

clippy-mcp receives the validated identity as Authorization: Bearer <JWT>. Environment values must stay aligned with the AIRS registration:

MCP_OIDC_ISSUER=https://auth.dev.cdot.io/realms/truffles
MCP_OIDC_AUDIENCE=clippy
MCP_OIDC_AUTHORIZED_PARTY=clippy-mcp-client
MCP_OIDC_REQUIRED_SCOPE=mcp.invoke
MCP_OIDC_WORKSPACE=ws-produc-985697
MCP_OIDC_REQUIRED_ROLE=stack-clippy:mcp.invoke   # optional; rejects forwarded tokens without the stack role

The server derives the JWKS URI from the issuer, permits only RS256, requires expiry/issued-at, and checks aud, azp, scope, and portkey_workspace. /healthz is the only excluded path.

Secret ownership

Clippy Chat and AI Security Academy are separate stacks. They coexist under the one truffles Keycloak realm, but their secret stores are isolated and nothing for Clippy lives in an Academy Conjur branch, vault, namespace, or Keycloak stack.

Source of truth since 2026-09-06: Conjur OSS branch data/clippy/* (talos-cluster conjur/policy/data-clippy.yml), consumed by External Secrets Operator through SecretStore/conjur in the clippy namespace as system:serviceaccount:clippy:eso-conjur. Application, Postgres, MCP tool, Keycloak route-client (clippy-mcp-client, clippy-inference-client) and Harbor pull credentials all live there.
The dedicated Clippy Chat 1Password vault (gphlxcldfinqyzo6jn7sa674sa) is escrow only — scripts/sync-clippy-secrets-to-1password.sh copies live values into it; nothing reconciles from it any more.
Kubernetes Secrets are reconciled objects, not the source of truth.
Never copy secret values into manifests, Argo parameters, Forgejo variables, evidence, or docs.
Never put admin passwords, client secrets, gateway keys, or JWTs in argv.
Use stdin for token request bodies and curl header files (-H @-).

:::info Vault boundary — resolved 2026-08-22 Earlier revisions of this page required clippy-mcp-client in AI Security Academy - Runtime, and Forgejo issue #22 tracked migrating it there. That was backwards: Clippy must not depend on an Academy vault at all. #22 is closed as invalid.

The real drift was the other direction — clippy-postgres, clippy-app, and clippy-mcp-secrets were sourced from the shared AI Security Academy vault. They now come from the dedicated Clippy Chat vault, as do the Keycloak client credentials.

Changing an itemPath is not a one-line edit: clippy-app is consumed via envFrom and its secretKeyRefs are non-optional, so a vault the Connect server cannot resolve puts every clippy-chat pod into CreateContainerConfigError — and clippy-postgres with it. Always prove Connect access with a throwaway canary item first, and keep migration (same values, new source) in a different window from rotation (new value) so a failure is never ambiguous. :::

Granting Connect access to a new vault

The operator runs in Connect mode: OP_CONNECT_HOST=http://onepassword-connect:8080 with a bearer token from the onepassword-token secret. That token is a JWT whose 1password.com/vts claim is the vault allowlist — it cannot be edited, only reissued. 1Password service accounts (ops_…) are a different mechanism and grant the operator nothing.

To add a vault: grant the Connect server integration access to it, issue a new access token covering all required vaults, update onepassword-token (durably, via talos-cluster/onepassword — a hand-patched secret is reverted on the next Argo sync), then restart both deployments so connect-sync pulls the new vault. Verify before relying on it:

kubectl -n onepassword-system port-forward deploy/onepassword-connect 18080:8080 &
TOKEN=$(kubectl -n onepassword-system get secret onepassword-token -o jsonpath='{.data.token}' | base64 -d)
curl -s -H "Authorization: Bearer ${TOKEN}" http://127.0.0.1:18080/v1/vaults | jq -r '.[].name'

A token only ever gets token scope ∩ server vault access, so checking the decoded JWT alone is not sufficient — query the running server.

Rotation procedure

Rotate the credential only in the Clippy Chat vault, which owns every Clippy secret.
Wait for clippy-mcp-client Secret reconciliation.
Prove the app can mint a token from the live Secret and verify selected claims.
Run initialize, tools/list, and safe tools/call through AIRS.
Run direct internal app-to-MCP coverage.
Restart clippy-chat; confirm 2/2 Ready and no token-mint errors.
Repeat the positive and negative matrix.
Retain old credential only for the approved overlap window; then revoke it.

Changing a confidential-client secret does not invalidate already issued JWTs. Plan for the configured token lifetime and AIRS validation-cache behavior.

JWKS key rotation

Portkey caches JWKS, and clippy-mcp caches the JWK set for five minutes. During rotation:

publish the new signing key before issuing tokens with its kid;
retain the old public key until all old tokens and caches expire;
mint a new token and run the full positive path;
verify malformed/old-key tokens fail after retirement;
remove the old key only after the overlap window.

Never solve a rotation problem by disabling claim or signature validation.

Deployment procedure

Clippy is a Forgejo repository. The normal path is:

Forgejo issue and cdot65/ branch;
tests and review in Forgejo;
Forgejo CI builds immutable digest-pinned images in Harbor;
manifest commit references the published digest;
Argo CD syncs the immutable manifest;
rollout readiness completes;
execute E2E testing;
record production evidence.

GitHub is only a push mirror and Docusaurus publication target. Do not open the authoritative implementation issue or pull request there.

Troubleshooting by status

Symptom	Likely boundary	Checks
`401 Authentication required`	Gateway authentication	`x-portkey-api-key` present; org claims/key valid
`401` with both headers	JWT validation	Bearer prefix, signature, issuer, audience, `azp`, scope, expiry
`403 wrong workspace`	Routing/tenant policy	URL workspace equals `portkey_workspace` and registration
`500` only after prior token use	AIRS validation cache + upstream denial	Confirm destination logged `401`; retry fresh token
`500` for positive request	Upstream/service failure	AIRS logs, endpoints, MCP readiness, destination auth logs
`200` initialize but empty tools	MCP protocol/readiness defect	Fail immediately; inspect tools/list body and server logs
App reports token request failure	Keycloak or reconciled Secret	token URL/client ID, Secret age, client enabled, credential parity
App works; external AIRS fails	Gateway registration/header contract	Compare AIRS policy, route, and `X-Auth-Token: Bearer`
AIRS works; direct app route fails	Destination enforcement/app credentials	app token claims and `clippy-mcp` environment

Logs without credential leakage

Safe fields:

timestamp, route, HTTP status, request correlation ID;
selected claim names/decisions, never the encoded JWT;
issuer, audience, client ID, scope names, workspace;
image digest, pod name, restart count;
MCP method/tool name and JSON-RPC error code.

Unsafe fields:

raw headers, form bodies, full token responses;
cookies, client secrets, gateway keys, admin credentials;
decoded claims containing customer personal data unless specifically approved.

Search application logs for stable error classes, not tokens:

kubectl -n clippy logs deployment/clippy-mcp --since=15m |
  grep -E 'unauthorized|invalid forwarded token|missing required scope'
kubectl -n clippy logs deployment/clippy-chat --since=15m |
  grep -E 'MCP token request failed|invalid MCP token response'

Silent publishing failure

Observed 2026-08-21 → 2026-08-22. build-and-push declares needs: [application, kubernetes, mcp]. When mcp fails the job is skipped, and Forgejo reports a skipped job as success in the combined commit status. The pipeline stops shipping while every check reads green.

Root cause that time: mcp-server/tests/test_manifest.py pins the app image by tag and digest as literal constants, and a manifest re-pin did not update them. Harbor held no artifact for any commit after 3eed1fd for roughly a day, and nothing surfaced it.

The registry is the ground truth, not the commit status. Check it with the cluster's own read-only pull robot:

auth=$(kubectl -n clippy get secret harbor-pull-secret \
  -o jsonpath='{.data.\.dockerconfigjson}' | base64 -d | jq -r '.auths[].auth')
tok=$(curl -s -H "Authorization: Basic ${auth}" \
  'https://registry.cdot.io/service/token?service=harbor-registry&scope=repository:clippy/clippy-chat:pull' \
  | jq -r .token)
curl -s -H "Authorization: Bearer ${tok}" \
  https://registry.cdot.io/v2/clippy/clippy-chat/tags/list | jq -r '.tags[]' | grep '^sha-'

Whenever k8s/20-app.yaml is re-pinned, update test_manifest.py in the same commit.

Rollback conditions

Rollback or stop the rollout if any occurs:

a positive request is not HTTP 200;
exact tools differ before/after restart;
a missing or mismatched credential reaches a tool;
Clippy/Agent/inference identities cross stack boundaries;
destination credentials or required mappings are not exact;
a Secret cannot reconcile from its intended vault;
evidence includes secret material;
readiness exceeds its bounded deadline.

Rollback must restore the complete captured state: AIRS registration, JWKS configuration, workload token URLs/registries, Argo application state, annotations, and source-client status. After rollback, rerun the previous realm's positive matrix and all cross-stack denials.

Incident containment

For suspected credential exposure:

identify which layer leaked: gateway key, client secret, or issued JWT;
revoke/rotate only the exact credential first;
disable the affected client if active abuse is possible;
preserve secret-free request IDs/timestamps for investigation;
confirm both AIRS and destination denials with newly invalid credentials;
restore via 1Password/Keycloak declarative ownership;
execute the complete E2E plan before reopening access.

Do not publish captured credentials to Forgejo, GitHub, Obsidian, chat, or incident tickets.

Production inventory​

AIRS registration contract​

Destination enforcement​

Secret ownership​

Granting Connect access to a new vault​

Rotation procedure​

JWKS key rotation​

Deployment procedure​

Troubleshooting by status​

Logs without credential leakage​

Silent publishing failure​

Rollback conditions​

Incident containment​