NeMo Guardrails + MCP Gateway Integration

Preliminary design document for securing MCP Gateway traffic with NeMo Guardrails, orchestrated by the TrustyAI operator.

Problem Statement

MCP Gateway processes Model Context Protocol requests (tool calls, prompt fetches) through Envoy. Today, NeMo Guardrails deploys as a standalone service with no awareness of the gateway. Wiring them together requires manual deployment of the Envoy Guardrails Plugin Adapter and manual creation of EnvoyFilter resources.

The goal is to let the TrustyAI operator handle this wiring automatically: when a user specifies a gateway reference in the NemoGuardrails CR, the operator deploys the adapter and creates the Envoy integration resources, connecting guardrails to the gateway's request/response pipeline.

Components

Component	Role	Current Owner
TrustyAI Operator	Deploys and manages NemoGuardrails CRs	`trustyai-service-operator/`
NeMo Guardrails	Guardrails engine (PII detection, content safety, etc.)	`NeMo-Guardrails/`
MCP Gateway	Envoy-based gateway for MCP protocol (broker, router, controller)	`mcp-gateway/`
Envoy Guardrails Plugin Adapter	ext-proc bridging Envoy to NeMo Guardrails	`envoy-guardrails-plugin-adapter/`

Current Deployment Topology (Before Integration)

Namespace: gateway-system
  └── Gateway/mcp-gateway (Istio-managed Envoy proxy)

Namespace: istio-system
  └── EnvoyFilter (injects MCP router ext_proc into Gateway)

Namespace: mcp-system
  ├── Deployment/mcp-broker-router (Broker :8080, Router ext_proc :50051)
  ├── Deployment/mcp-controller
  ├── ConfigMap/mcp-gateway-config
  └── Secret/mcp-aggregated-credentials

Namespace: <user-namespace>  (e.g., my-app)
  └── NemoGuardrails CR → Operator creates:
      ├── Deployment/<cr-name>         (NeMo server on :8000)
      ├── Service/<cr-name>            (ClusterIP, port 80 → 8000)
      ├── ConfigMap/<cr-name>-ca-bundle
      └── Route/<cr-name>              (OpenShift only)

Key point: in the current state, NeMo Guardrails and MCP Gateway are completely independent. The Envoy Guardrails Plugin Adapter exists as a manual deployment with no operator management.

Target Deployment Topology (With Integration)

When spec.gatewayRef is set on the NemoGuardrails CR:

Namespace: gateway-system
  └── Gateway/mcp-gateway (Envoy proxy, now with TWO ext_proc filters)

Namespace: istio-system
  ├── EnvoyFilter/mcp-router-extproc     (existing, for MCP routing)
  └── EnvoyFilter/<cr-name>-guardrails   (NEW, created by operator)

Namespace: mcp-system
  ├── Deployment/mcp-broker-router
  ├── Deployment/mcp-controller
  └── ...

Namespace: <user-namespace>
  └── NemoGuardrails CR (with gatewayRef) → Operator creates:
      ├── Deployment/<cr-name>              (NeMo server on :8000, unchanged)
      ├── Service/<cr-name>                 (ClusterIP, unchanged)
      ├── Deployment/<cr-name>-adapter      (NEW: Envoy adapter ext-proc on :50052)
      ├── Service/<cr-name>-adapter         (NEW: ClusterIP, port 50052)
      ├── ConfigMap/<cr-name>-adapter-config (NEW: plugin config pointing to NeMo service)
      └── ConfigMap/<cr-name>-ca-bundle

When spec.gatewayRef is not set, behavior is identical to today (standalone NeMo Guardrails service, no adapter, no EnvoyFilter).

Request Flow (Integrated)

Client
  │
  ▼
Gateway (Envoy, gateway-system)
  │
  ├──[1]──► Guardrails Adapter ext-proc (:50052, <user-namespace>)
  │           │
  │           ├── Parse MCP message (tools/call, prompts/get)
  │           ├── Invoke TOOL_PRE_INVOKE / PROMPT_PRE_FETCH hooks
  │           ├── Call NeMo Guardrails service (:8000, <user-namespace>)
  │           │     └── Run configured detectors (PII, content safety, etc.)
  │           ├── If violation: return deny response (x-mcp-denied header)
  │           └── If allowed: return continue
  │
  ├──[2]──► MCP Router ext-proc (:50051, mcp-system)
  │           └── Add routing headers, auth headers
  │
  ▼
MCP Broker (:8080, mcp-system)
  │
  ▼
Upstream MCP Server(s)
  │
  ▼  (response path)
Gateway (Envoy)
  │
  ├──[3]──► Guardrails Adapter ext-proc (TOOL_POST_INVOKE hooks)
  │           └── Validate output against guardrails
  │
  ▼
Client

Filter ordering: The guardrails adapter ext-proc runs BEFORE the MCP router ext-proc. This means:

Input guardrails can reject a request before any routing/auth processing happens
Output guardrails can inspect/modify the response after the tool executes

CRD Changes

Current NemoGuardrails Spec

type NemoGuardrailsSpec struct {
    NemoConfigs    []NemoConfig     `json:"nemoConfigs"`
    CABundleConfig *CABundleConfig  `json:"caBundleConfig,omitempty"`
    Env            []corev1.EnvVar  `json:"env,omitempty"`
}

Proposed Addition

type NemoGuardrailsSpec struct {
    NemoConfigs    []NemoConfig     `json:"nemoConfigs"`
    CABundleConfig *CABundleConfig  `json:"caBundleConfig,omitempty"`
    Env            []corev1.EnvVar  `json:"env,omitempty"`

    // GatewayRef is an optional reference to an MCP Gateway instance.
    // When set, the operator deploys the Envoy Guardrails Plugin Adapter
    // and creates the EnvoyFilter to wire guardrails into the gateway's
    // request/response pipeline.
    // When omitted, NeMo Guardrails deploys as a standalone service.
    GatewayRef     *GatewayRef      `json:"gatewayRef,omitempty"`
}

type GatewayRef struct {
    // Name of the Gateway resource.
    Name      string `json:"name"`
    // Namespace of the Gateway resource.
    Namespace string `json:"namespace"`
    // Port on the Gateway listener to attach the guardrails filter to.
    // Defaults to 8080.
    Port      int32  `json:"port,omitempty"`
}

Example CR: Standalone (unchanged)

apiVersion: trustyai.opendatahub.io/v1alpha1
kind: NemoGuardrails
metadata:
  name: my-guardrails
  namespace: my-app
spec:
  nemoConfigs:
    - name: content-safety
      configMaps:
        - content-safety-config
      default: true

Example CR: Gateway-Integrated

apiVersion: trustyai.opendatahub.io/v1alpha1
kind: NemoGuardrails
metadata:
  name: my-guardrails
  namespace: my-app
spec:
  nemoConfigs:
    - name: content-safety
      configMaps:
        - content-safety-config
      default: true
  gatewayRef:
    name: mcp-gateway
    namespace: gateway-system

What the Operator Does When gatewayRef Is Set

In addition to the standard resources (Deployment, Service, ConfigMaps for NeMo), the operator creates three additional resources:

1. ConfigMap: `<cr-name>-adapter-config`

Generated plugin configuration that points the adapter to the co-deployed NeMo Guardrails service. The operator builds this from the CR's nemoConfigs, so the user never writes adapter config manually.

apiVersion: v1
kind: ConfigMap
metadata:
  name: my-guardrails-adapter-config
  namespace: my-app
data:
  config.yaml: |
    plugins:
      - name: "NemoCheck"
        kind: "plugins.examples.nemocheckinternal.plugin.NemoCheckv2"
        description: "NeMo Guardrails check via operator-managed service"
        hooks: ["tool_pre_invoke", "tool_post_invoke"]
        mode: "enforce"
        priority: 100
        config:
          checkserver_url: "http://my-guardrails.my-app.svc.cluster.local:80/v1/guardrail/checks"
    plugin_settings:
      parallel_execution_within_band: true
      plugin_timeout: 30
      fail_on_plugin_error: false

The checkserver_url is deterministic: http://<cr-name>.<namespace>.svc.cluster.local:80/v1/guardrail/checks.

2. Deployment + Service: `<cr-name>-adapter`

The Envoy Guardrails Plugin Adapter, deployed in the same namespace as the CR.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-guardrails-adapter
  namespace: my-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: my-guardrails-adapter
  template:
    spec:
      containers:
        - name: plugins-adapter
          image: <adapter-image-from-operator-configmap>
          ports:
            - containerPort: 50052
              protocol: TCP
          env:
            - name: PLUGIN_MANAGER_CONFIG
              value: "/config/config.yaml"
            - name: PLUGINS_ENABLED
              value: "true"
            - name: LOGLEVEL
              value: "INFO"
          volumeMounts:
            - name: adapter-config
              mountPath: /config
      volumes:
        - name: adapter-config
          configMap:
            name: my-guardrails-adapter-config
---
apiVersion: v1
kind: Service
metadata:
  name: my-guardrails-adapter
  namespace: my-app
spec:
  selector:
    app: my-guardrails-adapter
  ports:
    - port: 50052
      targetPort: 50052
      protocol: TCP

3. EnvoyFilter: `<cr-name>-guardrails`

Created in istio-system (or the namespace controlling the gateway) to inject the guardrails ext-proc filter into the Gateway's Envoy proxy.

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: my-guardrails-guardrails
  namespace: istio-system
spec:
  workloadSelector:
    labels:
      gateway.io/name: mcp-gateway      # from gatewayRef.name
  configPatches:
    - applyTo: HTTP_FILTER
      match:
        context: GATEWAY
        listener:
          portNumber: 8080               # from gatewayRef.port
          filterChain:
            filter:
              name: envoy.filters.network.http_connection_manager
              subFilter:
                name: envoy.filters.http.router
      patch:
        operation: INSERT_BEFORE
        value:
          name: envoy.filters.http.ext_proc.guardrails
          typed_config:
            "@type": type.googleapis.com/envoy.extensions.filters.http.ext_proc.v3.ExternalProcessor
            failure_mode_allow: false
            message_timeout: 10s
            processing_mode:
              request_header_mode: SEND
              response_header_mode: SEND
              request_body_mode: STREAMED
              response_body_mode: STREAMED
              request_trailer_mode: SKIP
              response_trailer_mode: SKIP
            grpc_service:
              envoy_grpc:
                cluster_name: "outbound|50052||my-guardrails-adapter.my-app.svc.cluster.local"

Note: The EnvoyFilter is created in istio-system because that is where Istio processes filter configuration for gateways. The operator needs RBAC permissions to create EnvoyFilter resources in that namespace.

Lifecycle

Creation

User creates NemoGuardrails CR with gatewayRef
Operator reconciles: a. Creates NeMo Guardrails Deployment + Service (existing logic, unchanged) b. Detects gatewayRef is set c. Generates adapter config ConfigMap (pointing to the NeMo service) d. Creates adapter Deployment + Service e. Creates EnvoyFilter in istio-system
Envoy picks up the new filter, starts sending MCP traffic through the adapter
Adapter calls NeMo Guardrails service for each tool call / prompt fetch

Update

nemoConfigs change: Operator updates NeMo Deployment and adapter ConfigMap (the checkserver_url stays the same since it's based on the service name)
gatewayRef added: Operator creates adapter + EnvoyFilter (standalone → integrated)
gatewayRef removed: Operator deletes adapter + EnvoyFilter (integrated → standalone)
gatewayRef modified (different gateway): Operator updates/recreates EnvoyFilter

Deletion

User deletes NemoGuardrails CR
Operator finalizer runs: a. Deletes EnvoyFilter from istio-system b. OwnerReference-based garbage collection handles:
- Adapter Deployment + Service
- Adapter ConfigMap
- NeMo Deployment + Service
- CA ConfigMaps

The EnvoyFilter requires explicit cleanup in the finalizer because it lives in a different namespace (istio-system) and cannot use OwnerReference (which is namespace-scoped).

RBAC Implications

The operator currently has permissions scoped to the CR's namespace plus ClusterRoleBinding management. With this change, it additionally needs:

Resource	Namespace	Verbs	Reason
`envoyfilters.networking.istio.io`	`istio-system`	get, list, watch, create, update, patch, delete	Create/manage the guardrails EnvoyFilter

This is a cross-namespace permission. The operator's ClusterRole needs to include EnvoyFilter management.

What Does NOT Change

Guardrails configuration: Users configure NeMo Guardrails the same way (ConfigMaps with actions.py, config.yaml, flows.co). No new config format.
Detectors and rails: The content safety, PII detection, jailbreak detection, and any other rails work identically.
MCP Gateway: No changes to the MCP Gateway codebase. The gateway already supports multiple ext-proc filters via EnvoyFilter.
Adapter plugin logic: The adapter's ext-proc server and plugin system are unchanged. The operator just deploys and configures it.

Open Questions

Filter ordering: Should the guardrails ext-proc run before or after the MCP router ext-proc? Running it first means we can reject bad requests before any routing, but it also means the guardrails adapter must independently parse MCP messages. Running it after means the router has already set headers, which the adapter could use. Current proposal: guardrails first (fail fast).
Adapter image source: Where does the operator get the adapter container image? Options:
- Add a key to the operator's existing trustyai-service-operator-config ConfigMap (consistent with how nemo-guardrails-image is managed today)
- Specify it in the CR spec
EnvoyFilter namespace: The design assumes istio-system. Should this be configurable via the CR or derived from the gateway reference?
Multiple NemoGuardrails CRs referencing the same gateway: Should this be allowed? If so, how do the multiple ext-proc filters interact? If not, the operator should enforce uniqueness via validation.
Health checking: Should the EnvoyFilter be created only after both the NeMo service and adapter are ready (Deployment conditions), or immediately? Creating it before the adapter is ready would cause Envoy to reject requests (failure_mode_allow: false).
Adapter namespace: The current design places the adapter in the CR's namespace. An alternative is to deploy it in mcp-system alongside the broker. The CR namespace is simpler for ownership/cleanup but requires cross-namespace gRPC from the gateway.

References

Repositories

Component	Repository
TrustyAI Operator	https://github.com/trustyai-explainability/trustyai-service-operator
NeMo Guardrails	https://github.com/trustyai-explainability/NeMo-Guardrails
MCP Gateway	https://github.com/kuadrant/mcp-gateway
Envoy Guardrails Plugin Adapter	https://github.com/kagenti/plugins-adapter
WG AI Gateway	https://github.com/kubernetes-sigs/wg-ai-gateway

Key Source Files

TrustyAI Operator -- NemoGuardrails CRD and Controller

MCP Gateway -- Gateway and EnvoyFilter Definitions

Envoy Guardrails Plugin Adapter -- ext-proc and Plugin System

NeMo Guardrails -- Guardrails Engine

danielezonca/nemo-guardrails-mcp-gateway-integration.md

Select an option

No results found

Select an option

No results found

NeMo Guardrails + MCP Gateway Integration

Problem Statement

Components

Current Deployment Topology (Before Integration)

Target Deployment Topology (With Integration)

Request Flow (Integrated)

CRD Changes

Current NemoGuardrails Spec

Proposed Addition

Example CR: Standalone (unchanged)

Example CR: Gateway-Integrated

What the Operator Does When gatewayRef Is Set

1. ConfigMap: `<cr-name>-adapter-config`

2. Deployment + Service: `<cr-name>-adapter`

3. EnvoyFilter: `<cr-name>-guardrails`

Lifecycle

Creation

Update

Deletion

RBAC Implications

What Does NOT Change

Open Questions

References

Repositories

Key Source Files

danielezonca/nemo-guardrails-mcp-gateway-integration.md

NeMo Guardrails + MCP Gateway Integration

Problem Statement

Components

Current Deployment Topology (Before Integration)

Target Deployment Topology (With Integration)

Request Flow (Integrated)

CRD Changes

Current NemoGuardrails Spec

Proposed Addition

Example CR: Standalone (unchanged)

Example CR: Gateway-Integrated

What the Operator Does When gatewayRef Is Set

1. ConfigMap: <cr-name>-adapter-config

2. Deployment + Service: <cr-name>-adapter

3. EnvoyFilter: <cr-name>-guardrails

Lifecycle

Creation

Update

Deletion

RBAC Implications

What Does NOT Change

Open Questions

References

Repositories

Key Source Files

1. ConfigMap: `<cr-name>-adapter-config`

2. Deployment + Service: `<cr-name>-adapter`

3. EnvoyFilter: `<cr-name>-guardrails`