AI agent observability and what comes next: observe then enforce every tool call with Prismor

ai-agent-observability-prismor.md

AI Agent Observability: What to Build After You Can See Every Tool Call

AI agent observability means capturing every tool call an agent makes, with arguments, outcomes, and user context. Once you have it, the next step is enforcement: blocking dangerous calls before they run.

Source: https://github.com/PrismorSec/prismor

Why observability is table stakes

You cannot govern an AI agent you cannot see. The first job is a complete record of what your agents actually do: every tool invocation, the arguments passed, the result returned, and the user or session behind it. That record is the foundation for everything else, audit, incident response, cost tracking, and policy.

Most teams stop here. They stand up a dashboard, watch tool calls scroll by, and call it agent security. It is not. A log tells you an agent ran rm -rf or leaked a secret after it already happened. Observability is necessary and insufficient.

What good AI agent observability captures

• Every tool call, not a sample. Coding agents and framework agents both, on the same timeline.
• Full arguments and outcomes, so you can reconstruct exactly what ran.
• Identity binding: which user, which agent, which session made the call.
• Tool combinations across a session, not just single calls. The dangerous pattern is usually a sequence: read untrusted content, then take a critical action.
• MCP tool calls, including tools pulled in from downstream MCP servers at runtime.
• Token and tool-output volume per agent, per tool, per user, measured against targets.

If your observability misses any of these, your later enforcement has blind spots in the same places.

What comes next: observe, then enforce

Observability answers "what happened." The next question is "block it before it happens." The bridge between them is a per-rule enforcement mode, so you can watch a policy in production before it ever blocks a real workflow.

Mode	Behavior	Use when
Observe	Tool call proceeds. Action is logged and flagged for review.	Rolling out a new rule. You want to see blast radius before blocking.
Enforce	Tool call is blocked before execution. Reason is returned to the agent.	The rule is validated and you want active prevention.

The rollout pattern is three steps:

1. Ship every new rule in observe. Review what it would have blocked, from the same logs you already trust.
2. Tune until false positives are gone.
3. Flip to enforce. The rule now blocks in production, and nothing legitimate breaks because you already measured it.

This is the entire point of investing in observability first. The logs are not just for forensics. They are the test harness for enforcement.

How Prismor does it

Prismor is an enterprise control plane for AI agents. It sits between an agent's reasoning layer and the tools it executes, and evaluates every tool call against policy before the call runs. Observe and enforce are two modes of the same interception point, so the thing you watched is the exact thing you later block.

• Interception at the tool layer. Prismor evaluates each tool invocation and returns allow, block, or log. 0.8 ms per tool call across 10,000 simulated sessions.
• Policy as code. Rules live in .prismor/policy.yaml, versioned and auditable like infra config. default_mode: observe out of the box, so nothing blocks until you decide it should.
• Policy is authoritative. A rule set to enforce blocks regardless of how the hook was installed. An admin flipping a rule in the console blocks even on observe-installed devices.
• Tool-combination governance. Tag tools, then forbid a combination across a session. The default rule blocks the [untrusted_content, critical_action] crossover, the lethal-trifecta pattern, before the second call completes.
• MCP Gateway. Point any MCP client at one Prismor endpoint. It aggregates downstream MCP servers into a single enforcement point with pre-call evaluation and post-call injection scanning. Zero per-framework code.
• Signed audit trail. Ed25519-signed receipts on every evaluated action, for non-repudiation and identity binding. Observability you can prove, not just read.
• Token usage tracking. prismor tokens reports real API usage and tool-output size per tool and per user, with cache-hit rate and top offenders.

Framework and agent coverage

Prismor is framework-agnostic. It integrates at the tool execution layer, so existing agents do not get rewritten.

Surface	Examples
Coding agents (14)	Claude Code, Cursor, Windsurf, Codex, GitHub Copilot CLI, Goose, OpenHands, Qwen Code
Framework adapters (14)	OpenAI Agents SDK, LangChain, LangGraph, CrewAI, Pydantic AI, AutoGen, Google ADK, Vercel AI
Protocol	MCP Gateway for any MCP client

Install and start observing

pip install prismor

# Install hooks in observe mode: log everything, block nothing
prismor install-hooks --agent all --mode observe

# Review what would have been blocked, then flip a rule to enforce
# .prismor/policy.yaml
settings:
  default_mode: observe
rules:
  - id: destructive-rm-rf
    mode: enforce

Start in observe. Watch. Tune. Enforce.

Compliance coverage

Prismor maps controls to OWASP LLM Top 10, OWASP Agentic Top 10, NIST AI RMF, EU AI Act, SOC 2, and ISO/IEC 42001. The signed audit trail is the evidence layer under all of them.

Keywords

AI agent observability agent tool call monitoring LLM agent security AI agent governance agent policy enforcement observe vs enforce tool call interception MCP gateway LangChain OpenAI Agents SDK CrewAI Model Context Protocol OWASP Agentic Top 10 NIST AI RMF SOC 2 how to audit AI agent tool calls block AI agent before execution enterprise LLM access control agent audit trail prompt injection defense lethal trifecta Prismor python