kishore-crowai

AI Agent Evaluation Spec

Status: draft for review. Background reading: agentEvaluationReference.md.

Goal

A reliable, low-setup way to answer "is the agent working properly?" — and, once a prompt or model changes, "did that make it better or worse?"

Non-goals for the first release: production traffic splitting, statistical A/B

kishore-crowai

CrowAI + IDFC Bank: Authenticated Experiment Authoring Model

Problem

IDFC wants to run experiments on authenticated banking pages. However for the reasons below, it's not straight forward to setup these tests.

• production authenticated pages are reachable only by real customers
• no test or dummy accounts exist on production
• authors cannot log in to production for design-time inspection
• compliance and audit controls limit direct production access