Skip to content

Instantly share code, notes, and snippets.

@clsandoval
Created January 16, 2026 07:25
Show Gist options
  • Select an option

  • Save clsandoval/0494657fa6baf9b946e2d1d5f3944048 to your computer and use it in GitHub Desktop.

Select an option

Save clsandoval/0494657fa6baf9b946e2d1d5f3944048 to your computer and use it in GitHub Desktop.
Decision Orchestrator Analysis - Updated to reflect actual codebase state (January 2025)

Competitor Overview

Comprehensive analysis of 26 competitor repositories analyzed for Decision AI product positioning


Executive Summary

This document provides a structured overview of all competitors analyzed during our research phase. Each competitor is categorized by market segment, with detailed profiles including value propositions, target audiences, key features, and user journey diagrams.

Key Finding: The market is fragmented across multiple niches. No single competitor addresses our full vision of unified context across platforms + modular AI sessions (deployable Claude instances) + trust-focused data science. This represents our blue ocean opportunity.

Current State vs. Vision:

IMPORTANT: This document describes both what exists TODAY in Decision Orchestrator and the FUTURE VISION. The current implementation is simpler than the vision described in some sections.

Feature Current State (What Exists) Future Vision
Sessions Discord bot launches Claude instances on Fly.io with system prompts Reusable "Decision Packs" as deployable repos
Templates Supabase database records (name, system_prompt, tools_config) pack.yaml manifest files with triggers
Deployment Dynamic Fly.io machine creation via workflow_tools Pre-built container registry + one-click deploy
Communication ACP (Agent Communication Protocol) via SSE Same, but with richer multi-session orchestration

Current Decision Orchestrator Architecture

What ACTUALLY exists today:

┌─────────────────────────────────────────────────────────────────────────────┐
│                    DECISION ORCHESTRATOR - CURRENT STATE                      │
├─────────────────────────────────────────────────────────────────────────────┤
│                                                                             │
│   Discord Bot (Primary Interface)                                            │
│   └── User sends message                                                     │
│       └── Workflow Executor runs with Claude API                             │
│           └── Uses workflow_tools:                                           │
│               ├── fly_app_tools (create_fly_app, deploy_fly_app, etc.)      │
│               ├── session_comm_tools (ACP: send_message_to_session)          │
│               ├── human_interaction_tools (wait_for_human_decision)          │
│               └── mcp_server_tools                                           │
│                                                                             │
│   Sessions (Claude instances on Fly.io)                                      │
│   └── Each session = Fly.io machine running Claude                           │
│       └── System prompt from session_templates table (Supabase)              │
│       └── Communicates via ACP (SSE endpoints)                               │
│                                                                             │
│   Templates (Supabase: session_templates table)                              │
│   └── Columns: name, system_prompt, tools_config, mcp_config, metadata      │
│                                                                             │
└─────────────────────────────────────────────────────────────────────────────┘

Market Landscape Map

┌─────────────────────────────────────────────────────────────────────────────────────────────┐
│                              COMPETITIVE LANDSCAPE 2025                                      │
├─────────────────────────────────────────────────────────────────────────────────────────────┤
│                                                                                              │
│  ORCHESTRATION & FRAMEWORKS                    OBSERVABILITY & EVALUATION                   │
│  ┌──────────────┐  ┌──────────────┐           ┌──────────────┐  ┌──────────────┐           │
│  │   CrewAI     │  │  LangGraph   │           │   Langfuse   │  │  Braintrust  │           │
│  │  Multi-Agent │  │  Stateful    │           │  LLM Tracing │  │   AI Evals   │           │
│  │  Tool Mgmt   │  │  Workflows   │           │  & Prompts   │  │  Regression  │           │
│  └──────────────┘  └──────────────┘           └──────────────┘  └──────────────┘           │
│                                                                                              │
│  ┌──────────────┐  ┌──────────────┐           ┌──────────────┐                             │
│  │    Swarm     │  │ Claude-Flow  │           │   AgentOps   │                             │
│  │  Lightweight │  │  Enterprise  │           │   Session    │                             │
│  │   Handoffs   │  │   Swarms     │           │   Replay     │                             │
│  └──────────────┘  └──────────────┘           └──────────────┘                             │
│                                                                                              │
│  ┌──────────────┐  ┌──────────────┐                                                        │
│  │   AutoGen    │  │  Pydantic-AI │                                                        │
│  │  Multi-Agent │  │  Type-Safe   │                                                        │
│  │   Groups     │  │   Agents     │                                                        │
│  └──────────────┘  └──────────────┘                                                        │
│                                                                                              │
├─────────────────────────────────────────────────────────────────────────────────────────────┤
│                                                                                              │
│  PLATFORMS & BUILDERS                          MEMORY & KNOWLEDGE                           │
│  ┌──────────────┐  ┌──────────────┐           ┌──────────────┐  ┌──────────────┐           │
│  │   LLMStack   │  │  VoltAgent   │           │  ChatMemory  │  │    Glean     │           │
│  │   No-Code    │  │  TypeScript  │           │  4-Tier Mem  │  │  Enterprise  │           │
│  │   Builder    │  │  Full-Stack  │           │  Hierarchy   │  │  Knowledge   │           │
│  └──────────────┘  └──────────────┘           └──────────────┘  └──────────────┘           │
│                                                                                              │
│  ┌──────────────┐  ┌──────────────┐                                                        │
│  │  BotSharp    │  │   Composio   │                                                        │
│  │  .NET Agent  │  │   500+ App   │                                                        │
│  │  Framework   │  │  Integrations│                                                        │
│  └──────────────┘  └──────────────┘                                                        │
│                                                                                              │
├─────────────────────────────────────────────────────────────────────────────────────────────┤
│                                                                                              │
│  CHAT & MULTI-PLATFORM                         MARKETING ANALYTICS (MMM)                    │
│  ┌──────────────┐  ┌──────────────┐           ┌──────────────┐  ┌──────────────┐           │
│  │   Dust.tt    │  │   Clawdbot   │           │   PyMC-      │  │   Google     │           │
│  │  Slack AI    │  │  8-Platform  │           │  Marketing   │  │   LWMMM      │           │
│  │  Assistants  │  │  Personal AI │           │  Bayesian    │  │              │           │
│  └──────────────┘  └──────────────┘           └──────────────┘  └──────────────┘           │
│                                                                                              │
│  ┌──────────────┐  ┌──────────────┐           ┌──────────────┐  ┌──────────────┐           │
│  │    KIRA      │  │   Runbear    │           │    Meta      │  │   Nielsen    │           │
│  │  Privacy-1st │  │  Tiered Bot  │           │    Robyn     │  │  Proprietary │           │
│  │  Desktop AI  │  │  Platform    │           │   AutoMMM    │  │    MMM       │           │
│  └──────────────┘  └──────────────┘           └──────────────┘  └──────────────┘           │
│                                                                                              │
├─────────────────────────────────────────────────────────────────────────────────────────────┤
│                                                                                              │
│  SKILLS & PACKS                                DEPLOY & ARTIFACTS                           │
│  ┌──────────────┐                              ┌──────────────┐  ┌──────────────┐           │
│  │   Awesome    │                              │   Replit     │  │   Hex AI     │           │
│  │   Claude     │                              │   Agent      │  │  Analytics   │           │
│  │   Skills     │                              │   Deploy     │  │  Artifacts   │           │
│  └──────────────┘                              └──────────────┘  └──────────────┘           │
│                                                                                              │
│                                                 ┌──────────────┐                            │
│                                                 │    v0.dev    │                            │
│                                                 │  AI-Gen UI   │                            │
│                                                 │   Preview    │                            │
│                                                 └──────────────┘                            │
│                                                                                              │
├─────────────────────────────────────────────────────────────────────────────────────────────┤
│                                                                                              │
│  TEMPLATE & DEPLOYMENT PLATFORMS (Inspiration for Future Pack System)                        │
│  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐                     │
│  │   Replit     │  │   Railway    │  │   Render     │  │   Vercel     │                     │
│  │  Templates   │  │  Templates   │  │  Blueprints  │  │  Templates   │                     │
│  │  Full Proj   │  │  One-Click   │  │  IaC Deploy  │  │  Frontend    │                     │
│  └──────────────┘  └──────────────┘  └──────────────┘  └──────────────┘                     │
│                                                                                              │
└─────────────────────────────────────────────────────────────────────────────────────────────┘

What Decision Orchestrator Does TODAY

Core Architecture (Implemented)

  1. Discord Bot - Primary user interface
  2. Workflow Executor - Uses Claude API with tools to process requests
  3. Session Management - Creates/manages Claude instances on Fly.io
  4. ACP Protocol - Inter-session communication via SSE endpoints
  5. Supabase Backend - Stores templates, sessions, and messages

Key Workflow Tools (Implemented)

From app/src/services/workflow_tools/:

# fly_app_tools.py - Fly.io deployment
- create_fly_app(app_name, region)
- deploy_fly_app(app_name, image, env_vars)
- list_fly_apps()
- scale_fly_app(app_name, count)
- restart_fly_machine(app_name, machine_id)
- get_fly_logs(app_name)
- set_fly_secrets(app_name, secrets)

# session_comm_tools.py - Inter-session communication (ACP)
- send_message_to_session(session_id, message)
- wait_for_human_decision(prompt, options)
- read_active_sessions()
- request_session_spawn(template_name)

# human_interaction_tools.py - User decisions
- request_human_approval(prompt)
- wait_for_human_input(prompt)

Session Templates (Supabase Schema)

CREATE TABLE session_templates (
    id UUID PRIMARY KEY,
    name TEXT UNIQUE NOT NULL,
    system_prompt TEXT NOT NULL,
    tools_config JSONB DEFAULT '{}',
    mcp_config JSONB DEFAULT '{}',
    metadata JSONB DEFAULT '{}',
    is_default BOOLEAN DEFAULT FALSE,
    created_at TIMESTAMPTZ,
    updated_at TIMESTAMPTZ
);

Gap Analysis: Our Opportunity

What NO Competitor Does

Gap Description Our Opportunity
Unified Context No one offers context continuity across Discord/Slack/Teams/CLI Interface Primitives + Shared Memory
Domain-Specific Evals No one has insight recovery benchmarks for analytics MMM Insight Recovery Experiments
Trust-First Data Science No one combines Bayesian causal + LLM + governance Trust Differentiators
Deployable AI Sessions No one does Discord → Fly.io Claude deployment seamlessly What we have TODAY
ACP Protocol No standard for inter-Claude communication Our implemented protocol

What We Should Adopt

Pattern From Why
ToolCollection CrewAI Best-in-class tool management
Thread-as-Boundary Dust.tt Essential for chat context
Statistical Evals Braintrust Right approach to AI testing
4-Tier Memory ChatMemory Complete hierarchy
Manifest Format Awesome Skills Proven skill structure
Streaming Progress Replit Great deploy UX
Bayesian Foundation PyMC-Marketing Trust through uncertainty

Complete Competitor List

# Competitor Category What They Do (One-Liner)
1 CrewAI Framework Multi-agent orchestration with role-based collaboration and tool collections
2 LangGraph Framework Stateful graph-based workflows for LLM applications
3 Swarm Framework Lightweight multi-agent handoffs (educational, by OpenAI)
4 Claude-Flow Framework Enterprise multi-agent swarms with neural learning and MCP
5 AutoGen Framework Microsoft's multi-agent conversation framework
6 Pydantic-AI Framework Type-safe Python agents with structured outputs
7 VoltAgent Platform TypeScript full-stack agent framework with VoltOps observability
8 LLMStack Platform No-code visual builder for AI agents and workflows
9 BotSharp Platform .NET/C# agent framework with plugin architecture
10 Composio SDK 500+ app integrations for AI agents
11 Langfuse Observability LLM tracing, prompt management, and evaluation
12 Braintrust Observability Statistical AI evaluation with regression detection
13 AgentOps Observability Agent session replay and cost tracking
14 ChatMemory Memory 4-tier hierarchical memory for AI assistants
15 Glean Knowledge Enterprise permission-aware knowledge search
16 Dust.tt Chat Thread-aware Slack AI assistants
17 Clawdbot Chat 8-platform personal AI assistant (desktop)
18 KIRA Chat Privacy-first desktop AI coworker
19 Runbear Chat Tiered Slack/Teams bot platform
20 Awesome Claude Skills Skills Open-source skill manifest patterns
21 PyMC-Marketing MMM Bayesian causal marketing mix modeling
22 Meta Robyn MMM Automated MMM with Pareto optimization
23 Replit Templates Templates Full project templates with instant deployment
24 Railway Templates Templates One-click deployable app templates
25 Render Blueprints Templates Infrastructure-as-code deployment templates
26 Vercel Templates Templates Frontend/fullstack starter templates

Master Comparison Table

Competitor Category Primary Language Deploy Model Key Differentiator Pricing
CrewAI Framework Python Library Role-based multi-agent + ToolCollection OSS
LangGraph Framework Python Library Stateful graphs + checkpointing OSS + Cloud
Swarm Framework Python Library Minimal primitives (educational) OSS
Claude-Flow Framework TypeScript Enterprise 54+ agents + neural learning OSS
AutoGen Framework Python Library Conversational multi-agent OSS
Pydantic-AI Framework Python Library Type safety + structured outputs OSS
VoltAgent Platform TypeScript Hybrid Full-stack + VoltOps console OSS + Cloud
LLMStack Platform Python Self-hosted No-code visual builder OSS + Cloud
BotSharp Platform C# Enterprise .NET ecosystem + plugins OSS
Composio SDK TypeScript Multi-framework 500+ integrations Freemium
Langfuse Observability TypeScript Self-hosted Tracing + prompt management OSS + Cloud
Braintrust Observability Python Cloud Statistical evals + regression Freemium
AgentOps Observability Python Cloud Session replay + cost tracking Freemium
ChatMemory Memory Python Library 4-tier hierarchy + pgvector OSS
Glean Knowledge - Enterprise Permission-aware search Enterprise
Dust.tt Chat - Cloud Thread-aware Slack AI Tiered
KIRA Chat Python Desktop Privacy-first, local-only OSS
Clawdbot Chat TypeScript Desktop 8-platform personal AI OSS
Runbear Chat - Cloud Tiered bot platform Tiered
Awesome Skills Skills - - Manifest format pattern OSS
PyMC-Marketing MMM Python Library Bayesian causal inference OSS
Robyn MMM R Library Automated Pareto optimization OSS
LightweightMMM MMM Python Library Google's Bayesian MMM OSS
Nielsen MMM - Service Industry standard Enterprise
Replit Agent Deploy - Cloud Zero-friction deploy Freemium
Hex AI Artifacts - Cloud Professional notebooks Tiered
v0.dev Artifacts - Cloud AI-generated UI preview Freemium
Replit Templates Templates - Cloud Full project templates Freemium
Railway Templates Templates - Cloud One-click deploy apps Freemium
Render Blueprints Templates YAML Cloud IaC deployment specs Freemium
Vercel Templates Templates - Cloud Frontend starters Freemium

Appendix: Repository URLs

Competitor GitHub/URL
CrewAI github.com/joaomdmoura/crewAI
LangGraph github.com/langchain-ai/langgraph
Swarm github.com/openai/swarm
Claude-Flow github.com/ruvnet/claude-flow
AutoGen github.com/microsoft/autogen
Pydantic-AI github.com/pydantic/pydantic-ai
VoltAgent github.com/voltagent/voltagent
LLMStack github.com/trypromptly/LLMStack
BotSharp github.com/SciSharp/BotSharp
Composio github.com/composiohq/composio
Langfuse github.com/langfuse/langfuse
Braintrust github.com/braintrustdata/braintrust-sdk
AgentOps github.com/AgentOps-AI/agentops
ChatMemory github.com/uezo/chatmemory
KIRA github.com/krafton-ai/KIRA
Clawdbot github.com/ClawdAI/clawdbot
Awesome Skills github.com/anthropics/awesome-claude-skills
PyMC-Marketing github.com/pymc-labs/pymc-marketing
Robyn github.com/facebookresearch/Robyn
LightweightMMM github.com/google/lightweight_mmm
Replit Templates replit.com/templates
Railway Templates railway.app/templates
Render Blueprints render.com/docs/infrastructure-as-code
Vercel Templates vercel.com/templates

Document generated for Decision AI competitive analysis - January 2025 26 competitors analyzed across 9 categories (including Template Platforms) Updated to reflect actual current state of Decision Orchestrator codebase

Session Templates & Future Pack Architecture

This document describes the CURRENT template system in Decision Orchestrator and the FUTURE VISION for "Decision Packs" as deployable repositories.


Important: Current State vs. Future Vision

CRITICAL DISTINCTION: The sophisticated "Decision Pack" system described in much of this document is a FUTURE VISION, not the current implementation. Today, Decision Orchestrator uses a simpler database-driven template system.

What EXISTS Today

Session Templates are database records in Supabase:

-- Current schema (from supabase/migrations/20251216204538_create_session_templates.sql)
CREATE TABLE session_templates (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    name TEXT UNIQUE NOT NULL,
    system_prompt TEXT NOT NULL,
    tools_config JSONB DEFAULT '{}',
    mcp_config JSONB DEFAULT '{}',
    metadata JSONB DEFAULT '{}',
    is_default BOOLEAN DEFAULT FALSE,
    created_at TIMESTAMPTZ DEFAULT now(),
    updated_at TIMESTAMPTZ DEFAULT now()
);

How templates work today:

  1. Templates are stored in the session_templates Supabase table
  2. When a session is spawned, the template's system_prompt is loaded
  3. The tools_config and mcp_config define available tools
  4. Sessions run as Claude instances on Fly.io machines

What is PLANNED (Future Vision)

The "Decision Pack" concept envisions:

  • Complete GitHub repositories as deployable units
  • pack.yaml manifest files for discovery
  • Dockerfile + fly.toml for containerization
  • Pre-built container registry
  • Pack marketplace and discovery system

Current Template System (Implemented)

Template Structure

┌─────────────────────────────────────────────────────────────────────────────┐
│                    CURRENT TEMPLATE SYSTEM                                   │
├─────────────────────────────────────────────────────────────────────────────┤
│                                                                             │
│   Supabase: session_templates table                                          │
│   ┌─────────────────────────────────────────────────────────────────────┐   │
│   │ id          │ UUID (primary key)                                     │   │
│   │ name        │ "marketing-analyst"                                    │   │
│   │ system_prompt│ "You are a marketing analysis assistant..."          │   │
│   │ tools_config│ {"allowed_tools": ["search", "calculate"]}            │   │
│   │ mcp_config  │ {"servers": [...]}                                     │   │
│   │ metadata    │ {"version": "1.0", "author": "team"}                  │   │
│   │ is_default  │ false                                                  │   │
│   └─────────────────────────────────────────────────────────────────────┘   │
│                                                                             │
│   How it's used:                                                             │
│   1. User requests session via Discord                                       │
│   2. Workflow executor queries session_templates by name                     │
│   3. System prompt and configs are loaded                                   │
│   4. Fly.io machine is created with this configuration                       │
│   5. Claude instance runs with the template's instructions                   │
│                                                                             │
└─────────────────────────────────────────────────────────────────────────────┘

Example Template Record

{
  "id": "550e8400-e29b-41d4-a716-446655440000",
  "name": "marketing-analyst",
  "system_prompt": "You are a marketing analytics assistant specializing in budget optimization and ROI analysis. You have access to marketing data and can help with attribution modeling, media mix analysis, and campaign performance tracking.",
  "tools_config": {
    "allowed_tools": ["search_web", "calculate", "create_chart"],
    "restrictions": []
  },
  "mcp_config": {
    "servers": []
  },
  "metadata": {
    "version": "1.0",
    "category": "analytics",
    "author": "pymc-labs"
  },
  "is_default": false
}

Creating a Template (Current Method)

Templates are created by inserting records into Supabase:

# Using Supabase client
from supabase import create_client

supabase = create_client(url, key)

template = {
    "name": "my-template",
    "system_prompt": "You are a helpful assistant...",
    "tools_config": {"allowed_tools": ["search"]},
    "mcp_config": {},
    "metadata": {"version": "1.0"}
}

result = supabase.table("session_templates").insert(template).execute()

Future Vision: Decision Packs (NOT YET IMPLEMENTED)

NOTE: Everything below this line describes a FUTURE architecture that does not yet exist in the codebase. It represents the vision from research documents like thoughts/dynamic-builder-flow/research/builder_as_claude_factory.md.

The Pack Vision

A Decision Pack would be a complete, deployable repository containing:

  • Dockerfile for containerization
  • All dependencies and tooling
  • Agent/skill logic and instructions
  • Manifest/configuration
  • Ready to deploy on Fly.io
┌─────────────────────────────────────────────────────────────────────────────┐
│                    FUTURE: DECISION PACK = DEPLOYABLE REPOSITORY             │
├─────────────────────────────────────────────────────────────────────────────┤
│                                                                             │
│  Current (Template):                    Future Vision (Pack):               │
│  ──────────────────                     ───────────────────────             │
│                                                                             │
│  ┌─────────────────────┐               ┌─────────────────────────────────┐ │
│  │ Supabase record:    │               │ github.com/org/pack-mmm         │ │
│  │ - name              │               │                                  │ │
│  │ - system_prompt     │               │ ├── Dockerfile        ← BUILD   │ │
│  │ - tools_config      │               │ ├── fly.toml          ← DEPLOY  │ │
│  │ - mcp_config        │               │ ├── requirements.txt  ← DEPS    │ │
│  └─────────────────────┘               │ ├── pack.yaml         ← MANIFEST│ │
│                                        │ ├── .claude/                    │ │
│  Just database config.                 │ │   ├── CLAUDE.md     ← BRAIN   │ │
│  Session created                       │ │   └── skills/                 │ │
│  dynamically.                          │ ├── acp-server/       ← COMMS   │ │
│                                        │ └── app/              ← CODE    │ │
│                                        └─────────────────────────────────┘ │
│                                                                             │
│                                        Complete. Self-contained. Deployable.│
│                                                                             │
└─────────────────────────────────────────────────────────────────────────────┘

Proposed Pack Manifest (pack.yaml) - FUTURE

# pack.yaml - Pack Manifest (PROPOSED - NOT YET IMPLEMENTED)
name: marketing-mmm
version: 2.1.0
description: Marketing Mix Modeling with Bayesian inference and budget optimization

# Identity
author: decision-ai
license: Apache-2.0
repository: github.com/decision-ai/pack-mmm

# Discovery triggers - WHEN should this pack be suggested?
triggers:
  keywords:
    - "marketing.*budget"
    - "ROI|ROAS"
    - "attribution"
    - "media mix"
  file_patterns:
    - "*.csv"           # Marketing data files
    - "*.parquet"
  contexts:
    - analytics
    - marketing
    - planning

# Capabilities - WHAT does this pack provide?
provides:
  - bayesian-inference
  - budget-optimization
  - uncertainty-quantification
  - channel-attribution

# Dependencies - WHAT other packs does this need?
requires:
  - name: data-loader
    version: ">=1.0"
  - name: visualization
    version: ">=2.0"
    optional: true

# Runtime requirements
runtime:
  base_image: python:3.11-slim
  memory: 2gb
  cpu: shared-2x
  gpu: false

# Environment variables needed
environment:
  required:
    - ANTHROPIC_API_KEY
  optional:
    - DATABASE_URL
    - REDIS_URL

# Health check
health:
  path: /healthz
  interval: 30s
  timeout: 5s

Proposed Pack Repository Structure - FUTURE

pack-{name}/                        # FUTURE VISION
│
├── Dockerfile                      # Container definition
├── fly.toml                        # Fly.io deployment config
├── .dockerignore                   # Build exclusions
├── requirements.txt                # Python dependencies
├── pack.yaml                       # Pack manifest
│
├── .claude/                        # Claude environment
│   ├── CLAUDE.md                   # Primary instructions
│   ├── skills/                     # Modular skill files
│   └── settings.json               # Claude configuration
│
├── acp-server/                     # ACP communication server
│   ├── server.py                   # WebSocket server
│   ├── tools/                      # Tool implementations
│   └── pyproject.toml
│
├── app/                            # Application code
│   ├── main.py                     # Entry point
│   └── ...                         # Domain-specific implementation
│
├── tests/                          # Pack tests
│   ├── test_pack.py
│   └── fixtures/
│
└── README.md                       # Pack documentation

Roadmap: From Templates to Packs

Phase 1: Current State (IMPLEMENTED)

  • Supabase session_templates table
  • Dynamic Fly.io machine creation
  • System prompts loaded from database
  • ACP communication between sessions

Phase 2: Builder Sessions (PLANNED)

From thoughts/shared/roadmap/builder-sessions.md:

  • "Builder Claude" that can create other Claude instances
  • Dynamic session creation from templates
  • Session-to-session spawning

Phase 3: Pack System (FUTURE VISION)

From thoughts/dynamic-builder-flow/research/builder_as_claude_factory.md:

  • Git repositories as deployable units
  • pack.yaml manifest for discovery
  • Container registry for pre-built images
  • Pack marketplace

Migration Path

How the current system could evolve to packs:

┌─────────────────────────────────────────────────────────────────────────────┐
│                        EVOLUTION PATH                                        │
├─────────────────────────────────────────────────────────────────────────────┤
│                                                                             │
│  TODAY                     NEAR-TERM                    FUTURE              │
│  ─────                     ─────────                    ──────              │
│                                                                             │
│  session_templates         session_templates            Pack repos          │
│  (Supabase)                + git_repo_url field         (GitHub)            │
│       │                          │                           │              │
│       ▼                          ▼                           ▼              │
│  system_prompt             system_prompt OR            .claude/CLAUDE.md    │
│  in database               .claude/ from repo          in repository        │
│       │                          │                           │              │
│       ▼                          ▼                           ▼              │
│  Dynamic Fly.io            Clone repo →                Pre-built images     │
│  machine creation          Build → Deploy              from registry        │
│                                                                             │
└─────────────────────────────────────────────────────────────────────────────┘

Backward Compatibility

The future pack system should support existing templates:

# Pseudocode for future compatibility layer
def resolve_template(name: str) -> SessionConfig:
    # Check if it's a pack (git repo)
    if is_pack_url(name):
        return load_from_pack_repo(name)

    # Fall back to database template
    template = supabase.table("session_templates").select("*").eq("name", name).single()
    return SessionConfig(
        system_prompt=template["system_prompt"],
        tools_config=template["tools_config"],
        mcp_config=template["mcp_config"]
    )

Summary

Aspect Current (Templates) Future (Packs)
Storage Supabase database GitHub repositories
Format JSON records YAML manifests + code
Deployment Dynamic Fly.io machines Pre-built containers
Discovery Query by name Trigger-based matching
Versioning metadata.version field Git tags
Distribution Database queries Container registry
Status IMPLEMENTED VISION

This document reflects the actual state of Decision Orchestrator as of January 2025. The pack system described is a future vision based on roadmap documents in the codebase.

Artifact Output Patterns

How AI agents produce, structure, and present actionable outputs to users


The Core Insight

An artifact is a discrete, actionable unit of agent output. Unlike conversational text, artifacts are:

  • Structured and machine-parseable
  • Designed for a specific next action
  • Often rendered specially in UI
  • Versioned and comparable

The best agents don't just respond—they produce artifacts that enable action.


Artifact Taxonomy

┌─────────────────────────────────────────────────────────────────────────────┐
│                          ARTIFACT TYPES                                      │
├─────────────────────────────────────────────────────────────────────────────┤
│                                                                             │
│  CODE ARTIFACTS                         DOCUMENT ARTIFACTS                   │
│  ──────────────                         ──────────────────                   │
│  ┌─────────────────┐                    ┌─────────────────┐                 │
│  │ Type: code      │                    │ Type: document  │                 │
│  │ Lang: python    │                    │ Format: markdown│                 │
│  │ Executable: yes │                    │ Sections: yes   │                 │
│  │ Testable: yes   │                    │ Exportable: yes │                 │
│  └─────────────────┘                    └─────────────────┘                 │
│                                                                             │
│  VISUAL ARTIFACTS                       DATA ARTIFACTS                       │
│  ────────────────                       ──────────────                       │
│  ┌─────────────────┐                    ┌─────────────────┐                 │
│  │ Type: visual    │                    │ Type: data      │                 │
│  │ Format: SVG     │                    │ Format: JSON    │                 │
│  │ Renderable: yes │                    │ Schema: defined │                 │
│  │ Interactive: ?  │                    │ Queryable: yes  │                 │
│  └─────────────────┘                    └─────────────────┘                 │
│                                                                             │
│  COMPOSITE ARTIFACTS                    ACTION ARTIFACTS                     │
│  ───────────────────                    ────────────────                     │
│  ┌─────────────────┐                    ┌─────────────────┐                 │
│  │ Type: report    │                    │ Type: action    │                 │
│  │ Contains:       │                    │ Executable: yes │                 │
│  │  - text         │                    │ Reversible: ?   │                 │
│  │  - charts       │                    │ Confirmable: yes│                 │
│  │  - tables       │                    └─────────────────┘                 │
│  │  - code         │                                                        │
│  └─────────────────┘                                                        │
│                                                                             │
└─────────────────────────────────────────────────────────────────────────────┘

The Artifact Lifecycle

┌─────────────────────────────────────────────────────────────────────────────┐
│                      ARTIFACT LIFECYCLE                                      │
├─────────────────────────────────────────────────────────────────────────────┤
│                                                                             │
│  ┌────────────┐    ┌────────────┐    ┌────────────┐    ┌────────────┐      │
│  │  GENERATE  │───►│  VALIDATE  │───►│  PRESENT   │───►│   ACTION   │      │
│  └────────────┘    └────────────┘    └────────────┘    └────────────┘      │
│        │                 │                 │                 │              │
│        ▼                 ▼                 ▼                 ▼              │
│  Agent creates     Schema check      UI renders       User acts on         │
│  structured        Type validation   Special display  artifact             │
│  output            Completeness      Action buttons   (copy, run, etc.)    │
│                                                                             │
│  ════════════════════════════════════════════════════════════════════════  │
│                                                                             │
│                         ITERATION LOOP                                       │
│                                                                             │
│     ┌─────────────────────────────────────────────────────────┐            │
│     │                                                          │            │
│     │    User: "Make the chart blue"                          │            │
│     │         │                                                │            │
│     │         ▼                                                │            │
│     │    ┌─────────┐     ┌─────────┐     ┌─────────┐         │            │
│     │    │Artifact │────►│ Update  │────►│Artifact │         │            │
│     │    │  v1     │     │ Command │     │  v2     │         │            │
│     │    └─────────┘     └─────────┘     └─────────┘         │            │
│     │                                         │               │            │
│     │                                         │               │            │
│     │    Both versions preserved for comparison/rollback      │            │
│     │                                                          │            │
│     └─────────────────────────────────────────────────────────┘            │
│                                                                             │
└─────────────────────────────────────────────────────────────────────────────┘

Structured Output Schema Pattern

The key insight: artifacts need schemas for reliability.

┌─────────────────────────────────────────────────────────────────────────────┐
│                    SCHEMA-DRIVEN ARTIFACTS                                   │
├─────────────────────────────────────────────────────────────────────────────┤
│                                                                             │
│  WITHOUT SCHEMA                          WITH SCHEMA                         │
│  ──────────────                          ───────────                         │
│                                                                             │
│  Agent output:                           Agent output:                       │
│  "The ROI is about 2.5x,                 {                                   │
│   give or take, and                        "artifact_type": "analysis",     │
│   you should probably                      "metrics": {                     │
│   increase digital spend"                    "roi": {                       │
│                                                "value": 2.5,               │
│  Problems:                                     "unit": "multiple",         │
│  • Can't extract metrics                       "confidence": 0.87          │
│  • Can't compare versions                    }                              │
│  • Can't feed to downstream                },                               │
│  • UI can't render specially               "recommendations": [             │
│                                              {                              │
│                                                "action": "increase_spend", │
│                                                "channel": "digital",       │
│                                                "amount_pct": 15            │
│                                              }                              │
│                                            ]                                │
│                                          }                                  │
│                                                                             │
│                                          Benefits:                          │
│                                          • Machine-parseable               │
│                                          • Validatable                     │
│                                          • Comparable                      │
│                                          • UI can render richly            │
│                                                                             │
└─────────────────────────────────────────────────────────────────────────────┘

Artifact Presentation Patterns

How artifacts are displayed to users:

┌─────────────────────────────────────────────────────────────────────────────┐
│                    PRESENTATION PATTERNS                                     │
├─────────────────────────────────────────────────────────────────────────────┤
│                                                                             │
│  PATTERN 1: INLINE EXPANSION                                                 │
│  ───────────────────────────                                                 │
│                                                                             │
│     Conversation:                                                            │
│     ┌────────────────────────────────────────┐                              │
│     │ User: "Generate a budget allocation"   │                              │
│     │                                        │                              │
│     │ Agent: "Here's the optimized budget:"  │                              │
│     │                                        │                              │
│     │ ┌────────────────────────────────────┐ │                              │
│     │ │ [ARTIFACT: Budget Table]           │ │ ◄── Rendered inline          │
│     │ │                                    │ │                              │
│     │ │ Channel  │ Current │ Recommended  │ │                              │
│     │ │ ─────────┼─────────┼────────────  │ │                              │
│     │ │ TV       │ $500K   │ $420K        │ │                              │
│     │ │ Digital  │ $300K   │ $380K        │ │                              │
│     │ │                                    │ │                              │
│     │ │ [Copy] [Export CSV] [Apply]       │ │ ◄── Action buttons           │
│     │ └────────────────────────────────────┘ │                              │
│     └────────────────────────────────────────┘                              │
│                                                                             │
│  PATTERN 2: SIDE PANEL                                                       │
│  ─────────────────────                                                       │
│                                                                             │
│     ┌─────────────────────┬─────────────────────────────────┐               │
│     │   Conversation      │     Artifact Panel               │               │
│     │                     │                                  │               │
│     │  User: "Show me     │  ┌───────────────────────────┐  │               │
│     │   the analysis"     │  │  [LIVE PREVIEW]           │  │               │
│     │                     │  │                           │  │               │
│     │  Agent: "I've       │  │   Chart renders here      │  │               │
│     │   created a viz..." │  │   Updates in real-time    │  │               │
│     │                     │  │   as conversation         │  │               │
│     │  User: "Make it     │  │   progresses              │  │               │
│     │   a bar chart"      │  │                           │  │               │
│     │                     │  └───────────────────────────┘  │               │
│     │  Agent: "Updated."  │                                  │               │
│     └─────────────────────┴─────────────────────────────────┘               │
│                                                                             │
│  PATTERN 3: ARTIFACT GALLERY                                                 │
│  ───────────────────────────                                                 │
│                                                                             │
│     Session artifacts:                                                       │
│     ┌─────────┐  ┌─────────┐  ┌─────────┐  ┌─────────┐                     │
│     │ Chart   │  │ Code    │  │ Report  │  │ Config  │                     │
│     │ v3      │  │ v2      │  │ v1      │  │ v1      │                     │
│     │ [view]  │  │ [view]  │  │ [view]  │  │ [view]  │                     │
│     └─────────┘  └─────────┘  └─────────┘  └─────────┘                     │
│         ▲                                                                   │
│         └── Click to expand, compare with previous versions                 │
│                                                                             │
└─────────────────────────────────────────────────────────────────────────────┘

Action Buttons Pattern

Every artifact should have clear next actions:

┌─────────────────────────────────────────────────────────────────────────────┐
│                      ACTION BUTTON TAXONOMY                                  │
├─────────────────────────────────────────────────────────────────────────────┤
│                                                                             │
│  CODE ARTIFACTS                                                              │
│  ┌─────────────────────────────────────────────────────────────────────┐   │
│  │  [Copy to Clipboard] [Run in Sandbox] [Insert into File] [Explain] │   │
│  └─────────────────────────────────────────────────────────────────────┘   │
│                                                                             │
│  DATA ARTIFACTS                                                              │
│  ┌─────────────────────────────────────────────────────────────────────┐   │
│  │  [Download CSV] [Download JSON] [Open in Spreadsheet] [Visualize]  │   │
│  └─────────────────────────────────────────────────────────────────────┘   │
│                                                                             │
│  VISUAL ARTIFACTS                                                            │
│  ┌─────────────────────────────────────────────────────────────────────┐   │
│  │  [Download PNG] [Download SVG] [Edit Colors] [Resize] [Share]      │   │
│  └─────────────────────────────────────────────────────────────────────┘   │
│                                                                             │
│  ANALYSIS ARTIFACTS                                                          │
│  ┌─────────────────────────────────────────────────────────────────────┐   │
│  │  [Export Report] [Schedule Re-run] [Share with Team] [Add to Dash] │   │
│  └─────────────────────────────────────────────────────────────────────┘   │
│                                                                             │
│  RECOMMENDATION ARTIFACTS                                                    │
│  ┌─────────────────────────────────────────────────────────────────────┐   │
│  │  [Apply Changes] [Modify Parameters] [Reject] [Request Explanation]│   │
│  └─────────────────────────────────────────────────────────────────────┘   │
│                                                                             │
└─────────────────────────────────────────────────────────────────────────────┘

Streaming & Progressive Rendering

For long-running artifact generation:

┌─────────────────────────────────────────────────────────────────────────────┐
│                   PROGRESSIVE ARTIFACT RENDERING                             │
├─────────────────────────────────────────────────────────────────────────────┤
│                                                                             │
│  USER REQUEST: "Generate a comprehensive marketing report"                   │
│                                                                             │
│  TIME ────────────────────────────────────────────────────────────────►     │
│                                                                             │
│  t=0s     ┌─────────────────────────────────────────────┐                  │
│           │  [ARTIFACT: Report]                          │                  │
│           │                                              │                  │
│           │  Status: Generating...                       │                  │
│           │  ████░░░░░░░░░░░░░░░░░░░░  15%              │                  │
│           │                                              │                  │
│           │  Sections:                                   │                  │
│           │  ✓ Executive Summary (ready)                │                  │
│           │  ⟳ Channel Analysis (in progress)           │                  │
│           │  ○ Recommendations (pending)                 │                  │
│           │                                              │                  │
│           │  [View Available Sections]                   │                  │
│           └─────────────────────────────────────────────┘                  │
│                                                                             │
│  t=30s    ┌─────────────────────────────────────────────┐                  │
│           │  [ARTIFACT: Report]                          │                  │
│           │                                              │                  │
│           │  Status: Generating...                       │                  │
│           │  ████████████░░░░░░░░░░░░  55%              │                  │
│           │                                              │                  │
│           │  Sections:                                   │                  │
│           │  ✓ Executive Summary                         │                  │
│           │  ✓ Channel Analysis                          │                  │
│           │  ✓ ROI Breakdown                             │                  │
│           │  ⟳ Recommendations (in progress)            │                  │
│           │  ○ Appendix (pending)                        │                  │
│           │                                              │                  │
│           │  [View Completed Sections]                   │                  │
│           └─────────────────────────────────────────────┘                  │
│                                                                             │
│  t=60s    ┌─────────────────────────────────────────────┐                  │
│           │  [ARTIFACT: Report]                          │                  │
│           │                                              │                  │
│           │  Status: Complete ✓                          │                  │
│           │  ████████████████████████  100%             │                  │
│           │                                              │                  │
│           │  All 5 sections ready                        │                  │
│           │                                              │                  │
│           │  [Download PDF] [Share] [Schedule Update]    │                  │
│           └─────────────────────────────────────────────┘                  │
│                                                                             │
└─────────────────────────────────────────────────────────────────────────────┘

Artifact Comparison & Versioning

┌─────────────────────────────────────────────────────────────────────────────┐
│                    ARTIFACT VERSION COMPARISON                               │
├─────────────────────────────────────────────────────────────────────────────┤
│                                                                             │
│  ┌─────────────────────────┬─────────────────────────┐                     │
│  │  Budget Plan v1         │  Budget Plan v2         │                     │
│  │  (Yesterday)            │  (After feedback)       │                     │
│  ├─────────────────────────┼─────────────────────────┤                     │
│  │                         │                         │                     │
│  │  TV:      $500K         │  TV:      $420K    ↓    │                     │
│  │  Digital: $300K         │  Digital: $380K    ↑    │                     │
│  │  Social:  $200K         │  Social:  $200K    =    │                     │
│  │                         │                         │                     │
│  │  Total ROI: 2.3x        │  Total ROI: 2.7x   ↑    │                     │
│  │                         │                         │                     │
│  └─────────────────────────┴─────────────────────────┘                     │
│                                                                             │
│  Changes highlighted:                                                        │
│  • Shifted $80K from TV to Digital                                          │
│  • Expected ROI improvement: +17%                                           │
│                                                                             │
│  [Revert to v1] [Keep v2] [Merge manually]                                  │
│                                                                             │
└─────────────────────────────────────────────────────────────────────────────┘

Key Design Principles

Principle Implementation
Schema-First Define artifact structure before generation
Progressive Display Show what's ready while generating rest
Clear Actions Every artifact has obvious next steps
Versioning Track changes, enable comparison & rollback
Export Flexibility Multiple formats for different consumers
Inline Context Keep artifacts near relevant conversation

Common Pitfalls

┌─────────────────────────────────────────────────────────────────────────────┐
│                       ARTIFACT ANTI-PATTERNS                                 │
├─────────────────────────────────────────────────────────────────────────────┤
│                                                                             │
│  PROBLEM: Wall of Text                  SOLUTION: Structured Artifact        │
│  ─────────────────────                  ────────────────────────             │
│                                                                             │
│  "Here's your analysis:                 ┌──────────────────────────┐        │
│   The ROI for TV is 2.1x               │ [Analysis Artifact]      │        │
│   which is lower than digital          │                          │        │
│   at 3.2x but social is only           │ ROI Summary Table        │        │
│   1.8x so you should..."               │ Key Insight: Digital > TV│        │
│                                         │ [See Full Report]        │        │
│  Can't scan, extract, or act            └──────────────────────────┘        │
│                                         Scannable, actionable               │
│                                                                             │
│  PROBLEM: No Next Steps                 SOLUTION: Action Buttons             │
│  ──────────────────────                 ────────────────────────             │
│                                                                             │
│  "Here's the code."                     ┌──────────────────────────┐        │
│                                         │ [Code Artifact]          │        │
│  User: "Now what?"                      │                          │        │
│                                         │ [Run] [Copy] [Test]      │        │
│                                         └──────────────────────────┘        │
│                                                                             │
│  PROBLEM: Lost Artifacts                SOLUTION: Artifact Gallery           │
│  ───────────────────────                ────────────────────────             │
│                                                                             │
│  User: "Where's that chart             Session artifacts persisted          │
│   you made earlier?"                    and browsable in sidebar            │
│                                                                             │
│  Scroll, scroll, scroll...              One-click to find any output        │
│                                                                             │
└─────────────────────────────────────────────────────────────────────────────┘

Great artifacts are not just outputs—they're starting points for the next action. The difference between "here's some text" and "here's a structured artifact with clear next steps" is the difference between a chatbot and a productive AI assistant.

Session Deployment UX

How Decision Orchestrator deploys Claude sessions from Discord to Fly.io


Important: Current State vs. Future Vision

This document describes both the CURRENT deployment flow (which exists today) and a FUTURE "pack-based" deployment vision (which does not yet exist).


What EXISTS Today: Discord → Fly.io Sessions

The current Decision Orchestrator deploys sessions through a Discord-driven workflow:

┌─────────────────────────────────────────────────────────────────────────────┐
│                    CURRENT DEPLOYMENT FLOW (IMPLEMENTED)                     │
├─────────────────────────────────────────────────────────────────────────────┤
│                                                                             │
│  1. User sends message in Discord                                            │
│     └── "Create a marketing analysis session"                                │
│                                                                             │
│  2. Discord Bot receives message                                             │
│     └── Routes to Workflow Executor                                          │
│                                                                             │
│  3. Workflow Executor (Claude API with tools)                                │
│     └── Uses workflow_tools to process request:                              │
│         ├── Queries session_templates table (Supabase)                       │
│         ├── Calls create_fly_app() with template config                      │
│         ├── Calls deploy_fly_app() with system prompt                        │
│         └── Returns session URL to user                                      │
│                                                                             │
│  4. Fly.io Machine Created                                                   │
│     └── Claude instance running with:                                        │
│         ├── System prompt from template                                      │
│         ├── Tools config from template                                       │
│         └── ACP server for communication                                     │
│                                                                             │
│  5. Session Ready                                                            │
│     └── User can communicate via:                                            │
│         ├── Discord (through orchestrator)                                   │
│         └── ACP protocol (SSE endpoints)                                     │
│                                                                             │
└─────────────────────────────────────────────────────────────────────────────┘

Current Workflow Tools (Implemented)

From app/src/services/workflow_tools/fly_app_tools.py:

# Actual tools available today
async def create_fly_app(app_name: str, region: str = "sjc") -> dict:
    """Create a new Fly.io application"""

async def deploy_fly_app(app_name: str, image: str, env_vars: dict) -> dict:
    """Deploy an image to a Fly.io app"""

async def list_fly_apps() -> list:
    """List all Fly.io apps in the organization"""

async def scale_fly_app(app_name: str, count: int) -> dict:
    """Scale the number of machines for an app"""

async def restart_fly_machine(app_name: str, machine_id: str) -> dict:
    """Restart a specific Fly.io machine"""

async def get_fly_logs(app_name: str, lines: int = 100) -> str:
    """Get recent logs from a Fly.io app"""

async def set_fly_secrets(app_name: str, secrets: dict) -> dict:
    """Set secrets/environment variables for an app"""

Current Session Communication (Implemented)

From app/src/services/workflow_tools/session_comm_tools.py:

# ACP (Agent Communication Protocol) tools
async def send_message_to_session(session_id: str, message: str) -> dict:
    """Send a message to another Claude session via ACP"""

async def wait_for_human_decision(prompt: str, options: list) -> str:
    """Wait for human to make a decision"""

async def read_active_sessions() -> list:
    """List all active Claude sessions"""

async def request_session_spawn(template_name: str) -> dict:
    """Request the orchestrator to spawn a new session"""

Current User Journey

┌─────────────────────────────────────────────────────────────────────────────┐
│                    CURRENT USER JOURNEY (IMPLEMENTED)                        │
├─────────────────────────────────────────────────────────────────────────────┤
│                                                                             │
│  USER: "Help me analyze my marketing data"                                   │
│                                                                             │
│       │                                                                     │
│       ▼                                                                     │
│  ╔═══════════════════════════════════════════════════════════════════════╗ │
│  ║  DISCORD BOT RECEIVES MESSAGE                                          ║ │
│  ╠═══════════════════════════════════════════════════════════════════════╣ │
│  ║                                                                       ║ │
│  ║   Bot identifies intent: "Create specialized session"                  ║ │
│  ║                                                                       ║ │
│  ╚═══════════════════════════════════════════════════════════════════════╝ │
│       │                                                                     │
│       ▼                                                                     │
│  ╔═══════════════════════════════════════════════════════════════════════╗ │
│  ║  WORKFLOW EXECUTOR PROCESSES REQUEST                                   ║ │
│  ╠═══════════════════════════════════════════════════════════════════════╣ │
│  ║                                                                       ║ │
│  ║   1. Query Supabase for "marketing-analyst" template                   ║ │
│  ║   2. Get system_prompt, tools_config, mcp_config                      ║ │
│  ║   3. Call create_fly_app("session-{uuid}", "sjc")                     ║ │
│  ║   4. Call deploy_fly_app(...) with template config                    ║ │
│  ║   5. Wait for health check to pass                                    ║ │
│  ║                                                                       ║ │
│  ╚═══════════════════════════════════════════════════════════════════════╝ │
│       │                                                                     │
│       ▼                                                                     │
│  ╔═══════════════════════════════════════════════════════════════════════╗ │
│  ║  SESSION DEPLOYED                                                      ║ │
│  ╠═══════════════════════════════════════════════════════════════════════╣ │
│  ║                                                                       ║ │
│  ║   "Your marketing analysis session is ready!"                          ║ │
│  ║                                                                       ║ │
│  ║   Session URL: https://session-abc123.fly.dev                          ║ │
│  ║   ACP Endpoint: wss://session-abc123.fly.dev:3017                      ║ │
│  ║                                                                       ║ │
│  ║   You can now chat with your specialized assistant.                    ║ │
│  ║                                                                       ║ │
│  ╚═══════════════════════════════════════════════════════════════════════╝ │
│                                                                             │
└─────────────────────────────────────────────────────────────────────────────┘

Template-Based Deployment (Current)

Today, sessions are created from database templates:

┌─────────────────────────────────────────────────────────────────────────────┐
│                    TEMPLATE-BASED DEPLOYMENT (CURRENT)                       │
├─────────────────────────────────────────────────────────────────────────────┤
│                                                                             │
│   Supabase: session_templates                                                │
│   ┌─────────────────────────────────────────────────────────────────────┐   │
│   │ name: "marketing-analyst"                                            │   │
│   │ system_prompt: "You are a marketing analytics assistant..."          │   │
│   │ tools_config: {"allowed_tools": [...]}                               │   │
│   │ mcp_config: {"servers": [...]}                                       │   │
│   └─────────────────────────────────────────────────────────────────────┘   │
│                               │                                              │
│                               ▼                                              │
│   Workflow Executor calls:                                                   │
│   ┌─────────────────────────────────────────────────────────────────────┐   │
│   │ create_fly_app("session-marketing-123", region="sjc")               │   │
│   │ deploy_fly_app(                                                      │   │
│   │     app_name="session-marketing-123",                                │   │
│   │     image="decision-orchestrator-session:latest",                    │   │
│   │     env_vars={                                                       │   │
│   │         "SYSTEM_PROMPT": template.system_prompt,                     │   │
│   │         "TOOLS_CONFIG": json.dumps(template.tools_config),           │   │
│   │         "ANTHROPIC_API_KEY": secrets.anthropic_key                   │   │
│   │     }                                                                │   │
│   │ )                                                                    │   │
│   └─────────────────────────────────────────────────────────────────────┘   │
│                               │                                              │
│                               ▼                                              │
│   Fly.io Machine Running:                                                    │
│   ┌─────────────────────────────────────────────────────────────────────┐   │
│   │ • Claude instance with loaded system prompt                          │   │
│   │ • ACP server for communication                                       │   │
│   │ • Tools available per config                                         │   │
│   └─────────────────────────────────────────────────────────────────────┘   │
│                                                                             │
└─────────────────────────────────────────────────────────────────────────────┘

ACP Communication (Implemented)

The Agent Communication Protocol enables inter-session messaging:

┌─────────────────────────────────────────────────────────────────────────────┐
│                    ACP PROTOCOL (IMPLEMENTED)                                │
├─────────────────────────────────────────────────────────────────────────────┤
│                                                                             │
│   From .claude/skills/acp-communication/SKILL.md:                            │
│                                                                             │
│   ACP is the inter-session messaging protocol using SSE endpoints.           │
│                                                                             │
│   ┌─────────────────┐         SSE          ┌─────────────────┐             │
│   │  Orchestrator   │ ─────────────────►   │   Session A     │             │
│   │  (Discord Bot)  │ ◄─────────────────   │   (Fly.io)      │             │
│   └─────────────────┘                      └─────────────────┘             │
│            │                                        │                       │
│            │          SSE                           │                       │
│            └────────────────────────────────────────┤                       │
│                                                     ▼                       │
│                                            ┌─────────────────┐             │
│                                            │   Session B     │             │
│                                            │   (Fly.io)      │             │
│                                            └─────────────────┘             │
│                                                                             │
│   Communication Methods:                                                     │
│   • send_message_to_session(session_id, message)                            │
│   • Sessions authenticate via JWT tokens                                    │
│   • SSE endpoints for real-time messaging                                   │
│                                                                             │
└─────────────────────────────────────────────────────────────────────────────┘

FUTURE VISION: Pack-Based Deployment

NOTE: Everything below describes a FUTURE vision that does NOT yet exist in the codebase.

The Pack Deploy Vision

┌─────────────────────────────────────────────────────────────────────────────┐
│              FUTURE: PACK-BASED DEPLOYMENT (NOT IMPLEMENTED)                 │
├─────────────────────────────────────────────────────────────────────────────┤
│                                                                             │
│  CURRENT (Template-based):                                                   │
│  ────────────────────────                                                   │
│                                                                             │
│  User → Discord → Query template → Create Fly app → Deploy dynamically      │
│                                                                             │
│  ════════════════════════════════════════════════════════════════════════   │
│                                                                             │
│  FUTURE VISION (Pack-based):                                                 │
│  ───────────────────────────                                                │
│                                                                             │
│  User picks pack → Clone repo → Add data/keys → Deploy pre-built image      │
│       ↑                              ↑             ↑                        │
│   Pack registry               User customization  Already has               │
│   with discovery              is optional         Dockerfile!               │
│                                                                             │
└─────────────────────────────────────────────────────────────────────────────┘

Proposed Pack Deploy Flow (Future)

┌─────────────────────────────────────────────────────────────────────────────┐
│                       FUTURE: PACK DEPLOY FLOW                               │
├─────────────────────────────────────────────────────────────────────────────┤
│                                                                             │
│  USER: "I want to run marketing mix modeling"                                │
│                                                                             │
│       │                                                                     │
│       ▼                                                                     │
│  ╔═══════════════════════════════════════════════════════════════════════╗ │
│  ║  PHASE 1: PACK DISCOVERY                                               ║ │
│  ╠═══════════════════════════════════════════════════════════════════════╣ │
│  ║                                                                       ║ │
│  ║   Search: [marketing mix modeling                    ]                ║ │
│  ║                                                                       ║ │
│  ║   ┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐        ║ │
│  ║   │ pack-mmm        │ │ pack-attribution│ │ pack-forecast   │        ║ │
│  ║   │ ★★★★★ (124)     │ │ ★★★★☆ (87)      │ │ ★★★★★ (201)     │        ║ │
│  ║   │ Bayesian MMM    │ │ Multi-touch     │ │ Time series     │        ║ │
│  ║   │ [Select]        │ │ [Select]        │ │ [Select]        │        ║ │
│  ║   └─────────────────┘ └─────────────────┘ └─────────────────┘        ║ │
│  ║                                                                       ║ │
│  ╚═══════════════════════════════════════════════════════════════════════╝ │
│       │                                                                     │
│       ▼                                                                     │
│  ╔═══════════════════════════════════════════════════════════════════════╗ │
│  ║  PHASE 2: PACK CONFIGURATION                                          ║ │
│  ╠═══════════════════════════════════════════════════════════════════════╣ │
│  ║                                                                       ║ │
│  ║   pack-mmm v2.1.0                                                     ║ │
│  ║   Repository: github.com/decision-ai/pack-mmm                         ║ │
│  ║                                                                       ║ │
│  ║   ┌─────────────────────────────────────────────────────────────┐    ║ │
│  ║   │ Required:                                                    │    ║ │
│  ║   │   ANTHROPIC_API_KEY: [________________________] (required)   │    ║ │
│  ║   │                                                              │    ║ │
│  ║   │ Upload data:                                                 │    ║ │
│  ║   │   [Drop marketing_data.csv here or click to upload]          │    ║ │
│  ║   │                                                              │    ║ │
│  ║   │ [Deploy Pack]                                                │    ║ │
│  ║   └─────────────────────────────────────────────────────────────┘    ║ │
│  ║                                                                       ║ │
│  ╚═══════════════════════════════════════════════════════════════════════╝ │
│       │                                                                     │
│       ▼                                                                     │
│  ╔═══════════════════════════════════════════════════════════════════════╗ │
│  ║  PHASE 3: PACK DEPLOYMENT                                              ║ │
│  ╠═══════════════════════════════════════════════════════════════════════╣ │
│  ║                                                                       ║ │
│  ║   ┌─────────────────────────────────────────────────────────────┐    ║ │
│  ║   │ Deploying pack-mmm...                                        │    ║ │
│  ║   │                                                              │    ║ │
│  ║   │ ✓ Using pre-built image (registry.fly.io/pack-mmm:v2.1.0)   │    ║ │
│  ║   │ ✓ Injecting your data                                        │    ║ │
│  ║   │ ⟳ Starting Fly machine...                                    │    ║ │
│  ║   │                                                              │    ║ │
│  ║   │ Note: Using pre-built image—no Docker build needed!          │    ║ │
│  ║   └─────────────────────────────────────────────────────────────┘    ║ │
│  ║                                                                       ║ │
│  ╚═══════════════════════════════════════════════════════════════════════╝ │
│                                                                             │
└─────────────────────────────────────────────────────────────────────────────┘

Comparison: Current vs. Future

Aspect Current (Templates) Future (Packs)
User Interface Discord chat Discord + Web UI
Template Source Supabase database GitHub repositories
Deployment Dynamic Fly.io creation Pre-built container images
Customization Limited (env vars only) Full repo customization
Discovery Query by name Trigger-based matching
Build Time Minimal (image exists) Zero (pre-built) or custom
Status IMPLEMENTED VISION

Key UX Principles

These principles apply to BOTH current and future systems:

Principle Why It Matters
Show Your Work Explain what was detected and why; builds trust
Smart Defaults Minimize decisions, but allow overrides
Real Progress Show actual deployment status, not fake animations
Graceful Pauses Interrupt for secrets without losing progress
Actionable Errors Don't just say "failed"; say what to do
Verify Success Prove it works (health check, response time)
Clear Next Steps Always show what comes after deployment

This document reflects the actual state of Decision Orchestrator as of January 2025. The pack-based deployment vision is based on roadmap documents in the codebase.

Decision Orchestrator Implementation Roadmap

Executive Summary

This document provides a realistic roadmap based on the ACTUAL current state of Decision Orchestrator and the FUTURE vision from roadmap documents in the codebase.

KEY INSIGHT: The original 16-week roadmap described building from scratch. However, significant infrastructure ALREADY EXISTS. This updated roadmap acknowledges what's built and focuses on what's needed next.


Current State: What ALREADY EXISTS

Implemented Components

Based on analysis of the actual codebase:

Component Status Location
Discord Bot ✅ IMPLEMENTED app/src/bot/
Workflow Executor ✅ IMPLEMENTED app/src/services/workflow_executor.py
Fly.io Deployment Tools ✅ IMPLEMENTED app/src/services/workflow_tools/fly_app_tools.py
ACP Communication ✅ IMPLEMENTED app/src/services/workflow_tools/session_comm_tools.py
Session Templates (DB) ✅ IMPLEMENTED Supabase session_templates table
Human Interaction Tools ✅ IMPLEMENTED app/src/services/workflow_tools/human_interaction_tools.py
MCP Server ✅ IMPLEMENTED app/src/services/mcp_server.py
Supabase Integration ✅ IMPLEMENTED Multiple migrations in supabase/migrations/

Architecture Diagram (Current)

┌─────────────────────────────────────────────────────────────────────────────┐
│                    DECISION ORCHESTRATOR - CURRENT STATE                     │
├─────────────────────────────────────────────────────────────────────────────┤
│                                                                             │
│   ┌─────────────┐     ┌──────────────────┐     ┌───────────────────┐       │
│   │   Discord   │────▶│  Workflow        │────▶│   Fly.io          │       │
│   │   Bot       │     │  Executor        │     │   Machines        │       │
│   └─────────────┘     │  (Claude API)    │     │   (Sessions)      │       │
│         │             └──────────────────┘     └───────────────────┘       │
│         │                     │                         │                  │
│         │                     │                         │                  │
│         ▼                     ▼                         ▼                  │
│   ┌─────────────┐     ┌──────────────────┐     ┌───────────────────┐       │
│   │  Supabase   │◀───▶│  Workflow Tools  │◀───▶│   ACP Protocol    │       │
│   │  Database   │     │  - fly_app_tools │     │   (SSE)           │       │
│   │             │     │  - session_comm  │     │                   │       │
│   │  Tables:    │     │  - human_tools   │     │                   │       │
│   │  - sessions │     │  - mcp_tools     │     │                   │       │
│   │  - templates│     └──────────────────┘     └───────────────────┘       │
│   │  - messages │                                                          │
│   └─────────────┘                                                          │
│                                                                             │
└─────────────────────────────────────────────────────────────────────────────┘

What's NOT Built Yet (From Roadmap Docs)

Based on thoughts/shared/roadmap/ and thoughts/dynamic-builder-flow/:

Component Status Source Document
Builder Sessions ❌ NOT YET builder-sessions.md
Decision Packs ❌ NOT YET builder_as_claude_factory.md
Pack Registry ❌ NOT YET Future vision
Container Registry ❌ NOT YET Future vision
Pack Marketplace ❌ NOT YET Future vision
Web UI ❌ NOT YET Future vision

Updated Roadmap: Building on What Exists

Phase 0: Foundation (COMPLETED)

✅ Already implemented:

  • Discord bot interface
  • Workflow executor with Claude API
  • Fly.io deployment tools (create, deploy, scale, logs)
  • ACP inter-session communication
  • Supabase database (templates, sessions, messages)
  • Human interaction tools

Phase 1: Builder Sessions (2-3 Weeks)

Goal: Enable sessions to spawn other sessions dynamically

From thoughts/shared/roadmap/builder-sessions.md:

Task Description Dependencies
Builder Claude Template Create a specialized template that can create sessions Templates exist
Session Spawning API Enhance request_session_spawn to accept more parameters session_comm_tools
Template Discovery Allow sessions to list and select templates session_templates
Session Lifecycle Manage session start/stop/restart from other sessions fly_app_tools
┌─────────────────────────────────────────────────────────────────────────────┐
│                    PHASE 1: BUILDER SESSIONS                                 │
├─────────────────────────────────────────────────────────────────────────────┤
│                                                                             │
│   Current:                              After Phase 1:                       │
│   ────────                              ─────────────                        │
│                                                                             │
│   Discord → Workflow → Session          Discord → Workflow → Builder        │
│                                                      │        Session       │
│                                                      │            │         │
│                                                      ▼            ▼         │
│                                                   Session A   Session B     │
│                                                      │            │         │
│                                                      └────┬───────┘         │
│                                                           ▼                 │
│                                                      Session C              │
│                                                                             │
│   Sessions can now spawn other sessions!                                     │
│                                                                             │
└─────────────────────────────────────────────────────────────────────────────┘

Phase 2: Enhanced Templates (2-3 Weeks)

Goal: Move templates from pure database to hybrid (DB + Git)

Task Description Dependencies
Git URL Field Add git_repo_url to session_templates Migration
Template Cloning If git_url present, clone and extract .claude/ Builder tools
Template Versioning Track template versions with git refs Templates
Template Sharing Export templates as git repositories Builder tools
-- Migration: Add git support to templates
ALTER TABLE session_templates
ADD COLUMN git_repo_url TEXT,
ADD COLUMN git_ref TEXT DEFAULT 'main',
ADD COLUMN use_git_config BOOLEAN DEFAULT FALSE;

Phase 3: Pack Foundation (4-6 Weeks)

Goal: Implement the "Decision Pack" vision from research docs

From thoughts/dynamic-builder-flow/research/builder_as_claude_factory.md:

Task Description Dependencies
Pack Schema Define pack.yaml manifest format None
Pack CLI decision-pack create, deploy, list Pack schema
Pack Builder Build containers from pack repos Fly CLI
Pack Registry JSON index of available packs GitHub repos
┌─────────────────────────────────────────────────────────────────────────────┐
│                    PHASE 3: PACK SYSTEM                                      │
├─────────────────────────────────────────────────────────────────────────────┤
│                                                                             │
│   Pack Repository Structure:                                                 │
│                                                                             │
│   pack-{name}/                                                              │
│   ├── pack.yaml           # Manifest with triggers, deps                    │
│   ├── Dockerfile          # Container definition                            │
│   ├── fly.toml            # Fly.io config                                   │
│   ├── .claude/                                                              │
│   │   ├── CLAUDE.md       # System prompt                                   │
│   │   └── skills/         # Additional skills                               │
│   ├── acp-server/         # ACP communication                               │
│   └── app/                # Application code                                │
│                                                                             │
│   Deploy Flow:                                                              │
│                                                                             │
│   1. User: "Deploy pack-mmm"                                                │
│   2. System: Clone github.com/org/pack-mmm                                  │
│   3. System: Read pack.yaml for config                                      │
│   4. System: fly deploy --remote-only                                       │
│   5. System: Return session URL                                             │
│                                                                             │
└─────────────────────────────────────────────────────────────────────────────┘

Phase 4: Pack Ecosystem (6-8 Weeks)

Goal: Build the marketplace and discovery system

Task Description Dependencies
Pre-built Images CI/CD for pack images Pack repos
Pack Discovery Search packs by triggers Pack registry
Pack Ratings User ratings and reviews Database
Pack Marketplace UI Web interface for browsing Frontend

Revised Timeline

Week:  1   2   3   4   5   6   7   8   9  10  11  12  13  14
       │   │   │   │   │   │   │   │   │   │   │   │   │   │

Phase 0: Foundation (ALREADY COMPLETE)
═══════════════════════════════════════════════════════════════

Phase 1: Builder Sessions
       ├───────────┤
       │ Session   │
       │ Spawning  │

Phase 2: Enhanced Templates
                   ├───────────┤
                   │ Git-based │
                   │ Templates │

Phase 3: Pack Foundation
                               ├───────────────────┤
                               │ Pack Schema +     │
                               │ CLI + Builder     │

Phase 4: Pack Ecosystem
                                                   ├───────────────────┤
                                                   │ Registry +        │
                                                   │ Marketplace       │

Milestones:  ◆ Builder     ◆ Templates    ◆ Packs      ◆ Marketplace
            Week 3        Week 6         Week 10      Week 14

Success Metrics

Current Baseline (Already Achieved)

Metric Current Value
Discord bot functional ✅ Yes
Session deployment ✅ Works via workflow tools
ACP communication ✅ Implemented
Template system ✅ Supabase-based

Phase Targets

Metric Phase 1 Phase 2 Phase 3 Phase 4
Session spawn from session
Git-based templates -
Pack deployment - -
Pack marketplace - - -
Active packs 0 0 5 20+
Deploy time (pre-built) N/A N/A <60s <30s

Risk Assessment

Risk Probability Impact Mitigation
Builder complexity Medium High Start with simple spawn, iterate
Pack schema changes Medium Medium Version schema from start
Container build times High Medium Pre-build and cache images
Discovery accuracy Medium Low Use simple keyword matching first

Resource Requirements

Current Team Allocation

  • Backend: Workflow tools, templates, builder
  • Platform: Fly.io integration, CI/CD
  • Frontend: Future marketplace UI

Infrastructure

  • Existing: Supabase, Fly.io, Discord
  • Needed: Container registry (Fly.io or ghcr.io)

Next Actions

Immediate (This Week)

  1. Review existing request_session_spawn implementation
  2. Design builder template system prompt
  3. Test session-to-session communication

Short-term (Next 2 Weeks)

  1. Implement enhanced session spawning
  2. Add git_repo_url to templates
  3. Create first builder template

Medium-term (Weeks 3-6)

  1. Design pack.yaml schema
  2. Build pack CLI
  3. Create 3-5 reference packs

Summary

What Status Timeline
Core Infrastructure ✅ DONE Complete
Builder Sessions 🔨 NEXT Weeks 1-3
Enhanced Templates 📋 PLANNED Weeks 4-6
Pack System 📋 PLANNED Weeks 7-10
Marketplace 📋 PLANNED Weeks 11-14

The original roadmap assumed starting from zero. In reality, significant infrastructure already exists:

  • Discord bot ✅
  • Workflow executor ✅
  • Fly.io tools ✅
  • ACP protocol ✅
  • Templates (DB) ✅

The remaining work focuses on:

  1. Builder Sessions - Sessions that spawn sessions
  2. Git-based Templates - Templates from repositories
  3. Pack System - Full deployable repo pattern
  4. Marketplace - Discovery and sharing

This document reflects the actual state of Decision Orchestrator as of January 2025. Timeline estimates are based on building incrementally on existing infrastructure.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment