Skip to content

Instantly share code, notes, and snippets.

@kyleledbetter
Created December 26, 2024 23:25
Show Gist options
  • Save kyleledbetter/000d7a97a56a08bcb3d3f34ccc57c5ef to your computer and use it in GitHub Desktop.
Save kyleledbetter/000d7a97a56a08bcb3d3f34ccc57c5ef to your computer and use it in GitHub Desktop.
feat(): add product requirements for LLM

Product Requirements Doc

Our application leverages the latest innovations in React 19 and Next.js 15 (App Router) to deliver powerful real-time collaboration features. User authentication and database management are powered by Supabase, with data being transitioned from Neon Postgres to Supabase Postgres and fully integrated through Drizzle ORM. We utilize the Vercel AI SDK (referenced in /documentation/ai-sdk) to orchestrate AI-driven interactions, and NextAuth.js to optionally authenticate users with oauth2 connecting third-party software assets.

Implementation Plan

  1. Use React 19 & Next.js 15

    • Ensure the project dependencies match the versions in the package.json snippet provided ("react": "19.0.0-rc-...", "next": "15.0.3-canary.2").
    • Check the configuration of Next.js App Router in the /app directory, verifying that routes align with Next.js 15 standards.
    • Ensure all dependencies are compatible with the versions of React and Next.js specified.
  2. Integrate Supabase for Primary Auth & Database

    • Replace Neon Postgres references with Supabase, ensuring Drizzle ORM migrations reference the new Supabase instance.
    • Update any existing .env or config files to include Supabase credentials.
    • Keep the current auth directory structure for next-auth, but change the primary auth for this app over to Supabase
    • Validate read/write operations in the new environment:
      • Ensure workspace and project data is properly structured in the new Supabase schema.
    • Create a types file for the Supabase schema to ensure consistency and type safety.
  3. Configure Drizzle ORM

    • Adjust Drizzle configurations to map the new Supabase schema (replacing any direct references to Neon Postgres).
    • Update migration scripts to reflect the supabase schema tables and relationships.
    • Run validations to ensure consistent data shapes for workspace, project, and chat entities.
  4. Add NextAuth.js for Optional OAuth 3rd Party Connections

    • Use NextAuth.js (currently in the “@next-auth” or “next-auth” package) to allow third-party connections.
    • Create an abstract infrastructure framework so next-auth can easily be used to add any oauth2 provider for third party integrations to pull data from.
    • Make sure next-auth only provides connections to external APIs, not the primary authentication (since Supabase handles that).
  5. Multiple Agents using the Vercel AI SDK

    • Import the Vercel AI SDK from “/documentation/ai-sdk” references; connect it to the chat functionality.
    • Add AI agent logic (Orchestrator and specialized agents) within the existing Chat modules:
      • Orchestrator Agent: the primary chat interface for the group chat coordinates the agents and surfaces suggestions in each chat session.
      • Requirements Agent: gather key Project Requirements from Workspace Context, Project Context, and User Context through the Orchestrator Agent.
      • Architect Agent: receives the requirements from the Requirements Agent and collective context to create the Technical Architecture Document for the project, calling the Search Agent to fetch relevant external context like API documentation, and factors in the data schema of file documents, Oauth2 integrations, and used REST APIs.
      • Designer Agent: factors in the Technical Architecture Document and the Project Requirements to create the UI/UX Design Document for the project, calling the Search Agent to fetch relevant external context like UI framework documentation, creates custom color themes for the framework used (like Shadcn with Tailwind CSS, or MUI Material UI)
      • Coding Agent: receives the Technical Architecture Document, UI/UX Design Document, and the Project Requirements to create the full-stack code for the project, ensuring it only uses the available libraries and dependencies installed, and the REST APIs and Oauth2 integrations that are available in the Workspace.
      • Quality Agent: reviews the streaming code from the Coding Agent and ensures is compatible and has no bugs, and corrects code with errors and overwrites the code with the corrected version. Reviews final rendered code for errors and corrects them by rewriting the code until there are no errors.
      • Search Agent: fetch relevant external data using Perplexity AI and Gemini search grounding
      • Scanning Agent: parse workspace/project file documents and fetched API data to create a Context Document for the AI Agents to use
      • Data Validation Agent: ensures the data is valid and consistent with the data schema of the file documents, and the REST APIs and Oauth2 integrations that are available in the Workspace, and that the numbers are accurate.
    • Ensure the Orchestrator Agent (which will use a reasoning LLM) coordinates these roles and surfaces suggestions in each chat session.
  6. Establish Workspaces & Projects Structure

    • Create or update tables/models for Workspaces and Projects in Supabase.
      • Workspaces store role-based access, shared settings, and user membership.
      • Projects inherit from Workspace, with project-specific roles, file storage references, and configuration overrides.
    • Integrate Supabase S3 storage for Project static file uploads (documents, images).
    • Project Chats are stored in Supabase, and are real-time synced to all connected users, pulling in all the context from the Workspace and Project
  7. Workspace and Project REST API Connection Storing

    • Provide an interface to store REST API credentials securely in Supabase for Workspace and Project level connections to be used in the AI Group Chat.
    • Ensure Projects can inherit or override REST API credentials from their parent Workspace.
    • Integrate these REST APIs and documents (via Scanning Agent's Context Document) in the AI Chat to build or update Artifacts with live data.
  8. Enable Real-Time Collaboration for AI Group Chats within Projects

    • Ensure Chat sessions in a Project allow multiple connected users simultaneously.
    • Use Supabase’s real-time capabilities (or alternative real-time mechanisms) to sync chat logs, agent outputs, and user input across all participants.
    • Users can chat with each other by mentioning them, or to the Orchestrator Agent, or to specific agents by mentioning them (like @architect, @designer, @coder, @quality, @search, @scanning, @data-validation)
  9. Generative UI and AI SDK Tools in Chat Messages

    • Use the Vercel AI SDK to render inline UI components in the chat message stream for confirmations, actions, responses, and other UI elements like:
      • Asking the user to confirm Search Queries
      • Asking the user to confirm the API calls to make, and updating the saved APIs in the Project
      • Asking the user if the Project documentation should be updated (like the Technical Architecture Document or the UI/UX Design Document)
      • Rendering inline Charts and Graphs for data validation (using echarts-for-react or recharts)
      • Generating an AI image in the chat message stream using Replicate.com FLUX
    • Use the Vercel AI SDK to create a UI for the AI Agents to use, and to create tools for the AI Agents to use.
  10. Generate AI Artifacts from Chat Sessions

  • Code the logic to create various artifact types (full-stack code, PDFs, slides, images, etc.) as outputs from AI Chat.
  • Artifact type is dynamically generated based on the requirements and the context of the project, and verified by the Group Chat users via a UI in the chat message stream.
  • Permit collaborative editing among real-time participants and AI agents before finalizing the Artifact.
  • Use Dynamic React Components to render the artifacts in the UI.
  • Complex artifacts and have multiple files (client and server side) to render dynamically.
  1. Product Statistics
  • Storage statistics total numbers of files, total numbers of artifacts, total numbers of chats, total numbers of users, total numbers of workspaces, total numbers of projects
  • Stats can be filtered by workspace, project, role, user, date range
  • Stats are stored in Supabase and can be queried by the user with their level of access and role
  • Supabase has a function to generate these stats and store them in the database which runs daily or on demand
  1. Token and Usage Tracking
  • Track the number of tokens used within the chat sessions, and the number of tokens available to the user
  • The number of tokens available to the user is based on the number of tokens available via the Project and Workspace settings
  • Tokens are tracked in Supabase at the Workspace level, and Projects within the Workspace can use them
  • Projects in Workspaces have budgets of tokens available to them, and can be overridden on the Project level by Workspace Admins
  • Workspace Owners can purchase budgets of tokens for their Workspace and Projects adhoc, and can be purchased via Stripe
  1. Stripe Integration
  • Integrate Stripe for billing and payments
  • Create a Stripe dashboard for the Workspace for Workspace Admins to manage the billing and payments
  • 3 levels of pricing plans:
    • Free (1000000 tokens) # 1 million tokens per workspace
    • Hobby (15000000 tokens) # 15 million tokens per workspace
    • Pro (50000000 tokens) # 50 million tokens per workspace
    • Enterprise (100000000 tokens) # 100 million tokens per workspace
  • Additionally purchase tokens for the Workspace and Projects adhoc via Stripe
    • Buckets of additional tokens available:
      • 1000000 tokens per workspace
      • 5000000 tokens per workspace
      • 10000000 tokens per workspace
  1. Audit Logging
  • Audit logging for all actions in the application, including chat sessions, artifacts, users, workspaces, projects, and roles
  • Audit logging is stored in Supabase at the Workspace level, Project level, and Chat level (and Globally for Super Users)
  1. Testing and Verification
  • Set up automated tests for:
    • Auth flows (Supabase primary, NextAuth.js OAuth).
    • Database migrations and Drizzle ORM.
    • AI chat orchestration and agent collaboration.
    • Workspace/Project role inheritance.
  • Verify real-time features handle concurrency (multiple user input streams, AI agent updates).
  1. Deployment
  • Code will be deployed to Vercel via Github Actions and Pull Requests.
  • Confirm environment variables for Supabase, NextAuth.js, and any third-party REST APIs are set in production.
  • Validate performance with lazy loading and minimal “use client” statements where necessary.
  • Test that production logs, metrics, and Next.js monitoring capture successful usage.
  1. Feature Flags
  • Setup logic to use Vercel Feature Flags to control the behavior of the application.
  1. Analytics
  • Setup providers and logic to use @vercel/analytics to track the behavior of the application.
  • Also setup logic for @vercel/speed-insights to track the performance of the application.
  • Use a provider framework to easily add any other analytics providers like Fullstory
  1. Observability
  • Setup providers and logic to plan for observability of the application.

Detailed Documentation

Workspace Workspaces are created for companies to manage their projects and teams. There are workspace-level roles and permissions, and workspace-level REST API connections. Workspaces build global context for the company, and are used to manage the projects and teams within the company.

-Workspace Settings

  • Workspace Name
  • Workspace Description
  • Workspace Billing
    • Plan (Free, Hobby, Pro, Enterprise)
    • Account Details (name, email, phone number, address, etc)
    • Stripe Integration (used to manage the billing and payments for the workspace)
  • Workspace Usage Dashboard (used to track the usage of the workspace, including the number of tokens used, the number of tokens available, the number of users, the number of projects, the number of chats, the number of artifacts, the number of files, the number of REST APIs, the number of Oauth2 integrations, the number of file storage, the number of environment variables, the number of users, the number of roles, the number of logs, the number of audits, the number of feature flags, the number of analytics, the number of observability)
  • Company Name
  • Company Email Domains (used to verify company email addresses, pulled from the email address of the user creating the workspace)
  • Workspace Logo (stored as base64 image in the database)
  • Workspace Color Palette (using a color palette generator for Shadcn/Tailwind CSS)
  • Workspace REST API Connections (register and store API keys for external REST APIs information, so Project users in this workspace can GET/POST/PUT/DELETE to them in their projects)
  • Workspace Oauth2 Connections (toggle available integrations like Hubspot, Salesforce, etc, so Project users in this workspace can connect to them in their projects)
  • Workspace File Storage (manage global file storage for the workspace, and the projects within the workspace)
  • Workspace Environment Variables (manage global environment variables for the workspace, and the projects within the workspace, used in features like image generation and chatbots in artifacts)
  • Workspace Users (manage users in the workspace, and their roles)
  • Workspace Roles:
    • Workspace Owner (single owner, can manage workspace settings, assign admins, all users and projects in the workspace)
    • Workspace Admins (can manage workspace settings, cannot assign admins, can manage all users and projects in the workspace)
    • Workspace Project Creators (can create and manage projects in the workspace, cannot manage workspace settings, cannot manage users in the workspace)
  • Workspace Default System Prompt (used to set the default system prompt for the chats in the workspace, and can be overridden by the project)
  • Workspace Context
    • Company Context (context about the company history, industry, regions, number of employees, etc)
    • Company Goals (context about the company goals, and the projects within the company)
    • Compliance Context (context about the company compliance requirements, and the projects within the company, which is factored into the Project Requirements and Technical Architecture Document for Projects for how data is used and how integrations are used)
    • Security Context (context about the company security requirements, and the projects within the company, which is factored into the Technical Architecture Document for Projects for how data is stored and accessed)

Project Projects are created for each project within a Workspace. Projects inherit the workspace-level REST API connections, and have project-level roles and permissions. Projects build project-level context for the project, and are used to manage the project and its teams.

-Project Settings

  • Project Name

  • Project Description

  • Project Usage Dashboard (used to track the usage of the project, including the number of tokens used, the number of tokens available, the number of users, the number of chats, the number of artifacts, the number of files, the number of REST APIs, the number of Oauth2 integrations, the number of file storage, the number of environment variables, the number of users, the number of roles, the number of logs, the number of audits, the number of feature flags, the number of analytics, the number of observability)

  • Project Logo (stored as base64 image in the database, overrides the workspace logo)

  • Project Color Palette (overrides the workspace color palette, using a color palette generator for Shadcn/Tailwind CSS)

  • Project REST API Connections (register and store API keys for external REST APIs information, so Project users in this project can GET/POST/PUT/DELETE to them in their project)

  • Project Oauth2 Connections (toggle specific integrations from those available in the workspace, like Hubspot, Salesforce, etc, so Project users in this project can connect to them in their project)

  • Project File Storage (manage global file storage for the project, and the projects within the workspace)

  • Project Environment Variables (manage global environment variables for the project, and the projects within the workspace, used in features like image generation and chatbots in artifacts)

  • Project URLs (a saved list of URLs that are relevant to the project context, like the project website, the project documentation, the project API documentation, the project code repository, etc)

  • Project Users (manage users in the project, and their roles)

  • Project Roles:

    • Project Owner (single owner, can manage project settings, assign admins, all users and projects in the project)
    • Project Managers (can manage project settings, cannot assign Project Managers, can manage all users and projects in the project)
    • Project Contributors (can create and manage chats in the project, cannot manage project settings, cannot manage users in the project)
    • Project Artifact Viewers (can view the artifacts in the project, cannot manage project settings, cannot manage users in the project, cannot create or manage chats in the project)
  • Project Default System Prompt (used to set the default system prompt for the chats in the project, and can be overridden by the project)

  • Project Context

    • Project Requirements Document (Project Requirements, factored into all other context in the project)
    • Project Technical Architecture Document (Technical architecture of the project, APIs and integrations used, data schemas, and how the project will be built)
    • Project UI/UX Design Document (UI/UX design of the project, including the UI framework and color palette, and overview of the UI components and how they will be used)
    • Project Code Document (Coding plan for the project, including the code architecture, the specific list of files and their contents, and how the project will be built)
    • Project Search Context (Search queries and results from Perplexity AI and Gemini search grounding, used to build the Technical Architecture Document and the UI/UX Design Document)
    • Project Data Context (Scanned Documentation about the data in files and databases of the project, including the data schema, and how the project will be built)
    • Project Quality Context (Quality checks and results from the Quality Agent, used to build the Code Document)
  • Project Artifacts (the primary output of the project, the full-stack code, PDFs, slides, images, etc, created in Chat sessions)

    • Artifact Title
    • Artifact Description
    • Artifact Type (code, pdf, slide, image, etc)
    • Artifact Files (the files of the artifact, stored in Supabase S3)
      • Artifact File Name
      • Artifact File Type
      • Artifact File Size
      • Artifact File Path (the path and file name of the artifact when building or rendering the app, like "app/api/chat/route.ts")
      • Artifact Main File (mark the file as the main file of the artifact when building or rendering the app, only one file can be the main file)
      • Artifact File Storage Path
      • Artifact File Created At
      • Artifact File Created By
      • Artifact File Updated At
      • Artifact File Updated By
    • Artifact URL (the URL of the artifact for public access)
    • Artifact Created At (the date and time the artifact was created)
    • Artifact Created By (the user that created the artifact)
    • Artifact Status (the status of the artifact, like pending, in progress, completed, or failed)
    • Artifact Progress (the progress of the artifact, like 10% complete, 20% complete, etc)
    • Artifact Errors (the errors of the artifact, if any)
    • Artifact Logs (the logs of the artifact, if any)
    • Artifact Sandbox (sandbox ID, template, user ID, project ID, artifact ID, created at, updated at)
    • Artifact Users (users that have access to the artifact, and their roles)

Chat Sessions Chats are creative group chat sessions started within a project, and are real-time synced to all connected users, pulling in all the context from the Workspace (and context within) and Project (and context within), and any data or files or messages within the chat history.

  • Chat ID
  • Chat Title
  • Chat Description
  • Chat Created At
  • Chat Created By
  • Chat Users (users that have access to the chat, and their roles. Workspace admins can access all chats in the workspace, Project Managers can access all chats in the project, Project Contributors can access all chats in the project, and Project Artifact Viewers can only access the artifacts in the chat)
  • Chat Messages (the messages in the chat, including the message text, the message type, the message created at, the message created by, the message status, the message errors, the message logs, the message sandbox, the message users)
  • Chat Artifacts (the artifacts in the chat, including the artifact title, the artifact description, the artifact type, the artifact files, the artifact URL, the artifact created at, the artifact created by, the artifact status, the artifact progress, the artifact errors, the artifact logs, the artifact sandbox, the artifact users)

Chat Messages

  • Chat Message ID
  • Chat Message Text
  • Chat Message Type (user, ai, system)
  • Chat Message Created At
  • Chat Message Created By (the user or agent that created the chat message)
  • Chat Message Tools (the tools available to the chat message, like the search tool, the scanning tool, the coding tool, the quality tool, the design tool, the code tool, the data validation tool)

API Endpoints

  • API Endpoint Name
  • API Endpoint URL
  • API Endpoint Authorization Header (ex: "Bearer", or "Basic", or "API Key")
  • API Endpoint Token (the token for the API, if any)
  • API Endpoint Custom Headers (custom headers for the API, if any)
  • API Endpoint Method (the method for the API, like GET, POST, PUT, DELETE)
  • API Endpoint Request Body (the request body for the API, if any)
  • API Endpoint Path (the starting path for the API response)
  • API Endpoint Schema (an abstract schema for the API, which is used to generate the API calls in the chat messages)

Users

  • Users are added to the global users table when they auth with Supabase, and are added to the workspace and project users tables when they are added to the workspace and project.
  • A supabase trigger copies the user from the Supabase auth.users table to the users table.
  • Another supabase trigger checks the user's email domain against the workspace's company email domains, and if it matches, the user is added to the matching workspace users table.
  • The users table columns match the Supabase auth.users table columns, and are extended with the workspace roles, project roles, and chat roles.
  • An additional profile jsonb column is included to extend the user profile
  • An additional status column is included with the statuses: invited, waitlist, active, inactive, banned, archived
  • A supabase trigger updates the status of the user when the user is added to the workspace or project, or when the user is invited to the workspace or project. If a user signs up without an email domain that matches the workspace's company email domains and without an invitation, the user is added to the waitlist.

Default Global Workspace The database is initialized with a default global workspace only managed by the Super User. A supabase trigger adds all new users to the global workspace users table.

Personal Project A supabase trigger creates a personal project for each user when they first sign up for the application. The personal project belongs to the global workspace and inherits all the settings from the global workspace. The personal project is used to manage the user's personal chats, artifacts, and files. The personal project is not visible to other users in the workspace, unless invited by the user. The personal project is archived but not deleted when the user is deleted from the global users table.


Implementation Checklists

Database & Authentication Setup

  • Design and implement Supabase schema

    • Workspace tables (settings, users, roles, API connections)
    • Project tables (settings, users, roles, artifacts)
    • Chat tables (sessions, messages, participants)
    • Context tables (requirements, architecture, design docs)
    • File storage references and metadata
    • Implement RLS policies for multi-user access
  • Configure authentication flows

    • Set up Supabase primary auth with email/password
    • Implement NextAuth.js for OAuth2 providers
    • Create auth middleware and protected routes
    • Design role-based access control (RBAC)

AI Agent Infrastructure

  • Implement core agent framework

    • Build Orchestrator Agent with reasoning capabilities
    • Create agent routing and coordination system
    • Implement agent state management
    • Design error handling and fallback systems
  • Develop specialized agents

    • Requirements Agent

      • Context extraction from workspace/project
      • Requirements document generation
      • Validation workflows
    • Architect Agent

      • Technical architecture planning
      • API integration design
      • Database schema recommendations
      • Security and compliance integration
    • Designer Agent

      • UI/UX document generation
      • Color theme creation
      • Component architecture planning
      • Accessibility considerations
    • Coding Agent

      • Code generation pipeline
      • Framework-specific implementations
      • Type safety enforcement
      • Integration with existing codebase
    • Quality Agent

      • Code review system
      • Error detection
      • Automated fixes
      • Performance optimization
    • Search Agent

      • Perplexity AI integration
      • Gemini search implementation
      • Result filtering and ranking
    • Scanning Agent

      • Document parsing system
      • Context extraction
      • Metadata generation
    • Data Validation Agent

      • Schema validation
      • Data consistency checks
      • API response validation

Build Workspaces

  • Manage All Workspaces (only for Super Users)
  • CRUD (Create, Read, Update, Delete) Workspaces
  • Workspace Settings Pages & API Endpoints
  • Workspace Manage Users & Roles Pages & API Endpoints
  • Workspace REST API Connections Pages & API Endpoints
  • Workspace Oauth2 Connections Pages & API Endpoints
  • Workspace File Storage Pages & API Endpoints
  • Workspace Environment Variables Pages & API Endpoints
  • Workspace Default System Prompt Pages & API Endpoints

Build Projects

  • Manage Projects (only for Workspace Admins and Super Users for all workspaces)
  • CRUD (Create, Read, Update, Delete) Projects
  • Project Settings Pages & API Endpoints
  • Project Manage Users & Roles Pages & API Endpoints
  • Project REST API Connections Pages & API Endpoints
  • Project Oauth2 Connections Pages & API Endpoints
  • Project File Storage Pages & API Endpoints
  • Project Environment Variables Pages & API Endpoints
  • Project Default System Prompt Pages & API Endpoints

Build Chat Sessions

  • Manage All Chat Sessions (only for Super Users)
  • New Chat Session Pages & API Endpoints
  • Collaborative Group Chat Page Features & API Endpoints
  • AI Agent Chat Page Features & API Endpoints

Build Chat Messages & Tools

  • Add Chat Message Tool Functions & UIs

Build Artifacts

  • Manage All Artifacts (only for Super Users)
  • Chat Artifact Creation
  • Invite Users to Artifact

Build Context Management

  • Build context system
    • Workspace context management
    • Project context management
    • Chat history integration
    • Document indexing and retrieval

Build Integration & APIs

  • Implement API management
    • REST API credential storage
    • Workspace REST API Connections
    • Project REST API Connections
  • Implement OAuth2 connection handling
    • Flexible next-auth integration for OAuth2 providers
    • Flexible next-auth integration for callback URLs
    • Store refresh tokens securely in Supabase
    • Workspace Oauth2 Connections
    • Project Oauth2 Connections

Real-Time Collaboration

  • Implement chat infrastructure

    • Real-time message syncing
    • User presence system
    • Message queuing and delivery
    • Typing indicators
    • Mention system (@user, @agent)
  • Build artifact collaboration

    • Real-time artifact editing
    • Version control system
    • Conflict resolution
    • Change tracking

UI Components & Interactions

  • Develop chat interface

    • Message streaming
    • Code block rendering
    • Inline UI components
    • Agent interaction controls
  • Create artifact viewers/editors

    • Code editor integration
    • PDF viewer
    • Image generation interface
    • Chart/graph rendering

Performance & Monitoring

  • Setup monitoring systems

    • @vercel/analytics integration
    • @vercel/speed-insights setup
    • Error tracking
    • Performance metrics
  • Implement optimizations

    • Lazy loading
    • Code splitting
    • Cache management
    • Database query optimization

Testing & Quality Assurance

  • Develop test suites
    • Unit tests for agents
    • Integration tests for API flows
    • E2E tests for critical paths
    • Performance benchmarks

Deployment & DevOps

  • Configure deployment pipeline
    • GitHub Actions setup
    • Environment management
    • Feature flag system
    • Rollback procedures

Documentation

  • Create documentation
    • API documentation
    • Agent interaction guides
    • User guides
    • Development documentation
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment