Resourcly Architecture Diagrams - System Architecture, Data Flow, and Network Topology

01-system-architecture.md

Resourcly System Architecture

AI-powered manufacturing inventory harmonization platform deployed on GCP (europe-west3).

flowchart TB
    subgraph Users["👤 Users"]
        Portal["Portal<br/>portal.resourcly.com"]
        Admin["Admin Dashboard"]
    end

    subgraph Auth["🔐 Authentication"]
        Firebase["Firebase Auth<br/>Google + Microsoft SSO"]
    end

    subgraph Frontend["📱 Frontend Services"]
        ReactPortal["React 19 + Vite<br/>Frontend"]
        ReactAdmin["React 19 + Vite<br/>Admin"]
    end

    subgraph Gateway["🚪 API Gateway"]
        GatewayAPI["Gateway<br/>Go 1.24 + Chi<br/>:8080"]
        GatewayWorkers["Gateway Workers<br/>Pub/Sub Consumers"]
    end

    subgraph MLServices["🤖 ML Services"]
        InternalAPI["Internal API<br/>FastAPI + Python<br/>:8000"]
        VertexSearch["Vertex AI<br/>Vector Search"]
    end

    subgraph DocWorkers["📄 Document Workers"]
        MistralOCR["Mistral OCR<br/>Pixtral + Claude"]
        DocWorker["Document Worker<br/>Embeddings"]
        CADWorker["CAD Worker<br/>DXF/STEP/JT"]
        TechDrawing["Technical Drawing<br/>Claude Vision"]
        BatchCoord["Batch Coordinator<br/>Anthropic Batch"]
    end

    subgraph SpecWorkers["📋 Spec Workers"]
        SpecExtract["Spec Extraction<br/>RAG Matching"]
        SpecDedup["Spec Deduplication"]
    end

    subgraph DataWorkers["📊 Data Workers"]
        DataSource["Data Source Workers<br/>Go + rclone"]
        Datalake["Datalake<br/>DuckDB + Parquet"]
    end

    subgraph Messaging["📬 Messaging"]
        PubSub["Cloud Pub/Sub<br/>13+ Topics + DLQs"]
    end

    subgraph Storage["💾 Data Stores"]
        PostgreSQL["Cloud SQL<br/>PostgreSQL 15<br/>+ pgvector"]
        Elasticsearch["Elastic Cloud<br/>Full-text Search"]
        GCS["Cloud Storage<br/>PDFs, Images, Parquet"]
        Redis["Memorystore Redis<br/>Caching"]
        Neo4j["Neo4j Aura<br/>Graph Visualization"]
        AnalyticsDB["Analytics PostgreSQL<br/>Aggregated Data"]
    end

    subgraph ExternalAI["🧠 External AI"]
        Mistral["Mistral AI<br/>Pixtral OCR"]
        Anthropic["Anthropic Claude<br/>Spec Extraction"]
        VertexEmbed["Vertex AI<br/>Embeddings API"]
        DocAI["Document AI<br/>OCR Processor"]
    end

    %% User Flow
    Portal --> Firebase
    Admin --> Firebase
    Firebase --> ReactPortal
    Firebase --> ReactAdmin
    ReactPortal --> GatewayAPI
    ReactAdmin --> GatewayAPI

    %% Gateway connections
    GatewayAPI --> PostgreSQL
    GatewayAPI --> Elasticsearch
    GatewayAPI --> Redis
    GatewayAPI --> Neo4j
    GatewayAPI --> GCS
    GatewayAPI --> PubSub
    GatewayAPI --> InternalAPI

    %% ML Service connections
    InternalAPI --> VertexSearch
    InternalAPI --> VertexEmbed

    %% Pub/Sub to Workers
    PubSub --> GatewayWorkers
    PubSub --> MistralOCR
    PubSub --> DocWorker
    PubSub --> CADWorker
    PubSub --> TechDrawing
    PubSub --> BatchCoord
    PubSub --> SpecExtract
    PubSub --> SpecDedup
    PubSub --> DataSource
    PubSub --> Datalake

    %% Worker to External AI
    MistralOCR --> Mistral
    MistralOCR --> Anthropic
    TechDrawing --> Anthropic
    BatchCoord --> Anthropic
    DocWorker --> VertexEmbed
    DocWorker --> DocAI

    %% Worker to Storage
    MistralOCR --> GCS
    DocWorker --> GCS
    CADWorker --> GCS
    DataSource --> GCS
    Datalake --> GCS
    Datalake --> AnalyticsDB

    %% Gateway Workers to Storage
    GatewayWorkers --> PostgreSQL
    GatewayWorkers --> Elasticsearch
    GatewayWorkers --> Neo4j

    classDef users fill:#e1f5fe,stroke:#01579b
    classDef auth fill:#fff3e0,stroke:#e65100
    classDef frontend fill:#f3e5f5,stroke:#7b1fa2
    classDef gateway fill:#e8f5e9,stroke:#2e7d32
    classDef ml fill:#fce4ec,stroke:#c2185b
    classDef workers fill:#fff8e1,stroke:#f9a825
    classDef messaging fill:#e0f2f1,stroke:#00695c
    classDef storage fill:#e3f2fd,stroke:#1565c0
    classDef external fill:#fafafa,stroke:#616161

    class Portal,Admin users
    class Firebase auth
    class ReactPortal,ReactAdmin frontend
    class GatewayAPI,GatewayWorkers gateway
    class InternalAPI,VertexSearch ml
    class MistralOCR,DocWorker,CADWorker,TechDrawing,BatchCoord,SpecExtract,SpecDedup,DataSource,Datalake workers
    class PubSub messaging
    class PostgreSQL,Elasticsearch,GCS,Redis,Neo4j,AnalyticsDB storage
    class Mistral,Anthropic,VertexEmbed,DocAI external

Loading

Key Architectural Principles

Principle	Description
Gateway-centric writes	Only Gateway writes to PostgreSQL and Elasticsearch
Event-driven	Workers communicate via Pub/Sub, not direct DB access
Multi-tenant	All data scoped by customer_id from Firebase Auth
Stateless K8s	Only stateless apps in GKE, managed services for state
100M+ scale	Vertex AI Vector Search with Tree-AH algorithm

02-data-flow.md

Resourcly Data Flow Diagram

Document processing pipeline from PDF upload to searchable inventory item.

flowchart TD
    subgraph Upload["📤 Document Upload"]
        User["User uploads PDF"]
        GatewayUpload["Gateway receives file"]
        StoreGCS["Store PDF in GCS"]
        PublishOCR["Publish to<br/>mistral-ocr topic"]
    end

    subgraph OCR["🔍 OCR Processing"]
        MistralWorker["Mistral OCR Worker"]
        MistralAI["Mistral AI Pixtral<br/>Extract raw text"]
        ClaudeExtract["Anthropic Claude<br/>Structured spec extraction"]
        PublishResults["Publish results"]
    end

    subgraph Embeddings["🧮 Embedding Generation"]
        DocWorker["Document Worker"]
        ExtractImages["Extract images<br/>PyMuPDF"]
        TextEmbed["Generate text embeddings<br/>Vertex AI text-embedding-005<br/>768 dimensions"]
        ImageEmbed["Generate image embeddings<br/>Vertex AI multimodal<br/>1408 dimensions"]
        UploadImages["Upload images to GCS"]
        PublishProcessed["Publish to<br/>document-processed"]
    end

    subgraph SpecProcessing["📋 Specification Processing"]
        SpecWorker["Spec Worker"]
        RAGMatch["RAG matching against<br/>Spec Registry"]
        UCUMConvert["UCUM unit conversion"]
        PublishSpecs["Publish to<br/>spec-matching-results"]
    end

    subgraph Persistence["💾 Data Persistence"]
        GatewayWorker["Gateway Worker"]
        InsertPG["INSERT into PostgreSQL<br/>documents, pages, images"]
        InsertVector["Store embeddings<br/>pgvector"]
        IndexES["Index in Elasticsearch<br/>full-text search"]
        SyncNeo4j["Sync to Neo4j<br/>similarity graph"]
        UpdateVertex["Update Vertex AI<br/>Vector Index"]
    end

    subgraph Analytics["📊 Analytics Pipeline"]
        DatalakeWorker["Datalake Worker"]
        WriteParquet["Write Parquet files<br/>to GCS"]
        DuckDBProcess["Process with DuckDB"]
        AggregateMetrics["Aggregate metrics"]
        SyncAnalytics["Sync to Analytics DB"]
    end

    subgraph Search["🔎 Search & Retrieval"]
        SearchRequest["User search query"]
        GatewaySearch["Gateway API"]
        VectorSearch["Vertex AI Vector Search<br/>ANN similarity"]
        FullTextSearch["Elasticsearch<br/>keyword search"]
        Combine["Combine & rank results"]
        ReturnResults["Return results"]
    end

    %% Upload flow
    User --> GatewayUpload
    GatewayUpload --> StoreGCS
    StoreGCS --> PublishOCR

    %% OCR flow
    PublishOCR --> MistralWorker
    MistralWorker --> MistralAI
    MistralAI --> ClaudeExtract
    ClaudeExtract --> PublishResults

    %% Branching from OCR results
    PublishResults -->|pdf-ocr-results| DocWorker
    PublishResults -->|spec-worker-input| SpecWorker
    PublishResults -->|ocr-completed| GatewayWorker

    %% Embedding flow
    DocWorker --> ExtractImages
    DocWorker --> TextEmbed
    ExtractImages --> ImageEmbed
    ExtractImages --> UploadImages
    TextEmbed --> PublishProcessed
    ImageEmbed --> PublishProcessed

    %% Spec flow
    SpecWorker --> RAGMatch
    RAGMatch --> UCUMConvert
    UCUMConvert --> PublishSpecs
    PublishSpecs --> GatewayWorker

    %% Persistence flow
    PublishProcessed --> GatewayWorker
    GatewayWorker --> InsertPG
    InsertPG --> InsertVector
    GatewayWorker --> IndexES
    GatewayWorker --> SyncNeo4j
    InsertVector --> UpdateVertex

    %% Analytics flow
    GatewayWorker -->|item-processed| DatalakeWorker
    DatalakeWorker --> WriteParquet
    WriteParquet --> DuckDBProcess
    DuckDBProcess --> AggregateMetrics
    AggregateMetrics --> SyncAnalytics

    %% Search flow
    SearchRequest --> GatewaySearch
    GatewaySearch --> VectorSearch
    GatewaySearch --> FullTextSearch
    VectorSearch --> Combine
    FullTextSearch --> Combine
    Combine --> ReturnResults

    classDef upload fill:#e8f5e9,stroke:#2e7d32
    classDef ocr fill:#fff3e0,stroke:#e65100
    classDef embed fill:#e3f2fd,stroke:#1565c0
    classDef spec fill:#f3e5f5,stroke:#7b1fa2
    classDef persist fill:#fce4ec,stroke:#c2185b
    classDef analytics fill:#e0f2f1,stroke:#00695c
    classDef search fill:#fff8e1,stroke:#f9a825

    class User,GatewayUpload,StoreGCS,PublishOCR upload
    class MistralWorker,MistralAI,ClaudeExtract,PublishResults ocr
    class DocWorker,ExtractImages,TextEmbed,ImageEmbed,UploadImages,PublishProcessed embed
    class SpecWorker,RAGMatch,UCUMConvert,PublishSpecs spec
    class GatewayWorker,InsertPG,InsertVector,IndexES,SyncNeo4j,UpdateVertex persist
    class DatalakeWorker,WriteParquet,DuckDBProcess,AggregateMetrics,SyncAnalytics analytics
    class SearchRequest,GatewaySearch,VectorSearch,FullTextSearch,Combine,ReturnResults search

Loading

Pub/Sub Topics

Topic	Publisher	Subscriber	Purpose
mistral-ocr	Gateway	Mistral OCR Worker	Trigger OCR + spec extraction
pdf-ocr-results	Mistral OCR	Document Worker	OCR text for embeddings
spec-worker-input	Mistral OCR	Spec Worker	Extracted specs for matching
ocr-completed	Mistral OCR	Gateway Worker	OCR done, update status
document-processed	Document Worker	Gateway Worker	Full processing complete
spec-matching-results	Spec Worker	Gateway Worker	Matched specifications
item-processed	Gateway	Datalake	Item upserted for analytics
analytics-aggregation	Cloud Scheduler	Datalake	Daily aggregation at 3 AM UTC

Embedding Dimensions

Type	Model	Dimensions	Use Case
Text	text-embedding-005	768	Document content, item descriptions
Image	multimodalembedding@001	1408	Product images, technical drawings

03-network-topology.md

Resourcly Network Topology

GCP infrastructure in europe-west3 (Frankfurt) with private GKE cluster.

flowchart TB
    subgraph Internet["🌐 Internet"]
        Users["Users"]
        ExternalAI["External AI Services<br/>Mistral, Anthropic"]
    end

    subgraph Cloudflare["☁️ Cloudflare"]
        CF_DNS["DNS + CDN<br/>resourcly.com<br/>resourcly.de<br/>resourcly.earth"]
    end

    subgraph GCP["☁️ Google Cloud Platform - europe-west3"]
        
        subgraph PublicEdge["Public Edge"]
            GLB["Cloud Load Balancer<br/>HTTPS"]
        end

        subgraph VPC["VPC: resourcly-vpc (10.0.0.0/20)"]
            
            subgraph GKESubnet["GKE Subnet: 10.0.0.0/20"]
                subgraph GKE["GKE Autopilot (Private Cluster)"]
                    
                    subgraph IngressNS["ingress-nginx"]
                        Ingress["Ingress Controller<br/>TLS Termination"]
                    end

                    subgraph FrontendNS["frontend / admin"]
                        Frontend["Frontend Pods"]
                        Admin["Admin Pods"]
                    end

                    subgraph GatewayNS["gateway"]
                        Gateway["Gateway Pods<br/>:8080"]
                        GatewayWorker["Gateway Workers"]
                    end

                    subgraph InternalAPINS["internal-api"]
                        InternalAPI["Internal API<br/>:8000"]
                    end

                    subgraph DocWorkersNS["document-workers"]
                        MistralOCR["Mistral OCR"]
                        DocWorker["Document Worker"]
                        CADWorker["CAD Worker"]
                        TechDrawing["Tech Drawing"]
                    end

                    subgraph SpecWorkersNS["spec-workers"]
                        SpecWorker["Spec Workers"]
                    end

                    subgraph DatalakeNS["datalake"]
                        Datalake["Datalake Workers"]
                    end

                    subgraph SystemNS["System Namespaces"]
                        ArgoCD["ArgoCD"]
                        CertManager["cert-manager"]
                        ExtSecrets["external-secrets"]
                        KEDA["KEDA"]
                    end
                end
            end

            subgraph PodCIDR["Pod CIDR: 10.4.0.0/14"]
                PodIPs["262,144 Pod IPs"]
            end

            subgraph ServiceCIDR["Service CIDR: 10.8.0.0/20"]
                ServiceIPs["4,096 Service IPs"]
            end

            subgraph DBSubnet["Database Subnet: 10.1.0.0/24"]
                CloudSQL["Cloud SQL PostgreSQL<br/>+ pgvector<br/>Private IP"]
                Redis["Memorystore Redis<br/>Private IP"]
                AnalyticsSQL["Analytics PostgreSQL<br/>Private IP"]
            end

            NAT["Cloud NAT<br/>Egress for Private Cluster"]
        end

        subgraph PSA["Private Service Access"]
            PSAConnect["VPC Peering<br/>Managed Services"]
        end

        subgraph ManagedServices["Managed GCP Services"]
            PubSub["Cloud Pub/Sub"]
            GCS["Cloud Storage<br/>Autoclass Tiering"]
            SecretMgr["Secret Manager"]
            VertexAI["Vertex AI<br/>Vector Search + Embeddings"]
            DocAI["Document AI<br/>OCR (eu location)"]
            IAP["Identity-Aware Proxy"]
        end

        subgraph Observability["Observability"]
            Logging["Cloud Logging"]
            LogRouter["Log Router"]
        end
    end

    subgraph ExternalManaged["External Managed Services"]
        Elastic["Elastic Cloud<br/>Elasticsearch"]
        Neo4j["Neo4j Aura<br/>3 Instances"]
        Firebase["Firebase Auth<br/>Identity Platform"]
        Datadog["Datadog EU<br/>APM + Logs + RUM"]
    end

    %% Internet to GCP
    Users --> CF_DNS
    CF_DNS --> GLB
    GLB --> Ingress

    %% Ingress to Services
    Ingress --> Frontend
    Ingress --> Admin
    Ingress --> Gateway

    %% Gateway connections
    Gateway --> InternalAPI
    Gateway --> CloudSQL
    Gateway --> Redis
    Gateway --> PubSub
    Gateway --> GCS

    %% Internal API to AI
    InternalAPI --> VertexAI

    %% Workers consume from PubSub
    PubSub --> GatewayWorker
    PubSub --> MistralOCR
    PubSub --> DocWorker
    PubSub --> CADWorker
    PubSub --> TechDrawing
    PubSub --> SpecWorker
    PubSub --> Datalake

    %% Workers to storage
    MistralOCR --> GCS
    DocWorker --> GCS
    DocWorker --> VertexAI
    DocWorker --> DocAI
    Datalake --> GCS
    Datalake --> AnalyticsSQL

    %% Gateway Worker persistence
    GatewayWorker --> CloudSQL
    GatewayWorker --> Elastic
    GatewayWorker --> Neo4j

    %% NAT for external egress
    MistralOCR --> NAT
    TechDrawing --> NAT
    NAT --> ExternalAI

    %% External secrets
    ExtSecrets --> SecretMgr

    %% Logging
    Logging --> LogRouter
    LogRouter --> Datadog

    %% Private Service Access
    CloudSQL --> PSAConnect
    Redis --> PSAConnect
    AnalyticsSQL --> PSAConnect
    PSAConnect --> DBSubnet

    %% Firebase Auth
    Gateway -.-> Firebase

    classDef internet fill:#ffebee,stroke:#c62828
    classDef cloudflare fill:#fff3e0,stroke:#e65100
    classDef edge fill:#e8f5e9,stroke:#2e7d32
    classDef gke fill:#e3f2fd,stroke:#1565c0
    classDef db fill:#fce4ec,stroke:#c2185b
    classDef managed fill:#f3e5f5,stroke:#7b1fa2
    classDef external fill:#e0f2f1,stroke:#00695c
    classDef vpc fill:#fff8e1,stroke:#f9a825

    class Users,ExternalAI internet
    class CF_DNS cloudflare
    class GLB,Ingress edge
    class Frontend,Admin,Gateway,GatewayWorker,InternalAPI,MistralOCR,DocWorker,CADWorker,TechDrawing,SpecWorker,Datalake,ArgoCD,CertManager,ExtSecrets,KEDA gke
    class CloudSQL,Redis,AnalyticsSQL db
    class PubSub,GCS,SecretMgr,VertexAI,DocAI,IAP,Logging,LogRouter managed
    class Elastic,Neo4j,Firebase,Datadog external
    class NAT,PSAConnect,PSAConnect,PodIPs,ServiceIPs vpc

Loading

Network CIDR Allocation

Subnet	CIDR	IPs	Purpose
GKE Nodes	10.0.0.0/20	4,096	Node IP addresses
Pod CIDR	10.4.0.0/14	262,144	Pod IP addresses
Service CIDR	10.8.0.0/20	4,096	Kubernetes Service IPs
Database	10.1.0.0/24	256	Cloud SQL, Memorystore

Security Features

Feature	Implementation
Private GKE	No public IPs on nodes
Workload Identity	K8s SA to GCP SA binding, no JSON keys
Private Google Access	Access GCP APIs without public IPs
Cloud NAT	Controlled egress for external AI calls
TLS Everywhere	cert-manager + Let's Encrypt
KMS Encryption	GKE secrets encrypted at rest
Non-root Containers	distroless or UID 1000

Ingress Flow

User Request
    │
    ▼
Cloudflare (DNS + CDN + DDoS)
    │
    ▼
Cloud Load Balancer (HTTPS, managed cert)
    │
    ▼
ingress-nginx (TLS termination, routing)
    │
    ├──▶ portal.resourcly.com  → Frontend
    ├──▶ admin.resourcly.com   → Admin  
    └──▶ api.resourcly.com     → Gateway