Sky Marshal: A full-stack smart surveillance ecosystem built on ESP32-CAM edge nodes, an event-driven Kafka pipeline, computer vision object detection, and cross-platform React/React Native clients. From $5 hardware to real-time annotated video streaming. Built by two Zimbabwean developers. 🇿🇼🛰️

skymarshal-architecture-explained.md

🛰️ Sky Marshal — Integrated Smart Surveillance Ecosystem

"Securing the future through integrated intelligence." — Tanya Mushonga

Sky Marshal is a full-stack, end-to-end smart surveillance platform built around a swarm of low-cost ESP32-CAM edge nodes. The system combines embedded firmware, event-driven microservices, a real-time computer vision pipeline, and cross-platform client applications into a cohesive, production-grade architecture.

This document is the canonical technical reference for the entire ecosystem.

---

📑 Table of Contents

1. Project Overview
2. Ecosystem & Repositories
3. Tech Stack & Rationale
4. System Architecture
5. Data Flow: End-to-End
   • Control Flow (Client → ESP32)
   • Stream Flow (ESP32 → Client)
6. API Reference
7. Data Models
8. Edge Firmware (ESP32-CAM)
9. Core Design Decisions
10. Security Model
11. Deployment Overview
12. Roadmap

---

🌐 Project Overview

Sky Marshal addresses a common gap in surveillance systems: the disconnect between low-cost edge hardware, cloud processing, and real-time operator interfaces. Most solutions are either expensive proprietary systems or fragile DIY setups with no coherent architecture.

Sky Marshal bridges this gap by:

• Using ESP32-CAM modules (~$5 each) as distributed edge nodes that are Wi-Fi enabled and require no special drivers.
• Routing all data through an event-driven Kafka pipeline to decouple ingestion from processing.
• Delivering annotated, real-time video to operators via WebSocket on both a web dashboard and a mobile app.
• Keeping control simple: the edge nodes poll for their configuration, making them resilient to network interruptions.

Key Capabilities:

Capability	Details
Live Video Streaming	JPEG frames streamed from ESP32-CAM via HTTP POST → WebSocket to clients
Object Detection	CV worker runs inference on ingested frames and annotates metadata
Remote Patrol Control	Operators activate/deactivate nodes from the dashboard or mobile app
GPS Tagging	Each frame is tagged with coordinates from the drone node
Multi-Client Broadcast	Both the web dashboard and mobile app receive the same live feed simultaneously

---

🛠️ Ecosystem & Repositories

The project is partitioned into five specialized repositories, each owning a distinct layer of responsibility.

#	Repository	Role	Language
1	skymarshal-api	Central orchestration hub, state management, auth	Node.js / TypeScript
2	skymarshal-mobile	Field monitoring and tactical control (mobile)	React Native / Expo
3	skymarshal-admin-dashboard	Centralized command center and historical analysis (web)	Next.js
4	iatos-camera	High-performance ingest gateway for raw video frames	Node.js / TypeScript
5	sky_marshal_firmware	Low-level firmware for ESP32-CAM edge nodes	C++ / Arduino

Each repository is independently deployable and communicates only through defined interfaces (REST, Kafka topics, WebSocket), ensuring strong separation of concerns.

---

🚀 Tech Stack & Rationale

💻 Frontend (Web & Mobile)

Technology	Usage	Why
Next.js	Admin Dashboard	SSR for fast initial load, API routes for BFF pattern, strong TypeScript support
React Native + Expo	Mobile App	Write once, deploy to iOS and Android; shares component logic with the web layer
NativeWind / Tailwind CSS	Styling (both platforms)	Single utility-first design language across web and mobile eliminates context switching

Strategic rationale: Anchoring all clients on a React-based stack maximises code reuse (hooks, utilities, design tokens) and means a single developer can work across all client surfaces without a context shift.

---

⚙️ Backend & Ingest

Technology	Usage	Why
Node.js + TypeScript	API & Ingest services	Non-blocking I/O is ideal for high-concurrency tasks like frame ingestion and WebSocket broadcasting
Express.js	HTTP API server	Lightweight, well-understood, flexible middleware model
Apache Kafka	Event bus between ingest and processing	Durable message queue that absorbs frame bursts and decouples producers from consumers
WebSockets	Real-time frame delivery to clients	Low-latency, full-duplex channel required for live video; REST polling is too slow
PostgreSQL	Relational data (users, drones, sessions, patrol logs)	ACID-compliant, ideal for structured operational data
MongoDB	Frame metadata & event logs	Schema-flexible document store fits the variability of frame annotation payloads

---

📡 Edge (Hardware)

Component	Details	Why
ESP32-CAM (AI Thinker)	Microcontroller + camera module	Built-in Wi-Fi, OV2640 camera sensor, and deep-sleep support in a single ~$5 unit
C++ / Arduino Framework	Firmware language	Maximum hardware control, rich library ecosystem for the ESP32, real-time performance
HTTP POST	Uplink transport	Stateless and firewall-friendly; nodes can recover from any network interruption without a persistent connection

---

🗺️ System Architecture

The system is divided into three logical zones: Edge, Cloud/Server, and Clients.

┌─────────────────────────────────────────────────────────────────┐
│                            CLIENTS                              │
│                                                                 │
│   ┌──────────────────────────┐   ┌──────────────────────────┐  │
│   │   Next.js Admin Dashboard│   │  React Native Mobile App │  │
│   │   (Web)                  │   │  (iOS / Android)         │  │
│   └────────────┬─────────────┘   └───────────┬──────────────┘  │
│                │ REST (Control)               │ WebSocket (Live) │
└────────────────┼─────────────────────────────┼─────────────────┘
                 │                             │
┌────────────────▼─────────────────────────────▼─────────────────┐
│                       CLOUD / SERVER                            │
│                                                                 │
│   ┌─────────────┐      ┌──────────────┐    ┌────────────────┐  │
│   │ SkyMarshal  │      │ IATOS Ingest │    │  CV Pipeline   │  │
│   │   API       │      │  Gateway     │───▶│  (Object Det.) │  │
│   │ (Port 3000) │      │  (Port 3003) │    └───────┬────────┘  │
│   └──────┬──────┘      └──────┬───────┘            │           │
│          │ DB R/W             │ Kafka Publish        │ Broadcast │
│   ┌──────▼──────────┐  ┌──────▼───────┐            │           │
│   │ PostgreSQL      │  │ Apache Kafka │◀───────────┘           │
│   │ MongoDB         │  │ (Event Bus)  │                         │
│   └─────────────────┘  └─────────────┘                         │
└────────────────────────────────┬────────────────────────────────┘
                                 │ HTTP Poll (Config)
                                 │ HTTP POST (Frames)
┌────────────────────────────────▼────────────────────────────────┐
│                            EDGE                                 │
│                                                                 │
│   ┌──────────────┐  ┌──────────────┐  ┌──────────────┐        │
│   │ ESP32-CAM #1 │  │ ESP32-CAM #2 │  │ ESP32-CAM #N │  ...  │
│   └──────────────┘  └──────────────┘  └──────────────┘        │
└─────────────────────────────────────────────────────────────────┘

---

🔄 Data Flow: End-to-End

🟢 A: Control Flow (Client → ESP32)

How an operator command propagates from the UI to the physical device.

Operator Action
      │
      ▼
[Next.js Dashboard]  ──── HTTPS POST ────▶  [SkyMarshal API]
                                                    │
                                          Validate session token
                                                    │
                                          Update drone config in DB
                                          SET is_active = true
                                          WHERE drone_id = "xyz"
                                                    │
                                                    ▼
                                          [ESP32 Node polls /config]
                                          GET /api/v1/streams/config/
                                          every 10 seconds
                                                    │
                                          ◀── { "is_active": true,
                                                "stream_id": "xyz" }
                                                    │
                                          ESP32 transitions:
                                          IDLE ──▶ STREAMING

Step-by-step:

1. Action — An operator on the Admin Dashboard or Mobile App activates "Active Patrol" for a specific drone node.
2. Request — The client sends an authenticated REST request (POST /api/v1/drones/{drone_id}/activate) to the SkyMarshal API with a valid session token.
3. Validation — The API verifies the token and the operator's permissions for that drone.
4. State Update — The API sets is_active = true for the specified drone_id in the PostgreSQL database.
5. Edge Sync — The ESP32 node, which polls /api/v1/streams/config/{drone_id} every 10 seconds, receives the updated configuration on its next poll.
6. Transition — The firmware parses the JSON response and transitions its internal state machine from IDLE to STREAMING.

---

🔵 B: Stream Flow (ESP32 → Client)

How captured video frames travel from the camera sensor to the operator's screen.

[OV2640 Camera Sensor]
        │
        ▼ JPEG Capture
[ESP32 Firmware]
        │
        ▼ Base64 Encode + JSON Wrap
        │  { drone_id, frame_number, gps, frame_data }
        │
        ▼ HTTP POST
[IATOS Ingest Gateway :3003]
        │
        ▼ Validate + Publish
[Apache Kafka Topic: "raw-frames"]
        │
        ▼ Consume
[CV Pipeline Worker]
   - Decode Base64
   - Run Object Detection (persons, vehicles, etc.)
   - Annotate frame metadata
        │
        ▼ Publish annotated event
[Frame Service]
        │
        ├──── WebSocket ──▶ [Next.js Dashboard]
        └──── WebSocket ──▶ [React Native Mobile App]
                                    │
                              Render on Canvas
                              with real-time overlays

Step-by-step:

1. Capture — The OV2640 camera sensor on the ESP32-CAM captures a JPEG frame.
2. Encoding — The firmware encodes the binary frame as Base64 and wraps it in a JSON payload containing drone_id, frame_number, timestamp, and gps coordinates.
3. Ingestion — The ESP32 POSTs the JSON packet to the IATOS Ingest Gateway (Port 3003) via HTTP.
4. Queuing — IATOS validates the payload, strips sensitive routing headers, and publishes the message to the raw-frames Kafka topic.
5. Annotation (Async) — A CV worker consumes the Kafka message, decodes the Base64 frame, runs object detection inference (e.g., detecting persons and vehicles), and publishes an annotated event to a downstream topic.
6. Distribution — The Frame Broadcast Service consumes the annotated events and pushes them over WebSocket to all subscribed clients.
7. Render — The Admin Dashboard and Mobile App receive each frame event and render it on an HTML/Native Canvas element, overlaying bounding boxes, drone ID, GPS, and timestamp.

---

📡 API Reference

All API routes are served from the skymarshal-api service. Base URL: https://<host>/api/v1

Authentication uses Bearer token (JWT) in the Authorization header.

Drone Management

Method	Endpoint	Description
GET	/drones	List all registered drone nodes
GET	/drones/{drone_id}	Get status and metadata for a specific drone
POST	/drones/{drone_id}/activate	Activate a drone's patrol mode
POST	/drones/{drone_id}/deactivate	Deactivate a drone (transitions to IDLE)

Stream Configuration (Polled by ESP32)

Method	Endpoint	Description
GET	/streams/config/{drone_id}	Returns current config for the drone (is_active, stream_id, etc.)

Example Response:

{
  "drone_id": "drone-003",
  "is_active": true,
  "stream_id": "stream-abc-xyz",
  "poll_interval_ms": 10000,
  "resolution": "SVGA"
}

Frame Ingest (IATOS — Port 3003)

Method	Endpoint	Description
POST	/ingest/frame	Accepts a frame payload from an ESP32 node

Frame Payload Schema:

{
  "drone_id": "drone-003",
  "stream_id": "stream-abc-xyz",
  "frame_number": 1042,
  "timestamp": "2025-04-10T14:32:00Z",
  "gps": {
    "lat": -17.8292,
    "lng": 31.0522,
    "alt_m": 45.2
  },
  "frame_data": "<Base64-encoded JPEG string>"
}

Authentication

Method	Endpoint	Description
POST	/auth/login	Returns a JWT for an operator
POST	/auth/refresh	Refreshes an expiring JWT
POST	/auth/logout	Invalidates the current session

---

🗂️ Data Models

Drone (PostgreSQL)

CREATE TABLE drones (
  id           UUID PRIMARY KEY DEFAULT gen_random_uuid(),
  name         VARCHAR(100) NOT NULL,
  is_active    BOOLEAN DEFAULT FALSE,
  stream_id    VARCHAR(100),
  last_seen_at TIMESTAMP,
  created_at   TIMESTAMP DEFAULT NOW()
);

FrameEvent (MongoDB)

{
  "_id": "ObjectId",
  "drone_id": "drone-003",
  "stream_id": "stream-abc-xyz",
  "frame_number": 1042,
  "timestamp": "2025-04-10T14:32:00.000Z",
  "gps": { "lat": -17.8292, "lng": 31.0522, "alt_m": 45.2 },
  "detections": [
    { "label": "person", "confidence": 0.94, "bbox": [120, 80, 200, 340] },
    { "label": "vehicle", "confidence": 0.87, "bbox": [400, 150, 580, 290] }
  ],
  "raw_stored": false
}

---

🔧 Edge Firmware (ESP32-CAM)

The firmware (sky_marshal_firmware) operates as a deterministic state machine.

State Machine

         ┌──────────┐
  Boot ──▶  INIT    │
         └────┬─────┘
              │ Wi-Fi connected
         ┌────▼─────┐
         │  POLLING │◀──────────────────┐
         └────┬─────┘                   │
              │ is_active == true        │
         ┌────▼─────┐   is_active=false  │
         │ STREAMING │──────────────────┘
         └──────────┘

Key Firmware Parameters

Parameter	Default	Description
POLL_INTERVAL_MS	10000	How often the node polls the API for its config
FRAME_QUALITY	12	JPEG quality (0–63; lower = higher quality)
FRAME_SIZE	FRAMESIZE_SVGA	Resolution (SVGA = 800×600)
POST_ENDPOINT	http://<host>:3003/ingest/frame	IATOS ingest URL
CONFIG_ENDPOINT	http://<host>:3000/api/v1/streams/config/{id}	API config poll URL

Frame Transmission Loop (Pseudo-code)

void streamingLoop() {
  camera_fb_t* frame = esp_camera_fb_get();          // Capture frame
  String b64 = base64::encode(frame->buf, frame->len); // Encode

  StaticJsonDocument<512> payload;
  payload["drone_id"]    = DRONE_ID;
  payload["frame_number"] = frameCount++;
  payload["gps"]         = buildGpsPayload();
  payload["frame_data"]  = b64;

  http.begin(POST_ENDPOINT);
  http.addHeader("Content-Type", "application/json");
  http.POST(payload.as<String>());                   // Send to IATOS
  http.end();
  esp_camera_fb_return(frame);                       // Free buffer
}

---

🧠 Core Design Decisions

1. Decoupled Ingest (IATOS vs. API)

Separating the high-bandwidth frame ingestion layer (IATOS, Port 3003) from the business logic API (Port 3000) ensures that a burst of frames from multiple nodes never degrades the responsiveness of the admin interface. Each service can be independently scaled horizontally.

2. Kafka as the Event Backbone

Without Kafka, a burst of frames hitting the CV worker directly would cause bottlenecks or dropped frames. Kafka acts as a durable buffer — IATOS writes fast, and the CV workers consume at their own pace. This also makes it trivial to add future consumers (e.g., an archival service or alert engine) without touching existing code.

3. Base64 over Raw Binary

While Base64 adds ~33% overhead to payload size, it was the correct first-version choice: it eliminates multipart form handling on both the ESP32 and the Node.js ingest server, works natively with JSON, and is trivially debuggable (any HTTP inspector can read it). A future version can move to binary streaming (e.g., multipart/octet-stream) for bandwidth efficiency.

4. Poll-Sync Model for Control

Rather than maintaining a persistent MQTT or WebSocket connection from the ESP32, nodes poll the API for their configuration every 10 seconds. This makes nodes resilient to network drops (they simply re-poll on reconnect), requires no broker sidecar on the edge, and works behind NAT/firewalls without port forwarding.

5. PostgreSQL + MongoDB Hybrid Storage

Structured, relational data (users, drone registry, patrol sessions, audit logs) lives in PostgreSQL for ACID guarantees. Semi-structured, high-volume frame event data lives in MongoDB, where its schema flexibility accommodates evolving detection payload shapes without costly migrations.

---

🔐 Security Model

Layer	Mechanism
Client → API	JWT Bearer token, HTTPS
API → DB	Internal network only, credentials via environment variables
ESP32 → IATOS	drone_id + stream_id validated on every ingest request
IATOS → Kafka	Internal broker, no public exposure
WebSocket	Token validated on connection upgrade

Note: For production deployments, ESP32 nodes should use TLS-enabled endpoints and a pre-shared device key for additional ingest authentication.

---

🚢 Deployment Overview

The server-side services are containerised and can be orchestrated with Docker Compose for development or Kubernetes for production.

# docker-compose.yml (development)
services:
  api:
    build: ./skymarshal-api
    ports: ["3000:3000"]
    environment:
      - DATABASE_URL=postgres://...
      - KAFKA_BROKER=kafka:9092

  iatos:
    build: ./iatos-camera
    ports: ["3003:3003"]
    environment:
      - KAFKA_BROKER=kafka:9092

  kafka:
    image: confluentinc/cp-kafka:7.6.0
    ports: ["9092:9092"]

  postgres:
    image: postgres:16
    volumes: ["pgdata:/var/lib/postgresql/data"]

  mongo:
    image: mongo:7
    volumes: ["mongodata:/data/db"]

ESP32 Firmware Flash

# Using PlatformIO
cd sky_marshal_firmware
pio run --target upload --upload-port /dev/ttyUSB0

---

🗺️ Roadmap

Feature	Status
Core ingest pipeline (ESP32 → IATOS → Kafka)	✅ Complete
Admin Dashboard (Next.js)	✅ Complete
Mobile App (React Native)	✅ Complete
Object Detection CV Worker	✅ Complete
Binary frame streaming (replace Base64)	🔄 Planned
MQTT control channel (replace HTTP poll)	🔄 Planned
Multi-operator role-based access control	🔄 Planned
Persistent video archival service	🔄 Planned
Alert engine (motion triggers, geofence breach)	🔄 Planned
OTA firmware update mechanism	🔄 Planned

---

👥 Contributors

	Name	Role	GitHub	Location
🛰️	Tanya Mushonga	Lead Architect & Full-Stack Engineer — system design, API, firmware, mobile & web	github.com/TanyaMushonga	Harare, Zimbabwe 🇿🇼
⚡	Malwande Moyo	Software Developer & Electronics Engineer — embedded systems, hardware integration	github.com/malwandemoyo	Bulawayo, Zimbabwe 🇿🇼

---

Sky Marshal is an independently designed and built surveillance ecosystem. All architecture, firmware, and application code is original work.