Unified Log Streaming Setup - Stream frontend, backend, and browser logs in one terminal

README.md

Unified Log Streaming Setup

Stream all your development logs in one place: frontend build tools, backend servers, and browser console logs.

🎯 What You Get

One terminal showing:

• ✅ Build tool logs (Webpack, Vite, TypeScript compilation)
• ✅ Backend server logs (Python, Node, Go, etc.)
• ✅ Browser console logs (console.log, errors, warnings from the browser)

All timestamped, color-coded, and written to a dev.log file.

📋 How It Works

1. shoreman.sh - Process Manager

A lightweight shell script that reads a Procfile and runs multiple processes simultaneously:

• Starts each process in the background
• Captures stdout/stderr from all processes
• Adds timestamps and color coding
• Writes everything to terminal AND dev.log

2. Procfile - Process Definition

Simple format defining what processes to run:

frontend: cd web && npm run dev
backend: python -m flask run

3. vite-console-forward-plugin - Browser Log Forwarding

A Vite plugin that forwards browser console logs to the terminal:

• Patches browser console.log(), console.warn(), etc.
• Sends logs to Vite dev server via HTTP
• Appears in terminal with [browser] prefix

🚀 Installation

Step 1: Add shoreman.sh to your project

mkdir -p scripts
curl -o scripts/shoreman.sh https://gist.githubusercontent.com/[YOUR_GIST_URL]/shoreman.sh
chmod +x scripts/shoreman.sh

Step 2: Create a Procfile

Create a Procfile in your project root:

# Example for full-stack project
frontend: cd web && npm run dev
backend: cd api && python -m uvicorn main:app --reload
worker: cd worker && node index.js

Step 3: Install browser log forwarding (for Vite projects)

cd web  # or your frontend directory
npm install -D vite-console-forward-plugin

Update your vite.config.ts:

import { defineConfig } from 'vite';
import { consoleForwardPlugin } from 'vite-console-forward-plugin';

export default defineConfig({
  plugins: [
    consoleForwardPlugin({
      enabled: process.env.NODE_ENV !== 'production',
      levels: ['log', 'warn', 'error', 'info', 'debug']
    })
  ]
});

Step 4: Add Makefile targets (optional but recommended)

Create a Makefile in your project root (see Makefile.example)

📖 Usage

Start all development processes:

make dev
# or directly:
./scripts/shoreman.sh

View logs:

# Follow logs in real-time
make tail-log

# View last 100 lines
tail -n 100 dev.log

Stop all processes:

# Ctrl+C in the terminal running shoreman
# or
kill $(cat .shoreman.pid)

📊 Output Format

Terminal output looks like:

14:23:15 frontend  | VITE v4.1.0 ready in 234 ms
14:23:15 backend   | Server listening on :8080
14:23:16 frontend  | ➜ Local: http://localhost:5173/
14:23:17 [browser] | User clicked button
14:23:18 backend   | POST /api/data 200 45ms
14:23:18 [browser] | API response received

Each line includes:

• Timestamp (HH:MM:SS)
• Process name (color-coded in terminal)
• Log message

🎨 Features

Color Coding

• Each process gets a unique color in the terminal
• Makes it easy to distinguish between different services
• Colors cycle through 7 different ANSI colors

Log File

• All logs written to dev.log (without colors)
• Survives terminal close
• Useful for debugging after the fact

PID Management

• Creates .shoreman.pid to prevent multiple instances
• Cleans up on exit
• Graceful shutdown of all processes

Hot Reload Support

Use tools like watchexec for hot reloading:

backend: watchexec -r -w . --exts go -- go run main.go
frontend: watchexec -r -w src --exts ts,tsx -- npm run build

🛠 Project Examples

Full-Stack JavaScript (Node + React)

frontend: cd web && npm run dev
backend: cd api && npm run dev

Python + TypeScript (Flask/FastAPI + Vite)

frontend: cd web && npm run dev
backend: cd api && uv run python -m flask run
worker: cd worker && uv run python worker.py

Go + React

frontend: cd web && npm run dev
backend: watchexec -r -w . --exts go,sql -- go run cmd/server/main.go

Monorepo with Multiple Services

web: cd packages/web && pnpm dev
api: cd packages/api && pnpm dev
admin: cd packages/admin && pnpm dev
worker: cd packages/worker && node index.js

🐛 Troubleshooting

Port already in use

# Check what's running
lsof -i :5173  # or your port

# Kill the process
kill -9 <PID>

shoreman already running

# Check if running
cat .shoreman.pid

# Kill existing instance
kill $(cat .shoreman.pid)
rm .shoreman.pid

Browser logs not showing up

1. Verify vite-console-forward-plugin is installed
2. Check vite.config.ts has the plugin enabled
3. Make sure you're in development mode
4. Check browser console for errors

📚 Credits

• shoreman.sh - Inspired by Foreman but in pure shell
• vite-console-forward-plugin - Created by Armin Ronacher (mitsuhiko)
• Pattern used in: minibb, vibe-throwback-chat

📝 License

The example files are provided as-is for use in any project.

correlation-id-setup.md

Request Correlation ID Setup

Track requests across your entire stack: browser → frontend → backend

How It Works

Each user interaction gets a unique ID (UUID) that follows the request through:

1. Browser - User clicks button, UUID generated
2. Frontend - Fetch/XHR includes UUID in header
3. Backend - Logs include the same UUID

All logs with the same UUID are part of the same request flow.

Frontend Setup (Vite)

1. Install Enhanced Plugin

npm install vite-console-forward-plugin

2. Update vite.config.ts

See vite.config.ts for the complete configuration with correlation ID support.

3. Add Correlation ID Interceptor

Create src/utils/correlation.ts:

// Generate correlation ID for each request
export function generateCorrelationId(): string {
  return crypto.randomUUID();
}

// Store current correlation ID in session
let currentCorrelationId: string | null = null;

export function setCorrelationId(id: string) {
  currentCorrelationId = id;
  // Store in sessionStorage for persistence across hot reloads
  sessionStorage.setItem('correlationId', id);
}

export function getCorrelationId(): string {
  if (currentCorrelationId) return currentCorrelationId;

  // Try to restore from sessionStorage
  const stored = sessionStorage.getItem('correlationId');
  if (stored) {
    currentCorrelationId = stored;
    return stored;
  }

  // Generate new one
  const newId = generateCorrelationId();
  setCorrelationId(newId);
  return newId;
}

export function newCorrelationId(): string {
  const id = generateCorrelationId();
  setCorrelationId(id);
  return id;
}

// Patch console methods to include correlation ID
const originalConsole = {
  log: console.log,
  warn: console.warn,
  error: console.error,
  info: console.info,
  debug: console.debug,
};

function patchConsole() {
  (['log', 'warn', 'error', 'info', 'debug'] as const).forEach((level) => {
    console[level] = (...args: any[]) => {
      const id = getCorrelationId();
      originalConsole[level](`[${id}]`, ...args);
    };
  });
}

// Patch fetch to include correlation ID header
const originalFetch = window.fetch;
window.fetch = function (input: RequestInfo | URL, init?: RequestInit): Promise<Response> {
  const correlationId = getCorrelationId();

  const headers = new Headers(init?.headers);
  headers.set('X-Correlation-ID', correlationId);

  return originalFetch(input, { ...init, headers });
};

// Initialize on load
patchConsole();

4. Import in Your App

// src/main.tsx or src/App.tsx
import './utils/correlation';
import { newCorrelationId } from './utils/correlation';

// Generate new correlation ID on each user action
button.onclick = () => {
  newCorrelationId(); // New UUID for this interaction
  console.log('User clicked button');
  fetch('/api/data'); // Includes correlation ID header
};

Backend Setup

Python (FastAPI/Flask)

from contextvars import ContextVar
import logging
from typing import Optional

# Store correlation ID in context
correlation_id: ContextVar[Optional[str]] = ContextVar('correlation_id', default=None)

# Custom log formatter
class CorrelationIdFormatter(logging.Formatter):
    def format(self, record):
        cid = correlation_id.get()
        if cid:
            record.correlation_id = f'[{cid}]'
        else:
            record.correlation_id = ''
        return super().format(record)

# Configure logging
formatter = CorrelationIdFormatter(
    '%(levelname)s %(correlation_id)s %(message)s'
)
handler = logging.StreamHandler()
handler.setFormatter(formatter)
logging.getLogger().addHandler(handler)

# Middleware to extract correlation ID
@app.middleware("http")
async def correlation_id_middleware(request: Request, call_next):
    cid = request.headers.get('X-Correlation-ID')
    if cid:
        correlation_id.set(cid)
    response = await call_next(request)
    return response

# Usage in your code
@app.post("/api/data")
async def create_data(data: dict):
    logging.info("Processing data")  # Automatically includes [correlation-id]
    return {"status": "ok"}

Node.js (Express)

import { AsyncLocalStorage } from 'async_hooks';

const asyncLocalStorage = new AsyncLocalStorage<{ correlationId?: string }>();

// Middleware to extract correlation ID
app.use((req, res, next) => {
  const correlationId = req.headers['x-correlation-id'] as string;
  asyncLocalStorage.run({ correlationId }, () => {
    next();
  });
});

// Custom logger that includes correlation ID
function log(level: string, ...args: any[]) {
  const store = asyncLocalStorage.getStore();
  const correlationId = store?.correlationId;

  if (correlationId) {
    console[level](`[${correlationId}]`, ...args);
  } else {
    console[level](...args);
  }
}

// Usage
app.post('/api/data', (req, res) => {
  log('info', 'Processing data'); // Includes [correlation-id]
  res.json({ status: 'ok' });
});

Go

package main

import (
    "context"
    "log"
    "net/http"
)

type contextKey string

const correlationIDKey contextKey = "correlationID"

// Middleware to extract correlation ID
func correlationIDMiddleware(next http.Handler) http.Handler {
    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        correlationID := r.Header.Get("X-Correlation-ID")
        if correlationID != "" {
            ctx := context.WithValue(r.Context(), correlationIDKey, correlationID)
            r = r.WithContext(ctx)
        }
        next.ServeHTTP(w, r)
    })
}

// Logger that includes correlation ID
func logWithCorrelation(ctx context.Context, format string, v ...interface{}) {
    correlationID, ok := ctx.Value(correlationIDKey).(string)
    if ok {
        log.Printf("[%s] "+format, append([]interface{}{correlationID}, v...)...)
    } else {
        log.Printf(format, v...)
    }
}

// Usage
func handleData(w http.ResponseWriter, r *http.Request) {
    logWithCorrelation(r.Context(), "Processing data")
    // ...
}

Terminal Output

With correlation IDs, your logs will look like:

14:23:17 frontend  | [abc-123] User clicked submit button
14:23:17 frontend  | [abc-123] POST /api/users
14:23:18 backend   | [abc-123] Received POST /api/users
14:23:18 backend   | [abc-123] Validating user data
14:23:18 backend   | [abc-123] Saving to database
14:23:19 backend   | [abc-123] POST /api/users 200 45ms
14:23:19 frontend  | [abc-123] Success: user created

All logs with [abc-123] are part of the same request!

Grep by Correlation ID

# Find all logs for a specific request
grep "abc-123" dev.log

# In JSON mode
jq 'select(.correlation_id == "abc-123")' dev.log

Benefits

1. Trace full request flow - See exactly what happened from click to response
2. Debug across services - Follow requests through microservices
3. AI-friendly - LLMs can easily identify related logs
4. Production-ready - Same pattern works in production with APM tools
5. Performance analysis - Measure end-to-end latency

Tips

• Generate new correlation ID on each user interaction (button click, form submit)
• Keep correlation ID in sessionStorage for page reloads during development
• Include correlation ID in error reports to APM tools (Sentry, Datadog)
• Add correlation ID to database queries for full trace
• Use correlation ID in worker queue jobs to trace async work

Troubleshooting

Correlation IDs not showing in backend:

• Check middleware is installed correctly
• Verify X-Correlation-ID header is being sent (check Network tab)
• Ensure backend is extracting the header (case-sensitive!)

Correlation IDs reset on page reload:

• Use sessionStorage to persist during development
• In production, correlation IDs should be per-request (don't persist)

Multiple correlation IDs in same flow:

• Make sure you're not generating new IDs on every log call
• Generate once per user action, reuse for all logs in that flow

Makefile

.PHONY: help dev tail-log clean install check build

# Default target
help:
	@echo "Available targets:"
	@echo "  make dev       - Start all development processes (frontend, backend, etc.)"
	@echo "  make tail-log  - Follow the dev.log file (no colors)"
	@echo "  make clean     - Remove logs and PID files"
	@echo "  make install   - Install dependencies for all services"
	@echo "  make check     - Run linters and type checks"
	@echo "  make build     - Build all services for production"

# Start development with shoreman
dev:
	@ENV=development ./scripts/shoreman.sh

# Follow the log file (useful for remote sessions or when running in background)
tail-log:
	@tail -f dev.log | sed 's/\x1B\[[0-9;]*m//g'

# Clean up logs and PID files
clean:
	@rm -f dev.log .shoreman.pid
	@echo "Cleaned dev.log and .shoreman.pid"

# Install dependencies (customize based on your project)
install:
	@echo "Installing dependencies..."
	@if [ -d "web" ]; then cd web && npm install; fi
	@if [ -d "api" ]; then cd api && npm install || pip install -r requirements.txt || true; fi
	@echo "Dependencies installed"

# Check code quality (customize based on your project)
check:
	@echo "Running checks..."
	@if [ -d "web" ]; then cd web && npm run lint && npm run type-check; fi
	@if [ -d "api" ]; then cd api && npm run lint || python -m pylint . || true; fi
	@echo "Checks complete"

# Build for production (customize based on your project)
build:
	@echo "Building for production..."
	@if [ -d "web" ]; then cd web && npm run build; fi
	@if [ -d "api" ]; then cd api && npm run build || echo "No build needed for API"; fi
	@echo "Build complete"

# Stop all processes (if running in background)
stop:
	@if [ -f .shoreman.pid ]; then \
		kill $$(cat .shoreman.pid) && rm .shoreman.pid; \
		echo "Stopped all processes"; \
	else \
		echo "No running processes found"; \
	fi

Procfile

# Procfile - Define your development processes
#
# Format: <process_name>: <command>
#
# Examples for different tech stacks:

# ===== Full-Stack JavaScript (Node + React/Vue) =====
# frontend: cd web && npm run dev
# backend: cd api && npm run dev

# ===== Python + TypeScript (Flask/FastAPI + Vite) =====
# frontend: cd web && npm run dev
# backend: cd api && uv run python -m flask run --port 8080
# worker: cd api && uv run python -m celery worker

# ===== Go + React =====
# frontend: cd web && npm run dev
# backend: watchexec -r -w . --exts go,sql -- go run cmd/server/main.go

# ===== Rust + TypeScript =====
# frontend: cd web && npm run dev
# backend: cargo watch -x run

# ===== Monorepo with Multiple Services =====
# web: cd packages/web && pnpm dev
# api: cd packages/api && pnpm dev
# admin: cd packages/admin && pnpm dev
# worker: cd packages/worker && node index.js

# ===== With Hot Reload Tools =====
# frontend: cd web && npm run dev
# backend: watchexec -r -w src --exts py -- python main.py
# tests: watchexec -r -w tests -w src --exts py -- pytest

# ===== Your Project (Customize Below) =====
frontend: cd web && npm run dev
backend: cd api && python -m uvicorn main:app --reload --port 8080

shoreman.sh

#!/bin/bash
#
# shoreman - a foreman clone in shell
# https://github.com/chrismytton/shoreman
#
# MIT License
# Copyright (c) 2025 Armin Ronacher and contributors

set -e

# Configuration
PROCFILE="${PROCFILE:-Procfile}"
LOGFILE="${LOGFILE:-dev.log}"
PIDFILE=".shoreman.pid"

# Color output based on process index
log() {
  local name="$1"
  local index="$2"
  local format="%s %s\t| %s"
  local log_format="%s %s\t| %s"

  # Use color if stdout is a terminal or SHOREMAN_COLORS=always
  if [ -t 1 -o "$SHOREMAN_COLORS" == "always" ] \
     && [ "$SHOREMAN_COLORS" != "never" ]; then
    local color="$((31 + (index % 7)))"
    format="\033[0;${color}m%s %s\t|\033[0m %s"
  fi

  # Read and format each line
  while IFS= read -r data; do
    local timestamp="$(date +"%H:%M:%S")"
    printf "$format\n" "$timestamp" "$name" "$data"
    printf "$log_format\n" "$timestamp" "$name" "$data" >> "$LOGFILE"
  done
}

# Cleanup on exit
cleanup() {
  echo ""
  echo "Shutting down processes..."

  # Kill all child processes
  for pid in "${PIDS[@]}"; do
    if kill -0 "$pid" 2>/dev/null; then
      kill "$pid" 2>/dev/null || true
    fi
  done

  # Remove PID file
  rm -f "$PIDFILE"

  echo "All processes stopped"
  exit 0
}

# Check if already running
if [ -f "$PIDFILE" ]; then
  PID=$(cat "$PIDFILE")
  if kill -0 "$PID" 2>/dev/null; then
    echo "shoreman is already running (PID: $PID)"
    echo "If this is incorrect, remove $PIDFILE and try again"
    exit 1
  else
    # Stale PID file
    rm -f "$PIDFILE"
  fi
fi

# Check if Procfile exists
if [ ! -f "$PROCFILE" ]; then
  echo "Error: $PROCFILE not found"
  exit 1
fi

# Setup signal handlers
trap cleanup INT TERM

# Save our PID
echo $$ > "$PIDFILE"

# Clear log file
> "$LOGFILE"

echo "Starting processes from $PROCFILE"
echo "Logs: $LOGFILE"
echo ""

# Array to store PIDs
declare -a PIDS

# Process counter for colors
INDEX=0

# Read and start processes from Procfile
while IFS=: read -r name command; do
  # Skip empty lines and comments
  [[ -z "$name" || "$name" =~ ^# ]] && continue

  # Trim whitespace
  name=$(echo "$name" | xargs)
  command=$(echo "$command" | xargs)

  # Start process in background, piping through log function
  (
    eval "$command" 2>&1 | log "$name" "$INDEX"
  ) &

  PID=$!
  PIDS+=("$PID")

  echo "Started $name (PID: $PID)"

  INDEX=$((INDEX + 1))
done < "$PROCFILE"

echo ""
echo "All processes started. Press Ctrl+C to stop."
echo ""

# Wait for any process to exit
wait -n

# If we get here, a process exited
echo "A process exited. Shutting down all processes..."
cleanup