TMC: FINAL Institutional Master Plan & Comparative Audit

ComparativeAudit.md

Comparative Audit: TMC vs. UBDCC

This document provides a comprehensive analysis and audit of the Tardis Management Cluster (TMC) design compared to the existing Unicorn Binance Depth Cache Cluster (UBDCC) infrastructure.

1. Architectural Maturity & Philosophy

Attribute	UBDCC (Current)	TMC (Target Design)	Audit Note
Philosophy	"Deep & Narrow" (Binance Focused)	"Broad & Unified" (Multi-Exchange)	TMC inherits UBDCC's distributed maturity while expanding scope.
Design Pattern	Master-Worker-Proxy (Python/Cython)	Master-Worker-Proxy (Node.js/TS)	TMC adopts the same proven pattern but aligns with the Tardis ecosystem.
Control Plane	Python mgmt node	Node.js master node	Both use in-memory thread-safe state stores with replication.
Data Plane	unicorn-binance-local-depth-cache	tardis-machine	TMC leverages a more feature-rich, multi-exchange engine.

---

2. Data Integrity & Reconstruction

2.1 UBDCC (Institutional Standard for Binance)

• Strengths: Implements strict sequence ID validation (u == prev_U + 1). Any gap triggers an immediate hard-reset of the cache.
• Audit: Highly reliable for Binance. Its native integration with the Binance protocol ensures zero drift.

2.2 TMC (The Institutional Evolution)

• Strengths: Normalized schema across 30+ exchanges. Built-in compression and caching for historical replay.
• Gaps (Fixed in TMC Design): Standard Tardis setups lack active drift detection. The TMC design "fills" this by adding the Consistency Validation Layer (Periodic REST snapshot audit), bringing it up to UBDCC's institutional standard.
• Audit: By porting UBDCC's "Sequence Gap" and "Drift Detection" logic into the Tardis engine, TMC achieves parity in data integrity while supporting a much broader market.

---

3. Scalability & Performance

3.1 Scaling Strategy

• UBDCC: Scales by spawning multiple DCN pods (one per core). Orchestration is K8s-native but Python-bound (GIL).
• TMC: Scales by spawning multiple Worker pods managing tardis-machine. Node.js's non-blocking I/O is ideally suited for high-throughput WS proxying.
• Audit: TMC's "Resource-Weighted" distribution (Event Loop Lag metrics) is a more modern approach to rebalancing than UBDCC's "Unused Longest" strategy.

3.2 Throughput

• UBDCC: Cython-compiled modules provide high performance for Binance data.
• TMC: tardis-machine is a highly optimized C++/JS engine capable of 100k+ msg/s.
• Audit: TMC is expected to outperform UBDCC in multi-exchange scenarios due to the underlying engine's optimizations for high-density historical and real-time streams.

---

4. Fault Tolerance & Self-Healing

4.1 State Resilience

• UBDCC: Database is replicated to all nodes. On Master failure, any node can provide the state.
• TMC: Adopts the Circular Backup protocol from UBDCC. The ClusterDB is replicated to all TMC-Worker nodes.
• Audit: Parity achieved. TMC's "Master Recovery" logic is a direct port of UBDCC's successful self-healing mechanism.

4.2 Failover Path

• UBDCC: Proxy retries different DCNs for the same market.
• TMC: Gateway implements Hot-Standby Redundancy (First-to-Arrive signals).
• Audit: TMC's failover logic is more advanced for trading, focusing on signal latency (picking the fastest stream) rather than just availability.

---

5. Use Case Suitability

Use Case	UBDCC	TMC
Live Binance Trading	⭐⭐⭐⭐⭐	⭐⭐⭐⭐⭐
Multi-Exchange Research	❌	⭐⭐⭐⭐⭐
Historical Replay	❌	⭐⭐⭐⭐⭐
Ultra-Low Latency Streams	⭐⭐⭐⭐	⭐⭐⭐⭐⭐
Institutional Resilience	⭐⭐⭐⭐⭐	⭐⭐⭐⭐⭐

---

6. Final Audit Conclusion

The TMC design is a successful institutional evolution of the UBDCC project. It preserves the distributed orchestration and self-healing reliability of UBDCC while leveraging the Tardis ecosystem to provide:

1. Multi-exchange support (30+ vs 1).
2. Unified Historical/Real-time access (Full parity for backtesting).
3. Modernized Scaling (Event Loop metrics vs process-based).
4. Enhanced Signal Integrity (Active auditing vs passive consumption).

Recommendation: Proceed with TMC implementation as it represents a significant upgrade in both capability and performance for institutional quant operations.

MasterPlan.md

TMC: Institutional Architecture & Implementation Master Plan

1. Executive Summary & Rationale

The Tardis Management Cluster (TMC) is a globally scalable, institutional-grade market data infrastructure. It evolves tardis-machine and tardis-node by applying distributed systems patterns—proven in the UBDCC project—to overcome the inherent limitations of standard exchange APIs.

1.1 The Institutional Problem

• Rate Limit Constraints: Standard REST APIs are throttled by weight-based limits, making large-scale historical collection impossible without massive IP rotation.
• Archive Fragmentation: Inconsistent archive formats across 30+ exchanges hinder unified research.
• State Reconstruction Gap: Archives typically lack the initial snapshots required for order book bootstrapping.

1.2 The TMC Solution

1. Distributed Orchestration: A Master-Worker cluster that balances thousands of symbols across optimized Node.js processes.
2. High-Fidelity "Self-Healing" Reconstruction: Porting UBDCC's strict sequence-validation and checksum-driven drift detection into the Tardis engine.
3. Transparent Local Caching: Fetching once from the cloud and serving infinitely from local NVMe storage at wire speeds.

---

2. Requirements

2.1 Functional

• Multi-Exchange Ingestion: Support for all 30+ Tardis-integrated exchanges.
• Real-time & Historical Parity: Unified API for both live trading streams and backtesting.
• Active Drift Detection: Real-time audit of local cache against REST snapshots and native checksums.
• Self-Healing Master: Distributed state recovery via worker-replicated backups.

2.2 Performance (High-Frequency Ready)

• Sub-millisecond Internal Latency: Optimized zero-copy data paths.
• Extreme Throughput: Support for 100k+ messages per second per worker.
• Horizontal Scalability: Seamlessly add worker nodes as symbol counts grow.

---

3. High-Performance Technology Stack

TMC utilizes a Node.js Native, "Expert-Level" Stack to minimize the "bridge tax" and maximize reliability:

Layer	Technology	Institutional Rationale
Networking	uWebSockets.js	C++ core; 5-10x throughput of Fastify/Express; minimal memory overhead.
Concurrency	SharedArrayBuffer	Enables Zero-Copy state sharing between worker threads and the main API thread.
Memory	TypedArrays (Float64)	Prevents GC pressure by storing book levels in contiguous memory outside the JS heap.
Serialization	MessagePack	Faster and more compact than JSON for cluster-internal communication.
Logging	Pino	Extremely low-overhead logging for high-frequency event auditing.
Observability	prom-client	Real-time monitoring of Event Loop Lag and Worker Load.

---

4. System Architecture: Control vs. Data Plane

4.1 TMC-Master (The Control Plane)

• Role: Cluster brain; manages node registry, symbol distribution, and state replication.
• Stack: Fastify + AJV + ClusterDB (In-memory with v8 serialization for fast backups).

4.2 TMC-Worker (The Management Wrapper)

• Role: Spawns and configures a local tardis-machine instance.
• Performance Logic: Uses a dedicated Worker Thread pool for normalization. Communicates with the main thread via SharedArrayBuffer for zero-latency book access.

4.3 TMC-Gateway (The Data Plane Proxy)

• Role: High-performance router that proxies client requests to the correct worker node.
• Logic: Uses a zero-copy WebSocket proxy-chain in uWebSockets.js.

---

5. Detailed Data & Code Flows

5.1 Orchestration Flow

1. Registration: Worker reports metrics (Event Loop Lag, Memory) to Master.
2. Assignment: Master assigns symbols using a Resource-Weighted Round Robin strategy.
3. Bootstrap: Worker configures tardis-machine and starts the reconstruction thread.

5.2 The Consistency Audit (Self-Healing)

• Layer 1 (Gap Detection): Continuous sequence-ID monitoring (seq == last_seq + 1).
• Layer 2 (Checksum): Real-time validation using exchange-native checksums (e.g., ADLER32).
• Layer 3 (Drift Audit): Periodic (5-15 min) validation against fresh REST snapshots.
• Recovery: Automatic "Purge & Resync" on any failure.

5.3 Live Trading Path

• First-to-Arrive Redundancy: Gateway can subscribe to multiple redundant workers for a symbol and forward the fastest signal to the bot.

---

6. Implementation Roadmap

Phase 1: Cluster Skeleton

• Implement TMC-Shared (replicated state DB with SharedArrayBuffer hooks).
• Build Master-Worker heartbeat and registration (Fastify).

Phase 2: High-Performance Data Engine

• Build the LifecycleManager to wrap tardis-machine.
• Implement the Worker Thread pool with TypedArray-based book storage.
• Implement the Consistency Validator (Checksum & Snapshot logic).

Phase 3: Access & Gateway

• Build the TMC-Gateway using uWebSockets.js for zero-copy proxying.
• Implement Master Recovery (pulling state from Workers on startup).

Phase 4: Production Hardening

• Add resource-weighted distribution based on Event Loop Lag.
• Build the institutional monitoring dashboard (Prometheus/Grafana).