donbr

Title: Integrating Arize Phoenix and Apache Iceberg for Local Telemetry Data Management and Querying

Authors: Don Branson

Abstract

In modern data observability workflows, capturing and managing telemetry data is crucial for debugging and improving machine learning systems. This paper demonstrates the integration of Arize Phoenix, an open-source observability platform, with Apache Iceberg, a high-performance table format for data lakes, to create a scalable and efficient local telemetry data management and querying system. We present a step-by-step implementation for capturing telemetry data as Parquet files using Arize Phoenix on a local system and using Apache Iceberg to enable schema evolution, time travel, and efficient queries. This solution bridges the gap between data observability and data lake management for machine learning monitoring.

Proposal: Enhanced Interactive Knowledge Graph (IKG) for Real-Time Situational Awareness

Objective

To design and develop a multi-agent, dynamic, and interactive knowledge graph (IKG) system that integrates real-time data fusion, ontology-based semantic layers, and advanced human-in-the-loop capabilities. This system will enhance situational awareness (SA) for Air Force operations by allowing autonomous agents to dynamically manage, analyze, and adapt knowledge graphs, ensuring timely and effective decision-making in rapidly evolving environments.

Code Examples from smolagents Documentation

╭──────────────────────────────────────────────────────────── New run ────────────────────────────────────────────────────────────╮ │ │ │ Correlate TP53BP1 protein interactions with mRNA co-expression patterns in TCGA breast cancer data. │ │ │ │ If you have the required libraries and functions to complete the task: │ │ 1. run the code │ │ 2. limit the response to the first 5 records for initial validation │ │

Knowledge Graph Construction

Some fundamental concepts

Data Ingestion

NOTE: mermaid.js diagrams below are based on some inspiring content from the Connected Data London 2024: Entity Resolved Knowledge Graphs masterclass.

graph TD

🚀 Multi-Stakeholder Perspective: Air Cargo Disruptions During COVID-19 (One-Week Interval)

📌 Global Meta Event: COVID-19 Outbreak (March 2020 - Global Lockdowns, Border Closures, & Supply Chain Shocks)
📆 Time Window: March 16–22, 2020 (One-Week Interval of Major Cargo Disruptions)
🌍 Disruption: Rapid lockdowns, flight cancellations, PPE (Personal Protective Equipment) demand spikes, and re-routing bottlenecks.

Key Stakeholders & Their Perspectives:

📦 Shipper (Medical Equipment Manufacturer) → Focus: Single Critical PPE Shipment
✈ Airline Cargo Operations (Major Carrier) → Focus: Hundreds of disrupted flights & rerouted shipments
🌏 Freight Forwarder (Global Logistics Company) → Focus: Managing demand spikes, re-bookings, and rate volatility

Crisis Management - a personal perspective

The COVID-19 pandemic highlighted critical gaps in our crisis response capabilities, particularly in an industry where I worked directly. This paper examines the requirements for effective training and preparedness during acute phases of the outbreak.

A fundamental challenge lies in preparing both human operators and their supporting AI systems to respond to unprecedented scenarios. Crisis events, by their nature, often fall outside of the standard validation parameters used to test AI systems' generalization capabilities.

The ongoing refinement of the Graph Simulation Pipeline and associated open-source datasets represents a crucial element in advancing crisis preparedness. These tools enhance our ability to transform overwhelming data streams into actionable intelligence during evolving situations.

The framework incorporates two key components:

Replay Mode: leverages tools like NetworkX and TigerGraph to model temporal dependencies using historical crisis

Knowledge Graph Pipeline

Core Components:

Prefect 3.1 Cloud
Great Expectations
dbt
Hugging Face Datasets

adding critical layers of autonomy, adaptability, and trust to crisis management frameworks.

merging these ideas into a hybrid reference architecture and implementation blueprint
structured for crisis-ready IoT systems.

Reference Architecture Diagram (Textual Representation)

Layers	Components	Enhancements