exokb-ui: Unified ingestion service design report

exokb-ui-report.md

exokb-ui: Unified Ingestion Service for the Cross-Source Knowledge Base

Date: 2026-07-28
Author: exonomyapp
Status: Design proposal (updated with Gemini Antigravity findings)
Related: exokb knowledge store, kb_ingest.py

---

Problem

The exokb (cross-source knowledge base) ingests conversation history from three sources — OpenCode, Hermes, and Antigravity — but has no unified, reliable, automated trigger that ensures all three are polled and ingested in a timely manner.

Current state of triggers:

Source	Trigger mechanism	Status
OpenCode	TypeScript plugin fires kb_ingest.py --sources opencode on session.idle	Working (but only when OpenCode is running, and only covers OpenCode)
Hermes	Systemd path watcher on /home/exocrat/.hermes/state.db	Disabled — not actively monitoring
Antigravity (old IDE)	Nothing exists	Never triggered, extractor script has broken paths
Antigravity (Gemini)	Nothing exists	Not discovered until 2026-07-28 session

The Hermes path watcher and Antigravity extractor were each built as point solutions. Neither is reliably active. We need a single daemon that owns the ingestion schedule for all sources.

---

Solution: exokb-app

A system service with an embedded HTML UI that:

• Polls each source database every 3 seconds using the most natural monotonic column for that source (auto-increment ID or timestamp)
• Ingests only new rows since the last poll, using persisted watermarks
• Writes into the shared exokb.db using INSERT OR REPLACE for idempotency
• Provides a browser-accessible dashboard to monitor and manage each source

---

Architecture

Source Watermark Strategies

Each source has a different monotonic column that serves as its natural watermark:

Source	Database	Granular unit	Watermark column	Watermark type
OpenCode	/home/exocrat/.local/share/opencode/opencode.db	part (sub-message turn)	part.time_created (epoch ms)	Time-based
Hermes	/home/exocrat/.hermes/state.db	messages (conversation message)	messages.id (auto-increment)	ID-based
Antigravity (old IDE)	/home/exocrat/.local/share/antigravity/antigravity.db	turns (session turn)	turns.id (auto-increment)	ID-based
Antigravity (Gemini)	/home/exocrat/.gemini/antigravity/conversations/*.db	steps (conversation step)	steps.idx (auto-increment)	ID-based

For each source, the poller queries MAX(watermark_column) first. If unchanged since the last poll, the full fetch is skipped. If changed, it queries all rows where the watermark exceeds the stored high-water mark, inserts them into exokb.db, and advances the watermark.

Poll Loop

Every 3 seconds, for each enabled source:
  1. Check MAX(watermark) from source DB
  2. If same as stored watermark → skip
  3. If higher:
     a. Acquire exclusive lock on exokb.db (via _ingest_locks)
     b. SELECT new rows WHERE watermark > stored
     c. INSERT OR REPLACE into history table
     d. Update _ingest_state watermark
     e. Release lock

The exclusive lock (already implemented in ingest_coordinator.py via BEGIN EXCLUSIVE) prevents concurrent extractors from colliding. Since this service replaces the separate extractors, it will be the sole writer.

HTTP Server and UI

A lightweight HTTP server serves:

• Dashboard (/) — HTML page showing a card for each source:
  • Source name and DB path
  • Status indicator (green = polling ok, red = error, grey = disabled)
  • Last poll timestamp (human-readable local time)
  • Watermark value
  • Rows ingested since start
  • Toggle enable/disable
  • "Ingest now" button (manual trigger)
  • Last error message (if any)
• FTS search (/search) — Full-text search over ingested conversation history powered by the FTS5 index on the history table
• API (/ingest/{name}) — Trigger immediate poll for one source

Directory Layout

/home/exocrat/code/exokb-app/
├── exokb_app/
│   ├── __init__.py
│   ├── __main__.py            # entry point: starts scheduler + HTTP server
│   ├── server.py              # FastHTML app: dashboard routes and rendering
│   ├── poller.py              # APScheduler job definitions
│   ├── state.py               # shared poll status dict
│   ├── adapters/
│   │   ├── __init__.py
│   │   ├── base.py            # abstract base class for source adapters
│   │   ├── opencode.py        # OpenCode adapter
│   │   ├── hermes.py          # Hermes adapter
│   │   ├── antigravity.py     # Old Antigravity IDE adapter
│   │   └── gemini_antigravity.py  # New Gemini Antigravity adapter (TODO)
│   └── tests/
│       ├── __init__.py
│       ├── test_base.py
│       ├── test_opencode.py
│       ├── test_hermes.py
│       └── test_antigravity.py
├── docs/
│   ├── plans/
│   │   └── exokb-app.md       # implementation plan
│   └── reports/
│       └── exokb-ui-report.md # this report
├── tests/
│   └── test_e2e.py
├── fixtures/
├── pyproject.toml
├── systemd/
│   └── exokb-app.service
└── .gitignore

Source Adapter Interface

Each source adapter implements:

class SourceAdapter(ABC):
    @property
    def name(self) -> str: ...
    @property
    def db_path(self) -> Path: ...
    def get_watermark(self) -> int: ...
    def fetch_new_rows(self, since_watermark: int) -> list[dict]: ...
    def row_to_history(self, row: dict) -> dict: ...

This makes adding a new source (e.g. Gemini Antigravity, Zed IDE, TRAE) trivial — just write a new adapter class.

State Persistence

The service reuses the existing _ingest_state table in exokb.db:

CREATE TABLE _ingest_state (
    source             TEXT PRIMARY KEY,
    last_max_id        INTEGER,
    last_created_at_ms INTEGER,
    updated_at         TEXT
);

---

Discovery: Two Antigravity Installations

On 2026-07-28 it was discovered that this system has two separate Antigravity installations, each with its own conversation store:

1. Old Antigravity IDE (antigravity-ide)

Property	Value
Config dir	~/.config/antigravity-ide/
Database	~/.local/share/antigravity/antigravity.db
Schema	Simple SQL tables (turns, sessions)
Last activity	Jun 30, 2026
Extractor status	Existing adapters/antigravity.py handles this source

The old IDE's turns table contains plain-text columns (role, content, tool_name, tool_args, tool_result) that map directly to the history table format. The existing adapter works.

2. New Gemini Antigravity (Antigravity — capital A)

Property	Value
Config dir	~/.gemini/antigravity/
Database	~/.gemini/antigravity/conversations/<uuid>.db (one per conversation)
Schema	Protobuf blobs in SQLite (steps, trajectory_meta, gen_metadata, executor_metadata)
Last activity	Jul 28, 2026 (today)
Extractor status	Not yet implemented — needs a new adapter

Gemini Antigravity Database Schema

Each conversation is a separate SQLite database. Key tables:

trajectory_meta     — one row per conversation trajectory
  trajectory_id     TEXT PRIMARY KEY
  cascade_id        TEXT (matches the DB filename UUID)
  trajectory_type   INTEGER
  source            INTEGER

steps               — ordered conversation steps
  idx               INTEGER PRIMARY KEY (monotonic, usable as watermark)
  step_type         INTEGER (14=conversation root, 15=user, 8=tool, 98=etc.)
  status            INTEGER
  step_payload      BLOB (protobuf-encoded content)
  task_details      BLOB
  render_info       BLOB
  ...

The step_payload blob contains protobuf-encoded conversation content that must be decoded to extract user messages, assistant responses, and tool calls. Step type 14 contains the conversation title/metadata, type 15 contains user messages, and type 8 contains tool/view_file operations.

Adapter Requirements

A new adapters/gemini_antigravity.py must:

1. Discover conversation DBs in ~/.gemini/antigravity/conversations/ (or track them via a watermark table)
2. Read steps.idx as the monotonic watermark
3. Decode step_payload protobuf blobs to extract text content
4. Map decoded content to the history table format with source name gemini_antigravity

The protobuf schema is not yet fully reverse-engineered — the blobs contain embedded strings, JSON tool definitions, and binary fields that need a decoder.

---

Deployment

The service runs as a systemd user service (Type=simple):

[Unit]
Description=exokb-app: unified ingestion service

[Service]
Type=simple
ExecStart=/usr/bin/python3 -m exokb_app
WorkingDirectory=/home/exocrat/code/exokb-app
Restart=on-failure

[Install]
WantedBy=default.target

The HTTP server listens on localhost port 8765.

---

Open Questions

1. Gemini Antigravity protobuf schema — The step_payload blobs need a protobuf decoder. The .proto file or field definitions need to be extracted from the Gemini Antigravity installation or reverse-engineered.
2. Multiple Antigravity watermarks — Should the two Antigravity sources share a single watermark namespace in _ingest_state, or use separate entries (antigravity vs gemini_antigravity)?
3. Zed IDE as source #5 — Zed stores chat history at ~/.local/share/zed/db/. Adding it would follow the same adapter pattern.
4. Conversation discovery — The Gemini Antigravity adapter needs to discover new conversation DB files appearing in the conversations directory. This could use a directory-scan watermark or inotify.

---

Appendix: Why Not Keep the Old Approach

Old approach	Problem
Separate extractor scripts per source	Duplicate logic, inconsistent patterns, stale paths
Systemd path watchers	Each needs its own .path + .service unit; all must be enabled independently; no unified dashboard
OpenCode plugin only	Only runs when OpenCode is running; only handles one source

Additional Notes:

Several specialized Python libraries and system utilities can eliminate nearly all of the boilerplate code required for the proposed ingestion daemon and monitoring dashboard.

Dashboard and HTTP Layer

Constructing a custom HTTP server using standard libraries requires manual routing, template rendering, and client-side JavaScript for interactive updates. FastHTML accelerates this process by enabling the entire interface and state management to be written purely in Python. Because FastHTML natively integrates HTMX, interactive actions such as triggering an immediate ingestion run or toggling a database source perform server-side partial updates without writing client-side code or managing external template files. Alternatively, Flask or Bottle provide conventional micro-framework routing if a standard REST JSON structure is preferred.

Database Interaction and Ingestion

Writing raw SQL queries for schema mapping, state retrieval, and idempotent record insertion introduces repetitive data-handling code. sqlite-utils provides high-level Python abstractions tailored specifically for SQLite. It automates primary key upserts, schema creation, dynamic record transformations, and high-watermark queries in a few concise function calls. For an even more lightweight alternative, FastLite offers minimal data API functions that streamline INSERT OR REPLACE execution and state table persistence directly against SQLite files.

Polling and Execution Control

Implementing background polling via manual while loops and time sleep intervals introduces subtle failure modes, such as thread deadlocks, unhandled exception terminations, and execution timing drift. APScheduler (Advanced Python Scheduler) provides dedicated background execution management. It runs independent polling intervals across adapters concurrently, catches adapter-level errors cleanly, and prevents overlapping task executions if a source database query experiences unexpected latency.

Daemon Supervision

Daemonizing python scripts manually requires writing log redirection, pid-file management, and automatic recovery logic. Systemd user units handle background supervision natively within the operating system. Systemd provides automatic restarts on failure, environment variable isolation, dependency management, and centralized log aggregation via journalctl, requiring no supplementary process monitoring tools.