PRJ-041 — Incident Management: Log Monitoring & AI-Driven Incident Pipeline

Author: Claudia
Date: 2026-03-30
Status: Research (S01)
Project: PRJ-041
Related: T-00379 (GitHub Issue #1), RSH-005 (Lobster pipelines), PRJ-039 (CI/CD)

1. Problem Statement

We have ~100MB of logs generated daily across OpenClaw gateway and the RapidoFab plugin. Nobody reads them. Errors accumulate silently until something visibly breaks — at which point we're in reactive firefighting mode.

Current pain points:

3,654 ERROR/WARN lines in a single day's FAB log (2026-03-29)
Gateway error log at 68MB and growing
No alerting — errors discovered hours or days later
No grouping — the same recurring error generates thousands of lines
No trend tracking — can't tell if things are getting better or worse
FAULT declarations are manual and post-hoc (we discover problems, not prevent them)

Goal: Transform raw log noise into actionable incidents with AI-assisted classification, automatic deduplication, and integration with FAB task management.

2. Decision: PostHog vs. Custom (Option C: Hybrid)

2.1 PostHog — Why Not

PostHog is excellent for web application monitoring with SDK instrumentation. Our use case is fundamentally different:

PostHog Strength	Our Reality
JavaScript SDK captures exceptions in-app	We have text log files from a Node.js daemon
Session replay links errors to user actions	No browser sessions — backend process
Error grouping by stack trace	Log lines, not structured exceptions
Cloud or self-hosted with full UI	Self-host = Docker + Postgres + Redis + ClickHouse

To use PostHog, we'd need to:

Build a log parser (same work as custom)
Transform log lines into PostHog event format
Send via posthog-node SDK
Either host PostHog infrastructure or send logs to cloud

The integration cost equals the custom solution cost, plus PostHog overhead.

PostHog Error Tracking free tier: 100K exceptions/month — generous, but irrelevant since the bottleneck is integration, not capacity.

2.2 Custom + LLM (Option C) — Why Yes

Zero external dependencies — Lobster + local logs + llm_task.invoke
Zero tokens for signal extraction (Layer 1) — pure grep/awk
Minimal tokens for classification (Layer 2) — a few hundred per scan via llm_task.invoke
Native FAB integration (Layer 3) — incidents become tasks directly
Matches Rupert's directive: "formalise as many automated procedures as possible as lobster pipelines"
Data stays local — no third-party exfiltration of error logs

3. Architecture: Three-Layer Pipeline

┌──────────────────────────────────────────────────────────┐
│                    LOG SOURCES                            │
│  ~/.openclaw/logs/gateway.err.log     (68MB, growing)    │
│  ~/.openclaw/logs/gateway.log         (2.8MB)            │
│  ~/.openclaw/logs/rapido-fab.log      (today's log)      │
│  ~/.openclaw/logs/rapido-fab.YYYY-MM-DD.log (archived)   │
│  ~/Library/Logs/openclaw/gateway.err.log (older logs)    │
│  ~/Library/Logs/openclaw/otm-*.log    (legacy OTM)       │
└────────────────────────┬─────────────────────────────────┘
                         │
                         ▼
┌──────────────────────────────────────────────────────────┐
│  LAYER 1: SIGNAL EXTRACTION (Lobster, zero tokens)       │
│                                                          │
│  Cron: every 15 min                                      │
│  • Read byte offset from state (state.get last-offset)   │
│  • tail log files from offset                            │
│  • grep ERROR|WARN|exception|fail|crash                  │
│  • Parse structured fields:                              │
│    - timestamp, level, component, message                 │
│  • Emit JSON signal array                                │
│  • Update offset (state.set last-offset)                 │
│                                                          │
│  Output: signals.jsonl (append-only)                     │
│  { ts, level, component, message, source, line }         │
└────────────────────────┬─────────────────────────────────┘
                         │
                         ▼
┌──────────────────────────────────────────────────────────┐
│  LAYER 2: INCIDENT AGGREGATION (LLM, minimal tokens)    │
│                                                          │
│  Cron: every 1 hour (or on-demand after L1)              │
│  • Read new signals since last aggregation               │
│  • Batch signals (max 50 per LLM call)                   │
│  • LLM classifies via llm_task.invoke:                   │
│    - Group signals into incidents (dedup/merge)           │
│    - Assign severity: critical/high/medium/low/noise      │
│    - Identify root cause hypothesis                       │
│    - Suggest fix action                                   │
│    - Match against known incident patterns                │
│  • Merge with existing open incidents (same root cause)   │
│  • Store in incidents.jsonl                               │
│                                                          │
│  Output: incidents.jsonl                                 │
│  { id, severity, category, rootCause, suggestedFix,      │
│    signalCount, firstSeen, lastSeen, status, taskId }    │
└────────────────────────┬─────────────────────────────────┘
                         │
                         ▼
┌──────────────────────────────────────────────────────────┐
│  LAYER 3: INCIDENT PROCESSING (mixed)                    │
│                                                          │
│  Triggered by L2 when new incident or severity change    │
│  • Critical/High → immediate Telegram alert              │
│  • Any new incident → create FAB task (fab_task create)  │
│  • Recurring incident → update existing task count       │
│  • Resolved incident → close FAB task                    │
│  • Weekly → trend report (new vs resolved vs recurring)  │
│  • Monthly → incident review for process improvement     │
│                                                          │
│  Integration points:                                     │
│  • FAB tasks (fab_task tool via clawd.invoke)             │
│  • GitHub Issues (future, via PRJ-038)                   │
│  • Telegram notifications (clawd.invoke → message)       │
│  • Slack #inbox-actions (clawd.invoke → message)         │
│  • FAULT declarations (auto-draft when critical)         │
└──────────────────────────────────────────────────────────┘

4. Log Sources Inventory

4.1 Current Log Files

File	Format	Size	Rotation	Content
`~/.openclaw/logs/gateway.err.log`	Semi-structured text	68MB (no rotation!)	None	All stderr: model fallbacks, drain errors, memory failures, Slack WS timeouts, plugin errors
`~/.openclaw/logs/gateway.log`	Semi-structured text	2.8MB	None	Normal operations: session starts, tool calls, completions
`~/.openclaw/logs/rapido-fab.log`	Structured `[timestamp] [LEVEL] [component] message`	~12MB/day	Daily (`.YYYY-MM-DD.log`)	FAB plugin: dispatch, completion, services, HTTP, tools
`~/.openclaw/logs/config-audit.jsonl`	JSONL	149KB	None	Config change audit trail
`~/Library/Logs/openclaw/gateway.err.log`	Text	25MB	Stale (old location)	Legacy gateway errors
`~/Library/Logs/openclaw/otm-*.log`	Text	~8MB total	Stale	Legacy OTM services

4.2 Log Format Samples

Gateway err.log (semi-structured):

2026-03-30T12:35:12.724+02:00 [model-fallback/decision] model fallback decision: decision=candidate_failed ...
2026-03-30T12:35:12.725+02:00 Embedded agent failed before reply: Gateway is draining for restart
2026-03-30T12:37:33.969+02:00 [memory] qmd search failed: Error: qmd search ... failed (code 1): sqlite-vec extension is unavailable

rapido-fab.log (structured FabLogger):

[2026-03-29T00:24:01.071Z] [ERROR] [fab:dispatch] [fab:services:dispatch] SPAWN-CLI-FAIL: CLI spawn failed for agent=devdas task=T-00358
[2026-03-29T00:00:00.107Z] [ INFO] [fab:register:gateway] register() #47 [gateway] {"argv":["gateway","--port","18789"],...}

4.3 Known Error Categories (from current logs)

Category	Example	Frequency	Severity
Model fallback	`candidate_failed requested=sonnet next=opus`	High	Low (self-heals)
Gateway draining	`Gateway is draining for restart; new tasks are not accepted`	Medium	Low (transient)
Memory/qmd failure	`sqlite-vec extension is unavailable`, `database is locked`	Medium	Medium
Slack WS timeout	`A pong wasn't received from the server before the timeout`	High	Low (auto-reconnects)
FAB dispatch failure	`SPAWN-CLI-FAIL: CLI spawn failed for agent=X task=T-NNNNN`	Medium	High
Subagent announce fail	`Subagent completion direct announce failed`	Low	Medium
Plugin errors	Plugin not found, plugin disabled	Low	High (feature broken)

5. Incident Data Model

5.1 Signal (Layer 1 output)

interface Signal {
  id: string;              // UUID
  timestamp: string;       // ISO 8601
  source: string;          // "gateway.err" | "rapido-fab" | "gateway"
  level: "ERROR" | "WARN" | "INFO";
  component: string;       // "[memory]", "[fab:dispatch]", etc.
  message: string;         // Raw log message (truncated at 500 chars)
  lineNumber: number;      // For dedup reference
}

5.2 Incident (Layer 2 output)

interface Incident {
  id: string;              // INC-NNNN
  status: "new" | "open" | "investigating" | "resolved" | "noise";
  severity: "critical" | "high" | "medium" | "low" | "noise";
  category: string;        // "model-fallback", "dispatch-failure", "memory-db", etc.
  title: string;           // Human-readable summary
  rootCause: string;       // AI hypothesis
  suggestedFix: string;    // AI suggestion
  signalIds: string[];     // Signal references
  signalCount: number;     // Total signals in this incident
  firstSeen: string;       // ISO 8601
  lastSeen: string;        // ISO 8601
  taskId?: string;         // FAB task if created (T-NNNNN)
  faultId?: string;        // FAULT declaration if created
  resolvedAt?: string;     // When marked resolved
  notes: string[];         // Human or AI notes
}

5.3 Storage

~/.openclaw/incident-mgmt/signals.jsonl — append-only signal log
~/.openclaw/incident-mgmt/incidents.json — active incident registry
~/.openclaw/incident-mgmt/state.json — scan offsets, last aggregation time
~/.openclaw/incident-mgmt/archive/YYYY-MM/ — monthly archives

6. LLM Classification Prompt Design

6.1 Signal → Incident Classification

{
  "prompt": "You are an incident classifier for an AI agent platform (OpenClaw + RapidoFab). Given a batch of error signals from system logs, group them into incidents.\n\nFor each incident:\n1. Assign a severity (critical/high/medium/low/noise)\n2. Identify the category (model-fallback, dispatch-failure, memory-db, slack-ws, gateway-drain, plugin-error, unknown)\n3. Write a human-readable title\n4. Hypothesize the root cause\n5. Suggest a fix action\n6. List which signal IDs belong to this incident\n\nRules:\n- Multiple signals with the same root cause = ONE incident\n- 'noise' severity = known benign patterns (e.g., model fallback that self-heals, Slack WS reconnect)\n- 'critical' = data loss risk, service down, or security concern\n- If a signal matches a known pattern from the patterns list, use the pre-defined category\n\nKnown benign patterns (classify as noise unless frequency is abnormal):\n- Model fallback decisions (self-healing)\n- Gateway draining during restart\n- Slack WebSocket pong timeouts (auto-reconnect)\n\nReturn JSON array of incidents.",
  "schema": {
    "type": "array",
    "items": {
      "type": "object",
      "properties": {
        "title": { "type": "string" },
        "severity": { "enum": ["critical", "high", "medium", "low", "noise"] },
        "category": { "type": "string" },
        "rootCause": { "type": "string" },
        "suggestedFix": { "type": "string" },
        "signalIds": { "type": "array", "items": { "type": "string" } }
      },
      "required": ["title", "severity", "category", "rootCause", "suggestedFix", "signalIds"]
    }
  }
}

6.2 Token Budget

Input: ~50 signals × ~200 chars each = ~10K chars ≈ ~2.5K tokens
Output: ~5-10 incidents × ~300 chars each = ~3K chars ≈ ~750 tokens
Total per scan: ~3-4K tokens (Haiku = ~$0.001 per scan)
At 4 scans/day: ~$0.004/day = effectively free

6.3 Model Choice

haiku for routine classification (fast, cheap)
sonnet for weekly incident reviews (deeper analysis)
Configure via llm_task.invoke --model haiku

7. Cron Schedule

Cron	Frequency	Pipeline	Token Cost
`incident-signal-scan`	Every 15 min	L1: signal extraction	Zero
`incident-aggregate`	Every 1 hour	L2: LLM classification	~3-4K tokens
`incident-daily-report`	Daily 8am	L3: summary to Telegram	~1K tokens
`incident-weekly-review`	Monday 9am	L3: trend analysis	~5K tokens (sonnet)

Total weekly cost: ~150K tokens ≈ ~$0.04 (Haiku) + ~$0.015 (one Sonnet call) ≈ $0.06/week

8. Notification Strategy

Severity	Action	Latency
Critical	Telegram alert + FAB task (critical priority) + sound notification	≤15 min (next L1 scan)
High	Telegram alert + FAB task (high priority)	≤1 hour (next L2 aggregation)
Medium	Included in daily report + FAB task (normal priority)	≤24 hours
Low	Included in weekly report only	≤7 days
Noise	Suppressed (logged for trend tracking)	—

Critical Alert Format (Telegram)

🚨 INCIDENT INC-0042 [CRITICAL]
FAB Dispatch Failure — agent spawn consistently failing

Root cause: Devdas agent config missing from openclaw.json
Suggested fix: Add devdas to agents.list in config, restart gateway
Signals: 23 errors in last 15 min
First seen: 12:30 | Last seen: 12:45

Task created: T-00385

9. Integration with FAB Task Management

9.1 Automatic Task Creation

When Layer 2 identifies a new incident (severity ≥ medium):

clawd.invoke --tool fab_task --action create --args-json '{
  "caller": "orchestrator",
  "callerName": "claudia",
  "title": "INC-0042: FAB Dispatch Failure",
  "description": "**Incident:** ...\n**Root Cause:** ...\n**Suggested Fix:** ...\n**Signals:** 23 errors\n**First Seen:** ...",
  "priority": "high",
  "type": "action",
  "assignee": "sylvain",
  "project": "PRJ-041"
}'

9.2 Auto-Resolution

When an incident's signals stop recurring for >24 hours:

Mark incident as resolved
Update FAB task with resolution note
If task was created → move to agent_done with result: "Auto-resolved — no recurrence in 24h"

9.3 FAULT Auto-Draft

When a critical incident is confirmed:

Draft a FAULT-NNN document from template
Pre-fill: timeline, signals, root cause hypothesis
Alert Claudia to review and finalize

10. Implementation Steps

Step	Description	Effort	Dependencies
S01 ✅	Research & architecture (this doc)	Done	—
S02	Build signal extractor (Lobster pipeline + bash)	1 day	Lobster installed ✅
S03	Build LLM aggregator (Lobster + llm_task.invoke)	1 day	S02, llm-task plugin enabled
S04	Build incident processor (FAB integration + notifications)	1 day	S02, S03
S05	Configure crons + first production run	0.5 day	S02-S04
S06	Daily/weekly reporting	0.5 day	S05
S07	Log rotation & hygiene	0.5 day	S05

Total estimated effort: ~5 days

11. Open Questions

Log rotation — gateway.err.log is 68MB with no rotation. Should we add logrotate as part of this project, or separate task for Sylvain?
Storage location — ~/.openclaw/incident-mgmt/ or /Users/claudia/Documents/WORK/PROJECTS/PRJ-041/data/? Former is closer to the system, latter follows workspace hygiene.
Incident ID scheme — INC-NNNN sequential, or date-prefixed like INC-20260330-001?
Known patterns file — should we maintain a patterns.json with known benign patterns that bypass LLM classification? Saves tokens on recurring noise.
Gateway log format — OpenClaw's gateway logs are semi-structured (no consistent format). Should we request structured logging from the OpenClaw team, or build a robust parser for the current format?
Escalation path — for critical incidents outside working hours (23:00-08:00), should we still alert immediately or queue for morning?
Incident lifecycle — should resolved incidents auto-archive after 30 days, or keep all history for trend analysis?
Dashboard — should we add an incident view to the FAB dashboard (Devdas task), or is Telegram + weekly reports sufficient for V1?

12. Success Criteria

Zero silent errors — every ERROR in logs is captured as a signal within 15 minutes
Incident dedup ratio — 100:1 or better (100 signals → ≤1 incident for same root cause)
False positive rate — <10% of incidents classified as noise on review
Mean time to detection — <1 hour for high/critical, <24 hours for medium
Cost — <$1/month in LLM tokens
Uptime — cron runs reliably (monitored by existing healthcheck cron)

RupertBarrow/INCIDENT-MANAGEMENT-RESEARCH.md

Select an option

No results found