Skip to content

Instantly share code, notes, and snippets.

@mdp

mdp/POSTHOG.md

Created May 18, 2026 19:49
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save mdp/b7afcb8642154a77c5ac6f28091d2d3d to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save mdp/b7afcb8642154a77c5ac6f28091d2d3d to your computer and use it in GitHub Desktop.
Download ZIP
PostHog x BigBrother: Inception Debugging & Observability Strategy
Raw
POSTHOG.md

🦔 PostHog × BigBrother: "Inception" Debugging & Observability Strategy

Author: AI Analysis
Date: May 18, 2026
Status: Exploration & Case-Making

1. Executive Summary

BigBrother already has three parallel PostHog integrations wired into its runtime — OTel LLM tracing, console-log ingestion, and structured error events. The PostHog MCP server is connected but unused by the agent itself. This creates an untapped "inception" loop: the agent sends telemetry to PostHog, then queries that same telemetry through the MCP server to debug itself, understand costs, and improve its own behavior.

The gap is not technical infrastructure. It's agent-level skills and tool access. This report maps what exists, what's missing, and the concrete steps to close the loop.

2. Current PostHog Footprint in BigBrother

2.1 What's Already Wired

Layer Technology Files Status
LLM Traces @posthog/ai/otel (PostHogSpanProcessor), OTel SDK src/telemetry/instrumentation.ts, posthog-processor.ts, model-aliases.ts ✅ Live in daemon mode
Log Ingestion OTel BatchLogRecordProcessor → OTLP → PostHog /i/v1/logs src/telemetry/instrumentation.ts, src/logging.ts ✅ Live — console.* is patched to emit OTel log records
Error Events posthog-node SDK (capture, captureException) src/telemetry/error-events.ts ✅ Live — 6 event types defined
Cost Attribution Model alias normalization for OpenRouter → PostHog price table match posthog-processor.ts, model-aliases.ts ✅ Live
Discord Alerts notifySerious() posts to Discord on fatal errors discord-notify.ts ✅ Live
PostHog MCP Server Connected at https://mcp.posthog.com/mcp with POSTHOG_API_KEY src/config/mcp-servers.json ✅ Connected, but no skill uses posthog_* tools

2.2 Error Events Currently Captured

Event When Captures
agent.llm_error Exception inside withAiSpan runId, functionId, spanName, errorName, errorMessage
cron.job_failed Scheduler task throws jobId, jobName, errorName, errorMessage
email.process_failed Email thread handler throws threadId, errorName, errorMessage
email.attachment_ocr_failed Mistral OCR fails threadId, filename, errorMessage
agent.doom_loop Defined, not wired —
agent.step_limit Defined, partially wired in harness —

2.3 PostHog MCP Server Connection

{
  "id": "posthog",
  "url": "https://mcp.posthog.com/mcp",
  "enabled": true,
  "toolPrefix": "posthog",
  "headers": { "Authorization": "Bearer ${POSTHOG_API_KEY}" }
}

The PostHog MCP server provides tools for:

  • HogQL queries — run arbitrary SQL/HogQL against PostHog data (events, persons, spans, logs)
  • Feature flag management — list, create, update flags
  • Experiment management — manage AB tests
  • Dashboard/insight access — read dashboards and insights
  • Person/cohort management — query user data and cohorts
  • Annotation creation — mark events on timelines

These tools are loaded into getAllMcpTools() under the posthog_ prefix, available to any Cline skill run — but no skill frontmatter declares them, so they're never assembled into any tool belt.

3. The Inception Loop: How It Should Work

┌─────────────────────────────────────────────────────────┐
│                                                         │
│   BigBrother Agent Run                                  │
│   (runClineSkill)                                       │
│       │                                                 │
│       ├──→ LLM calls, tool calls, errors                │
│       │       │                                         │
│       │       ▼                                         │
│       │   ┌─────────────────┐                           │
│       │   │  PostHog        │  OTel spans + logs        │
│       │   │  (telemetry)    │  + error events           │
│       │   └────────┬────────┘                           │
│       │            │                                    │
│       │            ▼                                    │
│       │   ┌─────────────────┐                           │
│       │   │  PostHog        │  Stored as spans,         │
│       │   │  (storage)      │  events, log records      │
│       │   └────────┬────────┘                           │
│       │            │                                    │
│       │            ▼                                    │
│       │   ┌─────────────────┐                           │
│       │   │  PostHog MCP    │  Queryable via HogQL      │
│       │   │  Server         │  through posthog_* tools  │
│       │   └────────┬────────┘                           │
│       │            │                                    │
│       └──→ Agent queries own telemetry ←──────────────┘
│               │
│               ▼
│       "Why did I fail last run?"
│       "How much did that meeting-prep cost?"
│       "Show me the logs from the failed application-scoring"
│       → Self-corrects, includes context in output
│
└─────────────────────────────────────────────────────────┘

Key insight: BigBrother's own functionId (e.g., meeting-prep, deep-research, application-scoring) is already attached to every span and error event. The agent can query by these identifiers to find its own history.

4. The Case: What Agent-Level PostHog Access Unlocks

4.1 Self-Debugging ("Why did I fail?")

Today, when a task fails, the error is captured in PostHog but the agent itself has no awareness of it on subsequent runs. With posthog_run_hogql_query in its tool belt, the agent can:

-- On startup: "Check if my last run failed"
SELECT properties.run_id, properties.function_id, properties.error_message
FROM events
WHERE event = 'agent.llm_error'
  AND timestamp > now() - INTERVAL 1 HOUR
ORDER BY timestamp DESC
LIMIT 5

Value: The agent can tell you "I had 3 failures in the last hour, all in application-scoring, all from the same company with missing founder data" — without a human opening PostHog.

4.2 Cost Awareness

withAiSpan attaches ai.usage.* tokens + gen_ai.response.model to every span. PostHog computes $ai_total_cost_usd server-side. The agent can query:

-- "How much did yesterday's meeting-prep cost per partner?"
SELECT
    properties.partner_name ?? 'unknown' as partner,
    count() as runs,
    sum(ai_total_cost_usd) as total_cost
FROM spans
WHERE function_id = 'meeting-prep'
  AND timestamp > now() - INTERVAL 1 DAY
GROUP BY partner
ORDER BY total_cost DESC

Value: The agent can include cost summaries in its daily standup message. Detect runaway spend on a particular task or model choice.

4.3 Log-Based Debugging

Every console.log/warn/error/debug statement is routed to PostHog logs via OTel. The agent can retrieve its own logs:

-- "Show me the error logs from my most recent run"
SELECT body, severity_text, timestamp
FROM logs
WHERE trace_id = '${traceId}'  -- from the OTel trace
  AND severity_text = 'ERROR'
ORDER BY timestamp

Value: Deep debugging without SSH access to the production container. Logs are in PostHog, queryable by the agent itself.

4.4 Pattern Discovery & Self-Improvement

With access to historical data, the agent can ask meta-questions:

-- "Which skill produces the most errors?"
SELECT properties.function_id, count() as errors
FROM events
WHERE event = 'agent.llm_error'
  AND timestamp > now() - INTERVAL 7 DAY
GROUP BY properties.function_id
ORDER BY errors DESC

-- "What's the average step count per completed task?"
SELECT function_id, avg(iterations)
FROM spans
WHERE status = 'OK'
  AND timestamp > now() - INTERVAL 7 DAY
GROUP BY function_id

-- "Which tool does the agent call most frequently before hitting max_iterations?"
SELECT properties.repeated_tool, count() as occurrences
FROM events
WHERE event = 'agent.step_limit'
  AND timestamp > now() - INTERVAL 7 DAY
GROUP BY properties.repeated_tool
ORDER BY occurrences DESC

Value: Data-driven decisions about prompt engineering, tool choices, and task design.

4.5 Incident Response

When notifySerious() fires a Discord alert, the agent could automatically:

  1. Query PostHog for the past 5 minutes of agent.llm_error events
  2. Pull the relevant log records
  3. Post a rich root-cause summary to Discord

Currently the Discord alert is a blunt message.slice(0, 2000). With self-querying, it could include:

🚨 email.sweep_failed
InboxPoller threw: Connection timeout to google-workspace MCP
Context: 3 prior failures in the last 10 minutes. Last successful sweep was at 14:23 UTC.
Related: 2 email.process_failed events from the same thread (threadId: 1923abcd).
Logs: Connection reset by peer at google-workspace MCP endpoint.

5. Gaps Between Current State and Ideal

5.1 Agent-Level Gaps

Gap Severity Current State Desired State
No skill declares posthog_* tools 🔴 Blocking MCP server connected, tools loaded, but no skill includes them in tools: frontmatter A self-diagnose or query-telemetry skill with tools: [posthog_run_hogql_query]
Agent can't initiate HogQL queries 🔴 Blocking No mechanism for agent to request a HogQL query Light tool queryPostHog() or skill with HogQL tool
Runs without telemetry have no functionId 🟡 Major Some runClineSkill calls lack functionId, making queries impossible Implement ERROR_PLAN.md — add functionId to all calls
Tool-level tracing is flat 🟡 Moderate One OTel span per runClineSkill call — individual tool calls are not traced Wire onToolEvent → span events or child spans
Error events not consistently fired 🟡 Moderate Error plan exists but is unimplemented Wire captureDiscordCommandError, captureEmailSweepError, captureFatalError

5.2 Infrastructure Gaps

Gap Severity Current State Desired State
No source maps in PostHog 🟢 Minor Stack traces from Docker are minified JS Upload source maps for grouped error deduplication
PostHog server-side alerts not configured 🟢 Minor Error alerting is via notifySerious() in-process Plus PostHog-based threshold alerts (e.g., >5 errors/10min → Discord webhook)
CLI runs have no telemetry 🟢 Minor startTelemetry() only runs in daemon mode pnpm cli could optionally enable telemetry

6. Implementation Roadmap

Phase 1: Foundation (High Impact, Low Effort)

Goal: Give the agent a way to query its own telemetry, and ensure every run is tagged with a functionId.

Step What Files Effort
1.1 Create self-diagnose skill that declares posthog_run_hogql_query in its tools src/skills/self-diagnose/SKILL.md Small
1.2 Register self-diagnose in chat skill menus (channels: [discord, whatsapp]) SKILL.md frontmatter Trivial
1.3 Implement ERROR_PLAN.md: add functionId/properties to ClineSkillInput src/cline/harness.ts Medium
1.4 Wire functionId into all task runClineSkill calls src/tasks/*.ts Medium
1.5 Add missing error events: captureDiscordCommandError, captureEmailSweepError, captureFatalError src/telemetry/error-events.ts, src/channels/discord.ts, src/channels/email/pipeline.ts, src/index.ts Small

Outcome: Agent can now say posthog_run_hogql_query({'query': 'SELECT count() FROM events WHERE event = \'agent.llm_error\' AND timestamp > now() - interval 1 day'}) and get answers.

Phase 2: Enhanced Tracing (Medium Impact, Medium Effort)

Goal: Per-tool-call visibility in PostHog.

Step What Files Effort
2.1 Wire onToolEvent into withAiSpan to add span events for each tool call src/cline/harness.ts Medium
2.2 Attach timing metadata (start/end timestamps, duration) to tool span events src/cline/harness.ts Small
2.3 Capture tool-level error information as structured span attributes src/cline/harness.ts Small

Outcome: PostHog shows a flame graph of each tool call within a skill run — "application-scoring failed on notion_query_database with timeout after 30s."

Phase 3: Self-Healing & Feedback (High Impact, High Effort)

Goal: The agent reacts to its own telemetry.

Step What Files Effort
3.1 Build autoDiagnose(): after a failed run, automatically query PostHog for context src/cline/harness.ts, new file Medium
3.2 Build auto-incident-response: on notifySerious-class errors, agent queries PostHog and posts rich context to Discord src/telemetry/error-events.ts Medium
3.3 Build daily-health-check task: agent queries PostHog for yesterday's stats, posts to Discord src/tasks/health-check.ts Medium
3.4 Upload source maps to PostHog for JS stack trace grouping Dockerfile, CI Small

Outcome: The agent is self-aware. It knows when it's failing, why, and can tell the team.

Phase 4: Advanced (Niche)

Step What Effort
4.1 Agent-created PostHog dashboards for team-facing stats Medium
4.2 Agent-created PostHog annotations on deploy/restart events Small
4.3 Cost-optimization agent: queries model-level cost breakdown, recommends cheaper models for low-stakes tasks Medium

7. Skill Design: self-diagnose

Here's the concrete shape of the first skill:

---
name: self-diagnose
description: Query BigBrother's own PostHog telemetry — recent errors, cost, logs, and run stats.
channels: [discord, whatsapp]
trigger: "debug a failure or check health"
tools: [posthog_run_hogql_query]
---

You are a diagnostic assistant for the BigBrother agent system.

Given a failure context or a question about recent operations, query PostHog using
`posthog_run_hogql_query` to find relevant telemetry.

Standard queries to run:

1. **Recent agent errors:**
   ```sql
   SELECT timestamp, properties.run_id, properties.function_id, properties.error_message
   FROM events WHERE event = 'agent.llm_error'
   ORDER BY timestamp DESC LIMIT 10

  1. Cost by function (last 24h):

    SELECT function_id, count() as runs, sum(ai_total_cost_usd) as total_cost
FROM spans WHERE timestamp > now() - interval 1 day
GROUP BY function_id ORDER BY total_cost DESC

  2. Logs for a specific run:

    SELECT body, severity_text, timestamp
FROM logs WHERE trace_id = '<trace_id>'
ORDER BY timestamp

Summarize findings concisely. If the runId or functionId is provided by the caller, scope queries to that context. If not, show the broad picture of recent activity.


---

## 8. Data Model: How Queries Map to PostHog Data

| BigBrother Concept | PostHog Data Type | Key Fields |
|-------------------|------------------|-----------|
| Agent Run (`runClineSkill`) | OTel Span | `name: "ai.cline.<functionId>"`, `span_id`, `trace_id`, `attributes.ai.run.id`, `attributes.ai.function.id` |
| Tool call | (Proposed) Span Event on parent span | `name: "tool.start.<toolName>"`, `attributes.input`, `attributes.output`, `attributes.error` |
| Error | PostHog Event | `event: "agent.llm_error"`, `properties.run_id`, `properties.function_id`, `properties.error_message` |
| Console log | OTel Log Record | `body`, `severity_text`, `trace_id` (correlated to span trace) |
| Model cost | Span attribute → PostHog AI | `gen_ai.response.model`, `ai.usage.input_tokens`, `ai.usage.output_tokens`, computed `$ai_total_cost_usd` |
| Cron failure | PostHog Event | `event: "cron.job_failed"`, `properties.job_id` |
| Discord command failure | (Not yet sent) | N/A |

---

## 9. Open Questions

These emerged during the analysis and need partner input:

| Question | Why It Matters |
|----------|---------------|
| Should the agent auto-query PostHog on failure without being asked, or wait for a human to invoke the skill? | Changes whether error-handling logic in `runClineSkill` grows a PostHog query step |
| How much context should the agent include in self-diagnosis output? | Error message + functionId + timestamp is safe. Full stack traces in Discord could leak via alert channels |
| Should the PostHog MCP server's HogQL capability be exposed as a light tool (narrower scope) rather than through an MCP tool? | Light tools have defined input schemas; MCP tools pass raw JSON. A light tool could enforce read-only queries |
| What's the retention on BigBrother's PostHog project? | Determines how far back the agent can query |
| Are there PostHog RBAC concerns — does the MCP key have read access to all data in the project? | The agent shouldn't accidentally read product analytics data via HogQL queries meant for telemetry |

---

## 10. Risks & Mitigations

| Risk | Likelihood | Mitigation |
|------|-----------|------------|
| Agent enters infinite query loop (keeps asking PostHog why it failed, fails again, repeats) | Low | Add query budget — max 3 diagnostic queries per run |
| HogQL query costs (PostHog bills by query volume) | Low | Limit agent-triggered queries; use PostHog's `LIMIT` clause |
| Circular dependency: agent can't run skill because PostHog MCP is down | Low | The `self-diagnose` skill is optional — no run depends on it for core functionality |
| Telemetry data volume grows unbounded | Medium | Already handled by PostHog's volume-based pricing; log level filtering could reduce noise |

---

## 11. Conclusion

BigBrother has a sophisticated but **one-way** relationship with PostHog: it emits telemetry but never reads it back. Closing this loop turns PostHog from a passive observability sink into an **active runtime feedback system**.

The fastest path to value is **Phase 1** — a single `self-diagnose` skill that gives the agent `posthog_run_hogql_query`, plus the ERROR_PLAN.md implementation to ensure every run is tagged with a `functionId`. This is ~2-3 days of work and unlocks the core inception debugging capability.
The full vision — auto-incident-response, self-healing, cost optimization — can layer on top incrementally as the telemetry data becomes richer and the agent gains confidence querying it.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment