Observability-Driven Development: From Logs to Telemetry in TypeScript

You ship a feature at 4:37 PM on a Friday. At 4:42 PM, PagerDuty fires. Users are seeing HTTP 500s. You SSH into the production box, tail the logs, and see... nothing useful. The error message is Cannot read properties of undefined. No stack trace beyond your own code. No request ID to trace through the upstream services. No metric to tell you whether this affects 3 users or 30,000.

This is the moment you realize that console.log is not observability.

Observability-driven development (ODD) is the practice of instrumenting your application as you build it — not after it breaks. It treats logs, traces, and metrics as first-class deliverables, not afterthoughts. This article walks through a production-ready observability stack for TypeScript/Node.js using OpenTelemetry, structured logging with Pino, and metrics with Prometheus.

The Three Pillars, TypeScript Edition

Observability has three signals. Most teams have one (logs) and pretend it counts as three.

Logs tell you what happened at a point in time: "user 8472 attempted checkout, total $42.00, payment declined."
Metrics tell you how many times it happened and how fast: "checkout latency p99: 3.2s, error rate: 0.8%."
Traces tell you the journey across services: "checkout → inventory-service (230ms) → payment-gateway (1.8s) → notification (45ms)."

Together they form a triangulation: logs give you the what, metrics give you the scale, and traces give you the where.

The problem is that most Node.js teams stop at console.log — sometimes with JSON.stringify — and call it a day. Structured logging alone is not observability any more than a speedometer is a flight data recorder. You need all three signals, wired together with shared context (trace ID, span ID, request ID), emitted via open standards so you are not locked into a vendor.

When to Invest (and When Not To)

Observability is not free. Instrumentation adds latency, memory overhead, and code complexity. The decision framework:

Invest when:

Your application has more than one service (HTTP → worker → database → cache).
You cannot reproduce production issues locally.
Your mean time to resolution (MTTR) is measured in hours, not minutes.
You are shipping to production multiple times per day.

Skip when:

You are building an internal prototype with exactly one user.
You have a monolithic application with synchronous request handling and zero downstream calls.
Your traffic is under 100 requests per minute and you can afford to console.log everything.

For everything else, the stack described below adds roughly 200 lines of configuration, 2-5% CPU overhead, and pays for itself the first time you answer "which service is slow?" in 30 seconds instead of 30 minutes.

Structured Logging: Pino Over Everything

Stop using console.log. It is unbuffered, untyped, and writes to stdout with no metadata. The Node.js standard logging library is Pino — it is the fastest JSON logger in the ecosystem and has OpenTelemetry integration built in.

Before we write code, a word on logging conventions. A structured log line is a JSON object, not a string. Every key you add becomes a filterable, indexable dimension in your log aggregator. The rule: log context as structured fields and the message as a human-readable string.

Good:

{"level":"info","userId":"8472","cartTotal":42.00,"paymentMethod":"stripe","msg":"checkout completed"}

Bad:

{"level":"info","msg":"checkout completed for user 8472, total $42.00 via stripe"}

The second format requires regex parsing to answer "how many users paid with Stripe?" The first is a COUNT query in Grafana Loki.

Log levels are not decorative. They determine what your on-call engineer sees at 3 AM:

trace: Internal state transitions, variable values. Never enabled in production.
debug: Developer-oriented diagnostics. Occasionally enabled in production for specific users via dynamic log level.
info: Business events: "user signed up," "order placed," "payment processed." This is your default production level.
warn: Degraded but not broken: "retry attempt 2/3," "cache miss, falling back to database," "rate limit approaching."
error: Something failed that should not have failed: "database connection refused," "payment gateway timeout."
fatal: The process cannot continue: "port already in use," "unable to decrypt secrets."

Here is a logger factory that wires together structured output, trace context propagation, and environment-aware log levels:

// logger.ts
import pino from "pino";

export interface LogContext {
  requestId?: string;
  userId?: string;
  traceId?: string;
  spanId?: string;
  [key: string]: unknown;
}

const isProduction = process.env.NODE_ENV === "production";

export const logger = pino({
  level: process.env.LOG_LEVEL ?? (isProduction ? "info" : "debug"),
  // In production, emit raw JSON for log aggregation tools (Datadog, Grafana Loki, etc).
  // In development, use pino-pretty for human-readable output.
  transport: isProduction
    ? undefined
    : { target: "pino-pretty", options: { colorize: true } },
  // Merge OpenTelemetry trace context into every log line automatically.
  mixin() {
    const activeSpan = trace.getActiveSpan();
    if (!activeSpan) return {};
    const spanContext = activeSpan.spanContext();
    return {
      trace_id: spanContext.traceId,
      span_id: spanContext.spanId,
    };
  },
  serializers: {
    err: pino.stdSerializers.err,
    req: pino.stdSerializers.req,
    res: pino.stdSerializers.res,
  },
});

// Type-safe child loggers let you bind context once and propagate it.
export function childLogger(context: LogContext) {
  return logger.child(context);
}

Key decisions in this setup:

mixin() injects trace context automatically. Every log line carries trace_id and span_id without the developer remembering to pass them. This is the bridge between logs and traces.
childLogger creates scoped loggers. Call it once per request (in middleware) with requestId and userId, then pass the child logger downstream. Every log from that request carries the same identifiers.
pino.stdSerializers handle Error objects correctly — they capture message, stack, and cause so you never see [Object object] in production logs.

Usage in an Express middleware:

// middleware/requestContext.ts
import { Request, Response, NextFunction } from "express";
import { randomUUID } from "crypto";
import { childLogger } from "../logger";

declare global {
  namespace Express {
    interface Request {
      log: ReturnType<typeof childLogger>;
    }
  }
}

export function requestContext(req: Request, _res: Response, next: NextFunction) {
  const requestId = (req.headers["x-request-id"] as string) ?? randomUUID();
  req.log = childLogger({
    requestId,
    userId: (req as any).user?.id,
    method: req.method,
    path: req.path,
  });
  next();
}

Now every handler gets a pre-configured logger. No more console.log("checkout failed", userId) — just req.log.info({ cartTotal }, "checkout initiated").

Distributed Tracing: OpenTelemetry for Node.js

Logs show you one service. Traces show you the whole call chain. OpenTelemetry (OTel) is the CNCF standard — it works with Jaeger, Grafana Tempo, Datadog, Honeycomb, New Relic, and every other vendor.

The challenge in TypeScript is that OTel's Node.js SDK requires explicit setup before any other imports. Here is a production tracing bootstrap:

// tracing.ts — MUST be imported before any other module
import { NodeSDK } from "@opentelemetry/sdk-node";
import { OTLPTraceExporter } from "@opentelemetry/exporter-trace-otlp-http";
import { BatchSpanProcessor } from "@opentelemetry/sdk-trace-node";
import { HttpInstrumentation } from "@opentelemetry/instrumentation-http";
import { ExpressInstrumentation } from "@opentelemetry/instrumentation-express";
import { PinoInstrumentation } from "@opentelemetry/instrumentation-pino";
import { PgInstrumentation } from "@opentelemetry/instrumentation-pg";
import { RedisInstrumentation } from "@opentelemetry/instrumentation-ioredis";
import { Resource } from "@opentelemetry/resources";
import { ATTR_SERVICE_NAME, ATTR_SERVICE_VERSION } from "@opentelemetry/semantic-conventions";
import { diag, DiagConsoleLogger, DiagLogLevel } from "@opentelemetry/api";

if (process.env.OTEL_ENABLED !== "false") {
  // Enable SDK-level diagnostics during development; quiet in production.
  diag.setLogger(new DiagConsoleLogger(), DiagLogLevel.ERROR);

  const sdk = new NodeSDK({
    resource: new Resource({
      [ATTR_SERVICE_NAME]: process.env.OTEL_SERVICE_NAME ?? "checkout-service",
      [ATTR_SERVICE_VERSION]: process.env.npm_package_version ?? "0.0.0",
    }),
    spanProcessors: [
      new BatchSpanProcessor(
        new OTLPTraceExporter({
          url: process.env.OTEL_EXPORTER_OTLP_ENDPOINT ?? "http://localhost:4318/v1/traces",
        }),
        {
          // Batch up to 512 spans for 5 seconds before sending — avoids
          // overwhelming the collector with high-frequency spans.
          maxQueueSize: 2048,
          maxExportBatchSize: 512,
          scheduledDelayMillis: 5000,
        }
      ),
    ],
    instrumentations: [
      new HttpInstrumentation({ ignoreIncomingRequestHook: (req) =>
        req.url?.startsWith("/health") ?? false
      }),
      new ExpressInstrumentation(),
      new PinoInstrumentation(),
      new PgInstrumentation(),
      new RedisInstrumentation(),
    ],
  });

  sdk.start();

  // Graceful shutdown: flush pending spans before the process exits.
  const shutdown = async () => {
    try {
      await sdk.shutdown();
    } finally {
      process.exit(0);
    }
  };
  process.on("SIGTERM", shutdown);
  process.on("SIGINT", shutdown);
}

Critical details:

BatchSpanProcessor with scheduledDelayMillis: 5000 batching is essential. Sending every span synchronously will add 20-50ms to each request. Batching reduces this to near zero.
ignoreIncomingRequestHook filters health checks — without this, Kubernetes liveness probes will generate thousands of useless spans.
sdk.shutdown() on SIGTERM ensures in-flight spans are flushed before the container exits. Without this, you lose the last 5 seconds of trace data on every deploy.
The file must be imported first in your entry point — OTel monkey-patches http, express, pg, and ioredis before your application touches them.

Custom Spans: Tracing Your Business Logic

Auto-instrumentation covers HTTP and database calls. But the most valuable spans are your business logic. The checkout function that calls three downstream services? The calculateTax function that queries four databases? These are the spans that tell you why your p99 latency is 4 seconds.

// services/checkout.ts
import { trace, SpanStatusCode } from "@opentelemetry/api";
import { logger } from "../logger";

const tracer = trace.getTracer("checkout-service");

export async function processCheckout(
  cartId: string,
  paymentToken: string
): Promise<{ orderId: string }> {
  return tracer.startActiveSpan("checkout.process", async (span) => {
    try {
      span.setAttribute("cart.id", cartId);

      const inventory = await tracer.startActiveSpan(
        "checkout.reserveInventory",
        async (invSpan) => {
          try {
            const result = await reserveInventory(cartId);
            invSpan.setAttribute("items.reserved", result.items.length);
            return result;
          } catch (err) {
            invSpan.setStatus({ code: SpanStatusCode.ERROR, message: String(err) });
            throw err;
          } finally {
            invSpan.end();
          }
        }
      );

      const payment = await tracer.startActiveSpan(
        "checkout.processPayment",
        async (paySpan) => {
          try {
            paySpan.setAttribute("payment.amount", inventory.total);
            return await processPayment(paymentToken, inventory.total);
          } catch (err) {
            paySpan.setStatus({ code: SpanStatusCode.ERROR, message: String(err) });
            paySpan.recordException(err as Error);
            throw err;
          } finally {
            paySpan.end();
          }
        }
      );

      span.setAttribute("order.payment_id", payment.id);
      span.setStatus({ code: SpanStatusCode.OK });
      return { orderId: payment.orderId };
    } catch (err) {
      span.setStatus({ code: SpanStatusCode.ERROR, message: String(err) });
      span.recordException(err as Error);
      throw err;
    } finally {
      span.end();
    }
  });
}

This pattern — startActiveSpan, set attributes, set status, recordException, end() in finally — gives you:

Flame graphs showing exact time spent in reserveInventory vs processPayment.
Error tracking with full exception details (stack trace, message, type) via recordException.
Business dimensions like cart.id and payment.amount that let you filter traces by customer or order value.

Context Propagation: How Trace IDs Cross Service Boundaries

All three signals are useless if they cannot be correlated. The trace ID is the glue. When service A calls service B, the trace ID must travel with the HTTP request so that B's spans appear in the same trace as A's.

OpenTelemetry handles this automatically if you use the instrumented HTTP client. But many teams use fetch, axios, or a gRPC client that needs explicit header propagation. Here is how to propagate trace context manually when making outbound HTTP calls:

// httpClient.ts — context-aware fetch wrapper
import { trace, propagation, context, ROOT_CONTEXT } from "@opentelemetry/api";

export async function tracedFetch(
  url: string,
  options: RequestInit = {}
): Promise<Response> {
  const span = trace.getActiveSpan();
  const headers: Record<string, string> = { ...(options.headers as Record<string, string> ?? {}) };

  if (span) {
    // Inject the current trace context into W3C Trace Context headers.
    // Downstream services using OTel will automatically connect their spans.
    propagation.inject(
      trace.setSpan(context.active(), span),
      headers,
      {
        set(carrier, key, value) {
          carrier[key] = value;
        },
      }
    );
  }

  return fetch(url, { ...options, headers });
}

The W3C Trace Context standard defines two headers: traceparent (trace ID, span ID, flags) and tracestate (vendor-specific data). propagation.inject writes both. Any OTel-instrumented service that receives these headers automatically creates child spans under the same trace — no code changes needed on the receiving end.

If you use axios, the same pattern applies via an interceptor:

// axiosInterceptor.ts
import axios from "axios";
import { trace, propagation, context } from "@opentelemetry/api";

axios.interceptors.request.use((config) => {
  const activeSpan = trace.getActiveSpan();
  if (activeSpan) {
    propagation.inject(
      trace.setSpan(context.active(), activeSpan),
      config.headers as Record<string, string>,
      { set: (carrier, key, value) => { carrier[key] = value; } }
    );
  }
  return config;
});

Without this propagation, your trace graph shows service A's span and service B's span as two separate, disconnected trees. You lose the "distributed" in distributed tracing.

An Incident, Walked Through

Theory is clean. Practice is messy. Here is what this stack looks like during a real incident.

3:17 AM. PagerDuty fires: checkout_error_rate > 1% for 5 minutes. You open the metrics dashboard. The checkout_total{status="error"} counter shows a spike starting at 3:12 AM. No recent deployments — the deploy log is clean. The error is not correlated with a bad release.

3:18 AM. You open the error tracker. The most recent errors all share the same pattern: PaymentGatewayTimeout: request to https://api.stripe.com timed out after 30000ms. You grab the trace_id from one of the errors.

3:19 AM. You paste the trace ID into Grafana Tempo. The trace shows: POST /api/checkout → checkout.process (31.2s) → checkout.processPayment (30.0s) → POST https://api.stripe.com/v1/charges (30.0s). The payment gateway is timing out. You check Stripe's status page — they show degraded performance in us-east-1.

3:21 AM. You know exactly what is broken and who is responsible. You add a note to the incident channel: "Stripe API degraded in us-east-1. Not our bug. Monitoring until resolved." Total time from alert to diagnosis: 4 minutes.

Without observability: you would have spent 4 minutes just tailing logs, 10 minutes grepping for "timeout," 15 minutes suspecting your own code, and 30 minutes before concluding it was Stripe. The difference is not the tools — it is the data.

Metrics: Counters, Histograms, and Gauges

Traces answer "why is this request slow?" Metrics answer "is the system healthy right now?" You need both. OpenTelemetry provides a metrics API, but for production Node.js, Prometheus with prom-client is more battle-tested and has better ecosystem support.

// metrics.ts
import { collectDefaultMetrics, Counter, Histogram, Gauge, register } from "prom-client";
import { Request, Response, NextFunction } from "express";

// Enable default Node.js metrics: heap size, event loop lag, open handles, etc.
collectDefaultMetrics({ prefix: "app_" });

// Business-level counters.
export const checkoutTotal = new Counter({
  name: "checkout_total",
  help: "Total number of checkout attempts",
  labelNames: ["status", "payment_method"],
});

// Histograms with meaningful latency buckets.
// Default buckets (5ms–10s) are useless for most APIs. Define your own.
export const checkoutDuration = new Histogram({
  name: "checkout_duration_seconds",
  help: "Checkout processing time in seconds",
  labelNames: ["status"],
  buckets: [0.1, 0.25, 0.5, 1, 2.5, 5, 10, 30],
});

// Gauges for point-in-time state.
export const activeCheckouts = new Gauge({
  name: "checkout_active",
  help: "Number of checkouts currently in progress",
});

// Express middleware: request rate + duration for every route.
export function metricsMiddleware(req: Request, res: Response, next: NextFunction) {
  const end = checkoutDuration.startTimer({ status: "started" });
  activeCheckouts.inc();

  res.on("finish", () => {
    activeCheckouts.dec();
    const status = res.statusCode < 400 ? "success" : "error";
    end({ status });
    checkoutTotal.inc({ status, payment_method: "unknown" });
  });

  next();
}

// Expose /metrics endpoint for Prometheus scraping.
export function metricsEndpoint(req: Request, res: Response) {
  res.set("Content-Type", register.contentType);
  register.metrics().then((data) => res.end(data));
}

Bucket design is the most common mistake in Prometheus histograms. The default buckets span from 5ms to 10 seconds in exponential steps — they are fine for HTTP request latency but useless for checkout processing (which might take 0.5s to 5s). Define buckets to match your actual performance profile: the bucket boundaries should surround your p50, p95, and p99 latencies.

The Missing Signal: Error Tracking

Logs, traces, and metrics are the three official pillars. In production, there is a fourth signal: errors. Not just rate — full context (stack trace, user ID, request payload, surrounding log lines, trace ID).

OpenTelemetry's recordException captures the error on the span, but spans are sampled. In high-traffic systems, you may sample only 10% of traces. If a rare error occurs in the 90% you discard, you will never see it.

The solution: a dedicated error tracker. Sentry is the industry default, but you can build a lightweight alternative that hooks into your existing logger:

// errorTracker.ts
import { logger } from "./logger";
import { trace } from "@opentelemetry/api";

interface ErrorContext {
  userId?: string;
  requestId?: string;
  body?: unknown;
  headers?: Record<string, string>;
}

const errors: Array<{ timestamp: string; error: Error; context: ErrorContext; traceId?: string }> = [];
const MAX_IN_MEMORY_ERRORS = 100;

export function captureError(error: Error, context: ErrorContext = {}) {
  const activeSpan = trace.getActiveSpan();
  const traceId = activeSpan?.spanContext().traceId;

  // Log with full context — your log aggregator indexes this.
  logger.error({ err: error, trace_id: traceId, ...context }, "application error");

  // Keep a rolling window for the /debug/errors admin endpoint.
  errors.push({ timestamp: new Date().toISOString(), error, context, traceId });
  if (errors.length > MAX_IN_MEMORY_ERRORS) {
    errors.shift();
  }

  // In production, also send to Sentry, Datadog, or your internal error service.
}

This dual-write pattern — structured log + in-memory buffer — gives you real-time access during incidents without depending on an external error tracking service that might also be having issues.

Wiring It All Together: A Production Entry Point

The key insight of ODD is that instrumentation is not a middleware you bolt on. It is the scaffolding your entire application runs on. Here is how a production entry point wires logging, tracing, and metrics before anything else:

// server.ts — the FIRST file Node.js loads
import "./tracing";        // Must be first: initializes OTel SDK
import express from "express";
import { logger } from "./logger";
import { requestContext } from "./middleware/requestContext";
import { metricsMiddleware, metricsEndpoint } from "./metrics";
import { checkoutRouter } from "./routes/checkout";

const app = express();

// Layer 1: Request-scoped logging — binds requestId/userId to every log line.
app.use(requestContext);

// Layer 2: Metrics for every route — latency + throughput + error rate.
app.use(metricsMiddleware);

// Layer 3: Business routes.
app.use("/api/checkout", checkoutRouter);

// Layer 4: Health check (excluded from tracing via HttpInstrumentation config).
app.get("/health", (_req, res) => res.json({ status: "ok" }));

// Layer 5: Metrics scrape endpoint for Prometheus.
app.get("/metrics", metricsEndpoint);

const PORT = parseInt(process.env.PORT ?? "3000", 10);
app.listen(PORT, () => {
  logger.info({ port: PORT }, "server started");
});

The ordering matters. Tracing must be initialized before Express (so OTel can patch http.createServer). Request context must run before metrics middleware (so logs carry request IDs). Metrics middleware must run before route handlers (so every request is counted).

Sampler Strategy: You Cannot Keep Everything

In a system processing 10,000 requests per second, full tracing generates roughly 50,000 spans per second (5 spans per request on average). Storing all of them costs thousands of dollars per month in trace storage.

OpenTelemetry supports three sampling strategies. Choose based on your traffic:

AlwaysOn (development): Keep every span. Use for local development and staging. Gives you complete visibility at the cost of 5-10% CPU overhead.

Probability (production, high traffic): Sample 10% of traces. This is the default recommendation. 10% of 10,000 req/s = 1,000 traces/s, which is manageable for most trace backends. The catch: rare errors that occur in fewer than 10% of requests will be missed.

Tail-based (production, high value): Sample 100% initially, then decide at the collector whether to keep each trace based on whether it contains errors or exceeds a latency threshold. This requires a tail-sampling processor in your OTel collector pipeline — more operational complexity, but zero blind spots for errors.

For most teams, the pragmatic choice is Probability at 10% with error tracking as a side channel. Use captureError to send all errors to Sentry (regardless of sampling decision), and use traces for latency analysis and dependency mapping.

The OpenTelemetry Collector: Your Telemetry Router

Your application emits telemetry. Something needs to receive it, process it, and route it to the right backend. That something is the OpenTelemetry Collector — a standalone binary that acts as a telemetry proxy between your application and your observability backends.

Without the collector, every service points directly at Jaeger, Prometheus, and Sentry. Change any backend, and you redeploy every service. With the collector, your services point at a single endpoint (localhost:4318), and the collector handles routing.

A minimal production configuration:

# otelcol-config.yaml — OpenTelemetry Collector configuration
receivers:
  otlp:
    protocols:
      http:
        endpoint: 0.0.0.0:4318
      grpc:
        endpoint: 0.0.0.0:4317

processors:
  # Drop health check and metrics scrape spans — they are noise.
  filter:
    spans:
      exclude:
        match_type: strict
        span_names:
          - GET /health
          - GET /metrics

  # Batch spans to reduce backend load.
  batch:
    timeout: 5s
    send_batch_size: 512

  # Add Kubernetes metadata to every span.
  k8sattributes:
    extract:
      metadata:
        - k8s.pod.name
        - k8s.namespace.name
        - k8s.deployment.name

  # Tail sampling: keep all errors regardless of probability.
  tail_sampling:
    policies:
      - name: keep-errors
        type: status_code
        status_code: { status_codes: [ERROR] }
      - name: probabilistic-normal
        type: probabilistic
        probabilistic: { sampling_percentage: 10 }

exporters:
  otlp/tempo:
    endpoint: tempo:4317
    tls:
      insecure: true
  prometheus:
    endpoint: 0.0.0.0:8889

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [filter, batch, k8sattributes, tail_sampling]
      exporters: [otlp/tempo]
    metrics:
      receivers: [otlp]
      processors: [batch, k8sattributes]
      exporters: [prometheus]

Three details in this config that save real money:

The filter processor drops health checks. In Kubernetes, a single deployment with 3 replicas probed every 10 seconds generates 25,920 health check spans per day. Filtering them reduces trace storage costs by 5-15%.
tail_sampling with keep-errors ensures you never lose error traces regardless of sampling rate. This is the operational equivalent of the captureError function — errors bypass sampling at the collector level.
k8sattributes tags every span with pod name, namespace, and deployment. When a specific pod is misbehaving, you can filter traces to that pod without instrumenting your code.

Run the collector as a sidecar in Kubernetes (same pod, different container) or as a DaemonSet on every node. The sidecar pattern adds ~50MB of memory per pod but eliminates network overhead — your app sends telemetry to localhost:4318 with sub-millisecond latency.

Decision Tree: Which Tool for Which Problem

When you are on call and something breaks, you should not be guessing which tool to check. The decision tree:

Is the service returning errors? → Check the error tracker (Sentry, or your /debug/errors endpoint). Read the stack trace, user ID, and request body.
Is the service slow? → Check traces. Find the slowest span in the flame graph. Is it the database query? The payment gateway? The serialization step?
Is the error rate spiking? → Check metrics. Look at checkout_total{status="error"} over time. Correlate with deployment timestamps.
What happened at 3:17 AM? → Check logs. Filter by trace_id from the error tracker, then read the surrounding log lines for that request.

If you try to use logs to figure out why a service is slow, you will spend an hour grep-ing timestamps. If you try to use metrics to debug a specific user's failed request, you will find nothing. Each signal answers a specific question. Build all three.

Production Checklist

Before you ship this to production, verify each item:

What You Should Skip

Not every application needs this stack. Here is the honest assessment:

Single-service monoliths with no downstream calls: Skip tracing. Structured logging (Pino) + error tracking (Sentry) is sufficient. Distributed tracing adds complexity with no benefit when there is nothing distributed.
Internal CLI tools: Skip everything. console.error is fine.
Static sites or CDN-hosted SPAs: Skip backend instrumentation. Use browser RUM (real user monitoring) instead — the signal is on the client, not the server.
Lambda functions with sub-100ms execution: Skip OpenTelemetry. The cold-start cost of initializing the OTel SDK (50-200ms) can double your latency. Use AWS X-Ray natively or structured logging only.

The Takeaway

Observability-driven development is not about installing more packages. It is about a mindset shift: you instrument during development, not after the outage. Every function you write should answer: "If this breaks in production at 3 AM, what will I need to know?"

The answer is usually: the inputs, the outputs, the duration, and the caller. If your code cannot answer those questions without a debugger, you have not instrumented it enough.

Start with Pino. Add prom-client. Wire OpenTelemetry last — it has the highest operational cost but also the highest payoff when you have more than one service. By the time your next PagerDuty alert fires, you will know exactly which span to click.

Code examples in this article use OpenTelemetry JS v1.x, Pino v9.x, and prom-client v15.x. All patterns are compatible with Node.js 20 LTS and 22.

chengyixu/observability-driven-nodejs-typescript-20260509.md

Select an option

No results found