Skip to content

Instantly share code, notes, and snippets.

DSM monitor alert analysis — past 3 months

Date generated: 2026-04-29

Query and scope

Used pup against Datadog org2 and queried Event Management for monitor alert events matching:

source:alert team:data-streams-monitoring status:(error OR warn)
[
{
"distance": 0.018641136,
"date": "Apr 26, 2012, 5:13:15 PM"
},
{
"distance": 1.8330450400000002,
"date": "May 5, 2012, 8:51:47 PM"
},
{

Option 1: Axios

  • Our current style
  • Easier for the success path
  • Error path is untyped
let cells: Cell[] // = ...

Option 1: Axios

  • Our current style
  • Easier for the success path
  • Error path is untyped
let cells: Cell[] // = ...
type AlertLevel =
  | "crit"
  | "warn"
  | "info"
  | "ok"
  | "unknown"

type AlertGenerator = {
  id: string
  1. Low frame rate in large dashboards
  2. Long auto-refresh load times for cells with large queries
    • When a cell is refreshed, all data for the cell retrieved over the network, rather than just the delta of what has changed
    • Our CSV responses use gzip/DEFLATE compression, but we might be able to do better (by using compression techniques designed for time series)
  3. HTTP connection concurrency limits can slow down dashboard loading
  • We open 1+ HTTP connections for each cell in a dashboard, but are limited to ~6 concurrent connections over all. So if slow loading dashboard cells take the connections first, every other request must wait for those cells to complete before starting to load (including requests for fast cells)

Our current approach, all on the main thread:

  1. Fetch response
  2. Parse it
  3. Render it

Our goal is to avoid parsing data on the main thread, since it blocks higher-priority rendering and causes a drop in frame rate ("jank") as a result.

The approach we are currently implementing:

# Match on all types of devices but joysticks
#
# If you want to configure your devices, do not copy this file.
# Instead, use a config snippet that contains something like this:
#
# Section "InputClass"
# Identifier "something or other"
# MatchDriver "libinput"
#
# MatchIsTouchpad "on"
acl 2.2.53-1
adobe-source-code-pro-fonts 2.030ro+1.050it-5
adwaita-icon-theme 3.32.0-1
alacritty 0.2.9-1
alsa-lib 1.1.8-1
alsa-plugins 1.1.8-1
alsa-utils 1.1.8-1
aom 1.0.0-1
archlinux-keyring 20190123-2
argon2 20171227-3