diegoeche

widget.js: Why Widgets Are Slow

The Madness

widget.js is a 6,163-line, 260KB single file that every Stamped merchant loads on every page of their store.

It sometimes loads twice. We've observed this in the wild but the exact cause isn't fully diagnosed. There are at least two mechanisms that can inject it: the Shopify ScriptTag API (registered programmatically) and a manual <script> tag in theme.liquid. The code tries to detect duplicates before creating a new ScriptTag, but it never removes the other copy. There are runtime guards to prevent double-initialization, but by then the browser has already downloaded and parsed the full 107KB a second time. The guard is like locking the front door after the elephant is already in the living room.

It bundles jQuery 1.11 (2014 vintage). On load, it checks if the store already has jQuery. If not, it fetches it from Google's CDN as a blocking network request before anything else can happen. If the store does have jQuery, it hijacks th

diegoeche

The Varnish Cache Incident: A Post-Mortem in Three Facepalms

March 26, 2026 — prod namespace

---

Background

Varnish was getting OOM-killed. We increased memory limits. The OOMs stopped, but the cache hit ratio looked wrong. We went to investigate and found the dashboard itself was lying to us. What followed was a cascade of "wait, that's not right either" moments.

diegoeche

Why Was The Widget So Slow?

The Restaurant Analogy

Imagine a restaurant with 4 waiters (that's Kestrel's thread pool).

The old code worked like this: a waiter takes your order, walks to the kitchen, and stands there staring at the chef until the food is ready. They don't take any other orders while waiting. If 4 tables order at the same time, every waiter is standing in the kitchen doing nothing, and the 5th table? They can't even get a glass of water. That's thread starvation.

The fix: the waiter drops off the order, goes back to serve other tables, and picks up the food when the kitchen rings the bell. That's await.

diegoeche

MSSQL Index Bloat Analysis — Prod (2026-03-13)

Queries Used

1. Space bloat (via sys.allocation_units)

SELECT TOP 30
  OBJECT_NAME(p.object_id) AS TableName,
 i.name AS IndexName,

diegoeche

Widget Cache Layer — Design & Progress

Goal: Cache widget API responses (reviews, badges) with Varnish on NVMe SSD to reduce latency and backend load.

Architecture:

                    ┌───────────────────────────────────────────────┐
                    │      widget-cache node (m6id.2xlarge)         │

diegoeche

Battle Report: SQL Server Meltdown & Recovery — March 9, 2026

The Scene

Sunday morning. Merchants can't load review widgets. Dashboard inaccessible. Churn reported. The legacy .NET monolith (app-backend-api) running on a self-managed SQL Server (r6a.16xlarge — 64 vCPUs, 512 GB RAM) was completely unresponsive.

What followed was a 12-hour debugging marathon that uncovered five cascading failures, a year-old regression, and a SQL Server death spiral that required a force-kill to resolve.

---

diegoeche

k8s-mvp Environment Structure

How the k8s-mvp repository environments are structured using Kustomize overlays and GitOps.

Directory Layout

clusters/platform/
├── base/                          # Shared service definitions
│   ├── app-backend-api/

diegoeche

Models::Audience.first
=> #<FormsHub::Models::Audience @values={:id=>"ccdd57f2-d24f-4ff7-9356-b41f82756c94", :iid=>17, :account_id=>"0166cbcb-7827-4698-a267-fec98676db0b", :type=>"company", :name=>"Paid companies", :description=>nil, :facebook_account_id=>"act_10150428762524283", :facebook_audience_id=>"6103550209001", :company_segment_id=>"c6eddfbe-23ff-4287-b31c-9d3291c89da3", :person_segment_id=>nil, :company_exclusion_segment_id=>nil, :person_exclusion_segment_id=>nil, :role=>nil, :seniority=>nil, :search=>nil, :path=>nil, :status=>"active", :created_at=>2018-04-04 02:05:48 UTC, :updated_at=>2018-10-11 10:42:55 UTC, :last_sync_at=>nil, :archived_at=>nil}>

diegoeche

const axios = require('axios')
const sleep = require('sleep')
const base64 = require('base-64')

const appId = process.env["GEENY_APPLICATION_ID"]
const host = process.env["GEENY_APPLICATION_BROKER_URL"]

const brokerConfig = {
  appId: appId,
  messageTypeId: '54121087-14f1-4c2a-835f-117681618cc9', // incoming Develco messageType

diegoeche

const axios = require('axios')
const sleep = require('sleep')
const base64 = require('base-64')

const appId = process.env["GEENY_APPLICATION_ID"]
const host = process.env["GEENY_APPLICATION_BROKER_URL"]

const brokerConfig = {
  appId: appId,
  messageTypeId: '54121087-14f1-4c2a-835f-117681618cc9', // incoming Develco messageType