nickmerwin

Production Outage Post-Mortem — 2026-04-08

TL;DR

Production was down ~2 hours (7:30–9:30 AM ET). No data loss.

Cause: ES license expired → web workers hung → thundering herd on recovery → exposed latent N+1-style COUNT queries in the asset show endpoint that couldn't survive 130 concurrent connections. DB pinned at 100% CPU. Compounded by ~100K queued Sidekiq jobs and a Kintzing API client hitting us at 140 req/s.

Fix: Shipped 5 targeted fixes during the incident — eliminated 6+ expensive COUNT queries per asset show (upload progress counts, contribution counts, requirement checks). These now use cached counter columns or are skipped entirely for done uploads and API traffic.

nickmerwin

Saved Meals — New Feature

Great news — we just built out the Saved Meals feature you asked about! Here's what you can do now:

Create a saved meal by telling the coach:

"Create a meal called Cold Cereal: 3/4 cup Bob's Old Country Style Muesli, 1/4 cup granola, 1/4 cup trail mix, 1/2 cup unsweetened hemp milk, 1/2 cup kefir, and 1 scoop whey protein powder"

The coach will look up the macros for each ingredient and save it as a reusable template.

nickmerwin

DataBody — Claude API Payload: Thread 877 (2026-02-27)

Last message sent at 2026-02-27 23:54 UTC (3:54 PM PST).
User: Apple private relay, timezone: America/Los_Angeles
This is the session showing the 1,357 (header) vs 1,170 (AI) calorie discrepancy.

---

Root Cause (discovered via post-mortem)

nickmerwin

FROM phusion/baseimage:0.9.11
# https://github.com/phusion/baseimage-docker
MAINTAINER Nick Merwin <[email protected]>

ENV HOME /root
RUN /etc/my_init.d/00_regen_ssh_host_keys.sh
CMD ["/sbin/my_init"]

# ==============================================================================
# install deps

nickmerwin

yo yo yo yo\n yo yo yo

nickmerwin

Keybase proof

I hereby claim:

• I am nickmerwin on github.
• I am mer (https://keybase.io/mer) on keybase.
• I have a public key whose fingerprint is 24AE 1631 875E 8746 1B75 23D7 449A B548 4E59 8A2B

To claim this, I am signing this object:

nickmerwin

module Sunspot
  module SessionProxy
    class TandemSessionProxy < AbstractSessionProxy
      
      attr_reader :search_session
      
      delegate :new_search, :search, :config,
                :new_more_like_this, :more_like_this,
                :delete_dirty, :delete_dirty?,
                :to => :search_session

nickmerwin

# Heroku S3 Database backup task
#   by Nick Merwin (Lemur Heavy Industries) 10.08.09
#   * dumps db to yaml, gzip's and sends to S3
# 
# Setup:
# 1) replace APP_NAME and BACKUP_BUCKET with your info
# 2) add config/s3.yml like so (same as Paperclip's): 
#    production:
#      access_key_id: ...
#      secret_access_key: ...