The Immortal Daemon: Why "Outlives Every Session" Becomes a Process Leak

A field guide to garbage-collecting the long-lived background processes you spawn — and why your "memory leak" is often nothing to do with memory.

TL;DR

A background daemon designed to "outlive every session" quietly became "outlive everything, forever." It accumulated orphaned copies (one per sleep/wake, port change, or false-death respawn) until a pile of them sat idle holding nothing.

The crucial misdirection: it was reported as a memory leak, but the heap was fine. The leak was at the process layer — whole OS processes that no language runtime will ever garbage-collect. The operating system reaps a process only when it exits, and this one had no condition under which it would ever choose to exit.

The rule: anything you spawn detached must be able to answer "under what condition do I die?" If the answer is "never," you have written a leak with extra steps.

The shape of the bug (generic)

You have a resource that should be shared by many short-lived clients: a port, a GPU context, a browser tab, a database connection pool, a file lock. Binding it per-client is wasteful or impossible, so you hoist it into one long-lived daemon that all clients connect to:

client A ─┐
client B ─┼──▶  shared daemon  ──▶  the scarce resource
client C ─┘     (owns the port,
                 outlives clients)

The daemon is auto-spawned on first need, detached from its parent, and intentionally outlives every client so the next client reuses it instead of paying cold-start. This is a good pattern. It is also a trap, because the design naturally grows three of the four lifecycle verbs and silently omits the fourth:

Verb	Usually implemented?	What it does
Spawn	✅	Start the daemon when absent
Dedupe	✅	A second daemon that loses the bind race exits (`EADDRINUSE` → exit 0)
Respawn	✅	Client treats "connection refused" as daemon-dead and starts a new one ("self-healing")
Self-terminate	❌	Die when no longer useful. The missing verb.

Each implemented verb is individually correct. The gap is the one nobody writes, because nothing fails when it's absent — the daemon just... stays.

Why it actually accumulates

A daemon that successfully acquired the resource has, in most naive implementations, only two exit paths:

A termination signal (SIGTERM/SIGINT) — but nothing in the system ever sends one.
Losing the initial acquisition race — which only matters in the first second of life.

So once a daemon is past second one, it is effectively immortal. Now watch how copies breed:

Sleep/wake. The laptop suspends. On resume, the OS has quietly torn down the daemon's listening socket, but the process is still alive with a now-dead handle keeping its event loop spinning. It is no longer serving anyone. The next client can't reach it → spawns a fresh daemon that grabs the freed port. Now there are two. The old one will never leave.
Configuration drift. A client started once with a different port/path (a debug run, an env override, a half-finished migration). That daemon bound a different resource and coexists forever with the canonical one. Cross-config duplicates never collide, so the dedupe guard never fires.
"Self-healing" respawn. This is the cruel one: the very mechanism meant to make crashes invisible manufactures orphans. Every time a client misjudges the daemon as dead (a slow health check, a transient hang) and respawns, the "dead" daemon that was merely napping is now a permanent orphan. Robustness against death created immortality.

None of these are exotic. Over a week of normal laptop use, they compound into a heap of identical processes — in the case that prompted this writeup, ~18 of them, each holding tens of megabytes, none reachable, none ever exiting.

Why "memory leak" sends you the wrong way

The symptom — RAM creeping up, the machine getting sluggish — reads as a classic heap leak. So you reach for the heap tools: snapshots, retained-size diffs, hunting an unbounded Map or a forgotten event listener.

You will find nothing, because the daemon's in-process state is bounded and correct. The leaked bytes are not in one process's heap; they are N whole processes, each with a perfectly healthy heap. No garbage collector — not the language's, not the runtime's — has any authority over an OS process. The only "GC" for processes is:

the process exiting (reaped by its parent / init), or
you, with kill.

If neither happens, the bytes stay. Going one layer too low (the heap) when the leak is one layer up (the process table) is the single biggest time-sink in this class of bug.

Heuristic: In any system with more than one process, when someone says "memory leak," run ps and look at the process count and PPID before you open a heap profiler. "Memory" is describing the symptom, not the layer.

How to see it (the evidence lives in runtime state, not source)

Source review will not reveal this bug — the code looks correct. The proof lives in the live process table and the logs.

Count and parentage. Orphaned daemons get re-parented to init/launchd (PPID 1). Many processes with the same command line and PPID 1 is the signature:

# How many copies are running, and who owns the resource right now?
ps -axww -o pid,ppid,etime,rss,command | grep '[m]y-daemon'
lsof -nP -iTCP:<port> -sTCP:LISTEN     # which ONE actually holds the port

If five processes match but only one holds the port, the other four are orphans doing nothing.

The log is the smoking gun. In the case behind this report, the daemon log showed two daemons that had bound different ports (a drift no code review would have predicted), and the "idempotent bind-race" guard's log line had fired zero times in the entire history — proving the dedupe path that everyone assumed was protecting them had literally never engaged. You cannot deduce that from the source; you read it from what the system actually did.

Heuristic: For lifecycle bugs, trust the runtime artifacts (process table, logs, pidfiles) over your reading of the code. The code describes intent; the log describes events.

The fix: give the daemon a death condition

The repair is not clever. It is simply writing the fourth verb. The daemon should periodically ask "am I still worth running?" and exit when the answer is no. Two conditions cover almost every case:

Superseded. A newer daemon has taken over. Detect it: the shared registration (pidfile / lock / registry key) now names a different, live process. If so, step aside immediately.
Idle. No client has touched the daemon for longer than an idle TTL, and the live keepalive (if any) is gone. The resource is no longer in use; exit and let the next client respawn a fresh one.

Pseudocode for a watchdog, language-agnostic:

every WATCHDOG_INTERVAL (e.g. 30s), low priority, never keeping the process alive on its own:

    registered = read_shared_registration()          # pidfile / lock / registry
    if registered.pid != my_pid and is_alive(registered.pid):
        exit("superseded")                            # a newer daemon owns it now

    busy = clients_connected() or work_in_flight() > 0
    idle_since = busy ? null : (idle_since ?? now())
    if IDLE_TTL > 0 and idle_since and now() - idle_since >= IDLE_TTL:
        exit("idle")

And make the watchdog itself not a reason to stay alive (e.g. an unref'd timer, a daemon thread, a low-priority tick). The genuine work — the open socket, the real connections — should be the only thing holding the process up. When the work is gone and the watchdog fires, the process is free to die.

Two implementation notes that matter:

Make the decision a pure function. should_reap(connected, in_flight, idle_since, now, ttl, registered_pid, my_pid, registered_alive) → {reap, reason}. Timing-sensitive lifecycle logic is miserable to test against real processes and clocks; a pure function unit-tests in microseconds and is where the subtle bugs hide (off-by-one on the TTL, supersede-vs-idle priority, a missing registration that shouldn't trigger anything).
Don't let cleanup clobber your successor. On exit, only remove the shared registration (pidfile/lock) if it still names you. A superseded daemon that blindly deletes the pidfile erases the new daemon's registration, breaking the very supersede-detection that's supposed to keep the system to one daemon.

Gotchas

The keepalive is what makes "idle" safe. If clients send periodic heartbeats while genuinely in use (e.g. a UI polling every 25s), then "idle" reliably means the user is actually gone, not the user is thinking. Without such a signal, you must define idleness carefully or you'll reap a daemon someone is mid-way through using. Set the TTL comfortably longer than the longest legitimate gap between uses.
Idle-exit must be paired with respawn. Reaping is only safe because clients already know how to start a daemon on demand. If your clients assume the daemon is always up, adding idle-exit turns one bug into another. Verify the spawn-on-absent path works before you enable self-termination.
"Self-healing" without "self-terminating" is a leak generator, not a safety feature. If you respawn on suspected death, you must also let the falsely-presumed-dead instance reap itself. Otherwise every health-check false positive is a permanent orphan.
Blind pkill is the wrong cleanup. A "kill all daemons" command frees the resource but also murders the live, in-use one and any active client sessions. Provide a targeted reaper that keeps the one currently holding the resource and removes only the orphans:
```
keep=$(lsof -ti tcp:<port> | head -1)
for pid in $(pgrep -f my-daemon); do
    [ "$pid" != "$keep" ] && kill "$pid"
done
```
Surface the count in your health/doctor tooling. "Exactly one daemon should exist" is an invariant; make a doctor check assert it and warn when violated. An invariant nobody monitors is an invariant that silently breaks.
Never blind-kill a process you don't own. Supersede-detection should only act on a registration whose pid you can prove is yours-lineage (it matches a pidfile you wrote). A foreign process squatting your port is a different problem — report it, don't kill -9 a stranger.
unref/daemon-thread the watchdog. If the watchdog timer itself keeps the event loop alive, an otherwise-idle process never becomes eligible to exit naturally and your "die when the real work is done" logic is defeated.

A checklist for any long-lived process you spawn

Before you ship a detached, auto-spawned, shared daemon, answer all four:

Spawn — how does it start when absent?
Dedupe — what happens when two try to start at once?
Respawn — how does a client recover if it died?
Self-terminate — under what condition does it choose to exit? (superseded? idle past TTL? parent gone?)

Plus the hygiene:

On exit, does it clean up its registration only if still its own?
Is the reaping decision a pure, unit-tested function?
Is there a targeted reap command (keep the live one) and a doctor check on the "exactly one" invariant?
Does the child exit when its parent dies (stdin EOF / broken pipe / PR_SET_PDEATHSIG-style guard)? Orphaned children leak the same way orphaned daemons do.

The one-liner

OS processes have no garbage collector. A long-lived process you spawn is memory you allocated by hand — and like any manual allocation, if you never write the free, it leaks. "Outlives every session" is not a lifetime; it's the absence of one. Give every daemon a reason to die.

htlin222/the-immortal-daemon.md

Select an option

No results found