Websocket Deployments — Enemy of the State

enemy-of-the-state.md

Websocket Deployments — Enemy of the State

How do you deploy new code continuously into a system that needs to support stateful, uninterruptible computations?

Some workloads are inherently stateful

As web developers, we're used to valuing stateless computation, for good reason – reducing state makes everything easier. Servers (or cloud instances) hold no essential state, delegating any important data to databases like Postgres and Redis. This is a great architecture. Among other benefits, it makes zero-downtime deployments a breeze: to deploy a new code version, you simply spin up a set of new instances with the new git sha, add them to the load balancer, and kill the old instances after draining them of traffic (perhaps after 60 s).

However, some workloads don't fit this "short-duration request & response" mold. As an example, consider connecting OpenAI Realtime to Twilio's API, which requires you to:

• accept a websocket connection from Twilio
• listen to messages on the websocket, which contain bits of audio
• after making some optional changes, relay the bits of audio on to OpenAI Realtime, again via a websocket
• keep doing this for the duration of the call, which could last up to 30 minutes

Note that we're dealing with two pieces of essential state which can't be handed off to a database:

1. An incoming websocket connection from Twilio
2. An outgoing websocket connection to OpenAI

If we drop either of those connections due to a server restart, we inevitably drop the ongoing call and end up with a bad customer experience.

How do we square continuous deployment with stateful services?

We now have two desiderata:

1. We want to be able to deploy as often as necessary. In a busy monolith, we might deploy new code 10 times per hour, or more.

2. We need to keep a stateful process active for up to 30 minute which cannot be interrupted.

How can you make both of these possible at once? The problem, of course, is that when a new code version is deployed, the server is typically stopped and restarted, or a new cloud instance is created which cannot inherit open TCP connections from the old instance. I think this is an interesting engineering challenge.

I'd like to hear from you

Like any engineer, when I hear a problem like this, my mind starts coming up with possible solutions. (Would AWS lambda work for this use case? It might help with the incoming websocket, but I think it won't be able to maintain the outgoing websocket connection.) But instead of writing up my half-baked thoughts, I'm curious what you think. Let me know in the comments below!

Thanks @jbmoelker! On the "Decouple" approach, I can see how that would deal with incoming websocket connection, but the use case includes an outgoing websocket as well (to talk to OpenAI). Is it possible to maintain an outgoing connection in a stateless, serverless way?

I haven't done this exact thing. But I would think I would bind a Cloudflare Durable Object to both Twillio and OpenAI and have that work with a Cloudflare Worker for computation. So as long as that process is running that Worker would stay alive and so would the connections to Twillio and OpenAI. But I would really need to test this 😊

On Connection Draining, I think this could be a good approach. I'm mostly familiar with AWS load balancers and its ECS service, which has a deregistration_delay.timeout_seconds property.

deregistration_delay.timeout_seconds
The amount of time for Elastic Load Balancing to wait before deregistering a target. The range is 0–3600 seconds. The default value is 300 seconds.

So the idea would be

• set a maximum number of minutes for a process (e.g. 30 min as maximum call length)
• set the load balancer to continue relaying traffic on the tcp connection for up to that timeout