Effect workflows entity durability patterns and comparison to temporal

entity-durability-patterns.md

Entity Durability Patterns in Effect Cluster

Understanding how state survives failures - and how to test it.

The Two Durability Concerns

When an entity "dies" (runner crashes, shard reassigned), there are two separate things to worry about:

Concern	Question	Effect Cluster Solution
Message durability	Did the request survive?	MessageStorage persists pending messages
State durability	Did the entity's in-memory state survive?	You must handle this yourself

This is different from Temporal, where workflow local variables are automatically tracked via event history replay.

What MessageStorage Actually Does

┌─────────────────────────────────────────────────────────────┐
│                    Message Lifecycle                         │
├─────────────────────────────────────────────────────────────┤
│  1. Message arrives      → Stored in MessageStorage         │
│  2. Entity processes     → Handler runs, side effects happen│
│  3. Response stored      → Message marked complete          │
└─────────────────────────────────────────────────────────────┘

If entity dies during step 2:

• Message is still in storage (step 1 completed)
• Response not stored yet (step 3 didn't happen)
• Message will be redelivered to new entity instance

But the entity's in-memory state is lost. The new instance starts fresh.

The Erlang Mental Model

Effect Cluster entities are like Erlang processes:

┌──────────────────────────────────────────────────────────────┐
│  Erlang/OTP Pattern          │  Effect Cluster Equivalent   │
├──────────────────────────────────────────────────────────────┤
│  Process state = RAM         │  Entity state = RAM          │
│  Process crash = state lost  │  Entity crash = state lost   │
│  Supervisor restarts process │  Sharding restarts entity    │
│  Use mnesia/ETS for durable  │  Use DB for durable state    │
│  "Let it crash" philosophy   │  Typed errors + recovery     │
└──────────────────────────────────────────────────────────────┘

Key insight: Processes/entities are cheap and ephemeral. Important state belongs in durable storage.

Pattern 1: Volatile State (Cache/Rate Limiter)

When state loss is acceptable:

const RateLimiter = Entity.make("RateLimiter", [
  Rpc.make("Check", { success: Schema.Struct({ allowed: Schema.Boolean }) })
])

const RateLimiterLive = RateLimiter.toLayer(Effect.gen(function*() {
  // Volatile state - reset on restart is fine
  let count = 0
  let windowStart = yield* Clock.currentTimeMillis

  return {
    Check: () => Effect.gen(function*() {
      const now = yield* Clock.currentTimeMillis
      if (now - windowStart > 60_000) {
        count = 0
        windowStart = now
      }
      count++
      return { allowed: count <= 100 }
    })
  }
}))

Testing volatile state:

it.effect("resets on entity restart", () => Effect.gen(function*() {
  const client = yield* RateLimiter.client
  const limiter = client("api-key-1")

  // Use up some quota
  yield* limiter.Check()
  yield* limiter.Check()

  // Simulate entity restart by killing the shard
  yield* Sharding.terminateEntity(EntityAddress.make("RateLimiter", "api-key-1"))

  // New entity instance - count should be 0 again
  const result = yield* limiter.Check()
  expect(result.allowed).toBe(true)  // Fresh start
}))

Pattern 2: External State (Database-Backed)

For business-critical state that must survive failures:

const Order = Entity.make("Order", [
  Rpc.make("Create", {
    payload: Schema.Struct({ customerId: Schema.String, items: Schema.Array(Item) }),
    success: OrderState
  }),
  Rpc.make("ChargePayment", { payload: Schema.Number, success: PaymentResult }),
  Rpc.make("GetState", { success: OrderState })
])

const OrderLive = Order.toLayer(Effect.gen(function*() {
  const db = yield* Database
  const entityId = yield* Entity.currentId

  // Load existing state on entity creation
  let order = yield* db.orders.findById(entityId).pipe(
    Effect.orElseSucceed(() => null as OrderState | null)
  )

  return {
    Create: ({ payload }) => Effect.gen(function*() {
      order = {
        id: entityId,
        customerId: payload.customerId,
        items: payload.items,
        status: 'created',
        createdAt: yield* Clock.currentTimeMillis
      }
      yield* db.orders.upsert(entityId, order)  // Persist immediately
      return order
    }),

    ChargePayment: ({ payload: amount }) => Effect.gen(function*() {
      if (!order) {
        return yield* Effect.fail(new OrderNotFound({ entityId }))
      }

      // Idempotency: already charged?
      if (order.paymentId) {
        return { alreadyCharged: true, paymentId: order.paymentId }
      }

      // Charge with idempotency key (safe to retry)
      const result = yield* Stripe.charge({
        customerId: order.customerId,
        amount,
        idempotencyKey: `order-${entityId}-payment`
      })

      // Update state and persist
      order = { ...order, paymentId: result.id, status: 'paid' }
      yield* db.orders.upsert(entityId, order)

      return result
    }),

    GetState: () => Effect.gen(function*() {
      if (!order) {
        return yield* Effect.fail(new OrderNotFound({ entityId }))
      }
      return order
    })
  }
}))

Testing state persistence across restarts:

describe("Order entity durability", () => {
  it.effect("survives entity restart", () => Effect.gen(function*() {
    const orderClient = yield* Order.client
    const order = orderClient("order-123")

    // Create order
    yield* order.Create({
      customerId: "cust-1",
      items: [{ sku: "WIDGET", quantity: 2 }]
    })

    // Charge payment
    yield* order.ChargePayment(99.99)

    // Verify state before restart
    const stateBefore = yield* order.GetState()
    expect(stateBefore.status).toBe("paid")
    expect(stateBefore.paymentId).toBeDefined()

    // Simulate catastrophic failure - entity dies
    yield* Sharding.terminateEntity(EntityAddress.make("Order", "order-123"))

    // Entity restarts, loads from DB
    const stateAfter = yield* order.GetState()

    // State survived!
    expect(stateAfter.status).toBe("paid")
    expect(stateAfter.paymentId).toBe(stateBefore.paymentId)
  }))

  it.effect("handles duplicate charge attempts (idempotency)", () => Effect.gen(function*() {
    const orderClient = yield* Order.client
    const order = orderClient("order-456")

    yield* order.Create({ customerId: "cust-2", items: [] })

    // First charge succeeds
    const result1 = yield* order.ChargePayment(50.00)
    expect(result1.alreadyCharged).toBeUndefined()

    // Simulate crash mid-response (message will be redelivered)
    yield* Sharding.terminateEntity(EntityAddress.make("Order", "order-456"))

    // Second charge attempt (retry after crash)
    const result2 = yield* order.ChargePayment(50.00)

    // Idempotency: returns cached result, doesn't double-charge
    expect(result2.alreadyCharged).toBe(true)
    expect(result2.paymentId).toBe(result1.paymentId)
  }))
})

Pattern 3: Event Sourcing (Full Replay)

Store all messages as events, replay from beginning to rebuild state:

const Account = Entity.make("Account", [
  Rpc.make("Deposit", { payload: Schema.Number, success: Schema.Number }),
  Rpc.make("Withdraw", { payload: Schema.Number, success: Schema.Number }),
  Rpc.make("GetBalance", { success: Schema.Number })
])

const AccountLive = Account.toLayer(Effect.gen(function*() {
  const eventStore = yield* EventStore
  const entityId = yield* Entity.currentId

  // Replay all historical events to rebuild state
  const events = yield* eventStore.getEvents(entityId)
  let balance = events.reduce((acc, event) => {
    switch (event.type) {
      case 'Deposited': return acc + event.amount
      case 'Withdrawn': return acc - event.amount
      default: return acc
    }
  }, 0)

  return {
    Deposit: ({ payload: amount }) => Effect.gen(function*() {
      yield* eventStore.append(entityId, { type: 'Deposited', amount })
      balance += amount
      return balance
    }),

    Withdraw: ({ payload: amount }) => Effect.gen(function*() {
      if (balance < amount) {
        return yield* Effect.fail(new InsufficientFunds({ balance, requested: amount }))
      }
      yield* eventStore.append(entityId, { type: 'Withdrawn', amount })
      balance -= amount
      return balance
    }),

    GetBalance: () => Effect.succeed(balance)
  }
}))

Testing event sourcing:

describe("Account event sourcing", () => {
  it.effect("rebuilds state from event history", () => Effect.gen(function*() {
    const accountClient = yield* Account.client
    const account = accountClient("acct-789")

    // Perform operations
    yield* account.Deposit(100)
    yield* account.Withdraw(30)
    yield* account.Deposit(50)

    const balanceBefore = yield* account.GetBalance()
    expect(balanceBefore).toBe(120)  // 100 - 30 + 50

    // Kill entity
    yield* Sharding.terminateEntity(EntityAddress.make("Account", "acct-789"))

    // New entity replays events
    const balanceAfter = yield* account.GetBalance()
    expect(balanceAfter).toBe(120)  // Same! Rebuilt from events
  }))

  it.effect("event history is append-only", () => Effect.gen(function*() {
    const eventStore = yield* EventStore
    const accountClient = yield* Account.client
    const account = accountClient("acct-audit")

    yield* account.Deposit(100)
    yield* account.Withdraw(25)

    // Verify audit trail
    const events = yield* eventStore.getEvents("acct-audit")
    expect(events).toEqual([
      { type: 'Deposited', amount: 100 },
      { type: 'Withdrawn', amount: 25 }
    ])
  }))
})

When Entities Don't Die: Effect Error Handling

Important nuance: application errors don't kill entities.

ChargePayment: ({ payload: amount }) => Effect.gen(function*() {
  const result = yield* Stripe.charge(order!.customerId, amount).pipe(
    Effect.catchTag("PaymentDeclined", (err) =>
      // Error caught - entity keeps running!
      // State unchanged, error returned to caller
      Effect.fail(new PaymentFailed({ reason: err.message }))
    )
  )

  // Only reach here on success
  order!.paymentId = result.id
  return result
})

Entities only truly die when:

• Runner process crashes (OOM, kill -9, hardware failure)
• Shard reassigned (runner leaves cluster)
• Explicit termination (testing, shutdown)

Testing error handling (entity survives):

it.effect("entity survives payment failure", () => Effect.gen(function*() {
  const orderClient = yield* Order.client
  const order = orderClient("order-error-test")

  yield* order.Create({ customerId: "bad-card-customer", items: [] })

  // Payment fails
  const result = yield* order.ChargePayment(100).pipe(
    Effect.either
  )
  expect(Either.isLeft(result)).toBe(true)

  // Entity still alive, state intact
  const state = yield* order.GetState()
  expect(state.status).toBe("created")  // Not changed
  expect(state.paymentId).toBeUndefined()  // No payment recorded

  // Can retry with different amount or fix card
  // Entity didn't restart, in-memory state preserved
}))

Comparison: Temporal vs Effect Cluster Durability

Scenario	Temporal	Effect Cluster
Worker dies mid-activity	Activity retries automatically	Message retries with MessageStorage
Worker dies between steps	Workflow resumes at last completed step	Entity restarts, must load state from DB
Application error	Workflow handles typed error	Entity handles typed error, keeps running
State persistence	Automatic via event history	Manual via DB, event store, or snapshots
Replay mechanism	Re-run workflow, skip completed activities	Re-process pending messages OR replay events

Testing Checklist for Durable Entities

describe("Entity Durability Checklist", () => {
  // 1. State survives restart
  it.effect("persists state to storage")
  it.effect("loads state on startup")
  it.effect("survives entity termination")

  // 2. Operations are idempotent
  it.effect("handles duplicate requests safely")
  it.effect("uses idempotency keys for external calls")

  // 3. Error handling doesn't corrupt state
  it.effect("rolls back on failure")
  it.effect("entity survives application errors")

  // 4. Concurrent access is safe
  it.effect("handles concurrent messages correctly")
  it.effect("maintains consistency under load")
})

Summary

Pattern	When to Use	State Location	Complexity
Volatile	Caches, rate limiters	Memory only	Simple
External DB	Business entities	Database	Medium
Event Sourcing	Audit trails, CQRS	Event store	Complex

The key insight from Erlang applies here:

Entities are ephemeral. Design for recovery. Keep important state in durable storage.

temporal-to-effect-cluster-mental-model.md

From Temporal Activities to Effect Cluster Entities

A guide for developers familiar with Temporal who want to understand how Effect Cluster's sharding model changes the way you design distributed systems.

The Core Mental Shift

Temporal	Effect Cluster
Activities are stateless functions	Entities are stateful actors
Any worker can run any activity	Entity X always runs on one specific runner
State lives in database	State can live in entity memory
Worker pools are interchangeable	Entity locality is guaranteed

Concrete Examples

Example 1: Order Processing

Temporal Approach (Activity-based):

// Activities - stateless, any worker can execute
const activities = {
  async chargePayment(orderId: string, amount: number) {
    const order = await db.orders.findById(orderId)  // Load state
    const result = await stripe.charge(order.customerId, amount)
    await db.orders.update(orderId, { paymentId: result.id })  // Save state
    return result
  },

  async reserveInventory(orderId: string) {
    const order = await db.orders.findById(orderId)  // Load state again
    for (const item of order.items) {
      await db.inventory.decrement(item.sku, item.quantity)
    }
    await db.orders.update(orderId, { status: 'reserved' })  // Save state
  },

  async sendConfirmation(orderId: string) {
    const order = await db.orders.findById(orderId)  // Load state yet again
    await email.send(order.customerEmail, 'Order confirmed!')
  }
}

// Workflow orchestrates activities
async function orderWorkflow(orderId: string) {
  await activities.chargePayment(orderId, 99.99)
  await activities.reserveInventory(orderId)
  await activities.sendConfirmation(orderId)
}

Key characteristics:

• Each activity loads order from database (no shared memory)
• Any worker in the pool can execute any activity
• State consistency via database transactions
• Activities are pure functions with side effects

Effect Cluster Approach (Entity-based):

// Define the Order entity protocol
const Order = Entity.make("Order", [
  Rpc.make("Create", {
    payload: Schema.Struct({ customerId: Schema.String, items: Schema.Array(Item) }),
    success: Schema.String
  }),
  Rpc.make("ChargePayment", { payload: Schema.Number, success: PaymentResult }),
  Rpc.make("ReserveInventory", { success: Schema.Boolean }),
  Rpc.make("SendConfirmation", { success: Schema.Boolean })
])

// Entity implementation - stateful, long-lived
const OrderLive = Order.toLayer(Effect.gen(function*() {
  // State lives IN the entity - not fetched each time
  let order: OrderState | null = null

  return {
    Create: ({ payload }) => Effect.gen(function*() {
      order = {
        customerId: payload.customerId,
        items: payload.items,
        status: 'created'
      }
      return "created"
    }),

    ChargePayment: ({ payload: amount }) => Effect.gen(function*() {
      // order is already in memory - no database fetch!
      const result = yield* Stripe.charge(order!.customerId, amount)
      order!.paymentId = result.id
      order!.status = 'paid'
      return result
    }),

    ReserveInventory: () => Effect.gen(function*() {
      // Still have order in memory
      for (const item of order!.items) {
        yield* Inventory.reserve(item.sku, item.quantity)
      }
      order!.status = 'reserved'
      return true
    }),

    SendConfirmation: () => Effect.gen(function*() {
      yield* Email.send(order!.customerEmail, 'Order confirmed!')
      order!.status = 'confirmed'
      return true
    })
  }
}))

// Client sends messages to specific entity instance
const orderClient = yield* Order.client
const order123 = orderClient("order-123")  // All messages go to same entity instance

yield* order123.Create({ customerId: "cust-1", items: [...] })
yield* order123.ChargePayment(99.99)
yield* order123.ReserveInventory()
yield* order123.SendConfirmation()

Key differences:

• Entity holds state in memory across all operations
• All messages to order-123 route to the same runner (entity locality)
• No repeated database lookups for the same order
• Entity is an actor that processes messages sequentially

---

Example 2: Rate Limiting

Temporal Approach:

// Need external coordination for rate limiting
const activities = {
  async checkRateLimit(apiKey: string) {
    // Must use Redis or similar for coordination
    const count = await redis.incr(`ratelimit:${apiKey}`)
    await redis.expire(`ratelimit:${apiKey}`, 60)

    if (count > 100) {
      throw new RateLimitExceeded()
    }
    return { remaining: 100 - count }
  },

  async processRequest(apiKey: string, request: Request) {
    await activities.checkRateLimit(apiKey)
    // ... process
  }
}

Problem: Every rate limit check hits Redis. At high scale, Redis becomes a bottleneck.

Effect Cluster Approach:

const RateLimiter = Entity.make("RateLimiter", [
  Rpc.make("CheckLimit", { success: Schema.Struct({ allowed: Schema.Boolean, remaining: Schema.Number }) })
])

const RateLimiterLive = RateLimiter.toLayer(Effect.gen(function*() {
  // State is IN MEMORY - no Redis needed!
  let count = 0
  let windowStart = Date.now()
  const limit = 100
  const windowMs = 60_000

  return {
    CheckLimit: () => Effect.gen(function*() {
      const now = yield* Clock.currentTimeMillis

      // Reset window if needed
      if (now - windowStart > windowMs) {
        count = 0
        windowStart = now
      }

      count++
      return {
        allowed: count <= limit,
        remaining: Math.max(0, limit - count)
      }
    })
  }
}))

// All requests for api-key-123 go to SAME entity instance
const limiterClient = yield* RateLimiter.client
const limiter = limiterClient("api-key-123")

const { allowed } = yield* limiter.CheckLimit()

Why this works:

• Entity ID = API key
• All requests for same API key → same shard → same runner → same entity instance
• Rate limit state is in memory, no external coordination
• Horizontally scalable: different API keys go to different runners

---

Example 3: User Session / Shopping Cart

Temporal Approach:

const activities = {
  async addToCart(userId: string, item: Item) {
    // Load cart from database
    const cart = await db.carts.findById(userId) || { items: [] }
    cart.items.push(item)
    await db.carts.upsert(userId, cart)
    return cart
  },

  async removeFromCart(userId: string, itemId: string) {
    const cart = await db.carts.findById(userId)
    cart.items = cart.items.filter(i => i.id !== itemId)
    await db.carts.upsert(userId, cart)
    return cart
  },

  async getCart(userId: string) {
    return await db.carts.findById(userId)
  }
}

Every operation hits database. For high-traffic users, this creates database load.

Effect Cluster Approach:

const ShoppingCart = Entity.make("ShoppingCart", [
  Rpc.make("Add", { payload: Item, success: Cart }),
  Rpc.make("Remove", { payload: Schema.String, success: Cart }),
  Rpc.make("Get", { success: Cart })
])

const ShoppingCartLive = ShoppingCart.toLayer(Effect.gen(function*() {
  // Cart state lives in entity memory
  const items: Item[] = []

  return {
    Add: ({ payload: item }) => Effect.sync(() => {
      items.push(item)
      return { items }
    }),

    Remove: ({ payload: itemId }) => Effect.sync(() => {
      const idx = items.findIndex(i => i.id === itemId)
      if (idx >= 0) items.splice(idx, 1)
      return { items }
    }),

    Get: () => Effect.sync(() => ({ items }))
  }
}))

User alice always routes to the same entity instance - her cart is in memory, zero database reads for subsequent operations.

---

Key Considerations When Migrating

1. Entity Granularity

Question: What should be an entity?

• Too coarse (one entity for all orders): Bottleneck, no parallelism
• Too fine (entity per order line item): Overhead, harder to maintain consistency
• Just right (entity per order): Natural consistency boundary

Rule of thumb: Entity = thing that needs consistent state across operations

2. State Persistence

In Temporal, state is always in the database. In Effect Cluster:

// Option A: Volatile state (lost on crash)
// Good for: caches, rate limiters, temporary aggregations

// Option B: Persisted messages (state reconstructed from message log)
// Good for: important business entities that need durability

With MessageStorage, Effect Cluster replays messages after restart to reconstruct state.

3. Cross-Entity Coordination

Temporal: Workflow can call multiple activities, each potentially on different workers.

Effect Cluster: Each entity is independent. For cross-entity operations:

// DON'T try to call another entity from within an entity handler
// DO coordinate from the client/orchestration layer

const orderClient = yield* Order.client
const inventoryClient = yield* Inventory.client

// Client orchestrates across entities
yield* orderClient("order-123").ReservePayment()
yield* inventoryClient("sku-456").DecrementStock(5)
yield* orderClient("order-123").ConfirmReservation()

4. Idempotency

Temporal: Built-in via workflow execution ID and activity deduplication.

Effect Cluster: Built-in when using MessageStorage - same message ID won't be processed twice.

// Messages have unique IDs, storage tracks processed messages
// Retries are safe - already-processed messages return cached result

5. Scaling Characteristics

Scenario	Temporal	Effect Cluster
Add more workers	All workers share all work	Shards rebalance, ~1/N entities move
Hot entity (one user doing lots)	Fine - activities distribute	Bottleneck on single runner
Many entities, even load	Good	Excellent (automatic distribution)

Hot entity mitigation: Split into sub-entities or add caching layer.

---

When to Use Which Model

Temporal (Activity-based) is better when:

• Work is truly stateless or state is naturally in database
• You need workflow-level orchestration with complex control flow
• Activities are long-running and independently retriable
• You want activity-level visibility in the Temporal UI

Effect Cluster (Entity-based) is better when:

• You have natural entity boundaries (users, orders, accounts)
• Multiple operations on same entity should share in-memory state
• You want to avoid database round-trips for hot entities
• Rate limiting or coordination per-entity is needed
• You're building event-sourced or actor-model systems

---

Summary

The mental model shift:

Temporal:  "I have functions (activities) that workers execute"
           State → Database → Activity reads it → Activity writes it

Effect:    "I have actors (entities) that receive messages"
           Messages → Entity (keeps state in memory) → Responds

Both are valid distributed computing models. Temporal is more like a distributed function executor. Effect Cluster is more like a distributed actor system with automatic sharding.