Why can't I start a service?

0-service-start.md

Why can't I start a service?

• I can start a thread, if I want shared memory parallelism
• I can start a process, if I want shared storage parallelism
• I cannot start a service, if I want shared network parallelism

What would it take to add a start API?

Overly literal examples

Threads: shared memory parallelism

from threading import Thread

def task(d, k, v):
  d[k] = v

values = {}

threads = [
  Thread(target=task, args=(values, 1, 'hello')),
  Thread(target=task, args=(values, 2, 'world')),
]

for t in threads:
  t.start()

for t in threads:
  t.join()

assert values == {1: 'hello', 2: 'world'}

Processes: shared storage parallelism

from multiprocessing import Process
from pathlib import Path

def task(k, v):
  open(k, 'w+').write(v)

processes = [
  Process(target=task, args=('1.txt', 'hello')),
  Process(target=task, args=('2.txt', 'world')),
]

if __name__ == '__main__':
  assert Path('1.txt').exists() == False
  assert Path('2.txt').exists() == False

  for p in processes:
    p.start()

  for p in processes:
    p.join()

  assert open('1.txt', 'r').read() == 'hello'
  assert open('2.txt', 'r').read() == 'world'

Services: shared network parallelism

# Why can't I do this? 🤷🏾‍♂️

with Redis() as redis_service:
  services = [
    Container(
      image="redis",
      command="redis-cli", args=('-h', 'redis.local', 'SET', '1', 'hello'),
      links={'redis': redis_service}
    ),
    Container(
      image="redis",
      command="redis-cli", args=('-h', 'redis.local', 'SET', '2', 'world'),
      links={'redis': redis_service}
    ),
  ]
  
  redis_client = redis.Redis(host=redis_service.address)

  assert redis_client.get('1') == None
  assert redis_client.get('2') == None

  for s in services:
    s.start()

  for s in services:
    s.join()

  assert redis_client.get('1') == 'hello'
  assert redis_client.get('2') == 'world'

Map of the Land

The caternetes paper got me thinking, and so the question I'm asking at least recapitulates:

1. Spencer Baugh (catern)'s rsyscall
2. Plan9's cpu server
3. Erlang's OTP

In terms of ecosystems, there's really just containers and virtual machines; sorry jails and chroot/pledge/unveil.

Virtual Machines

Too high level. Or low-level, depending on your point-of-view. AFAICT no standard way to inherit (or pass)...

• Networking fabric
• Inputs / output / error streams
• Service links
• Resource contraints
• ... ?

Perhaps this is too quick of a dismissal; Firecracker works for Lambda/Fargate, Kata, and Fly.io.

Containers

• systemd's container interface (see also: systemd-nspawn / machinectl)
• Docker's Engine API as accessed via /var/run/docker.sock
• k8s pods can access the API ...
  • ... but wow, no. However, the kublet's API seems reasonable
• Both k8s and Docker front containerd's CRI as accessed via /var/run/containerd.sock

Define away big problems

Complexity is conserved; is cannot be removed or hidden, only rearranged. — Law of conservation of complexity

Internal Networking

This is a hard enough problem that there's a Container Networking Interface (CNI) standard!

IM-extrememly-HO, connections in and out of a service are fundamentally "links" between services.

Basically, services should only be able to talk with:

1. Their parent service and services it linked
2. Their child services

Idea: services connected and isolated via a mesh network ala tailscale

External networking

Flippantly: Tailscale Funnel.

Seriously: 🤷🏾‍♂️

Public egress/ingress is as hard as we let it be. CNI is all about interfaces, IPs, routes, and DNS.

Defer it. Interfaces are first-class capabilities exposed by a service host node ala Plan 9's ip device.

Storage

A capability that's divisable (range of total capacity for blocks, sub-directory for filesystems). Exposed by service host nodes or dedicated (network accessible) nodes.

Yes, of course Kubernetes has a Container Storage Interface (CSI).

Would it be the worst to put (non-ephemeral) storage under the network-accessible capability bucket? Yes, that's fine.

Memory

The currently running service inherits a larger available pool than it has reserved. That available portion is a divisible capability. Available >= Reserved >= In-Use.

Is there a way to simplify this model to Available >= In-Use? No— that would play hell with garbage collection. It also would require coordination across service host boundaries.

Capabilities

• Every service has an address (IPv6 ULA)
• A parent can access a child's address
• A parent can give a child access to any addresses the parent can access (inc. itself)

Further Research

• cpu command in Go, inspired by the Plan 9 cpu command
• Using Tailscale with Docker
• Fly.io: Working with the Machines API
  • Idea: FLAME but a multilingual supervision tree via WASM

init.go

package cmd

import (
	"context"
	"fmt"
	"time"

	"github.com/quad/flytree/service"
	"github.com/redis/go-redis/v9"
	"github.com/spf13/cobra"
	"github.com/thejerf/suture/v4"
	"golang.org/x/sync/errgroup"
)

func run(cmd *cobra.Command, args []string) error {
	ctx := cmd.Context()

	redis_ctx, redis_cancel := context.WithCancel(ctx)
	defer redis_cancel()
	redis_srv := service.Redis()
	go redis_srv.Serve(redis_ctx)

	services := []suture.Service{
		service.Container{
			Image:     "redis",
			Command:   "redis-cli",
			Arguments: []string{"-h", redis_srv.Address(), "SET", "1", "hello"},
		},
		service.Container{
			Image:     "redis",
			Command:   "redis-cli",
			Arguments: []string{"-h", redis_srv.Address(), "SET", "2", "world"},
		},
	}

	eg, eg_ctx := errgroup.WithContext(ctx)
	for _, srv := range services {
		srv := srv // 🙄

		eg.Go(func() error {
			timeout_ctx, timeout_cancel := context.WithTimeout(eg_ctx, 10*time.Second)
			defer timeout_cancel()
			return srv.Serve(timeout_ctx)
		})
	}

	if err := eg.Wait(); err != nil {
		panic(err)
	}

	redis_client := redis.NewClient(&redis.Options{
		Addr:     redis_srv.Address(),
		Password: "",
		DB:       0,
	})

	assertGet(redis_client, ctx, "1", "hello")
	assertGet(redis_client, ctx, "2", "world")

	return nil
}

func assertGet(c *redis.Client, ctx context.Context, key string, expected string) {
	if actual, err := c.Get(ctx, key).Result(); err != nil {
		panic(err)
	} else if actual != expected {
		panic(fmt.Errorf("[%v] %v != %v", key, actual, expected))
	}
}

var initCmd = &cobra.Command{
	Use:   "init",
	Short: "Top-level supervisor process",
	RunE:  run,
}

func init() {
	rootCmd.AddCommand(initCmd)
}