This document explores design options for a new parallel execution feature in Dagger, enabling functions to be automatically split and executed across multiple workers.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| #!/bin/bash | |
| set -ex | |
| update() { | |
| mkdir -p ~/bin | |
| curl -L -o ~/bin/dev.sh https://gist.githubusercontent.com/shykes/7bf0704be14a0a9288d7dfe98617c208/raw/dev.sh | |
| chmod +x ~/bin/dev.sh | |
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| registryURL: string | |
| runRegistry: docker.#Run & { | |
| args: ["registry", "-h", registryURL] // bla bla | |
| // Wait for registry port to be available | |
| hooks: post: waitRegistry: { | |
| args: """ | |
| while true; do | |
| curl -L -S "localhost:foo" && exit | |
| sleep 1 |
Help me plan a new "collaborative parallel execution" feature in dagger. The general idea:
- Dagger functions can be annotated with a new directive @parallel (+parallel in Go etc) which indicates to the engine that this function can be executed in parallel
- The dimensions of paralellism are controlled by arguments. Any argument to a parallel function that is a list, is interpreted by the engine as a dimension of parallelism. The contract is that when a client invokes a paralle function, the engine may split that "frontend" invocation into N "backend" invocations, with each invocation receiving a slice of the frontend list argument. The function implementor is responsible for guarranteeing correct behavior of the function regardless of the splitting.
- Only return types that can be reliably merged in any order (I don't know the fancy terminology) is allowed. Example: Directory is allowed because you can always merge N directories together. Chan
Dagger Design: Part 1 - Workspaces and Modules
This proposal is part 1 of an multi-part proposal to simplify the "design knot" - an interconnected set of design and ux problems blocking implementation of a wide range of features and improvements
Builds on Part 1: Workspaces and Modules
Builds on Part 1: Module vs. Workspace and Part 2: Workspace API