Host provisioning tool config

config.toml

# Outline of a TOML-based DSL for configuring hosts. For now, we assume we execute these "on the box".

# Imports are processed first, and given the correct precedence as if they were passed on the command
# line. If an import uses a resolver
imports = [
   "/path/to/local/something.toml",
   "bucket1://some/remote.toml"
]

# Bucket is a resolver type. Configure a bucket called bucket1.
# We require the right "secret" env vars to be set to access private buckets.
[bucket.bucket1]
url = "nyc3.digitaloceanspaces.com"
name = "coleman"

# A bucket we can reference as "bucket2"
[bucket.bucket2]
url = "nyc3.digitaloceanspaces.com"
name = "coleman2"


# Declarative state that can be applied.
[[user]]
name = "coleman"
home = "/home/coleman"
groups = ["wheel", "something"]
ssh_public_key = "bucket1://ssh-keys/coleman.pub"
ssh_private_key = "bucket2://ssh-keys/coleman"

# make another user
[[user]]
name = "lilb"
home = "/home/lilb"
groups = ["wheel", "admin"]
ssh_public_key = "bucket1://ssh-keys/lilb.pub"
ssh_private_key = "bucket2://ssh-keys/lilb"

[[group]]
name = "admin"

[[group]]
name = "something"

[[file]]
src = "bucket1://path/to/website.service"
dest = "/usr/lib/systemd/system/website.service"
perms = 644
user = "root"
group = "root"

# Add a systemd service, enable it, and make sure it is running. Expects a unit file to be on disk.
[[service]]
name = "website"
enabled = true
running = true

# Run some command.
[[exec]]
name = "identifier" # need a name for exec types, just to provide a ref to allow overrides?
cmd = "rm -rf /"

usage.md

Overview

Usage: local

Set the appropriate environment variables and pass a sequence of configs to the tool.

$ hpt config1.toml config2.toml

Types

Resolvers (aka data sources)

Resolvers are things like S3 buckets that we can "get" data or files from. They can be referenced by name given a string with this structure (NOTE: this might change):

{name}://{path}

An example reference to a resolver is "bucket1://ssh-keys/coleman.pub", which will point to the bucket declared like this:

[bucket.bucket1]
url = "nyc3.digitaloceanspaces.com"
name = "coleman"

Compiler phases

• Pass 1: build a list of resolver names; can fail if there is a bad reference
• Pass 1.5: if imports reference a remote config, fetch the config and put it in the right place; goto Pass 1
• Pass 2: merge all configs into a final data structure that can be executed; apply overrides

Execution precedence by type (defunct)

---

UPDATE There is no "natural precedence", and a purely declarative config for host configuration seems difficult without embedding an unacceptable amount of complexity/options into the block-level object-ish APIs (like [[user]]). Consider the example of changing the user's default shell from bash to ion shell. This means we need to do a clone and a build (via some [[exec]] for rustup and cargo) to get the ion binary onto the system, and we need to edit /etc/shells to tell the system that this non-standard shell is available. Only then can we create the user, and easily assign the shell at the time of user creation.

While in this particular example we could accomplish changing the shell after the clone and after the installation of rust and after the cargo build/install of ion with chsh -s /usr/local/bin/ion username in an [[exec]] block, this means users will have to have a LOT of [[exec]] blocks that refer back to other parts of the config. Our custom shell example would look (something) like this.

# The fantasy of natural precedence
[[user]]
name = "someone"

# get ion source
[[clone]]
url = "https://github.com/redox-os/ion"
dest = "/opt/ion"

# get rust 
[[exec]]
script = "curl https://sh.rustup.rs -sSf | sh"

[[exec]]
script = "cargo build --release"
cwd = "/opt/ion"

[[exec]]
script = "echo \"/opt/ion/target/linux/bin/ion\" >> /etc/shells"

[[exec]]
script = "chsh -s /usr/local/bin/ion someone"

If we embrace natural precedence, we can't expect any [[exec]] block to run before the [[user]] block. But, as it turns out, being able to rely on your own custom order lets you do this.

# instead, config blocks execute in order

# get ion source
[[clone]]
url = "https://github.com/redox-os/ion"
dest = "/opt/ion"

# get rust 
[[exec]]
script = "curl https://sh.rustup.rs -sSf | sh"

[[exec]]
script = "cargo build --release"
cwd = "/opt/ion"

[[exec]]
script = "echo \"/opt/ion/target/linux/bin/ion\" >> /etc/shells"

[[user]]
name = "someone"
shell = "/opt/ion/target/linux/bin/ion"

The second version has one less instruction. It also locates the user's shell configuration inside the user block itself, which is cleaner.

This is all to say that host configuration needs script or procedure-like behavior. We need to be able to make state A happen before state B, regardless of the state's type. This really starts to matter when making lower level changes to the system, like installing custom shells.

---

Since we want to support merging configs, we need to have a default ordering of types that will be applied. Within each type, each "apply" will be run in the order it appears in the config, unless it is subsequently overridden.

We don't have resolvers here because they are not "applied", but are instead merely instantiated during our execution phase.

The gambit here is that unix-ish systems have a "natural" precedence. For instance, groups and users are applied before files, because files can be owned by those users. And systemd services are applied after files, because they require a file (the "unit file") to exist as a service at all.

1. group
2. user
3. files
4. service
5. exec

Ansible does this differently. It executes things in the other they appear in a playbook, except in the case of "dependent execution", iirc. Dependent/conditional execution is not something we will support right away, but we should be able to add it later by adding attributes to our applyables.

In any case, if you're relying on something like this, you're probably doing something hacky totally normal we should support?

Can't do that in toml, fwiw.

You're definitely right that giving up a natural ordering means we lose "merging" configs. In practical terms, this doesn't lose us too much.

But really we'll just need to kick the tires and see how it all shakes out. I do want to install custom shells, custom builds of tmux and neovim, etc. As soon as I tried to do that I hit some problems mentioned above.

How SaltStack does it: https://docs.saltstack.com/en/latest/ref/states/requisites.html
How Ansible does it (see "handlers"): https://serversforhackers.com/c/an-ansible2-tutorial