(started as a response during a private conversation about how using multiple languages could get confusing or overly complex, when discussing how Nix is used as a general purpose configuration language)
So the more I gained experience with Nix, the more I started to personally feel that the main issue was (a lack of) clean boundaries. So I do think there is room for multiple languages, but the key is to be consistant in what they are used for. From that perspective I think there is room for a few different languages. Something like:
- nix as a DSL for packaging (not config)
- some actual programming language for config (because I despise yaml templating)
- Nickel: still doesn't really have a "killer-app" to demonstrate its value, but on the surface it seems promising
- Dhall: I actually love that dhall is not turing complete. If its type system wasn't so damn rigid, it would be my preferred choice here, but unfortunately that is not the case
- Cue: it seems like cue is starting to win this battle. I see it being picked up in a lot of places nowadays, but maybe that's just me
- TOML for static configuration that actually benefits the tooling by remaining static (e.g. to keep evaluation bounded/predictable)
when it comes to static config I actually wish something like strictyaml had more cross-language support, because I actually think yaml's syntax is far more concise than TOMLs. But the problem is that yaml is horrendously ambiguous and has all kinds of anti-features. So I'm just trying to be practical in my choices above.
The real can of worms is the config boundary. There are a ton of problems I believe I have identified:
- we don't want to rewrite the config world in a new langauge
- e.g. if we had a language with a type system, okay great, but now we have to write types for every config ever? not so great (this is already the problem we are hitting with the growing number of nixos modules out in the wild, which is compounded by the fact that Nix is a wildly inappropriate choice for type validation)
- we do want to be able to abstract common config and use proper programming constructs to do so
- the pain of YAML templating I think proves this point for me
I guess the problem is kinda that there is an inherent conflict in these goals. Tools use all sorts of config strategies, and it is really impossible to abstract over them all, or even a significant portion, without hitting one of these undesirable problems.
I guess one final point is to address the question that comes up, and since we don't have a good answer, has apparently become the de facto solution, regardless of its drawbacks: "why not just use Nix for everything".
Well, because then we end up right where we are:
- no answer to the problems above, with complexity that is just compounding out of control (all the nixos modules out in the wild, and the maintenance and documentation burden they induce)
- no real strategy to speak of to actual tackle this unbounded complexity
- no easy way to dial it back and contain it (so it just keeps growing even more out of control)
These problems and others are the main motivation for eka's plugin system. As much as possible we should avoid rewriting any config at all outside the expected format that each specific program expects. Instead, if we need or want to share values between them, we use one of the specialized config languages above as "glue", and pass the result across a well defined API boundary to a Nix evaluation context.
This is the main motivation for eka's envisioned plugin system. Many questions are still outstanding, but the overall architecture, at least according to my understanding of the problems above, seems to be sound, as far as I can tell.
One last piece worth mentioning is when and why we might want certain parts of config to remain static. The answer would be something like, "when it sits along the critical evaluation path". Take the atom manifest itself, which is a static toml file. If we have to run some kind of evaluation process just to determine dependencies we already start to fall below the standard expected from other ecosystems. So instead, certain information should be contained within a static boundary (TOML manifest) to avoid the extra cost that would significantly reduce either the real performance cost, or the theoretical complexity of evaluating a large number of build requests in a Nix like system.
One problem we face in Nix is the explosion of variants produced when we want to account for things like cross-compilation, or other variables that may effect the end result of the produced artifact, such as whether we want a static binary or not, etc, etc. For that I believe a statically defined matrix of variants which we pass into to Nix at runtime is far more appropriate than the cambrian explosion of pkgsCross.* variants in current nixpkgs.
It is by no means final, but here is a theoretical example of a fairly minimal static matrix DSL which could be declared in a TOML, to demonstrate the concept:
[[pkg.my.awesome.package]]
# Build:?Host:?Target:?{*}
matrix = [
"x86_64-linux:{static,musl}:{store,deb,oci}",
# expansion
"x86_64-linux:{aarch64-linux,riscv64-linux}:{gnu,musl,libc}:{static, dyn}",
# {^build|host|target} from immediately above
# {*}: repeats the list immediately preceeding it
"{^host}:{*}:static" # {aarch64-linux,riscv64-linux}:{aarch64-linux,riscv64-linux}:static
]
# specify in reverse to disable
sparse = [
# disable the static gnu builds for both host architectures
"static:gnu:*"
]Of course there is some "hidden" complexity here, in the form of having to parse the matrix dsl, but the point is that on the surface it is static, that extra complexity is hidden and well-defined (inside eka, i.e. performant rust code) so that it is fast and never introduces a significant enough penalty to be problematic.