Nix language changes

nix-lang.md

This document contains some ideas for additions to the Nix language.

Motivation

The Nix package manager, Nixpkgs and NixOS currently have several problems:

• Poor discoverability of package options. Package functions have function arguments like enableFoo, but there is no way for the Nix UI to discover them, let alone to provide programmatic ways to change them. Likewise, there is a global configuration mechanism (config.*) but the Nix UI doesn't know about it.

• Ad-hoc override/configuration mechanisms in Nixpkgs: .override., overrideDerivation, super: self:, overlays, config.*.

• Very inefficient evaluation mechanism. To get metadata about a package, nix-env needs to evaluate a derivation, just to be able to extract its name attribute and other metadata. Quite apart from being inefficient, it's not guaranteed that evaluation succeeds (e.g. a package might have an assert stdenv.isLinux, so on macOS you can't even see the package).

  Also, derivations and package function need to hang on to their inputs to make .override and .overrideDerivation work. This makes it impossible to garbage-collect inputs.

• The functional style of package definitions is verbose and of questionable value. That is, package functions need to specify dependencies in three places: 1) at the use site, e.g. buildInputs = [ libfoo ]; 2) as a function argument, e.g. { libfoo }: ...; and 3) at the call site, e.g. import package.nix { inherit libfoo; }. The last one can be elided using callPackage but that's an ugly hack.

• The NixOS module system is implemented as a library function, which causes oddities like the need for a mkIf function (due to a lack of laziness in the language when merging attribute sets). Evaluation is also pretty expensive.

• Inconsistent configuration styles between NixOS (the module system) and Nixpkgs (function arguments, and an ad hoc config attribute set).

• nix-env has no notion of package plugins. For example, there is no way to install flashplayer into the firefox package. Nix doesn't know that firefox is a wrapper script generator that takes a list of plugins and that flashplayer is something that can be plugged into that list. Nixpkgs has numerous other examples of such "composable" packages, e.g. TeX (texlive.combine) and GHC (ghcWithPackages).

  Such compositions can be expressed in Nix expressions (e.g. ghcWithPackages (pkgs: [pkgs.mtl])), but how to do so is not discoverable except by searching the Internet or grepping the Nixpkgs sources.

It's worth noting that the Nix language is intended as a DSL for package and configuration management, but it has no notions of "packages" or "configurations".

Solution: Configurations

The solution to these problems is to add a language feature named configurations to make Nix more suitable as an actual DSL for package/configuration management. Configurations are essentially an implementation of the NixOS module system as a language feature. The goal is that this provides us with a better override mechanism for packages and can replace the current module system with a more performant implementation.

A configuration is essentially a nested attribute set: it contains values (plus some metadata) indexed by an attribute path. Configurations are composed out of one or module configuration modules. A configuration module is a syntactic construct that vaguely looks like a recursive attribute set using angle brackets:

module1 = <
  foo = 123;
  bar = true;
  a.b.c = if bar then foo else foo * 2;
>;

This configuration module by itself describes a configuration with three options named foo, bar and a.b.c. Selecting an attribute from a configuration module causes it to be evaluated into a configuration (which is an opaque value, not an attribute set!). E.g.

module1.a.b.c => 123

Note that option values can refer to other options in the configuration, so the value of a.b.c. can depend of the values of foo and bar. (In contrast to the NixOS module system, you don't have to write config. in front of every option use.)

Configuration modules can be composed with other configuration modules to create more complex configurations. This is done by extending another module. For example:

module2 = <
  extends module1;
  bar = false;
>;

Now,

module2.a.b.c => 246

Thus, modules can override the values of options declared in another module.

TBD: the alternative is this quasi-functional style:

module2 = module1 <
  bar = false;
>;

i.e. a configuration is extended by applying a module to it. This is particularly nice in conjunction with derivation builders (see below). This also allows extending a configuration with a plain old attrset, e.g. module2 = module1 { bar = false; }.

A crucial property here is that forward references are not allowed. That is, the option values in a particular module cannot reference options that have not been declared in that module or one of its ancestors (i.e. the modules it extends). For example, this doesn't work:

module3 = <
  foo = bar;
>;

module4 = <
  extends module3;
  bar = 1;
>;

Here the evaluation of module3 will fail because there is no bar in scope. This is in contrast to the NixOS module system, which allows any module to reference any other option, which makes it hard to reason about the dependencies of a module.

A few restrictions: option names cannot be computed dynamically. For example, this is not allowed:

bad1 = <
  foo.${bar} = ...;
>;

Also, options cannot be a prefix of a previously defined option, as in

bad2 = <
  extends module1;
  a.b = { c = 100; }; # wrong
  a.b.c = 100; # right
>;

Configuration option fields

[Annotations? Attributes?]

Configuration options, like NixOS options, are not just values. They can have some metadata such as documentation, a type, default values etc. The syntax of an option is as follows:

<ATTR-PATH> (| <FIELD-NAME> <EXPR>?)* (= <EXPR)?;

For example:

module5 = <

  networking.firewall.enable
    | doc "Whether to enable the firewall."
    | type types.bool
    | default false;

  networking.firewall.allowedTCPPorts
    | doc "TCP ports to be opened in the firewall."
    | type types.list types.int
    = [];

>;

TODO: syntax bikeshedding.

The following fields are supported:

• value: An option value at priority level 100. = <X> is sugar for | value `.

• default: An option value at priority level 1500. Thus defaults are ignored (not merged) when there are any higher-priority values.

• doc: A description of the option in some sensible format.

• example: An example value. May be repeated. Unlike in the NixOS module system, there is no literalExample because the parser can store the source text of the example field. (The example needs to be a syntactically valid Expr, but it doesn't need to evaluate.)

• type: The type of the option. This should be a configuration with merge and check members. TODO: to support future incremental evaluation, maybe merge should be a fold, rather than a function that takes a list of all option values.

• if: A Boolean value that determines whether the option value is to be used, e.g.

  <
    environment.systemPackages
      | if enableFoo
      = [ foo ];
  >
  

• prio: The numerical priority of the value. When computing the final value of the option, all values that have a lower priority that the highest-priority value(s) are discarded.

• label, before and after: A string value used to topologically sort option values prior to merging. The NixOS module system uses a numeric value for this, but strings are nicer for things like ordering of the activation script:

  <
    activationScript
      | label "users-groups"
      = "... populate /etc/{passwd, group} ... ";
  
    # This activation script fragment needs to run after users/groups
    # have been created.
    activationScript
      | after "users-groups"
      = "install -d -o fnord -g /var/foo";
  >
  

• final: Specifies that the value of the option cannot be changed by later modules.

• scope: An attrset added to the lexical scope of the evaluation of values of this option, equivalent to putting with <scope>; around every value. Typical use case: allowing different pkgs sets to be in scope for different options for cross-compilation (e.g. you can have options hostBuildInputs and nativeBuildInputs with different package sets in scope).

Any other field name gives a syntax error. Only the initial definition of an option can specify doc, type and example.

Note: there is no merge field. If you want another merge policy you should adapt the type.

TODO: flip priorities? "Higher priority" is ambiguous.

Configuration module fields

Probably it's also nice to have some global fields for a configuration module. E.g.

• doc: A description of the configuration.

• globalScope: Like scope, but for all options in this configuration. Typical use case would be globalScope = pkgs.

• fieldScope: Like scope, but for the evaluation of option fields other than values/defaults. For example, fieldScope = types would allow getting rid of types. in option types.

More syntactic sugar

Configuration modules could have an if construct to enable/disable a group of values more concisely. This is similar to mkIf in the NixOS module system, which pushes the conditional down into the options.

<
  enableFoo
    | type bool
    = false;

  if enableFoo {
    environment.systemPackages = [ foo ];
    systemd.units.foo = ...;
  };

  # The above is equivalent to:
  environment.systemPackages | if enableFoo = [ foo ];
  systemd.units | if enableFoo = { foo = ...; };
>

However, this might not be very useful if we get rid of most enable options (see below).

Configurations as derivation builders

Currently Nix uses functions as the mechanism to abstract over common derivation patterns. For example, stdenv.mkDerivation, buildPythonPackage and fetchurl are all functions that take some inputs and return a derivation. The problem with this approach is that the act of applying a function consumes its inputs, leaving us without the ability to override inputs or to query the inputs. This led to hacks like .override, .overrideDerivation, passthru, meta and so on. Also, functions lack a documentation mechanism.

We can replace such functions with configuration modules that produce a drv output option. For example, here is a very bare-bones derivation configuration module:

builders.derivation = <

  # Interface

  name
    | doc "Name of the derivation, used in the Nix store path."
    | type types.str
    | example "openssl";

  version
    | doc "Version of the derivation, used in the Nix store path."
    | type types.str
    | example "1.0.2"
    = "";

  builder
    | doc "Command to be executed to build the derivation."
    | type types.path
    | example "${bash}/bin/sh";

  args
    | doc "Arguments passed to the builder."
    | type types.unique (types.list types.str)
    = [];

  outputs
    | doc "Symbolic names of the outputs of this derivation."
    | type types.list types.str
    = "out";

  env
    | doc "Structured values passed to the builder."
    | type types.attrsOf types.str
    = { inherit outputs; };

  # Implementation

  drv
    | doc "The resulting store derivation."
    | final
    = builtins.derivation ({
        name = "${name}-${version}";
        inherit builder args;
      } // env);

>;

We can then build other abstractions on top of each other:

# This is basically stdenv.
builders.generic = <

  extends builders.derivation;

  # Interface

  buildInputs
    | doc "Dependencies of this derivation."
    | type types.list types.package # FIXME
    = [];

  phases
    | doc "Names of build phases."
    | type types.list types.str
    | example ["build" "install"];

  # Implementation

  builder = "${bash}/bin/bash";

  args = [ "-c" stdenv/generic/setup.sh ];

  env = { inherit phases buildInputs; };

  ...
>;

# A package is a derivation that can be installed by nix-env.
builder.package = <

  description
    | doc "A short (one-line) description of the package."
    | type types.str;

  homepage = ...;
  longDescription = ...;

  supportedPlatforms
    | doc "List of platforms on which this package is supported."
    | type types.list types.platform;

  # nix-env won't show this package if `supported` evaluates to false.
  supported
    | doc "Whether the package is supported on the target platform."
    | type types.bool
    = elem system supportedPlatforms;

  # Nix will refuse to build this package if `enabled` evaluates to false.
  enabled
    | doc "Whether the package can be built."
    | type types.bool
    = (free || config.allowUnfree) && supported;

>;

# Global Nixpkgs configuration.
config = <

  allowUnfree
    | doc "Whether to allow unfree software to be installed."
    | type types.bool
    = false;

>;

builders.unixPackage = <

  extends builders.generic builders.package;

  src
    | doc "The source tarball of the package."
    | type types.path;

  configureFlags
    | doc "Flags passed to the package's configure script."
    | type types.list types.str
    = [];

  # Create some phases.
  phases | label "unpack" = ["unpack"];
  phases | label "configure" after "unpack" = ["configure"];
  phases | label "build" after "configure" = ["build"];
  phases | label "install" after "build" = ["install"];

  configurePhase
    | doc "Shell code to configure the package."
    | type type.lines
    = "./configure --prefix=${placeholder out} $configureFlags";

  buildPhase
    | doc "Shell code to build the package."
    | type type.lines
    = "make";

  installPhase
    | doc "Shell code to build the package."
    | type type.lines
    = "make install";

  env = { inherit configurePhase ...; };
>;

# An actual package.
pkgs.hello = <
  extends builders.unixPackage;

  name = "hello";
  version = "1.12";
  description = "A program that produces a familiar, friendly greeting";
  license = licenses.gpl;

  enableGUI
    | doc "Enable GTK+ support."
    | type types.bool
    | default false;

  # This uses the functional notation for extending a configuration:
  src = builders.fetchurl { url = ...; sha256 = ...; };

  buildInputs = if enableGUI then [ gtk ] else [];
  # or equivalently:
  buildInputs | if enableGUI = gtk;
>;

Since configuration values are computed lazily, a tool like nix-env -qa only needs to evaluate attributes like name and enabled. It doesn't need to evaluate drv.

Package overrides

# To override the source of Hello.
pkgs.hello = <
  src.url = http://example.org/my-hello.tar.gz;

  # Note: this is sugar for:
  src = < url = http://example.org/my-hello.tar.gz; >;
  # I.e. it adds a module to the configuration 'src'.
>;

# To enable a feature in Hello.
pkgs.hello = <
  enableGUI = true;
>;

# To add something to installPhase.
pkgs.hello = <
  installPhase | after default = "mv $out/bin/hello $out/bin/my-hello";
>;

# To change a dependency of Hello in Hello *only*.
pkgs.hello = <
  targetPackages = < extends pkgs; gtk = < src = ./my-gtk.tar.gz; >; >;
>;

# To replace Hello entirely. FIXME
pkgs.hello | prio 0 = pkgs.my-hello;
pkgs.my-hello = < ... >;

Plugins

TODO

pkgs.firefox = <
  extends builders.unixPackage;

  plugins
    | doc = "Set of enabled plugins."
    | type = subsetOf pkgs.mozillaPlugins;
    # this type should implicitly set `scope` to the given attrset
    = [];

  installPhase = "... generate firefox wrapper script ...";
>;

pkgs.mozillaPlugins.flashplayer = < ... >;

To enable:

pkgs.firefox = <
  plugins = [ flashplayer ];
>;

User interface

Having explicit package options and attribute annotations allows nix to show and modify options. E.g.

$ nix query-package nixpkgs.firefox
Available options:

- enablePulseAudio: Enable sound via PulseAudio.
Default value: true
Current value: N/A

- enableOfficialBranding
...

- plugins: Set of enabled plugins.
Possible values:
  nixpkgs.flashplayer
  nixpkgs.google-talk
  ...

$ nix install nixpkgs.firefox --with enableOfficialBranding --without enablePulseAudio

$ nix modify-package nixpkgs.firefox --with enablePulseAudio --add plugins nixpkgs.flashplayer

TBD: we probably don't want to show only end-user options like enablePulseAudio by default, rather than every option inherited from builders like buildInputs or name. So we need some way to specify which options should be shown in what context (like the internal flag in the current NixOS module system).

Nixpkgs/NixOS structure

TODO: how to hook everything up.

At top-level, Nixpkgs should evaluate to a set (or configuration?) of configurations:

{
  lib = <
    ...;
  >;

  builders = <
    derivation = ...;
    generic = ...;
    package = ...;
    unixPackage = ...;
    fetchurl = ...;
    ...;
  >;

  pkgs = <
    glibc = import .../glibc;
    hello = import .../hello;
  >;

  # NixOS modules
  modules = <
    etc = import modules/.../etc.nix;
    ...;
    top-level = import modules/.../top-level.nix;
    ...
    # The minimal NixOS system.
    base = < extends etc top-level initrd kernel ...; >;

    # Non-default stuff.
    kde = import modules/.../kde.nix;
  >;
}

We should no longer include every NixOS module by default. Instead of using enable options, you just inherit NixOS modules into your system configuration, e.g.

myConfig = <
  extends modules.base modules.kde modules.nginx ...;
  ...
>;

NixOS services

In the NixOS Of The Future, most modules shouldn't be top-level configuration modules, since those 1) violate POLA: you don't want the PostgreSQL module to have the ability to change your X11 configuration; and 2) are not functional: you can't easily instantiate a module multiple times, e.g. to have multiple PostgreSQL instances.

Instead most NixOS modules should be implemented as extensions of smaller configuration types. For example, builders.service allows the construction of a service that declares some systemd unit, and only has access to (say) /var/services/<service-name> and the closure of the unit in the Nix store. (This is essentially a light-weight container.)

builders.service = <
  name
    | doc "The identifier of the service."
    | type str;

  systemdUnits
    | doc "The systemd units that implement this service."
    | type attrsOf /* instances of builders.systemUnit */
    = [];

  ...
>;

So PostgreSQL can be extension of this type:

services.postgresql = builders.service <
  # Interface
  port
    | doc "TCP port on which the server listens."
    | type int
    = 5432;

  # Implementation
  systemdUnits =
    ... a systemd service that runs postgresql, using
    /var/services/<service-name> as the data directory ...;

>;

NixOS at top-level has a systemServices option that allows services to be hooked into the system:

modules.services = <

  systemServices
    | doc "Set of isolated system services"
    | type attrsOf /* instances of builders.service */;
    ;

  systemd.units =
    ... the concatenation of the systemdUnits values of all systemServices ..;

>;

A NixOS configuration can then instantiate multiple PostgreSQL instances:

<
  systemServices.postgresql-prod = services.postgresql < >;
  systemServices.postgresql-test = services.postgresql < port = 12345; >;
>

Include mechanism

The idea is to provide an include mechanism as an alternative to import. The expression include ./foo.nix is equivalent to replacing the expression with the contents of ./foo.nix. Thus foo.nix has access to the lexical scope at the site where it is included.

This means that we can write all-packages.nix as a long list of inputs:

<
  pkgs = <
    git = include ./git.nix;
    openssl = include ./openssl.nix;
    ...
  >;
>

where the package expressions look like

builders.unixPackage <
  name = "git";
  version = "2.3.4";
  ...
  buildInputs = [ openssl ];
>

That is, git.nix is no longer a function that takes builders as an argument. Rather it just assumes that builders exists in the lexical scope, thus preventing the need for ugly hacks like callPackage. This might seem like a radical departure from how we expressed dependencies in Nixpkgs, but in fact it's already widely in use, see e.g. perl-packages.nix or python-packages.nix where most dependencies are obtained from the surrounding rec set. include simply makes it possible to split such package sets into multiple files.

Note: The include mechanism must be lazy, i.e., the inclusion shouldn't be done eagerly while parsing the including file, since we don't want all-packages.nix to eagerly all its referenced files.

Note: This is not a token-based include mechanism like C's #include. That is, include E is only valid as an expression (i.e. the grammar will have a non-terminal expr: INCLUDE expr), and the contents of the included file is parsed as an expr. Thus you cannot write something like

{ x = 123;
  include ./other-attrs.nix
}

TODO: this may not be needed since option values are evaluated in a dynamic scope anyway. E.g. in the example above openssl doesn't have to be provided as a function argument anyway because it doesn't refer to the lexical scope but to the scope set by the buildInputs option. However, builders does need to come from somewhere, and it would be annoying if we had to change the whole file into a function just to pass in builders. OTOH, if we used the extends syntax instead, then maybe this could be avoided: its argument could be evaluated in a dynamic scope.

Discussion

Memory / performance impact

Hopefully Nix configurations can lead to less CPU and memory-hungry Nixpkgs/NixOS evaluation. There are some aspects to this:

• Configuration values should be implemented internally by keeping weak pointers to option values. Thus option values can be garbage collected (and re-evaluated from the saved thunk if necessary). Of course, this is a risky strategy because the thunk might be bigger than the WHNF. So if an option value is "small" (like an integer or small string) it makes sense to overwrite the thunk by a non-weak pointer to the resulting value.

• Getting rid of pass-thru derivation attributes like .override, .overrideDerivation, .meta etc. means that after evaluating a drv option value of a package configuration, we can garbage-collect the entire package configuration (assuming nothing else is keeping it alive).

• Not including every NixOS module by default should save a huge amount of time and memory.

• Doing the module system in C++ rather than in a slow purely functional language should give a big speedup.

Incremental evaluation?

Can we make NixOS module evaluation incremental? E.g. in a NixOps module with multiple machines that share a base configuration:

myBase = <
  extends modules.nixos modules.kde ...;
  networking.firewall.allowedTCPPorts = [ 1234 ];
>;

machine1 = myBase < networking.hostname = "machine1"; >;
machine2 = myBase < networking.hostname = "machine2"; >;
...

it would be nice if the evaluation of the configurations of machine1 and machine2 only needs to recompute the options that are affected by the change to networking.hostname.

Do we need "super"?

The self: super: style has a self attribute to refer to the final set, and super attribute to refer to the previous set in the chain. (It should really be named final: previous:.) In configuration modules, any reference is to the final configuration, so no special notation for referring to self is required. But do we need an equivalent for super?

The typical use case is to extend a previous value, e.g.

buildInputs = super.buildInputs ++ [ libfoo ];

However, merge functions remove the need for this: we just concatenate all definitions of buildInputs, just as is done in the NixOS module system, i.e.

buildInputs = [ libfoo ];

If we want to throw away previous definitions, we do:

buildInputs override = [ libfoo ];

(This corresponds with mkForce in the module system.)

The experience with the NixOS module system suggests that this is sufficient and we don't need a super keyword.

Strict argument checking

Historically, Nixpkgs has been full of builder functions that accept an open set of arguments (...), e.g. stdenv.mkDerivation. This is bad for error checking since a typo in an argument won't be detected. Thus, configuration modules like builders.package or builders.generic have a closed set of inputs in the sense that they cannot refer to options that haven't been declared yet. Also, builders.derivation doesn't pass through arbitrary arguments to the builder environment; instead the environment has to be specified explicitly (by adding values to the env option).

This should be fine because the use of environment variables really predates the introduction of string interpolation. Originally, if (say) buildPhase needed access to some variable foo, we would write

derivation {
  inherit foo;
  buildPhase = "... $foo ... ";
}

But nowadays we can just write

<
  buildPhase = "... ${foo} ... ";
>

Migration path

TODO

Nice work on this! It's great that there are proposals to big changes like "package". Derivations are a useful concept, but adding new things like this should help to make Nix feel a more "normal" package manager.

On metadata for packages: I wrote the "PackageKit" backend for Nix a year ago (it's still broken, sorry everyone!) and found that lots of the things in Nix don't map very well to the PackageKit standard. I definitely think it would be a good idea to try to reuse this standard where possible (obviously some deviations are allowed). For instance, here is the header file that every PackageKit backend implements:

https://github.com/hughsie/PackageKit/blob/master/src/pk-backend.h

It expects us to keep track of information like:

• author (different from maintainer)
• "groups"
• mime types
• "categories"
• EULA
• "update details"
• "providers"

Some of these are certainly of debatable value but I think we some of this information is valuable. In fact, I think packagekit allows you to not provide most of this information (but certain UI items might look out of place in some apps using package kit).

So, I have some feedback on the ideas expressed here, but first of all I want to emphasize one particular argument: I would be strongly against adding an include mechanism of any kind.

Right now, Nix as a language has a rather beneficial property, namely the ability to reliably follow references. If something is in scope, that means it is visibly declared somewhere in the same file; in a let ... in block, as a function argument, and so on. This makes it very easy to navigate even large amounts of Nix code without being familiar with it, as you can quite literally 'follow the crumbs'.

Adding an include system would remove this property from the language; it would introduce a 'treasure hunt' where a referenced identifier could have been declared in any included file. This especially does not scale; the more includes you add, the more files you will need to go through to find an identifier. This makes understanding existing Nix code an unnecessarily arduous task.

If we want an easier way to include package modules, then I can't see why that wouldn't be possible without includes; for example:

packageSet {{
    git = include ./git.nix;
    openssl = include ./openssl.nix;
}}

... where packageSet would simply be a function that, for each of the items within the attribute set, injects dependencies as needed, in whatever fashion is supported by whichever package declaration format is eventually chosen. This would make the problem of dependency injection a userland-solvable problem, without making it a language-level concern.

I haven't thought too much about the specifics, but especially with the introduction of extensible attribute sets, this should definitely be possible without adding includes.

---

As for the rest of my feedback:

• Extensible attribute sets are a great idea. I'm not certain that they should be deep-merged by default, however; from my experience in JS-land, default deep-merges often trip people up.

  Another point to consider here is that often, neither a deep merge nor a shallow merge are desired - but rather, the author would want a 'cutoff point' at which a 'sub-object' is treated as a wholly replaceable but non-extensible unit, and everything up to that point can be deep-merged. This applies especially in situations where the author of the attribute sets does not control the merging algorithm used in the end, such as for plugin/option specifications exposed by a package.

  I suspect that extensible vs. regular sets would provide a natural way to indicate this, but additional syntax may be required.

• Annotations make sense; however, I find neither of the suggested syntaxes to be ergonomic. I'm not sure how to improve it, although I'm wondering whether a postfix syntax may be a better option here; or possibly even a completely separate syntax, where the annotation metadata is specified separately from the (default) value, with a distinct syntax. For example:

  {
    someOption = false;
    someOption::description = "Description goes here";
    someOption::type = Boolean;
  }

• Partial ordering for priorities is a great idea, but I'm not convinced that it needs to be a language-level feature; it seems like this is implementable in userland, perhaps with a builtin for efficient sorting of lists (of attribute sets).

• Plugins are a tricky point. A major issue is that every application has a subtly different plugin mechanism, and there's some variance in how stringent the expectations of applications are; some applications are totally okay with a bunch of symlinks for plugins, whereas others mandate that the plugins be stored directly in the application prefix, whereas yet others expect them to be in the user's home directory.

  I don't think this can be fully solved on a language level; however, a major sticking point that does probably require cooperation from the language and/or package manager, is the ability to do 'partial builds'. Right now, a change in plugins/mods typically requires a full rebuild of the application, and games have to be built with all their assets included in the build. This creates significant bloat, and in the case of games with large assets it simply means that the entire game isn't built in Hydra as the derivation would get too big due to the assets.

  It would be useful to have some way to do 'partial builds' of some sort, where different aspects of an application can be separately built without breaking any of the assumptions that applications have about where and how assets or plugins are exposed. I don't know whether it's possible, but support for partial builds as well as an 'interface standard' for how to make plugin configuration work in packages, would probably fully solve the plugin issue as it stands.

• It's not clear to me what problem the delayed functions are meant to solve.

• It's also not clear to me what exactly the buildable / package functions solve that can't be solved on a userland level. Surely, preventing evaluation of derivations when inspecting the metadata would just involve wrapping the buildable item in a function?

• I don't fully understand the problem case that's described in the "Overriding dependencies" section. Why would the stated example "override nss globally"?

• Given a solid implementation of extensible attribute sets, I don't see why we'd still need super.

• Yes, we definitely need an escape hatch for actual environment variables; these are often needed for custom setup scripts. I also vaguely recall there being some cases where environment variables were intentionally used in place of string concatenation, but I don't recall what or where exactly those cases were.

Some further thoughts:

• I'd like to see a better mechanism for using third-party 'Nix libraries'; it would allow many of the current ergonomics problems to be solved by userland implementations, possibly multiple competing ones, and would also allow for more extensive out-of-tree abstractions for eg. dealing with WINE versions or particular plugin architectures. This doesn't necessarily need to take the form of a package manager like npm, but something functionally equivalent is what I'd be thinking of here.
• There are currently two major sticking points for less conventional cases; out-of-store secrets and fixed-hash derivations for local files. I'm not sure what the progress is on either of these, but it would probably be worth considering whether they can be solved as a part of these language changes or not.

Please ping me on IRC (joepie91 in #nixos) if you have any further questions or thoughts about the above points, since Gist doesn't send out notifications for replies, nor for explicit mentions.

Additional comment: with the out-of-band :: annotation syntax I suggested, it would also open up the possibility of extending the annotations for an extensible attribute set, rather than just its value, and without having to re-define all the existing annotations. The same would apply to any other out-of-band syntax.

@edolstra Something that feels like it's missing from the usual symmetry within Nix: Just as we have a way to turn an attribute set into a lexical scope using with, there could be two builtins for:

• capturing the current lexical scope (minus builtins) as an attribute set, so it can be passed as a value;
• "clearing" the current lexical scope (minus builtins), so that clearing and using with on a capture recovers the exact lexical context of the capture

The functional style of package definitions is verbose and of questionable value. That is, package functions need to specify dependencies in three places: 1) at the use site, e.g. buildInputs = [ libfoo ]; 2) as a function argument, e.g. { libfoo }: ...; and 3) at the call site, e.g. import package.nix { inherit libfoo; }. The last one can be elided using callPackage but that's an ugly hack.

Until there is a way to reflect into expression and tell which inputs it uses, it is useful to specify things twice. Really, if I could hower cursor over package definition in IDE and see it's "inputs" (source inputs, build-time inputs, runtime inputs, build-time inputs inherited from mkDerivation, any other level-2 dependecies) separated into groups, it would be OK to specify things once.

Follow up on this discussion in NixOS/nix#2511

Also see my critique of this in NixOS/nixpkgs#56227 (comment) where I argue that 80% of the problems this tries to address can be solved by simply adding optional types to attrsets.

On a whole, these changes seems nice to me as someone who has been someone stuck in the learning curve; the syntax overall looks easier to follow than most of what I have seen.

Syntactic bikeshedding here - is there no way to have the {{ ... }} double curley braces for configurations like earlier versions of this gist showed? I think its somehow more reminiscent of attribute sets and will be more familiar to newcomers. Wouldn't it be nice to reserve the < and > characters for other operands too?

Looking forward to see it in action!

I am in love with this.

(Except for lexically-magical include. But everything else is 💖.)

It looks to me as if Jsonnet was covertly inspired by nix. Why not have a look: https://jsonnet.org/articles/design.html?

Discourse Topic on Comparing nix with jsonnet

Three thoughts about the proposed include:

1. Right now, Nix as a language has a rather beneficial property, namely the ability to reliably follow references. If something is in scope, that means it is visibly declared somewhere in the same file;

   While this is mostly true, and I agree it's important, with breaks this.

2. Minor quibble: I think just having the two keywords include and import is going to be super confusing for people learning the language. Something like haskell's import {#- SOURCE -#} seems better to me, where there is a clearly delineated default option (import), and an optional source import that is (a) clear what it does, and (b) clearly dispreferred. Obviously we don't have pragmas in Nix, but something along those lines.

3. I'm wondering if we can resolve people's problems with the proposed include (without getting rid of what we want) by having a syntax that lists at the top of the file the variables that you expect to be in scope when the file is included. (this would be required for any file you want to include).

The systemServices idea can be implemented or prototyped in the existing NixOS. This can be exploited for testing the dynamics and usability of this new approach, without committing to C++ code and allowing more of the community to contribute.

1. Add a module that declares config.systemServices as an attrsOf (submoduleWith {})
2. Add a module that reads from config.systemServices and writes to systemd.*, environment.*, etc.
3. Write functions that produce modules to be loaded into systemServices, such as the postgres one.
4. config.systemServices.mydb.imports = [ (postgres { }) ]. Note the With in submoduleWith earlier to make this syntax possible.
5. (Bonus points) write a module that reads from config.systemServices and writes to launchd.*, load it into nix-darwin. You've built the mythical service abstraction layer.

can be implemented or prototyped in the existing NixOS

Forgot about this comment, but did that for RFC 163.

• NixOS/nixpkgs#267111
• NixOS/rfcs#163

3. Write functions that produce modules to be loaded into systemServices, such as the postgres one.

This was a mistake. Don't write functions that produce modules, unless you really have to. They're already functions.
types.deferredModule didn't exist yet, which is for modules that are still abstract - not merged with the other modules into a submodule or evalModules call.

This is awesome. Is this still an "eventually" idea? Or has the nix ecosystem since matured/(become complex) enough that it is no longer feasible?

The Nix team is currently trying to finish other eventually-level ideas that already have implementations in the form of experimental features, that need to be made sustainable first.