Packaging and dependency management is foundational in software deployment, but is often implemented so that outcomes depend on state and are not repeatable. This has implications for reproducing computational experiments, which require repeatable deployment of environments whenever possible. For example, the popular package managers Conda and Pip are both examples of imperative and stateful implementations. In contrast, the Nix package manager is a functional package manager that implements fully repeatable environment deployment in a systematic manner. The talk will introduce the Nix package manager and demonstrate using Nix to generate repeatable Python environments as well as environments involving multiple languages.
Scientific computation is becoming more complex and requires sharing computational experiments involving hundreds of dependencies between scientists using different platforms. Moreover, simply maintaining a computational environment for a single user on a single platform can be challenging if the environment requires multiple languages or multiple package managers each requiring deployment in different global or local locations. The talk will discuss how the Nix package manager overcomes common drawbacks associated with stateful package managers.
Nix is a purely functional package manager that addresses a range of problems (such as dependency hell) associated with the model of imperative execution implemented in most widely used package managers (see the Nix paper). An imperative model implies that the outcome from any action depends on the state of the existing system and/or the previous executed commands. Nix uses a functional model ensuring that any Nix action is independent of the current state of the system ensuring that builds are deterministic and reproducible. The Nix programming language (used by the Nix package manager) is a purely functional language that supports writing to the file system, but only in unique sub-directories with unique identifiers (a purely functional store) so that existing files can never be overwritten during installations. Nix is language agnostic and supports deployment of development environments with multiple languages, which are often difficult to manage.
The talk provides an overview of generating new Nix expressions using existing Pip distributions. A number of different scientific computing scenarios are examined including:
-
a build requiring an old version of Scipy that requires a custom Nix recipe,
-
an environment involving legacy Python 2 code and a number of unsupported packages,
-
a data processing web app using Jupyter, Ruby and Javascript,
-
and, using Nix with Travis CI.
The talk introduces the notion of functional programming and why Nix does not support imperative installation of Python packages. Some tricks are demonstrated including how to use Pip from inside the Nix shell.
The Nix package manager has the potential to become a fundamental tool for scientific Python deployment addressing many issues associated with reproducibility. However, Nix suffers from a steep learning curve when acquiring proficiency in the Nix language and inadequate documentation for Python users. Additionally, Nix does not currently have support for Windows. One of the main goals of this talk is to generate sufficient interest in Nix within the scientific Python community to eventually address some of these usability issues for Python developers.