Okay, I've got a need to build Firefox from source, and I'd like to do that on a remote machine, and then copy build result back to my laptop. With Nix, using bastion host. I'll note details of my successful adventure.
Here's the list of resources I've used actively:
- https://nixos.wiki/wiki/Distributed_build
- https://nixos.org/nix/manual/#conf-trusted-substituters and various other configuration parameters
- https://github.com/NixOS/nixpkgs/blob/release-19.09/nixos/modules/services/misc/nix-daemon.nix which contains mappings from Nix config names to NixOS config names (damn, they are different!)
Here's my setup:
- local machine - ThinkPad 8cores 16Gb RAM, NixOS
- remote machine - some 48-core 512Gb RAM, Ubuntu, with Nix installed in multi-user way
- jump host - a bastion host, Ubuntu, ssh to remote machine works only from jump host
I wanted to build nixpkgs.firefox-esr-52, it is last one to support NPAPI plugins (I need icedtea for fucking remains of Java-applet in my bank). It stopped being built by Nixpkgs Hydra because it is "insecure". And previously built firefox stopped working because of some GLIBC version mismatch....
Let's make ssh remote_host seamless. Append to ~/.ssh/ssh_config:
Host remote_host
ProxyCommand ssh -W remote_host:1221 ubuntu@jump_host
User ubuntu
If ssh remote_host now works, remember, it is not enough! This should work for root too, but you can't edit /root/.ssh/ssh_config. It just doesn't work. You have to edit /etc/ssh/ssh_config or set extra config through NixOS options:
programs.ssh.extraConfig = ''
Host remote_host
ProxyCommand ssh -i /root/.ssh/my_key -W remote_host:1221 ubuntu@jump_host
IdentityFile /root/.ssh/my_key
User ubuntu
'';
You should play with various SSH config params until ssh remote_host works as root. For example, note that you should have your private key copied to root home, with root owner.
NOTE: Ideally you should connect to remote host as root, but I didn't have such an opportunity. This will make a bit pain later.
Signing should prevent you to download malware build artifacts. Even when we use SSH private key to reach our builder. But oh well, let's do this.
Remote host:
nix-store --generate-binary-cache-key builder-name cache-priv-key.pem cache-pub-key.pemecho "\nsecret-key-files = $PWD/cache-priv-key.pem" >> /etc/nix/nix.confsudo $(which nix) sign-paths --all -k cache-priv-key.pem
(if you ask why $(which nix) - it is because by default sudo on Ubuntu limits PATH envvar.)
Local host:
nix.binaryCachePublicKeys = [
"builder-name:dauKsezR2tY+HsLpeH7hzx8LcCTdPvUPBoJ/3Y=" # this one is cache-pub-key.pem content
];
nix.trustedBinaryCaches = [
"ssh-ng://remote_host"
"ssh://remote_host"
];
VERY IMPORTANT: don't set nix.binaryCaches here. nix.trustedBinaryCaches sets optional builders, nix.binaryCaches sets default builders.
So, when you do nix-build '<nixpkgs>' -A firefox-esr-52 you download everything from cache, except firefox itself. So we want to build only firefox remotely, and everything else should be downloaded directly.
The build is like this:
- first we generate .drv for firefox package
- then we copy this .drv to remote builder
- then we SSH and build .drv on a remote builder
- then we copy result back
- from now on, our local
nixcommands will know about package and won't try to build it from source.
This is described in NixOS/nix#1639 (comment). I'll give direct commands on what was done.
It is actually an overlay:
self: super: let
in {
ff_java = (import <nixpkgs> {
config = {
allowUnfree = true;
permittedInsecurePackages = [
"firefox-52.9.0esr"
"firefox-esr-unwrapped-52.9.0esr"
];
firefox = {
ffmpegSupport = false;
icedtea = true;
};
};
}).firefox-esr-52;
}
in /home/danbst/.config/nixpkgs/overlays/firefox_java.nix
$ nix-instantiate '<nixpkgs>' -A ff_java
warning: you did not specify '--add-root'; the result might be removed by the garbage collector
/nix/store/ynpj76a9nfdrx82zqirfvvrldw2gf3fn-firefox-52.9.0esr.drv
$ nix-copy-closure --to remote_host -s /nix/store/yqg1sd13z89vc4z81dkbjnb9hjsy28z9-firefox-52.9.0esr.drv
The -s param is important both as optimization and a fix for a problem. It says "do downloads where possible on a remote side". If you don't set it, it will copy your local .drv-s to remote host. But then you can have a problem - not all your .drv-s are trusted on remote builder! -s soemhow fixes this problem.
$ ssh remote_host
remote_host $ nix build /nix/store/yqg1sd13z89vc4z81dkbjnb9hjsy28z9-firefox-52.9.0esr.drv
.... nice Nix2.0 build output ...
.... unfortunately, Nix2.0 doesn't show the path to package built, so we have to run old nix-build. It will be no-op later ...
remote_host $ nix-build /nix/store/yqg1sd13z89vc4z81dkbjnb9hjsy28z9-firefox-52.9.0esr.drv
/nix/store/9x839fn56fyc6ar3wpgp6x9h5zn91m1j-firefox-52.9.0esr
But because we've built it not as root, we have to sign our build result. Easiest is to do again
$ sudo $(which nix) sign-paths --all -k cache-priv-key.pem
$ nix-copy-closure --from remote_host -s /nix/store/9x839fn56fyc6ar3wpgp6x9h5zn91m1j-firefox-52.9.0esr
And that's it! I have now firefox-52 on my laptop. Last thing -- pin it in as garbage root:
$ nix-env -f '<nixpkgs>' -iA ff_java -p /nix/var/nix/profiles/per-user/danbst/firefox-java
I've tried also distributed builder config:
nix.distributedBuilds = true;
nix.buildMachines = [
{
hostName = "remote_host";
system = "x86_64-linux";
maxJobs = 32;
}
];
Well, I was able to launch it
$ nix build -vvvvvvvvvv --option extra-substituters ssh-ng://remote_host --option max-jobs 0 /nix/store/yqg1sd13z89vc4z81dkbjnb9hjsy28z9-firefox-52.9.0esr.drv
however it started downloading Firefox sources to my laptop. I've canceled it before it started sending Firefox sources from my latop to remote host 🤦
If you're still interested in this, you can add
--builders-use-substitutesto yournix buildso that the sources are downloaded straight to the remote host.