The end goal of this experiment is to produce a statically-linked rust executable on Linux that has absolutely no dependencies (other than the kernel interfaces). We cannot do this on a glibc-based system because glibc does not support being statically linked. We need to link against musl (musl-libc.org) which supports being statically linked (and also has a smaller footprint than glibc).
From the reasoning above, the first step we need to do is to produce a toolchain that targets musl. As we are running under a glibc-based distro, we will need a cross toolchain that runs on host triple x86_64-unknown-linux-gnu and generates binaries for the target triple x86_64-unknown-linux-musl.
The second step is to patch rustc to generate binaries for the target triple x86_64-unknown-linux-musl. As rustc depends on LLVM to do most of the heavylifting, we need to patch LLVM as well. These patches are pretty trivial though (mostly just to patch the build system to recognize the new target triple), because musl and glibc are mostly compatible (on a source level, not on a binary level).
The next step is to build rustc that targets x86_64-unknown-linux-musl. rustc itself is inherently a cross compiler, meaning that one rustc is able to generate binaries for all supported targets. However, the standard libraries are usually only compiled for the host triple (because it simply takes too much time to compile for each and every supported target triples). The rustc build system will need to be changed slightly in order for us to do this.
Now that we've built a rustc that can target x86_64-unknown-linux-musl, to generate a musl-linked executable we simply invoke rustc and specify the required target triple. However, the executable generated this way will be linked against musl dynamically. It appears that when invoking the system linker (in our case, x86_64-unknown-linux-musl-gcc), rustc always tells it to linked to libc dynamically.
There are two ways to fix this. We can modify the rust compiler so that it calls the system linker to link against libc statically. Or we can just manually invoke the system linker ourselves. For this quick exercise the latter approach suffices. So this is what we are going to do.
The easiest way to get a cross compiler that targets musl is the musl-cross script (https://github.com/GregorR/musl-cross). Clone this repo (which appears to be more actively maintained than the other one mentioned in the readme file):
git clone https://github.com/GregorR/musl-cross
If you just run the ./build.sh script now it will happily run and produce a musl-targeting cross toolchain. However, it will fail to compile some of the LLVM libraries because of an issue in musl's implementation of the standard C headers (stdio.h and sys/stat.h). The issue is that some of the preprocessor symbols defined in said C headers pollute the LLVM namespace.
To fix this you need to apply the following patch to musl:
https://gist.githubusercontent.com/cl91/bb927df2525738502131/raw/b16eeab3d855986b294ae80fc4cbfc7630b45932/0001-Fix-namespace-pollution-of-preprocessor-macros.patch
Apply this to the latest musl master (I believe any version after 1.1.6 will work, although I have not tested them.). You can clone musl from git://git.musl-libc.org/musl.
The build system of LLVM expects to find cross toolchain with prefix x86_64-unknown-linux-musl if we are building for said target. The cross prefix that the musl-cross script creates is slightly different: x86_64-linux-musl. To fix this we modify the build script to symlink the cross toolchain to the correct cross prefix:
https://gist.githubusercontent.com/cl91/bb927df2525738502131/raw/cf6b243b7454fd1d29c9d42b134d16d73774e147/0001-Symlink-to-cross-prefix-x86_64-unknown-linux-musl.patch
Now run the build script ./build.sh. You might want to modify CC_BASE_PREFIX in both config.sh and config-static.sh to something other than /opt/cross if you want your cross toolchain to be installed somewhere else. In my case, I set it to $HOME/cross.
Rustc links to libedit which is a library for command line editing. To build rustc for the target triple we need libedit and all its dependencies built for x86_64-unknown-linux-musl. libedit depends on ncurses. Therefore the next step is to cross compile ncurses and libedit for the target triple.
Get ncurses from here:
https://ftp.gnu.org/gnu/ncurses/ncurses-5.9.tar.gz
Once you downloaded it, untar it and cd into the source directory.
I am assuming that the cross toolchain is installed into $HOME/cross.
export CROSS_TOOLS=$HOME/cross
export PATH=$PATH:$CROSS_TOOLS/x86_64-linux-musl/bin
You need to modify this accordingly.
Before compiling ncurses you need to modify the config.sub script to recognize our target triple.
wget https://gist.githubusercontent.com/cl91/bb927df2525738502131/raw/f5e6bc9485d5d48bc7466623c310f5b64f950656/config-sub-musl.patch
patch -p1 < config-sub-musl.patch
We now build ncurses for the cross architecture:
./configure --prefix=$CROSS_TOOLS/x86_64-linux-musl/x86_64-linux-musl --host=x86_64-unknown-linux-musl
make && make install
Note that if you are hitting the tic error, you need to make sure that the host $PATH comes before the cross tools.
Now we need to build libedit. It appears that the build system for libedit has already been patched to recognize musl. We simply configure it and build it:
wget http://www.thrysoee.dk/editline/libedit-20141030-3.1.tar.gz
cd libedit-20141029-3.1
./configure --prefix=$CROSS_TOOLS/x86_64-linux-musl/x86_64-linux-musl --host=x86_64-unknown-linux-musl
make && make install
The object files that rustc generates uses symbols from libgcc for exception handling. It appears that the build script for the cross toolchain only copies the dynamic version of this library but not the static ones. We need to copy them manually:
cd <path-to-musl-cross>/gcc-4.9.2/build2
cp x86_64-linux-musl/libgcc/libgcc*.a $CROSS_TOOLS/x86_64-linux-musl/x86_64-linux-musl/lib
From now on I mostly followed http://github.jfet.org/Rust_cross_bootstrapping.html with all occurrences of arm-unknown-linux-gnueabi replaced by x86_64-unknown-linux-musl. First of all, both the build system and the compiler code need to be patched to recognize our new target triple. Also, the build system needs to be tweaked slightly according to the article.
git clone https://github.com/rust-lang/rust
cd rust
git submodule update --init
git checkout dd6c4a8f15bc04dae7720af69d4a534d93c85c0a
git checkout -b musl
You need the patch from https://gist.githubusercontent.com/cl91/bb927df2525738502131/raw/87ada1d66e6129d3c104c16ed48b0891f36c124c/0001-Add-target-x86_64-unknown-linux-musl.patch for rustc and https://gist.githubusercontent.com/cl91/bb927df2525738502131/raw/e695ab47dca4d4d91fe880499ca16e17b2c688f1/0001-Add-support-for-linux-musl-in-build-system.-Fix-glib.patch for llvm.
cd src/llvm
git apply ~/0001-Add-support-for-linux-musl-in-build-system.-Fix-glib.patch
git commit -a -m "Add musl support"
cd ../..
git apply ~/0001-Add-target-x86_64-unknown-linux-musl.patch
git add .
git commit -a -m "Add musl support"
Now we configure rustc:
mkdir -p $CROSS_TOOLS/rust/{var/lib,etc}
./configure --prefix=$CROSS_TOOLS/rust --host=x86_64-unknown-linux-gnu --disable-llvm-assertions --target=x86_64-unknown-linux-gnu,x86_64-unknown-linux-musl --localstatedir=$CROSS_TOOLS/rust/var/lib --sysconfdir=$CROSS_TOOLS/rust/etc
We need to tweak the build system slightly so that it knows where to find our LLVM built for the target triple:
chmod 0644 config.mk
sed 's/x86_64\(.\)unknown.linux.gnu/[chang@arch rust]$ grep 'CFG_LLVM_[BI]' \
sed 's/x86_64\(.\)unknown.linux.gnu/x86_64\1unknown\1linux\1musl/g' \
>> config.mk
We need to build LLVM for both the host and the cross target. We first build it for the host triple:
cd x86_64-unknown-linux-gnu/llvm
../../src/llvm/configure --enable-target=x86_64 --enable-optimized --disable-assertions --disable-docs --enable-bindings=none --disable-terminfo --disable-zlib --disable-libffi --with-python=/usr/bin/python2.7
make -j$(nproc)
And then for the target triple:
cd x86_64-unknown-linux-musl
mkdir rustllvm # you need this for rustc build system to work
mkdir llvm
cd llvm
../../src/llvm/configure --enable-target=x86_64 --enable-optimized --disable-assertions --disable-docs --enable-bindings=none --disable-terminfo --disable-zlib --disable-libffi --with-python=/usr/bin/python2.7 --host=x86_64-unknown-linux-musl --target=x86_64-unknown-linux-musl
make -j$(nproc)
According to the article, you need to manually tweak the build system to invoke the llvm-config tool built for the host triple rather than the target triple:
cd Release/bin
mv llvm-config llvm-config-musl
ln -s ../../BuildTools/Release/bin/llvm-config .
# (Now test to be sure this works.)
./llvm-config --cxxflags
# (You should see some CXX flags printed out here!)
Finally, we are ready to build rustc. This takes a long long time. So you may need to grab some coffee, or do some squats to kill the time.
cd ../../../.. # this brings us back to the rust source directory
make -j$(nproc)
By default, when generating executables, rustc will statically link all rust dependencies but dynamically link all external dependencies, including system libraries such as libc and libgcc (on gcc-based systems).
Therefore we need to manually invoke the cross linker to link against musl statically. As a test, let's build the hello world program:
fn main() {
println!("hello, world!");
}
We first compile it into object file:
export RUSTSRC=$HOME/src/rust-lang
export RUSTLIB=$RUSTSRC/x86_64-unknown-linux-gnu/stage2/lib/rustlib/x86_64-unknown-linux-musl/lib
export RUSTEXE=$RUSTSRC/x86_64-unknown-linux-gnu/stage2/bin/rustc
LD_LIBRARY_PATH=$RUSTLIB $RUSTEXE -C opt-level=3 -C lto --target=x86_64-unknown-linux-musl --emit=obj -o a.o a.rs
You need to modify the $RUSTSRC accordingly.
We then invoke the cross linker directly:
x86_64-unknown-linux-musl-gcc a.o -nodefaultlibs -Wl,-s -Wl,-Bstatic $RUSTLIB/liballoc-4e7c5e5c.rlib $RUSTLIB/libstd-4e7c5e5c.rlib $RUSTLIB/libmorestack.a $RUSTLIB/libcompiler-rt.a -lgcc_eh -lc
The -nodefaultlibs tells the cross linker to not link against libc. We then manually add libc and other dependencies into the executable. We use -Wl,-Bstatic to tell the linker to link statically.
Hooray! We have now built a rusc program that is statically linked against libc and all dependencies, and therefore has absolutely no dependencies. To verify this:
ldd a.out
You should be able to see not a dynamic executable. Run it:
a.out
=> hello, world!
Following this path, it is actually possible to produce a statically-linked rustc. However, the resulting compiler is not functional as the compiler plugins require dynamic linking.
The dynamically linked compiler does work. From the cross rustc built above, we can bootstrap a native rustc that runs on a musl-based environment.
LD_LIBRARY_PATH=$RUSTLIB $RUSTEXE --cfg stage2 -O -cfg rtopt \
-C linker=x86_64-unknown-linux-musl-g++ -C ar=x86_64-unknown-linux-musl-ar \
--cfg debug -C prefer-dynamic --target=x86_64-unknown-linux-musl \
-o x86_64-unknown-linux-gnu/stage2/lib/rustlib/x86_64-unknown-linux-musl/bin/rustc --cfg rustc ./src/driver/driver.rs
LD_LIBRARY_PATH=$RUSTLIB $RUSTEXE --cfg stage2 -O -cfg rtopt \
-C linker=x86_64-unknown-linux-musl-g++ -C ar=x86_64-unknown-linux-musl-ar \
--cfg debug -C prefer-dynamic --target=x86_64-unknown-linux-musl \
-o x86_64-unknown-linux-gnu/stage2/lib/rustlib/x86_64-unknown-linux-musl/bin/rustdoc --cfg rustdoc ./src/driver/driver.rs
Tar them up according to the directory structure of a standard rustc distribution, and then copy it into a musl-native environment (for instance, Alpine Linux).
You then need a native musl toolchain. You can get it from a musl-based distro such as Alpine Linux, or you can build for yourself. The musl-cross script can be modified very slightly to do this. All you need to do is to modify the --host and --target passed to the configure script in binutils and gcc to be x86_64-unknown-linux-musl. You may need to patch gmp, mpr, and mpfr (patches are in this gist repo).
You can then use the native rustc to boostrap a rustc build in a musl environment. To have a reasonable assurity that the resulting compiler indeed works, I ran the test suite for the compiler and the standard library. Stack overflow detection tests failed, but all other compiler tests are successful. The logs can be seen at https://gist.github.com/cl91/d24bb5b75248b0214fa3
Why do you build LLVM with musl? It would make most sense to just use the host rustc.
As a side note, some distributions (like Debian derivatives) offer a
musl-gccpackage, which is more convenient to use if invoking linker manually. You're saved from building your own musl as well.