ubuntu-18.04-buildbot-setup.md

Setting up new Ubuntu 18.04 buildbot on Azure

Recording of my steps to setup Ubuntu 18.04 buildbot VM on July 22, 2020. The bot is responsible for WebKit and Firefox builds.

1. Install clang-9. Default clang-6 is too old to compile Firefox - we need a new one.

~$ sudo apt-get install clang-9
~$ sudo update-alternatives --install /usr/bin/clang++ clang++ /usr/bin/clang++-9 100
~$ sudo update-alternatives --install /usr/bin/clang clang /usr/bin/clang-9 100

2. Azure VM has /mnt that belongs to root. Create a cron folder that will hold all our cron-based processing

~$ cd /mnt
/mnt$ sudo mkdir cron
/mnt$ sudo chown azureuser:azureuser cron

3. Clone Playwright into cron directory

/mnt$ cd /mnt/cron
/mnt/cron$ git clone https://github.com/microsoft/playwright

4. Install NVM to manage Node.js + NPM versions. Instructions generally live at NVM official readme. As of July 2020, I did the following:

/mnt/cron$ curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.35.3/install.sh | bash
/mnt/cron$ source ~/.bashrc # to get updated bin paths

5. Install Node.js + Npm using NVM

/mnt/cron$ nvm install stable

6. Just in case, install all dependencies for Playwright project

/mnt/cron$ cd /mnt/cron/playwright
/mnt/cron/playwright$ npm install

7. Prepare checkouts for both WebKit and Firefox browsers (this takes quite a while!).

/mnt/cron/playwright$ ./browser_patches/prepare_checkout.sh webkit
/mnt/cron/playwright$ ./browser_patches/prepare_checkout.sh firefox

8. Install dependencies for WebKit GTK and WebKit WPE

/mnt/cron/playwright$ ./browser_patches/webkit/checkout/Tools/gtk/install-dependencies
/mnt/cron/playwright$ ./browser_patches/webkit/checkout/Tools/wpe/install-dependencies

9. Build WebKit

/mnt/cron/playwright$ ./browser_patches/webkit/build.sh

Succeeded! 🎉 Webkit browser toolchain is ready.

10. Bootstrap env for Firefox build

/mnt/cron/playwright$ cd ./browser_patches/firefox/checkout/
/mnt/cron/playwright/browser_patches/firefox/checkout$ ./mach bootstrap

11. Build Firefox

/mnt/cron/playwright/browser_patches/firefox/checkout$ cd /mnt/cron/playwright
/mnt/cron/playwright$ ./browser_patches/firefox/build.sh

❌ The compilation failed for me with the following error:

0:08.85 error: options `-C embed-bitcode=no` and `-C lto` are incompatible
0:09.04 error: could not compile `gkrust`.

12. Attempt to re-bootstrap environment with recent firefox version (btw we need to roll Firefox!) and build it.

/mnt/cron/playwright$ cd ./browser_patches/firefox/checkout
/mnt/cron/playwright/browser_patches/firefox/checkout$ git checkout beta
/mnt/cron/playwright/browser_patches/firefox/checkout$ ./mach bootstrap
/mnt/cron/playwright/browser_patches/firefox/checkout$ ../build.sh

13. The build failed since it wasn't a clean build.. Cleaned up, switched back to playwright-build branch and try build it with the tip-of-tree toolchains. I remember this helped once.

/mnt/cron/playwright/browser_patches/firefox/checkout$ git checkout playwright-build
/mnt/cron/playwright/browser_patches/firefox/checkout$ ../clean.sh
/mnt/cron/playwright/browser_patches/firefox/checkout$ ../build.sh

❌ No luck though - it failed again with the same error.

60:01.15 error: options `-C embed-bitcode=no` and `-C lto` are incompatible
60:01.15 error: could not compile `gkrust`.

14. Ok, let's try compiling tip-of-tree again, this time with clean-build. Maybe we should roll Firefox first? According to rust update policy for firefox, it looks like Rust version is not hermetic to the build and should be managed separately.

/mnt/cron/playwright/browser_patches/firefox/checkout$ git checkout beta
/mnt/cron/playwright/browser_patches/firefox/checkout$ ../clean.sh
/mnt/cron/playwright/browser_patches/firefox/checkout$ ../build.sh

❌ Build Failed with the same error.

Reading the rust update policy for firefox accurately, it looks like we should use a particular version of rust to compile browser. Since beta at this moment is 79, which requires rust 1.44, and I currently have 1.45 used - this might be the problem.

15. Install rust 1.42 (known to compile our playwright-build branch of firefox), re-bootstrap and try compiling our branch with it

/mnt/cron/playwright/browser_patches/firefox/checkout$ rustup default 1.42.0
/mnt/cron/playwright/browser_patches/firefox/checkout$ git checkout playwright-build
/mnt/cron/playwright/browser_patches/firefox/checkout$ ./mach bootstrap
/mnt/cron/playwright/browser_patches/firefox/checkout$ ../clean.sh
/mnt/cron/playwright/browser_patches/firefox/checkout$ ../build.sh

🎉 Yay! 🎉 Firefox browser toolchain is ready!

TAKEAWAY rust version has to be managed separately and should match Firefox matrix at rust update policy for firefox

16. Install Azure CLI using official docs

/mnt/cron/playwright$ curl -sL https://aka.ms/InstallAzureCLIDeb | sudo bash

17. Setup crontab to run buildbot scripts. Important parts here are:
    • MAILTO="" to disable mail spam
    • PATH="..." has to be copied from the actual azureuser path, since cron jobs run with a very basic PATH.

MAILTO=""
PATH="/home/azureuser/.cargo/bin:/home/azureuser/.nvm/versions/node/v14.6.0/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin"
AZ_ACCOUNT_KEY="<KEY>"
AZ_ACCOUNT_NAME="<NAME>"
TELEGRAM_BOT_KEY="<TG_KEY>"
*/5 * * * * /mnt/cron/playwright/browser_patches/buildbots/buildbot-ubuntu-18.04.sh >/tmp/buildbot-cronjob.log