chrome @ home

notes.md

Notes on compiling chromium using an EC2 cluster

This note lays out the machine specifications and configurations used to get a distributed Chome build working on cloud compute (in this case AWS EC2) using the icecc tool.

These links were a good starting point:

• https://chromium.googlesource.com/chromium/src/+/master/docs/linux_faster_builds.md
• https://github.com/icecc/icecream
• http://mkollaro.github.io/2015/05/08/compiling-chromium-with-clang-and-icecc/

but I couldn't fine a full walk through, so hopefully we can make it easier for others.

(with @maxogden)

Machines we need to orchestrate

The idea behind iccee is that one machine acts as a scheduler, and one or more machines do the compilation via the icecc daemon. In a typical configuration, you'd setup one machine with the shceduler, and one or more machines with daemens. One thing I found confusing at first: it's ok to run a daemon on the scheduler! It's optional, but useful in testing, because it means you can get a full distributed build working with just two nodes. Finally, you trigger the build itself from a machine on the same network that can also be running a daemon. I think running a daemon on this machine is optional too, but againt this lets you test with a minimal cluster.

I was able to get it all working on cloud compute (at least EC2) so long as the neccessary ports were open on all the nodes. It was also very useful to run the monitoring tool icemon on one of the nodes (I ran it on the scheduler), but make sure you have -X forwarding as it's GUI-only.

I used the following set of machines:

SCHEDULER

• runs the icecc scheduler
• runs an icecc deamon (optional)

MASTER

• runs ninja
• runs an icecc daemon (optional)

WORKER

• runs an icecc daemon

WORKER

• runs an icecc daemon

etc

This should just require two Dockerfiles, one for the MASTER, and one for the SCHEDULER + WORKERs. See comment below.

Instance types

The following worked fine:

ubuntu aws image, m3.large, 100GB storage

Make sure to open the following ports:

80, 8765, 10245

Machine configuration

MASTER

sudo apt-get update

sudo apt-get install git

git clone https://chromium.googlesource.com/chromium/tools/depot_tools.git

echo 'export PATH=$PATH:/home/ubuntu/depot_tools' >> ~/.bash_profile

source ~/.bash_profile

fetch --nohooks --no-history chromium

echo 'deb http://us.archive.ubuntu.com/ubuntu/ trusty multiverse' | sudo tee -a /etc/apt/sources.list echo 'deb-src http://us.archive.ubuntu.com/ubuntu/ trusty multiverse' | sudo tee -a /etc/apt/sources.list echo 'deb http://us.archive.ubuntu.com/ubuntu/ trusty-updates multiverse' | sudo tee -a /etc/apt/sources.list echo 'deb-src http://us.archive.ubuntu.com/ubuntu/ trusty-updates multiverse' | sudo tee -a /etc/apt/sources.list

cd src sudo ./build/install-build-deps.sh

sudo apt-get install icecc

gclient runhooks

SCHEDULER

sudo apt-get update

sudo apt-get install icecc icecc-monitor

WORKER

(cna be same as SCHEDULER, though the monitor is optional)

Job running

To run on the shceduler:

icecc-scheduler -n chrome -u ubuntu -vv

sudo iceccd -u ubuntu -vv -s SCHEDULER_IP -n chrome

To run on each worker:

sudo iceccd -u ubuntu -vv -s SCHEDULER_IP -n chrome

To run on the master (once the rest is setup):

sudo iceccd -u ubuntu -vv -s SERVER_IP -n chrome

export GYP_DEFINES="$GYP_DEFINES clang=0 linux_use_debug_fission=0 linux_use_bundled_binutils=0"

ninja -j 10 -C out/Release chrome

Other notes:

In debugging how all this works, it was very useful to run the monitoring tool icemon (great name!)

AFAIK this tool is GUI only, so I just used -X forwarding and ran it on the scheduler with

icemon -n chrome

work in progress Dockerfiles (one with icecc, one with icecc and chromium source + depot_tools)

docker pull maxogden/icecc:

FROM ubuntu:14.04

RUN echo 'deb http://us.archive.ubuntu.com/ubuntu/ trusty multiverse' >> /etc/apt/sources.list
RUN echo 'deb-src http://us.archive.ubuntu.com/ubuntu/ trusty multiverse' >> /etc/apt/sources.list
RUN echo 'deb http://us.archive.ubuntu.com/ubuntu/ trusty-updates multiverse' >> /etc/apt/sources.list
RUN echo 'deb-src http://us.archive.ubuntu.com/ubuntu/ trusty-updates multiverse' >> /etc/apt/sources.list
RUN sudo apt-get update -y
RUN sudo apt-get install git icecc python -y

docker pull maxogden/chromium-icecream:

FROM ubuntu:14.04

RUN echo 'deb http://us.archive.ubuntu.com/ubuntu/ trusty multiverse' >> /etc/apt/sources.list
RUN echo 'deb-src http://us.archive.ubuntu.com/ubuntu/ trusty multiverse' >> /etc/apt/sources.list
RUN echo 'deb http://us.archive.ubuntu.com/ubuntu/ trusty-updates multiverse' >> /etc/apt/sources.list
RUN echo 'deb-src http://us.archive.ubuntu.com/ubuntu/ trusty-updates multiverse' >> /etc/apt/sources.list
RUN sudo apt-get update -y
RUN sudo apt-get install git icecc python -y

WORKDIR /root
RUN git clone https://chromium.googlesource.com/chromium/tools/depot_tools.git
RUN echo 'export PATH=$PATH:/root/depot_tools' >> /root/.bashrc
ENV PATH=$PATH:/root/depot_tools
RUN fetch --nohooks --no-history chromium
# RUN cd src sudo ./build/install-build-deps.sh
# RUN gclient runhooks