Skip to content

Instantly share code, notes, and snippets.

@acmiyaguchi

acmiyaguchi/buildprops.json

Last active August 29, 2015 14:26
Show Gist options
  • Save acmiyaguchi/7a546ff246c42ba48dd9 to your computer and use it in GitHub Desktop.
Save acmiyaguchi/7a546ff246c42ba48dd9 to your computer and use it in GitHub Desktop.
Download ZIP
Windows Taskcluster
Raw
buildprops.json
{
"properties": {
"buildnumber": 6573,
"product": "firefox",
"script_repo_revision": "production",
"branch": "try",
"repository": "",
"buildername": "WINNT 5.2 try build",
"basedir": "/c/builds/moz2_slave/try-w32-0000000000000000000000",
"project": "",
"platform": "win32",
"master": "http://buildbot-master87.bb.releng.scl3.mozilla.com:8101/",
"commit_titles": [
"try: -b do -p all -u mochitest-o -t none"
],
"scheduler": "try-firefox",
"repo_path": "try",
"slavename": "b-2008-ix-0062",
"builduid": "a55c9943a9ba404c985f823c90242ee4",
"revision": "b01bbc3bb8ddb47a4c1523805a9ed5b4b693757c"
},
"sourcestamp": {
"repository": "",
"hasPatch": false,
"project": "",
"branch": "try",
"changes": [
{
"category": null,
"files": [
{
"url": null,
"name": "accessible/base/nsAccessibilityService.cpp"
},
{
"url": null,
"name": "accessible/tests/mochitest/treeupdate/test_textleaf.html"
}
],
"repository": "",
"rev": "b01bbc3bb8ddb47a4c1523805a9ed5b4b693757c",
"who": "[email protected]",
"when": 1438714337,
"number": 6182936,
"comments": "try: -b do -p all -u mochitest-o -t none",
"project": "",
"at": "Tue 04 Aug 2015 11:52:17",
"branch": "try",
"revlink": "https://hg.mozilla.org/try/rev/b01bbc3bb8ddb47a4c1523805a9ed5b4b693757c",
"properties": [
[
"commit_titles",
[
"try: -b do -p all -u mochitest-o -t none"
],
"BaseHgPoller"
]
],
"revision": "b01bbc3bb8ddb47a4c1523805a9ed5b4b693757c"
}
],
"revision": "b01bbc3bb8ddb47a4c1523805a9ed5b4b693757c"
}
}
Raw
steps.md 
# Install chocolatey
c:/> @powershell -NoProfile -ExecutionPolicy Bypass -Command "iex ((new-object net.webclient).DownloadString('https://chocolatey.org/install.ps1'))" && SET PATH=%PATH%;%ALLUSERSPROFILE%\chocolatey\bin

choco install visualstudiocommunity2013
choco install vcredist2008 

$client = new-object net.webclient

# Enable dotnet 3.5
# https://technet.microsoft.com/en-us/library/Dn482071.aspx
# install directx sdk
$client.DownloadFile("http://download.microsoft.com/download/A/E/7/AE743F1F-632B-4809-87A9-AA1BB3458E31/DXSDK_Jun10.exe", "DXSDK_Jun10.exe")
Start-Process "DXSDK_Jun10.exe" -ArgumentList "/silent" 

# Install mozillabuild
$client.DownloadFile("https://ftp.mozilla.org/pub/mozilla.org/mozilla/libraries/win32/MozillaBuildSetup-Latest.exe", "MozillaBuildSetup-Latest.exe")
Start-Process "MozillaBuildSetup-Latest.exe"

#install dependenencies for source
choco install nodejs.install 
npm install -g [email protected]

Notes

  • Following both the Win64 Machine Setup on the wiki, and the build prerequisites on MDN.

    I figure that we don't need some of the packages on the first image, so I'm installing dependencies as I go. ffledgling gave me access to one of his images, but I may or may not use it pending on the difficulty of installing the current dependencies. It seems managable so far. I may decide to follow mrrrgn's linux64 script, which seems to be relatively clean. If mozharness ends up being cross platform in terms of configuration, I can probably reuse a lot of it.

    [Mozilla-Central] /testing/docker/desktop-build

  • Running on an m3.large instance. This will probably change to a m3.xlarge instance once I have all the setup done.

    jonasfj> amiyaguchi, for development just launch one... we like c3.xlarge or c3.2xlarge for building... just use whatever you think you'll use in production...

  • Ran out of space on the m3.large instance, which is limited to 35GBs. Followed the directions on how to expand the ebs-volume

Raw
windows_cmd.md 
./mozharness/scripts/fx_desktop_build.py `
--config builds/releng_base_windows_32_builds.py `
--disable-mock `
--no-setup-mock `
--no-clone-tools `
--no-clobber `
--no-update `
--no-upload-files `
--no-sendchange `
--log-level=debug `
--work-dir=C:\build `
--no-action=generate-build-stats

./mozharness/scripts/fx_desktop_build.py \
--config builds/releng_base_windows_32_builds.py \
--disable-mock \
--no-setup-mock \
--no-clone-tools \
--no-clobber \
--no-update \
--no-upload-files \
--no-sendchange \
--log-level=debug \
--work-dir=build \
--no-action=generate-build-stats \
--branch=try \
--build-pool=taskcluster

C:\mozilla-build\msys\bin\bash --login -c "python /c/Users/cltbld/sources/mozharness/scripts/fx_desktop_build.py --config builds/releng_base_windows_32_builds.py --disable-mock --no-setup-mock --no-update --no-upload-files --log-level=debug --branch=try --build-pool=taskcluster"

python mozharness/scripts/fx_desktop_build.py --config builds/releng_base_windows_32_builds.py --disable-mock --no-setup-mock --no-update --no-upload-files --no-clobber --no-checkout-sources --log-level=debug --branch=try --build-pool=taskcluster

Where are the buildprops.json made? I need a way to create them before they are made so I can actually do stuff.

You must run the mozharness script inside of a bash environment that has been started from a cmd, not powershell. Otherwiset the path environmental variable for %system32% doesn't get replaced.

eyJhbGciOiJIUzI1NiJ9.eyJpc3MiOiJyYTIiLCJqdGkiOiJ0NjIiLCJ0eXAiOiJ1c3IifQ.ILHNmWvSVN93bd7ioPd_y71Tyn0-4MjJCtMs3laa6DU

@acmiyaguchi
Copy link
Author

eyJhbGciOiJIUzI1NiJ9.eyJpc3MiOiJyYTIiLCJqdGkiOiJ0NjMiLCJ0eXAiOiJ1c3IifQ.X3kv5j2XjNBng6U-deWKY3cwL8vwt_Z5c63obYxYCC8

@acmiyaguchi
Copy link
Author 

{'TMP': 'C:\\Users\\cltbld\\AppData\\Local\\Temp', 'USERNAME': 'cltbld', 'COMPUT
ERNAME': 'B-2008-EC2-LOAN', 'VS110COMNTOOLS': 'C:\\Program Files (x86)\\Microsof
t Visual Studio 11.0\\Common7\\Tools\\', 'USERDOMAIN': 'B-2008-EC2-LOAN', 'VS100
COMNTOOLS': 'c:\\Program Files (x86)\\Microsoft Visual Studio 10.0\\Common7\\Too
ls\\', 'KTS_VERSION': '1.19c', 'PSMODULEPATH': 'C:\\windows\\system32\\WindowsPo
werShell\\v1.0\\Modules\\', 'COMMONPROGRAMFILES': 'C:\\Program Files (x86)\\Comm
on Files', 'PROCESSOR_IDENTIFIER': 'Intel64 Family 6 Model 62 Stepping 4, Genuin
eIntel', 'PROGRAMFILES': 'C:\\Program Files (x86)', 'PROCESSOR_REVISION': '3e04'
, 'DNSSUFFIX': 'winbuild.releng.scl3.mozilla.com', 'PATH': 'C:\\windows\\system3
2;C:\\windows;C:\\windows\\System32\\Wbem;C:\\windows\\System32\\WindowsPowerShe
ll\\v1.0\\;C:\\mozilla-build;C:\\mozilla-build\\python27;C:\\mozilla-build\\pyth
on27\\Scripts;C:\\mozilla-build\\msys\\bin;C:\\mozilla-build\\vim\\vim72;C:\\moz
illa-build\\wget;C:\\mozilla-build\\info-zip;C:\\CoreUtils\\bin;C:\\mozilla-buil
d\\buildbotve\\scripts;c:\\Program Files (x86)\\Microsoft SQL Server\\100\\Tools
\\Binn\\;c:\\Program Files\\Microsoft SQL Server\\100\\Tools\\Binn\\;c:\\Program
 Files\\Microsoft SQL Server\\100\\DTS\\Binn\\;C:\\Program Files (x86)\\Windows
Kits\\8.0\\Windows Performance Toolkit\\;C:\\Program Files (x86)\\Windows Kits\\
8.1\\Windows Performance Toolkit\\;C:\\Program Files\\Microsoft SQL Server\\110\
\Tools\\Binn\\;C:\\Program Files (x86)\\Microsoft SDKs\\TypeScript\\1.0\\;C:\\mo
zilla-build\\hg', 'PROGRAMFILES(X86)': 'C:\\Program Files (x86)', 'WINDOWS_TRACI
NG_FLAGS': '3', 'TEMP': 'C:\\Users\\cltbld\\AppData\\Local\\Temp', 'COMMONPROGRA
MFILES(X86)': 'C:\\Program Files (x86)\\Common Files', 'PROCESSOR_ARCHITECTURE':
 'x86', 'SYSTEMROOT': 'C:\\windows', 'ALLUSERSPROFILE': 'C:\\ProgramData', 'LOCA
LAPPDATA': 'C:\\Users\\cltbld\\AppData\\Local', 'HOMEPATH': '\\Users\\cltbld', '
VS120COMNTOOLS': 'C:\\Program Files (x86)\\Microsoft Visual Studio 12.0\\Common7
\\Tools\\', 'PROGRAMW6432': 'C:\\Program Files', 'TEST1': 'testie', 'LOGONSERVER
': '\\\\B-2008-EC2-LOAN', 'PROMPT': '$P$G', 'SESSIONNAME': 'RDP-Tcp#0', 'PROGRAM
DATA': 'C:\\ProgramData', 'DCLOCATION': 'SCL3', 'PATHEXT': '.COM;.EXE;.BAT;.CMD;
.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.MSC', 'CLIENTNAME': 'Anthonys-MacBoo', 'FP_NO_HOST
_CHECK': 'NO', 'WINDIR': 'C:\\windows', 'WINDOWS_TRACING_LOGFILE': 'C:\\BVTBin\\
Tests\\installpackage\\csilogfile.log', 'HOMEDRIVE': 'C:', 'SYSTEMDRIVE': 'C:',
'DXSDK_DIR': 'C:\\Tools\\sdks\\dx10\\', 'COMSPEC': 'C:\\windows\\system32\\cmd.e
xe', 'NUMBER_OF_PROCESSORS': '2', 'APPDATA': 'C:\\Users\\cltbld\\AppData\\Roamin
g', 'PROCESSOR_LEVEL': '6', 'KTS_HOME': 'C:\\Program Files\\KTS', 'PROCESSOR_ARC
HITEW6432': 'AMD64', 'COMMONPROGRAMW6432': 'C:\\Program Files\\Common Files', 'O
S': 'Windows_NT', 'PUBLIC': 'C:\\Users\\Public', 'USERPROFILE': 'C:\\Users\\cltb
ld'}

@acmiyaguchi
Copy link
Author 

'c:/mozilla-build/python27/python' '-u' 'scripts/scripts/fx_desktop_build.py' '--config' 'builds/releng_base_windows_64_builds.py' '--config' 'balrog/production.py' '--branch' 'mozilla-inbound' '--build-pool' 'production'

@acmiyaguchi
Copy link
Author

https://queue.taskcluster.net/v1/task/eh5dnf7dQWuJEv6SP0bZ8w/artifacts/public/hg.mozilla.org/mozilla-central.tar.gz

@acmiyaguchi
Copy link
Author

  • Download buildbotve folder and install into mozilla-build
  • Install pywin32 for 32bit windows
  • Download tooltool
  • make clean
  • make

@acmiyaguchi
Copy link
Author

22:22:09 INFO - Copy/paste: c:\Users\Administrator\sources\build\venv\Scripts\pip install --download-cache c:\Users\Administrator\sources\build\venv\cache --timeout 120 requests==2.2.1

22:22:10 ERROR - Traceback (most recent call last):
22:22:10 INFO - File "c:\Users\Administrator\sources\build\venv\Scripts\pip-script.py", line 9, in
22:22:10 INFO - load_entry_point('pip==0.8.2', 'console_scripts', 'pip')()
22:22:10 INFO - File "c:\Users\Administrator\sources\build\venv\lib\site-packages\distribute-0.6.14-py2.7.egg\pkg
_resources.py", line 305, in load_entry_point
22:22:10 INFO - return get_distribution(dist).load_entry_point(group, name)
22:22:10 INFO - File "c:\Users\Administrator\sources\build\venv\lib\site-packages\distribute-0.6.14-py2.7.egg\pkg
_resources.py", line 2244, in load_entry_point
22:22:10 INFO - return ep.load()
22:22:10 INFO - File "c:\Users\Administrator\sources\build\venv\lib\site-packages\distribute-0.6.14-py2.7.egg\pkg
resources.py", line 1954, in load
22:22:10 INFO - entry = import(self.module_name, globals(),globals(), ['name'])
22:22:10 INFO - File "c:\Users\Administrator\sources\build\venv\lib\site-packages\pip-0.8.2-py2.7.egg\pip__init
_.py", line 10, in
22:22:10 INFO - from pip.basecommand import command_dict, load_command, load_all_commands, command_names
22:22:10 INFO - File "c:\Users\Administrator\sources\build\venv\lib\site-packages\pip-0.8.2-py2.7.egg\pip\basecom
mand.py", line 6, in
22:22:10 INFO - import socket
22:22:10 INFO - File "c:\mozilla-build\python\Lib\socket.py", line 47, in
22:22:10 INFO - import _socket
22:22:10 INFO - ImportError: DLL load failed: The specified procedure could not be found.
22:22:10 WARNING - Return code: 1

@acmiyaguchi
Copy link
Author

The windows 2008 server edition serves its own version of python27.dll that gets served in the registry somewhere. It needs to be removed before the install will work properly.

@acmiyaguchi
Copy link
Author

General

  • What does the buildbot master set up?
  • I've heard that the build infrastructure has quite the history behind it. Has mozharness depreciated much of the buildbot functionality? What does buildbot still do, and what dependencies are left behind.
  • How are the mercurial repositories set up? How are they updated from run to run if the slaves are cleaned up every time.

Specific

  • Can you explain buildprops.json: Its generated by the buildbot master?
  • What is the buildbotve, and why is it necessary.
  • Is there a specific version of mozilla-build that is running on the slave images at the moment? Why is there such a difference between lastest mozilla build and the 2008 server image?

@acmiyaguchi
Copy link
Author

https://api.pub.build.mozilla.org/

@acmiyaguchi
Copy link
Author

@acmiyaguchi
Copy link
Author

https://pastebin.mozilla.org/8843093

@acmiyaguchi
Copy link
Author

export SCCACHE_DIR to use local directory, which doesn't use boto for uploads. See https://bugzilla.mozilla.org/show_bug.cgi?id=1000718

@acmiyaguchi
Copy link
Author

A minor update with progress on this issue.

With the image I got from the relops (spot-y-2008-2015-08-13-08-57, a try image), I've managed to get closer to a working builder. Currently, I am still unable to build anything successfully.

I'll outline a few of the work arounds I've had to make in order to get a semi working build.

  • Update mozilla-build from 1.9 to 1.10
    This resolves a puzzling issue where mach configure (and in turn mach build) fails to read the nightly mozconfig pulled from browser/config/mozconfigs/win32/nightly during a mozharness build. The error looks like the following when run with start-shell-msvc-2013.bat.

    0:07.47 client.mk:117: *** missing separator (did you mean TAB instead of 8 spaces?). Stop.

When started with start-shell-l10n.bat, configuration succeeds, but compilation fails since none of the compilers have been added to the path.

  • Disable remote sscache (to a certain extent)
    This involves setting SCCACHE_DIR= to a local directory, otherwise boto will fail due to authorization errors. This is probably due to the lack of a .boto with valid keys, which can otherwise be solved later.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment