Skip to content

Instantly share code, notes, and snippets.

@acmorrow
Created January 23, 2021 14:50
Show Gist options
  • Save acmorrow/f729ee772b6e1446e0887de984452512 to your computer and use it in GitHub Desktop.
Save acmorrow/f729ee772b6e1446e0887de984452512 to your computer and use it in GitHub Desktop.
/usr/bin/time /opt/mongodbtoolchain/v3/bin/python3 buildscripts/scons.py --dbg=on --opt=on --link-model=dynamic --implicit-cache --cache --experimental-visibility-support=on --install-action=hardlink --build-tools=next --disable-warnings-as-errors --modules= --build-dir=$(pwd)/build/toolpy37 --cache-dir=$(pwd)/cache -j300 install-unittests --debug=time 2>&1 | tee toolpy37.cleanbuild.log
/usr/bin/time python3.8 buildscripts/scons.py --dbg=on --opt=on --link-model=dynamic --implicit-cache --cache --experimental-visibility-support=on --install-action=hardlink --build-tools=next --disable-warnings-as-errors --modules= --build-dir=$(pwd)/build/syspy38 --cache-dir=$(pwd)/cache -j300 install-unittests --debug=time 2>&1 | tee syspy38.cleanbuild.log
\rm -rf build/toolpy37/cached build/toolpy37/install
/usr/bin/time /opt/mongodbtoolchain/v3/bin/python3 buildscripts/scons.py --dbg=on --opt=on --link-model=dynamic --implicit-cache --cache --experimental-visibility-support=on --install-action=hardlink --build-tools=next --disable-warnings-as-errors --modules= --build-dir=$(pwd)/build/toolpy37 --cache-dir=$(pwd)/cache -j12 install-unittests --debug=time 2>&1 | tee toolpy37.cacheextractbuild.log
\rm -rf build/syspy38/cached build/syspy38/install
/usr/bin/time python3.8 buildscripts/scons.py --dbg=on --opt=on --link-model=dynamic --implicit-cache --cache --experimental-visibility-support=on --install-action=hardlink --build-tools=next --disable-warnings-as-errors --modules= --build-dir=$(pwd)/build/syspy38 --cache-dir=$(pwd)/cache -j12 install-unittests --debug=time 2>&1 | tee syspy38.cacheextractbuild.log
/usr/bin/time /opt/mongodbtoolchain/v3/bin/python3 buildscripts/scons.py --dbg=on --opt=on --link-model=dynamic --implicit-cache --cache --experimental-visibility-support=on --install-action=hardlink --build-tools=next --disable-warnings-as-errors --modules= --build-dir=$(pwd)/build/toolpy37 --cache-dir=$(pwd)/cache -j12 install-unittests --debug=time 2>&1 | tee toolpy37.noopbuild.log
/usr/bin/time python3.8 buildscripts/scons.py --dbg=on --opt=on --link-model=dynamic --implicit-cache --cache --experimental-visibility-support=on --install-action=hardlink --build-tools=next --disable-warnings-as-errors --modules= --build-dir=$(pwd)/build/syspy38 --cache-dir=$(pwd)/cache -j12 install-unittests --debug=time 2>&1 | tee syspy38.noopbuild.log
@acmorrow
Copy link
Author

acmorrow commented Jan 23, 2021

  • I'm comparing "mongodb toolchain" py37 to system py38. It is possible we just did something dumb in our py37 build that accounts for the difference. If someone who has a capable system with both 37 and 38+ wants to try it out, that'd be a good experiment. Anyway, you will need to adjust the scripts to get your own py37 unless you have our toolchain, which we don't really give out.
  • You will need a checkout of https://github.com/mongodb/mongo. I did this experiment on or around 75f5e2bb5240a7061b3d40c59a5238590221813c.
  • The cleanbuild runs say -j300 because I have access to an icecream cluster. You will need to turn that down to something your local system can support.
  • The cacheextract and noop builds say -j12 because that is how many local cores I have. It is interesting to think about what running a pure cache extraction with -j300 does when you only have 12 local cores. Perhaps the cache extraction should be protectable with a semaphore to constrain local concurrency.
  • You will need a properly constructed python environment. I didn't use a virtualenv here because I want two compare different pythons, so I have the contents of etc/pip/compile-requirements.txt from the mongodb tree loaded up with pip install --user -r.
  • The install-unittests build is a pretty big target. You could go with something smaller with something like install-visibility-test, or install-all to get as big as possible.
  • This uses the vendored copy of SCons, which is 3.x. Change buildscripts/scons.py to something else if you want to try invoking different SConses.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment