01-building-mlvm-openjdk.md

Building MLVM version of OpenJDK (Java 1.8) on Mac OS X

Take a look at the comments and code in update.sh.

Requires a working version of Java 1.8.

I use create_tarball.sh to make a copy of the build to share.

OpenJDK 1.7 builds for Mac OS X:

• Building OpenJDK (Java 1.7) on Mac OS X
• OpenJDK 7 and 8 for OS/X Snow and Lion

update.sh

#!/bin/sh
#
# Building the Java 1.8 Da Vinci Machine (mlvm) on Mac OS X
#
# filename: update.sh
# latest version available here: http://gist.github.com/243072
#
# Stephen Bannasch
#
# When this script runs it assumes:
#   Apple's Mac OX X Xcode developer tools (and command-line tools) are installed
#   hg (mercurial) is installed
#   The forest extension to hg is installed
#   The hsx/hotspot-comp port forest is checked out into: ./sources/
#   The mlvm patchsets are checked out into: ./patches/
#   A working version of Java 1.8 is installed.
#   jtreg 4.1 is installed and the linux/jtreg shell script is on the path
#     see: http://openjdk.java.net/jtreg/
#
# References:
#   http://wikis.sun.com/display/mlvm
#   http://wikis.sun.com/display/mlvm/Building
#   http://hg.openjdk.java.net/mlvm/mlvm/file/tip/README.txt
#   http://bit.ly/8TKqcS
#   http://code.google.com/p/openjdk-osx-build/
#   http://hg.openjdk.java.net/jdk7/build/raw-file/tip/README-builds.html
#   http://sourceforge.net/projects/jibx/files/
#   http://blogs.sun.com/jjg/entry/building_javac_for_jdk7
#   http://blogs.sun.com/kto/entry/anatomy_of_the_jdk_build
#
#   http://hgbook.red-bean.com/read/advanced-uses-of-mercurial-queues.html
#
# Prerequisites:
#   Mac OS X Java 1.8
#     A working version of Java 1.8
#     Binary distributions are available here: http://code.google.com/p/openjdk-osx-build/
#
#  Once installed you should be able to successfully run this command:
#     $ `/usr/libexec/java_home --version 1.8`/bin/java -version
#
#   and see something like this:
#     openjdk version "1.8.0-internal"
#     OpenJDK Runtime Environment (build 1.8.0-internal-stephen_2012_03_07_18_02-b00)
#     OpenJDK 64-Bit Server VM (build 23.0-b16, mixed mode)
#
# Apple's Mac OX X Xcode developer tools (and command-line tools) are installed
#   Newer versions of Xcode might require this command to select the path to the Xcode
#   folder for Xcode command-line BSD tools:
#
#     sudo xcode-select -switch /Applications/Xcode.app/Contents/Developer
#
# Basic setup after installing prerequisites:
#   mkdir mlvm
#   cd mlvm
#   curl http://gist.github.com/raw/243072/update.sh -o update.sh
#   hg fclone http://hg.openjdk.java.net/hsx/hotspot-comp sources
#   hg fclone http://hg.openjdk.java.net/mlvm/mlvm patches
#
# Building:
#   cd mlvm
#   ./update.sh
#
# After building this script will first test if
# the new build can report it's version number:
#
#   $ build/macosx-x86_64/j2sdk-image/bin/java -version
#   openjdk version "1.8.0-internal"
#   OpenJDK Runtime Environment (build 1.8.0-internal-stephen_2012_07_11_11_41-b00)
#   OpenJDK 64-Bit Server VM (build 24.0-b16, mixed mode)
#
# Then it will use jtreg to run the following jtreg-based unit tests:
#
#   jdk/test/java/lang/invoke/
#   jdk/test/java/dyn/CoroutineTest.java
#
# Additionaly Swing is tested by running running SwingSet2.jar.
# This makes sure the X11 freetype libraries are being integrated correctly
#
# More about MVLM Guarded Patches
#
# Enable a guarded patch:
#   +foo
#   -/foo    *this form is a double negative. It may be needed (besides plain +foo)
#             due to the rules hg uses to total up the effects of all the guards
#
# Disable a guarded patch:
#   /foo
#
# See:
#   hg qguard --help
#   hg qguard -l
#
# Here's a list of the possible guards for the different directories of
# the build (produced Jul 15 2012):
#
# sources
#   [sources ]$ hg qguard -l
#   macosx-7113349.patch: -/macosx +1533dfab9903
#
# jdk
#    [jdk ]$ hg qguard -l
#    meth-lazy-7023639.patch: -/meth +78f1f4e4e9c7
#    meth-lazy-7023639.init.patch: -/meth +78f1f4e4e9c7
#    meth-lazy-7023639.bmh.patch: -/meth +78f1f4e4e9c7
#    meth-7177472.patch: -buildable
#    indy.pack.patch: -/meth +78f1f4e4e9c7 -buildable
#    meth.patch: -/meth +78f1f4e4e9c7
#    dyncast.patch: +dyncast -/dyncast
#    inti.patch: +inti -/inti -buildable
#    callcc_old.patch: +callcc_old -/callcc_old
#    continuation.patch: +continuation -/continuation
#    tailc.patch: +tailc -/tailc
#    anonk.patch: -/anonk +78f1f4e4e9c7
#    coro.patch: +coro -/coro
#    tuple-tsig.patch: -/tuple -testable
#
# hotspot
#    [hotspot ]$ hg qguard -l
#    meth-lazy-7023639.patch: -/meth +ed21db7b3fda
#    meth-lazy-7023639.review.patch: -/meth +ed21db7b3fda
#    meth.patch: -/meth +ed21db7b3fda
#    meth.proj.patch: -/meth +projects
#    anonk.proj.patch: -/anonk +projects
#    tagu.patch: +tagu -/tagu +ed21db7b3fda -buildable
#    inti.patch: +inti -/inti +d1605aabd0a1 +jdk7-b30 -buildable
#    callcc_old.patch: +callcc_old -/callcc_old +d6d1af32c5f9 -testable
#    continuation.patch: +continuation -/continuation +4afae810a801
#    tailc.patch: +tailc-lazy -/tailc -tailc-eager
#    tailc-eager.patch: +tailc-lazy -/tailc -tailc-lazy
#    hotswap.patch: +hotswap -/hotswap
#    hotswaplight.patch: +hotswaplight -/hotswaplight
#    coro-meth.patch: +coro -/meth -/coro
#    coro.patch: +coro -/coro
#    tuple-tsig.patch: +tuple -/tuple -testable
#
# langtools
#    [langtools ]$ hg qguard -l
#    netbeans-project.patch: +netbeans-project
#    dump-stack-on-error.patch: +dump-stack-on-error
#    indy-bsm-6964498.patch: -/meth +cfc480e5499d -/experimental -testable
#    indy-args-6984311.patch: -/indy +cfc480e5499d -/experimental -testable
#    meth-ldc-6939203.patch: -/meth +cfc480e5499d -/experimental -testable
#    meth-ver-6949040.patch: -/meth +cfc480e5499d -testable
#    meth-anno.patch: -/meth +cfc480e5499d -/experimental -testable
#    dyncast.patch: +dyncast -/dyncast
#    tailc.patch: +tailc -/tailc
#

MLVM=$(pwd)
JDK_UPDATE_VERSION=00
SYSTEM_JAVA_1_8=`/usr/libexec/java_home --version 1.8`
BUILD_JOBS=$(sysctl -n hw.ncpu)
SWINGSET2_JAR=/Users/stephen/dev/java/SwingSet2/SwingSet2.jar

BUILD="build/macosx-x86_64/j2sdk-bundle/jdk1.8.0.jdk"
BUILT_IMAGE="$BUILD/Contents/Home/bin/java"
JTREG_INVOKE="jtreg -XX:+UnlockDiagnosticVMOptions -XX:+UnlockExperimentalVMOptions -ignore:quiet -concurrency:$BUILD_JOBS -jdk:$BUILD/Contents/Home"

#
# Specify which MLVM guarded patches should be applied in the shell variable: guards
#
export davinci=$(pwd)
export guards="buildable testable"

#
# process optional arguments
#
TEST_ONLY=0
UPDATE_SOURCE_AND_PATCHES=1
SOURCE_AND_PATCHES_ONLY=0
CORO=""
while [ "$1" != "" ]; do
  case $1 in
    -t | --test )                   shift
      TEST_ONLY=1
      ;;
    -b | --build-and-test-only )    shift
      UPDATE_SOURCE_AND_PATCHES=0
      ;;
    -s | --source-and-patches-only ) shift
      SOURCE_AND_PATCHES_ONLY=1
      ;;
    -c | --coro )                   shift
      CORO="coro"
      export guards="$guards $CORO"
      ;;
    -h | --help )
      echo '
usage: ./update.sh [[-t, --test ] | [-s, --source-and-patches-only] | [-b, --build-and-test-only] [-c, --coro ] | [-h, --help]]'
      exit
      ;;
  esac
done

# If COMMAND_MODE=legacy then the archive command: 'ar' will not create or update
# catalog entries when creating an archive resulting in errors like this:
#
#   libfdlibm.i586.a, archive has no table of contents
#
# This only appears to be a problem when using iTerm as the terminal program.
# Using the Terminal.app program that comes with Mac OS X sets COMMAND_MODE=unix2003.
# See: http://old.nabble.com/Why-ar-doesn't-call-ranlib-on-Mac--td22319721.html
if [ "$COMMAND_MODE" = "legacy" ]
then
  echo '
*** changing COMMAND_MODE from legacy to unix2003
'
  export COMMAND_MODE=unix2003
fi

########################################
#
# Functions
#
########################################

# --------------------------------------
#
# function: prerequisites
#
# test for prerequisites
#
# --------------------------------------

function prerequisites
{
  if [ ! -e $SWINGSET2_JAR ]
  then
    echo "
*** SwingSet2 jar file not found at this path: $SWINGSET2_JAR
"
    exit
  fi
}

# --------------------------------------
#
# function: updatesource
#
# update source and patch forests and integrate patches into source tree
#
# --------------------------------------

function updatesource
{
  cd $MLVM/sources

  echo "
*** using MLVM Guards: $guards

*** popping previously applied patches to sources/ and jdk8 hsx/hotspot-comp sources: hotspot, jdk, and langtools ..."

  echo '
[sources] '
  hg qpop -a

  echo '
[hotspot] '
  cd hotspot; hg qpop -a; cd -

  echo '
[jdk] '
  cd jdk; hg qpop -a; cd -

  echo '
[langtools] '
  cd langtools; hg qpop -a; cd -

  echo '
*** updating hsx/hotspot-comp sources...
  '
  hg fpull -u

  echo '
*** updating mlvm patchsets ...
  '
  cd ../patches
  hg fpull -u
  cd ..

  echo '
*** integrating mlvm patchsets into hsx/hotspot-comp sources ...

*** creating symbolic links to the patch directories from the corresponding .hg directories of the full source
  '
  bash patches/make/link-patch-dirs.sh sources patches
  ln -s ../../../patches/hotspot sources/hotspot/.hg/patches
  ln -s ../../../patches/jdk sources/jdk/.hg/patches
  ls -il patches/hotspot/series sources/hotspot/.hg/patches/series

  echo '
*** running: hg qselect --pop $guards
  '
  sh patches/make/each-patch-repo.sh "hg qselect --pop $guards" '$(sh $davinci/patches/make/current-release.sh)'

  echo '
*** running: hg qselect; hg qunapplied
  '
  sh patches/make/each-patch-repo.sh "hg qselect; hg qunapplied"

  echo '
*** running: hg update -r
  '
  sh patches/make/each-patch-repo.sh "hg update -r" '$(sh $davinci/patches/make/current-release.sh)'

  echo '
*** running: hg qpush -a
  '
  sh patches/make/each-patch-repo.sh "hg qpush -a"
}

# --------------------------------------
#
# function: build
#
# build MLVM
#
# --------------------------------------

function build
{
  cd $MLVM/sources
  echo '
*** cleaning previous build products ...
  '

  # delete the previous build dir in the background if it exists
  if [ -e build ]
  then
  mv build build.del; rm -rf build.del &
  fi

  # Remove file with versioned name of last build and symbolic link to last build.
  # Used when referring to a build when copying to /usr/local or creating a tarball.
  if [ -e .last_build ]
  then
  echo '
*** removing references to last build
  '
  buildname=`cat .last_build`
  rm -f .last_build
  rm -f $buildname
  fi

  # Save a name with the date to use when referring to this build when
  # copying to ~/Library/Java/JavaVirtualMachines or creating a tarball later.
  buildname=1.8.0_$JDK_UPDATE_VERSION-mlvm-`date "+%Y_%m_%d".jdk`

  echo '
*** building davinci (mlvm) ...
  '
  export LC_ALL=C
  export LANG=C
  unset CLASSPATH
  unset JAVA_HOME
  unset LD_LIBRARY_PATH

  # include the following to enable a FASTDEBUG build:
  #    DEBUG_NAME=fastdebug
  # and remove the following:
  #   SKIP_FASTDEBUG_BUILD=true
  #   SKIP_DEBUG_BUILD=true

  sets="
    LD_LIBRARY_PATH=
    CLASSPATH=
    JAVA_HOME=
    LC_ALL=C
    LANG=C
    LP64=1
    OPENJDK=true
    ARCH_DATA_MODEL=64
    ALT_BOOTDIR=$SYSTEM_JAVA_1_8
    ALT_JDK_IMPORT_PATH=$SYSTEM_JAVA_1_8
    ANT_HOME=/usr/share/ant
    NO_DOCS=true
    NO_DEMOS=true
    NO_SAMPLES=true
    BUILD_LANGTOOLS=true
    BUILD_HOTSPOT=true
    BUILD_JDK=true
    BUILD_CORBA=false
    BUILD_DEPLOY=false
    BUILD_INSTALL=false
    BUILD_JAXP=false
    BUILD_JAXWS=false
    BUILD_SPONSORS=false
    HOTSPOT_BUILD_JOBS=$BUILD_JOBS
    SKIP_FASTDEBUG_BUILD=true
    SKIP_DEBUG_BUILD=true
    SA_APPLE_BOOT_JAVA=true
    ALWAYS_PASS_TEST_GAMMA=true
    ALLOW_DOWNLOADS=true
    CC=$(  xcrun -find gcc )
    CXX=$( xcrun -find g++ )
  "

  echo '
***  executing sets into the environment ...
  '
  # Execute the above sets, into the environment.
  for s in $sets; do eval export $s; done

  echo '
*** preview sets in command line  ...
  '
  # Preview sets in command line
  for s
  do case $s in
     *'[ ;]'*) break;;
     *'='*) eval "$s";;
     *) break;;
     esac
  done

  echo '
*** running make ...
  '
  # Run make, or whatever, in the resulting environment.
  eval "${@-make}"

  echo "
*** create symbolic link to last build and persist $buildname into .last_build
  "
  ln -Ffs $BUILD/ $buildname
  echo $buildname > .last_build
}

# --------------------------------------
#
# function: testbuild
#
# test most recent build
#   1. java -version
#   2. jtreg: jdk/test/java/lang/invoke
#      ** if CORO build enabled (-c, --coro) also: jdk/test/java/dyn/CoroutineTest.java
#   3. SwingSet2.jar
#
# --------------------------------------

function testbuild
{
  cd $MLVM/sources
  if [ -e $BUILT_IMAGE ]
  then
    echo "
*** smoketest build:

$BUILT_IMAGE -version"
    $BUILT_IMAGE -version

    echo "
*** running jdk/test/java/lang/invoke tests:

$JTREG_INVOKE  -v:summary jdk/test/java/lang/invoke/"
    $JTREG_INVOKE  -v:summary jdk/test/java/lang/invoke/

    if [ -n "$CORO" ]
    then
      echo "
*** running jdk/test/java/dyn/CoroutineTest.java test:

$JTREG_INVOKE -v:summary jdk/test/java/dyn/CoroutineTest.java"
      $JTREG_INVOKE -v:summary jdk/test/java/dyn/CoroutineTest.java
    fi

    echo "
*** testing Swing by running SwingSet2.jar:

./$BUILT_IMAGE -jar $SWINGSET2_JAR"
    ./$BUILT_IMAGE -jar $SWINGSET2_JAR&
  else
    echo "$BUILT_IMAGE does not exist, rebuild MLVM with ./update.sh"; exit
  fi
}

# --------------------------------------
#
# function: epilog
#
# describe how to install new jdk bundle
#
# --------------------------------------

function epilog
{
  if [ -e .last_build ]
    then buildname=`cat .last_build`
    else echo "file '.last_build' does not exist"; exit
  fi
  echo "
*** if the build worked install with this command:

cp -r sources/$buildname ~/Library/Java/JavaVirtualMachines

*** after installation the new Java should appear in the list generated when running this command:

/usr/libexec/java_home --version 1.8 -V
  "
}

########################################
#
# Main
#
########################################

prerequisites

if [ "$TEST_ONLY" = "0" ]
then
  if [ "$UPDATE_SOURCE_AND_PATCHES" = "1" ]
  then
    updatesource
  fi
  if [ "$SOURCE_AND_PATCHES_ONLY" = "0" ]
  then
    build
  fi
fi

if [ "$SOURCE_AND_PATCHES_ONLY" = "0" ]
then
  testbuild
  epilog
fi