Skip to content

Instantly share code, notes, and snippets.

@myui
Created January 5, 2018 12:25
Show Gist options
  • Save myui/1b9991e26271b7052c6914d8cf83ee63 to your computer and use it in GitHub Desktop.
Save myui/1b9991e26271b7052c6914d8cf83ee63 to your computer and use it in GitHub Desktop.

Apache Hivemall Release Guide

This document describes the release process of Apache Hivemall particularly for Release Managers.

For general information of the Apache Incubator release process, please refer Incubator Release Management and ASF Release Poloicy page.

Prerequisites

If it is your first time doing an Apache release, then there is some initial setup involved. Follow this guide for the initial setup.

  1. Notifying the community the overall plan for the release
  2. Ensure JIRA Issues are appropriately tagged for the Release
    • Check this link for CLOSED/RESOLVED issues that does not have FixVersion.
    • Then, ensure that all JIRA issues that are addressed in this release are marked with the release version in the FixVersion field of the issue. Here is an example for v0.5.0 release.

UPDATE CHANGELOG

  1. Create a release note in JIRA following this instructions.
  2. Download a release note as ChangeLog.html using JIRA.

TEST YOUR SETTINGS

Try installing locally artifacts with activation apache-release profile. The following command will build artifacts, sources and sign.

Note: Apache Hivemall keeps compatibility to Java 7.

# JDK 7 is required for packaging
$ export JAVA_HOME=`/usr/libexec/java_home -v 1.7`

# Install xgboost jar to Maven local repository
$ mvn -Pcompile-xgboost validate

# Try to create artifacts
$ mvn -Papache-release clean install

Ensure that all unit tests passes. License check by Apache RAT (mvn apache-rat:check) will be ran by the above command.

Verify Signatures of Release Artifacts

cd target/
for file in `find . -type f -iname '*.asc'`; do
  gpg --verify ${file} 
done

SNAPSHOT RELEASE

PREPARE YOUR POMS FOR RELEASE

1) Switch to master syncing to ASF master

$ git checkout master
$ git fetch
$ git pull # or, git reset --hard asf/master

2) Set version string for a snapshot

$ version=X.Y.Z
# RC should start at 1 and increment if early release candidates fail to release
$ rc=1
# $ echo "${version}-incubating-SNAPSHOT"

$ ./bin/set_version.sh --pom --version "${version}-incubating-SNAPSHOT"

Note: --pom option SHOULD only used for SNAPSHOT release.

$ git commit -a -m "Prepare for the next Snapshot release of X.Y.Z"
$ git push asf master

3) Perform a dryRun

$ version=X.Y.Z
# RC should start at 1 and increment if early release candidates fail to release
$ rc=1
$ next_version=X.(Y+1).Z

# Confirm that version and rc is defined.
$ echo "Release version: ${version}-incubating-rc${rc}"
$ echo "Development version: ${next_version}-incubating-SNAPSHOT"

$ mvn -Papache-release release:prepare \
-DautoVersionSubmodules=true -DdryRun=true \
-Darguments='-Dmaven.test.skip.exec=true' -DskipTests=true -Dmaven.test.skip=true \
-Dtag=v${version}-rc${rc} -DreleaseVersion=${version}-incubating-rc${rc} -DdevelopmentVersion=${next_version}-incubating-SNAPSHOT

Please provide the next SNAPSHOT version for next release WITHOUT -rcX e.g., as follows:

What is the new development version for "Apache Hivemall"?: X.(Y+1).Z-incubating-SNAPSHOT

PUBLISH A SNAPSHOT

$ mvn deploy -Darguments='-Dmaven.test.skip.exec=true' -DskipTests=true -Dmaven.test.skip=true 

Note: You SHOULD verify the deployment under the Maven Snapshot repository at Apache.

PREPARE THE RELEASE

BRANCH THE RELEASE

1) Create a branch

$ git checkout -b vX.Y.Z

**Note: ** Branch name starts with v and does not include -rcX.

2) Send e-mail announcing the release branch

To: [email protected]
CC: [email protected]
Subject: New release branch X.Y.Z

Hello Hivemall developers and friends,

We now have a release branch for X.Y.Z release. Trunk has been bumped to X.Y.Z-SNAPSHOT.

  https://github.com/apache/incubator-hivemall/tree/vX.Y.Z

I'll be going over the JIRAs to move every non-blocker from this release to the next release. Release voting will be posted soon.

PREPARE A RELEASE

1) Set a release version

# $ version=X.Y.Z
# $ rc=1

# Confirm that version and rc is defined.
$ echo ${version}-incubating-rc${rc}
X.Y.Z-incubating-rc1

2) Update version strings in source codes.

$ ./bin/set_version.sh --version "${version}-incubating-rc${rc}"

# Check list of files to be committed.
$ git stauts
$ git commit -a -m "Bumped version string to ${version}-incubating-rc${rc}"

3) Prepare sets the version numbers in POM, creates a tag, and pushes it to git.

$ mvn -Papache-release release:clean release:prepare \
-DautoVersionSubmodules=true -DdryRun=false \
-Darguments='-Dmaven.test.skip.exec=true' -DskipTests=true -Dmaven.test.skip=true \
-Dtag=v${version}-rc${rc} -DreleaseVersion=${version}-incubating-rc${rc} -DdevelopmentVersion=${next_version}-incubating-SNAPSHOT

4) Update version strings for the development deversion

$ ./bin/set_version.sh --version "${next_version}-incubating-SNAPSHOT"

# Check list of files to be committed.
$ git stauts
$ git commit --amend -a

5) Push release branch and tag to remote ASF repository

# Push the release branch
$ git push asf v${version}

# Push the release tag
$ git push asf v${version}-rc${rc}

STAGE THE RELEASE FOR A VOTE

The release will automatically be inserted into a temporary staging repository for you.

$ mvn -Papache-release release:perform \
-Darguments='-Dmaven.test.skip.exec=true' -DskipTests=true -Dmaven.test.skip=true \
-Dgoals=deploy -DlocalRepoDirectory=. -DlocalCheckout=true 

Verify nexus release artifacts

  1. Verify the staged artifacts in the nexus repo
  • Go to https://repository.apache.org/ and login
  • Under Build Promotion, click Staging Repositories
  • In the Staging Repositories tab there should be a line with profile org.apache.hivemall
  • Navigate through the artifact tree and make sure that all javadoc, sources, tests, and jars have .asc (GPG signature) and .md5 files. Refer the ASF page for artifacts verification.
  1. Close the nexus staging repo

Attach signatures for shaded jars

Shaded jars does not have signatures. So, attach signatures to them as follows:

cd target/

# Sign to the artifacts created by maven-shade-plugin
for f in `ls hivemall-*-with-dependencies.jar`; do
  gpg --armor --output ${f}.asc --detach-sig ${f}
  gpg --print-md MD5 ${f} > ${f}.md5
  gpg --print-md SHA1 ${f} > ${f}.sha1
done

# Verify GPG sign
for file in `find . -type f -iname 'hivemall-*-with-dependencies.jar.asc'`; do
  echo ${file}
  gpg --verify ${file} 
  echo
done

Upload the artifacts via subversion to a staging area

  • Prepare release artifacts in SVN repository
# Checkout release SVN repository
$ mkdir -p dist/dev/incubator
$ cd dist/dev/incubator
$ svn co https://dist.apache.org/repos/dist/dev/incubator/hivemall/
$ cd hivemall

# Download release artifacts
$ wget -e robots=off --no-check-certificate \
 -r -np --reject=html,txt,tmp -nH --cut-dirs=7 \
 https://repository.apache.org/content/repositories/orgapachehivemall-1001/org/apache/hivemall/hivemall/${version}-incubating-rc${rc}/
 
# Put ChangeLog
$ cd ${version}-incubating-rc${rc}
# Put ChangeLog generated by JIRA
$ cp ~/Downloads/ChangeLog.html .

# Put Shaded jars
$ cp ~/hivemall/target/hivemall-*-with-dependencies.jar* .
  • Push release arfifacts to ASF svn repository
# cd dist/dev/incubator/hivemall
# ls ${version}-incubating-rc${rc}

svn add ${version}-incubating-rc${rc}/
svn commit -m "Put hivemall version ${version}-incubating-rc${rc} artifacts"

PPMC voting

Create a VOTE email thread on [email protected] to record votes as replies.

To: [email protected]
CC: [email protected]
Subject: [VOTE] Apache Hivemall <release version> Release

Hi all,

Apache Hivmeall 0.5.0 release candidate #1 (the first Apache release!) is now available for a vote
within dev community. 

Links to various release artifacts are given below. Please review and cast your vote.

    - The source tarball, including signatures, digests, ChangeLog, etc.:
      https://dist.apache.org/repos/dist/dev/incubator/hivemall/0.5.0-incubating-rc1/
	- Sources for the release:
	  https://dist.apache.org/repos/dist/dev/incubator/hivemall/0.5.0-incubating-rc1/hivemall-0.5.0-incubating-rc1-source-release.zip
	  https://dist.apache.org/repos/dist/dev/incubator/hivemall/0.5.0-incubating-rc1/hivemall-0.5.0-incubating-rc1-source-release.zip.asc (PGP Signature)
	  https://dist.apache.org/repos/dist/dev/incubator/hivemall/0.5.0-incubating-rc1/hivemall-0.5.0-incubating-rc1-source-release.zip.md5 (MD5 Hash)
    - Git tag for the release:
      https://git-wip-us.apache.org/repos/asf?p=incubator-hivemall.git;a=shortlog;h=refs/tags/v0.5.0-rc1
    - The Nexus Staging URL:
      https://repository.apache.org/content/repositories/orgapachehivemall-1001/
    - KEYS file for verification:
      https://dist.apache.org/repos/dist/dev/incubator/hivemall/KEYS
    - For information about the contents of this release, see:
      https://dist.apache.org/repos/dist/dev/incubator/hivemall/0.5.0-incubating-rc1/ChangeLog.html
      
You can find Podling releases policies in 
  https://incubator.apache.org/policy/incubation.html#releases
  http://www.apache.org/legal/release-policy.html

The vote will be open for at least 72 hours and until necessary number of votes are reached. 
At lease three +1 from PPMC members are required and we welcome your vote.

[ ] +1  approve (Release this package as Apache Hivemall 0.5.0-incubating-rc1)
[ ] +0  no opinion
[ ] -1  disapprove (and reason why)

Here is my +1 (binding).

Thanks,
Makoto

IPMC Voting

What if vote succeed, then vote in [email protected].

To: [email protected]
Subject: [VOTE] Apache Hivemall <release version> Release

Hi all,

The Apache Hivemall community has voted on and approved a proposal to release Apache Hivemall 0.5.0-rc1 (the first Apache release). Apache Hivemall is a library for machine learning for Apache Hive/Spark/Pig, incubating since 2016-09-13.

We now kindly request that the Incubator PMC members review and vote on this incubator release candidate.

The PPMC vote thread is located here:
    <link to the dev voting thread>

Links to various release artifacts are given below.

    - The source tarball, including signatures, digests, ChangeLog, etc.:
      https://dist.apache.org/repos/dist/dev/incubator/hivemall/0.5.0-incubating-rc1/
    - KEYS file for verification:
      https://dist.apache.org/repos/dist/dev/incubator/hivemall/KEYS
    - Git tag for the release:
      https://git-wip-us.apache.org/repos/asf?p=incubator-hivemall.git;a=shortlog;h=refs/tags/v0.5.0-rc1
    - The Nexus Staging URL:
      https://repository.apache.org/content/repositories/orgapachehivemall-1001/
    - For information about the contents of this release, see:
      https://dist.apache.org/repos/dist/dev/incubator/hivemall/0.5.0-incubating-rc1/ChangeLog.html

Please vote accordingly:

[ ] +1  approve (Release this package as Apache Hivemall 0.5.0-incubating-rc1)
[ ] -1  disapprove (and reason why)

The vote will be open for at least 72 hours.

Thanks,
Makoto
on behalf of Apache Hivemall PPMC

Once 72 hours has passed (which is generally preferred) and/or at least three +1 (binding) votes have been cast with no -1 (binding) votes, send an email closing the vote and pronouncing the release candidate a success.

To: [email protected]
Subject: [RESULT][VOTE]: Apache Hivemall <release version> Release

Hi all,

The Apache Hivemall <release version> vote is now closed and has passed as follows:

 [number] +1 (binding) votes
 [number] -1 (binding) votes

The Apache Hivemall (incubating) community will proceed with the release.

Thanks,
Makoto
on behalf of Apache Hivemall PPMC

Finalize release

Update JIRA

Update the JIRA versions page to close all issues, mark the version as "released", and set the date to the date that the release was approved. You may also need to make a new release entry for the next release.

Merge release branch for the next development iteration

If IPMC vote succeed, then merge the release branch into the master branch.

Publish the websit

Update download page etc.

Announcing the release

Make an announcement about the release on the [email protected], [email protected], [email protected], and [email protected] list as per the Apache Announcement Mailing Lists page.

Release Prerequisites

This document describes the initial setup procedures for making a release of Apache Hivemall.

If it is your first time doing an Apache release, then there is some initial setup involved. You should read this page before proceeding.

Software

You would need the following softwares for building and making a release:

  • gpg2
  • git client
  • svn client (svn is still mandatory in the ASF distribution process. Don't ask me why.)
  • JDK 7 (not JDK 8 nor 9 to support Java 7 or later)
  • maven (>=v3.3.1)

Installation

# on Mac
$ brew install gpg gpg-agent pinentry-mac
$ brew intall svn
$ brew install maven
$ brew install npm
$ npm install gitbook-cli -g

Java 7 setup (Optional)

We recommend to use jEnv for Java 7 environment setup on Mac OS X.

$ /usr/libexec/java_home -v 1.7
  /Library/Java/JavaVirtualMachines/jdk1.7.0_80.jdk/Contents/Home
$ jenv add `/usr/libexec/java_home -v 1.7`
$ jenv versions
* system (set by /Users/myui/.jenv/version)
  1.7
  1.7.0.80
  oracle64-1.7.0.80

# configure to use Java 7 for Hivemall
$ cd incubator-hivemall
$ jenv local 1.7
$ java -version
  java version "1.7.0_80"

PGP signing

To perform a release, you need to add signatures to release packages. See the Signing Releases page for information on how to do this.

In a nutshell, you'll need to follow the instructions at How To OpenPGP to generate a new code signing key and publish the public key in various places.

Setting up signing keys

1). Generate a key-pair with gpg using this instruction. The program's default values should be fine. Please use a signing key with an ASF email address (i.e. [email protected]). Generated Keys should be RSA with at least 4096 bits.

$ gpg --full-generate-key

Here is my key.

$ gpg --list-key --keyid-format LONG

pub   rsa4096/93F4D08DC8CE801B 2017-11-01 [SC]
      7A6BA1A10CC6ABF47159152193F4D08DC8CE801B
uid                 [ultimate] Makoto Yui (CODE SIGNING KEY) <[email protected]>
sub   rsa4096/C3F1C8E219A64221 2017-11-01 [E]

Public key is 93F4D08DC8CE801B in the above case.

2). Send your public PGP key to a public keyserver.

$ gpg --keyserver pgp.mit.edu --send-keys <your-public-pgp-key>

3). Update the PGP key fingerprint of your account on id.apache.org. Find your PGP key fingerprint by

$ gpg --fingerprint

4). Update KEYS file in the git repo to your public key be listed in it.

$ export YOUR_NAME="Makoto Yui"
$ (gpg --list-sigs ${YOUR_NAME} && gpg --armor --export ${YOUR_NAME} && echo) >> KEYS

# Update git repository
$ git add KEYS
$ git commit -m "Added the public key of YOUR NAME"
$ git push origin master

5). Add your public key to KEYS file in the subversion repository:

- dev: https://dist.apache.org/repos/dist/dev/hivemall/KEYS
- release: https://dist.apache.org/repos/dist/release/hivemall/KEYS

# checkout dist repos
$ svn co --depth immediates https://dist.apache.org/repos/dist dist
$ cd dist
$ svn up --set-depth infinity dev/incubator/hivemall
$ svn up --set-depth infinity release/incubator/hivemall

# edit KEYS files
$ svn add * --force
$ svn ci -m "Updated KEYS file of Incubator Hivemall" && svn up

6). Once you have followed these instructions, you should have:

* Your public key viewable at https://people.apache.org/keys/committer/your-asf-id.asc
* Your public key also viewable at https://people.apache.org/keys/group/hivemall.asc

Configure PGP signing on git (optional)

After completing this, you should also configure git to use your key for signing.

If your signing key is identified by 01234567, then you can configure git with:

$ git config --global user.signingkey 01234567

If you are using gpg, you'll then need to tell git to use it.

$ git config --global gpg.program gpg

You can enable GPG signing always true on the particular git repository by:

$ git config commit.gpgsign true

To sign all commits by default in any local repository on your computer, run

$ git config --global commit.gpgsign true

Use pientry to omit typing a passphrase for each commit.

$ echo "pinentry-program /usr/local/bin/pinentry-mac" >> ~/.gnupg/gpg-agent.conf
$ echo -e "use-agent\nno-tty" >> ~/.gnupg/gpg.conf

Tips: You may get an error about a passphrase not being provided when signing with git. If this happens try running the command below, which should case the passphrase prompt to show in the terminal.

export GPG_TTY=`tty`

Configure Maven for publishing artifacts

Update your ~/.m2/settings.xml following this instructions.

<settings>
...
  <servers>
    <!-- To publish a snapshot of some part of Maven -->
    <server>
      <id>apache.snapshots.https</id>
      <username> <!-- YOUR APACHE LDAP USERNAME --> </username>
      <password> <!-- YOUR APACHE LDAP PASSWORD (encrypted) --> </password>
    </server>
    <!-- To stage a release of some part of Maven -->
    <server>
      <id>apache.releases.https</id>
      <username> <!-- YOUR APACHE LDAP USERNAME --> </username>
      <password> <!-- YOUR APACHE LDAP PASSWORD (encrypted) --> </password>
    </server>
   ...
  </servers>
</settings>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment