Jenkins Installation Howto

jenkins-pt1

=======================================================================
= LUXURIATING THE BENEFITS OF CONTINUOUS INTEGRATION - A SHORT HOW-TO =
=======================================================================
              2012 Nikos Vasilakis, nvas@seas.upenn.edu

TODO: SVN-option, add a post-commit hook on the svn server


0. Introduction

The document is devided as follows:
  * Installation: Get jenkins up and running
  * Post-Installation: Configure jenkins plug-ins, users and jobs.
  * Project-structure: Augment each project locally in order to
    have it running with jenkins.

1. Installation

To install jenkins we need to add the repository to our machine.

On RedHat-based distributions we issue the following series:

  1. add the repository
      sudo wget -O /etc/yum.repos.d/jenkins.repo \
      http://pkg.jenkins-ci.org/redhat/jenkins.repo
  
  2. add the GPG key
      sudo rpm --import \
      http://pkg.jenkins-ci.org/redhat/jenkins-ci.org.key
  
  3. install jenkins
      sudo yum install jenkins

On Debian-based we issue:

  1. fetch the key
      wget -q -O - http://pkg.jenkins-ci.org/debian/jenkins-ci.org.key \
      | sudo apt-key add -
  
  2. add the repository to the sources
      sudo sh -c 'echo deb http://pkg.jenkins-ci.org/debian binary/ > \
      /etc/apt/sources.list.d/jenkins.list'
  
  3. re-synchronize index file
      sudo aptitude update
  
  4. install jenkins
      sudo aptitude install jenkins

Either  way,  we   can  validate  jenkins  is  installed   by  going  to
http://localhost:8080/jenkins. Et.. voilÃ! Now,  to configure jenkins to
be on port 80, we need to  install the mod-proxy module. On fedora it is
probably  installed,  on  ubuntu  it should  be  "sudo  apt-get  install
libapache2-mod-proxy-html". Now, to enable:

sudo a2enmod proxy
sudo a2enmod proxy_http

We can configure our proxy as follows (this worked in the past on ubuntu
server; however, I did not try it on Amazon EC2 Demo).

ProxyPass         /jenkins  http://localhost:8080/jenkins
ProxyPassReverse  /jenkins  http://localhost:8080/jenkins
ProxyRequests     Off

# Local reverse proxy authorization override
# Most unix distribution deny proxy by default 
# (ie /etc/apache2/mods-enabled/proxy.conf in Ubuntu)
<Proxy http://localhost:8081/jenkins*>
  Order deny,allow
  Allow from all
</Proxy>

As a final  note, there are many more installation  options for Jenkins.
You may  want to consider  running jenkins as  a WAR behind  a container
like Tomcat,  Jetty etc. (currently it  uses Winstone, a small  and fast
servlet  container).  You  might  also consider  enhancing  security  by
creating  a dedicated  user to  run  jenkins (with  the permissions  you
want).

2. Post-installation

After installation, there  are a number of things we  have to configure.
Go to  jenkins' home page and  select "Manage Jenkins" from  the menu on
the  left. Select  "Manage  Users"  and then  "Create  User"  to add  an
administrator (consider also  adding the rest of the users  who will use
the system).

After adding  users, we need  to add  permissions. Again, go  to "Manage
jenkins" and select "Configure System".  Here you can configure a number
of settings. Most importantly, click  on "Enable Security" radio button,
and this  will populate a  number of  settings. In the  "Access Control"
level you can configure:

  * "Security Realm":  where to draw  users from  -- in our  case select
    "Jenkins's own  user database".  Because creating accounts  for many
    students can be  time consuming, you can consider  enabling users to
    sign up;  or even  "import/authenticate" via  LDAP or  Unix database
    (both are applicable for eniac setup and can be automated for future
    classes of students).

  * "Authorization": what  users can do --  in our case I  would suggest
    matrix  based security  which is  quite granular.  The administrator
    should  have  everything  enabled,  while  other  users  would  only
    need  a number  of options  (e.g., overall:  read/run scripts,  job:
    read/discover/build/cancel, view: all).

Moreover, we would need to configure  system variables like ant and JDK.
Assuming that ant and JDK are installed (there are copious resources for
these) we  just add the  paths to the  installations. For Amazon  EC2, I
used /usr/share/ant for  ant and /usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0
for the JDK.

2.1 Add plugins

Now,  in  order  to  add  plugins, we  go  to  "Manage  Jenkins">"Manage
plugins". First  of all, we  install the updates  if there are  any. The
following are  installed in  my setup  (some of  these are  installed by
default):

  * javadoc [+]
  * Maven 2 Project plugin [+]
  * ant [+]
  * Static Analysis Utilities (this is a prerequisite for other plugins)
  * PMD plugin
  * Checkstyle plugin
  * Static Analysis Collector plugin
  * Jenkins SSH Slaves plugin [+]
  * Jenkins Translation Assistance plugin [+]
  * Jenkins CVS Plug-in [+]
  * Jenkins Subversion Plug-in
  * SLOCCount Plugin

[+] I *think* that these plugins were installed by default.

Please browse in the plugin repository,  since you may find more plugins
that feet  your needs.  For instance,  the other day  I came  across the
"Green Balls" plugin  which changes successful build icons  from blue to
green  balls (red  stay the  same). Or  the "Scriptability"  plugin that
allows to programmatically access jenkins' interface.

2.2 Setup users

2.3 Setup jobs


3. Project structure

Using ant,  the project will need  to have some targets  responsible for
running jUnit  tests through  ant. Please  make sure  that you  have ant
optional jars (which contain the junit). The important thing here is the
directory for  storing junit output  which will  be read by  jenkins (to
make things easy, it should better  be the same across all projects). An
example would be (second target is the important part):

    <!-- grab library jars, it contains junit jar-->
    <path id="junit-lib">
        <fileset dir="${library.dir}">
            <include name="**/*.jar"/>
        </fileset>
        <pathelement location="${target.dir}" />
    </path>

    <!-- run tests and output xml to be read by jenkins -->
    <target name="junit" depends="compile">
        <mkdir dir="${report.dir}"/>
        <junit fork="no" printsummary="yes" showoutput="no">
            <classpath refid="junit-lib" />
            <formatter type="xml"/>
            <batchtest todir="${basedir}/distribution">
                  <fileset dir="${src.dir}">
            <include name="**/*Test*.java" />
          </fileset>
            </batchtest>
        </junit>
    </target>

    <!-- generates html report -->
    <target name="junitreport" depends="junit">
        <junitreport todir="${report.dir}">
            <fileset dir="${basedir}/distribution" includes="TEST-*.xml"/>
            <report todir="${report.dir}"/>
        </junitreport>
    </target>

Now, there are many ways to  "send" (synchronize) the local project with
the one in the jenkins job. Here I describe three possible solutions (in
increasing level of complexity for  the administrator). In all versions,
the user  invokes a (555-)ci-build  script which takes care  of updating
jenkins and invoking the build.

The easiest way would be to set the script to rsync the archived project
to a  typical unix multi-user  server, and send  the file to  the user's
html folder. Then  it invokes the job  for the user which  in turn calls
wget  to  fetch the  latest  version.  This has  minimal  administration
hassle. An example (555-)ci-build script would be:

    DIR=$(cd $(dirname "$0"); pwd)
    echo "Base dir: $DIR"
    cd $DIR/../../
    echo "Now on: `pwd`"
    if [ -d "build" ]; then
        rm -rf build
    fi
    mkdir build;
    cp -r $DIR/../* build;
    # remove Intellij IDEA files (it should be the case for eclipse etc)
    rm build/*.iml
    echo "Archiving.."
    tar -cvzf build.tar.gz build
    echo "Copying to remote.."
    # This can be a dedicated server
    rsync -av --progress build.tar.gz nvas@eniac.seas.upenn.edu:~/html
    sleep 1;
    echo "Launching build.."
    # Launch the build as shown below -- the server domain 
    # name would also be different
    /usr/bin/wget -O - --auth-no-challenge --user=myUserName \
    --password=myPassWord \
    'http://ec2-50-16-77-74.compute-1.amazonaws.com:8080/view/All/job/HW3MS1/build?token=myToken'

The typical  "Build" part of  the job (see section  2.3 on how  to setup
jobs) would  contain an  "Execute shell" build  target which  would wget
from the remote (or local) machine, afer doing some cleanup. *Note* that
it fetches  official tests  from a  different user (it  could also  be a
different server):

    if [ -d "build" ]; then
        rm -rf build
    fi
    if [ -f build.tar.gz ];
    then
        rm build.tar.gz
    fi
    wget http://www.seas.upenn.edu/~nvas/build.tar.gz
    tar -xvzf build.tar.gz
    cd build
    # apply more tests from various sources, either local or remote
    wget http://www.seas.upenn.edu/~ahae/HW3Test.java
    mv HW3Test.java src/test/edu/upenn/cis/cis555/youtube
    # ant checkstyle and other code-quality targets described below
    ant junit # or ant junitreport

A slightly  more complex way would  be to rsync directly  on the jenkins
server. Given  that each user has  an account with minimum  quota and no
other  permissions, the  script rsyncs  to  each user's  home folder.  A
slight modification is needed to the job: Go to the job's web interface>
"Configure"  and  enable "This  build  is  parametrized". Add  a  String
parameter with name  USER and default value no-user  (or another value).
The (555-)ci-build script  would be the same as  before, only containing
the same url for both rsync and launch. See the changes below:

    DIR=$(cd $(dirname "$0"); pwd)
    echo "Base dir: $DIR"
    cd $DIR/../../
    echo "Now on: `pwd`"
    if [ -d "build" ]; then
        rm -rf build
    fi
    mkdir build;
    cp -r $DIR/../* build;
    # remove Intellij IDEA files (it should be the case for eclipse etc)
    rm build/*.iml
    echo "Archiving.."
    tar -cvzf build.tar.gz build
    echo "Copying to remote.."
    ####################################################################
    # This is important change 1: The host is the same as jenkins server
    ####################################################################
    rsync -av --progress build.tar.gz \
    nvas@ec2-50-16-77-74.compute-1.amazonaws.com:~/
    sleep 1;
    echo "Launching build.."
    # Launch the build as shown below -- the server domain 
    # name would also be different
    ####################################################################
    # This is important change 2: last part of the resource is 
    # buildWithParameters instead of build and there another parameter
    # with name USER and value the user who launches the build
    ####################################################################
    /usr/bin/wget -O - --auth-no-challenge --user=myUserName \
    --password=myPassWord \
    'http://ec2-50-16-77-74.compute-1.amazonaws.com:8080/view/All/job/HW3MS1/buildWithParameters?token=myToken&USER=ahae'

The typical  "Build" part of  the job (see section  2.3 on how  to setup
jobs) again contains an  "Execute shell" build  target. This time the
script copies from the user's home folder (which is passed as the $USER
environmental variable from the url above):

    if [ -d "build" ]; then
        rm -rf build
    fi
    if [ -f build.tar.gz ];
    then
        rm build.tar.gz
    fi
    ####################################################################
    # This is important change: instead of wget we mv from $USER to .
    ####################################################################
    mv /home/$USER/build.tar.gz .
    tar -xvzf build.tar.gz
    cd build
    # apply more tests from various sources, either local or remote
    wget http://www.seas.upenn.edu/~ahae/HW3Test.java
    mv HW3Test.java src/test/edu/upenn/cis/cis555/youtube
    # ant checkstyle and other code-quality targets described below
    ant junit # or ant junitreport

The  third  way would  be  for  jenkins to  check  out  from svn  before
launching the build. This way the server builds the latest version. This
option is  more complex, since it  also needs the administration  of the
svn server.  Another drawback of this  way is that users  will always be
forced to  build the  latest version,  even if they  want to  build some
versions earlier  -- there is no  easy way of overcoming  this with svn,
although by using git or mercurial can be solved easier.

Since subversion plugin is installed, we can go to the job's web
interface > "Configure" and scroll down to the "Source Code Management"
section. Select "Subversion", and add the parameters (URL etc)

*Idea*: Since  students will  use svn,  a nice  idea would  be to  add a
post-commit hook to  the svn server for each repository  that only wgets
the url as described above. This way students will understand the actual
meaning of *continuous* integration as used in the industry.

3.4 More ant targets

We  can add  some targets  whose sole  purpose will  be to  measure code
statistics  (like javadoc  or code  quality metrics).  There is  a great
chapter for  code metrics among other  things in the book  "Jenkins: The
Definitive Guide" by John Ferguson  Smart (Publisher is O'Reilly Media).
We first define extra  ant targets for each tool that  we want to invoke
and then call  these targets before junit. Also, for  each tool, we need
to have  installed the respective  plugin (as described in  section 2.1)
which  reads  the output  created  from  the  respective ant  call.  For
instance,