performance-tests-docker-junit-eclipse.md

Functional Performance Tests For Java

Since there seems to be multiple definitions of functional tests here is mine:

A functional test is a test which will test the exposed application features (e.g.: Internal or REST API's ) with less or no mocks compared to unit tests. Functional tests may also interact with databases or other external services.

For Gentics Mesh we already have a lot of functional tests but no performance tests. It is therefore hard to tell whether a change or even a dependency update actually improves or degrades the service performance.

The Gentics Mesh functional tests are written using JUnit. Please keep in mind that a JUnit test must not necessarily be a unit test. In case of mesh we have a lot of functional tests which startup an embedded mesh instance and invoke regular REST requests via HTTP.

So lets see what kind of performance test are common:

== Load Performance Tests Those tests usually test the system load performance. JMeter Tests and Gatling Tests are tools which are commonly employed for those tests. Testsituations must be created or recorded upfront. Gatling and JMeter for example can as far as i know only be used for testing HTTP APIs. There are various Jenkins CI Plugins with allow you to plot neat graphs that visualize your jmeter test results. I think it is even possible to define test thresholds.

== Stopwatch Performance Tests

Typical stopwatch tests are tests in which you just measure the duration it takes to execute a specific task. This is a very easy method to write performance tests since you can reuse your regular functional tests.

A typical mesh stopwatch performance test in a simplified form could look like this:

@Test
public void testCreatePerformance() {
  // Warmup the system caches etc
  warmup();
  // Start the stopwatch
  mark();
  // Invoke the create node action 1000 times
  for(int i = 0; i < 1000; i++) {
    createNode();
  }
  // Stop the stopwatch and assert the duration
  assertDuration(5000);
}

There are a few things that may be note worthy:

Performance tests should contain a warmup phase in order to smooth out test results.
It may also be interesting to test for performance degradation. Sometimes the 100th request is fast and the performance degades slightly with each next request.
Any performance test is highly hardware specific. A stop watch test is no different. Executing the test within your IDE on your computer may produce very different results from executing the test on your continous integration system.
The same test stopwatch test setup can also be used to for memory leaks. Using JMX it is possible to check the heap memory size. You can invoke you action multiple times and check whether the heap size exceeds a specific limit. Of course you would need to invoke a System.gc() in order to clear the heap. Please note that the System.gc() call in java is just a recomendation. The JVM may or may not execute the GC request. This makes those tests a bit tricky.

Stopwatch tests can also be used to assert the performance of specific internal calls like graph database traversals, cache access speed.

While researching options for executing stopwatch performance tests I was wondering whether it is possible to execute a JUnit test remotely and starting it from within my eclipse. Executing the test remotely would allow me to run the test on a dedicated server and thus the performance would always be the same since the hardware is the same. My CI would also utilize this server for excuting the test. I could even setup two identical servers in order to be able to write performance tests while also exceute CI tests using the other server.

=== Docker to the rescue

It is possible to create a docker container which contains all needed test classes. Instead of executing the regular $JAVA_HOME/bin/java executable a bash script will be invoked which creates a test image and executes the jvm within that image.

I just copied duplicated my JDK 1.8 folder and replaced the regular java binary with the bash script below. In eclipse i only had to add this modified JDK folder and invoke the test using that JDK. This way the bash script below was executed by my Eclipse IDE. Next i needed to prepare the base image. The base image needs to contain the JDK. In my case i used the the registry.office/docker-jenkins-slave:latest image which is an internal jenkins slave docker image which we use to execute our Jenkins Jobs.

The bash script will splitup the classpath and copy all needed files and folders to the BUILDDIR folder. After that a docker build will be invoked which create a docker image that contains all those files and folders. After that docker run will be used to execute the JVM within the image. It is important to note that your system hostname needs to be specified. The JVM process within the docker image will connect to your IDE in order to visualize the JUnit test results. Finally the build docker image is no longer needed and can be purged from the docker host.

Please keep in mind that the bash script and process will most likely not work on windows but it should be possible to transform the windows classpath to a classpath which linux understands.

#!/bin/bash

set -o nounset
set -o errexit

BUILDDIR=/tmp/eclipse-docker
DOCKERHOST=tcp://cetus.office:2375

mkdir -p $BUILDDIR
cd $BUILDDIR

echo "FROM registry.office/docker-jenkins-slave:latest" > Dockerfile
echo "ADD opt /opt" >> Dockerfile
echo "ADD home /home" >> Dockerfile
echo "Adding files to context folder.."
paths=$(echo $@ | tr ":" "\n")
for path in $paths ; do
        # Only handle file paths
        if [[ $path == /* ]] ; then
          dirname=$(dirname $path)
          relDirname=${dirname:1}
      mkdir -p $relDirname
      cp -ra $path $relDirname
        fi 
done
echo "Invoking docker build.."
docker -H $DOCKERHOST build -t eclipse-docker .

echo "Invoking java.."
docker -H $DOCKERHOST run --rm eclipse-docker /opt/java/jdk1.8.0_74/bin/java  $@ -host $HOSTNAME

echo "Cleanup.."
docker -H $DOCKERHOST rmi eclipse-docker

Jotschi/performance-tests-docker-junit-eclipse.md