Reproducing the benchmarks of the Maven vs Gradle performance page

README.md

Gradle vs Maven performance reproduction instructions

Install the Gradle Profiler

This page explains how to reproduce the Gradle vs Maven performance numbers yourself. For that, you need to install the Gradle profiler, a tool which will automate benchmarking by running several builds in a row, with the appropriate options (it also runs Maven builds).

Generate the test projects

Our performance comparison uses 4 test projects:

• a large, monolithic app
  • single project
  • 50000 source files and 50000 test source files
• a medium sized, multi-project build
  • 100 subprojects
  • each subproject has 100 source files and 100 test source files
• a large, multi-project build
  • 500 subprojects
  • each subproject has 100 source files and 100 test source files
• Apache Commons Lang

The first 3 can be found in the Gradle repository. To generate a test project, checkout the Gradle sources, then execute:

• ./gradlew largeMonolithicJavaProject for the large monolithic build
• ./gradlew mediumJavaMultiProject for the medium multiproject build
• ./gradlew largeJavaMultiProject for the large multiproject build

The test projects will then be found in subprojects/performance/build/<test project name>.

The last project is a real-world project (Apache Commons Lang that needs to be setup separately:

git clone https://github.com/gradle/performance-comparisons.git commons-lang --branch commons-lang --depth 1
cd commons-lang

This will clone the repository which is a copy of the original Commons Lang repository, but containing our converted to Gradle build and the scenario file.

Running a performance test

Make sure you have installed the Gradle Profiler and that it's available on your PATH.

Measuring the performance of Gradle

Change directory to a test project and run:

gradle-profiler --gradle-version 4.0 --benchmark --scenario-file performance.scenarios --project-dir .

This will execute all scenarios by default, which can take a significant amount of time. If you want to run one or more specific scenarios, just add them explicitly to the command line:

gradle-profiler --gradle-version 4.0 --benchmark --scenario-file performance.scenarios --project-dir . <scenario1> <scenario2> ...

Measuring the performance of Apache Maven

First make sure you have properly set the MAVEN_HOME and MAVEN_OPTS environment variables:

export MAVEN_HOME=/path/to/apache-maven-3.5.0
export MAVEN_OPTS="-Xms2g -Xmx2g"

Then execute:

gradle-profiler --benchmark --scenario-file performance.scenarios --project-dir . --maven --warmups 2 --iterations 2

Note: For the large monolithic case, increase Maven memory to 3g instead of 2g, otherwise it won't run. Gradle will use 2g for this scenario.

You will notice that we only use 2 warmups and runs for Maven. There are multiple reasons for this:

• Maven doesn't have a daemon, so doesn't benefit from subsequent runs of the same build
• Maven is significantly slower. A test that would take minutes to complete with Gradle would take several hours with Maven

Note about the scenarios for Maven

You will notice that for Apache Maven, we run clean compile instead of compile. It may, at first glance, look like an unfair comparison, but it's for an important reason: Maven's incremental compiler is completely broken and will miss changes. An example of serious bug can be found here.

Well, I have actually tried running your tests and the result is that maven is faster for the clean build. I have generated "mediumJavaMultiProject" project.

"mediumJavaMultiProject" is a project which has 100 modules. Each module has 100 Java source files and 100 Java unit test files. So the total number of Java files is 20000 (10000 source files and 10000 unit test files). For comparison, apache flink project has approximately 40 modules (As of February of 2021) and total number of Java files is 11573 plus 1981 Scala files.

In gradle.properties you have

org.gradle.parallel=true
org.gradle.workers.max=4

which is equivalent of mvn -T 4
However checking pom.xml I have found strange things - for some reason ;-) you set fork=true for the maven compiler plugin. Why would you do that? I have never had the need to set fork=true. And yes, obviously it affects performance. So what I did I disabled it and configured maven to use java 15.

And to test I run
gradle clean followed by gradle classes. I run this on a VM on my laptop. So machine is not extremely fast. First run of gradle classes (daemon is not started yet) takes 1 minute and 11 seconds. Next runs when gradle daemon is running are taking about 52-54 seconds (I mean running gradle classes after gradle clean).
After this I run mvn clean followed by mvn -T 4 compile. And compilation by maven runs in about 33 seconds which is faster even compared to gradle+daemon combination.

I bet the reason of poor maven performance was in fork=true setting.

clean task for both maven and gradle is around 2 seconds so it does not impact the summary at all.

Now let's talk about incremental compilation.

Yes, maven does not have it by default and if you want it you will need to use 3rd party module - http://takari.io/book/40-lifecycle.html with jdt compiler. And yes, gradle does have it by default without the need of external components.

Let's look at the example. Imagine situation when I change only 5 classes (e.g. add new property) in the 5 different modules:

project0/src/main/java/org/gradle/test/performance/mediumjavamultiproject/project0/p4/Production80.java
project10/src/main/java/org/gradle/test/performance/mediumjavamultiproject/project10/p50/Production1000.java
project20/src/main/java/org/gradle/test/performance/mediumjavamultiproject/project20/p100/Production2000.java
project30/src/main/java/org/gradle/test/performance/mediumjavamultiproject/project30/p150/Production3000.java
project40/src/main/java/org/gradle/test/performance/mediumjavamultiproject/project40/p200/Production4000.java

For gradle this means detect changes and recompile only 5 classes. For maven without takari-lifecycle this means detect changes and recompile 500 files since default maven configuration recompiles entire module if it detects even simple change.

For maven execution takes about 11 seconds mvn -T 4 compile.

Now I execute following commands

serhiy@serhiy-vb-204:~/mvntest/mediumJavaMultiProject$ ../gradle-6.8.3/bin/gradle clean

BUILD SUCCESSFUL in 2s
100 actionable tasks: 100 executed
serhiy@serhiy-vb-204:~/mvntest/mediumJavaMultiProject$ ../gradle-6.8.3/bin/gradle classes

BUILD SUCCESSFUL in 55s
100 actionable tasks: 100 executed

after this I modify same 5 files from different modules and run compilation with maven.

There is no difference at all for such scenario. I mean recompiling all classes of 5 modules with maven takes 11 seconds and detecting the change and recompiling only 5 files from different modules takes same 11-12 seconds. By the way, when you configure takari-lifecycle with incremental build for maven it will still be around same 12 seconds :-) of compilation time.

I think that it is still possible to come up with some scenario where gradle will be slightly faster. But that would be synthetic scenario. For me personally modification of 5 different modules at once is covering most of real world scenarios any developer will have. Also do not forget that with maven recompiling entire project with 100 modules and 10000 Java source files takes less than 35 seconds is also very fast.

Gradle is more actively developed compared to maven. But still it has bugs. You have link to maven bug. But running is:issue is:open incremental label:a:bug query today (3/12/2021) I see a number of open defects with interesting names like

• Kotlin build fails when run with Gradle daemon #15196
• Incremental compilation should be supported with JDK 15 #14744
• Gradle always does full recompilation after getting result of compileJava task from Build cache #14917
• No java incremental compile after compile failure - "unable to get source-classes mapping relationship from last compilation." #13601
• Java incremental compilation api/impl usages separation does not work between modules #13499
• Dependent classes not recompiled when Android resource is added on Gradle 6.3 #12995

So in reality gradle users will still be doing clean build to avoid issues.