Performance! 3 reasons to stick to Java 8 for the moment

It is a smart thing to move to newer versions of Java! Support such as security updates and new features are just two of them but there are many more. Performance might be a reason to stick to Java 8 though. In this blog post I'll show some results of performance tests I have conducted showing Java 11 has slower startup times and slightly slower throughput compared to Java 8 when using the same Java code. Native images (a GraalVM feature) have greatly reduced startup time and memory usage at the cost of throughput. You can only compile Java 8 byte-code to a native image though (at the moment).

Worse performance for Java 11

I've done extensive measures on performance of different JVMs and different microservice frameworks running on them. You can browse the scripts used here and view a presentation which describes the test set-up here. The test I did was create minimal implementations of several frameworks, compiled them to Java 8 and Java 11 byte-code respectively, ran them in Docker containers on a specific JVM and put load on them. I send a message, waited for a response and then I send the next message (all synchronous). I've done the same test, which ran for 15 minutes per framework/JVM combination (millions of requests), several times and the results are reproducible. I paid specific attention to make sure the load test and the JVM used separate resources. Also, I first wrote the measurements in memory before writing them to disk at the end of the test. I did this to be able to measure sub-millisecond differences between the JVMs and frameworks and not my disk performance.

Slightly worse throughput (response times)

For every framework (Akka, Vert.x, Spring Boot, WebFlux, Micronaut, Quarkus, Spring Fu) I noticed that the performance on Java 8 (tested on Azul Zing, Oracle JDK, OpenJDK and OpenJ9) was slightly worse than on Java 11 (though less than a tenth of a millisecond). For OpenJ9 it is beneficial to go to Java 11 though when running Akka or Spring Boot.

This could be related to the different garbage collection algorithm which is used by default. For Java 8 this is Parallel GC while for Java 11 this is G1 GC. G1 GC with 2Gb of heap performs slightly worse during my test than Parallel GC as you can see in the below graph. When reducing the heap though, G1 GC starts to outperform Parallel GC.

I have not tried Java 11 with Parallel GC (-XX:+UseParallelGC) to confirm the drop in performance is caused by the GC algorithm. As you can see though the performance difference from Java 8 to Java 11 is consistent over the different JVMs (with exception of Akka, Spring Boot on OpenJ9) using different GC algorithms. For example Zing 11 performs slightly worse then Zing 8 and Zing did not make the change in the default GC algorithm. Thus it is likely that when using the same algorithm for OpenJDK and Oracle JDK 8 and 11, there will still be slightly worse performance for JDK 11.

Worse startup time

As you can see in the above graph for the same Spring Boot application, startup time, running on a JVM with 2Gb of heap, for Java 11, was longer. I just showed the graph for Spring Boot but the same holds true for the other microservice frameworks.

GraalVM native compilation only supports Java 8

GraalVM 19 is currently (1st of June 2019) only available in a Java 8 variant. The native compilation option is still in an early adopters phase at the moment, but, even though throughput is worse (see the above graphs, Substrate VM), start-up time is much better!  Throughput might be worse because pre-compilation could prevent some runtime optimizations (I have not confirmed this). See for example the below graph of Quarkus startup time on different JVMs. Also it is said memory usage is better when using native images but I've not confirmed this yet by own measurements.

If you want to use native images, for the moment you are stuck on Java 8. A good use-case for native images can be serverless applications where startup time matters and memory footprint plays a role in determining how much the call costs you.

Finally

Java 11 specific features were not used

These tests were performed using the same code but compiled to Java 8 and Java 11 byte-code respectively to exclude effects of compilation (although OpenJDK was used to compile to Java 8 and 11 byte-code in all cases). Usually you would write different code (using new features) when writing it to run on Java 11. For example, the Java Platform Module System (JPMS) could have been implemented or maybe more efficient APIs could have been used. Thus potentially performance on Java 11 could be better than on Java 8.

Disclaimer

Use these results at your own risk! My test situation was specific and might differ from your own situation thus performing your own tests and base your decisions on those results is recommended. This blog post just indicates that moving from Java 8 to Java 11 might not improve performance and prevents you from using native compilation at the moment.