Microservice framework startup time on different JVMs

When developing microservices, a fast startup time is useful. It can for example reduce the amount of time a rolling upgrade of instances takes and reduce build time thus shortening development cycles. When running your code using a 'serverless' framework such as for example Knative or FnProject, scaling and getting the first instance ready is faster.

When you want to reduce startup time, an obvious thing to look at is ahead of time (AOT) compilation such as provided as an early adopter plugin as part of GraalVM. Several frameworks already support this out of the box such as Helidon SE, Quarkus and Micronaut. Spring will probably follow with version 5.3 Q2 2020. AOT code, although it is fast to startup, still shows differences per framework. Which framework produces the native executable which is fastest to start?

If you need specific libraries which cannot be natively compiled (not even when using the Tracing Agent), using Java the old-fashioned JIT way is also an option. You will not achieve start-up times near AOT start-up times but by choosing the right framework and JVM, it can still be acceptable.

In this blog post I'll provide some measures which I did on start-up times of minimal implementations of several frameworks and an implementation with only Java SE. I've looked at both JIT and AOT (wherever this was possible) and ran the code on different JVMs.

Disclaimer

These measures have been conducted on specific hardware (my laptop) using specific test scripts on specific though minimal implementations (but comparable). This is of course not the same as a full blown application running in production on specialized hardware. Use these measures as an inspiration to get a general idea about what differences between startup time might be. If you want to know for sure if these differences are similar for you, conduct your own tests which are representative for your situation on your hardware and see for yourself.

Setup

At the end of this blog post you can find a list of framework versions which have been used for my tests. The framework implementations which I've used can be found here.

Measuring startup time

In order to determine the startup time, I looked at the text line in the logging where the framework indicated it was ready.

• Helidon SE
  WEB server is up!
• Micronaut
  Startup completed in
• Microprofile (Open Liberty)
  server is ready to run a smarter planet
• Spring Boot and related
  JVM running for
• Vert.x
  Succeeded in deploying verticle
• Akka
  Server online at
• Quarkus
  started in

I wanted to measure the time between the java command to run the JAR file and the first occurance of the above lines. I found the magic on how to do this here. Based on this I could execute the following to get the wanted behavior.

expect -c "spawn JAVACOMMAND; expect \"STRING_TO_LOOK_FOR\" { close }" > /dev/null 2>&1

Next I needed to time that command. In order to do that, I did the following:

ts=$(date +%s%N)

expect ...

echo ADDITIONAL_INFO_ABOUT_MEASURE,$((($(date +%s%N) - $ts)/1000000)) >> output.txt

I did this instead of using the time command because of the higher accuracy and because of the way I piped the expect output to /dev/null.

Implementing a timeout

I noticed sometimes the expect command left my process running. I did not dive into the specifics as to why this happened, but it caused subsequent tests to fail since the port was already claimed. I installed the 'timelimit' tool and specified both a WARN and KILL signal timeout (timelimit -t30 -T30). After that I did a 'killall -9 java' just to be sure. The tests ran a long time and during that time I couldn't use my laptop for other things (it would have disturbed the tests). Having to redo a run can be frustrating and is time consuming. Thus I want to be sure that after a run the java process is gone.

JVM arguments

I used the following JVM arguments:

java8222cmd="/jvms/java-8-openjdk-amd64/bin/java -Xmx1024m -Xms1024m -XX:+UseG1GC -XX:+UseStringDeduplication -jar"

java1104cmd="/jvms/java-11-openjdk-amd64/bin/java -Xmx1024m -Xms1024m -XX:+UseG1GC -XX:+UseStringDeduplication -jar"

javaopenj9222="/jvms/jdk8u222-b10/bin/java -Xmx1024m -Xms1024m -Xshareclasses:name=Cache1 -jar"

javaoracle8221="/jvms/jdk1.8.0_221/bin/java -Xmx1024m -Xms1024m -XX:+UseG1GC -XX:+UseStringDeduplication -jar"

The script to execute the test and collect data

I created the following script to execute my test and collect the data. The Microprofile fat JAR generated a large temp directory on each run which was not cleaned up after exiting. This quickly filled my HD. I needed to clean it 'manually' in my script after a run.

Results

The raw measures can be found here. The script used to process the measures can be found here.

JIT

The results of the JIT compiled code can be seen in the image above. 

• Of the JVMs, OpenJ9 is the fastest to start for every framework. Oracle JDK 8u221 (8u222 was not available yet at the time of writing) and OpenJDK 8u222 show almost no difference.
• Of the frameworks, Vert.x, Helidon and Quarkus are the fastest. 
• Java 11 is slightly slower than Java 8 (in case of OpenJDK). Since this could be caused by different default garbage collection algorithms I forced them both to G1GC. This has been previously confirmed here. Those results cannot be compared to this blog post one-on-one since the tests in this blog post have been executed (after 'priming') on JVMs running directly on Linux (and on different JVM version). In the other test, they were running in Docker containers. In a Docker container, more needs to be done to bring up an application such as loading libraries which are present in the container, while when running a JVM directly on an OS, shared libraries are usually already loaded, especially after priming.
• I also have measures of Azul Zing. However I tried using the ReadyNow! and Compile Stashing features to reduce startup time, I did not manage to get the startup time even close to the other JVMs. Since my estimate is I must have done something wrong, I have not published these results here.

AOT

JIT compiled code startup time does not appear to correspond to AOT code start-up time in ranking. Micronaut does better than Quarkus in the AOT area. Do notice the scale on the axis. AOT code is a lot faster to startup compared to JIT code.

Finally

File sizes

See the below graph for the size of the fat JAR files which were tested. The difference in file size is not sufficient to explain the differences in startup time. Akka is for example quite fast to startup but the file size is relatively large. The Open Liberty fat JAR is huge compared to the others but its start-up time is much less than to be expected based on that size. The no framework JAR is not shown but it was around 4Kb.

Servlet engines

The frameworks use different servlet engines to provide REST functionality. The servlet engine alone however says little about the start-up time as you can see below. Quarkus, one of the frameworks which is quick to start-up, uses Reactor Netty, but so do Spring WebFlux and Spring Fu which are not fast to start at all.

Other reasons?

An obvious reason Spring might be slow to start, can be because of the way it does its classpath scanning during startup. This can cause it to be slower than it could be without this. Micronaut for example, processes annotations during compile time taking away this action during startup and making it faster to start.

It could be a framework reports itself to be ready while the things it hosts might not be fully loaded. Can you trust a framework which says it is ready to accept requests? Maybe some frameworks only load specific classes at runtime when a service is called and others preload everything before reporting ready. This can give a skewed measure of startup time of a framework. What I could have done is fire off the start command in the background and then fire of requests to the service and record the first time a request is successfully handled as the start-up time. I however didn't. This might be something for the future.