A recent report of bad result from Quarkus warranted some further investigation. On the face of it the results looked bad, really bad, for Quarkus.

tl;dr

The test that was shared with us is a simple load test that updates a database via REST invocations;

1. A load generator creates a continuous stream of HTTP POST requests to a REST api. In this case wrk

2. A Quarkus application process the request via RESTEasy Reactive

3. The Quarkus application queries and updates a MySQL database instance via Hibernate Reactive

The source code for the test can be found here: https://github.com/thiagohora/tutorials/tree/fix_jmeter_test

To learn more about creating Reactive Applications with Quarkus, please read the Getting Started With Reactive guide

Initial results for Quarkus were not promising;

That was 105 requests in 60 seconds, with 10 errors. Only 95 requests had been successfully sent in 60 seconds, or 1.75 req/sec

Running the comparison test on my machine;

Overall, the request rate that Quarkus could support was only 1.75 req/sec!! Ok, so it wasn’t 600 times slower, but it was 192 times slower on my machine.

but…​ something was not correct, Quarkus was displaying the following exception in the service logs;

An initial investigation showed that the number of open MySQL connections during the test was very high: 96 open connections

And checking the number of inserts the application had managed to perform within 1minutes;

There was obviously something wrong with the database connections! Each connection was committing only a single value to the database and no more progress was being made. The number of entries in the database tallied exactly with the number of successful HTTP requests.

Reviewing the CPU time for the Quarkus process confirmed that no further work was being done after the initial 95 commits to the database, the application was deadlocked;

A quick review of the code revealed the deadlocking issue;

Ah Ha! the endpoint is annotated with @Transactional. The application is using Hibernate Reactive, so instead we need to use the @ReactiveTransactional annotation. For further details, please read the Simplified Hibernate Reactive with Panache guide. This can be confusing, but conversations have started about how to clarify the different requirements and warn users if there is an issue.

Let’s try again:

390.21 req/sec!! that’s much better!!

With the test fixed, we can see a lot more data in the database table;

but…​ my hard disk on my machine was making a lot of noise during the test! The Quarkus result of 390.21 req/sec is suspiciously similar to the comparison baseline of 336.86 req/sec, and…​

The application is using less than 0.5 cores on a 32 core machine…​ hmm!

Let’s move the database files to a faster disk;

and re-run the test

Sit back, Relax and Profit! 25,599.85 req/sec!

the Quarkus process is now using 4.5 cores…​

but…​ the system is 60% idle

We still have a bottleneck outside of the application, most likely within MySQL or we are still I/O bound!

At this point, we have a couple of options, we can either;

The easiest option is to simply constrain the application.

We will remove the MySQL/System bottleneck by constraining the application to 4 CPU cores, therefore reducing the maximum load the application can drive to the database. We achieve this by running the application in docker;

and re-running the test;

Ok, so we are not at Max Throughput, but we have removed the system outside of the application as a bottleneck. The bottleneck is NOW the application

Further investigation showed that Quarkus is not running with TLS enabled between the application and database, so database network traffic is running un-encrypted. Let’s fix that;

and re-run

This provided us with a final, comparable throughput result of 14,955.61 req/sec

Is Quarkus really 600x times slower than Framework X/Y/Z? Of course not!

On my machine;

1. the initial result was 1.75 req/sec.

2. fixing the application brought that up to 390.21 req/sec

3. fixing some of the system bottlenecks gave us 25,599.85 req/sec

4. constraining the application, so that a fairer comparison with other frameworks can be made resulted in 18,667.87 req/sec

5. and finally, enabling TLS encryption to the database gives a final result of 14,955.61 req/sec

Where does that leave Quarkus compared to Framework X/Y/Z?

Do these results show that Quarkus is quick? Well kinda, they hint at it, but there are still issues with the methodology that need resolving.

However, when faced with a benchmark result, especially one that does not appear to make sense, there are a number of steps you can take to validate the result;

• Fix the application: Are there errors? Is the test functioning as expected? If there are errors, resolve them

• Ensure the application is the bottleneck: What are the limiting factors for the test? Is the test CPU, Network I/O, Disk I/O bound?

• Do not stop evaluating the test when you see a "good" result. For comparisons, you need to ensure that every framework is the limiting factor for performance and not the system.

• Chose how to constrain the application: either by scaling up the system, or scaling down the application.

• Validate that all frameworks are doing the same work. For comparisons, are the frameworks performing the same work?

• Ensure al frameworks are providing the same level of security. Are the semantics the same? e.g. same TLS encoding? same db transaction isolation levels?