NBomber 2.0

Hey folks, we were busy for nearly half a year working on a new big release of NBomber. I am so happy to announce that we finally completed it!

In this release, we focused on improving engine performance, fixing RAM consumption issues, improving UI/UX, including reporting, extending API to provide flexibility but keep it SIMPLE as before.

UI/UX Improvements#

Console#

In the previous version, we always had a desire to improve the console output. The main pain point for us was the progress bar, which did not scale well while resizing the console, and sometimes it literally broke down when you are writing log messages to the console. The second most important problem for us was the table rendering, which was also displayed crookedly while resizing windows, and we wanted to fix this. We started considering Spectre.Console project for a long time as a good candidate for replacement and then we started a smooth transition. It was good with tables, but we still had some problems with the progress bar. Since for logs, we use Serilog, we figure out that we need to develop a proper integration between Serilog and Spectre.Console to fix our issue. After all, we developed Serilog.Sinks.SpectreConsole that nicely integrates Serilog with Spectre.Console. It's open-source, and you can use it for your integrations too. Now our console UI is much more stable, and I hope more beautiful :)

HTML report#

We use Vue.js, which amazed us with its simplicity and minimalism for rendering HTML. Compared to the previous version, we have significantly expanded the functionality by adding: fail stats, status codes, hints analyzer results. You can take a look at the new HTML report here Also, we switched to Google Charts for rendering charts since it's free and under Apache 2.0 license.

Runtime Improvements#

In this section, we will discuss some internals of NBomber that improve the performance and stability of the system.

Stats collecting#

We refactored our stats collecting module to use the actor model via F# MailboxProcessor and to have one StatsScenarioActor per scenario. Also, we send metrics by batches to eliminate the overload of one actor by many concurrent ScenarioActors that execute steps and send metrics.

Minimizing RAM usage#

Compression of stats results#

We refactored our statistics module to fix the issue with memory growing footprint for long-running tests. Now you can run tests that will work for years and have a stable (usually small) memory footprint. This is achieved essentially due to one important optimization: all results are located in memory but decomposed into buckets with the same or very similar results. Each such bucket has its own counter, which is monotonically growing. Due to this optimization, we got quite effective compression. Aside from this, such optimizations are often used in time-series databases where many metrics of the same value or very similar can arrive at one point in time.

Default step timeout#

Another important thing was introducing default timeout for steps. NBomber v1 did not support any timeouts out of the box, and this task shifted to the shoulders of the developers. For example, the HTTP plugin added its own timeout, which was quite sane. But on the other hand, everyone needs such a basic thing as a timeout, and forcing everyone to implement it is not a good idea. Also, newbies often weren't aware of this, and afterward, it led to the thread pool starvation problem. It especially was visible when testing very slow API using a big request rate, say 2-4K requests per sec from 1 node. It leads to many tasks (.NET Task) was activated, consumed RAM, and never finished but growing. In the new version, each step by default contains a timeout of 1 second and can be changed by the user.

Removed uneccesary memory allocations#

We refactored a few places where we did extra memory allocations on every step invocation. For example, StepContext was previously created on every invocation.

Task computation expression#

Internally, we use FsToolkit.ErrorHandling library that provides handy F# computation expressions and corresponding extensions. With the latest updates to this library for work with Task, we have automatically switched to using Ply library, which provides a low overhead Task computation expression.

Low-level optimizations#

GC tuning#

Also, we tuned GC a bit. At runtime, we set:

And also, for the project settings, we apply:

New Statistics#

One of the major changes we had been working on is the statistics module. Initially, we didn't really want to change it, but using NBomber v1 in production, we often ran into bottlenecks or questions regarding providing advanced statistics.

Ok and Fail stats#

An important addition to the previous statistics was that we started tracking the fail stats with the full scope, including data transfer for fails, latency, and percentiles.

Now you can write your assertions for fails stats too. Also, you will see them in reporting, including real-time reporting too.

Unit of measure#

In the new version, we changed basic data types to represent latency and data transfer.

Latency#

We started using float instead of integer to expand the range of possible results for latency, for example: [0.2 ms, 1.5 ms, 9.3 ms]. We were pushed to this by the case of testing a high-speed in-memory database that had a latency of less than 1 ms, and in our reports, we saw it as 0 ms instead of 0.1 - 0.5 ms. Just for the record, most load test tools represent latency as integer, meaning using them, you cannot cover such cases since your results will be 0 ms.

Data transfer#

We switched to using Byte instead of KB to keep parity with standard tools for data transfer stats.

In addition, we provide helper functions for converting Byte to KB and MB. This can be useful when you want to assert on KB or MB.

Hint Analyzer#

In this release, we have integrated a new HintAnalyzer feature that allows to analise the results of statistics and, based on this, displays hints. HintAnalyzer may refer not only to the statistics data but also to the use of certain plugins. Any NBomber plugin can add new analyzers. For example, PingPlugin pings the target host before starting any test, and after that, it analyzes received results and prints hints.

plugin stats: NBomber.Plugins.Network.PingPlugin

HostStatusAddressRound Trip TimeTime to LiveDon't FragmentBuffer Sizenbomber.comSuccess104.248.140.12858 ms128False32 bytes

Here's a hint by PingPlugin.

sourcenamehintWorkerPluginNBomber.Plugins.Network.PingPluginPhysical latency to host: 'nbomber.com' is bigger than 2ms which is not appropriate for load testing. You should run your test in an environment with very small latency.

Also, HintAnalyzer can suggest some other hints related to the usage of NBomber.

sourcenamehintScenariosimple_httpStep 'fetch_html_page' in scenario 'simple_http' didn't track data transfer. In order to track data transfer, you should use Response.Ok(sizeInBytes: value)

You can disable HintAnalyzer if you want

New API#

In the new version, we have added a couple of additions that improve the convenience of writing test scripts without adding new abstractions to NBomber itself. We are very scrupulous about adding any new features to NBomber as they can introduce additional abstractions and hence additional complexity. Our initial goal was to build an easy-to-use framework with as few abstractions as possible. Therefore, any new feature that we consider should be easy to understand and harmoniously fit into NBomber.

Status code#

We have added the ability to specify any status codes you want or your protocol, or your API service returns.

Also, we provide statistics on status codes.

Step invocation count#

For certain tests, it may be necessary to understand the current step invocation count. For example, you want to change the behavior of your step execution when the step's invocation counter is reached 100 invocations. For such cases, you need to start a counter and manually increment it. NBomber now supports this out of the box.

Step timeout#

In the new version, each step by default contains a timeout of 1 second and can be changed by the user.

Dynamic step order#

We decided to provide the ability to change the order of the steps at runtime dynamically. Also, it allows changing the number of steps per scenario iteration. This feature opens up a new horizon of possibilities for writing load tests. We came up with this idea from a real case when we tested the database, and we needed to introduce a certain randomity into our tests. For a better understanding, I suggest looking at an example:

Basically, with this feature, you can introduce convenient load distribution. For example, you can define a scenario where you will test Redis database with the following request distribution: 30% for write requests and 70% for read requests.

Scenario info#

It's a context property that contains info about the current Scenario. The main use case is to use ScenarioInfo.ThreadId as correlation id for your requests. Another one is to use ScenarioInfo.ThreadNumber for partition your requests. For example, you have a database, and you want to split all your requests into 4 partitions.

TimeSpan extensions#

We added TimeSpan extensions for F# and C# to have a more expressive API.

Init-only scenarios#

NBomber had the ability to provide Scenario initialization via Scenario.init in the very first version. It had a mandatory restriction to create a Scenario you should provide at least one Step.

This was convenient until the moment when several scenarios had a dependency on the same initialization. In this case, only one initializer should be executed, and it's getting tricky since NBomber v1 didn't provide any functionality for this.

In NBomber v2, we can define Scenario without steps and have only the init or clean function.

LoadSimulation inject random#

Injects a random number of scenario copies (threads) per 1 sec during a given duration. Use it when you want to maintain a random rate of requests without being affected by the target system's performance under test.

Logs#

In the new version, we write a log file to the current session folder.

HTTP plugin#

NBomber.Http has been slightly modified to make it easier to be used. In the previous version, this plugin provided an HttpStep wrapper over the standard Step that hid many details and added extra magic. In the new version, we have reworked the approach with DSL plugins to use standard NBomber abstractions instead of creating new ones. Now, the HTTP plugin contains HttpClientFactory and has an HTTP module with functions for building HTTP requests.

InfluxDb plugin#

InfluxDb plugin is supplemented with tracking fail stats and load simulation value. In Grafana, you can render your load simulation timeline and compare it with what your target system can handle.