E-Series performance with Elasticsearch

Introduction

I have a lengthy “generic” post on Elasticsearch 7 with NetApp storage and while all arrays mentioned therein have sweet spots, E-Series is my favorite because it works great for all scenarios, large, small, Hot, Warm, Cold, and even snapshots (if you prepare a software S3 server backed by E-Series storage).

It works great because most Elasticsearch users need what E-Series offers: reliability, simplicity, performance, scalability, cost-effectiveness.

Instead of maintaining storage on every Elasticsearch node, you maintain it in one place, and use it from all Elasticsearch nodes. Rather than having two replicas (i.e. RF=3), you make one (RF=1) or none.

I wrote about these things in other posts (including one VM-based performance test with EF280) you can use the search feature to find them), but I wanted to re-check those claims with more testing which is never easy as I don’t have lab where such tests are possible and easy to conduct.

Environment

Configuration:

• Elasticsearch 8.6.2 in Docker on Rocky Linux 9
• One box with 2 x Intel Xeon 6136 CPUs and 96 GB RAM
• One EF-Series EF570 with SSDs (iSER, i.e. iSCSI over IB), with both inputs and Elasticsearch storage on DDP (RAID6-like disk pool)

On this system I ran Elasticsearch two ways:

• Single large Docker container with zero replicas (copies)
• Three Docker containers with one replica (one secondary copy, RF=2)

Tests:

• General performance metrics - why, because that’s what most people do
• SIEM - very popular Elasticsearch workload
• Kubernetes metrics - same as general, but more of it

What I wanted to see:

• One vs. three nodes
• One vs. two copies of data
• Impact of different workloads on Docker (server) resources
• Comparison in indexing performance between different data sources
• Other minor tests (batch size, number of clients, etc.)

Although this isn’t nearly enough to be able to size E-Series for specific workloads, it’s better than guessing it based on what I had before with Elasticsearch 7 (much simpler, less extensive, less comprehensive).

In all tests I used the same Rocky Linux box to run Kibana, and clients to submit data, plus other (idle) services weren’t shut down. I wasn’t looking for “maximum performance” since I knew I’d run out of CPU resources no matter what I did.

Elasticsearch performance in Docker

Unlike Splunk, Elasticsearch has supported Docker (and Kubernetes) for a while and has been production-ready for years. So there’s nothing new here - it works, and it works well. It also runs well on Kubernetes on E-Series (see this post for more).

But I don’t spend much time using it in Docker and Kubernetes, so I appreciate these opportunities.

With one Elasticsearch instance, CPU utilization varied, but generally didn’t go above 50%.

With three Elasticsearch instances and one replica, CPUs can get more busy. IO wait time is still close to nothing.

Simple Web log indexing

This is HTTP log on a single Docker instance (so, no copies). This is simple, and therefore relatively faster than other data types: 400,000 documents/second.

EF570 needed less than 200 MB/s for that. I marked IOPS (~2,000) and MB/s (~200) to illustrate that IO requests are close to 100 kB on average.

With three nodes and one replica, we’d probably see twice as much. Before I move on to that, let’s see the effect of “force-merge one segment” on storage: > 1 GB/s from a single container.

You may say it’s “just” 537 MB/s write, which is true, but data has to be read off disk and written back, so in this case 1,148 MB/s isn’t something to be taken lightly when one has 10 or 20 Elasticsearch nodes.

This SIEM workload was “heavier” than the plain Web log, and a single Docker node could do around 110,000 documents per second.

The array wasn’t busy, just 400-600 MB/s, which is perhaps 10% of its maximum.

I didn’t see more than 750 MB/s write with SIEM data. (The blue (read) line appears because JSON data is read from SSD DDP before it’s sent to Elasticsearch to be indexed and ingested.)

What about search tests? Not much. With E-Series and SSDs, search is much less likely to cause storage saturation. More on that below.

Time-series database (TSDB) for performance monitoring use cases

This is something you may need in a medium to large Kubernetes environment.

On the array:

TSDB indexing with one Elasticsearch node and no replicas peaked at below 300 MB/s.

Indexing latency

This wasn’t unique to TSDB data, but to all tests: with Elasticsearch in Docker using 20 CPUs, latency was negligible.

Another example:

It’s not until three Elasticsearch instances are used, and data written twice (1 replica, i.e. RF=2), that we can observe differences between runs.

Search latency

Searches of hot data are often satisfied from RAM, and due to indexes living on SSDs, only complex searches took more than 10ms.

As far as I could tell most of that latency was software-related, in order to select and filter data.

In this example producing aggregate report is what took a long time (compare that with a peak latency of 0.5ms during data ingestion and indexing).

Not that CPU utilization during search is always high (it varies, obviously, between one and several vCPUs so it’s relatively light to begin with), but storage utilization was very low (usually well below 100 MB/s).

Part of it is because EF570 storage controller can read-cache many GB/s of data, but even without that I doubt search would present any problems: E-Series is roughly twice as fast when reading data from flash media.

The little read (blue line) bump at 6:10am was one of the more demanding search tests.

Effect of Elasticsearch replicas

Here we have three Elasticsearch nodes in three Docker containers and one replica, so indexing has a 100% overhead. Still, the effect on IO latency is not even noticeable.

Why? Because that 200-300 MB/s in writes (without replicas that’d be 100-200 MB/s) is still too little to move the needle. Each bump here is one of seven indexing runs from the screenshot above.

TSDB indexing with one node and no replica was 36K/s and with three nodes and one replica it’s 44K/s (and 73K/s before replicas are considered). Why?

I figure that’s because three nodes make better use of OS resources - whereas a single ES container couldn’t make use of more than 20-22 vCPUs, three ES in Docker easily used 32-34 vCPUs.

The added IO from making an extra replica was easy to spot (vs. TSDB with no replicas earlier): now we write at 250 MB/s per each container.

And if we really push it and use 24 submitters to 3 Elasticsearch servers, it’s possible to write at close to 1 GB/s without any performance tuning.

Should we use Elasticsearch replicas with E-Series

For Hot Tier, I’d say yes.

NetApp recommends two copies for HDFS with E-Series as well. It’s usually a small fraction of data, and it protects both data and uptime from filesystem corruption, server crashes, Elasticsearch and OS upgrades, etc.

Two copies on the same array can use the same DDP or RAID10, but also separate (however, given that it’s protected and redundant storage, it’d be very unusual to do that).

A replica on each array can protect you from array, rack or site failures. I wrote about that here.

For Warm Tier, the risk of data loss is slightly lower and the impact of temporary access to data (which can be restored after a server is fixed, for example) is usually not as large, so it may make more sense to use just one copy.

Should we use E-Series’ snapshots with Elasticsearch

Sometimes people ask if they can use E-Series snapshots to protect Elasticsearch data. Yes, it’s possible - but why?

Restoring a snapshot likely means several minutes of data loss due to going back to the most recent snapshot an additional one-two minutes due to downtime.

Snapshots can be useful to protect application and data before an upgrade or during scheduled downtime, but generally don’t make a lot of sense in Elasticsearch environments.

Folks who want to protect data should use Cross-Cluster Replication (aka CCR) and protect it across racks or sites.

I guess you could say having a snapshot could help you recover from ransomware or restore a mistakenly deleted index, and that’s true.

But due to the scale-out nature of Elasticsearch indexes, reverting to a snapshot would cause some data loss in Hot Tier across the board, so maybe the only situations where the admin would make that trade-off is in the case of a catastrophic human error or freshly-discovered ransomware attack (where taking a snapshot and restoring from a previous one gets you back online, but without access to logs you probably need to analyze what just happened until you clone that last snapshot with potentially corrupt data). This post talks E-Series snapshots with anti-ransomware considerations.

Saturating E-Series controllers

Current entry-level model EF300 is similar to the previous mid-range EF570 which I used for this post. EF600 can do more than twice as much.

Let’s consider CPU utilization on an idle EF570 (similar to EF300 available today): 30%.

While running one of those Elasticsearch indexing tests and indexing 80,000 documents per second, combined read and write throughput was over 1 GB/s.

What did that do to array’s CPU utilization? It barely moved the needle: 33% (on one of the controllers).

Both EF300 and EF600 have plenty of horsepower for heavy Elasticsearch workloads.

In this environment we’d need several servers to get even close - maybe half a dozen for EF570 or a dozen for EF600.

Sixteen log submitters are enough to keep three containers on this server busy (second top output at the top) and adding more just makes the CPUs work harder.

Indexing performance with 16 submitters doesn’t work slower, but it does result in a lower latency.

So there’s no advantage in “pushing” these containers to work harder. We’d need more servers.

Performance observations

Elasticsearch 8 running in Docker connected to an EF570 performed similarly to Elasticsearch 7 in VMs connected to an EF280. Why? Because in both cases I had just one server to run everything and it ran out of resources.

That was expected, but in this round of testing I found a bit more about the impact of input data, replication, number of containers, number of workers, batch sizes, and such.

No matter what I did with 20-30 cores, it barely registered on E-Series: even the entry-level EF300 should be enough to serve multiple Elasticsearch nodes, whereas the mid-range EF600 could be good even for one-two dozen of them (depending on workload).

While indexing performance varied depending on type of data and number of replicas, we saw that we generally got 100-200 MB/s of write workload per node, and in extreme cases over 1 GB/s in combined read and write performance per node.

So, assuming one replica and 500 MB/s indexing performance, an EF300 sized to 5,000 MB/s write could be enough for half a dozen Elasticsearch nodes with 32-64 cores. This isn’t a sizing rule (and completely ignores search and other workloads), but together with examples given above provides a better idea of how E-Series should be sized for E-Series. It’d take relatively more smaller nodes - especially smaller VMs with 8-16 vCPUs - to saturate an EF600 with > 16 NVMe disks.

No QLC for Hot Tier

I wrote about 15TB QLC SSD support on E-Series. Those are (relatively) affordable, but won’t work as Hot Tier.

They may be fine as Warm Tier, though.

Summary

When it comes to Elasticsearch workloads all-flash E-Series arrays do what’s required, and don’t what isn’t.

Containerized Elasticsearch works very well, and there’s no particular reason to use VMs for Elasticsearch: a simple approach is to run it in Docker containers on bare metal servers, and a sophisticated approach is something like Elasticsearch on Kubernetes (see that post on DirectPV) - that requires more skills, but some management tasks (including container and storage management) are probably easier once you get used to it.