An updated runtime performance benchmark of all five programming languages supported by AWS Lambda

AWS recently announced their support for both C# (Net Core 2.0) and Go programming languages for Lambda functions.

As an engineer who maintained serverless-golang, I was curious to learn how the performance of each runtime compared — especially after participating in some insightful discussions with other developers on this topic.

To find out for myself, I developed a series of performance tests using the Serverless Framework — and observed some very interesting results.

The 2018 AWS Lambda Performance Benchmark

My benchmarks were based on the performance testing and comparisons of AWS Lambda functions that Yan Cui conducted in 2017. For consistency, I forked Yan’s codebase and added some extra testing templates for 2018.

Similar to the original performance tests, we’ll ignore the initial cold start time — and focus only on the duration metric to compare runtime performance between the different languages.

All of the code used for the test is available in my GitHub repository.

Each Lambda function is configured with the default 1024MB memory set by Serverless Framework and deployed to North Virginia (us-east-1). Upon deployment there are a total of 12 Lambda functions.

To manage the performance load test, I created a script that calls artillery.io. The script executes a load test across all twelve APIs over a one-hour period:

Observation 1 — .Net Core 2.0 significantly outperforms

Both C# and F# on .Net Core 2.0 exceeds all expectations and outperforms all other runtimes in average duration. AWS Lambda developers on .Net Core should consider .Net Core 2.0 as the default — and upgrade all existing projects still using 1.0.

Observing the average durations on the graph, both C# and F# on .Net Core 2.0 are consistently lower than other runtimes throughout the 1-hour period:

Observation 2 — Go performance is comparable to Java

The runtime performance of Go is very similar to Java — with slightly less variance. There are no significant performance gains from using either UPX compression or the Python Shim.

For the 1-hour period, the average duration of Go is exactly the same as Java:

Observation 3 — Consistent performance of compiled vs dynamic

In the original 2017 benchmark results, the compiled languages of Java and .Net Core 1.0 were clearly more consistent compared to the dynamic languages — Python and Node.js.

Based on the latest 2018 performance testing results — we are no longer observing any significant variances between Java and the newer compiled languages (.Net Core 2.0 and Go), and the original dynamic languages.

In 2018, we’ve observed much more consistency amongst all the languages. It’s safe to assume that while new languages are being added to AWS Lambda, existing languages are not forgotten — and are being continuously improved.

Observation 4 — Go packages are bigger than Java

Go binaries compiled with -s and -w linker flags results in a bigger package size (2.3mb) than Java (2.0mb). By using upx, it is possible to shrink the binary size — but without any significant gains.

C# has the smallest footprint out of all compiled languages with 201KB, whereas F# adds on about an extra 1MB of dependencies.

Final thoughts

With Go and .Net Core 2.0 support, AWS continues to lead the FaaS and Furious race as the most mature provider — with the widest range of supported languages. I am really pleased to see how much the performance and stability of .Net Core 2.0 tooling for AWS Lambda has improved since its initial 1.0 release.

As an advocate of the Go programming language, it still feels like the Go runtime needs some work. Based on the results of the performance test, .Net Core 2.0 can execute up to 3x faster than Go.

I suspect the performance benefits of .Net Core 2.0 over Go are a result of some specific under-the-hood optimizations at container level, and potentially JSON serialization enhancements or caching. Although disappointed with the results, I’m confident that the performance of Go will improve as the AWS Lambda platform continues to quickly evolve.

The benchmarks results don’t reflect production use cases — and ultimately the differences in performances are quite subtle. However, compared to last year, architects and engineers should have more confidence in choosing from all the available runtimes when considering AWS Lambda functions in 2018.

Let me know your thoughts in the comments below, or connect with me directly on Twitter @yunzhilin.

Yun is the VP of Engineering APAC at Contino — a global consultancy that enables highly regulated organizations accelerate innovation through modernized software delivery.

This article is also available in simplified Chinese here and Russian here

Edit: I wish to thank the community for all your comments and feedback. I have taken them on-board and will incorporate them into the next benchmark article. This time with multiple cloud vendors.