Zero To Production in Rust

Zero To Productionis a book that I will be writing in the open, publishing one chapter at a time on this blog. If you’d like to be notified when an episode comes out you can subscribe to the mailing list:

Chapter #1

1. 
   
2. Installing The Rust Toolchain
   
   • 2.1. Compilation targets
   • 2.2. Release channels
   • 2.3. What toolchains do we need?
3. 
   
4. • 
     
   • 
     
   • 4.3. What should I use?
5. Continuous Integration
   
   • 
     
   • 5.2. Ready-to-go CI pipelines

1. Getting Started

There is more to a programming language than the language itself: tooling is a key element of the experience of using the language.

The same applies to many other technologies (e.g. RPC frameworks like gRPC or Apache Avro) and it often has a disproportionate impact on the uptake (or the demise) of the technology itself.

Tooling should therefore be treated as a first-class concern both when designing and teaching the language itself.

The Rust community has put tooling at the forefront since its early days: it shows.

We are now going to take a brief tour of a set of tools and utilities that are going to be useful in our journey. Some of them are officially supported by the Rust organisation, others are built and maintained by the community.

2. Installing The Rust Toolchain

There are various ways to install Rust on your system, but we are going to focus on the recommended path: via rustup .

Instructions on how to install rustup itself can be found at https://rustup.rs .

rustup is more than a Rust installer - its main value proposition is toolchain management .

A toolchain is the combination of a compilation target and a release channel .

2.1. Compilation targets

The main purpose of the Rust compiler is to convert Rust code into machine code - a set of instructions that your CPU and operating system can understand and execute.

Therefore you need a different backend of the Rust compiler for each compilation target , i.e. for each platform (e.g. 64-bit Linux or 64-bit OSX) you want to produce a running executable for.

The Rust project strives to support a broad range of compilation targets with various level of guarantees. Targets are split into tiers , from “guaranteed-to-work” Tier 1 to “best-effort” Tier 3.

An exhaustive and up-to-date list can be found here .

2.2. Release channels

The Rust compiler itself is a living piece of software: it continuously evolves and improves with the daily contributions of hundreds of volunteers.

The Rust project strives for stability without stagnation . Quoting from Rust’s documentation :

[..] you should never have to fear upgrading to a new version of stable Rust. Each upgrade should be painless,

but should also bring you new features, fewer bugs, and faster compile times.

That is why, for application development, you should generally rely on the latest released version of the compiler to run, build and test your software - the so-called stable channel.

A new version of the compiler is released on the stable channel every six weeks- the latest version at the time of writing is v1.43.1 .

There are two other release channels:

• beta , the candidate for the next release;
• nightly , built from the master branch of rust-lang/rust every night, thus the name.

Testing your software using the beta compiler is one of the many ways to support the Rust project - it helps catching bugs before the release date.

nightly serves a different purpose: it gives early adopters access to unfinished featuresbefore they are released (or even on track to be stabilised!).

I would invite you to think twice if you are planning to run production software on top of the nightly compiler: it’s called unstable for a reason.

2.3. What toolchains do we need?

Installing rustup will give you out of the box the latest stable compiler with your host platform as a target.

Some of the tools we will be using on our development machine (e.g macro expansion) will rely on the nightly compiler. While nightly is discouraged for production workloads it is not a big deal if something fails on our local machine - we can live with that.

You can install the nightly compiler by running

1rustup toolchain install nightly --allow-downgrade

Some components of the bundle installed by rustup might be broken/missing on the latest nightly release: --allow-downgrade tells rustup to find and install the latest nightly where all the needed components are available.

We are only specifying the release channel, nightly - rustup uses our host platform as default for the target. On my system, if I wanted to be explicit, I would have to use

1rustup toolchain install nightly-x86_64-unknown-linux-gnu --allow-downgrade

You can update your toolchains with rustup update , while rustup toolchain list will give you an overview of what is installed on your system.

We will not need (or perform) any cross-compiling - our production workloads will be running in containers, hence we do not need to cross-compile from our development machine to the target host used in our production environment.

3. Project Setup

A toolchain installation via rustup bundles together various components.

One of them is the Rust compiler itself, rustc . You can check it out with

1rustc --version

You will not be spending a lot of quality time working directly with rustc - your main interface for building and testing Rust applications will be cargo , Rust’s build tool.

You can double-check everything is up and running with

1cargo --version

Let’s use cargo to create the skeleton of the project we will be working on for the whole book:

1cargo new zero2prod

You should have a new zero2prod folder, with the following file structure:

1zero2prod
2├── Cargo.toml
3├── .gitignore
4├── .git
5└── src
6   └── main.rs

The project is already a git repository, out of the box.

If you are planning on hosting the project on GitHub, you just need to create a new empty repository and run

1cd zero2prod
2git commit -am "Project skeleton"
3git remote add origin [email protected]:YourGitHubNickName/zero2prod.git
4git push -u origin master

We will be using GitHub as a reference given its popularity and the recently released GitHub Actions feature for CI pipelines, but you are of course free to choose any other git hosting solution (or none at all).

4. IDEs

The project skeleton is ready, it is now time to fire up your favourite editor so that we can start messing around with it.

Different people have different preferences but I would argue that the bare minimum you want to have, especially if you are starting out with a new programming language, is a setup that supports syntax highlighting, code navigation and code completion.

Syntax highlighting gives you immediate feedback on glaring syntax errors, while code navigation and code completion enable “exploratory” programming: jumping in and out of the source of your dependencies, quick access to the available methods on a struct or an enum you imported from a crate without having to continuously switch between your editor and doc.rs .

You have two main options for your IDE setup: rust-analyzer and IntelliJ Rust.

4.1. Rust-analyzer

rust-analyzer is an implementation of the Language Server Protocol for Rust.

The Language Server Protocol makes it easy to leverage rust-analyzer in many different editors, including but not limited to VS Code, Emacs, Vim/NeoVim and Sublime Text 3.

Editor-specific setup instructions can be found here .

4.2. IntelliJ Rust

IntelliJ Rust provides Rust support to the suite of editors developed by JetBrains.

If you don’t have a JetBrains license, IntelliJ IDEA is available for free and supports IntelliJ Rust.

If you have a JetBrains license, CLion is your go-to editor for Rust in JetBrains’ IDE suite.

4.3. What should I use?

As of May 2020, IntelliJ Rust should be preferred.

Although rust-analyzer is promising and has shown incredible progress over the last year, it is still quite far from delivering an IDE experience on par with what IntelliJ Rust offers today.

On the other hand, IntelliJ Rust forces you to work with a JetBrains’ IDE, which you might or might not be willing to. If you’d like to stick to your editor of choice look for its rust-analyzer integration/plugin.

It is worth mentioning that rust-analyzer is part of a larger library-ification effort taking place within the Rust compiler: there is overlap between rust-analyzer and rustc , with a lot of duplicated effort.

Evolving the compiler’s codebase into a set of re-usable modules will allow rust-analyzer to leverage an increasingly larger subset of the compiler codebase, unlocking the on-demand analysis capabilities required to offer a top-notch IDE experience.

An interesting space to keep an eye on in the future.

5. Continuous Integration

Toolchain, installed.

Project skeleton, done.

IDE, ready.

One last thing to look at before we get into the details of what we will be building: our Continuous Integration (CI) pipeline .

In trunk-based development we should be able to deploy our master branch at any point in time.

Every member of the team can branch off from master , develop a small feature or fix a bug, merge back into master and release to our users.

Continuous Integration empowers each member of the team to integrate their changes into the main branch multiple times a day.

This has powerful ripple effects.

Some are tangible and easy to spot: it reduces the chances of having to sort out messy merge conflicts due to long-lived branches. Nobody likes merge conflicts.

Some are subtler: Continuous Integration tightens the feedback loop . You are less likely to go off on your own and develop for days or weeks just to find out that the approach you have chosen is not endorsed by the rest of the team or it would not integrate well with the rest of the project.

It forces you to engage with your teammates earlier than when it feels comfortable, course-correcting if necessary when it is still easy to do so (and nobody’s is likely to get offended).

How do we make it possible?

With a collection of automated checks running on every commit - our CI pipeline .

If one of the checks fails you cannot merge to master - as simple as that.

CI pipelines often go beyond ensuring code health: they are a good place to perform a series of additional important checks - e.g. scanning our dependency tree for known vulnerabilities, linting, formatting, etc.

We will run through the different checks that you might want to run as part of the CI pipeline of your Rust projects, introducing the associated tools as we go along.

We will then provide a set of ready-made CI pipelines for some of the major CI providers.

5.1. CI steps

5.1.1. Tests

If your CI pipeline had a single step, it should be testing.

Tests are a first-class concept in the Rust ecosystem and you can leverage cargo to run your unit and integration tests:

1cargo test

cargo test also takes care of building the project before running tests, hence you do not need to run cargo build beforehand (even though most pipelines will invoke cargo build before running tests to cache dependencies).

5.1.2. Code coverage

Many articles have been written on the pros and cons of measuring code coverage.

While using code coverage as a quality check has several drawbacks I do argue that it is a quick way to collect information and spot if some portions of the codebase have been overlooked over time and are indeed poorly tested.

The easiest way to measure code coverage of a Rust project is via cargo tarpaulin , a cargo subcommand developed by xd009642 . You can install tarpaulin with

1# At the time of writing tarpaulin only supports x86_64 CPU architectures running Linux.
2cargo install cargo-tarpaulin

while

1cargo tarpaulin --ignore-tests

will compute code coverage for you application code, ignoring your test functions.

tarpaulin can be used to upload code coverage metrics to popular services like Codecov or Coveralls - instructions can be found in tarpaulin ’s README .

5.1.3. Linting

Writing idiomatic code in any programming language requires time and practice.

It is easy at the beginning of your learning journey to end up with fairly convoluted solutions to problems that could otherwise be tackled with a much simpler approach.

Static analysis can help: in the same way a compiler steps through your code to ensure it conforms to the language rules and constraints, a linter will try to spot unidiomatic code, overly-complex constructs and common mistakes/inefficiencies.

The Rust team maintains clippy , the official Rust linter.

clippy is included in the set of components installed by rustup if you are using the default profile. Often CI environments use rustup ’s minimal profile, which does not include clippy .

You can easily install it with

1rustup component add clippy

If it is already installed the command is a no-op.

You can run clippy on your project with

1cargo clippy

In our CI pipeline we would like to fail the linter check if clippy emits any warnings.

We can achieve it with

1cargo clippy -- -D warnings

Static analysis is not infallible: from time to time clippy might suggest changes that you do not believe to be either correct or desirable.

You can mute a warning using the #[allow(clippy::lint_name)] attribute on the affected code block or disable the noisy lint altogether for the whole project with a configuration line in clippy.toml or a project-level #![allow(clippy::lint_name)] directive.

Details on the available lints and how to tune them for your specific purposes can be found in clippy ’s README .

5.1.4. Formatting

Most organizations have more than one line of defence for the master branch: one is provided by the CI pipeline checks, the other is often a pull request review.

A lot can be said on what distinguishes a value-adding PR review process from a soul-sucking one - no need to re-open the whole debate here.

I know for sure what should not be the focus of a good PR review: formatting nitpicks - e.g. Can you add a newline here? , I think we have a trailing whitespace there! , etc.

Let machines deal with formatting while reviewers focus on architecture, testing thoroughness, reliability, observability. Automated formatting removes a distraction from the complex equation of the PR review process. You might dislike this or that formatting choice, but the complete erasure of formatting bikeshedding is worth the minor discomfort.

rustfmt is the official Rust formatter.

Just like clippy , rustfmt is included in the set of default components installed by rustup . If missing, you can easily install it with

1rustup component add rustfmt

You can format your whole project with

1cargo fmt

In our CI pipeline we will add a formatting step

1cargo fmt -- --check

It will fail when a commit contains unformatted code, printing the difference to the console.

You can tune rustfmt for a project with a configuration file, rustfmt.toml . Details can be found in rustfmt ’s README .

5.1.5. Security vulnerabilities

cargo makes it very easy to leverage existing crates in the ecosystem to solve the problem at hand.

On the flip side, each of those crates might hide an exploitable vulnerability that could compromise the security posture of your software.

The Rust Secure Code working group maintains an Advisory Database - an up-to-date collection of reported vulnerabilities for crates published on crates.io .

They also provide cargo-audit , a convenient cargo sub-command to check if vulnerabilities have been reported for any of the crates in the dependency tree of your project.

You can install it with

1cargo install cargo-audit

Once installed, run

1cargo audit

to scan your dependency tree.

We will be running cargo-audit as part of our CI pipeline, on every commit.

We will also run it on a daily schedule to stay on top of new vulnerabilities for dependencies of projects that we might not be actively working on at the moment but are still running in our production environment!

5.2. Ready-to-go CI pipelines

Give a man a fish, and you feed him for a day. Teach a man to fish, and you feed him for a lifetime.

Hopefully I have taught you enough to go out there and stitch together a solid CI pipeline for your Rust projects.

We should also be honest and admit that it can take multiple hours of fidgeting around to learn how to use the specific flavour of configuration language used by a CI provider and the debugging experience can often be quite painful, with long feedback cycles.

I have thus decided to collect a set of ready-made configuration files for the most popular CI providers - the exact steps we just described, ready to be embedded in your project repository:

• GitHub Actions ;
• CircleCI ;
• Travis .

It is often much easier to tweak an existing setup to suite your specific needs than to write a new one from scratch.

Feel free to get in touch if you would like to provide a template for a CI provider that is currently not covered in the list above.

Zero To Productionis a book that I will be writing in the open, publishing one chapter at a time on this blog. If you’d like to be notified when an episode comes out you can subscribe to the mailing list:

Book ToC

The Table of Contents is provisional and might change over time. The draft below is the most accurate picture at this point in time.

• Who Is This Book For
• What Is This Book About

1. Getting Started (this blog post)
   
   • Installing The Rust Toolchain
   • Project Setup
   • IDEs
   • Continuous Integration
2. Our Driving Example
   
   • What Should Our Newsletter Do?
   • Working In Iterations
3. Sign Up A New Subscriber
   
   • Application Scaffolding
   • Our First Integration Test
   • Reading Request Data
   • Adding A Database
   • Persisting A New Subscriber
4. Publish A Newsletter Issue
   
   • Writing A REST Client
   • Mocking Third-Party APIs
5. Logging
   
   • The Facade Pattern
   • Logging Levels
   • Log Setup
6. Reject Invalid Subscribers
   
   • Result
   • Modeling With Types #1
7. Survive Delivery Failures
   
   • Simulating API Errors
8. Tracing
   
   • Structured Logging
   • Spans
   • OpenTelemetry
   • Jaeger
9. Send A Confirmation Email On Sign Up
   
   • Migrating Your Database
   • Modeling With Types #2
   • Handling Confirmations
   • Send Newsletter Only To Confirmed Subscribers
10. Metrics
    
    • Prometheus
    • Grafana
11. Send Emails Asynchronously
    
    • Adding A Message Broker
    • Enqueueing Tasks
12. Fulfilling Email Tasks
    
    • Adding An Actor Queue Worker
    • Basic Retries
    • Failure Injection
    • Idempotency
13. Benchmarking
    
    • Cargo bench
    • Criterion
    • Load testing