A short guide on reducing inbox clutter from automatically generated email updates.

Introduction

At Flexport, we communicate with our customers — and their software — in many ways. API and EDI integrations are ideal because they allow structured data to flow seamlessly with no humans involved. Sometimes, however, emails are necessary to facilitate coordination.

To save our operations teams’ time, we send certain emails automatically using Rails ActionMailer. While the approach works great in general, we’ve had to overcome a few challenges related to email threading. This post outlines the problem and how we solved it.

One benefit of manual, conversation-based emails is that replies create threaded email conversations. Most email clients thread messages to make them easier to follow. Some Flexport customers reported that their inboxes were being filled by automated emails that weren’t being threaded properly. To make sure subsequent updates for a given request were not cluttering ourusers’ inboxes, we needed our automated emails to appear grouped together, just like a hand-typed email would.

One small thread for mail, or one giant mess for mail clients?

To understand how to recreate this behavior programmatically, let’s look at how replies work in the mail client.

How Email Threading Works

Before an email client sends an email, it generates a uniqueheader called a Message-ID that is added to the message. Message-IDs are supposed to be unique, and have a specified format as defined in the Internet Message Format standards. A Message-ID is used to reference an individual email across the client.

When a user clicks ‘Reply’ to an email in Gmail, the reply has its own Message-ID generated, distinct from the original. A new header is also added to the reply, called References, and its value is the Message-ID from the original email. This is fairly self explanatory: this header tells the email client that this email “references” the one it is replying to.

Every additional reply will append another reference to this header, corresponding to all of the entries in the thread. However, even if References only contains the Message-ID of the original message, most email clients will thread all the messages with that References value together.

Additionally, email clients will add the “Re:” prefix to messages, to visually signal that they are “Regarding” a previous message. In Gmail, this subject only appears in the inbox view, and not in the email thread. This can be confirmed by clicking ‘More’ next to the ‘Reply’ button and clicking ‘Show Original’.

While Google’s Gmail “magic” may sometimes group emails together with one or neither of the above techniques, using both can guarantee that they will be visually threaded together.

Using ActionMailer

Based on this specification for Message-ID, it is actually fairly straightforward to send programmatic email replies:

1. Set the References header of the reply to the Message-ID of the original.
2. Change the subject of the reply to “Re: <original subject>”

Let’s take a look at how to accomplish this using Ruby’s ActionMailer. Below is a simple code example MoonLandingService that automatically sends emails from a lunar expedition back to earth:

Mailer Class

Mailer Service

The send_landing_updates method first creates and sends an email to start the thread. Then, the generated Message-ID header of the message is accessed using .message_id. This can be stored in email logs for further use, or as additional metadata inside a more complex model.

Then the user can create the reply and set the references header to the Message-ID of the first. It is important to note that it is surrounded by the < and > symbol: the .message-id call will return the Message-ID without these symbols, but they are needed to properly format the header.

And that’s it! Following this pattern will result in a string of messages like this:

Inbox viewConversation view

Other Things to Consider

The above example highlights a simple usage of the References header for email threading. In reality, these threads can be complex, with multiple users replying to different emails in the thread. Additionally, replying to an email that was not initially sent programmatically requires retrieving the Message-ID of that email from the web server (or some other technique). Wes Morgan’s blog post Understanding Email Headers goes into more details about how these headers work.

I hope this helps anyone working on automating email infrastructure. Feedback or questions welcome!