How Spotify almost had its email marketing shut down because of spam traps

By Derek du Preez

October 22, 2021

Audio mode

Spotify is one of the world's largest online music streaming services and sends billions of marketing emails to its users every year. However, in December 2020 the team responsible for email campaigns was contacted by Twilio SendGrid - the platform it uses to manage email marketing - and was told that it was close to being shut down. 

Why? Because Spotify didn't realize that it was sending millions of emails to what are known as ‘spam traps': bad actors that are looking to harm Spotify's systems. Matt Gioe, Engineering Manager at Spotify, and the person heading up the email campaign office, was speaking at Twilio's Signal event this week, where he warned other organizations that they need to get to grips with both the art and science of email, in order to avoid a similar situation. 

The problem was brought to Spotify's attention in December 2020, during the company's largest campaign of the year, Holiday 2020. During this time Spotify sends approximately 1 billion emails to 170 email subscribers across 92 countries over roughly a 30 day period. It was in the middle of this huge campaign that Twilio SendGrid got in touch. Gioe said: 

I received a series of pretty frantic emails, messages and phone calls from three Twilio employees, two of them who I've worked with before, and one that I'd actually never heard of. 

It turns out this third person was Len Shneyder, who is the VP of Industry Relations at Twilio. Len wanted me to know…that if I don't do something, I'm going to get shut down…and by me, he really meant Spotify, and by Spotify he meant all of Spotify's emails. So this is pretty terrifying.

This kind of came out of the blue. This was actually my first marketing campaign that I'd ever worked on. But you can imagine that I was not having a great day.

Gioe said that the reason he's sharing this story is because the exact circumstances could happen to any organization, particularly ones with the scale of Spotify. However, whilst there is a lot of ‘science' that is involved with email marketing, Gioe warned that this problem was not going to be solved with technical engineering. He said: 

This type of catastrophic conversation is not going to be saved through good software engineering. Success in email is not something that is simply an engineering domain. When you fall into issues, a lot of the time it's something that can be solved by just great software -  if you're not reaching out to people, you're not serving enough messages, you're not actually sending enough emails fast enough. - these are things that any software engineer can handle. 

But there's another class of issues that is not software engineering related and this is important to understand as well. And the distinction between these two things is the difference between science and art. On the one hand, email has a lot to say about science, but on the other hand it has a lot to say about art. It's a specific domain, it has a lot of rules that are not particularly obvious, and you need to really be aware of both.

Spam traps

As already noted, the Holiday campaign is Spotify's largest and most important email campaign across the company. Gioe's team sits in between Twilio SendGrid and the rest of the Spotify organization, in order to coordinate campaigns across the Spotify organization. Everything comes through Gioe's team, in order to avoid "chaos". 

Gioe shared some charts which showed the success of emails being received by Spotify users, which by and large are positive. However, looking closely at certain snapshots in time, Spotify was having approximately 1 million failures. Gioe said: 

As a software engineer, you look at 120 million successes and 1 million failures, and that doesn't seem so bad. But when you're talking about the art of email sending it is actually quite bad. 

One chart Gioe shared, which Spotify wasn't aware of at the time, showed what are known as spam traps. He explained: 

In this third chart, which is really the crux of the issue, is the chart that I wasn't aware of in the middle of holiday 2020. These are the spam traps that we were hitting - and spam traps is not a concept that I'd never heard of before. Spam traps are literally emails that exist only to expose poor sending behaviours, they're only set up to catch you sending bad emails. 

We knew that we were hitting these spam traps because Spamhaus is an organization who monitors these types of things. When Spamhaus catches you sending all of these bad emails to spam traps, they let the world know. And unfortunately Spamhaus, spam traps, these are not things that any of the great software engineers throughout history had any idea about. These are not the things that my heroes had any idea about. 

The more of these that you're hitting, the more it reflects poorly on your reputation as an email sender. And this is all completely non-obvious, you don't even know that you're doing this. You don't know that you're doing this until you're about to get shut down. 

Set up for failure

As one can imagine, being made aware of spam traps and potentially being exposed by Spamhaus in the middle of one the largest marketing campaigns of the year is less than enviable. Gioe described it as "the worst possible moment". 

As such, Spotify spun up a ‘war room' between himself and the people at Twilio SendGrid to understand how it could avoid sending emails to these bad actors and reduce types of domain specific, non engineering bad outcomes. The discussions centered around analyzing the entire lifecycle of Spotify's user history. Gioe explained: 

How do we actually get to send users emails? And what we found was that, functionally speaking, our email programme was set up for failure.

And this is not really that surprising. We're very far away from the point of user registration, that's not the case in many other companies, or many other mailing lists, but we found that there were actually two main components of Spotify that were really hurting us.

On the one hand, users are not, and historically have not been, required to verify their identity when signing up for a Spotify account. You don't have to prove that you're a person to get a Spotify account. And the other side of that is that we are subject to pretty rampant fraud. There is a heavy incentive to create bad accounts and there's a pretty heavy incentive for bad actors to try and harm our systems.

Gioe said that when you combine a lack of verification with the sheer reality of attempted fraud at Spotify, what you end up with is a section of your user base that is harmful. He added: 

We didn't know any of this. This is so far away from us that we didn't have any concept that this was happening. And that's not really that surprising. Even though we have great engineers and even though we're hitting pretty incredible scale, we also have a pretty young email programme and our domain knowledge is pretty inexperienced.

The art of sending emails 

Gioe and the team at Twilio SendGrid now think of email campaigns at Spotify in two ways, as has already been noted: art and science. On the science side, tools are built to empower scale, reliability, and to send emails quickly. However, it's the art side of email marketing that Spotify has had to learn. Gioe said: 

But what we had to develop a growing appreciation for was that we had to, in certain circumstances, actually send fewer emails, because every email that you send is potentially a risk to your programme. And the act of doing that is an art, you have to understand the domain intimately. You need to insert intelligent guardrails that monitor this risk and you need to set up a system that minimizes this risk. And a successful email programme at the end of the day is a function of both. 

And if it hadn't been for Twilio, Spotify could have found itself in a much worse situation. Gioe said: 

We couldn't have done this without Twilio SendGrid. The way that I like to think about this is that Twilio SendGrid made the hard stuff easy and the really hard stuff easier. The science, the rhard stuff, hitting scale, It takes good engineers, it takes a great architectural plan, but it is difficult, it's non trivial. On the other hand, the art of email sending is difficult and Twilio SendGrid really made it easier for us to get that done. 

They took the hard issue of scale off the plate, we never really had to think about it. But when we hit these issues that were more domain specific, when we really hit these issues that had to do with the art of email sending, they were also there to teach us. They were there to humble us. And without them I don't think that we would have saved our email programme and we wouldn't have such a successful programme going forward. 

So when you think about your own email programme, I encourage you to think about both sides. I find it to be a bit of a healthy and sane framework to think about these two sides of email sending, in terms of science and in terms of art.