How to Try Out Experimental Netflix Features

Netflix is obsessed with testing; the company tests every new feature before rolling it out widely. Here's how it works and how you can opt in.

There are plenty of reasons why Netflix is one of, if not the, best streaming services around. It offers a wealth of inventive programming, has an easy-to-use interface, and rarely suffers from technical hitches. But how does Netflix get it so right?

The answer is thorough testing. Every single change to Netflix, from small app tweaks to major algorithm changes, goes through multiple rounds of detailed testing. We're going to explore how and why the company does this, and how you can opt in to test Netflix features.

How to Opt In to Netflix Tests

If you want to ensure you're always experiencing the cutting-edge version of Netflix, you need to opt in to receive tests and previews. By default, everyone is included in this, but here's how to check that you didn't opt out previously:

1. From the Netflix website, hover over your profile picture in the top-right.
2. Click Account.
3. Next to Settings, click Test participation.
4. Turn Include me in tests and previews on and click Done.

Netflix notes that "the setting does not apply to tests regarding security, anti-fraud or enforcement of the Netflix Terms of Use". Essentially, you can't opt in or out of changes made to these elements.

It's important to note that while opting out of this test participation program excludes you from seeing new trial features, it doesn't mean that you won't be exposed to A/B testing of existing features—more on that later.

Why Does Netflix Experiment and How Does It Work?

Netflix tests everything: registration, playback, recommendations, search... you name it, it's tested. The reason for this is that Netflix wants to check its assumptions. It doesn't matter how hard a Netflix employee pushes for an idea—data informs decisions, not opinion. Otherwise, a change could fail drastically because it was based on a hunch.

The company does this through A/B testing. At its core, A/B testing is a way of comparing two versions of something to see which is the most successful. The measure of success isn't always the same; it depends on the change. The experiment might aim to see which version gets more people to register, or to discover new Netflix shows, or to stay watching for as long as possible.

Netflix isn't just testing two versions, though. It has such an expansive user base, not only in terms of raw numbers, but also in demographics and behavior. It means that Netflix can separate its users into many experimental groups, trialing ever-so-slightly tweaked versions to see which is truly optimal.

As detailed on the Netflix Technology Blog, Netflix assigns its users to these groups through batch and real-time allocation. Batch assigns a fixed set of members to an experiment who are known to meet the trial's criteria. Real-time is more flexible, shuffling users in and out of tests as they interact more with the platform.

At any one time, you could be part of many tests. Netflix has an internal tool which lets a department track what experiments other teams are performing, to ensure a user isn't assigned to multiple conflicting tests (e.g., two tests which change the placement of the same button). These tests can run for days, weeks, or months.

In another post on the Netflix Technology Blog, the company admits that it tests hundreds of changes every year, but only a small percentage of these roll out globally. The reason for this is that the A/B tests show that most of the experiments don't improve the platform.

But Netflix doesn't see these as failures. As the blog reads:

When our members vote down new product experiences with their actions, we still learn a lot about their preferences, what works (and does not work!) for different member cohorts, and where there may, or may not be, opportunities for innovation.

This leads to new hypotheses and tests; some of these may be successful. Plus, as the member base grows, and its expectations and preferences change, Netflix revisits old experiments to see if they now provide value.

Ultimately, while the overall experience is the same, this means that the Netflix you use isn't necessarily identical to your friends or family. Your homepage may feature different rows, the playback buttons may look different, you could unknowingly have a different recommendation algorithm, and so on.

What Features Has Netflix Tested?

Netflix is always testing changes and new features. Some of these never make the light of day, while others are now beloved components of the platform. Examples of past Netflix experiments include:

• Changing the size of the tiles on the homepage to see which encouraged users to watch.
• Changing the tone of the call-to-action button on the homepage to see which encouraged users to click through to sign up.
• Introducing collections of content curated by Netflix's creatives, rather than an algorithm, to see if it encouraged users to watch something outside their usual recommendations.
• Charging subscribers who share their Netflix account outside their household, to see if revenue grew through existing customers.
• Introducing a button that displays something psuedo-randomly for someone to watch (based on their watch history and recommendations) to see if it increased watch time and decreased time spent scrolling. This later became the Netflix "Play Something" button.

As mentioned earlier, you have the opportunity of receiving some of these larger tests (new features like the shuffle button) by opting in to test participation via your Netflix account. However, you'll always receive A/B tests for smaller adjustments, no matter your setting.

As an example, you may have noticed that the image tile used to represent a movie or show on your homepage changes periodically, or looks different from your friend's—this is because Netflix displays an image it thinks you're more likely to find appealing.

Tests aren't always user-facing, though. For example, Netflix is constantly tweaking the technology it uses to power its streaming, to ensure the perfect balance between quality and consistency. There's nothing worse than a slow streaming service.

Netflix Is Continuingly Improving

Netflix is an evolving platform, changing constantly to keep existing subscribers happy and bring new customers in. That doesn't mean Netflix is perfect—far from it, since there are many helpful features missing from Netflix—but that your behavior should eventually lead to improvements.