The Registration Test Results Netflix Never Expected

This post is adapted from Anna Blaylock’s talk at Manifesto SF.

“Come on Netflix. Why can’t I browse titles before I sign up?”

It’s a sentiment that many Netflix visitors have expressed at one point or another. And by many, I mean 46% of visitors surveyed. According to a their data, 46% of survey respondents said that making titles of Netflix content available to browse was the 1 thing they wanted to know more about before signing up.

It makes total sense to let potential customers take a peek right? If you share the movie and show titles they’d be getting from Netflix, they’re probably going to complete signups at a much higher rate.

We know that demonstrating value early in the onboarding process is a fundamental principle in the onboarding process. So how is it that one of the most data driven companies in Silicon Valley doesn’t do this?

Anna Blaylock, a Product Designer at Netflix wondered this too when she first started at the company. She knew that visitors were pining for this, so she posed the question to the team. “Why don’t we allow users to view titles before sign up?”

Blaylock wasn’t the only team member with this question in mind. While she wasn’t the first to have the idea, she was the catalyst that sparked the series of A/B tests that followed.

The team was really confident that this change would help improve signups. The only question on their minds was, “How do we best implement it?” They designed a clever series of back-to-back tests that would pit variants against each other to determine the optimal solution.

The test would start out with the control (not allowing users to browse titles) vs. Variant 1 (where users could scroll different genres, click on a title, get a synopsis, and see all the titles). The winner of the first test would become the new control and go on to face Variant 2. The process would repeat with variant 3 and the rest of the possibilities until they determined the best onboarding flow. This would allow Netflix to learn quickly, and determine the optimal solution out of the 5 very quickly.

As they ran the experiment, they found that the original page (without any title searching abilities) beat variant 1.

Then it went on to beat variant 2.

And then variant 3.

And 4.

And 5.

The team was shocked by the results. The survey data and best practices learned had suggested the exact opposite. Everyone had thought that allowing visitors to preview titles would increase signups. So they dug deep to figure out the reasons why the original performed best. Here’s what they learned.

“Don’t confuse the meal with the menu.”

“Netflix is all about the experience,” says Blaylock. Just as a restaurant dining experience isn’t solely about the food on the menu, Netflix’s experience isn’t just about the titles. There’s something magical about being able to just visit the site, have your favorite shows or personalized suggestions pop up, click once with your mouse, and start watching. This incredibly smooth experience isn’t something that a web page can just describe to users, it’s something that has to be shown.

Simplify Choices

Each of the variants built for the tests increased the number of choices and possible paths for Netflix visitors to follow. The original only had one choice: Start Your Free Month. Blaylock says that the low barrier to entry outweighs the need for users to see all their content before signing up.

Users Don’t Always Know What They Want

While survey had shown that 46% of users wanted to browse titles before signing up for Netflix, the tests proved otherwise. Testing reveals our assumptions. That’s why it’s so important to run tests and trust the data. If Netflix had simply followed best practices, they would have been scrambling trying to figure out why their new onboarding was performing so poorly.

A/B testing allowed the team to quickly invalidate their hypotheses, without disrupting the user experience, or significantly affecting their key metrics.

Blaylock left us with one great final quote: “Your assumptions are your windows on the world. Scrub them off every once in awhile, or the light won’t come in.” – Isaac Asimov