How VSCO & Twitter Deploy New Features to Millions of Mobile Users

How do you update an app without breaking the user experience when you have over 20 million users? 200 million?

When it comes to mobile apps, the stakes are higher than they ever were on web. Mobile users have an incredibly low tolerance for buggy apps, with only 16% willing to try a failing app more than twice.

To make matters worse, the app stores make responding to any problems a difficult hurdle, requiring additional time-costly app reviews that can take days, if not a whole week. Even their “expedited reviews” are estimated to take 1-2 days, far from fast enough if you’re quickly bleeding users.

When you only have a few thousand users, these hurdles are annoying. When your app has hundreds of thousands or even millions of MAUs, they’re critical.

So I asked VSCO (and formerly Twitter) Engineer Chris Burnett how his teams deploy mobile app updates to millions of users without breaking the user experience.

Decider by Twitter & VSCO

“Once you start shipping things like this, you never want to go back,” says Burnett.

While 99.999% of apps release updates to their entire userbase in one fell swoop, many leading apps have already matured their release processes far beyond that.

Both Twitter and VSCO use a release management mechanism called Decider (better known as feature flags). Feature flags allow mobile teams to ship mobile app updates with confidence, knowing that they can respond immediately to any unforeseen bugs, crashes, or other any problems.

So what exactly are they?

Feature Flags are a technique that allows teams to enable or disable features at any time, without having to deploy an update. Instead, they can deploy a feature or update in an off state, then flip a switch to turn it on to any subset of users, immediately – no need to wait on app store approvals.

Burnett was first introduced to feature flags for mobile while working at Twitter. It was a critical piece of framework that he knew would revolutionize the way VSCO’s teams approached mobile. “Nothing ships if it’s not behind decider,” he says about his former company. VSCO has adopted a similar mindset.

“It changes everything.”

“It changes everything,” says Vance Pham, an iOS engineer at VSCO. With it they’re able to:

Take more risks, knowing that they can respond quickly to unforeseen issues
Instantly control staged rollouts and rollbacks on a granular level
Segment and test on specific users

Take More Risks on Mobile

Anytime you ship code, you’re taking a risk. “The highest correlation to failure in a system like ours, not just Twitter but any big system, is software development error,” says Raffi Krikorian, the VP of Platform Engineering at Twitter.

That’s not to say mobile engineers are doing a bad job though.

“You can be confident in your code…but what code does in prod can be totally different than what you expected,” says Burnett. That’s why a well-thought-out deployment system is so critical. The worst thing you can do to your mobile users is ruin the user experience.

With the current system of pushing one button to release to all users, something is almost guaranteed to break, leaving your users frustrated and your team scrambling to fix the issues.

With Decider, teams can easily fix any user issues. At VSCO “anything that’s risky, they all go behind deciders,” says Burnett.

Using Decider to turn off new updates is incredibly easy. One flip of a switch and end users’ apps will instantly roll back to their previous state. It’s a speedy fix that prioritizes the user experience above all else, but it also allows them to take more risks that previously may have been too precarious.

Instant, Granular Control of Staged Rollouts (and Rollbacks)

Feature flagging also allows you to perform staged rollouts with granular control. You can choose which segments and percentage of users can see an update, then increase or decrease the number of affected users at any time.

If there are every any problems, you can instantly roll back a change and fix the problem for any affected users. Unlike the old methods, you’re suddenly able to ensure that you’re responding timely to any user issues, and providing the best UX possible.

Segment Users and Test

The segmentation aspect of feature flags also really shines when it comes to analytics and testing. At VSCO, the team uses an employee segment to internally test new updates on the production apps.

Additionally, Burnett says that the team uses Decider to track how new features affect user behavior. They tie these new features to a metric, then bucket users in test groups to figure out if those metrics increase. A basic form of A/B testing.

This methodology allows VSCO to measure the effectiveness of new features, and enables them to iterate on a minimum viable feature using data from actual users, a much better method than building in a vacuum and hoping it connects.

In addition to changing the way they deploy new features, feature flags also allow them to start segmenting their users. Suddenly, they can start thinking about how to configure their app for users by geography or even user behavior.

This Is Where Mobile Is Headed

While there’s little we can do to change the outdated app store review processes, feature flagging gives the control back to mobile teams, enabling them to iterate quickly, take more risks, maintain the user experience, and deploy with confidence. Many of the top mobile companies such as Facebook, Uber, Lyft, Dropbox, and others are already using these techniques to provide their users with a better experience. Will your team?

Find out More about Feature Flags here.