Yahoo Growth: How We Reached #5 on the App Store

Getting Yahoo Mail into the top 5 free apps in the App Store is no easy feat, so it’s pretty safe to say the growth team at Yahoo knows what they’re talking about. Over the course of 10 weeks, they ran 122 tests that increased conversions by 13x, a huge factor in their amazing success.

When we had the chance to host Yahoo Growth’s s Kevin Li and Sergei Sorokin at our Fast Iteration event, we were ecstatic. There, they shared key insights for mobile growth teams about the importance of testing and how incremental changes compound massive wins equivalent to even “big swing” projects. You know, the kinds of things you want to share with your boss.

Copying “Best Practices” Is Bad Practice

“best practice audits” where you collect how other products do things like on-boarding, checkout, etc. are hazardous to your design [1/3]

— Luke Wroblewski (@lukew) August 5, 2015

rather than coming to conclusions through iteration use & testing you assume others did & got to a best practice. usually they didn’t [2/3]

— Luke Wroblewski (@lukew) August 5, 2015

and once you see solutions, it’s very hard to un-see them so they end up strongly influencing your design. [3/3]

— Luke Wroblewski (@lukew) August 5, 2015

Just as Luke W explained, copying others isn’t a technique that’ll help you build amazing products that build presence and engage/retain users. The problem is that users’ desires, needs, and expectations will all vary from app to app. You end up assuming that something will work and, “the one thing you should not do is assume you know something,” shares Sergei.

That’s why testing is so important. It allows you to generate deeper customer insights through analytics. The better you understand your customers, the better you can serve their needs and desires.

In Retrospect, Effective Changes Seem Obvious

“The things that ended up working were the things that in retrospect were like ‘oh yeah well that’s obvious. If I were the person receiving this I would totally respond better.’ But when you’re in the mode and when you’re coming at it from the other side it’s really hard to think about.” said Sergei.

That’s why their testing methodology was so important to the app’s success. “It’s not just good enough for somebody to come up and say ‘I have this perfect idea. it has to work.’ Well, guess what? We tested 122 of them and most of them didn’t.” Rather than hoping the perfect idea would come along, they used a structured methodology experimenting with incremental iterations.

The truth is, while you may have a brilliant idea or a “best practice” that you want to implement, you really don’t know how users will react to it. Testing it reveals exactly how users behave in accordance to your change so that you can A) learn quickly about the effects of your change on user behavior and B) demonstrate and learn from it using deep analytics.

Testing Isn’t a Magic Pill

“You do one small tweak and in one week you have a massive improvement and you’re like, ‘Oh my God I am a genius. Like, I am a growth hacker expert.’ And then you release and go into week two, week three, week four, and your week on week improvement is like 3 percent, 4 percent…” quipped Li.

Some days you’ll have big wins, but testing won’t necessarily provide tremendous results week after week. Some days, the process is going to be discouraging, and that’s okay. Numerous tests and incremental gains are just a part of it. It’s important to remember that each test gives you deeper customer insights through the analytics. Even though some tests and incremental gains may feel like failures, they’re actually giving you incredibly valuable feedback as to what doesn’t work with your customers and why.

Stick to Your Guns

Just because you’re not making big wins week after week doesn’t mean you should give up or change strategies, even if there’s pressure from the top. Li shares that when he got pulled into a product review one day, he came out incredibly frustrated. “Well, week 3 and week 4 you have like very little improvement,” said Li, quoting his manager. “Your end goal is seemingly small. I don’t think you’re being ambitious enough. Why don’t you 4x that goal and come back.”

Meetings like this can derail the entire process because of the immense pressures to perform. Thankfully, the team shook it off and continued to tap into their wealth of experience. They kept trudging through the process and a week later, saw a massive spike that resulted from their compounding efforts.

Incremental Changes Compound

“If there’s one takeaway it’s just that it’s okay to do small wins. Small wins are good, they will compound. If you’re doing it right the end result will be massive,” says Li. When you’re constantly testing and improving your product, it’s easy to get sucked up in the incremental results and forget how tremendous the power of compounding can be.

Those 3%, 4% gains that the growth team made paid off. That’s because like financial gains, each of the small increases from tests compound together to produce staggering increases. Combined together with the results of all the tests, they increased conversions by over 1000%.

So don’t forget to celebrate the small wins as well. “We celebrated every 3 percent, every 4 percent, and every 5 percent improvement,” shared Sergei. And you should too. Each increase is an improvement. While it may not be that 15% or 30% gain you’re hoping for, don’t forget that these changes combined are what produce exponential growth. The culmination of their efforts was a huge contributor to the success of the Yahoo Mail app, and teams would be wise to follow suit.

Wrapping It Up

Li and Sorokin were able to compound their effects for conversions because they were methodical and relentless when implementing their proven methodology. Throughout their 10 weeks of testing, they gained invaluable insights into their user base and learned a lot about their own product. Testing is vital to the growth process, so make sure you know how to demonstrate that to others as well.

Want to attend our next event? Make sure you sign up for our newsletter at the bottom of this page!