Storytelling with data: Lessons learned from Bump

Bump used A/B testing before it was cool. Google acquired Bump last year, in part because the people at Bump really understand data, mobile, and user experience. Earlier this week, we spoke with Chris Perry, former product manager and lead data scientist at Bump. Chris took a break from his Google projects to talk with us about:

1. How data doesn’t win arguments. It’s the story around the data that wins.
2. The ups and downs of Bump and Flock.

Connect with Chris

Apptimize: Many small companies brush off A/B testing because they don’t think they have the time/resources for it. How did you develop an experimentation culture at Bump?

Chris: The key is telling a good story with the data. As a data scientist, I used to think that data always won. But in reality, a lot of people ignore data and go with their gut. To be helpful, data need context and awareness. People will feel much more comfortable with data-driven decisions if the data tell a story they can understand and buy into.

Data is also easily misinterpreted. We found that if you give users a button, they will click it. The data will show you that people clicked the button, but that doesn’t necessarily mean the version with the button is a good one. You really need to understand the story you’re trying to tell and plan tests with the right metrics in mind.

A: What helped Bump and Flock grow? What hurt adoption?

C: Getting into the iPhone commercial was definitely a huge boost to downloads. That created your hockey stick adoption graph, but growth remained consistently solid after that. We had some acceleration when we launched photo sharing and messaging features for Bump, but aside from those times, we mostly focused on engagement.

Flock never grew in the way Bump did. The idea was that we’d take the best of photo sharing from Bump and create an entire app around it since it was such a popular feature on Bump. But the core premise of Flock made the growth model more challenging. We created this app that required as little interaction from the user as possible. When you downloaded Flock, it told you it would work in the background and alert you when you needed to do something. The fundamental problem with that was that people wanted to experience the app when they first downloaded it.

A: Did you learn anything surprising from A/B testing?

C: It took a while to convince people that you need to have a control group when making a test. It’s more intuitive to just allot 5% of your users to a variant and compare the results to the 95% that are using your app as usual. But there are a lot of unpredictable things that may happen just because there’s a switch turning on somewhere. You need to turn that switch to the baseline version of your app for another 5% to have a control group. This definitely may not be intuitive even if you are an amazing engineer in your field.

Being data-driven isn’t rocket science per se, but you need to have someone who knows when to use a t-test versus a z-test and push others to think in a statistical way. I know Apptimize does a lot of this for the user, but we started A/B testing before solutions like Apptimize existed so we had to do all this ourselves.