Contradictions in Mobile App Design Research

There is a lot of literature about how to build great apps–what works, what doesn’t. But something I found while researching email marketing subject lines is that a lot of research is contradictory. Mailchimp’s data told me that subject lines with the word “newsletter” generated more opens on average for emails. Adestra’s report said the exact opposite. Both were calculated over hundreds of thousands, if not millions, of emails. I’m sure both are right based on the (possibly self-selected) samples that use those respective services. But what will work for my specific audience? I didn’t know until I tested it out for myself.

This led me to think about all the mobile A/B testing variants that improve one app but completely fail for another. The moral of the story is to read as much as you can, but do not be afraid to defy conventional wisdom. Look to those who came before you for ideas but always test theories for yourself.

How Robot Invader did this

Most of the literature about the differences between iOS and Android will tell you that users spend more on iOS (I even wrote about it in this blog post). But the team at Robot Invader, creators of the mobile games, Wind-up Knight and Rise of the Blobs, have thoroughly tested pricing and revenues for in-app purchases in their games and have found no difference in conversion rates for purchase or the amount spent per person between the iOS and Android versions of their games.

Their new game, Wind-up Knight 2, takes this learning into account, so the pricing strategy between the platform versions are identical. This is an amazing insight because now Robot Invader can move on to testing more unknowns. And since testing on Android is easier than testing on iPhone, they can A/B test a lot on Android and port many of those conclusions to iPhone to simplify testing on iOS.

The idea that iOS users spend more than Android user wasn’t the only convention that they found to be untrue for their games. They also found that:

• iOS isn’t necessarily more profitable than Android
• Localizing your app for as many languages as possible was about as profitable as only creating Japanese and English versions
• iPad engagement was actually slightly lower than iPhone engagement

You can find more details on the myths they’ve busted here.

How KAYAK did this

Mobile A/B testers at KAYAK were told that having a message assuring customers of the safety of submitting credit card information at checkout would only remind customers of the insecurities of mobile commerce and therefore decrease conversions to purchase. KAYAK tested this and found that reassuring customers actually helped improve sales.

Read more about mobile A/B testing at KAYAK.

We hear a lot about this particular topic going both ways. For some apps, a message related to checkout security helps customers feel at ease. For other apps, it reminds customers that giving your credit card info can be dangerous, which hurts the app’s purchase completion rates. Which types of customers does your app have? You won’t know until you test it.