Apr 1, 2014
There is a lot of literature about how to build great apps–what works, what doesn’t. But something I found while researching email marketing subject lines is that a lot of research is contradictory. Mailchimp’s data told me that subject lines with the word “newsletter” generated more opens on average for emails. Adestra’s report said the exact opposite. Both were calculated over hundreds of thousands, if not millions, of emails. I’m sure both are right based on the (possibly self-selected) samples that use those respective services. But what will work for my specific audience? I didn’t know until I tested it out for myself.
This led me to think about all the mobile A/B testing variants that improve one app but completely fail for another. The moral of the story is to read as much as you can, but do not be afraid to defy conventional wisdom. Look to those who came before you for ideas but always test theories for yourself.
Most of the literature about the differences between iOS and Android will tell you that users spend more on iOS (I even wrote about it in this blog post). But the team at Robot Invader, creators of the mobile games, Wind-up Knight and Rise of the Blobs, have thoroughly tested pricing and revenues for in-app purchases in their games and have found no difference in conversion rates for purchase or the amount spent per person between the iOS and Android versions of their games.
Their new game, Wind-up Knight 2, takes this learning into account, so the pricing strategy between the platform versions are identical. This is an amazing insight because now Robot Invader can move on to testing more unknowns. And since testing on Android is easier than testing on iPhone, they can A/B test a lot on Android and port many of those conclusions to iPhone to simplify testing on iOS.
The idea that iOS users spend more than Android user wasn’t the only convention that they found to be untrue for their games. They also found that:
You can find more details on the myths they’ve busted here.
Mobile A/B testers at KAYAK were told that having a message assuring customers of the safety of submitting credit card information at checkout would only remind customers of the insecurities of mobile commerce and therefore decrease conversions to purchase. KAYAK tested this and found that reassuring customers actually helped improve sales.
Read more about mobile A/B testing at KAYAK.
We hear a lot about this particular topic going both ways. For some apps, a message related to checkout security helps customers feel at ease. For other apps, it reminds customers that giving your credit card info can be dangerous, which hurts the app’s purchase completion rates. Which types of customers does your app have? You won’t know until you test it.
Executives face an ongoing challenge of creating amazing customer experiences while also keeping costs at bay. This tug of war between satisfying customers and reducing costs takes center stage in stores, branches, and call centers. In particular, organizations have typically...Read More
1. Is your conversion rate good? 2. Are your engagement and retention rates good? 3. Are the new features and design changes you make actually improving your top goals? 4. How can you grow your user base? 5. What actually...Read More
We hear from customers that planning out your second, third, and fourth A/B tests is one of the hardest things to. Many app managers have a first test in mind when they start experimenting and planning out a series of...Read More