5 tips for A/B testing on Android versus iOS

If you have an Android and iOS app, your two versions are probably similar but not exactly the same. Your users are probably similar but not exactly the same. So make sure you don’t test on one platform and assume the results are the same for both. Here are some ideas of things to test that have yielded different results for some of our customers.

1. Try a different revenue strategy

   Despite Android’s now dominant market share in device sales/activations/users, iOS still accounts for a larger portion of dollars spent while on the device. iOS ad prices being higher than those of Android are yet another indication of this.

   What to test: Think about testing multiple revenue strategies on both platforms. You might get different results. For example, if you’re an ecommerce app, try featuring items of different price tiers on your home page. Variant A might be featuring all items with a low price point, variant B at middle price point, and variant C at a premium.

2. Test different engagement strategies

   Studies also show that iOS users tend to spend more time in-app than Android users. This is especially true of iPad of users. I’m sure there’s some correlation here with the amount of money spent that I mentioned above.

   What to test: Try different user flows that require different amounts of engagement. For example, if you’re testing a game with in-app purchases, try showing the first purchasable item to the user at different points in the game experience.

3. Account for different out of app experiences

   Nuances in the way an OS does things is very important to the overall user experience. The fact that Android has 3 to 4 buttons (back, home, menu, and sometimes search) on the the phone versus iOS’s singular home button makes the navigation of any given app very different. The way users interact with notifications on iOS versus Android is also very different. iOS tends to take users from a notification into the app while Android allows the user to do many more tasks without launching the app. There also just many more combinations of app client/OS version/hardware on Android than on iOS.

   What to test: Experiment with different user flows. A user might hit the back button to go back to a previous page on Android, but iOS users don’t have that option. You might want to test providing a back button to iOS users. Alternatively, going back may be an action you don’t want users to take, so you can try to test variants to reduce the click of a back button on Android.

4. See if differences in performance matter

   There are a handful of different iOS devices, but there are many, many more Android device with varying quality of hardware and versions of software. Something that works fast enough on iOS may not work fast enough on Android. You can find what crashes your app with some pre-release testing, but you can’t really effectively evaluate the trade-off of performance versus feature without A/B testing.

   What to test: A/B test your features before doing a full launch. You might have a feature that causes your app to have a slight delay in the user experience. It might only cause a 0.2 sec. delay in iOS which users may tolerate given the benefit of the feature, but it might cause anywhere from a 0.2 to 2 second delay on Android devices which may turn out to be a deal breaker for users.

5. Test more on Android because it’s easier

   With our Visual Apptimizer, we’ve eliminated the need to resubmit your iOS app into the Apple app store every time you want to make a design, copy, or layout change to your app, but any test involving a change to your app’s binary will always require new approval. Getting a new version of your app approved by Google is easier and takes less time than doing so for Apple.

   What to test: When it comes to large flow redesigns that necessitate code redevelopment, try different things on Android first because the Google Play store approval is easier to get through. Gain learnings there, and then refine your theories on the two platforms separately.