How to Evaluate a Mobile A/B Testing Solution

How do you know which mobile A/B testing solution is right for you? Ask for a demo. Signing up for a trial is a great start. Seeing a live demo is even better. With a live demo, you can watch how the solution actually works with your app. You can also learn about features that you may not discover right away or that may be inaccessible with the free trial. In this post, we’ve provided a list of seven topics to consider and discuss while evaluating the demo.

1. Platform Compatibility

   Is your app on iOS, Android or both? Do you need to A/B test on both? It’s important that your A/B testing solution works on the OS that you need. Most mobile A/B testing platforms offer different features for Android and iOS so make sure that the demo doesn’t just show you the more developed OS and imply that all features are available on both.

2. Visual Editor

   Will non-technical peers want to create their own A/B tests? A visual editor empowers product managers, designers, and producers to develop tests on their own. In turn, developer resources can focus on implementing more complex tests and updates.

   Tip: Find out what the visual editor allows you to edit. Many A/B testing frameworks tout a visual editor, but they’re not all created equal. Most visual editors can handle copy and colors. But what about images, layout, and visibility adjustments?

3. Analytics

   Not all A/B testing frameworks offer analytics to calculate the performance of your app variants. This leaves you to manually calculate statistical significance to see if you’ve reached a conclusion on your tests. You may have to spend thousands of dollars on an analytics framework and then invest more in integrating the two. Ideally, your A/B testing service should already have analytics baked in.

   However, if you already have an analytics framework in place in your app, definitely make sure that the new A/B testing platform can integrate easily with your existing systems (and have the capability to integrate for you).

4. Collaboration

   Was your app built by a team? If so, your app testing platform should also support collaboration. Team-oriented platforms make test development easier and support an experimental corporate culture. When everyone understands how testing works, everyone generates optimization ideas and is working to improve the app in a data-driven way.

5. Reliability, Security, and Trust

   The most obvious of the requirements is also the hardest to check. The testing SDK should integrate seamlessly with your app so users don’t perceive any differences in their experience with the app. You know that the app needs to be secure with your and your users’ data. You know that the A/B testing platform should give reliable data about which variant is performing better. But does your A/B testing solution actually provide this or just claims they do? There are a few key questions that provide insight on how developers are approaching these issues. If you ask these questions, the engineers should have answers. If the engineers sound confused, then they probably haven’t thought about the problem, and the reliability of the whole system may be questionable. Ask them:

   • What happens when an app user is offline? Do they still receive the A/B test? What if their user data can’t be transmitted back to your servers?
   • How do you ensure data accuracy? (Listen for responses related to server redundancy and consideration of fringe cases.)
   • How do you handle the first run experience? (Some A/B testing frameworks cause a lag in the user experience while the app is downloading the new app variant. Make sure the A/B testing framework is optimizing for a smooth user experience.)

6. Customer service

   Since mobile apps are complex, testing is always going to be more straight forward for websites, emails or social media. So is the A/B testing service going to be responsive and helpful when something is confusing or doesn’t work as expected? Are the engineers experts in the field? Can they solve your problem(s) quickly?

7. Advanced Features

   In addition to the core functionality above, there are some auxiliary features that you may need depending on internal circumstances:

   • If you have fewer than 10,000 monthly active users, it might be difficult to run more than one or two tests at a time. If you’re able to cross allocate users (when a user can be part of two tests at once), you could get more data out of each user. For example, if a user is in a test for the start of the app, they could potentially be a great subject for a test in the latter part of the app without the results of the two tests contaminating each other. Do you want an A/B testing solution that enables you to get more data from fewer users?
   • Targeting: Can you send tests to specific users based on type? For example, can you exclude geographies with high piracy rates when testing something related to purchase? Can you test a different flow on returning users versus new users? Can you exclude certain OS’s or devices from tests because there’s a known bug in some hardware/software versions?
   • Segmentation and results filtering: Can you slice results data along user groups? For example, if a certain app variant is doing poorly and you suspect it’s because that variant is crashing on iOS 6.1, can you see the data from all users except those on iOS 6.1? Can you look at the results from men versus women?

So ask for a demo. Play around with the free trial. Push the platform’s limits. See how the SDK really works. And keep in mind that A/B testing mobile apps is complicated. Mobile apps are not like websites, emails, or social copy. Every app is constructed differently, so the best way to know if a platform will work for you is to see it in action.