Mar 21, 2014
How do you know which mobile A/B testing solution is right for you? Ask for a demo. Signing up for a trial is a great start. Seeing a live demo is even better. With a live demo, you can watch how the solution actually works with your app. You can also learn about features that you may not discover right away or that may be inaccessible with the free trial. In this post, we’ve provided a list of seven topics to consider and discuss while evaluating the demo.
Is your app on iOS, Android or both? Do you need to A/B test on both? It’s important that your A/B testing solution works on the OS that you need. Most mobile A/B testing platforms offer different features for Android and iOS so make sure that the demo doesn’t just show you the more developed OS and imply that all features are available on both.
Will non-technical peers want to create their own A/B tests? A visual editor empowers product managers, designers, and producers to develop tests on their own. In turn, developer resources can focus on implementing more complex tests and updates.
Tip: Find out what the visual editor allows you to edit. Many A/B testing frameworks tout a visual editor, but they’re not all created equal. Most visual editors can handle copy and colors. But what about images, layout, and visibility adjustments?
Not all A/B testing frameworks offer analytics to calculate the performance of your app variants. This leaves you to manually calculate statistical significance to see if you’ve reached a conclusion on your tests. You may have to spend thousands of dollars on an analytics framework and then invest more in integrating the two. Ideally, your A/B testing service should already have analytics baked in.
However, if you already have an analytics framework in place in your app, definitely make sure that the new A/B testing platform can integrate easily with your existing systems (and have the capability to integrate for you).
Was your app built by a team? If so, your app testing platform should also support collaboration. Team-oriented platforms make test development easier and support an experimental corporate culture. When everyone understands how testing works, everyone generates optimization ideas and is working to improve the app in a data-driven way.
The most obvious of the requirements is also the hardest to check. The testing SDK should integrate seamlessly with your app so users don’t perceive any differences in their experience with the app. You know that the app needs to be secure with your and your users’ data. You know that the A/B testing platform should give reliable data about which variant is performing better. But does your A/B testing solution actually provide this or just claims they do? There are a few key questions that provide insight on how developers are approaching these issues. If you ask these questions, the engineers should have answers. If the engineers sound confused, then they probably haven’t thought about the problem, and the reliability of the whole system may be questionable. Ask them:
Since mobile apps are complex, testing is always going to be more straight forward for websites, emails or social media. So is the A/B testing service going to be responsive and helpful when something is confusing or doesn’t work as expected? Are the engineers experts in the field? Can they solve your problem(s) quickly?
In addition to the core functionality above, there are some auxiliary features that you may need depending on internal circumstances:
So ask for a demo. Play around with the free trial. Push the platform’s limits. See how the SDK really works. And keep in mind that A/B testing mobile apps is complicated. Mobile apps are not like websites, emails, or social copy. Every app is constructed differently, so the best way to know if a platform will work for you is to see it in action.
We hear from customers that planning out your second, third, and fourth A/B tests is one of the hardest things to. Many app managers have a first test in mind when they start experimenting and planning out a series of...Read More
I talk to a lot of app developers and product managers on a regular basis about A/B testing their mobile app. The feedback I hear the most is: “I know I need to start A/B testing, but we just haven’t...Read More
There is a lot of literature about how to build great apps–what works, what doesn’t. But something I found while researching email marketing subject lines is that a lot of research is contradictory. Mailchimp’s data told me that subject lines...Read More