What we learned from web A/B testing… and what we didn’t

These days, it’s common to have a both a web and mobile experience that are core to your business. One probably reels in more revenue than the other, but if you’ve developed an experimental culture, there’s a good chance that you’re a/b testing and optimizing both. And you typically test more on web because it’s easier (no iTunes regulation, don’t need to wait for users to update, etc.). But you can’t just test on web and assume it applies to mobile. No, that would be too easy. Here are some DOs and DON’Ts for leveraging web a/b testing into your mobile testing strategy.

DO build a separate implementation strategy for mobile optimization. Mobile technology is very different from web technology in two important ways:

1. Websites can typically handle a lot of client side logic via JavaScript. This is not typically the case with native mobile apps. A lot of client side logic not only slows the user experience, but is more likely to cause bugs.
2. When bugs do happen, they’re much easier to fix on a website because shipping a new version of the site is easy. Users are able to see the new version instantly — you don’t need to wait for any third party approvals. Not so with mobile. Not only will you need for a new version of the app to be approved by the Android or iOS stores, you also need to wait for users to update.

DON’T assume what works on web works on mobile. While it’s likely that your users are coming to both, there are many nuanced differences between the web user and the mobile user:

1. There may be segments of your user base that prefer one platform over the other, so you may have different users on web than on mobile.
2. The context for phone (on the go?), tablet (second screen?), and web (at a desk?) are very different from each other, so users might be looking for different things when using each type of device.
3. Each platform has its own idiosyncrasies that affect your user flow. For example, your user can interact with your app via notifications on mobile. This does not happen on web.

DO leverage learnings from web to create hypotheses for mobile. Since it is much easier to test on web, test on web more. Maybe you’ll find that out of 10 different images you tested on web, one type of image wins, two others are close, and seven perform terribly. Take the top three and retest them on mobile. The winner might be different, but you’ve reduced the number of versions you need to create for mobile.

DON’T dissociate mobile and web so much that users get conflicting messages. There’s a chance that someone seeing variant A on mobile might be seeing variant B on web. Make sure that you plan tests so that these version don’t conflict. There are some big changes that users might not notice but small changes that users might. For example, if you’re testing out a different check out flow on mobile than web, users mostly expect that web and mobile check out experiences are different. However, if you’re testing item pricing, customer might notice that a price they see on web is different from the same item on mobile.

DO utilize a mobile-first solution like Apptimize. We were born on mobile and know mobile A/B testing technology and philosophy better than anyone. Our framework not only lets you A/B test but also has other features that are vital to mobile product managers and developers such as feature flagging/toggling and slow rollout. Try it today or email us at contact@apptimize.com to learn more about enterprise solutions.