State of Mobile Experimentation: What Product Managers A/B Test and Why

Product teams that build sticky products in today’s proliferating technology landscape have to embrace a culture of experimentation and A/B testing to know what users want. Empowering innovation among product teams is at the heart of what we do –that’s why we conducted research to obtain insight into what product managers are testing and why.

For our first Mobile Experimentation Report, we surveyed product managers and their teams to learn the primary objectives fueling their experiments and optimization efforts.

Here is what we found:

1) Everyone focuses on monetization but executives think about retention and engagement much more than product managers do

When it comes to their Key Performance Indicators (KPIs), respondents with director-level to C-suite positions were more focused on generating retention/engagement than those who held individual contributing roles.

We asked, “What is your primary Key Performance Indicator (KPI)?” Respondents could select from seven KPIs: Retention/Engagement, Overall Customer Experience, Monetization/Total Revenue, Monthly Active Users, User Acquisition, Conversion/Onboarding/Activation, Other.

Among those who held a C-level job title, there was an even split between Retention/Engagement and Monetization/Total Revenue, each chosen by 40% of the executives.

We also found that a majority of those at the director and vice president levels looked at Retention/Engagement (25%) and Monetization/Total Revenue (28%) as their core KPIs. Similar to C-level executives, Directors and Vice Presidents didn’t prioritize other core objectives such as Monthly Active Users or Conversion/Onboarding/Activation nearly half as much.

 

 

 

Interestingly, while a significant portion of individual contributors, 31%, saw Monetization/Total Revenue as their main KPI, they didn’t view Retention/Engagement as highly as their executive counterparts. Only 16% chose Retention/Engagement as their primary KPI, the same percentage as those who cared about Conversion/Onboarding/Activation.

High-growth companies often mobilize around “The One Metric That Matters”, but in this case, there are two metrics vying for that top spot. C-level executives and senior-level leaders consider both revenue-generation and retention metrics as critical to their success. Studies have shown that companies see a smaller number of customers contribute a larger amount of their revenue. Retention goes hand in hand with revenue since executives are looking to retain their best customers while sustaining growth with happy customers overall.

Individual contributors should also take note of what executives consider as their guiding metric. To be better aligned with their superiors, more individual contributors should prioritize Retention/Engagement higher and worry less about other metrics such as Monthly Active Users.

2) Easier user experiences and new features are key strategies

To propel their goals forward, product managers and their teams are looking to streamline user experiences and introduce new features as key strategies. Allowing for multiple selections, we asked, “What strategies are you taking to achieve your KPI goal?”.

Coming in first, around 68% of survey respondents selected “Easier UX.” In a close second, a preferred strategy among 65% of respondents was the launch of new features. Increased personalization and better onboarding tied for third place, selected by 57% of respondents.

When we examine the results by role, Marketing was the only group that didn’t have a majority selecting an easier UX as one of their strategies. Instead, nearly 60% of marketers see Increased Personalization as a strategy and about half also selected Better Content, Better Onboarding and New Features.

 

Overall, respondents dismissed the strategy of increasing the number of promotions, with only 12% seeing it as helpful to achieve their KPIs. The second least popular strategy was using location-specific information.

There is a synergy behind product teams looking to make the UX easier to use and add new features at the same time. It makes sense that tests around a new feature also accompany tests around how the new feature is presented in the UX. Through experimentation, they can tackle challenges around the discovery, onboarding and adoption of new features. For example, have seen customers launch a new feature, such as new payment options, that are paired with an improved UX, design and/or flow.

However, product managers should exercise caution when changing the user experience. Users can fiercely favor a particular experience and interpret a well-intentioned UX change as a brand faux paus (remember Snapchat?). Product managers can avoid backlash by making sure their changes or features are tested among different users, especially the most loyal users, and rolled out incrementally over time.

3) Product teams are increasing experiments in monetization, engagement and retention

Consistent with the most popular KPI, our research shows that product managers’ experiments were focused on Monetization and Engagement/Retention.

We asked product teams to indicate the number of tests they were currently running in each category and how that number would would change in 2018. The majority of respondents had planned to increase experiments in three categories: Monetization (53%), Engagement/Retention/Re-activation (59%) and Onboarding/Activation/Registration (56%).

Here are some tests you can run in these categories:

Monetization

Many apps that monetize through ads can A/B test their formats, copy, creatives, placements and how they perform with different audiences. For example, if an app wants to display native ad, it can test whether placing it toward the top or middle would increase the conversion rates. Another monetization test to try how the duration of a video ad affects click-throughs in the short-term and app engagement in the long-term.

To increase subscription rates, try testing how you display the annual subscription rate, such as yearly and monthly. Another test is to see whether adding a user review or other form of validation to a paywall rather than a promotional statement would increase subscriptions. Read the Runtastic case study to learn more.

Another way to increase revenue is to optimize checkout flows so users can go from browsing to completing an order faster. Food ordering app Delivery Hero wanted to test whether changing from a scroll menu to an accordion style menu would help more customers successfully complete their orders. They ran an A/B test on a condensed, accordion menu style and saw a 4.5% relative increase in click through rates and order conversions.

Onboarding / Activation / Registration

Just as first impressions matter, the onboarding experience is a critical moment for an app to get right. A poor onboarding flow could lead to users closing the app before they even log in or sign up.

Music and entertainment platform Vevo was using a standard 4-step onboarding tutorial to let readers know the value of the app and encourage them to sign up or sign in. However, their product team wanted to optimize the onboarding tutorial to maximize logins and new signups. They tested a new onboarding flow that took new users to a login screen right after a brief splash screen. It turns out that users preferred a short and sweet onboarding flow, which resulted in a nearly 10% lift in logins and 6% increase in account sign-ups.

Sometimes, users just need a little extra incentive as shopping rewards app Ibotta learned. The app increased their registration rate by 6.6% by highlighting a $10 welcome bonus on their pre-registration screen. They expanded their experiment cross-platform and saw similar results with a “$10 Welcome Bonus!” banner on desktop and mobile-web version of their website.

Automation is also technique you might test in the onboarding and registration stage. The trick is find the right balance between making registration easier while ensuring the user is getting the appropriate information. For Virgin Mobile UAE, their previous registration flow forced users to decide on a mobile number in order to successfully register their account. They ran experiments on whether more users would sign up if they were automatically assigned a mobile number, with the option to change it later. In conclusion, Virgin Mobile UAE saw an increase in successful sign-ups and decreased their drop-off by 18% by automating mobile number assignments instead of allowing the user to select their own.

Retention / Engagement / Re-Activation

Every app has a different formula for nudging users to open their apps and to have longer sessions. Only through A/B testing can product managers know when a feature can go from being helpful to annoying, or event times distracting.

We’ve seen a number of our customers test content recommendation algorithms as a way to boost engagement and retention. For retail and media apps, experiments can help determine whether providing recommendations based on favorited items and shows or based on actual browsing and viewing history leads to better engagement and retention rates.

Often times, push notifications are helpful reminders but product managers must first get users’ permissions. This is where tests on how to get users to allow and opt in to push notifications come in handy. You can run tests on which types of copy, such as benefits to the user, increases opt-ins to push notifications,

Although we mentioned that testing onboarding flows can increase sign-ups, a solid onboarding flow is also critical to long-term engagement. Users may not properly fill out their profiles, which affects their curated experience and impacts their engagement. They may also need to be educated on how key features work in order to get the most out of the app. That’s why it’s important to test onboarding flows that help mature new users to loyal ones when it comes to retention and engagement.

Get more sign-ups and engagement with tips from our user onboarding guide: