Video provider uses experiments to recommend content their viewers watch longer

The Customer

A leading content provider dramatically increased episode watch time by server-side testing their personalized recommendations feature with Apptimize. Instead of simply testing the feature on OTT (over-the-top) set-top devices, they were able to test across mobile, web, and OTT, creating a holistic understanding of user cohorts, giving them insight and inspiration for future feature experiments.

The Challenge

Personalization based on recommendation algorithms is improving outcomes for businesses around the globe, but there are untapped pools of information companies aren’t leveraging because it’s siloed by platform instead of tracked by user. One leading content provider unlocked the power of server-side data to enable cohort testing, targeting and true personalized cross-platform experiences with Apptimize.

The company was facing stagnant user engagement and wanted to improve a key metric: user time spent watching video content. Prior to utilizing Apptimize experiments, users saw the same content based on what content was most popular overall.

The Hypothesis

In order to do this, they looked into introducing a personalization algorithm to suggest content to users based on their view history, positing that if users were presented with content that they found more relevant to their interests, they would be more likely to engage, watch more videos and spend more time overall on the platform.

The Test

To execute their new strategy, the company enlisted Apptimize to enable them to test their theory across their entire content distribution platform ecosystem, including mobile, web, mobile web, and OTT. To gain a full understanding of how their users are interacting with personalized content throughout these mediums, they made the decision to employ Apptimize’s server-side A/B testing capabilities. This ensured that user experiences during the test was never siloed — server-side cohorts were created using Apptimize and then used for personalization and targeting across all platforms.

The company thus took a data-driven approach to their bold hypothesis that content personalization would lead to increased time spent on content consumption. Instead of simply releasing features that they thought may work, they took a targeted, staged approach.

The Results

When the company rolled out its content personalization tests with Apptimize, the results were initially surprising. They initially saw a 5.3% decrease in shows started across all their content distribution platforms — this was discouraging and not what they had hoped for. But, this also wasn’t the key metric they were tracking. Though shows started is an important metric for a video content company to track, this was not the primary metric this test was aimed at increasing — they wanted to increase total watch time per user, which they did, by 10.2%.

The time that each viewer watched an episode, and how often they finished it, dramatically increased. This led to an increase in the overall time each user spent watching shows on the company’s platforms. The personalization test proved that the company’s hypothesis was correct — personalized content based on customer view history increased customer watch time. The test pointed to signs of future engagement: there was also a 4.9% increase in the number of shows added to users’ watch lists. Content consumption is rapidly evolving and this company responded with its cross-platform content delivery strategy to meet their viewers on their preferred devices, wherever they are.

This increase of 10.2% in watch time and 4.9% in content added to users’ queues leads to increased advertising revenue and customer retention. Growth metrics mean nothing when they’re not tied to the company’s business objectives. The primary objectives, in this case, are to increase ad revenue and maintain and grow their user base. These two objectives are met by increasing the total amount of time users engage with video content.

This video content provider dramatically increased episode watch time by testing their personalized recommendation feature with Apptimize. They tested across their entire content distribution ecosystem, creating a holistic understanding of user cohorts. They were able to increase total watch time by 10.2%, which leads to increased ad revenue, creating more opportunities for testing and optimization within their ads.

Thus, testing how users respond to an implemented change has shifted the company’s focus to phased data-driven innovation.

The Takeaway

In developing an innovative mobile experience, the company’s mobile team has turned to experimentation to help them learn which new features and user experiences are truly effective at improving user experience, and which should be cut. The technology and a mobile strategy team from Apptimize helped transform the way the company develops their mobile apps. Using Apptimize’s mobile A/B testing platform, the company has uncovered some unexpected results that are directly leading to product improvements.

Personalization based on recommendation algorithms is improving outcomes for businesses around the globe, but there are untapped pools of information that companies may not be using because it’s siloed by platform instead of tracked by user. This company unlocked the power of server-side data to enable cohort testing and targeting and true personalized cross-platform experiences.

Apptimize is the best mobile-focused client and server-side optimization engine available today for testing content recommendation algorithms. Content companies, from news organizations to movie producers, are vying for audience attention. Apptimize is the only optimization engine that’s built with those unique users in mind. We allow you to create a consistent user experience no matter how many channels the user interacts within one day. Make sure your content is reaching its audience with server-side recommendation algorithm testing.

We have a set of client and server-side SDKs that product teams are using to enable new types of mobile experiments, access better data, and reach their end users in a more strategic, targeted way. Schedule a custom demo with our testing experts now.