Defining Tracking Events On the Fly: Charity Miles Case Study

The App

Charity Miles is an innovative app with a fresh take on running. By partnering with different corporate sponsors and charities, they help change the world as well as people’s lives through exercise. Emphasizing action over words, Charity Miles sponsors donate money to charities their users choose for every mile they walk, run, or bike with the app. Together with their network, they make the world a better place by telling stories through actions.

The Problem

As with any other app, one of the hardest things to do is to create effective growth campaigns that help your product and message spread. One of the ways that Charity Miles did this was by creating a sharing feature that allows users to post about their achievements and money earned. The team wanted to try out a few different methods to see how they could increase shares, and reach more potential users.

The A/B Test

Charity Miles decided to test a different call to action wanted to see if tweaking the messaging would increase the number of shares, and help get their message to a wider audience.

While the old call to action got the job done, they wanted to see if having more humorous and quirky copy would do a better job of converting. They used the Visual Apptimizer to quickly change the text and launch their experiment.

Surprise Complications

A little while after the experiment deployed, the team noticed something peculiar: the key conversion events for button presses weren’t being triggered, even though users were seeing the variants. Concerned, they contacted Apptimize to help triage the situation. Together, they discovered a small mistake. Two lines of code used in their proprietary analytics platform to track the clicks had been left out of the latest release.

Every once in a while, stuff like this happens. You just have to suck it up and push it out on the next release. Or do you?

Instead of calling it a day and redeploying to the App Store for a future release, the teams worked together to figure out a potential solution.

The Fix

Working together, the teams created a clever solution to track the data, without having to re-deploy new code. Instead of coding a solution, they defined new metrics for both the Facebook and Twitter share buttons using the Visual Apptimizer. Then, they simply copied and re-launched the experiment, and started seeing results trickle in within minutes.

Even with a minor setback, Charity Miles was able to get the experiment relaunched and back on track within 30 minutes, something that would be unfathomable without the tool. As this doesn’t qualify for an expedited review through the App Store, even if Charity Miles redeployed the code immediately, they would have had to wait approximately 9 days just for their review to be approved.

Results

The team saved more than a week in lost time by using the Visual Apptimizer. When you’re on a mission and growing, that time makes all the difference in the world. As for the results? The test results showed small decreases for both Facebook and Twitter shares. However, neither of the differences were statistically significant, suggesting that this change in language didn’t have a large impact on user behavior.

Even though the test didn’t produce statistically significant results, the team has still learned that this messaging would not have produced the desired results. Furthermore, they’ve been able to gain a better understanding of their user base, as well as more clues as to what types of changes would have a more significant impact for their fans.

Using testing, Charity Miles was able to avoid using up valuable development resources to measure the impact of a simple messaging change. Instead, they spent a fraction of the time that it would have taken with a regular dev cycle, and got solid data in return.