HotelTonight A/B Tests and Finds Surprising Results

HotelTonight skyrocketed up the app lists in the last several years to take a leading position in the mobile travel market primarily because they are fast moving company with a laser focus on user experience and design. At Apptimize, we often hear that mobile app teams struggle with the fine line between design led UX and data driven conversion optimization. The HotelTonight team has figured out how data (qualitative and quantitative) and design aesthetic can work nicely together to create a cohesive strategy. Last week, I visited HotelTonight’s beautiful office in downtown SF to chat with Audrey Tsang, their Director of Product, about how they have integrated A/B testing to their culture.

Apptimize: How has A/B testing affected the way you think about user experience?

Audrey: Without measuring, you don’t know how you’re doing, right? Without measuring, you can’t improve. So A/B testing is a great way to do that. Our inventory changes a lot day to day, market-to-market. There’s all kinds of things that impact the user’s experience and in order to control for that, we have to be able to test simultaneously. So really A/B testing helps us bring down the noise so we can draw out the insights.

A/B testing has allowed us to question the assumptions that we make about our user experience. It’s made us revisit some assumptions that we had held dear. Some exciting “oh…” moments have been where we really thought that something would make a difference and what we ended up finding out was that it doesn’t make any difference at all.

Apptimize: Can you tell us an experiment that really surprised you?

Audrey: Some of the hardest to change parts are the ones that say, take the most rebuilding to do. And we had one experiment where we really wanted to question that. It would have cut out a big step in our checkout flow. And we’ve always relied on that step before. What A/B testing allowed us to do was take a risk and test an alternative way that reduced friction. We could evaluate whether or not it was worth it, and then do the real hard work of fixing the architecture to match that change if it proved worthwhile.

A/B testing has been instrumental in helping us refine and iterate on our core product.

Apptimize: With all the experiments that you want to run, how do you manage and prioritize everyone’s ideas?

Audrey: We keep an experiment backlog that anyone in the company can contribute to. We then take it and scope it out, which involves back of the envelope projections on impact to the user problem that we’re trying to solve or the hypothesis we’re making. From there, we can prioritize the backlog for further investigation before building and running the experiment.

Before A/B testing, we still had a feature backlog. We had something called a war room process where, every quarter, we decide on the features and the strategic initiatives we’re going to focus on for the quarter. That process has a similar democratic approach to it where we solicit ideas from the whole company and this is to give a cross-functional view on what needs to be changed and what can be improved upon.

With A/B testing, we can iterate faster by creating experiments as opposed to full blown features. So while those same ideas have kind of lived on in both lists throughout the history of the company, we’ve been able to modify how we work a little bit with the experiments. We’ve been able to move faster, iterate faster, and take a few more risks.