What You Need to Know When Deciding to Build or Buy Your Experimentation Software

You’ve already come to the conclusion that experimentation is the best way to create the optimal user experience. You know that a better user experience can improve engagement, retention and revenue and thus helps you grow your product and organizational KPIs faster. This article will answer the question of whether you should build or buy software.

Experimentation testing gives you cold, hard data to answer any kind of question about how to improve your product or service. By presenting your customers with different versions of your product, you can measure the cause-and-effect relationship between any changes and the impact on KPIs.

While A/B testing is simple in concept, it is hard to execute well, especially given the fragmented customer journey across digital and physical channels today. Now that you’ve made the right decision that experimentation is a priority for your business, you need to choose a framework that will enable you to rapidly run experiments and accurately analyze them.

Should you build or should you buy?

Here are the key considerations that you should keep in mind while answering this question.

1. How many extra engineers do you have?

Your team is really good at building whatever product they are building (probably better than our engineers at that!). The basis of all SaaS products is letting the experts build what they build best. In a world as complicated as product development, we believe it’s important to have a system built by a team of experts (who have scaled a product for the likes of Viber, Vevo, Strava, and Wall Street Journal).

2. Who at the organization needs to be able to use it?

One big problem with building in-house software is that it’s often not “democratic,” or accessible by the people who need to use it to achieve their KPIs. There are multiple teams that have an ongoing interest in using experimentation to meet their goals (product, marketing, engineering, and data to name a few), and the usage of the tool will vary amongst these different stakeholders. Satisfying the requirements of different users can be very difficult, something that requires a deep understanding of each of these personas and their goals.

3. What is the cost of ongoing maintenance?

The cost of building your own platform isn’t just limited to the upfront development; you also have to consider the ongoing maintenance. Studies show that 75% of the total cost of software can be attributed to maintenance. Around a quarter of that goes to fixing bugs once the software is deployed, another quarter goes to adapting the code to a changing business environment, and the remaining half goes to enhancing the feature set for future iterations. You incur all of those future costs when you ship a “light-weight” custom A/B testing framework, because the implicit expectation is that you will fix, adapt and evolve the framework as the business grows.

4. How fast do you want to innovate?

If you do have the time and resources to build a successful experimentation framework, it’s important to think high-level and long-term about the requirements that you will have down road as you develop a robust culture of experimentation. We have found that in-house solutions are often built for a specific use case and are ultimately limiting as the experimentation roadmap develops. Without years of knowledge built up through practice and observation, it’s hard to come up with an end product that will fit nearly all possible edge cases.

5. How fast do you want to start experimenting?

When buying an existing solution, integration and implementation is seamless and typically takes less than a few weeks. Building an experimentation solution, however, can take months– even years. And while you’re building in-house, your competitors are already experimenting, iterating and growing.

For more detail on the topic, download our free whitepaper.