“Bite, chew, spit.”

Brad Haley has a bit of an odd job. He’s one of the team members that tries out new, wild food items that the Hardee’s/Carls Jr. franchise is experimenting with, like the philly cheesesteak burger, the Napa burger (caramelized onions, Merlot glaze, and arugula), the Mac & Cheese burger, or the clambake burger (fried clams, corn salsa, Narragansett lager glaze).

As the CMO of CKE foods, Haley’s job is to test out the experimental items and determine their likelihood for mainstream success. So he and a group of testers try out hundreds of new food ideas in their taste lab.

And that’s where the spit buckets come in (just like in wine tasting). I mean, you can only eat so many burgers in one sitting right?

“Bite, chew, spit.”

Haley’s job exists because it’s expensive to roll out new products. Just imagine the sheer scale of rolling out new recipes, ingredients, and the training involved. So they try their best to validate the success of a new release as cheaply as possible.

We run into the same problem on mobile: rolling out app updates is expensive. Even a simple change can be teeth-gnashingly painful. With the press of a button, you deploy a new update to thousands, if not billions of users.

If a new update doesn’t hit KPIs, you’ve lost months of development time and effort.

So how do we avoid this? By validating changes as early in the process as possible. This week, we’re going to break down 5 techniques to validate mobile features.

User Testing

User testing is one of the most powerful tools you can use to quickly and cheaply validate features. It helps you:

• Figure out whether your product communicates clearly
• Determine if its easy to use
• Figure out how discoverable the new feature is
• Determine where and why people are getting stuck in your product.

On mobile, the UX is even more vital than on web. Users have far lower levels of tolerance for poorly designed apps since they’re often on the go and controls are less precise.

One company that has made this an essential part of their culture is HotelTonight. Every 2 weeks, they invite both HotelTonight users and non-users to their SF office to get early feedback on whatever they’re working on at the time.

In her interview with AppMasters, Audrey Tsang (the director or product) mentioned that they’ll oftentimes use Craigslist or Taskrabbit to get new non-HotelTonight-user testers. In exchange they offer either hotel credits or an Amazon gift card.

What’s fantastic about user tests is that they allow you to gain an outside perspective.

“It’s really important to building great customer experiences. Because if we build in a black box without getting any feedback, what pops out almost never hits the markets.” – Audrey Tsang.

Making user tests a regular part of your development process is crucial to gaining outside perspective about your product. Utilizing tools such as UserTesting.com or just hiring testers from online websites makes it cheap and easy to do so.

For more on how to run a lightweight user test, check out Jason Hreha’s article: How to Run a Cheap, Fast, and Easy User Test or visit UserTesting.com to try out their service.

Dogfooding

Dogfooding refers to the process in which a company uses its own product for testing purposes. This process allows them to uncover bugs or usability issues well before any end user sees the product.

Facebook is one company well known for their dogfooding process.

“Facebook forcibly updates employees to the most recent beta version of apps like Facebook for Android and Facebook Messenger” – Josh Constine, TechCrunch

In doing so, Facebook actually does a few key things:

They allow employees to test on their personal devices. Instead of forcing employees to use test devices just to test out new features, Facebook utilizes feature flags to enable internal updates on employees’ personal devices. This ensures that they’ll be integrated more deeply into daily routines, rather than tested for a few minutes then set aside.

It “solicits immediate and brutally honest feedback.” Because employees are forcibly upgraded, they’re forced to actually use the products in their day-to-day lives, and will be quite vocal about what they do and don’t like.

It allows for precise reporting. Facebook built a specific function to help them get contextual data surrounding a crash. Aptly named “rage shake” it allows employees to violently shake their phones to auto log the phone’s current state and send it to Facebook’s QA team.

“Fail early. Fail often. But fail internally.” -Jason Mark, Fast Company.

The most pressing problem with internal testing is having to sideload apps onto personal phones or distributing test devices to employees. To make this whole process seamless, utilize feature flags by creating a segment of internal users, then enable the feature for only those specific phones.

Beta Testing

Beta testing expands your reach to a larger segment of users. In addition, they’re less biased and less tolerant for buggy apps than your employees are. This stage is a critical step towards releasing a validated feature, because its the first time outside parties will give you feedback about the new changes.

So our goals for beta testing should be to:

• How non-internal users utilize the product. While internal testers will know what to look for and play around with in a new release, beta testers give you a chance to sit back and observe how people outside of your team will interact with the changes. Using your analytics platform, track funnels and events related to the feature to determine usage.
• Identify bugs. With a much larger scale deployment, we want to identify as many bugs as possible so that the full release will be as polished as possible. Tools such as Apteligent or Crashlytics can help you monitor performance for beta testers.
• Determine real-world performance. Along with end-user device performance, we also want to take time in the beta testing stage to figure out how servers and associated backend will perform on a wider scale.
• Generate product awareness. Beta testing is also your first semi-public release, so people will notice the changes. For higher-profile companies, this also means publicity as publications scrutinize the new direction your team is taking your product.

A/B Testing

A/B Testing is a way to get quantitative data that answers the following questions:

• Will real users use the feature? Nothing gives you better insights into your users than you actual users themselves. A/B testing utilizes a sample of your users to gain quantitative data about adoption and usage patterns.
• Is there a significant improvement to the user experience? Looking at the stats of actual users will tell you whether these new features will move the needle on your KPIs such as increasing retention, engagement, or LTV.

A/B testing gives you valuable, unbiased data about how your feature performs in the wild. Unlike other methods above, it gives you quantifiable data from real users. This allows you to make deductions with a high level of confidence on whether you’ll have a positive or negative impact.

Field Testing

Field tests are experiments conducted under actual use conditions, instead of under controlled conditions in a laboratory. They’re the last step before you deploy to your entire userbase.

With the typical mobile release model, teams are forced to deploy a new change to 100% of their users with the click of a button. Unfortunately this has a few major side effects. Mainly that jumping from beta testing with

Running field tests allows you to:

• Stage rollouts. Instead of releasing to 100% at once, you can slowly increase allocation of the new feature until you’ve deployed to your entire audience.
• Get feedback on real-world performance. Unlike the other tests, field testing gives you qualitative data from your users themselves. You’ll get reviews, comments, and other feedback from the non-early adopter crowd.
• Instantly rollback features that aren’t performing. Unlike staged rollouts or normal deployments, using feature flags for releases means that you can instantly rollback your app to a previous version, without going through the app store.
• Test stability for a large population. With beta testing, you’re only deploying to
• Garner word-of-mouth buzz. What’s great about releasing to your actual users is that you generate word of mouth buzz amongst your users. If users love your new release, they’ll talk amongst their networks and create organic growth and re-engagement.

It’s All About Early Validation

Rolling out new features doesn’t have to be an expensive process. Using user testing, dogfooding, beta testing, A/B testing, and field testing allows teams to validate features before releasing to their entire audience.

Utilizing Feature Flags to run field tests and staged rollouts make it incredibly easy. Using advanced targeting, you can choose exactly who you want to deploy a feature to.

To try running internal, beta, or field tests seamlessly get started with our free Feature Flags trial , or learn more on our documentation here.