A/B Testing Glossary

A/B testing helps product managers to rapidly grow and optimize digital experiences. Our A/B testing glossary breaks down commonly used terms and concepts used when A/B testing across different channels – mobile, web, server and OTT.

 

A

A/B Testing

An experiment comparing two versions of a web page or mobile app to see which one performs better.

Read full description

A/A Testing

A/A testing is a form of A/B testing, in which both the control and variation are identical.

Read full description

Allocation

The number of users (usually a percentage of your total user population) assigned to an A/B test or feature flag.

App Downloads

The number of downloads a mobile app receives.

B

Beta Release

The software development phase following the alpha (sometimes after the canary or pilot) release. The feature is usually stable and complete but likely to have bugs.

C

Canary Release

The earliest stable release of a new feature or product following internal QA and the alpha release.

Control

A variant of an experiment that is the same as the original app (no change) against which new variants are compared.

Click-Through Rate (CTR)

The percentage of visitors/users who clicked on a link compared to the page/screen’s total visitors.

Conversion

The number of users who performed a certain action (e.g. click, purchase, registration, etc.)

Conversion Rate

The percentage of conversions compared to total users.

Conversion Rate Optimization

An iterative, data-driven process to increase conversion rates.

Customer Lifetime Value

An estimate of the average revenue that your customers will generate throughout their lifespan as a customer.

D

Daily Active Users (DAU)

The number of users that visited a website or mobile app on a given day.

Dark Launch

The first release of a new feature’s code hidden behind a feature flag. A dark launch is the first step to a gradual, staged, or phased rollout.

Deep Link

A link that leads directly to a specific screen of a mobile app that is not the homescreen.

E

Events

A user activity or behavior on a website or mobile app being tracked by analytics.

Exclusive Experiments

An A/B test or experiment whose participants are not simultaneously allotted into any other experiments.

Experiment

An A/B test.

F

Feature Flags

A code switch or toggle that contains a feature’s code so that it can be independently turned on/off. Feature Flags separates the code development process from the deployment process.

Feature Rollout

The software development process of gradually introducing a new feature to a set of users.

Funnel

An order of steps a customer or user takes toward a key action.

Funnel Conversion Rate

Percentage of users who progressed through all steps of a funnel out of all the users who enter into a funnel by converting on the first step of the funnel.

G

Goal

An event or combination of events that measure the success of an experiment.

H

Heatmap

A visual representation showing where users interacted with a website or mobile app using colors to indicated areas of higher engagement.

I

iOS A/B Testing

The process of A/B testing an iOS app despite the App Store approval process.

iOS Feature Flags

The process gradually rolling out new features on an iOS app despite the App Store approval process.

L

Landing Page

The first webpage that users land on.

Landing Page Optimization

The process of improving elements on a landing page to increase conversions.

Lifetime Value

An estimate of the average revenue that your customers will generate throughout their lifespan as a customer.

Lift

Percentage better or worse a variant is compared to the control in an experiment or A/B test.

M

Monthly Active Users (MAU)

The number of unique users visiting a website or mobile in a given month.

DAU/MAU

A ratio of daily active users to monthly active users. A common measure of stickiness.

Mobile Analytics

The measurement and analysis of data to inform an understanding of user behavior across a mobile app.

Mobile App A/B Testing

The practice of A/B testing mobile apps despite App Store and Play Store approval processes.

Mobile App Optimization

The process of using controlled experimentation to improve a mobile app’s ability to drive business goals.

Mobile Retention

The retention rate of users in a mobile app.

Multivariate Testing

A variation of A/B testing where multiple changes are tested in the same experiment.

An experiment comparing two versions of a web page or mobile app to see which one performs better.

Read full description

Multivariate Testing vs A/B Testing

Learn about the key differences between A/B testing and multivariate testing, and when is the best time to use each approach.

O

Onboarding

The start of the customer experience with a digital product, oftentimes the first session or the first run experience.

OTT A/B Testing

The process of A/B testing video streaming app made for a TV viewing device.

P

Participant

A user in an experiment’s population who has seen a variant.

Population

Users who meet the criteria to be entered into an experiment or A/B test.

Purchase Funnel

The path taken by a potential customer through a website or app as they move towards becoming a customer.

Python A/B Testing

The process of A/B testing on a server running Python.

R

React Native A/B Testing

The process of A/B testing on a mobile app developed using the React Native framework.

S

Sales Funnel

The path taken by a potential customer through a website or app as they move towards becoming a customer.

Search Engine Optimization

The process of enhancing webpages to rank higher in search engines results.

Server-Side A/B Testing

A form of experimentation where unlike in client-side testing, the variations of a test are rendered directly on the web server before it is delivered to the client.

Shopping Cart Abandonment

When a potential customer starts the check out process for an online order but drops out of the process before completing the purchase.

Split Testing

A strategy for conducting controlled, randomized experiments with the goal of improving a conversion metric on a website or mobile app. A synonym of A/B testing.

Squeeze Page

A landing page which is specifically designed to ‘squeeze’ e-mail addresses out of visitors and prospects.

Statistical Significance

The likelihood that the difference in conversion rates between a given variation and the baseline is not due to random chance (rejects the null hypothesis).

Visitor Segmentation

The process of dividing your visitors to your website or mobile app based on a specific criteria, such as demography, geography, or user behavior.

T

Type 1 Error

A statistics term used to refer to an error that is made in testing when a conclusive winner is declared although the test is actually inconclusive.

Type 2 Error

A statistics term used to refer to a testing error that is made when no conclusive winner is declared between a control and a variation when there actually should be one.

U

Unique Selling Point

The competitive differentiators that a business has over its competitors.

Usability Testing

A method of evaluating a website or app’s readiness for release by testing it with real users who are part of your target audience.

User Testing

A method to get qualitative user feedback via user interviews.

User Flow

The path taken by a prototypical user on a website or app to complete a task.

User Journey Map

A diagram of the entire user experience, starting with initial contact or discovery, and continuing through the process of engagement into long-term loyalty and advocacy.

V

Value Proposition

The essence of the benefit that your product or service provides to the customer.

Viewable Impression

A standard measure of ad viewability defined by the IAB to be an ad which appears at least 50% on screen for more than one second.

W

Web Analytics

The measurement and analysis of data to inform an understanding of user behavior across web pages.

Website Optimization

The process of using controlled experimentation to improve a website’s ability to drive business goals.

Website Personalization

The process of creating customized experiences for visitors to a website.