Glossary

A/B Testing

Advanced Verify

An advanced option before launching an experiment that allows you to review the experiment in an interactive manner.

Allocate

Setting the size of an experiment’s population.

Attributes

Attributes defined by you that can be used for targeting and segmentation.

Control

A variant of an experiment that is the same as the original app.

Event

A specific piece of data with associated properties that we track in our analytics system.

Exclusive/Non-exclusive

Toggle that allows experiments to run in parallel with other experiments on a specific user’s device.

Experiment

An A/B test that includes a control and variant(s).

Funnel conversion rate

Percentage of users who progressed through all steps of a funnel out of all the users who enter into a funnel by converting on the first step of the funnel.

Goal

Event or combination of events that measure the success of an experiment.

Launch

Taking an experiment live so that users will start participating.

System Attributes

A way to select a subset of your users for an experiment.

Variant

A version of an app. Variants that you create are typically different than the original app.

Results

Conversions count

Number of unique users who triggered the event at least once.

Conversion Rate

Percentage of participants who have converted (i.e. fulfilled the goal at least once).

Filter

An expression using an attribute, operator, and criteria used to sift through data.

Impact

Raw change in conversion rate from control to variant.

Lift

Percentage better or worse a variant is compared to the control in an experiment.

Number of Sessions

Number of sessions is how many foreground to background transitions that we see.

Occurrences

Number of times the goal event was triggered.

Occurrences per user

Average number of occurrences per participant.

Participant

A user in an experiment’s population who has seen a variant. We determine this by whether the mobile devices runs the code that determines variant treatment.

Population

All users in an experiment who could potentially see a variant. Some users will be allocated into an experiment but never see a variant because they don’t navigate to that part of the app.

Retention

The number of users who have a second session- open the app at least a second time, in a given period of time.

Screens Viewed per Session

On Android this is the number of activities, popups, and fragments we measure during a session. On iOS it is the number of view controllers shown that we measure per session. Note that since this is how we measure screens view, it’s possible given how differently apps are architectured that we overcount the number of screens viewed, however since in an experiment you’re measuring versus a control this doesn’t adversely affect your experiment results.

Session

A session is when a user is in your app, specifically measured by the time between when your app is foregrounded to when the app is backgrounded.

Session Interval

The session interval is time between consecutive sessions, specifically the time between when the app is backgrounded to when it is next foregrounded by a user.

Session Length

Session length is measured by the length of time between a foreground and a background.

Significance

Standard method for statistical significance by confidence. Statistical significance is the probability that the difference in the goal metric between variant and control is not due to random chance.

Trajectory

Direction of lift.

Value

Total sum of all the numeric values associated with goal event.

Value per user

Total event value per participant.