Programmatic Code Block A/B Experiments

The Apptimize Code Block interface is a flexible, code-based to A/B test anything you can imagine. If you can code it, you can A/B test it. The behavior for each variant is pre-defined in your app’s code. Code blocks require an explicitly named section of code for each variant and each experiment.

We’ll walk through a code block experiment example of testing a new login user flow by adding a variant for a guest flow option. Our example app, Urban Attic, is a retail commerce app that lets user log in, browse catalog, and make purchases. In the original flow, the user logs in and enters in information about themselves to make browsing and purchasing experience personalized. In our experiment, we’ll make a new variant that add a button on the login flow to sign in as guest, and then later in purchase flow asks user to login and personal information before paying. Let’s get started!

Here is the original variant in the app: 1) login page, 2) account registration, 3) checkout from bag, 4) payment page. Screenshot showing original variant: image 1 of 4, login screen

Here is the new variant “Added Guest Flow” that users allocated to variant will see: 1) login page with guest option, 2) checkout from bag, 3) account registration, 4) payment page.

1. Create a new experiment. You can do this by clicking on the Create button on the top right of the dashboard list.

2. In Step 1 Details, the default experiment name will be “New Experiment 1,” but you can change it to something more meaningful. We’ll call our example experiment “Add GuestFlow”, and can enter notes or tags to help describe the experiment for other users.

3. Then we will continue to Step 2 Goals and setup our primary goal, which in this case may be a funnel of users who click “Checkout” and click “Pay” to finish purchase.

4. Continue to Step 3 Configuration. For Experiment type, select Code Block. By default, your experiment will have one variant called original which is your current app and will be displayed to users not participating in the experiment. For each additional variant you wish to add, click the Add Variant button, and then name the resulting variant. In this example, we name the new variant “Guest Flow”. You can also add screenshots to each variant to remind yourself of experiment change you are testing.

5. After you have added a variant, you will see a code snippet at the bottom of the page (select the Swift tab if you are writing in Swift and refer to this FAQ). Notice that the code snippet references a code block variable, which must match in your code, and allows the code block to be reused in future experiments. In this case, we set the code block variable to the same name as the experiment, “addGuestFlow”. Paste the snippet into the place in your app’s code where you wish to run the experiment, and then insert code for each experiment in the designated areas. To run the experiment, we wrap the original code hook around the existing code, and add a guest button on login and change the layout order for the new variant we created.

Android (Java)Objective CSwiftJavaScriptAndroid (Kotlin)Node.jsPythonRest APIRoku

Android (Java)

Apptimize.runTest("Add GuestFlow", new ApptimizeTest() {
  @Override
  public void baseline() {
    // Variant: original
    loginWithGuestButton.setVisibility(View.INVISIBLE);
    useGuestFlow = false;
  }

  @SuppressWarnings("unused")
  public void variation1() {
    // Variant: Guest Flow
    loginWithGuestButton.setVisibility(View.VISIBLE);
    useGuestFlow = true;
  });

Note that you can use the same code block variable multiple times throughout your app and across multiple platforms at the same time. Apptimize will ensure the user is consistently assigned to the same variant each time the same code block variable is referenced.

6. Click Continue to Target & Launch at the bottom or Step 4 Target & Launch in the upper sub-navigation to setup who will see the experiment. In this case, we set the allocation to 10% of our users exclusively. Then add a filter so only new users will see this experiment, since current users have already had to login and cannot participate in the experiment.

7. Next, we’ll continue to review for launch. Confirm that the variant names, allocation percentage (10%), our target (“user is new”) are correct, and have the option to preview variants with the Preview Variants tool or set up early access and QA using Preview Groups tool in Step 5 Preview & Launch. When ready, we can start the experiment by clicking the “Launch Experiment” button at the bottom, which will make the experiment move from “draft” to “running” state, and end users will see the experiment on their devices within roughly 15 minutes.

You can find more information about stopping the experiment, setting a winner and interpreting reading results in A/B Experiment Results.

You can also find documentation on our classes and methods in our SDK documentation.