Create an A/B test - Optimize Help

Create an A/B test

Follow these steps to create a simple A/B test.

An A/B test is a randomized experiment using two or more variants of the same web page (A and B). Variant A is the original and variant B through n each contain at least one element that is modified from the original.

In this article:

Before you can start testing, you need something to test on. Rather than randomly testing items on your homepage or adapting your checkout flow, start small. Test a change to a call-to-action (CTA), change the color of a button, or remove an extraneous form field. Once you're comfortable creating variants and experiments, you can expand the scope of your testing.

Find more idea starters in the Example experiments article.

Create a hypothesis

Before diving into your first experiment you should identify a problem and create a hypothesis that's backed up by data.

Start by identifying a problem that you want to solve. For example, have your conversions dropped off? Have traffic patterns changed? Have your demographics shifted? A close examination of trends in your Google Analytics behavior reports is a great place to start.

Once you've identified a problem, assemble a team within your organization and solicit their opinion about the cause of the problem. Use feedback from this cross-functional team to form your hypothesis, an educated guess that you'll validate or invalidate with experimentation.

Example hypothesis

Changing the color of the 'Add to cart' button from blue to green will increase revenue by 10 percent."

After you've identified a problem (low conversions), and worked with your team on a hypothesis (changing the button color) your ready to test your hypothesis on your website.

Create an A/B test

In this experiment, we’ll test changing the color of a button:

1. Go to your Optimize Account (Main menu > Accounts).
2. Click on your Container name to get to the Experiments page.
3. Click CREATE EXPERIMENT.
4. Enter an Experiment name (up to 255 characters).
5. Enter an Editor page URL (the web page you'd like to test).
6. Click A/B test.
7. Click CREATE.

Use a fully resolved URL in the Editor Page field. Redirects aren't supported here.

The variants card

The top of the experiment page includes the variants card. This where you'll create the specific changes to your web page that you wish to test, called variants. You can create as many variants as you wish to test against your original page (the Editor page in Optimize).

Create a variant

To get started, click CREATE VARIANT (bottom right), enter a variant name, then click ADD. Repeat this process to create additional variants. When finished, you'll see a list of your new variants on the variants card.

To start making changes. click anywhere in the variant row (which will say "0 changes"). This will launch the Optimize visual editor – an overlay on top of your editor page consisting of two components: the app bar (at the top of the page) and the editor panel (floating in the lower right).

Start editing:

1. Click on any web page element you wish to edit (e.g. a button).
2. Use the editor panel to make a change (e.g. change the button color).
3. Click SAVE.
4. Continue making edits as necessary.
5. Click DONE.

Learn more about the visual editor.

Variant weighting

All variants are weighted equally by default in Optimize. A visitor who is included in your experiment has an equal chance of seeing any of your variants. If you want to direct more, less, or even all of your traffic to a specific variant, you can adjust your variant weights on the experiment details page.

Learn more about variant weighting.

The configuration card

Objectives tab

Configure your experiment objectives on the OBJECTIVES tab:

1. Select a Google Analytics view.
2. Select a Primary Objective.
3. (Optional) Click + ADD AN OBJECTIVE to add an objective. Optimize users can use up to three pre-selected objectives per experiment and see data for those objectives in Optimize reports. Optimize 360 customers can select additional objectives and view additional goals.
4. Add a description and hypothesis.
5. Click SAVE.

Targeting tab

Configure who and when to target on the TARGETING tab.

Who to target

The Who section of the Targeting tab is used to select the visitors to whom you wish to target your variant. Enter a numeric value (in tenths of a percent) or use the slider to specify the percentage of your visitors to include in your experiment.

When to target

The When section of the Targeting tab determines where the experiment is shown. When is evaluated each time a user visits the experiment page. Use the rules to set where your experiment is seen. To create a targeting rule, click AND, then select a targeting rule type.

You must create either a URL or path targeting rule to start your experiment. The quickest way to get started is to create a URL matching rule with the same Editor page URL that you used when creating your experiment.

Targeting rules

Optimize includes the following targeting rule types, which you can read more about in the following articles:

Learn more about targeting.

Start your experiment

Click START EXPERIMENT. When the status field says "running," your experiment is live on the web. Most updates happen within a minute.

How long should your experiment run?

Keep an experiment running until at least one of these conditions has been met:

1. Two weeks have passed, to account for cyclical variations in web traffic during the week.
2. At least one variant has a 95 percent probability to beat baseline.

Reports

To monitor a running experiment or see the results of a concluded experiment, click the Reporting tab at the top of the experiment detail page. The report is broken down into a series of cards that contain data about your experiment, including its status and how your variants perform against your objectives.

In addition to the reports included in Optimize, you can also see Optimize reports in Google Analytics. Sign in to Google Analytics, select the Reporting tab and select Behavior > Experiments in the report navigation. Learn more about Optimize reports.

Related resources

Measure

Measure