Medium: 9 Important Things to Remember for AB Test

There are lots of things that can go wrong [存在隐患] when you try to create AB tests. 

Goal & Motivation

Making it clear helps align the team toward the goal.

Hypothesis

Your hypothesis should be short and to the point. It should not run into a couple of pages.

You can use the following format for writing an A/B test hypothesis:

“By making “X” change, we expect “Y” result”

Test groups

An A/B test has two groups:

1. Control Group: When an A/B test begins users in this group do not see the new change. They continue to use the product in its original avatar.
2. Variant Group: Users in this group see the product in its new updated version.

Randomisation

Users should be put into control and variant groups at random without any specific rule.

Traffic split

As part of the experiment you need to specify how the traffic needs to be split between control and variant.

Duration

A/B experiments need to have a start and end date. (Keep seasonality in mind)

Variants

In most cases you will have a control group and variant. But at times you might want to test more than one variant. So you can do an A/B/C test where B and C are two variants to find out the winner!

Platforms and Devices

Users behave differently across platforms and devices. You will have to run different tests for each type to prove your hypothesis. What works for desktop might not work for mobile. What might work for Android users might not work for iOS users. You cannot extrapolate the results of one platform to say that it will work for the other.

Statistical Significance

“In the context of AB testing experiments, statistical significance is how likely it is that the difference between your experiment’s control version and test version isn’t due to error or random chance.”

Metrics Tracked

The A/B experiment should not hurt other important metrics at the cost of increasing the impact of the experiment.