Experiments

Comparing variants to improve your overall performance.

Experiments are a type of Experience that enable you to test different variations of your content to determine what resonates best with your visitors. You can set up one or multiple variants per Experiment, show them to all users or just specific Audiences, and analyze the performance either with your own tools or Ninetailed's Experience Insights.

Creating an Experiment

You can create an Experiment by creating a new entry of the type Ninetailed Experience in your Content Source and choosing "create experiment" in the config field. On top of the title (required) and the description (optional), you are now presented with a number of configuration options.

Primary Metric

The primary metric determines how the success of an experiment is measured (or in other words: what defines success). Selecting a primary:

• Helps to signal your team what the goal of your experiment is

• Will automatically select that metric in the Insights section

• Enables the multi-armed bandit distribution type

Distribution

The distribution of an Experiment decides what percentage of users that are part of this Experiment are allocated to which variant. You can also add or remove variants, enabling you to run A/B/n tests. Every Experiment consists of a control and at least one variant. The control represents your baseline content.

About the stickiness of variants:

For as long as the variant distribution is not changed for the given Experiment, variants are sticky for each user. This means that each user will always see the same variant of an Experiment. If the variant distribution is changed, some users might be re-allocated to see another variant.

There are three distribution models to choose from:

Manual distribution

Manually select what percentage of traffic should be distributed to each individual variant. The sum of all values must equal 100%. If you do not want to allocate all of your traffic towards the experiment in general, you can use traffic allocation.

Even split distribution

The traffic will be split evenly among all variants.

Multi-armed bandit distribution

Ninetailed can automatically and gradually allocate more traffic towards the winning variant. This smart capability is referred to as the multi-armed bandit approach. Ninetailed uses the combination of Experience Insights, your selected primary metric and other factors to update the variant distribution.

Multi-armed Bandit distribution

In order to select multi-armed bandit distribution, you need to select a primary metric.

Overall, the distribution closely resembles the Probability To Be Best in Experience Insights for the selected primary metric. However, it adjusts this percentage to also include an "exploration factor", making sure that every variant will always have a small percentage of traffic allocated to it. The variant distribution is evaluated and updated at least daily.

Traffic Allocation

Traffic allocation lets you allocate only a portion of all visitors that would be eligible for this Experiment. By default, 100% of your eligible traffic will be allocated to your Experiment.

Example: Imagine you have an Experiment running that is only shown to a "Returning Visitors" Audience, with a 50/50 split between two variants and a traffic allocation of 30%. We now have 1000 users visiting our website, and 200 of them are returning visitors. The traffic allocation will allocate 30%*200 users = 60 users. Each variant will be displayed to 50% of these users (30 users each).

Experiment traffic allocation example

About the stickiness of traffic allocation:

Similar to variant distribution, a user is permanently either allocated to an Experiment or not. Only if the traffic allocation changes, some users are reallocated. If the traffic allocation is increased, all users that have already seen the Experiment will be kept in that Experiment.

Difference Between Traffic Allocation and the Control Group

Visitors not participating in an experiment and those participating in the control group of an experiment may see the same content. This case arises when some of your visitors are not assigned to an experience. You should recognize those users as statistically distinct from one another. If you were to consider these users as belonging to a single group and compared their conversion rate against those seeing experiment variant content, you may introduce base rate fallacies in your analysis.

• The control group helps you understand the performance uplift of your variants. Only visitors that are specifically chosen to see this control group are counted towards your control group in any analytical tools such as Experience Insights.

• If you are running multiple Experiences on the same component, being in the control group will exclude the visitor from any other Experiences that have a lower priority at that time. In contrast, being excluded from an Experiment (either due to traffic allocation or Audience eligibility) will still evaluate any other Experiences that have a lower priority and potentially show a variant for these.

Audience

Choosing an Audience will set an eligibility requirement for this Experiment. Therefore, only users that are eligible for this Audience at that moment will see that experiment.

Components

Mapping components enable you to choose which entry should be exchanged with another one. Depending on the number of variants you have set up in your distribution configuration, you can map one or several variants.