Experiments
Comparing variants to improve your overall performance.
Experiments are a type of Experience that enable you to test different variations of your content to determine what resonates best with your visitors. You can set up one or multiple variants per Experiment, show them to all users or just specific Audiences, and analyze the performance either with your own tools or Ninetailed's Experience Insights.
Creating an Experiment
You can create an Experiment by creating a new entry of the type Ninetailed Experience in your Content Source and choosing "create experiment" in the config field. On top of the title (required) and the description (optional), you are now presented with a number of configuration options.
Distribution
The distribution of an Experiment decides what percentage of users that are part of this Experiment are allocated to which variant. You can also add or remove variants, enabling you to run A/B/n tests. Every Experiment consists of a control and at least one variant. The control represents your baseline content. In total, the sum of all percentages must always be 100%.
About the stickiness of variants:
For as long as the variant distribution is not changed for the given Experiment, variants are sticky for each user. This means that each user will always see the same variant of an Experiment. If the variant distribution is changed, some users might be re-allocated to see another variant.
Traffic Allocation
Traffic allocation lets you allocate only a portion of all visitors that would be eligible for this Experiment. By default, 100% of your eligible traffic will be allocated to your Experiment.
Example: Imagine you have an Experiment running that is only shown to a "Returning Visitors" Audience, with a 50/50 split between two variants and a traffic allocation of 30%. We now have 1000 users visiting our website, and 200 of them are returning visitors. The traffic allocation will allocate 30%*200 users = 60 users. Each variant will be displayed to 50% of these users (30 users each).
About the stickiness of traffic allocation:
Similar to variant distribution, a user is permanently either allocated to an Experiment or not. Only if the traffic allocation changes, some users are reallocated. If the traffic allocation is increased, all users that have already seen the Experiment will be kept in that Experiment.
Difference Between Traffic Allocation and the Control Group
Visitors not participating in an experiment and those participating in the control group of an experiment may see the same content. This case arises when some of your visitors are not assigned to an experience. You should recognize those users as statistically distinct from one another. If you were to consider these users as belonging to a single group and compared their conversion rate against those seeing experiment variant content, you may introduce base rate fallacies in your analysis.
The control group helps you understand the performance uplift of your variants. Only visitors that are specifically chosen to see this control group are counted towards your control group in any analytical tools such as Experience Insights.
If you are running multiple Experiences on the same component, being in the control group will exclude the visitor from any other Experiences that have a lower priority at that time. In contrast, being excluded from an Experiment (either due to traffic allocation or Audience eligibility) will still evaluate any other Experiences that have a lower priority and potentially show a variant for these.
Audience
Choosing an Audience will set an eligibility requirement for this Experiment. Therefore, only users that are eligible for this Audience at that moment will see that experiment.
Components
Mapping components enable you to choose which entry should be exchanged with another one. Depending on the number of variants you have set up in your distribution configuration, you can map one or several variants.
Last updated
