Skip to content
Experimentation Wiki

FAQ's

Getting Started

How do I know if my idea is suitable for an experiment?

If you think something might be worth testing, contact our team using your standard escalation process. We’ll set up a short meeting to talk it through and help decide if A/B testing is the right method. Even if it’s not, we can point you in the right direction.

How can I request an experiment?

Please use your standard escalation process to make an experiment request.

Where can I find out which experiments are currently live?

All experiments can be found on the Experimentation UI home page. Refer to the managing your experiments pages for more information.

Methodology on Duration and Traffic

What if I need to have a shorter timeframe for my experiment? What is the minimum duration?

Only run the experiment for 2-3 weeks if there are commercial, trading, or external factors dictation a shorter test, e.g. peak trading period, or changes need to be made in time for regulatory compliance, However, it is still necessary to have a minimum of 5k orders and 20k visitors per bucket. Please note, if the experiment runs for under 2 weeks, there is a high chance that the collected data is not representative, even if there are over 5k orders and 20k visitors per bucket. Main reasons for this include the following: data can be skewed by weekly or even daily phenomena, e.g. week-long sale, traffic of specific days can be under-representated, e.g. weekends. You can visit methodology for more information on traffic threshold and suggested duration of an experiment.

What if experiment’s data is not sufficient? What is the maximum duration?

If there is not enough traffic, 5-6 weeks would be recommended to have a minimum of 5k orders and 20k visitors per bucket. Please note, duraing an experiment there are many changes happening on the websites that could be affecting the results of the experiment, e.g. products going out of stock, new product releases, increase or decrease in the number of customers; however, if an experiment runs for longer than 6 weeks, it becomes increasingly more difficult to identify such changes. Moreover, if we require more than 6 weeks’ worth of data to see any meaningful difference in metrics, it may be the case that we are testing a change that is too subtle. Note that the last point mainly applies to changes that are seen by a very small percentage of a website’s traffic. However, in the case of testing a single small website, if we don’t have enough traffic in 6 weeks for 5k orders and 20k visitors per bucket, it might still be worth running the test for 6 weeks and seeing how it impacts performance metrics, even if we can’t perform significance testing with the same level of confidence. You can visit methodology for more information on traffic threshold and suggested duration of an experiment.

Planning an Experiment

How many variants should I include?

This depends on what you’re testing. More variants mean the test will need to run longer to produce valid results. If your site has low traffic, it’s usually better to keep the number of variants to a minimum. the maximum of variants the UI supports is 8 in total: one control and seven variants.

How do I decide which site or website to test on?

Some experiments are specific to a brand or a website of a brand. For example, suppose you want to compare text in the banner on a Netherlands website. A question you might want to ask: “Is it better in English version or Dutch version?”. In this case, running the experiment specifically on the Netherlands website of the brand is preferable. Otherwise, choose based on where the change is most relevant. For broader changes, it may be useful to run tests across multiple websites.

Do we have to run an A/B test for every change?

We strongly encourage running experiments before making major changes—since even small updates can impact user experience and financial performance—there are situations where it’s appropriate to proceed without testing. For example, if a change is required (like for regulatory reasons), we can roll it out directly without running an A/B test.

Are all changes worth testing?

In general, it’s a good idea to test changes whenever possible—even small ones. What may seem like a minor update (like a font change) can still influence user behaviour and business outcomes in unexpected ways.

Can I just make a change and check the data later?

We strongly recommend avoiding this approach. Making changes without a controlled A/B test can lead to misleading conclusions and potentially negative business impact that may be hard to reverse. While there are rare cases where post-change analysis is appropriate, this should always be discussed with the relevant teams in advance.

Running an Experiment

How do I choose the right success metric?

The success metric should reflect the main goal of your test. Supporting metrics help explain the result. These will be finalised with your facilitator in the test planning session.

What metrics are available?

The metrics you will need will depend on your goals. The UI can support a wide range of metrics, but you can a short list of common ones in the interpreting your results page.

How long will my experiment need to run?

This depends on traffic and the chosen metrics. Most experiments run for 2 to 6 weeks, but we usually reccommend 4 weeks. Our team will provide an estimate after your planning session.

When can my experiment go live?

Experiments typically go live Monday to Thursday. Friday launches are strongly discouraged due to limited weekend monitoring.

Can my experiment go live on a Friday?

Not recommended. If something breaks or goes wrong, there may be no one to catch it until Monday, which risks poor user experience and lost data.

During and After the Experiment

When will I see the results?

After 24 hrs an experiment has a gone live, you will be able to view your results by finding your experiment on the dashboards. For further information, consult the viewing experiment dashboard page.

Can I change the % split of variants mid-test?

Sometimes, but it is worth considering the consequences of doing so. Large changes to the split can affect the data collection and the results that appear on the dashboards. To mitigate the impact, you will need to run the experiment much longer than planned to ensure it is more representative of the new split. If you require a variant split change, you will need to contact our team to ensure we can update the data collection and reporting. This is so we can ensure the data is reflective of the split change and different reports can be produced reflecting the change before and after.

What does ‘experiment reach’ mean?

It refers to the percentage of revenue from sessions that the experiment could affect. For example, a list page test only includes sessions where users visit list/search pages.

What’s the difference between cumulative and non-cumulative results?

  • Cumulative: Shows the total effect over time.
  • Non-cumulative: Shows each individual day’s data.

General

I need to perform some actions that I can’t find in the UI. What should I do?

Currently there are some limitations to the UI that we are currently working on removing. In the mean time, please use your standard escalation process for the following actions:

  • Add CTR to specific pages

  • Add new metrics that are not listed in the UI

  • Target a subset of PDPs by Product ID (SKU list) or by URL (e.g. PDP price redesign)

  • Apply an experiment to a custom base event not listed in the UI (e.g. the App homepage)*

  • Apply an experiment to a specific segment (e.g. specific devices or new/returning visitors)

  • Update monitoring after editing an experiment (e.g. change dates or add sites)

  • Exclude specific dates while the experiment was live (e.g. if there was a bug on the site)

Note: This is not needed if the experiment was paused

*A custom base event in A/B testing is a specific user action you choose to measure success, so the test focuses on what matters most to your business rather than default actions like clicks or purchases. Need a question answered that’s not here? Contact us via the contact information page