Experimentation Process
When running experiments, we follow a structured four-phase process to ensure alignment, proper execution, and data-driven outcomes. Each phase has specific objectives and key activities that guide the experimentation lifecycle.
Phase 1: Proposal
The Proposal phase is where ideas are generated and evaluated. These ideas can come from anyone — product managers, developers, brand teams, analysts, etc.
Key Activities
Idea Generation
Any stakeholder can propose test ideas. In this phase, we assess the feasibility of the experiment and its prioritization. This includes evaluating whether the test is technically and operationally feasible, and whether it should be pursued based on its potential impact, required effort, and alignment with strategic goals.
If you have an idea for an A/B test, please follow your standard escalation process to submit it for review.
Phase 2: Planning
Once an idea is approved, we move into Planning, where the structure of the experiment is defined and communicated.
Key Activities
Test Plan Creation
Led by the Product Manager or Brand Team, the test plan outlines the hypothesis, variants, metrics, duration, targeting, and success criteria.
Experiment Preparation
Depending on the type of experiment, the Product Manager or Brand Team is responsible for preparing the experiment setup. This may involve collaborating with the Developer Team to implement code changes for standard experiments or configuring the experiment directly within the Campaigns module inside Teepee, which requires no code change. For detailed instructions, please refer to the Create Experiments section.
Stakeholder Communication
Product Manager or Brand Team sends out communication to relevant stakeholders to align on objectives, timelines, and expected impact.
Experiment Setup in UI
The experiment must be created in the Experimentation UI.
Ensure you have the appropriate access — refer to the UI Roles and Permissions page if you haven’t yet been granted access.
Phase 3: Live
During the Live phase, the experiment is actively running. This phase requires close monitoring to ensure integrity and to catch any anomalies early.
Key Activities
Automated Monitoring
Daily performance reports are available in the UI dashboard.
Live Investigation
Ongoing review for performance shifts, bugs, or unexpected behavior.
Extension and Early Termination Decisions
If needed, the test duration may be extended based on data sufficiency or observed performance trends. The experiment may also be terminated earlier than planned if it shows a significant negative impact on the site’s performance.
Phase 4: Post Live
After the experiment concludes, we move into the Post-Live phase, where results are analyzed and next steps are determined.
Key Activities
End-of-Test Analysis
The Experimentation Team provides a summary of the experiment’s final performance, including key metrics and insights.
Decision & Recommendations
Based on results, recommendations are made on whether to roll out, iterate, or discard the tested variant.
Further Analysis (if needed)
Additional analysis may be conducted to investigate specific segments, behaviors, or secondary outcomes.