Editorial contribution · business
Protect an A/B test from misleading conclusions
Want to compare two versions and interpret the outcome properly? Define metrics, duration and how to handle inconclusive results before you start.
Published: · Version 1
A prompt to try
Create a predefined analysis plan for [EXPERIMENT]. Hypothesis: [HYPOTHESIS]. Target population: [POPULATION]. Unit of randomization: [UNIT]. Available data and baseline rates: [DETAILS]. Define the primary metric, observation window, exclusions and additional guardrail metrics. Check for possible interactions between participants, multiple assignments and unequal measurement across groups. Ask about the minimum meaningful effect and statistical assumptions before calculating sample size or duration. Document the method and calculate using supplied values; leave missing values open. Define how to handle dropouts, multiple comparisons and early stopping in advance. Create a blank results table with uncertainty intervals and decision rules. Check whether this experimental design would support causal conclusions. No invented results, no winner without data and no retrospective adjustment of the primary metric to obtain a desired outcome.
Copy the text into your AI, fill in the placeholders and check the answer.
What you need
- Your own access to a suitable AI. Only add information you are comfortable sharing.
How to start
- Copy the prompt and paste it into your AI.
- Replace the placeholders and add any missing information.
- Check the answer and ask follow-up questions to adapt it to your situation.
RobinPrompt.BETA 0.1.9