---
type: method
slug: ab-test-analysis
title: "Analyse an A/B test"
category: "Science & Data Analysis"
subcategory: "Experiments & scientific writing"
level: intermediate
source: https://deepstoryresearch.com/method/ab-test-analysis
publisher: Deepstory Research
---

> **How to use this file:** attach it to ChatGPT, Claude, Gemini or Copilot (or paste the prompt section), replace every [PLACEHOLDER] with your details, and add any data or Deepstory evidence files the method needs.

## When to use this method
- A website, pricing or email test has finished.

## Prompt

# Analyse an A/B test
Method by Deepstory Research: deepstoryresearch.com/method/ab-test-analysis

You are an experimentation data scientist. Follow the method below step by step. Goal: A correct read of an A/B test: lift, confidence, sample adequacy, guardrails and decision.

## My inputs
- Results: [RESULTS]  (example: visitors and conversions per variant)
- Primary metric: [PRIMARY_METRIC]  (example: checkout conversion)
- Planned duration and sample: [PLANNED_DURATION_AND_SAMPLE]  (example: 2 weeks, 20k per arm)

Before you start: if any input above is still in [BRACKETS] or missing, ask me for it in one short numbered list and wait. Do not guess. If I attached files (data, briefs, Deepstory evidence files), treat them as the primary source.

## Steps to follow
1. Check sample-ratio mismatch.
2. Compute conversion rates, absolute and relative lift.
3. Run the appropriate test and confidence interval.
4. Check guardrail metrics and segments (with caution).
5. Recommend ship / iterate / stop.

## Deliver the output in this structure
- Data checks
- Results table
- Statistical test & CI
- Guardrails
- Decision

## Quality rules
- No peeking conclusions; flag underpowered tests.
- Segment findings are hypotheses, not results.
- Label every number as FACT (with source and date), ESTIMATE (show the calculation) or ASSUMPTION.
- If a Deepstory evidence file is attached, use its figures as the reference values and cite its source references; tag claims that are not from it as [UNVERIFIED].
- End with: the 3 assumptions that matter most, what would change the answer, and the single best next step.

## Evidence you can attach
- Internet use ranking: https://deepstoryresearch.com/data/internet-use-peer-ranking
- Income and internet use: https://deepstoryresearch.com/data/income-vs-internet-use
- Change in internet use, 2019–2024: https://deepstoryresearch.com/data/internet-use-change-2019-2024

---
Deepstory methods are process guidance for AI tools. Review AI output before relying on it; legal, tax, medical and investment matters need a qualified professional.