A/B Test Analysis and Significance Calculator

Description

## A/B Test Analysis

Analyze the following A/B test:

**Test Name:** [TEST_NAME]
**Hypothesis:** [HYPOTHESIS — what you expected to happen]
**Test Duration:** [DURATION]
**Traffic Split:** [SPLIT — 50/50, 70/30, etc.]

**Results Data:**
| Variant | Visitors | Conversions | Conv. Rate |
|---|---|---|---|
| Control (A) | [N_A] | [C_A] | [RATE_A]% |
| Treatment (B) | [N_B] | [C_B] | [RATE_B]% |

**Analysis:**

### 1. Conversion Rate Comparison
- Control rate: [RATE_A]%
- Treatment rate: [RATE_B]%
- Relative lift: [LIFT]%
- Absolute difference: [DIFF] percentage points

### 2. Statistical Significance
```python
from scipy.stats import norm
import numpy as np

# Two-proportion z-test
p_a = [C_A] / [N_A]
p_b = [C_B] / [N_B]
p_pool = ([C_A] + [C_B]) / ([N_A] + [N_B])
se = np.sqrt(p_pool * (1 - p_pool) * (1/[N_A] + 1/[N_B]))
z_score = (p_b - p_a) / se
p_value = 2 * (1 - norm.cdf(abs(z_score)))
```

- Z-score: [VALUE]
- P-value: [VALUE]
- Significant at 95%? [YES/NO]
- Confidence interval for difference: [[LOWER], [UPPER]]

### 3. Practical Significance
- Minimum detectable effect (MDE): [MDE]%
- Is the lift >= MDE? [YES/NO]
- Revenue impact projection: [PROJECTION]

### 4. Segment Analysis
- Did the treatment work equally across:
  - Device types (mobile/desktop)
  - New vs returning users
  - Geographic regions
  - Traffic sources

### 5. Recommendation
- **Decision:** [SHIP/ITERATE/KILL]
- **Confidence:** [HIGH/MEDIUM/LOW]
- **Reasoning:** [EXPLANATION]
- **Next steps:** [FOLLOW_UP_TESTS]

**Variables:** Replace all data values with your actual test results.

Free to copy and use. Compatible with Claude 4 Opus, GPT-5.

Usage Instructions

Tags

Compatible AI Models

Version History

Related Prompts