Analytics & evaluation

Open A/B Tests → Analytics for a test in the dashboard. Results are organized in five tabs:

• Sessions: Visitors assigned to a variant (after assignment timestamp).
• Conversions: Sessions with at least one goal event — not individual event counts. Multiple purchases in one session count as one conversion.
• Conversion rate: Conversions ÷ sessions per variant.
• Primary goal: By default, analytics use the test’s primary goal (e.g. purchase). Other events are only included if you filter explicitly in analytics.

Improvement (lift)
Relative gain vs. control: (variant − control) ÷ control.
Example: 10% → 15% = +50% improvement. If control is 0%, usertrax shows the absolute difference.

p-value
Probability the observed difference is due to chance. Lower = stronger evidence of a real effect.
usertrax uses a two-tailed test (difference in either direction).

Statistically significant
Yes only when both are true:

1. p-value is below the threshold for your confidence level (default 95% → p < 0.05)
2. Minimum sample size per variant is reached (default: 100 sessions, configurable per test)

Without enough data, a low p-value may still show No — the test needs more traffic.

Confidence interval (95% CI)
Wilson interval for the conversion rate: the true rate likely falls in this range. A narrower interval means more certainty.

Winner
A variant is marked as winner only if it is significantly better than control. With multiple test variants, the confidence level is Bonferroni-adjusted (stricter threshold) to reduce false positives.

Why does Variant A show +70% lift but “not significant”?
Lift alone is not enough — with few sessions, a large gap can still be random. Run the test longer until minimum sample size is reached.

Why do sessions and conversions differ?
Not every session converts. The rate is derived from both numbers.

What does “sample size adequate” mean?
Each variant (including control) has at least as many sessions as the test’s minimum sample size setting (default: 100).

Can I stop early?
Results before minimum sample size are exploratory — not a reliable basis for production rollouts.

What about 3+ variants?
Each variant is tested against control. More variants means a stricter significance threshold (Bonferroni).

// Example conversion with A/B test context
{
  "event": "signup",
  "ab_test_data": {
    "button_color_test": "variant_red",
    "headline_test": "variant_short"
  }
}