A/B testing & bandits
The A/B Testspage (also called the Experiment Lab) is where you measure the impact of changes you make to the storefront experience. Today it tracks three live experiments — the opening message, the system prompt, and the widget theme — and reports per-variant conversion outcomes with a confidence indicator so you don’t call a winner on noise.
Split tests, not auto-reallocating bandits
How variants are assigned
Assignment is deterministic by session id. A given shopper always sees the same variant, including on return visits, so a returning customer never gets a jarringly different greeting or theme. Splits are even across variants.
Opening-message experiment
The headline experiment splits new shoppers across two opening greetings and tracks what happens next. Author it right on the page:
- Variant A and Variant B — two messages, up to 500 characters each. They must differ.
- Run on the storefront widget — a toggle that enables or pauses the experiment. When paused, past data stays visible for reference but no new assignments are made.
For each variant you see assignments and three downstream rates: chat open, add to cart, and lead captured. A comparison table below shows, per outcome (opened chat, added to cart, captured lead, requested handoff), Variant A’s rate, Variant B’s rate, the absolute lift of B over A, and a confidence badge.
How confidence is judged
Zubby runs a one-tailed z-test for proportions and maps the z-score to a coarse label rather than publishing a p-value — the point is to discourage premature decisions:
| Label | Meaning |
|---|---|
| Very high confidence (≥99%) | z ≥ 2.33 |
| High confidence (≥95%) | z ≥ 1.96 |
| Moderate confidence (≥90%) | z ≥ 1.64 |
| Low confidence — keep running | below 1.64 |
| Insufficient data | a variant has zero assignments |
Wait for at least moderate confidence
System-prompt variants
Zubby also tracks which system-prompt variant each shopper session was assigned, so you can compare prompt strategies on quality rather than clicks. For each variant over the trailing 30 days you see conversations, assistant messages, average confidence, positive rate, escalation rate, and feedback counts.
- Positive rate — good feedback divided by total feedback.
- Escalation rate — wrong + incomplete + needs-handoff feedback divided by total feedback.
- Coverage — the share of all conversations that were tagged with a prompt variant.
A winner is only declared when the sample is large enough; otherwise the page reads “No winner yet” with the reason. See System prompt for how variants are authored.
Widget theme A/B
The theme test compares conversion by served widget appearance over 30 days. Variant A is your saved theme (the control); Variant B is the preset chosen in Widget Designer→ Targeting. Assignment is stable per shopper. The table shows shoppers, conversions, conversion rate, and lift versus control, with a confidence badge once a variant clears a minimum of 50 assignments. Until then it reads “Gathering data”.
Reading the results well
- Change one thing at a time. If you swap the opening message and the theme together, you can’t attribute the lift.
- Look at the outcome that matters for the change — opening-message tests should be judged on add-to-cart and leads, not just chat opens.
- Lift can be negative. A red lift means B is underperforming A; pause B rather than waiting for it to recover.
- “Leading” and “Winner” badges are signals, not commands — promote a variant manually by making it the saved default.
Upcoming experiments
Split tests for discount nudges, bundle recommendations, and post-purchase upsells are in progress. Experiment ideas are welcome — drop a note in Support.
Related
- Widget Designer — where the theme variant is configured.
- Analytics — store-wide conversion and attribution.
- Recovery — outcomes the experiments feed into.