A/B testing and experiment results

The A/B Tests page (also called the Experiment Lab) is where you measure the impact of changes you make to the storefront experience. It includes opening-message split tests, system-prompt variant quality tracking and widget theme A/B results. Their outcome definitions and decision rules differ; a badge is a decision aid, not proof of a causal effect or guaranteed future performance.

Split tests, not auto-reallocating bandits

Despite the page name, the current engine runs fixed-split A/B tests, not multi-armed bandits. Traffic is divided deterministically and stays put - Zubby surfaces the leading variant and an observed-signal label, but it never shifts traffic toward the winner mid-test. The one exception is the opt-in auto-promote described below, which ends the test outright when the implemented threshold is met. Adaptive bandit allocation is on the roadmap.

How variants are assigned

Assignment is deterministic by session id. A given shopper sees the same variant while that session identity persists. Clearing browser storage, changing devices or starting a new session can change assignment; this is not a permanent customer-level allocation. Two-arm opening-message tests split traffic according to the traffic slider (50/50 by default); adding a Variant C switches to an equal three-way split. Theme tests use their configured split; prompt variants use administrator-configured weights.

Opening-message experiment

The headline experiment splits new shoppers across two or three opening greetings and tracks what happens next. Author it right on the page:

Variant A and Variant B - two messages, up to 500 characters each. Every variant must have different copy.
Variant C (optional) - a third arm. Leave it empty for a classic A/B test; fill it in when you have two genuinely different challenger ideas and enough traffic to feed three arms. With C present the split is locked to an equal three-way division - each arm gets a third of the traffic. Each comparison needs adequate data, and B-vs-A and C-vs-A create multiple-comparison concerns. Duration depends on traffic, baseline rate, effect size and the decision plan; no fixed percentage increase in runtime is guaranteed.
Traffic to Variant B - a slider from 10% to 90% (two-arm tests only; disabled when C is set). Use 10–20% for a cautious rollout of reviewed copy. Never expose unsupported claims or unapproved discounts, even to a small audience. 50% converges fastest, because statistical power is maximized when both arms accumulate samples at the same rate.
Run on the storefront widget - a toggle that enables or pauses the experiment. When paused, past data stays visible for reference but no new assignments are made.
Auto-promote the winner - an opt-in checkbox, covered below.

For each variant you see assignments and four downstream rates: chat open, add to cart, purchased, and lead captured. A comparison table below shows, per outcome (opened chat, added to cart, purchased, captured lead, requested handoff), Variant A’s rate, the challenger’s rate, the lift over A, and an observed-signal badge. With a Variant C in play you get two tables - B vs A and C vs A - each judged against the control independently.

The Purchased (AI-attributed) outcome

Purchased counts sessions whose AI recommendations reached a completed purchase - it is attributed, not store-wide revenue. It is the outcome closest to money, but also the sparsest: most stores see far fewer purchases than add-to-carts, so purchase estimates may be especially uncertain. Read it as a guardrail on a result, and judge the running test on add-to-cart (the closest well-sampled proxy to revenue) - which is also the metric the recommended-winner badge and auto-promote use.

Changing copy starts a new test

Every test records when it started, and all stats on the page cover the current test only. Editing any variant’s copy archives the running test’s results into history and resets the stats window. This is deliberate: sessions assigned under the old copy would otherwise contaminate the new test’s numbers - shoppers who saw last week’s greeting have nothing to say about this week’s, and mixing them produces confident-looking nonsense. Re-enabling a paused test also restarts the window. Either way, metrics look back at most 90 days, so a long-forgotten test cannot drag ancient data into today’s decision.

Sample-size guidance

While a test runs, the page shows a rough fixed-sample planning estimate based on the currently observed add-to-cart gap between A and B, using a conventional 5% two-sided alpha and approximately 80% power formula. It also shows how many more assignments that means at current traffic. The observed gap can change; this estimate does not validate early stopping or establish that the effect is real. Small real differences need a lot of traffic, and if the estimate says thousands of sessions on a store doing dozens a day, consider testing bolder copy - bigger differences are cheaper to detect.

Auto-promote

Tick Auto-promote at |z| ≥ 2.33 and Zubby will end the test for you: when one arm leads on add-to-cart at an absolute pooled z-score of at least 2.33 and both arms have at least 200 sessions, the winner becomes your default greeting, the results are archived, and the experiment switches off - it fires at most once per test. This check runs when the A/B Tests page is rendered, not in a scheduled background job. It only arms on two-arm tests; three-arm promotion is manual. The 200-session floor and threshold are operational safeguards, not a validated sequential error-control method. Leave auto-promote off when you need to review purchases, guardrails and your predefined stopping plan before a change.

How the observed signal is labeled

The implementation calculates the pooled two-proportion z-score (challenger rate minus control rate, divided by the pooled standard error), then maps its absolute value to the labels below. These are exploratory score bands, not calibrated confidence levels or probabilities that a variant wins. The thresholds are unchanged; repeated looks, automatic stopping and multiple comparisons are not corrected.

Label	Meaning
Very strong observed signal	\|z\| ≥ 2.33
Strong observed signal	\|z\| ≥ 1.96
Moderate observed signal	\|z\| ≥ 1.64
Weak observed signal	below 1.64
Insufficient data	zero assignments or zero estimated standard error

Plan the decision before starting

Repeated checks, automatic stopping and multiple outcomes or challengers can inflate false-positive rates. The current implementation does not document a sequential or multiple-comparison correction. Predefine the primary outcome, meaningful effect size, sample or review horizon and safety guardrails. A minimum sample alone does not establish significance; obtain statistical review for consequential decisions.

Past tests: the archived history

Every opening-message test that ends is archived automatically - whether it ended because you promoted a winner, auto-promote fired, or you replaced the copy with a new test. The Past opening-message tests card lists the most recent twenty: when each test ended, which variant (if any) was promoted and why, and per-variant snapshots of the exact message copy, assignments, add-to-cart rate, and purchases. Use it to avoid re-running an idea that already lost, and to build an institutional memory of which greeting styles work for your audience - the archive keeps the receipts even after the widget config has moved on.

System-prompt variants

Zubby also tracks which system-prompt variant each shopper session was assigned, so you can compare prompt strategies on quality rather than clicks. For each variant over the trailing 30 days you see conversations, assistant messages, average confidence, positive rate, escalation rate, and feedback counts.

Positive rate - good feedback divided by total feedback.
Escalation rate - wrong + incomplete + needs-handoff feedback divided by total feedback.
Coverage - the share of all conversations that were tagged with a prompt variant.

Prompt variants are configured by platform administrators or deployment configuration, not authored in a merchant raw-prompt editor. The built-in fallback is a 90/10 control/challenger split; configured weights may differ. The merchant page reports observed variants. Its winner heuristic requires at least two variants with 100 conversations each and a score gap of 0.05 using positive feedback, escalation rate and model confidence. This is not the opening-message z-test or a significance guarantee. See System prompt for the supported merchant configuration controls.

Widget theme A/B

The theme test compares conversion by served widget appearance over 30 days. Variant A is your saved theme (the control); Variant B is the preset chosen in Widget Designer → Targeting. Assignment is stable while the session identity persists. The table shows shoppers, conversions, conversion rate, and lift versus control, with an observed-signal badge once a variant clears a minimum of 50 assignments. Until then it reads “Gathering data”.

Theme conversion window limitation

Assignments and conversion events must both fall within the trailing 30 days, and a conversion must occur at or after a matching assignment. The signal combines checkout, recovery-converted and AI add-to-cart events; it is not a purchase-only rate or proof of causal lift. Sessions with multiple theme assignments can appear in multiple variant rows. Review assignment stability and event coverage before choosing a theme.

Worked opening-message test (illustrative)

Use this when you want to compare two safe greetings while holding the catalog, offers and other widget settings steady. Do not use it as a test of discount policy, overall store profit or email recovery lift. You need workspace access; saving or promoting requires the settings capability. Opening-message tests are available on every plan; theme experiments require premium themes from Growth.

Open A/B Tests. Write A as “What are you shopping for today?” and B as “Need help choosing a size or comparing products?”. Both are synthetic examples to adapt to your store.
Leave Variant C empty, set Traffic to Variant B to 50%, and leave Auto-promote the winner off while you establish your review plan.
Set Run on the storefront widget and save. Verify both greetings in separate fresh test sessions, then confirm assignments appear on the results table.
At the predefined review point, compare add-to-cart per assigned session, check Purchased and handoffs, and sample conversations for misleading or unhelpful answers.
Use Make A/B the default greeting only after reviewing the evidence. Confirm the experiment ends, the greeting changes and a history entry preserves the copy and counts.

If assignments remain at zero, check that the experiment is enabled, the widget is installed and audience rules allow it, and that you are looking at the current test start. Editing copy or re-enabling a paused experiment resets its window. Missing purchases can reflect sparse data, identity coverage or recommendation attribution rather than no sales.

Reading the results well

Change one thing at a time. If you swap the opening message and the theme together, you can’t attribute the lift.
Look at the outcome that matters for the change - opening-message tests should be judged on add-to-cart (with Purchased as the sanity check), not just chat opens.
Lift can be negative. A red lift means the challenger is below A in the observed sample. Follow your predefined stopping and harm rules rather than ending whenever a noisy estimate turns red. The UI can recommend A when challengers cross its negative threshold.
“Recommended” and “Winner” badges are signals, not commands - unless you armed auto-promote, you end the test yourself with Make A/B/C the default greeting, which promotes that copy and archives the results.

Upcoming experiments

Split tests for discount nudges, bundle recommendations, and post-purchase upsells are in progress. Experiment ideas are welcome - drop a note in Support.

Widget Designer - where the theme variant is configured.
Analytics - store-wide conversion and attribution.
Recovery - outcomes the experiments feed into.