This test plan applies to existing authorized Shopify connections. New Shopify connections are coming soon; WooCommerce has a different linked-order reporting scope. An AI shopping assistant can help when a shopper has a question the product page does not resolve. It cannot repair an inaccurate catalog or prove its own value merely by appearing before an order. Start with a small set of buying decisions and measure the answers and resulting actions separately.
Build a test around real shopping decisions
For a sample footwear store, ask: Which shoes come in a wide fit under $150? A useful answer should match the size, price and available variant, explain the choice, and link to that product. This is an illustrative test, not a customer result. Include a sold-out size, an unclear request and a policy question so the assistant has to handle limits as well as successful recommendations.
- Check the named product and variant against the source store, including currency and stock.
- Open the product link and test add-to-cart on a test store. Do not assume a plausible answer means the cart is correct.
- Check that missing information leads to a clarification or human handoff rather than an invented claim.
Define each metric before comparing results
AI-influenced revenue gives last-touch credit when a purchased product was recommended to the same shopper within 30 minutes. It credits the full qualifying order, including items the assistant did not recommend. Revenue divided by the listed monthly plan price is a fee ratio, not profit-based ROI. Reconcile refunds, discounts and payment costs before discussing profit.
Interpret the holdout carefully
The optional holdout hides the chat assistant from a deterministic share of visitors. Standalone promotional surfaces remain visible. The current linked-order lift report requires Shopify cart tokens and enough observations in both arms; WooCommerce-only stores do not receive a lift figure. A display threshold is not proof of statistical significance. Record dates, consent coverage, traffic differences and other changes alongside the result.