Skip to content
Back to the blog
Engineering

How to test catalog-grounded AI answers

How catalog retrieval, inventory checks and knowledge sources support answers, with a practical test plan and honest limits.

ZZubby AI engineering · EngineeringPublished Updated 2 min read

A shopper asking for a waterproof jacket under a particular budget needs current product facts, not a fluent guess. Retrieval brings store catalog and knowledge into the answer. It improves the evidence available to the model but does not guarantee that the final answer is correct.

Follow one question through retrieval

Take the sample request: Show an in-stock jacket under $200 for a wet commute. The system retrieves candidates from the connected store, then uses product metadata and the current shopper context to choose relevant results. A relevant title alone is insufficient: check the actual variant price, availability and the product description that supports the waterproof claim.

  • Keep every lookup scoped to the connected store. A similar product from another tenant must never become a candidate.
  • Use structured filters and current inventory checks where available; an embedding is not a live stock ledger.
  • Read merchant policies from the knowledge source instead of inventing shipping or refund terms.
  • Treat retrieved descriptions as untrusted content, not instructions that can override the assistant rules.

Test the failure paths as well as search quality

Build a question set containing exact product names, spelling mistakes, variant questions, conflicting policies and products with missing attributes. For each answer, record whether the recommendation is relevant, its price and stock agree with the source, and its link works. Include requests where no product meets the constraints. The useful response may be a clarification or an honest no-match result.

Freshness is a separate check

A successful connection does not prove every product has finished indexing. Review import and indexing status, then change a test product and verify that the updated facts reach a new conversation. Record sync timing separately from model response time. A fast answer using old inventory is still wrong.

Search and AI discovery limits

Readable pages and consistent facts also help people and search engines understand a store. Google says its AI search features need no special AI text file or schema. A feed, llms.txt file or MCP endpoint does not guarantee indexing, recommendations or rankings.

Stop reading. Start shipping.

Try Zubby on your own store.

Connect WooCommerce, let the catalog sync, and test the agent on your own products. New Shopify connections are coming soon; Custom API requires Pro or above. Eligible accounts get a 14-day Growth trial.