Review Queue

The Review Queue is a triage surface for recent AI conversations. Its default view shows up to 100 recent conversations; filter chips narrow that list to low satisfaction, negative or frustrated sentiment, low confidence, or human escalation. Open the transcript before flagging: a flag records a concrete review signal, but it is not an automatic fix and does not guarantee that a future answer will improve.

Where it lives

Reviews → Moderation queue - filter chips with live counts, and a card per conversation with its quality badges.
Each card’s Open button jumps to the full transcript in Conversations, where you can also leave per-message feedback.

Old Review Queue bookmarks redirect here and preserve the selected filter. The same tab also shows PDP Q&A drafts: edit their question and answer against the product source, then approve or reject. Require approval keeps future drafts in review;Skip approval enables future auto-publication. Use the latter only after reviewing your catalog and accepting that review risk.

What lands in the queue

The queue shows the store’s most recent conversations (up to 100, newest first) and lets you cut them by four risk signals. Each chip shows a live count so you can see at a glance where the problems concentrate:

Filter	Criterion	Where the signal comes from
Low CSAT	Rating of 2/5 or below	The satisfaction survey shoppers see when a chat closes.
Negative sentiment	Labeled `negative` or `frustrated`	The enrichment classifier that scores every conversation after the fact. Frustrated is treated as explicitly negative.
Low confidence	Confidence score ≤ 0.40	The AI’s own self-assessment of how well-grounded its answers were. It is a triage signal, not calibrated proof that the answer was wrong or that a high-confidence answer is correct.
Escalated	Conversation was handed to a human	Any handoff, manual or automatic.

Each card carries the conversation’s AI-generated summary as its title plus badges for whichever signals apply - an Escalated badge, the CSAT score, the sentiment label, and the numeric confidence. A conversation can trip several filters at once; one that is simultaneously low-CSAT, negative, and escalated is exactly the one to read first.

Why these four signals

These signals can overlap and can be wrong. CSAT records shoppers who rate the chat; sentiment catches the ones who don’t fill in surveys; confidence helps select answers to inspect; escalations show human handoffs. None replaces reading the transcript and checking its source.

The review flow

Pick a filter. Start with Low CSAT - it is the smallest, highest-signal bucket. Move to Negative sentiment and Low confidence once it is clear.
Open the transcript. Read what the shopper asked and what the AI answered. Often the verdict takes ten seconds: the answer was wrong, incomplete, or should have gone to a human.
Review the exact message. Use per-message feedback in the transcript when the faulty answer is not the latest AI reply. Use Flag for retraining on the queue card only when that latest reply is the one you intend to mark wrong.

What flagging actually does

Flag for retraining is not a bookmark - it writes a concrete training signal. Zubby resolves the conversation’s most recent AI replyand records a wrong feedback entry on it, with a note marking it as flagged from the Review Queue. That is the exact same signal the per-message feedback form in the conversation view writes, so both paths feed one loop:

The flagged reply appears in the AI Improver inbox for correction - see Training & gaps inbox.
Negative feedback flows into reranking, demoting products the AI keeps recommending badly.
Aggregate volume shows up on the Overview AI Improver impact card, so you can see the loop working.

Flagging is idempotent per reviewer and message - clicking twice updates the same feedback row rather than double-counting - and every flag is written to the audit log with who flagged what.

Review before flagging

Flagging marks the latest assistant reply as wrong for the current reviewer. An incorrect flag can pollute quality and ranking signals, so read the transcript and use per-message feedback when the problem belongs to a different reply.

Why review sits after the conversation, not before

Zubby’s primary quality controls run in-line - the grounding guardrails, blocked-claims checks, and handoff triggers described in How the AI works can block or redirect defined risk cases before shoppers see them. They do not catch every incorrect or unhelpful reply. The Review Queue is the complementary after-the-fact layer: it catches the failures no pre-filter can, because they are only visible once the shopper has reacted - a technically-correct answer that frustrated someone, a confident-sounding reply built on thin retrieval. Blocking every AI reply on human approval would defeat the point of an instant agent; reviewing the scored aftermath keeps the speed and still closes the loop.

Gotchas

Sentiment, confidence, and summaries can arrive after the conversation itself. If new conversations lack badges, review Notifications and the support team and use the unfiltered Conversations view while enrichment catches up.
The queue reads the most recent 100 conversations per filter. It is a triage surface, not an archive - for historical digging use Conversations search.
“Queue is clear” under a specific filter does not mean all is well globally - check the counts on the other chips before celebrating.
The moderation queue requires both Training and Catalog permissions on your team role; see Team & roles.

Training & gaps inbox - where flagged replies get corrected.
Conversations overview - transcripts, scoring, and per-message feedback.
Human handoff - the escalations that feed the Escalated filter.
How the AI works - the in-line guardrails this queue backstops.