Training & gaps inbox

Every store has questions the AI can’t confidently answer on day one. The gaps inbox is the workflow that closes those holes. Zubby flags any conversation where the AI hit low retrieval confidence, escalated, or got a thumbs-down — then drafts a fix and queues it for your review. Accept it, edit it, or reject it; over a few weeks your AI gets meaningfully smarter.

What lands in the inbox

• Low-confidence answers — the AI gave a reply but flagged itself uncertain (retrieval score below threshold, no citations, or hedging language detected).
• Explicit escalations — the AI handed off because a question was out of scope.
• Negative shopper feedback — thumbs-down or frustrated language detected in the next turn.
• Repeated similar questions — when the same shape of question shows up across multiple conversations, the inbox groups them so you can fix once.

The triage workflow

Open Dashboard → AI → Training. Each row is a gap with:

1. The original shopper question and the AI’s reply.
2. A draft fix — usually a new knowledge-base entry, a tweak to the system prompt, or a “do not handle” rule that routes to a human.
3. Evidence — the citations the AI was working from (or the lack thereof) and which retrieval rank they landed at.
4. Three actions: Accept, Edit & accept, Reject.

Auto-promotion

Some suggestions are obviously correct — short, factual, well-sourced. Toggle Auto-promote under Training → Settings to let Zubby publish those automatically. The AI watches its own confidence: it only auto-promotes a suggestion when:

• The drafted KB entry contains a verbatim quote from an existing authoritative source (your shipping policy page, an official FAQ).
• The fix has been seen 3+ times across distinct conversations.
• No reviewer rejected a similar suggestion in the past 30 days.

Auto-promoted entries are logged with a banner so you can audit them. You can always demote one back to the inbox with a click.

Coverage metric

The number that matters: retrieval coverage, the share of questions answered with at least one citation above the confidence threshold. Healthy stores sit at 85-95%. Below 70%, the gaps inbox is probably backed up — open it and clear the top ten items.

Confidence threshold

Default threshold is a cosine similarity of 0.55. Lower it (e.g. 0.45) to make the AI more willing to answer; raise it (e.g. 0.65) to make it more cautious. For regulated verticals (supplements, finance), we recommend 0.7+ with mandatory citations.

Best-in-class workflow

The teams that get the most out of training do four things:

1. Triage the inbox once a day for the first two weeks. After that, a weekly cadence is enough.
2. Accept or edit, rarely reject. Even a partial fix improves coverage.
3. Use the “Replay last 50 conversations” button after big changes to see real impact.
4. Turn on auto-promotion after the first 50 manual accepts — by then the AI has learned which kinds of patches you typically approve.

Adjacent

• Knowledge base
• Human handoff
• Analytics & weekly review