AI Bot QA Analyzer

---
name: ai-bot-qa-analyzer
description: "Run a QA analysis on conversations handled by your AI chatbot via Rippit, starting from a Rippit worksheet URL or a list of conversation IDs. Enriches each conversation across the classification columns you define — whether it matches your primary issue category, whether a human agent participated, and what the conversation was actually about when the bot misfired — judging ONLY from the customer's own quotes, never from the bot's tags or metadata. Then reads aggregate counts and publishes an interactive HTML QA dashboard with donuts, bar charts, mismatch stats and a drill-down table. Use whenever the user pastes a worksheet URL or asks to run a bot-accuracy / routing-accuracy QA check through Rippit."
---
# AI Bot QA Analyzer

Runs a multi-column AI enrichment on a set of conversations that were handled by
your AI chatbot, then publishes a QA dashboard. Classifications are based **only
on customer quotes** — never on the bot's own tags or metadata — to validate
whether the bot's routing and escalation were accurate.

---

## First run — make it yours

**Look first.** Before asking anything, inspect what the user actually has:
- Call `list_data_sources` to see the connected sources.
- For the source most likely to hold conversation transcripts, call
  `describe_table` and note which columns could serve as the conversation ID,
  the transcript/body text, the author/sender field (to tell the bot apart from
  human agents), and any timestamp.
- If no source has readable transcript text (e.g. an ID-only CSV upload), say so
  plainly and ask whether to rebuild a worksheet from the main transcript table.

**Then ask, one question at a time, leading with what you found:**

1. **{{data_source}}** — "Point me at the conversations to QA. I found <sources> in
   your workspace; <best guess> looks like the transcript table — use that, or
   give me a worksheet URL or list of conversation IDs?" If the pointed-at
   worksheet has no enrichable transcript text, offer to build one from the main
   transcript table filtered to the supplied IDs.
2. **{{bot_identity}}** — "How do I recognise your AI bot in a transcript so I can
   separate it from human agents? A sender name, an admin ID, or an email works."
3. **{{primary_category}}** — "What is the primary issue category you want to
   validate the bot's routing against, and which customer-language signals define
   it? Enrichment 1 flags each conversation Yes/No/Insufficient data on this."
4. **{{misfire_topics}}** — "When a conversation is NOT {{primary_category}}, what
   topic buckets should misfires be classified into? Give me an ordered list."
5. **{{drilldown_focus}}** (optional) — "Want an optional deeper drill-down on one
   misfire topic? If so, which topic and what pattern should I look for?"

After the interview, restate the filled-in bindings for confirmation, then run.
On later runs, reuse the established bindings unless the user asks to change them.

---

## Step 0 — Resolve the worksheet

From {{data_source}}, obtain a worksheet with transcripts:
- A worksheet URL of the form `.../workbook/<workbookId>?tab=<worksheetId>` →
  extract the `tab=` value as the worksheetId.
- A plain worksheetId string → use directly.
- An ID-only CSV upload (no transcripts) → read the IDs, then build a proper
  worksheet from the main transcript table:

```
read_table(tableId=<csvWorksheetId>, limit=1000)
  → collect all conversation IDs from the ID column
create_worksheet(
  tableId=<main_transcript_table_id>,
  title="Bot QA – Full Transcript [date]",
  where=[{columnId: <id_column>, operator: "IN", value: [...ids]}]
)
  → use the new worksheetId for all enrichments below
```

---

## Step 1 — Column 1: Primary Category match?

```
enrich_worksheet(worksheetId=<id>, enrichments=[{
  name: "{{primary_category}} Issue?",
  outputType: {type: "enum", values: ["Yes", "No", "Insufficient data"]},
  instructions: """
    Read the full conversation transcript — chats handled by the AI bot.
    Determine, based ONLY on what the CUSTOMER wrote (not tags, labels, or bot
    actions), whether the customer's issue is genuinely about {{primary_category}},
    using the customer-language signals defined for it.
    - 'Yes' — the customer's own words clearly indicate this category.
    - 'No' — the customer's own words clearly indicate something else.
    - 'Insufficient data' — messages too short, ambiguous, or only button clicks.
  """
}])
```

Poll `get_enrich_status` until `done: true` before proceeding.

---

## Step 2 — Column 2: Agent Participated?

```
enrich_worksheet(worksheetId=<id>, enrichments=[{
  name: "Agent Participated?",
  outputType: {type: "enum", values: ["Yes", "No", "Insufficient data"]},
  instructions: """
    Determine whether a human agent (any person who is NOT the AI bot) actually
    sent a message in this conversation. The AI bot is identified by
    {{bot_identity}}. If at least one human teammate sent a visible reply,
    return 'Yes'. If only the bot and the customer participated, return 'No'.
    If unclear, 'Insufficient data'.
  """
}])
```

Poll until done.

---

## Step 3 — Column 3: Actual Topic (misfires only)

Only classifies rows where the primary-category column is 'No'.

```
enrich_worksheet(worksheetId=<id>, enrichments=[{
  name: "Actual Topic",
  outputType: {type: "enum", values: [ ...{{misfire_topics}}, "N/A" ]},
  instructions: """
    IMPORTANT: Only classify if the primary-category column is 'No'. For all
    other rows return 'N/A'.
    For 'No' cases: read the customer's own messages and determine which of the
    buckets in {{misfire_topics}} the conversation is actually about. Return
    'N/A' if the primary category was confirmed or data is insufficient.
  """
}])
```

Poll until done.

---

## Step 4 — Optional drill-down

Run only if the user asked for {{drilldown_focus}}, or if that topic is the
dominant misfire category. Scopes to Actual Topic = {{drilldown_focus}} rows.

```
enrich_worksheet(worksheetId=<id>, enrichments=[{
  name: "Drill-down flag",
  outputType: {type: "enum", values: ["Yes", "No", "N/A"]},
  instructions: """
    Only classify rows where 'Actual Topic' is {{drilldown_focus}}; others 'N/A'.
    For those rows, read ONLY the customer's messages before any human agent
    joined and flag the specific pattern the user defined for this drill-down.
  """
}])
```

---

## Step 5 — Read aggregates and build dashboard

```
aggregate_table(
  tableId=<worksheetId>,
  groupBy=[{columnId: "<primary_category_col>"}, {columnId: "<agent_col>"}],
  select=[{agg: "count", alias: "count", columnId: "<primary_category_col>"}]
)
```

Repeat for Actual Topic and the drill-down column if run. Rippit returns column
IDs like `Name__XXXXXX` — use those verbatim in aggregate calls.

Then publish an HTML dashboard (see Dashboard Spec).

---

## Dashboard Spec

Publish an interactive HTML artifact the user reads in the chat, with:

**Header**: worksheet name, source, date, method note ("Customer quotes only — not bot tags").

**Hero stats row**: primary-category Yes count (green), No count (red),
Insufficient data (amber), Agent participated (blue), and tag mismatches = rows
where primary-category ≠ Yes (red).

**Cards (grid)**:
1. Primary-category donut — Yes / No / Insufficient data
2. Agent Participated? donut
3. Tag Accuracy bar chart — Correct vs Mismatch with insight callout
4. Actual Topic bar chart for misfires (if Step 3 run)
5. Drill-down donut (if Step 4 run)
6. Full summary table with all columns

**Design tokens** (dark theme):
```
--bg: #0d1117   --surface: #161b22   --border: #30363d
--yes: #3fb950  --no: #f85149        --insuf: #d29922
--agent: #79c0ff --accent: #388bfd   --muted: #8b949e
```

Use Chart.js from cdnjs for donuts. Bars as pure CSS flex. No external images.

---

## Error handling

- **Enrichment fails immediately**: worksheet likely has no transcripts (ID-only
  upload). Rebuild from the main transcript table (Step 0).
- **Partial ID matches**: some IDs may not match (deleted/not synced). Note the
  count delta in the dashboard subtitle.
- **`Insufficient data` dominates**: conversations may be short menu-click flows.
  Flag in the dashboard insight.
- **All Agent Participated = Yes**: may be normal if the bot always escalates.
  Note it as expected behaviour.