AutoQA Performance Insights

# QA Performance Insights

Turn your QA classifiers into three things a CX leader can act on in the same pass: where the program is losing the most points (weighted opportunities), what to do about each one and who owns it (coaching / training / KB / process / tooling), and which conversations are genuinely worth celebrating (wins). All three come from the same graded population, so do them together.

## First run — make it yours

Before doing any work, interview the user one question at a time, in QA terms, and explain what each answer drives. Then restate the filled-in bindings for confirmation and run. On later runs, reuse these bindings unless the user asks to change them.

1. **Data source & classifiers** — "Which table holds your graded conversations, and what are the labels of the QA classifiers I should score against?" This tells me which already-computed columns to read (I read them, I do not re-grade them). Sets {{data_source}}.
2. **Scoring rubric** — "What point weight does each classifier carry, and which one (if any) is an auto-fail gate that zeroes the whole ticket?" This drives the composite score. Sets {{scoring_rubric}}.
3. **Time window** — "What time window should I analyze?" Default last 30 days. Sets {{time_window}}.
4. **Scope** — "Should I limit the analysis to a specific team, queue, or channel, or look at everything?" Sets {{scope_filter}} (if set, filter to it).
5. **Wins count** — "How many wins would you like me to feature at the end?" Default 3. Sets {{wins_count}}.
6. **Delight signal** — "What customer-delight signal should qualify a win beyond a clean scorecard — sentiment, CSAT, or a predictive-CSAT value?" Sets {{wins_delight_signal}}.
7. **Recognition destination** — "If you want wins shared, where should recognition messages go, and should I DM-test before posting anywhere shared?" Sets {{recognition_destination}}.

## The rubric

Use {{scoring_rubric}}: each weighted classifier carries a point value, values read yes/no/n/a. Any auto-fail gate classifier reads pass/fail and forces the ticket's score to 0 when it fails, regardless of the others.

Scoring per ticket: if the gate is fail, score = 0. Otherwise score = 100 x (points earned from yes answers) / (points possible from yes+no answers) — n/a is removed from both sides, never scored as 0 and never used to drop the row. If all weighted metrics are n/a, report the ticket separately rather than averaging it as 0 or dropping it.

The composite score does not exist as a Rippit column — it is business logic over raw classifiers, so it is computed by the bundled `scripts/compute_autoqa_score.py`, never by hand and never by an enrichment pass. Read that script's docstring for exact edge cases.

## Analysis engine

Rippit MCP tools: list_data_sources, describe_table, describe_column, read_table, aggregate_table, pivot_table, create_worksheet, enrich_worksheet + get_enrich_status, read_conversations, get_report_guide. The classifiers are already-computed columns on the source table — do not re-enrich them; read them. Enrichment enters only later to explain why a ticket failed a classifier.

**Cost discipline:** classifier-level fail rates and single-classifier cross-tabs are cheap aggregate_table/pivot_table calls — do the opportunity ranking that way with no row pulls. Row-level pulls are only needed for the blended composite score (distribution and per-agent leaderboard).

## Step 1 — Find the table and confirm the classifiers

1. list_data_sources, then describe_table on the ticket/conversation table in {{data_source}}.
2. Match the classifiers named in {{scoring_rubric}} by label; confirm each one's distinct values match what is expected (pass/fail for the gate; yes/no/n/a for weighted). If a label is missing or its values do not match, stop and ask the user.
3. Record the id column, the agent column (an agent-array column shows id and name inline), a created/updated date column, a topic/issue-type column, and any CSAT/sentiment columns.
4. Resolve column ids fresh via describe_table this run rather than hardcoding.

## Step 2 — Confirm scope (light checkpoint)

Apply {{time_window}}, {{scope_filter}}, and {{wins_count}} from the interview. Then check grading coverage: aggregate_table a count of graded rows (gate classifier not empty) vs. total rows in the window, and state the coverage percentage up front so every finding is understood as about the graded sample.

## Step 3 — Rank opportunities (cheap, no row pulls)

For each weighted classifier, one aggregate_table grouped by that column gives yes/no/n/a counts. From those exact counts:
- fail_rate = no / (yes + no) (n/a excluded from denominator).
- weighted_impact = points x fail_rate — the real ranking, not raw fail rate. Surface explicitly whenever the raw-rate and weighted rankings disagree.
- The gate classifier gets its own aggregate_table (pass/fail), reported separately and first as a critical/auto-fail rate, not folded into the weighted ranking.

pivot_table (rows = topic or agent, columns = the classifier's values) gets the same fail-rate cut by topic or agent with no row pulls — use it to spot where each opportunity concentrates.

## Step 4 — Compute the blended score (needs row pulls — only for this)

1. create_worksheet scoped to {{time_window}} and {{scope_filter}}, filtered to graded rows, narrowed to needed columns (ids, agent, date, classifiers, topic, CSAT/sentiment). Confirm rows landed with a small read_table (hydration is async).
2. read_table in pages (up to 1000 rows/call), appending each page to a local CSV/JSONL file.
3. Run scripts/compute_autoqa_score.py against the accumulated file, mapping resolved column ids to its flags and passing agent/topic columns. Read its docstring first.
4. Cross-check: the script's implied yes/no/n/a counts should match Step 3's aggregate_table numbers on the same slice. If not, fix the pull's scope, do not distrust the rubric.
5. Use per-agent output for the coaching leaderboard, per-topic as secondary confirmation, and the distribution for the headline and trend.

## Step 5 — Drill into root causes for the top opportunities

For the top 2-3 opportunities by weighted impact (offer full coverage), build a child worksheet filtered to that classifier's fail/no rows and enrich_worksheet (one batched call per classifier) with:
- root_cause_category — an enum (with a catch-all like unclear) naming the failure pattern. Draft starting categories per classifier only after checking a handful of real examples; pilot on ~50-75 rows first if the failing population is large.
- coaching_note — string: one actionable sentence a coach could hand an agent.
- evidence_quote — string: the verbatim line showing the failure (empty if none stands out).

Prefer the full failing population if under ~2,000-3,000 rows; sample representatively above that and say so.

**Route each root-cause cluster to an owner** after aggregating root_cause_category (on the aggregate, not per row):
- Concentrated in one or two agents -> individual coaching.
- Spread across many agents on the same category -> team training.
- Inherently about missing/unclear documentation -> KB/script update (if a help center is connected).
- Policy or process ambiguity -> process fix.
- System/tooling limitation -> escalate to product/ops.

Every cluster you report needs a named fix type and an owner, not just a percentage.

## Step 6 — Surface the wins

Build the candidate pool as: gate = pass AND all weighted metrics = yes (a plain where filter), cross-referenced with the delight signal in {{wins_delight_signal}} if those columns exist. From that qualified pool, apply your recognition ranking and delivery steps — quote verification against the raw transcript, DM-test-first, and never post to a shared channel without confirmation. Send to {{recognition_destination}} only per those guardrails. This step only changes which population feeds the ranking (QA-qualified tickets).

## Step 7 — Deliver the report

Call get_report_guide first. Lead with the coverage caveat, then the headline (overall/blended score, critical-fail rate, trend if supported), the weighted opportunity ranking, the routed action table (cluster -> root cause -> fix type -> owner), and the wins highlights. Render the score distribution and opportunity ranking as visuals per the report guide. Link the worksheet(s).

## Step 8 — Offer to make it recurring

Once the user is happy, offer a cadence (weekly/monthly digest for the ranking; whatever cadence they set for wins). A recurring run can't do Step 2's interactive checkpoint, so lock in the resolved window (relative to run date), scope, and any recognition destination as fixed config, and re-run Steps 1-7 fresh each period rather than reusing stale worksheets.

## Principles

- The gate classifier is a gate, not a weighted metric — report its fail rate separately and first.
- N/A is excluded, never a 0 and never a dropped row.
- Never hand-compute the composite score; use the script and cross-check its totals.
- Weighted impact beats raw fail rate for ranking; say so when they disagree.
- Ground every opportunity and win in real evidence, not just a percentage.
- Every routed action needs a fix type and a named owner.
- Wins require genuine customer delight, verified against transcript and sentiment/CSAT.
- State the grading-coverage caveat up front.
- Never post recognition to a shared channel without the DM-test-then-confirm guardrail.