Conversation Escalation Review

# Conversation Escalation Review

## First run — make it yours

Before doing any work, interview the user one question at a time to establish the run bindings, then restate them for confirmation before running. On later runs, reuse these bindings unless the user asks to change them.

1. What kind of entity do you want each analyst to review in isolation — agents, QA graders, or queues/categories? ({{entity_type}})
2. Should I review all active entities of that type, or a named subset? ({{entity_scope}})
3. What time window should the review cover? Give a date range or a clear relative range like 'last 7 days'. ({{time_window}})
4. Which Rippit table holds the conversation data I should read from? ({{data_source}})
5. Which Slack channels should the deep dive search for existing escalations? ({{slack_channels}})
6. Which Jira project should I search for existing tickets on flagged themes? ({{jira_project}})

After collecting answers, restate the filled-in bindings and confirm before fetching anything.

## What this is and why it's shaped this way

This is a fan-out / stop-gate / synthesize pipeline. One analyst reviews each entity in isolation, a manager pass reads all findings together to spot patterns no single analyst could see, and only the patterns worth investigating get a deeper look — checking whether they're already known via Slack escalation threads or Jira tickets. The stop-gate exists on purpose: it stops the pipeline from either drowning the user in N separate reports or from digging deep on every issue when only a few are cross-cutting.

Do not skip the stop-gate and go straight from Level 1 to a report. The whole value is catching the theme that shows up in several entities' findings that no single analyst would flag on its own.

## Engine: Agent-tool loop

If no Workflow scripting tool is available, every run uses the Agent tool directly: you (the manager) spawn Level-1 analyst agents in parallel batches, read their compact findings yourself, run the stop-gate as your own judgment call, then spawn Level-2 deep-dive agents the same way. Batch parallel Agent calls in groups of roughly 6-8 per message. If the entity list is larger, run additional batches rather than one call per entity in series.

## Step 0 — Nail down the run parameters

Confirm from the first-run interview: {{entity_type}}, {{entity_scope}}, and {{time_window}}. Never default the time window silently — a review run on the wrong week is worse than asking one question.

## Step 1 — Fetch the entity list fresh, every time

Never hardcode a list of entities — rosters and queue definitions change. At runtime:

1. `list_data_sources` to find {{data_source}} (the conversation/entity table).
2. `describe_table` on it to confirm the entity-identifying column and the conversation-date column.
3. `aggregate_table` grouped by that entity column, filtered to {{time_window}}, to get the live distinct entity list plus a conversation count per entity.
4. If {{entity_scope}} names a subset, filter to just those; otherwise keep all entities with at least a handful of conversations in the window.

Record `M` = total entities in scope for the coverage note ("N of M analyzed").

## Step 2 — Give each analyst a stable data slice

For each entity, call `create_worksheet` on the source table filtered to that entity + {{time_window}} (a `where` clause on the entity column and the date range). This gives each Level-1 analyst a fixed, reproducible set of rows to read via `read_table`.

## Step 3 — Fan-out: Level 1, one analyst per entity

Spawn one Agent (`subagent_type: "general-purpose"`) per entity. Every analyst gets the identical prompt template, with only the entity name and worksheet id swapped in. Prompt drift ruins the manager's ability to compare findings apples-to-apples.

### Level-1 analyst prompt template

```
You are reviewing conversations for {{entity_type}} "{entity_name}" over {{time_window}}.

Data: worksheet {worksheet_id} in Rippit (use read_table to pull the rows; each row is one conversation). Read the actual conversation content, not just metadata, wherever available.

Answer these six questions and return ONLY the structured JSON below — no prose outside it:

1. Volume: how many conversations are in scope, and does that look normal or unusual?
2. Common trends/themes: what topics recur across multiple conversations? For each, note how many conversations it appeared in and 2-3 example conversation IDs.
3. Issue resolution: for each notable issue, how was it actually resolved? If an internal back-office link or reference appears, note that one was referenced — do not open it or paste its URL.
4. Escalation signal: does anything suggest it was flagged or escalated internally (mentions of escalation, tier levels, team lead, PO)?
5. Jira signal: is there a Jira ticket number or reference visible anywhere?
6. Outliers: is there a single conversation odd or severe enough to flag on its own?

If you can't access some conversations or the worksheet is empty, say so in data_gaps — never guess or pad the numbers.
```

### Level-1 structured output contract

```json
{
  "entity_id": "",
  "entity_type": "agent|grader|queue",
  "conversations_in_scope": 0,
  "conversations_analyzed": 0,
  "common_themes": [{"theme": "", "frequency": 0, "example_conversation_ids": []}],
  "notable_issues": [{"issue": "", "resolution": "", "escalation_signal": false, "escalation_detail": "", "jira_reference": ""}],
  "outliers": [{"conversation_id": "", "description": ""}],
  "data_gaps": ""
}
```

## Step 4 — Stop-gate: the manager review (you, inline)

Once all Level-1 analysts return, read every finding plus the set as a whole. Nothing goes back to the user yet. Look for: a theme appearing in 2+ entities; a finding that contradicts expectation; a metric or theme outlier; a single-entity anomaly severe enough to matter alone. Rank what you find and cap it at 3-5 themes for Level 2. If nothing clears the bar, say so plainly rather than manufacturing a theme. Track coverage as "N of M entities analyzed".

## Step 5 — Level 2: Slack + Jira escalation cross-check

For each theme that cleared the stop-gate, or any single issue the user names directly, launch a fresh deep-dive Agent with no memory of Level 1, so the directive must be fully self-contained.

### Deep-dive directive template

```
Investigate whether this issue is already known to the business:

Theme: {theme description}
Evidence: seen in {entities_involved}, e.g. conversations {example_conversation_ids}
Question: was this already escalated in Slack, and/or does it already have a Jira ticket? If yes, what's the current status? If no, say so — this may be net-new.

Slack: resolve channel names first with slack_search_channels (exact naming may differ) against {{slack_channels}}. Then search those channels for threads matching keywords from the theme, within {{time_window}} plus a reasonable buffer.

Jira: search project {{jira_project}} using jira-search with JQL like `project = {{jira_project}} AND text ~ "<keywords>"`, sorted by updated date descending. Pull the issue key, status, and summary for genuine matches only.

Return ONLY the structured JSON below.
```

### Level-2 structured output contract

```json
{
  "theme": "",
  "entities_involved": [],
  "slack_findings": {"found": false, "channel": "", "thread_summary": "", "date": ""},
  "jira_findings": {"found": false, "issue_keys": [], "status": "", "summary": ""},
  "conclusion": "already tracked | partially tracked | net-new",
  "recommendation": ""
}
```

## Step 6 — Synthesize the final report (chat summary)

Merge Level 1 and Level 2 into one chat response, in prose (no invented names or numbers, flag anything ambiguous instead of guessing). Cover, in order: 1) Coverage — how many entities analyzed out of scope; 2) Per-entity summary — short and factual; 3) Cross-cutting themes — what the stop-gate surfaced and why; 4) Escalation cross-check results — already tracked in Slack/Jira or net-new, with actual references; 5) Open questions / recommended next steps. Never invent an entity name, conversation ID, Slack thread, or Jira ticket that wasn't returned by a tool call.

## Standalone escalation cross-check (skip the fan-out)

If the user just asks "was this already escalated?" or "does this have a Jira ticket?" about one specific issue, go straight to Step 5 and build the directive from what they describe.

## Security notes

- Reading Rippit, Slack, and Jira data for this analysis is read-only and needs no warning.
- Never put full customer PII, KYC document contents, or exact financial/transaction figures into the final report. Summarize or redact if they show up.
- If an internal back-office link appears in the source data, do not open it or paste it into the report without asking first.
- This skill only reads. Any write action (Jira comment, Slack post) needs the user's explicit go-ahead first, every time.

## First run: dry run on 2-3 entities

Before running the full fan-out for the first time (or after any change), run it on just 2-3 entities and show the Level-1 output shape and one Level-2 result before scaling. A flaw in the analyst prompt gets multiplied by every entity in the fan-out.