Chat of the Day — Behavior Discovery

# Chat of the Day — Weekly Behavior Discovery

## First run — make it yours

Before doing any work, interview the user one question at a time, then restate the filled-in bindings for confirmation before running. On later runs, reuse these bindings unless the user asks to change them.

1. "Which Slack channel holds the hand-picked QA examples I should learn the good and bad behavior patterns from?" ({{training_channel}})
2. "What behavior categories does that channel use, and where is the pinned message defining them that I should read before each run?" ({{behavior_categories}})
3. "Where should I keep the living criteria doc that stores behavior definitions, the last-run timestamp, and already-surfaced conversation IDs?" ({{knowledge_doc}})
4. "Which Rippit table holds the conversations I should scan each week?" ({{source_table}})
5. "How should I scope which agents/teams to include — a resolvable list of agent IDs, or scan unfiltered and note the caveat?" ({{scope_filter}})
6. "How many good and how many bad conversations should each weekly sample contain at most?" ({{sample_size}})
7. "Who should receive the weekly sample, and as what — a direct Slack DM, a channel post, or something else?" ({{delivery_destination}})

## The actual point of this skill (read this before anything else)

The {{training_channel}} posts are training data: real examples QA has already hand-picked to teach a behavior pattern. The deliverable is never the channel posts themselves — it is **new** conversations from Rippit, found this week, that show those same learned behaviors, so QA has fresh material without manually surfing live chats at scale.

If a run ends with "here are the channel posts, verified" and nothing new pulled from Rippit, that run failed at the one thing it exists to do.

## The behavior categories

Read the pinned definition message ({{behavior_categories}}) in {{training_channel}} yourself before each run, since it may be edited over time. Treat these as behavior categories, not just labels — extract the underlying *behavior* each is pointing at (e.g. "agent proactively investigated beyond the literal question" or "agent missed an earlier request buried in a long thread"), not just post titles.

## Message shape in the channel

Each post is typically a thread: the parent is a title with an emoji, the real content is in the first reply. Read the thread itself with the thread-read tool, since a channel-level history dump can collapse Slack's markdown link syntax (`<url|text>`) and hide links. Positive examples typically follow `Link:` / `What happened:` / `Why it matters:` plus a shoutout; anonymous ones follow `Situation:` / `What went wrong:` / `What good looks like instead:` with no link or name (label wording varies).

## Step 1: Build or refresh the behavior definitions

Read every relevant thread in {{training_channel}} (or since the last refresh — check {{knowledge_doc}} for what's already incorporated). For each, extract not just its category label but the concrete behavior it teaches: what the agent did or didn't do, and why it mattered. Group similar behaviors across posts rather than treating every post as its own rule.

Write these as instructions detailed enough to hand to someone who has never seen the channel and have them correctly judge a new, unrelated conversation — this is what you'll use as enrichment instructions in Step 3. Include concrete language pulled from the real posts, not invented examples. Run this fresh every run so definitions keep sharpening.

## Step 2: Maintain the criteria doc

Keep one living doc in {{knowledge_doc}} holding: the current behavior definitions in detailed enrichment form; the timestamp of the last successful run; the list of Rippit conversation IDs already surfaced in a previous week's deliverable (so Step 4 can exclude them); and a running note of edge cases and judgment calls. Update this doc *after* the run completes, so a failed or partial run doesn't corrupt the record.

## Step 3: Build the week's candidate pool in Rippit

Source table: {{source_table}}. Scope: apply {{scope_filter}}. If the desired grouping is a Rippit-UI-level group filter the MCP connector cannot resolve, don't fake it — scan the table unfiltered by group and note the limitation in your working notes only. If a resolvable list of agent IDs or a working filter becomes available, use it and drop the caveat.

Create a worksheet (`create_worksheet`) of the current week's conversations with these filters:

- `searchable_conversation_source_type` = `conversation` only. Exclude `email`, `admin_initiated`, and `sms` — non-chat sources massively inflate the pool with rows that have no agent chat behavior to evaluate. If a run sees a materially different count than expected, investigate rather than assuming it's fine.
- `team_assigned NOT EMPTY` — exclude conversations with no assigned team (automated/bot/system rows with no agent behavior).
- Filter on `first_seen` or `last_seen` for the week's date range, and exclude external_id values already recorded as surfaced in {{knowledge_doc}}.

Closure, reopening, retention, and bonus/free-spin request conversations are in scope for both good and bad sampling — don't exclude by category. Judge them on:
1. **Voice and tone** — respectful, patient, non-pushy? Pressuring a player who wants to leave is a real issue regardless of the outcome.
2. **Other unresolved requests** — did the player raise anything else (a pending withdrawal, deposit issue, game problem, unexplained charge) the agent didn't address? These should be handled regardless of the closure/retention outcome.

Never judge these on whether the account ended up closed or open. They remain eligible for the good side when the handling genuinely resembles what the channel itself has praised — match against the real channel examples, not a generic notion. If it's genuinely unclear whether a scenario reads good, bad, or neither, ask the user rather than guessing.

## Step 4: Enrich the worksheet against the learned behaviors

Use `enrich_worksheet` to add one AI column per learned behavior pattern (or a small number of well-designed columns — all columns in one call are filled in a single LLM read per row, so batch everything). Each column's instructions should be the detailed behavior description from Step 1, asking the model to judge whether the transcript shows that behavior with a real justification, not just a label.

This is the mechanism for finding similar behavior at scale — don't fall back to reading a handful of conversations yourself and generalizing; that's a sampling error that defeats the purpose.

## Step 5: Pick and verify the top candidates

Read back the enriched worksheet (`read_table` / `aggregate_table`) and pick the strongest matches per behavior category. Narrow to a shortlist per side with some margin above {{sample_size}} so weak matches can be dropped.

Fan sub-agents out over the shortlist, one per candidate, each given the conversation's external_id, the matched behavior, and the enrichment's justification. Instruct each to pull the full transcript with `read_conversations` and confirm the justification holds against the real conversation, and to check Slack for related context. Absence of a Slack hit is normal, not a failure.

## Step 6: Manager review (you, not a sub-agent)

Read every returned case. Reject anything where the transcript doesn't support the matched behavior, or where the match is technically correct but weak/generic. Apply the closure/retention judgment from Step 3 here too. Never pad to hit {{sample_size}} on either side — report exactly how many genuinely hold up. Cross-check against the doc's behavior definitions so the bar stays consistent week to week. The bar: would this actually be worth someone's attention if they saw it.

## Step 7: Deliver the weekly sample

Deliver to {{delivery_destination}} (`slack_send_message`) with the sample itself, written to be read directly in Slack — no attached file, no HTML, no separate report artifact. Keep it short and direct:

1. A short summary — how many good and how many bad were found this run, and anything genuinely worth flagging up front (e.g. a real compliance/RG concern, or a pattern showing up more than usual).
2. **Good chats** — each with enough context to understand what happened and why it's a good example (what the agent did, the behavior it matches, the conversation link), written the way a person would explain it to a colleague.
3. **Bad chats** — same structure and context, including the conversation link.

Do not include verification proof, sub-agent methodology, filter details, or row counts in the deliverable — that belongs in your own working notes. If either side has fewer than {{sample_size}} genuine matches, just say so in one line. Then update {{knowledge_doc}} with the newly surfaced conversation IDs and this run's timestamp.

## Things to never do here

- Never judge a closure/reopening/retention/bonus chat as good or bad based on the closure/retention outcome — judge on voice and tone and whether other player requests were left unaddressed.
- Never pull from non-chat sources (email, admin_initiated, sms).
- Never treat the channel posts as the deliverable to verify, rank, or fact-check — they are training signal only.
- Never guess or fake an unresolvable group filter — state the scope limitation in your working notes only.
- Never resurface a conversation already delivered in a previous week — check the surfaced-ID list first.
- Never generalize a behavior pattern by reading a handful of conversations yourself instead of enriching the full weekly pool.
- Never include customer PII beyond what's needed to identify and verify a case, and never include verification/methodology detail, in the final delivered message.