Overview

Overview

The new Agent Performance Console moves measurement from a single reply to the whole conversation, because that's where the real answer lives: did the guest actually get what they came for. Every campaign your agents run now points at one clear goal, so instead of guessing whether things are working, you see exactly how each one is resolving tasks and how that stacks up against your peers.

Access the Console

There are multiple ways to access the Agent Performance Console:

• Access the Console directly here: agent2.satis.fi

• From the Launchpad, click here:

• From the Content Studio, click on the following sign:

Why the measure changed

Early chat products answered questions, so we measured replies. The Quality Response Index (QRI) counted messages received, replies sent, and how relevant and complete those replies were. Each message was scored on its own.

Agents should be resolving tasks, which are only identifiable if you analyze the entire conversation. The conversation is a better unit of analysis because the task exists across turns; the messages are individual steps used to complete it.

A message-level evaluation asks: "Was this response good?" A conversation-level evaluation asks the more important question: "Did the agent successfully accomplish what the user was trying to do?"

A guest asks about parking, then gate openings, then bag policy, and only the full exchange reveals the one thing they were trying to do: get into the building on time. The task is identifiable only at the conversation level, therefore the console scores conversations, not messages.

The standard: resolution

Every conversation is evaluated against what the guest came to do. Each one receives a single verdict.

Resolved

The guest got what they came for, at the level they needed, within what the agent is built to handle.

Partial

The need was within the agent's remit and the agent moved the guest specifically toward it, but stopped short of the completeness or specificity the guest wanted.

Unresolved

The need went unmet: a generic runaround, conflicting information, or no usable path, or the request is something the agent cannot or will not do.

Resolution accounts for business needs, not only the guest's preference. If a guest asks for a refund the venue does not offer, and the agent says so clearly, the conversation is Resolved. If the agent leaves the guest without a clear answer, it is Unresolved.

Campaigns

An Agent Campaign is a structured, AI-driven mission assigned to a single agent. It aligns that agent's behavior with a strategic business outcome, and it is defined by what the guest is trying to accomplish rather than by topic.

Each campaign is anchored by a task completion statement: the plain definition of what a finished task looks like. Every conversation in the campaign is measured against that same bar.

Example. The Guest Experience agent's "Plan the Visit" campaign covers guests preparing for game day. It succeeds when their entry, policy, seating, and wayfinding questions are resolved with the right information.

Task completion rate=conversations resolved÷conversations attempted(resolved includes partially resolved)

Objectives and Key Results

OKRs organize what an agent should achieve and how it is measured.

Objective

The qualitative business outcome.

Key Result

One campaign. Scored 0 to 1.

Objective Score

Average of its Key Result scores.

The Key Result Score

The score is produced in two steps. First, measure the campaign's task completion rate. Second, score that rate against the goal for the campaign.

A score (0 to 1) and a task completion rate (0 to 100%) are different units. The score places your rate against the goal for that campaign. It is not the rate itself, and 0.7 does not mean 70%.

0.7 is success. It marks performance on par with a strong campaign running the same job under the same conditions. The task completion rate required to earn a 0.7 is not the same across campaigns: a hard campaign such as ticket sales reaches 0.7 at a lower rate than an easier one.

Setting goals

Every campaign has a goal. We set it using historical results across clients running that same campaign, adjusted for task difficulty. Task completion rate is the primary input.

Goal pending

A newly created campaign shows this marker in place of a score until we have enough data across the cohort to set a reliable benchmark. A goal reflects how comparable programs perform at the same job, so it cannot be set from a single client's campaign. While a goal is pending, the campaign still tracks its task completion rate. It simply is not scored yet. The Key Result Score begins reporting once the benchmark is established.

FAQ

General Questions

Can I get a login for someone else on my team?

Yes, you can get a login by creating a support ticket.

Can I export any of this to share with my leadership or board?

You can export the weekly briefing page by clicking on the top right button to share with the leadership team. You can also provide login access to the leadership team by submitting a support ticket.

Why does a section or a page say BETA? Is the data reliable, or is it still being tested?

The data that is shown is reliable. We’re still refining the representation of the data in the Beta section and we’re still refining some of the metrics we want to show on those pages.

Who at Satisfi can I ask if I want to dig deeper into data?

Use the support chat in the bottom-right corner for any Console-related questions. We’re also happy to set up a call with our leadership team to walk you through the Console and answer any questions. Reach out to your Account Manager for more details.

Data

What counts as a "conversation" versus a "message"?

A conversation is the full exchange, start to finish, not a single message. For example, a guest asking about parking, then gate openings, then bag policy is really one task ("get into the building on time") spread across several messages. The console evaluates the whole conversation, because the task is only identifiable at that level. A single message can look fine in isolation and still be part of a conversation that never actually resolved the guest's real need.

Is this data specific to my organization only, or does it include benchmark data mixed in anywhere?

Your own conversation and completion data is yours alone. The goal/benchmark you're scored against, however, is explicitly built from "historical results across clients running that same campaign (clients like you)", so the target you're compared to is cross-client, even though your underlying counts are not.

What does "excluded as non-actionable" mean, and what kind of conversations get excluded?

A non-actionable conversation is one with no question in it: nothing the agent could act on. They are about 2% of conversations, and most are sessions where nobody typed anything. A session where nobody asked anything cannot be resolved or unresolved.

Evals

Why does a campaign with a lower completion rate show as "Success" while another with a higher completion rate shows "Partial Progress"?

0.7 (Success) means "on par with a strong peer campaign of the same type," and the completion rate needed to hit that bar is different for every campaign because it's adjusted for task difficulty. A hard campaign (e.g., ticket sales) reaches its 0.7 "Success" mark at a lower raw completion rate than an easier campaign does. So completion percentages are not directly comparable across different campaigns, only the final score is.

What does "Volume Pending" actually mean?

It means this specific client's campaign has too little of its own conversation volume yet (e.g., <20 conversations attempted in a given period) to calculate anything.

Is the cohort median recalculated over time, or fixed for a given period?

The cohort median is a rolling benchmark that shows the median completion rate for a specific campaign among similar clients running that campaign during a given time period. To change the time period, you need to go to Campaign Overview and apply the date range there.

Resolutions

What does "Partially Resolved" actually cover?

Partially Resolved means: The need was within the agent's remit and the agent moved the guest specifically toward it, but stopped short of the completeness or specificity the guest wanted.

If a guest gets a good answer but doesn't take the intended action (like buying a ticket), does that count as resolved?

Yes. Resolution is about whether the guest got what they came for, at the level they needed, within what the agent is built to handle, not whether a specific business action followed.

How is resolution scored, is a human reviewing conversations, or is it automated?

Resolution is scored automatically. Every conversation, not each individual message, is evaluated as a whole against a defined standard: did the guest get what they came for, at the level they needed, within what the agent is built to handle. That's what produces the Resolved, Partial, and Unresolved verdicts you see in the console.