Building Quality Scorecards in Call: Items, Weights and Review
Design a fair quality scorecard for phone calls, then use Call to score, coach and calibrate: weights, critical items, automatic scoring and review queues.
Ask three managers what a good call sounds like and you will get three different answers. That is the problem a scorecard solves. It agrees, in advance and in writing, what you are looking for, so that a call is judged by the same standard whoever listens. Done well, it is fair to agents and useful to the business. Done badly, it becomes a box-ticking exercise that rewards the wrong things.

This guide covers how to design a scorecard, how to use one in Call, and how to keep scoring consistent. It builds on Call Quality Management (QA) Basics and How Many Calls Should You Review?.
Designing a scorecard
Start with outcomes, not behaviours. What do you want every caller to experience? Work backwards to the few behaviours that make that likely. Fewer, clearer items beat a long list.
Typical groups of items
- OpeningGreeted clearly, named the business, identified the need.
- UnderstandingAsked useful questions, listened, confirmed what the caller wanted.
- AccuracyGave correct information, followed process, recorded what was needed.
- ClosingConfirmed next steps, thanked the caller, ended politely.
Each item should be a question with a clear yes or no answer. "Did the agent confirm the caller's request before acting?" is checkable. "Was the agent professional?" is not, unless you define it. If two reviewers would argue about an item, rewrite it.
Weights and critical items
Not every item matters equally. A weight shows how much each item counts. Critical items are the few things that must never be missed, such as a required disclosure or handling a caller's request not to be called. In Call, a failed critical item sets the score to zero regardless of the other items. Use critical items sparingly. If more than a handful are critical, the score stops being informative.
| Item type | Example | Treatment |
|---|---|---|
| Critical | Required disclosure given | Fail zeroes the score |
| Heavily weighted | Resolved the caller's question | Large share of the score |
| Lightly weighted | Used the caller's name | Small share of the score |
| Not applicable | Transfer handled well, on a call with no transfer | Marked N/A and left out |
Scoring a call in Call
Open AI agents, then Review and coaching. The page is titled Quality and Coaching. The review queue lists calls waiting for a review, longest and most negative first, up to twelve at a time. It needs an active scorecard in your workspace.
Scoring one call
- Choose the callIn Review queue, find the call and press Score.
- Mark each itemChoose Pass, Fail or N/A. Items marked critical zero the score if they fail.
- Watch the scoreIt shows at the top right and turns green at or above the pass mark.
- Save the evaluationWrite what the agent should do differently in Comment, then press Save evaluation.
The comment is the most valuable part. A score says how a call went; a comment says what to do next. Be specific and kind: name the moment, say what you would try instead and, where possible, mention something that went well. Agents see their own results under My evaluations.
Automatic scoring, with care
Some items can be checked by looking for words. For those, Call offers automatic scoring: for an item, type the words that prove it in Words that evidence it, separated by commas, then choose Expect: said or not said. Press Save rules, and Score the last 7 days to apply them to recent calls. Japanese and English can be mixed.
- Use it only for items you can express as words, such as a required statement.
- Keep rules narrow and true. A blank item is better than a guessed one.
- Spot-check the results against a human review before you trust them.
- Never use it for tone, empathy or judgement.
How many calls, and which ones
You do not need to score every call. A fair sample, spread across agents, shifts and call types, tells you more than a large sample taken after complaints. Mix random calls with a few chosen on purpose: long calls, short calls, calls with a poor customer rating and calls that ended in a transfer. The review queue helps by listing longest and most negative calls first, but add a random element so that good calls get noticed too.
Tell agents how calls are chosen. When people believe sampling is fair, they accept the results; when they suspect they are being singled out, they discount them.
Keeping reviewers consistent
Calibration means having more than one reviewer score the same call and comparing the results. Call lists calls scored by more than one reviewer, with the spread between their scores. A spread above twenty points means reviewers read the scorecard differently, and you should agree on it together. Run a short session each month: listen, score alone, compare and discuss. See Running QA Calibration Sessions.
Calibration also finds scorecard problems. If reviewers keep disagreeing on one item, the item is vague. Rewrite it, add an example, or drop it.
Using the results
Scores matter only if they change something. The page shows the average score against the pass mark, evaluations that failed, agent scores over the last thirty days, and where the team is weakest, which ranks items by failure rate. The last list is gold for training: if one item fails often across the team, it is a process or training issue, not an individual one.
Using scores well or badly
| Well | Badly | |
|---|---|---|
| Purpose | Find what to coach and fix | Rank or punish individuals |
| Sample | Enough calls, from all agents, chosen fairly | A few calls after a complaint |
| Feedback | Specific, soon and private | A number in an email weeks later |
| Scorecard | Reviewed regularly with agents | Fixed and unexplained |
You can book coaching sessions from the same page and compare an agent's average score before and after each held session, to see whether coaching moved the score. See A Simple Framework for Coaching Agents on Calls and Fair Performance Reviews for Phone Agents.
A hypothetical scoring example
Suppose a small repair business reviews a call where a customer asks about a delayed part. The reviewer marks the opening as Pass, understanding as Pass, accuracy as Fail because the agent quoted an out-of-date delivery window, and closing as Pass. The call has no transfer, so that item is N/A and is left out of the score. Because accuracy carries a heavy weight, the score lands below the pass mark even though three of four groups went well.
The comment matters more than the number. A useful one reads: "You were warm and clear, and the customer knew what to expect. The delivery window you gave is the old one; check the status page before quoting, and say when you will confirm if it is not available." That sentence names what went well, what to fix and what to do instead. A bare "Fail: accuracy" teaches nothing.
Over a month, the same reviewer might notice that many failed accuracy items involve delivery windows. That is no longer an agent problem; it is a knowledge problem, and the right fix is to update the knowledge base so that every agent can find the right answer quickly. Scorecards earn their keep by revealing such patterns.
Pitfalls to avoid
- Too many items. Reviewers tire and scores drift.
- Scoring what is easy to measure rather than what matters to callers.
- Using scores as the only input to performance reviews.
- Never telling agents what is on the scorecard.
- Ignoring the customer's own feedback; compare scores with what customers said.
Scorecard launch checklist
- Fewer than a dozen items, each answerable with yes, no or not applicable.
- Weights agreed, and a small number of critical items.
- Agents have seen the scorecard and understand it.
- A pass mark that is realistic.
- A calibration session booked within the first month.
- A date to review the scorecard itself.
Frequently asked questions
What is a quality scorecard?
It is a short list of questions used to judge a call consistently. Each item is marked Pass, Fail or N/A and carries a weight showing how much it counts.
What is a critical item?
A critical item is one that must never be missed. In Call, if a critical item fails, the score is zero regardless of the other items.
Can Call score calls automatically?
For items that can be expressed as words said or not said, yes. You type the words that evidence the item and choose said or not said. Other items need a human reviewer.
How do I know whether reviewers agree?
Calibration lists calls scored by more than one reviewer with the spread between their scores. A large spread means the scorecard is read differently.
Who can score calls?
Admins and managers score from the review queue. Agents see their own evaluations and coaching sessions.
Does a high score mean a happy customer?
Not necessarily. Compare scores with what customers said in surveys and calls, and adjust the scorecard if they disagree.